2010 Aug 18 - Wed
Cygwin, Eclipse and Subversion Installation Notes
I have written several articles about Eclipse (the code editing UI) and it's
integration with subversion. This is an update of a few things to watch out for
with Eclipse, the Helios release. I do development on Linux as well as on Windows. In
this case my primary machine is a Windows machine running VMWare with several
guest Linux systems.
For the Linux systems with a GUI, I've used Cygwin to provide a mechanism of running the
Linux interfaces on my Windows interface. I have tried the VMWare Unity mechanism, but
on my multi-monitor system, it appears clunky and buggy.
When installing Cygwin, the key library to install is the 'xinit' library. This loads
all other necessary X11 libraries. Also include mintty in the Shells category for an
improved console experience.
As a side note for regular terminal operations in Cygwin, the following can be used with
mintty. Start 'ssh-agent mintty'.
mintty is explained at
http://code.google.com/p/mintty/.
Then use ssh-add to add a private key. The public key can be added to the ~/.ssh/authorized_key
files on the destination machines.
Anyway, for getting the GUI experience, use startxwin to start an xwindow terminal window.
Connect to the destination computer with 'ssh -l username -Y ipaddress'. At that point,
I run eclipse with '/usr/sbin/eclip[se/eclipse &'. The '&' forks the process and allows further
operations in the terminal window.
I've got ahead of myself here. To get eclipse installed, I downloaded the binaries from
eclipse.org, expanded them to a directory called eclipse. I then moved the directory to
/usr/sbin. Eclipse can then be started with '/usr/sbin/eclipse/eclipse'.
For version control, the Polaris subversion client is listed as a standard item
in the Collaboration items in the Eclipse New Software. After trying that, I wasn't very
pleased with the experience. It is not well integrated.
Instead, I removed the Polaris Subversive client and installed the Tigris.org
Subclipose Client.
The integration into Eclipse is much better. I used the SVNKit (Pure Java) connector
so as to obtain the svn+ssh://... tunnelling capability with a private key based login.
2010 Jan 29 - Fri
Migrating Bacula 2.x on Debian Etch to 3.x on Squeeze
Debian Etch, which is the current release, has the Bacula 2.x packages.
I needed to upgrade to the Bacula 3.x packages, which are located in debian/testing,
also known as the forthcoming Debian Squeeze release. In addition, since PostgreSQL 8.3 is
packaged in Etch, and PostgreSQL 8.4 is packaged in Squeeze/testing, a database
migration is also required.
I had attempted updating my sources.list file to testing and then running the
apt-get dist-upgrade process. This broke some dependences, and also broke on a
udev migration. I guess testing has more testing to do on the distribution
upgrade process.
In the end, I built a new Bacula service on a freshly installed Debian testing server.
The special consideration for this configuration is that it needs to handle backing
up servers across a WAN. As such, backups may travel through one or more firewalls. Through
such a configuration, it is very difficult to get the firewall ports opened for the various
necessary Bacula service ports. The better way to tackle this is through the use of
ssh' port local and remote port forwarding capability. Port 22 is becomes the only necessary
port to open on a firewall. The ssh-tunnel.sh script helps make this happen.
To build the server, when it came to package selection, I unselected all packages, and then chose
just the database package which installed PostgreSQL.
After the basic server finished installing and rebooted, I manually installed the following packages:
apt-get install bacula-common-pgsql
apt-get install bacula-client
apt-get install bacula-director-common
apt-get install bacula-director-pgsql
apt-get install bacula-sd-pgsql
apt-get install bacula-server
If starting with a new database, then dbconfig-common can be used. If migrating an older database, don't use
dbconfig-common, and use the manual methods I'll describe further on. There is further documentation in
/usr/share/doc/bacula-director-pgsql.
During installation of the bacula packages, a new user of 'bacula' is created, as well as a group called 'tape'.
The 'bacula' user has a home directory of /var/lib/bacula.
Into that directory, create a .ssh directory for any authorized_keys and known_hosts required. I also created a keys
subdirectory to hold the public/private keys for ssh'ing into other servers for processing backups. I called the
two files 'bacula' and 'bacula.pub'. These will be referenced in my customized ssh-tunnel.sh script.
Run
dpkg-reconfigure exim4-config
to reconfigure the mail system to allow outbound mail delivery.
My backups go onto a remote file share. I created an entry in /etc/fstab along the lines of:
10.1.1.1:/bu /mnt/nas nfs rw,hard,intr,async,nodev,nosuid 0 0
Ensure that the NFS client is installed through:
apt-get install nfs-common
In /etc/postgresql/8.4/main/pg_hba.conf, I have lines along:
host bacula bacula 127.0.0.1/32 trust
host bacula sysadmin 127.0.0.1/32 trust
local bacula bacula trust
local bacula sysadmin trust
As an aside, a useful command to find out database information is through the use of:
psql -l
When migrating the database to 8.4, there are modifcations to the pg_dump command required (which are
required to prevent import errors along the lines of 'ERROR: invalid byte sequence for encoding "UTF8"',
basically resolving the UTF-8 to SQL_ASCII issues in Bacula):
pg_dump -E SQL_ASCII -U bacula bacula > /var/lib/bacula/bacula.sql
On the new server, use the following to import the database:
dropdb bacula
su - postgres
psql
create role bacula;
create database bacula owner=bacula encoding='SQL_ASCII' template=template0;
/q
psql bacula </var/lib/bacula/bacula.sql
Basic instructions for updating the database from Bacula table version 10 to Bacula
table version 11 is found in /usr/share/bacula-director/update_postgresql_tables:
BEGIN;
ALTER TABLE file ALTER fileid TYPE bigint ;
ALTER TABLE basefiles ALTER fileid TYPE bigint;
ALTER TABLE job ADD COLUMN readbytes bigint default 0;
ALTER TABLE media ADD COLUMN ActionOnPurge smallint default 0;
ALTER TABLE pool ADD COLUMN ActionOnPurge smallint default 0;
-- Create a table like Job for long term statistics
CREATE TABLE JobHisto (LIKE Job);
CREATE INDEX jobhisto_idx ON JobHisto ( starttime );
UPDATE Version SET VersionId=11;
COMMIT;
Once the configuration files for the director, storage manager, and file manager are ready, bacula
can be managed through 'bconsole'.
My modified /etc/bacula/scripts/ssh-tunnel.sh looks like:
#!/bin/sh
# script for creating / stopping a ssh-tunnel to a backupclient
# Stephan Holl sholl@gmx.net
# Modified by Joshua Kugler joshua.kugler@uaf.edu
# Modified by Ray Burkholder ray@oneunified.net
#
#
# variables
USER=bacula
CLIENTADDR=$2
# CLIENTPORT is local end
CLIENTPORT=$3
#LOCAL=your.backup.server.host.name
# local is a local address and uses ssh's remote/local port forwarding
LOCAL=127.0.0.1
SSH=/usr/bin/ssh
SSHOPTIONS=-vfnCN2
LOG1=/var/lib/bacula/log1.log
LOG2=/var/lib/bacula/log2.log
#LOG1=/dev/null
#LOG2=/dev/null
# location of the public/private keys used with ssh to gain access to remote servers
KEY=/etc/bacula/keys/bacula
case "$1" in
start)
# create ssh-tunnel
echo "Starting SSH-tunnel to $CLIENTADDR..."
$SSH $SSHOPTIONS -o PreferredAuthentications=publickey -i $KEY -l $USER \
-R 9101:$LOCAL:9101 -R 9103:$LOCAL:9103 -L $CLIENTPORT:$LOCAL:9102 $CLIENTADDR \
>> $LOG1 2>> $LOG2
exit $?
;;
stop)
# remove tunnel
echo "Stopping SSH-tunnel to $CLIENTADDR..."
# find PID killem
PID=`ps ax | grep "$SSH $SSHOPTIONS -o PreferredAuthentications=publickey -i $KEY" \
| grep "$CLIENTADDR" | awk '{ print $1 }'`
kill $PID
exit $?
;;
*)
# usage:
echo " "
echo " Start SSH-tunnel to client-host"
echo " to bacula-director and storage-daemon"
echo " "
echo " USAGE:"
echo " ssh-tunnel.sh {start|stop} client.fqdn"
echo ""
exit 1
;;
esac
The links I used for getting started with ssh-tunnels are found at:
In /etc/hosts file, 127.0.0.1 should be the only line referring to the local server. The exteral port
ip address should be commented out:
127.0.0.1 localhost bu.example.com bu
#10.10.10.1 bu.example.com bu
In the bacula-dir.conf configuration file, a typical client configuration will look similar to:
Client {
Name = mail-fd
Address = 127.0.0.1
FDPort = 9130 # specific port for this client, allows multiple simultaneous backups
Catalog = MyCatalog
Password = "xxxxxx" # password for FileDaemon
File Retention = 120 days
Job Retention = 4 months
AutoPrune = yes # Prune expired Jobs/Files
}
The special characteristic of the above configuration is the use of a unique port number for FDPort.
Each client in the bacula-dir.conf should have a unique port number. This allows bacula to
tunnel via ssh to remote clients and redirect them to the storage manager on the
local server.
The definition of the storage device in bacula-dir.conf will have Address=127.0.0.1 and SDPort=9103.
The job description for each client should have something similar to:
Job {
Name = "mail-fd"
Client = mail-fd
JobDefs = "DefaultJob"
FileSet = "FileSet_mail"
Storage = storageSshClients
Write Bootstrap = "/var/lib/bacula/mail.bsr"
Priority = 12
Run Before Job = "/etc/bacula/scripts/ssh-tunnel.sh start mail.example.com 9130"
Run After Job = "/etc/bacula/scripts/ssh-tunnel.sh stop mail.example.com 9130"
}
When using Bacula in console mode, a useful command to find out the meaning of the
backup status codes:
*sqlquery
Entering SQL query mode.
Terminate each query with a semicolon.
Terminate query mode with a blank line.
Enter SQL query: select * from status;
+-----------+---------------------------------+
| jobstatus | jobstatuslong |
+-----------+---------------------------------+
| C | Created, not yet running |
| R | Running |
| B | Blocked |
| T | Completed successfully |
| E | Terminated with errors |
| e | Non-fatal error |
| f | Fatal error |
| D | Verify found differences |
| A | Canceled by user |
| F | Waiting for Client |
| S | Waiting for Storage daemon |
| m | Waiting for new media |
| M | Waiting for media mount |
| s | Waiting for storage resource |
| j | Waiting for job resource |
| c | Waiting for client resource |
| d | Waiting on maximum jobs |
| t | Waiting on start time |
| p | Waiting on higher priority jobs |
+-----------+---------------------------------+
Enter SQL query:
End query mode.
For the bacula entry in /etc/passwd, change /bin/false to be /bin/sh.
For each server to which will be connected via ssh, within the
context of the bacula user, use the following command to update ~/.ssh/known_hosts:
ssh -l bacula -i /etc/bacula/keys/bacula -v server.example.com
2010 Jan 12 - Tue
Creating a Cold Standby FreeBSD Machine
I have a FreeBSD machine for which I don't have original installation files, thus
rebuilding the machine and reinstalling software from the ground up on new hardware
could be a problem.
I have a few of avenues getting a suitable secondary machine up and running:
I didn't use the dump/restore commands as I don't have easy access to enough space for temporary
storage so I brute forced the scenario by copying data directly from the source machine partition
to the destination machine partition.
I was able to obtain almost identical hardware for the second machine so as not to have any
hardware compatibility issues. Drive sizes and memory are larger on the second machine.
FreeBSD has a FixIt feature, otherwise known as a live boot, which allows a machine to be booted and
analyzed from a cd. I could have used a recent version of FreeBSD to do this task, but instead,
I chose a version which was of the same major.minor version as that of the running system at the
old release archives.
Once booted off the cd, a number of tasks can be completed:
- create partitions on the disks
- label the partitions
- obtain network connectivity
- copy partitions from source machine to destination machine
- depending upon services started, protect the destination machine from the network
- reboot the destination machine
- log in and verify successful operation
- possibly change driver, service, partition, and network settings
On the source machine, with root privileges, use 'df' and 'df -h' to determine partition
labels and sizes. 'sysinstall' may need to be run to get a listing of all partitions and their sizes,
particularily for the swap partition. The command 'swapctl -l -k -s' will detail the swap device.
On a SCSI based system, drive devices are named in fashion to /dev/da0s1a, where
da0 is drive 0, s1 is slice 1, and a is the first labelled partition. a is typically / (the boot partition),
b is typically swap, c is a hidden device for accessing the full drive, and d and
subsequent drives are partition mappings. On this system I have d as /var,
e as /tmp, and f as /usr.
On the destination computer, I selected the Fixit menu item, then the 'CDROM/DVD' option,
which takes me to a prompt.
I then used 'sysinstall' to gain access to various utilities. The first one I use is FDisk
to allocate all space on the drive to FreeBSD and use 'w' to write the changes to disk.
On exiting, I select the option to 'Install the FreeBSD Boot Manager' onto the drive.
I then use the Label editor to create partitions. They are created in order of a, b, d, e, and f. Note
that c is skipped as part of the process. Write the changes with 'w' and quit.
Then use the Networking menu item, then the Interfaces menu item to set ip addresses for the
interface.
Upon exiting back to the prompt, an 'ipconfig' command should show the ip address as being set.
After confirming network access is available, partitions can be copied over:
- ssh source_ip "cat /dev/da0s1a" | cat > /dev/da0s1a
- ssh source_ip "cat /dev/da0s1b" | cat > /dev/da0s1b
- ssh source_ip "cat /dev/da0s1d" | cat > /dev/da0s1d
- ssh source_ip "cat /dev/da0s1e" | cat > /dev/da0s1e
- ssh source_ip "cat /dev/da0s1f" | cat > /dev/da0s1f
Once things are copied over, disconnect the network interfaces, exit the menus, and
restart the machine. The machine should boot as though it was the original. Watch
the boot messages carefully for any errors.
The utility of the above operation improves if a filesystem snapshot capability is available
incorporated, then a guaranteed 'instant in time' view is available.
2009 Oct 14 - Wed
Boost BJam Updated
With the version 1.40 of Boost, library names are decorated differently. To keep the old style library
decorations and naming style, the option "--layout=tagged" should work. So from my
2008/10/10 Boost Build Article, my typical command line should be:
bjam --layout=versioned --toolset=msvc-9.0 variant=debug threading=multi link=static runtime-link=static stage
2009 Sep 29 - Tue
Upgrade to KDE4: Black Screen, Obsidian Cursor
Today when upgrading my Debian Lenny/KDE to the latest version, I started having problems with KDE.
On my first upgrade, I did a simple 'apt-get update', 'apt-get upgrade' sequence. A bunch of packages were held back.
The end result was that I could log in to KDE, and could see a desktop, but I had no menu interface.
Considering that there were a bunch of packages being help back, I did a
'apt-get update', 'apt-get dist-upgrade' sequence. Upon logging into the KDE shell, all I saw was a black screen
and a shiny obsidian cursor.
It looks like the transition from KDE 3.5 to KDE 4.0 is not seamless in this Debian (Lenny) point release.
However, that isn't quite correct. In my /etc/apt/sources.list file I do have entries for testing and experimental.
So..., I may now be downloading testing or experimental releases.
In any case, the resolution to the problem appears to be to drop into the console and run one of these three commands:
'apt-get install kde-standard', 'apt-get install kde-minimal', or 'apt-get install kde-full'.
2009 Sep 22 - Tue
Updating WebGUI
WebGUI's Update Page has links to the various updates.
Upgrade information can be found at
Upgrading WebGUI.
To view the current upgrade history:
cd /data/WebGUI/sbin
perl upgrade.pl --history --doit
perl testEnvironment.pl
Stop Spectre:
cd /data/WebGUI/sbin
perl spectre.pl --shutdown
Make a backup of the files in /data/WebGUI/etc. The originals will be over-written, but the customized ones
should be ok after the upgrade.
Decompress the new archive over the old files (with the current version as of this writing):
cd /data
wget http://update.webgui.org/7.x.x/webgui-7.7.20-stable.tar.gz
tar -zxvf webgui-7.7.20-stable.tar.gz
Read the WebGUI/docs/gotcha.txt file.
Read the WebGUI/docs/changelog/7.x.x.txt to check out the latest changes.
Restart apache with '/etc/init.d/apache2 restart'.
Run the upgrade:
cd /data/WebGUI/sbin
perl upgrade.pl
perl upgrade.pl --doit --backupDir /data/bu/wg
Run testEnvironment.pl:
cd /data/WebGUI/sbin
perl testEnvironment.pl
Start Spectre:
cd /data/WebGUI/sbin
perl spectre.pl --daemon
Restart apache with '/etc/init.d/apache2 restart'.
2009 Aug 31 - Mon
Massaging CommunigatePro MIB For Cricket
CommuniGate Pro's web interface has a page which shows SNMP originated statistics.
On that same page, there is link for downloading the MIB file which defines the
values shown on that page.
Rather than going through all 100 or so MIB entries, I wrote an
AWK script to process the CommuniGate Pro MIB file
into a Defaults file which can read by Cricket, the SNMP collector/grapher.
After running the Defaults file with a CommuniGate Pro server for a while, I found that some of the
groupings didn't work very well by default. Several values are serveral orders of magnitude different from
other values in the same group. I did some manually editing to get values of like magnitude into their
own groups. Here is the resulting
Defaults.communigate file. I've left colouring
to the Cricket defaults, but at least it gets the values into my monitoring solution.
2009 Aug 19 - Wed
Debian Dpkg Install
From the Debian Security Announce List, a little short-cut for installing .deb packages:
wget url
will fetch the file for you
dpkg -i file.deb
will install the referenced file.
IPTables Mangle DSCP
From the Nanog mailing list, a way to force QOS packet marking on outbound packets might look like:
# iptables -t mangle -I OUTPUT -p tcp --sport 80 -j DSCP --set-dscp 0x1a
2009 Jul 31 - Fri
Installing OpenLDAP on Debian Lenny
Here are a few basic apt-get commands for the OpenLDAP installation. I have to look
into how TLS is actually implemented and configured.
apt-get install libsasl2-2 libgnutl26
apt-get install ldap-utils libsasl2-modules-ldap
apt-get install slapd libldap-2.4-2
Installing Asterisk 1.6.2.0 beta3 on Debian Lenny 5.0.2
Debian package manager has the Asterisk v1.4 flavour as a package,
but I wanted the latest to try out. Here is the work flow to get the basics in place:
Here are some pre-requisites to install. I havn't figured out the 'lua' bit yet:
apt-get install build-essential
apt-get install openssl
apt-get install libssl-dev
apt-get install libldap2-dev
apt-get install libncurses5-dev
apt-get install festival-dev festival
apt-get install curl libcurl4-openssl-dev
apt-get install lua5.1
apt-get install uw-mailutils
apt-get install libgsm1
apt-get install libiksemel3
apt-get install libogg0
apt-get install libspeex1 libspeexdsp1
apt-get install libtonezone1
apt-get install libvorbis0a libvorbisenc2
apt-get install doxygen
apt-get install postgresql-server-dev-8.3 postgresql-client-8.3
apt-get install libnewt-dev
apt-get install linux-headers-2.6.26-2-686
apt-get install libogg-dev
apt-get install libvorbis-dev
apt-get install liblua5.1-posix-dev
apt-get install libgsm1-dev
The basic hardware layer for the kernel is next. This includes dummy timers for
systems without additional telephony hardware.
d /usr/src
wget http://downloads.asterisk.org/pub/telephony/dahdi-linux/dahdi-linux-2.2.0.2.tar.gz
tar -zxvf dahdi-linux-2.2.0.2.tar.gz
cd dahdi-linux-2.2.0.2
make
make install
User space Dahdi tools are then built:
d /usr/src
wget http://downloads.asterisk.org/pub/telephony/dahdi-tools/dahdi-tools-2.2.0.tar.gz
tar -zxvf dahdi-tools-2.2.0.tar.gz
cd dahdi-tools-2.2.0
./configure \
--sysconfdir=/etc/ \
--libdir=/usr/lib \
--localstatedir=/var/local \
--datarootdir=/usr/share \
--includedir=/usr/include
make menuselect
make
make install
make config
This portion installs a recent beta releaes of the Asterisk engine:
cd /usr/src
wget http://downloads.asterisk.org/pub/telephony/asterisk/asterisk-1.6.2.0-beta3.tar.gz
tar -zxvf asterisk-1.6.2.0-beta3.tar.gz
cd asterisk-1.6.2.0-beta3
./configure \
--sysconfdir=/etc/ \
--libdir=/usr/lib \
--localstatedir=/var/local \
--datarootdir=/usr/share \
--includedir=/usr/include \
--disable-xmldoc
Ensure you've got all the various libraries, modules, bits and pieces attached:
make menuselect
If you are installing a system from scratch, the run all these. If you already have configuration files, skip the 'make samples'.
make
make install
make samples
make progdocs
If you are using PostgreSQL, build the database tables with:
su - postgres
psql template1
> create database asterisk;
> quite;
psql asterisk < /usr/src/asterisk-1.6.2.0-beta3/contrib/scripts/realtime_pgsql.sql
Then edit /etc/asterisk/res_pgsql.conf to add connection information. Other files you may need to edit include:
sip.conf
dahdi-channels.conf
cdr_manager.conf
cdr_pgsql.conf
cdr.conf
extensions.conf
iax.conf
Get things started with:
/etc/init.d/dahdi start
safe_asterisk
2009 Jul 24 - Fri
Debian Lenny with Sendmail, Dovecot, MailScanner, SpamAssassin: Part 6
I've spent the last articles writing about getting an open source email server up and running. So far so good.
My email logs show that a tremendous amount of spam is being blocked. One begins to wonder if there any real email
remaining any more.
During the building of this server, a number of web sites provided useful information for troubleshooting and for configuration.
I'm listing them here for reference before I close them out.
- http://www.linuxweblog.com/blogs/sandip/20080206/sendmail-accessdb-example:
provided useful explanations and examples of the interactions between the access database, the blacklist_recipients feature, the value part
of the map, and how to use the delay_checks feature for negative or positive exception handling.
- ZEN Return Codes: The 127.0.0.x return codes. Basically,
127.0.0.2 is for direct UBE sources, spam services, and ROKSO spammers;
127.0.0.4-8 are for illegal 3rd party exploits, including proxies, worms and trojan exploits; and
127.0.0.10-11 are for non-MTA IP address ranges set by outbound mail policy
- : a good description of the sendmail.mc file, it's options,
features, and ordering. It goes into some detail about special considerations of the VIRTUSER_DOMAIN_FILE. It also goes into uses
and configuration examples of the access file.
- SPF Setup Wizard: I'm not sure if the Sender Policy Framework (SPF)
is much in use, but this web size provides a wizard for it's DNS record creation.
- Sendmail Readme for Configuration: The original source
for configuring Sendmail.
- Linux Home Server HowTo: Sendmail: another article on
how to build a full-fledged email server. One key command for ensuring you havn't configured an open relay through a series of 19 tests:
'telnet relay-test.mail-abuse.org'. When run from the mail server, the server at relay-test.mail-abuse.org
will connect back to your server on port 25 and run the series of tests.
- sendmail.mc: this is the best organized and best
documented sample sendmail.mc file I've ever seen.
- xabean's sendmail.mc: example sendmail.mc with native macros and a milter, with
hotlinks to relevant sections in the
Sendmail Readme file.
- Hugo van der Kooij's sendmail.mc: looks like
he no longer runs sendmail, but here is his sendmail.mc with some native macros.
In some follow-up, I came across MailWatch,
which is a web-based front-end to MailScanner written in PHP, MySQL and JpGraph and
is available for free under the terms of the GNU Public License.
|
