2010 Jan 29 - Fri
Migrating Bacula 2.x on Debian Etch to 3.x on Squeeze
Debian Etch, which is the current release, has the Bacula 2.x packages.
I needed to upgrade to the Bacula 3.x packages, which are located in debian/testing,
also known as the forthcoming Debian Squeeze release. In addition, since PostgreSQL 8.3 is
packaged in Etch, and PostgreSQL 8.4 is packaged in Squeeze/testing, a database
migration is also required.
I had attempted updating my sources.list file to testing and then running the
apt-get dist-upgrade process. This broke some dependences, and also broke on a
udev migration. I guess testing has more testing to do on the distribution
upgrade process.
In the end, I built a new Bacula service on a freshly installed Debian testing server.
The special consideration for this configuration is that it needs to handle backing
up servers across a WAN. As such, backups may travel through one or more firewalls. Through
such a configuration, it is very difficult to get the firewall ports opened for the various
necessary Bacula service ports. The better way to tackle this is through the use of
ssh' port local and remote port forwarding capability. Port 22 is becomes the only necessary
port to open on a firewall. The ssh-tunnel.sh script helps make this happen.
To build the server, when it came to package selection, I unselected all packages, and then chose
just the database package which installed PostgreSQL.
After the basic server finished installing and rebooted, I manually installed the following packages:
apt-get install bacula-common-pgsql
apt-get install bacula-client
apt-get install bacula-director-common
apt-get install bacula-director-pgsql
apt-get install bacula-sd-pgsql
apt-get install bacula-server
If starting with a new database, then dbconfig-common can be used. If migrating an older database, don't use
dbconfig-common, and use the manual methods I'll describe further on. There is further documentation in
/usr/share/doc/bacula-director-pgsql.
During installation of the bacula packages, a new user of 'bacula' is created, as well as a group called 'tape'.
The 'bacula' user has a home directory of /var/lib/bacula.
Into that directory, create a .ssh directory for any authorized_keys and known_hosts required. I also created a keys
subdirectory to hold the public/private keys for ssh'ing into other servers for processing backups. I called the
two files 'bacula' and 'bacula.pub'. These will be referenced in my customized ssh-tunnel.sh script.
Run
dpkg-reconfigure exim4-config
to reconfigure the mail system to allow outbound mail delivery.
My backups go onto a remote file share. I created an entry in /etc/fstab along the lines of:
10.1.1.1:/bu /mnt/nas nfs rw,hard,intr,async,nodev,nosuid 0 0
Ensure that the NFS client is installed through:
apt-get install nfs-common
In /etc/postgresql/8.4/main/pg_hba.conf, I have lines along:
host bacula bacula 127.0.0.1/32 trust
host bacula sysadmin 127.0.0.1/32 trust
local bacula bacula trust
local bacula sysadmin trust
As an aside, a useful command to find out database information is through the use of:
psql -l
When migrating the database to 8.4, there are modifcations to the pg_dump command required (which are
required to prevent import errors along the lines of 'ERROR: invalid byte sequence for encoding "UTF8"',
basically resolving the UTF-8 to SQL_ASCII issues in Bacula):
pg_dump -E SQL_ASCII -U bacula bacula > /var/lib/bacula/bacula.sql
On the new server, use the following to import the database:
dropdb bacula
su - postgres
psql
create role bacula;
create database bacula owner=bacula encoding='SQL_ASCII' template=template0;
/q
psql bacula </var/lib/bacula/bacula.sql
Basic instructions for updating the database from Bacula table version 10 to Bacula
table version 11 is found in /usr/share/bacula-director/update_postgresql_tables:
BEGIN;
ALTER TABLE file ALTER fileid TYPE bigint ;
ALTER TABLE basefiles ALTER fileid TYPE bigint;
ALTER TABLE job ADD COLUMN readbytes bigint default 0;
ALTER TABLE media ADD COLUMN ActionOnPurge smallint default 0;
ALTER TABLE pool ADD COLUMN ActionOnPurge smallint default 0;
-- Create a table like Job for long term statistics
CREATE TABLE JobHisto (LIKE Job);
CREATE INDEX jobhisto_idx ON JobHisto ( starttime );
UPDATE Version SET VersionId=11;
COMMIT;
Once the configuration files for the director, storage manager, and file manager are ready, bacula
can be managed through 'bconsole'.
My modified /etc/bacula/scripts/ssh-tunnel.sh looks like:
#!/bin/sh
# script for creating / stopping a ssh-tunnel to a backupclient
# Stephan Holl sholl@gmx.net
# Modified by Joshua Kugler joshua.kugler@uaf.edu
# Modified by Ray Burkholder ray@oneunified.net
#
#
# variables
USER=bacula
CLIENTADDR=$2
# CLIENTPORT is local end
CLIENTPORT=$3
#LOCAL=your.backup.server.host.name
# local is a local address and uses ssh's remote/local port forwarding
LOCAL=127.0.0.1
SSH=/usr/bin/ssh
SSHOPTIONS=-vfnCN2
LOG1=/var/lib/bacula/log1.log
LOG2=/var/lib/bacula/log2.log
#LOG1=/dev/null
#LOG2=/dev/null
# location of the public/private keys used with ssh to gain access to remote servers
KEY=/etc/bacula/keys/bacula
case "$1" in
start)
# create ssh-tunnel
echo "Starting SSH-tunnel to $CLIENTADDR..."
$SSH $SSHOPTIONS -o PreferredAuthentications=publickey -i $KEY -l $USER \
-R 9101:$LOCAL:9101 -R 9103:$LOCAL:9103 -L $CLIENTPORT:$LOCAL:9102 $CLIENTADDR \
>> $LOG1 2>> $LOG2
exit $?
;;
stop)
# remove tunnel
echo "Stopping SSH-tunnel to $CLIENTADDR..."
# find PID killem
PID=`ps ax | grep "$SSH $SSHOPTIONS -o PreferredAuthentications=publickey -i $KEY" \
| grep "$CLIENTADDR" | awk '{ print $1 }'`
kill $PID
exit $?
;;
*)
# usage:
echo " "
echo " Start SSH-tunnel to client-host"
echo " to bacula-director and storage-daemon"
echo " "
echo " USAGE:"
echo " ssh-tunnel.sh {start|stop} client.fqdn"
echo ""
exit 1
;;
esac
The links I used for getting started with ssh-tunnels are found at:
In /etc/hosts file, 127.0.0.1 should be the only line referring to the local server. The exteral port
ip address should be commented out:
127.0.0.1 localhost bu.example.com bu
#10.10.10.1 bu.example.com bu
In the bacula-dir.conf configuration file, a typical client configuration will look similar to:
Client {
Name = mail-fd
Address = 127.0.0.1
FDPort = 9130 # specific port for this client, allows multiple simultaneous backups
Catalog = MyCatalog
Password = "xxxxxx" # password for FileDaemon
File Retention = 120 days
Job Retention = 4 months
AutoPrune = yes # Prune expired Jobs/Files
}
The special characteristic of the above configuration is the use of a unique port number for FDPort.
Each client in the bacula-dir.conf should have a unique port number. This allows bacula to
tunnel via ssh to remote clients and redirect them to the storage manager on the
local server.
The definition of the storage device in bacula-dir.conf will have Address=127.0.0.1 and SDPort=9103.
The job description for each client should have something similar to:
Job {
Name = "mail-fd"
Client = mail-fd
JobDefs = "DefaultJob"
FileSet = "FileSet_mail"
Storage = storageSshClients
Write Bootstrap = "/var/lib/bacula/mail.bsr"
Priority = 12
Run Before Job = "/etc/bacula/scripts/ssh-tunnel.sh start mail.example.com 9130"
Run After Job = "/etc/bacula/scripts/ssh-tunnel.sh stop mail.example.com 9130"
}
When using Bacula in console mode, a useful command to find out the meaning of the
backup status codes:
*sqlquery
Entering SQL query mode.
Terminate each query with a semicolon.
Terminate query mode with a blank line.
Enter SQL query: select * from status;
+-----------+---------------------------------+
| jobstatus | jobstatuslong |
+-----------+---------------------------------+
| C | Created, not yet running |
| R | Running |
| B | Blocked |
| T | Completed successfully |
| E | Terminated with errors |
| e | Non-fatal error |
| f | Fatal error |
| D | Verify found differences |
| A | Canceled by user |
| F | Waiting for Client |
| S | Waiting for Storage daemon |
| m | Waiting for new media |
| M | Waiting for media mount |
| s | Waiting for storage resource |
| j | Waiting for job resource |
| c | Waiting for client resource |
| d | Waiting on maximum jobs |
| t | Waiting on start time |
| p | Waiting on higher priority jobs |
+-----------+---------------------------------+
Enter SQL query:
End query mode.
For the bacula entry in /etc/passwd, change /bin/false to be /bin/sh.
For each server to which will be connected via ssh, within the
context of the bacula user, use the following command to update ~/.ssh/known_hosts:
ssh -l bacula -i /etc/bacula/keys/bacula -v server.example.com
[/OpenSource/Debian]
permanent link
2010 Jan 17 - Sun
Option Trading Scenarios
Most option trading books discuss the mechanics and mathematics of option trading. Rarely
do they offer up any sort of in-the-trenches useful guidelines for doing what when and how.
Some recent reading has enlightened me on some useful option trading scenarios.
The first one provides the dual mechanism of obtaining income and optionally
paying for protection on an underlying which is already in one's portfolio.
The process involves selling a kind of straddle: selling an out of the money call
and an out of the money put. For example, with TSL, as of Friday's close, is at $49.50.
The Feb 55 Call is at $1.75 and the Feb 48 Put is at $3.25. Selling this results in $5.00 premium.
If TSL stays within the range of $48 and $55 till Feb 19, the full premium is kept. If
the price goes above $55, the underlying will probably be exercised to result in a $5.50 profit,
which is the premium plus the amount the underlying goes to get to the strike price. If
the price goes below $48, the Put will go into the money, most likely causing it to be
exercised, and you'll end up buying the underlying at the Put strike price, which may be a
good thing if you are expecting the price to rebound. On the protection side, the premium
earned on the put and call sale could be use to purchase twice the number of puts at the next
out-of-the money price, which in this case would be Feb 45 Put.
This provides protection on the original underlying plus the
shares gained when the buyer exercised the puts.
Another strategy requires maintenance of no underlying. For a stock that you
think will go up due to some upcoming good news, such as an earnings announcement,
one could sell an out of the money put and use the funds to buy and out of the money call.
By careful selection of put and call strikes, the money gained by selling the put may
fully cover the price of buying the call. The downside of this is if the price of the
underlying goes below the put strike price. You may end up owning the underlying in this case, but
at least it was obtained at a lower price.
[/Trading/Options]
permanent link
They Who Have The Money
Sometime this last week, someone made the observation of the fact that there was a
'hidden buyer' buying a large amount of treasuries during the weekly US Treasury auctions.
There was speculation it might be the Chinese operating through private proxy parties.
The US government has a problem with all the money they are spending. Some one has
to loan it to them. Hence the regular treasury auctions.
A couple figures came to mind. I believe I recall seeing that the Chinese have over
one trillion US dollars due to trade imbalance between the US and China. Another
figure has to do do with the fact that the US government has added over a trillion dollars
to their defict through recent spending.
If you put those two numbers together, and tie them together with the 'hidden buyer'
observation, perhaps it could be said that the Chinese are converting their US dollar
currency holdings into US Treasury holdings. That way, not only have they made money
through their exports, but they make additional revenue through the yields on the
treasuries purchased.
[/Trading/MarketNotes]
permanent link
Short Interest, Solar, Lithium
Many commenters have been discussing Peak Oil, with Peak Oil being the fact that we've found all the easy
supplies of oil, most of which has been pumped, and that our supplies, regardless of type, are dwindling... the
peak has come and gone. To be more specific, the supply of oil, of which we only have a finite supply, is decreasing,
while world demand for oil is increasing. The first day of Economics 100 teaches us the supply / demand curve and the
effect on pricing. Oil prices can only go up, and up until the last dropped is supplied and consumed.
In the meantime, alternate sources of energy must be found to supply the world's unending requirement. For
sourcing some fraction of our energy needs, some are turning to the nearly non-ending supply of sunlight. One
way of converting sunlight into energy is through the utilization of solar panels. Solar panel technology
has improved recently, and has resulted in it becoming economically viable to generate electricity via these sunlight receptors.
A number of companies are in the business of producing solar panels. In a sideways fashion, I came across
TSL (Trinia Solar) yesterday. Someone had run a stock screener with the settings similar to:
- stocks with high put/call ratio
- minimum stock price of $10/share
- minimum put & call open interest of 10k contracts
- minimum average daily stock volume of 100k shares traded
- trading above its 20-day moving average
- near a 52 week high
TSL has risen briskly but has some pessimism from investors based upon puts easily outnumbering calls. If the
negative sentiment unwinds, it could rise further. Another area of measuring pessimism is through
monitoring short interest. The web size
Short Squeeze offers up information so one can see how many shares are sold sort for a
company. Short Interest can also be found within the Fundamental Record of the DTNIQ data stream.
In looking at TSL on Google today, Google shows that all similar solar stocks took a decline today. It turns out that
the German government will cut its solar financial incentive schem by 16 to 17 percent.
With a price correction like this, it might be interesting to look to getting into solar at this level. But the real moral
of the story is that a news service which tracks a portfolio's stock symbols is probably worth its weight in gold.
Moving along with the alternate energy theme, collected energy needs to be stored. Currently technology
appears to be focussing on Lithium as the mineral of choice. A common play appears to be WLC (Western Lithium). An
upcoming player as mentioned in Industrial Metals magazine might be GXY (Galaxy Resources).
[/Trading/MarketNotes]
permanent link
2010 Jan 14 - Thu
Definition of a Banana Republic
In a missive from Contrarian Profits newsletter today, they drew nice intimated
an interesting parallel between a Banana Republic and a certain relatively large,
very well known country. The recipe for a Banana Republic, as described by
Justice Litle, Editorial Director, Taipan Publishing Group.
Wikipedia defines "Banana Republic" as a pejorative term for a country that is
politically unstable, dependent on limited agriculture (e.g. bananas), and
ruled by a small, self-elected, wealthy, and corrupt clique.
Ingredients:
- out-of-control printing presses
- currency restrictions and controls
- strangling regulation and red tape
- aggressive nationalization of private assets
- extravagant social programs (bribing the poor)
- deeply corrupt financial structures (bribing the connected)
- crushing pressures on small business (extorting the middle class)
Directions:
- Combine fervent promises of "hope," "change" and "revolution" in demagogue crockpot. Bring mixture to a slow rolling boil.
- As mixture firms, stir in aggressive spending plans and "revolutionary" public adjustments. Sprinkle liberally with insider connections and oligarchic financial loopholes to maintain smooth consistency. Let simmer for a full election cycle on low-heat propaganda flame.
- Pour filling into flaky self-righteous crust. Top with blatant corruption, repressed scandal and outright nationalization. Bake in fiscal suicide oven until inflation thermometer registers 30%+ and insider cronies are sufficiently enriched.*
*As with a delicate souffle, the middle class must not experience complete collapse during this phase. If this happens, you have inadvertently followed the recipe for a coup.
[/Personal/Business]
permanent link
2010 Jan 12 - Tue
Creating a Cold Standby FreeBSD Machine
I have a FreeBSD machine for which I don't have original installation files, thus
rebuilding the machine and reinstalling software from the ground up on new hardware
could be a problem.
I have a few of avenues getting a suitable secondary machine up and running:
I didn't use the dump/restore commands as I don't have easy access to enough space for temporary
storage so I brute forced the scenario by copying data directly from the source machine partition
to the destination machine partition.
I was able to obtain almost identical hardware for the second machine so as not to have any
hardware compatibility issues. Drive sizes and memory are larger on the second machine.
FreeBSD has a FixIt feature, otherwise known as a live boot, which allows a machine to be booted and
analyzed from a cd. I could have used a recent version of FreeBSD to do this task, but instead,
I chose a version which was of the same major.minor version as that of the running system at the
old release archives.
Once booted off the cd, a number of tasks can be completed:
- create partitions on the disks
- label the partitions
- obtain network connectivity
- copy partitions from source machine to destination machine
- depending upon services started, protect the destination machine from the network
- reboot the destination machine
- log in and verify successful operation
- possibly change driver, service, partition, and network settings
On the source machine, with root privileges, use 'df' and 'df -h' to determine partition
labels and sizes. 'sysinstall' may need to be run to get a listing of all partitions and their sizes,
particularily for the swap partition. The command 'swapctl -l -k -s' will detail the swap device.
On a SCSI based system, drive devices are named in fashion to /dev/da0s1a, where
da0 is drive 0, s1 is slice 1, and a is the first labelled partition. a is typically / (the boot partition),
b is typically swap, c is a hidden device for accessing the full drive, and d and
subsequent drives are partition mappings. On this system I have d as /var,
e as /tmp, and f as /usr.
On the destination computer, I selected the Fixit menu item, then the 'CDROM/DVD' option,
which takes me to a prompt.
I then used 'sysinstall' to gain access to various utilities. The first one I use is FDisk
to allocate all space on the drive to FreeBSD and use 'w' to write the changes to disk.
On exiting, I select the option to 'Install the FreeBSD Boot Manager' onto the drive.
I then use the Label editor to create partitions. They are created in order of a, b, d, e, and f. Note
that c is skipped as part of the process. Write the changes with 'w' and quit.
Then use the Networking menu item, then the Interfaces menu item to set ip addresses for the
interface.
Upon exiting back to the prompt, an 'ipconfig' command should show the ip address as being set.
After confirming network access is available, partitions can be copied over:
- ssh source_ip "cat /dev/da0s1a" | cat > /dev/da0s1a
- ssh source_ip "cat /dev/da0s1b" | cat > /dev/da0s1b
- ssh source_ip "cat /dev/da0s1d" | cat > /dev/da0s1d
- ssh source_ip "cat /dev/da0s1e" | cat > /dev/da0s1e
- ssh source_ip "cat /dev/da0s1f" | cat > /dev/da0s1f
Once things are copied over, disconnect the network interfaces, exit the menus, and
restart the machine. The machine should boot as though it was the original. Watch
the boot messages carefully for any errors.
The utility of the above operation improves if a filesystem snapshot capability is available
incorporated, then a guaranteed 'instant in time' view is available.
[/OpenSource/FreeBSD]
permanent link
|
