2010 Jan 29 - Fri

Migrating Bacula 2.x on Debian Etch to 3.x on Squeeze

Debian Etch, which is the current release, has the Bacula 2.x packages. I needed to upgrade to the Bacula 3.x packages, which are located in debian/testing, also known as the forthcoming Debian Squeeze release. In addition, since PostgreSQL 8.3 is packaged in Etch, and PostgreSQL 8.4 is packaged in Squeeze/testing, a database migration is also required.

I had attempted updating my sources.list file to testing and then running the apt-get dist-upgrade process. This broke some dependences, and also broke on a udev migration. I guess testing has more testing to do on the distribution upgrade process.

In the end, I built a new Bacula service on a freshly installed Debian testing server.

The special consideration for this configuration is that it needs to handle backing up servers across a WAN. As such, backups may travel through one or more firewalls. Through such a configuration, it is very difficult to get the firewall ports opened for the various necessary Bacula service ports. The better way to tackle this is through the use of ssh' port local and remote port forwarding capability. Port 22 is becomes the only necessary port to open on a firewall. The ssh-tunnel.sh script helps make this happen.

To build the server, when it came to package selection, I unselected all packages, and then chose just the database package which installed PostgreSQL.

After the basic server finished installing and rebooted, I manually installed the following packages:

apt-get install bacula-common-pgsql
apt-get install bacula-client
apt-get install bacula-director-common
apt-get install bacula-director-pgsql
apt-get install bacula-sd-pgsql
apt-get install bacula-server

If starting with a new database, then dbconfig-common can be used. If migrating an older database, don't use dbconfig-common, and use the manual methods I'll describe further on. There is further documentation in /usr/share/doc/bacula-director-pgsql.

During installation of the bacula packages, a new user of 'bacula' is created, as well as a group called 'tape'. The 'bacula' user has a home directory of /var/lib/bacula.

Into that directory, create a .ssh directory for any authorized_keys and known_hosts required. I also created a keys subdirectory to hold the public/private keys for ssh'ing into other servers for processing backups. I called the two files 'bacula' and 'bacula.pub'. These will be referenced in my customized ssh-tunnel.sh script.

Run

dpkg-reconfigure exim4-config

to reconfigure the mail system to allow outbound mail delivery.

My backups go onto a remote file share. I created an entry in /etc/fstab along the lines of:

10.1.1.1:/bu /mnt/nas nfs rw,hard,intr,async,nodev,nosuid 0 0

Ensure that the NFS client is installed through:

apt-get install nfs-common

In /etc/postgresql/8.4/main/pg_hba.conf, I have lines along:

host    bacula  bacula          127.0.0.1/32            trust
host    bacula  sysadmin        127.0.0.1/32            trust
local   bacula  bacula          trust
local   bacula  sysadmin        trust

As an aside, a useful command to find out database information is through the use of:

psql -l

When migrating the database to 8.4, there are modifcations to the pg_dump command required (which are required to prevent import errors along the lines of 'ERROR: invalid byte sequence for encoding "UTF8"', basically resolving the UTF-8 to SQL_ASCII issues in Bacula):

pg_dump -E SQL_ASCII -U bacula bacula > /var/lib/bacula/bacula.sql

On the new server, use the following to import the database:

dropdb bacula
su - postgres
psql
create role bacula;
create database bacula owner=bacula encoding='SQL_ASCII' template=template0;
/q
psql bacula </var/lib/bacula/bacula.sql

Basic instructions for updating the database from Bacula table version 10 to Bacula table version 11 is found in /usr/share/bacula-director/update_postgresql_tables:

BEGIN;
ALTER TABLE file ALTER fileid TYPE bigint ;
ALTER TABLE basefiles ALTER fileid TYPE bigint;
ALTER TABLE job ADD COLUMN readbytes bigint default 0;
ALTER TABLE media ADD COLUMN ActionOnPurge smallint default 0;
ALTER TABLE pool ADD COLUMN ActionOnPurge smallint default 0;

-- Create a table like Job for long term statistics
CREATE TABLE JobHisto (LIKE Job);
CREATE INDEX jobhisto_idx ON JobHisto ( starttime );

UPDATE Version SET VersionId=11;
COMMIT;

Once the configuration files for the director, storage manager, and file manager are ready, bacula can be managed through 'bconsole'.

My modified /etc/bacula/scripts/ssh-tunnel.sh looks like:

#!/bin/sh
# script for creating / stopping a ssh-tunnel to a backupclient
# Stephan Holl sholl@gmx.net
# Modified by Joshua Kugler joshua.kugler@uaf.edu
# Modified by Ray Burkholder ray@oneunified.net
#
#

# variables
USER=bacula
CLIENTADDR=$2
# CLIENTPORT is local end
CLIENTPORT=$3
#LOCAL=your.backup.server.host.name
# local is a local address and uses ssh's remote/local port forwarding
LOCAL=127.0.0.1
SSH=/usr/bin/ssh
SSHOPTIONS=-vfnCN2
LOG1=/var/lib/bacula/log1.log
LOG2=/var/lib/bacula/log2.log
#LOG1=/dev/null
#LOG2=/dev/null
# location of the public/private keys used with ssh to gain access to remote servers
KEY=/etc/bacula/keys/bacula  

case "$1" in
 start)
    # create ssh-tunnel
        echo "Starting SSH-tunnel to $CLIENTADDR..."
        $SSH $SSHOPTIONS -o PreferredAuthentications=publickey -i $KEY -l $USER \
            -R 9101:$LOCAL:9101 -R 9103:$LOCAL:9103 -L $CLIENTPORT:$LOCAL:9102 $CLIENTADDR \
            >> $LOG1 2>> $LOG2
        exit $?
        ;;

 stop)
        # remove tunnel
        echo "Stopping SSH-tunnel to $CLIENTADDR..."
        # find PID killem
        PID=`ps ax | grep "$SSH $SSHOPTIONS -o PreferredAuthentications=publickey -i $KEY" \
             | grep "$CLIENTADDR" | awk '{ print $1 }'`
        kill $PID
        exit $?
        ;;
 *)
        #  usage:
        echo "             "
        echo "      Start SSH-tunnel to client-host"
        echo "      to bacula-director and storage-daemon"
        echo "            "
        echo "      USAGE:"
        echo "      ssh-tunnel.sh {start|stop} client.fqdn"
        echo ""
        exit 1
        ;;
esac

The links I used for getting started with ssh-tunnels are found at:

• /usr/share/doc/bacula-common/examples/ssh-tunnel-README.txt.gz
• http://wiki.bacula.org/doku.php?id=sshtunnel

In /etc/hosts file, 127.0.0.1 should be the only line referring to the local server. The exteral port ip address should be commented out:

127.0.0.1      localhost  bu.example.com        bu
#10.10.10.1    bu.example.com        bu

In the bacula-dir.conf configuration file, a typical client configuration will look similar to:

Client {
  Name = mail-fd
  Address = 127.0.0.1
  FDPort = 9130  # specific port for this client, allows multiple simultaneous backups
  Catalog = MyCatalog
  Password = "xxxxxx"          # password for FileDaemon
  File Retention = 120 days         
  Job Retention = 4 months          
  AutoPrune = yes                     # Prune expired Jobs/Files
}

The special characteristic of the above configuration is the use of a unique port number for FDPort. Each client in the bacula-dir.conf should have a unique port number. This allows bacula to tunnel via ssh to remote clients and redirect them to the storage manager on the local server.

The definition of the storage device in bacula-dir.conf will have Address=127.0.0.1 and SDPort=9103.

The job description for each client should have something similar to:

Job {
  Name = "mail-fd"
  Client = mail-fd
  JobDefs = "DefaultJob"
  FileSet = "FileSet_mail"
  Storage = storageSshClients
  Write Bootstrap = "/var/lib/bacula/mail.bsr"
  Priority = 12
  Run Before Job = "/etc/bacula/scripts/ssh-tunnel.sh start mail.example.com 9130"
  Run After  Job = "/etc/bacula/scripts/ssh-tunnel.sh stop  mail.example.com 9130"
}

When using Bacula in console mode, a useful command to find out the meaning of the backup status codes:

*sqlquery
Entering SQL query mode.
Terminate each query with a semicolon.
Terminate query mode with a blank line.
Enter SQL query: select * from status;
+-----------+---------------------------------+
| jobstatus | jobstatuslong                   |
+-----------+---------------------------------+
| C         | Created, not yet running        |
| R         | Running                         |
| B         | Blocked                         |
| T         | Completed successfully          |
| E         | Terminated with errors          |
| e         | Non-fatal error                 |
| f         | Fatal error                     |
| D         | Verify found differences        |
| A         | Canceled by user                |
| F         | Waiting for Client              |
| S         | Waiting for Storage daemon      |
| m         | Waiting for new media           |
| M         | Waiting for media mount         |
| s         | Waiting for storage resource    |
| j         | Waiting for job resource        |
| c         | Waiting for client resource     |
| d         | Waiting on maximum jobs         |
| t         | Waiting on start time           |
| p         | Waiting on higher priority jobs |
+-----------+---------------------------------+
Enter SQL query:
End query mode.

For the bacula entry in /etc/passwd, change /bin/false to be /bin/sh.

For each server to which will be connected via ssh, within the context of the bacula user, use the following command to update ~/.ssh/known_hosts:

ssh -l bacula -i /etc/bacula/keys/bacula -v server.example.com

[/OpenSource/Debian] permanent link

2010 Jan 17 - Sun

Option Trading Scenarios

Most option trading books discuss the mechanics and mathematics of option trading. Rarely do they offer up any sort of in-the-trenches useful guidelines for doing what when and how. Some recent reading has enlightened me on some useful option trading scenarios.

The first one provides the dual mechanism of obtaining income and optionally paying for protection on an underlying which is already in one's portfolio. The process involves selling a kind of straddle: selling an out of the money call and an out of the money put. For example, with TSL, as of Friday's close, is at $49.50. The Feb 55 Call is at $1.75 and the Feb 48 Put is at $3.25. Selling this results in $5.00 premium. If TSL stays within the range of $48 and $55 till Feb 19, the full premium is kept. If the price goes above $55, the underlying will probably be exercised to result in a $5.50 profit, which is the premium plus the amount the underlying goes to get to the strike price. If the price goes below $48, the Put will go into the money, most likely causing it to be exercised, and you'll end up buying the underlying at the Put strike price, which may be a good thing if you are expecting the price to rebound. On the protection side, the premium earned on the put and call sale could be use to purchase twice the number of puts at the next out-of-the money price, which in this case would be Feb 45 Put. This provides protection on the original underlying plus the shares gained when the buyer exercised the puts.

Another strategy requires maintenance of no underlying. For a stock that you think will go up due to some upcoming good news, such as an earnings announcement, one could sell an out of the money put and use the funds to buy and out of the money call. By careful selection of put and call strikes, the money gained by selling the put may fully cover the price of buying the call. The downside of this is if the price of the underlying goes below the put strike price. You may end up owning the underlying in this case, but at least it was obtained at a lower price.

[/Trading/Options] permanent link

They Who Have The Money

Sometime this last week, someone made the observation of the fact that there was a 'hidden buyer' buying a large amount of treasuries during the weekly US Treasury auctions. There was speculation it might be the Chinese operating through private proxy parties.

The US government has a problem with all the money they are spending. Some one has to loan it to them. Hence the regular treasury auctions.

A couple figures came to mind. I believe I recall seeing that the Chinese have over one trillion US dollars due to trade imbalance between the US and China. Another figure has to do do with the fact that the US government has added over a trillion dollars to their defict through recent spending.

If you put those two numbers together, and tie them together with the 'hidden buyer' observation, perhaps it could be said that the Chinese are converting their US dollar currency holdings into US Treasury holdings. That way, not only have they made money through their exports, but they make additional revenue through the yields on the treasuries purchased.

[/Trading/MarketNotes] permanent link

Short Interest, Solar, Lithium

Many commenters have been discussing Peak Oil, with Peak Oil being the fact that we've found all the easy supplies of oil, most of which has been pumped, and that our supplies, regardless of type, are dwindling... the peak has come and gone. To be more specific, the supply of oil, of which we only have a finite supply, is decreasing, while world demand for oil is increasing. The first day of Economics 100 teaches us the supply / demand curve and the effect on pricing. Oil prices can only go up, and up until the last dropped is supplied and consumed.

In the meantime, alternate sources of energy must be found to supply the world's unending requirement. For sourcing some fraction of our energy needs, some are turning to the nearly non-ending supply of sunlight. One way of converting sunlight into energy is through the utilization of solar panels. Solar panel technology has improved recently, and has resulted in it becoming economically viable to generate electricity via these sunlight receptors.

A number of companies are in the business of producing solar panels. In a sideways fashion, I came across TSL (Trinia Solar) yesterday. Someone had run a stock screener with the settings similar to:

• stocks with high put/call ratio
• minimum stock price of $10/share
• minimum put & call open interest of 10k contracts
• minimum average daily stock volume of 100k shares traded
• trading above its 20-day moving average
• near a 52 week high

TSL has risen briskly but has some pessimism from investors based upon puts easily outnumbering calls. If the negative sentiment unwinds, it could rise further. Another area of measuring pessimism is through monitoring short interest. The web size Short Squeeze offers up information so one can see how many shares are sold sort for a company. Short Interest can also be found within the Fundamental Record of the DTNIQ data stream.

In looking at TSL on Google today, Google shows that all similar solar stocks took a decline today. It turns out that the German government will cut its solar financial incentive schem by 16 to 17 percent.

With a price correction like this, it might be interesting to look to getting into solar at this level. But the real moral of the story is that a news service which tracks a portfolio's stock symbols is probably worth its weight in gold.

Moving along with the alternate energy theme, collected energy needs to be stored. Currently technology appears to be focussing on Lithium as the mineral of choice. A common play appears to be WLC (Western Lithium). An upcoming player as mentioned in Industrial Metals magazine might be GXY (Galaxy Resources).

[/Trading/MarketNotes] permanent link

2010 Jan 14 - Thu

Definition of a Banana Republic

In a missive from Contrarian Profits newsletter today, they drew nice intimated an interesting parallel between a Banana Republic and a certain relatively large, very well known country. The recipe for a Banana Republic, as described by Justice Litle, Editorial Director, Taipan Publishing Group.

Wikipedia defines "Banana Republic" as a pejorative term for a country that is politically unstable, dependent on limited agriculture (e.g. bananas), and ruled by a small, self-elected, wealthy, and corrupt clique.

Ingredients:

• out-of-control printing presses
• currency restrictions and controls
• strangling regulation and red tape
• aggressive nationalization of private assets
• extravagant social programs (bribing the poor)
• deeply corrupt financial structures (bribing the connected)
• crushing pressures on small business (extorting the middle class)

Directions:

• Combine fervent promises of "hope," "change" and "revolution" in demagogue crockpot. Bring mixture to a slow rolling boil.
• As mixture firms, stir in aggressive spending plans and "revolutionary" public adjustments. Sprinkle liberally with insider connections and oligarchic financial loopholes to maintain smooth consistency. Let simmer for a full election cycle on low-heat propaganda flame.
• Pour filling into flaky self-righteous crust. Top with blatant corruption, repressed scandal and outright nationalization. Bake in fiscal suicide oven until inflation thermometer registers 30%+ and insider cronies are sufficiently enriched.*

*As with a delicate souffle, the middle class must not experience complete collapse during this phase. If this happens, you have inadvertently followed the recipe for a coup.

[/Personal/Business] permanent link

2010 Jan 12 - Tue

Creating a Cold Standby FreeBSD Machine

I have a FreeBSD machine for which I don't have original installation files, thus rebuilding the machine and reinstalling software from the ground up on new hardware could be a problem.

I have a few of avenues getting a suitable secondary machine up and running:

• as I have bacula making backups, I could use the FreeBSD Bare Metal Recoveryas the system is still active, I could Restore a FreeBSD system using the fixit CD, or
• but in the end I chose to copy partitions over manually by using a variation of cat over ssh.

I didn't use the dump/restore commands as I don't have easy access to enough space for temporary storage so I brute forced the scenario by copying data directly from the source machine partition to the destination machine partition.

I was able to obtain almost identical hardware for the second machine so as not to have any hardware compatibility issues. Drive sizes and memory are larger on the second machine.

FreeBSD has a FixIt feature, otherwise known as a live boot, which allows a machine to be booted and analyzed from a cd. I could have used a recent version of FreeBSD to do this task, but instead, I chose a version which was of the same major.minor version as that of the running system at the old release archives.

Once booted off the cd, a number of tasks can be completed:

• create partitions on the disks
• label the partitions
• obtain network connectivity
• copy partitions from source machine to destination machine
• depending upon services started, protect the destination machine from the network
• reboot the destination machine
• log in and verify successful operation
• possibly change driver, service, partition, and network settings

On the source machine, with root privileges, use 'df' and 'df -h' to determine partition labels and sizes. 'sysinstall' may need to be run to get a listing of all partitions and their sizes, particularily for the swap partition. The command 'swapctl -l -k -s' will detail the swap device.

On a SCSI based system, drive devices are named in fashion to /dev/da0s1a, where da0 is drive 0, s1 is slice 1, and a is the first labelled partition. a is typically / (the boot partition), b is typically swap, c is a hidden device for accessing the full drive, and d and subsequent drives are partition mappings. On this system I have d as /var, e as /tmp, and f as /usr.

On the destination computer, I selected the Fixit menu item, then the 'CDROM/DVD' option, which takes me to a prompt. I then used 'sysinstall' to gain access to various utilities. The first one I use is FDisk to allocate all space on the drive to FreeBSD and use 'w' to write the changes to disk.

On exiting, I select the option to 'Install the FreeBSD Boot Manager' onto the drive.

I then use the Label editor to create partitions. They are created in order of a, b, d, e, and f. Note that c is skipped as part of the process. Write the changes with 'w' and quit.

Then use the Networking menu item, then the Interfaces menu item to set ip addresses for the interface.

Upon exiting back to the prompt, an 'ipconfig' command should show the ip address as being set. After confirming network access is available, partitions can be copied over:

• ssh source_ip "cat /dev/da0s1a" | cat > /dev/da0s1a
• ssh source_ip "cat /dev/da0s1b" | cat > /dev/da0s1b
• ssh source_ip "cat /dev/da0s1d" | cat > /dev/da0s1d
• ssh source_ip "cat /dev/da0s1e" | cat > /dev/da0s1e
• ssh source_ip "cat /dev/da0s1f" | cat > /dev/da0s1f

Once things are copied over, disconnect the network interfaces, exit the menus, and restart the machine. The machine should boot as though it was the original. Watch the boot messages carefully for any errors.

The utility of the above operation improves if a filesystem snapshot capability is available incorporated, then a guaranteed 'instant in time' view is available.

[/OpenSource/FreeBSD] permanent link

2009 Dec 22 - Tue

Global Warming: Should I Be Concerned?

For years, I've heard reports that the Earth is warming up, and as a consequence, something should be done about it. Primary symptoms of warming have been through reports of that Antartica's Ice Cap is reducing in size. A primary contributing factor which has been suggested as a primary contributor has been the combustion of various fossil fuels which added carbon dioxide to the atmosphere, which in turns traps reflected sunlight, which in turn warms the Earth/Atmosphere. Among other things.

I've been lead to believe that that global warming is a bad thing, and should be stopped, or even possibly reversed.

A fellow by the name of Mann recently presented a chart, now known as the 'hockey stick' chart, which shows a significant increase in average global temperatures. Popular media has become enamoured with this chart, particularily recently, what with all the latest brouhaha during the Copenhagen Climate talks.

From what I hear, developing nations want developed nations to give them billions of dollars in either cash or technology cut emissions. If developed nations can't even commit to cutting their own emissions, what is the point in giving some one else money in the illusion that they might also cut emissions.

Would it not be better to just put the hundreds of billions of dollars directly into alternate energy research? Obviously, there will be expensive gaffs along the way, but at least something direct may come of it to assist the developed as well as developing countries.

But then comes along some data that indicates that this may all be a moot point. A number of different scientific observers have put together data from many different sources, with the data indicating that we are by no means currently at a historically high global temperature. The middle ages (around 1000AD) saw higher temperatures than what we are currently experiencing. The time around 0 AD saw even higher temperatures. And 1000 years before that were even higher temperatures. And over the course of history, substantial temperature swings have been noted.

Mann's 'Hockey Stick' may be significant in recent history, but is it significant in the grand scheme of things? Is global warming an event which would have arrived regardless of human meddling?

One way or another, the climate is going to give us some interesting action over the next little while, for some value of little. It looks like I should be concerned, but due to completely different reasons.

For more climate oriented pointers, visit Watts Up With That?.

[/Personal] permanent link

2009 Dec 06 - Sun

ToS/DSCP Cheat Sheet

On the Flow-Tools email list, Craig Weinhold published a cheat sheet for how to treat IP Packet ToS (Type of Service) bits:

**** Pre-1998

The IPv4 ToS byte was part of the original 1981 definition of Internet Protocol 
in RFC 791, which specified a 3-bit precedence value and 3-bits of ToS attributes. 
In the tables below, "tos" values refer to the entire byte. 
In 1992, RFC 1349 added a fourth ToS attribute. 

  0x80  0x40  0x20  0x10  0x08  0x04  0x02  0x01
+-----+-----+-----+-----+-----+-----+-----+-----+
|     PRECEDENCE  |    TOS attributes     |  -  |
+-----+-----+-----+-----+-----+-----+-----+-----+

         PRECEDENCE                     TOS attributes

name            dec tos bin     name              dec tos bin
network           7 224 111     min-delay           8  16 1000
internet          6 192 110     max-throughput      4   8 0100
critical          5 160 101     max-reliability     2   4 0010
flash-override    4 128 100     min-monetary-cost   1   2 0001
flash             3  96 011     normal              0   0 0000
immediate         2  64 010
priority          1  32 001
routine           0   0 000


**** Post-1998 

RFC 2474 reworked the ToS as a 6-bit Differentiated Services Code Point (DSCP) 
and, soon after, RFC 3168 allocated the lowest two bits for 
Error Congestion Notification (ECN, 
an IP analogy of frame-relay FECN and ATM EFCI). 

  0x80  0x40  0x20  0x10  0x08  0x04  0x02  0x01
+-----+-----+-----+-----+-----+-----+-----+-----+
|                DSCP               |    ECN    |
+-----+-----+-----+-----+-----+-----+-----+-----+

                        DSCP

name    dec tos binary      name    dec  tos binary
AF11    10   40 001010      CS1       8   32 001000
AF12    12   48 001100      CS2      16   64 010000
AF13    14   56 001110      CS3      24   96 011000
AF21    18   72 010010      CS4      32  128 100000
AF22    20   80 010100      CS5      40  160 101000
AF23    22   88 010110      CS6      48  192 110000
AF31    26  104 011010      CS7      56  224 111000 
AF32    28  112 011100      EF       46  184 101110
AF33    30  120 011110      default   0    0 000000
AF41    34  136 100010      
AF42    36  144 100100      AF = assured forwarding
AF43    38  152 100110      EF = expedited forwarding
                            CS = class selector

   ECN (unrelated to QoS)
   00   Not-ECT  Not ECN-Capable Transport
   01   ECT(0)   ECN-Capable Transport 
   10   ECT(1)   ECN-Capable Transport 
   11   CE       Congestion Experienced

**** Notes on interpreting the ToS byte

The two definitions are complimentary for the upper 3-bits. This is good, since those three bits are often copied to/from the 3-bit class-of-service (CoS) field of layer-2 802.1p frames and the 3-bit experimental (EXP) field of MPLS frames. Bits 3-5, however, are fairly incompatible..

Thus, it's important not to oversimplify precedence/DSCP as a simple pecking order. In reality, each unique precedence/DSCP value conveys a packet's requirements for throughput, latency, and packet loss, three traits that are somewhat at odds with each other. And, any value can be assigned to any organizationally-unique purposes. For example,

• Packets with precedence 5 and/or DSCP EF are often serviced by priority queues, so they may delay packets with higher precedence/DSCP values.
• Within each AF level (e.g., AF2x includes AF21, AF22, and AF23), the higher values indicate a higher tolerance to packet loss. I.e., a congested interface should drop AF22 packets earlier than AF21 packets. In Cisco IOS, this behavior is implemented with DiffServ-complaint WRED ('random-detect dscp' on a class-map).
• Any DSCP value under CS6 can be assigned for any organizationally-unique use. For example, Precedence 1/DSCP CS1 is often assigned for use as a less-than-best-effort class called scavenger. To successfully implement the scavenger class, all network devices must agree to treat CS1 traffic worse than Precedence 0/DSCP default.

ECN (RFC 3168) is an emerging issue for traditional netflow collection and processing. The jist of ECN is that an intermediary router can, after sensing congestion, change the lower two bits of ToS to indicate congestion so that the hosts can slow themselves down. It's a L3 implementation of frame-relay FECN, essentially. Unfortunately, since the ToS field changes packet-to-packet and hop-to-hop, it also disrupts the traditional netflow 7-tuplet key (protocol, src/dst IP, src/dst port, ToS, input interface).

If you can, exclude ToS as a flow key on your netflow sources. Recent cisco IOS versions let you do this with flexible netflow while still exporting netflow v5.

[/Networks] permanent link

2009 Oct 26 - Mon

Machine Readable News and Algorithmic Trading

A-Team Research has released a special report called: Machine Readable News and Algorithmic Trading.

I've writing some code to accept a news release feed from DTNIQ/IQFeed. This report comes in handy for supplying some ideas on how to analyze and make use of the news feed. Here are some examples:

• When generating trading signals for high frequency traders and other alpha-seekers, it can be used to build sentiment measurement applications, stock screening applications and back-testing systems for trading algorithms.
• It can be used in support of market surveillance systems.
• This translates into simple stock-screening applications for individual securities or lists of stocks.
• It can mean the analysis of macroeconomic data to identify trends, correlations and other relationships.
• It can involve scanning key parameters to measure market sentiment.
• It could predict potentially volatile trading days, indicating which stocks or types of stock may be most affected.
• It can also be used to quickly derive directional signals from the marketplace, and set in play appropriate trading algorithms.

[/Trading/AutomatedTrading] permanent link

Bottom Line on Security in Windows 7, and Some Thoughts on MultiTouch

From SANS NewsBites vol. 11 Num. 84, 2009-10-23, NewsBites editorial board member John Pescatore says:

From a security perspective, Windows 7 offers definite improvements over Windows XP, but there is no major security reason to move to Windows 7 before it makes business sense. The biggest improvement in Windows desktop security comes from getting off of the IE6 browser and moving to IE8 or the latest version of Firefox - and you don't need Windows 7 to do that.

I've read that Windows 7 is somewhat faster but is better than Windows Vista. I havn't seen definitive reviews that Windows 7 is faster than Windows XP, or offers anything useful over and above what Windows XP offers as a development or user platform.

Well actually, I understand that Windows 7 has a multi-touch API built-in for when multi-touch devices become more ubiquitous. 10/GUI is one such interesting multi-touch method of CHI (Computer Human Interaction).

reacTIVision is an existing tangible multi-touch interaction framework. I've always thought that using a multi-touch interface with a DMX controlled lighting system would make for some very intersting busking capabilities for live concerts.

Anyway back to Windows 7, the EE Times Newsletter roving editor Rick Merritt asserts:

That all Microsoft has done with Windows 7 is not mess it up. "Imagine the response systems makers might have if Microsoft had actually enabled some cool new ideas," Merritt writes. "Call me a curmudgeon, but I think Microsoft is resting on its monopolistic backside." What was needed from Microsoft, of course, was an OS that advanced the state of the art. This is not the time for tech companies to play it safe, especially a company with pockets as deep as Microsoft's.

On the other hand, if I took the time out to evaluate real life workflows in the new Windows 7 environment, and the execution time differentials was minimal, I'd migrate just to stay with the latest thing. Some of the workflows I'd have to check would be:

• Editing video with Adobe Premiere Pro CS4: lots of drive activity and lots of multimedia interaction
• Compiling heavily templated Boost supported C++ programs in Visual Studio: lots of CPU and some drive activity
• Compiling heavily templated Boost supported C++ programs in a an KDE/Eclipse/GCC environment hosted in a VMWare Workstation environment: lots of cross operating system calls
• Running trading and news gathering applications with intensive cross thread messaging: cpu and network intensive

Can anyone offer up opinions on what they've encountered between Windows XP, Windows Vista, and Windows 7 in these various workflow environments from a speed/stability/effectiveness point of view?

Shortly after having written this, I saw an article published at Ars Technica which had a rather lengthy review regarding XP, Vista, and Windows 7 entitled Hasta la Vista, baby: Ars reviews Windows 7. Buried further back in the article makes reference to the fact that performance isn't much different among the three. The article does mention multi-touch, and indicates that it isn't very well integrated into the supplied applications.

Once I get some time, it looks like an upgrade to Windows 7 might be worth examining.

[/Personal/Technology] permanent link

2009 Oct 17 - Sat

Trader Urgency Indicator

In the LinkedIn Group Automated Trading Strategies, Alpesh Patel posted a Trader Urgency Indicator:

• Fix no. of ticks based on market traded( Let's say 100 ticks chart for S&P 500 Emini)
• Monitor the time taken to finish the bar
• You will notice that most successful breakout bars will finish in significatly less time showing trader urgency
• At trend exhaustion you will notice significantly more time taken to finish the bar

I think I wrote about Range Bars at one point in time. This is probably a variation on that theme.

[/Trading/TechnicalAnalysis] permanent link

Memory Leak Detection in MSVC 2008 C++

In Visual Studio, when building debug releases, I seem to recall that memory leak detection was automatically enabled. In Visual Studio 2008, memory leak detection is not automatically enabled. Code will need to be added to the source files to make it available.

Memory Leak Detection Enabling is a document in MSDN describing how to enable the ability. Basically, to enable the debug heap functions, include the following statements:

#define _CRTDBG_MAP_ALLOC
#include <stdlib.h>
#include <crtdbg.h>

Immediately before the program exits, include the following function call:

_CrtDumpMemoryLeaks();

One commenter indicates that this only works best with C code, ie, C code will get discriptive comments, but C++ code will get cryptic memory reports.

In order to get the file and line number to work you need to manually redefine new in your code. This is done by undefining new, and redefining it to point to the debug versions that take a file and line number.

During debugging, in Windows based applications, std::cout no longer sends text into the IDE's Output window. Instead, the function OutputDebugString( "..." ) needs to be used.

[/Personal/SoftwareDevelopment/CPP] permanent link