2008 Dec 28 - Sun

Modern Day Vikings

I don't have access to the paper, but the abstract looks interesting: Looting: The Economic Underworld of Bankruptcy for Profit. Sometimes I think that some companies do a business plan around this, or implement it through 20::20 hindsight. Back a few years ago, companies were laying fibre like crazy. Over capacity resulted. Many went 'under' and resurfaced with the assets but less debt overhead after writing investors off.

[/Trading/ReadingMaterial] permanent link

2008 Nov 27 - Thu

TFTP

There are a number of TFTP servers available. I had written about atftpd in a related article. Research indicates that tftpd-hpa is another popular tftp server. This article provides a few hints on its installation. Although designed for remote boot capability for PXE (and for handling larger files), I use it mostly for device configuration and image loading.

Basic installation on Debian is straightforward: apt-get install tftpd-hpa

I created a local sub-directory called /var/local/tftpd. Traditionally, people use /tftpboot but I wanted the files in the traditional /var/local location instead.

I modified /etc/default/tftpd-hpa to have this line: OPTIONS="-l -c -u tftpd -s /var/local/tftpd"

I added a user and group called tftpd.

I disabled the tftp entry in /etc/inetd.conf, and restarted inetd.

Starting the service: /etc/init.d/tftpd-hpa start

I use iptables for inbound/outbound protection, so needed to add rules for the tftp protocol. For protocol inspection the connection tracker is needed: modprobe ip_conntrack_tftp

[/OpenSource/Debian/Monitoring] permanent link

2008 Nov 24 - Mon

TTCP: Test TCP

A quick and simple tool for link bandwidth testing (aka Throughput Testing) is included in many flavours of Cisco's IOS. Although it is hidden and officially unsupported, it is documented and functional.

By running 'ttcp' from the command line in privileged mode on two different routers, one can test links between the routers.

Cisco documents the tool with Document 10340, Using Test TCP (TTCP) to Test Throughput.

A couple of other non-Cisco tools are available and maintain compatibility in order to perform link testing between most combinations of routers, Unix, Linux, and Windows platforms.

Netcordia has a Java based client, while Unix and Windows based client can be found at ttcp. The source compiled on Linux with no problem.

In The Story of the TTCP Program, Mike Muuss discusses some of the history of ttcp. It seems that he is the original author of the venerable ping program. In the same article, Mike illustrates a clever file transfer capability of ttcp, if effect being a UNIX "pipe" between two machines across a network. On the destination machine:

ttcp -r | tar xvpf -

and on the source machine:

tar cf - directory | ttcp -t dest_machine

and on possible intermediate machines:

ttcp -r | ttcp -t next_machine

A discussion of additional variants of ttcp can be found at ttcp/nttcp/nuttcp/iperf versions. It looks as though the version used by Cisco is a renamed nttcp. A version called nuttcp will echo traffic back.

iperf uses the same concept but uses a different name and includes different functionality for network perforamance analysis. I'm not sure if it is compatible with ttcp.

When using iperf, the -P option will run multiple tcp tests simultaneously. A document called Guide to Bulk Data Transfer over a WAN describes the use of iperf to determine data transfer rates under various scenarios.

Linux.com has an article called Benchmarking network performance with Network Pipemeter, LMbench, and nuttcp by Ben Martin.

[/Cisco] permanent link

2008 Nov 23 - Sun

Seasonalality Timing System

In Mark Hulbert's November 17, 2008 article called The long-term reasserts itself, he mentions the Seasonality Timing System (STS) designed by Norman Fosback of the Fosback's Fund Forecaster Newsletter.

The STS is designed around the fact that "the stock market has a bullish bias around the trading sessions immediately prior to each exchange holiday as well as those at the turns of each month." He indicates that "STS followers will not get back into the stock market until the close next Monday, so as to be fully invested for the sessions on Tuesday and Wednesday, the two trading sessions prior to the market's Thanksgiving holiday."

According to those remarks, trading for this upcoming week should give us a rebound.

The picture turns less rosy with Paul Farrell's November 19, 2008 article 30 reasons for Great Depression 2 by 2011. Basically he says more spending with little or no increased income is a recipe for further disasters.

Peter Brimelow in a November 20, 2008 article called Bears' glass half empty or half full?, writes about Dow Theorist Richard Russell indicates that the primary bear market has been reconfirmed, and things are headed lower, perhaps to around the 7286 (the 2002 Dow Low) and 7470 (half the bull market peak), which we touched Thursday and Friday, but were saved by the news of Timothy Geithner, now president of New York Federal Reserve, would be Obamas's Treasury Secretary.

Corey Rosenbloom has confirmation of the Interesting Fibonacci Development. We have a level of support at about 7500, and a trading range up to a level of Fibonacci and psychological resistance of about 10,000.

But then more bad news could be around the corner. Lots of interesting Economic news coming this week: Existing Home Sales on Monday, Tuesday with GDP and Consumer Confidence, and then the day before the US Thanksgiving we have Durable Goods, Personal Income, Jobless Claims, Consumer Sentiment, and New Home Sales.

[/Trading/MarketNotes] permanent link

2008 Nov 14 - Fri

Receding Recession Indicator

The last time the Dow was at current levels looks to be back in May of 2003. But going back a bit more, it was May of 2002 that the Dow dropped below 10,000. It hit a low of 7600 during the beginning of October 2002. December and January were relatively 'happy' months before the Dow retested 7700 in March 2003. It took a steady rise till December 2003 to cross back above the magic 10,000. The year 2004 saw a few minor dips below 10,000, but nothing serious. October 2007 seeems to have been the recent peak at around 14,000. It declined bit by bit until September/October of this year when it bit the dirt.

In the last few weeks, it hit a low of 8451 around Oct 10, another lower low October 27 of about 8175, and retested with a mid-low at 8282 on November 12.

All this to say that we haven't made any recent lower lows. Yet. Leonard Novy says a symmetrical triangle is forming prior to a head and shoulders finalization at a still lower level. We shall see.

And if history offers any pattern for the future, we could stay at this level for six to twelve months. Things could improve over the next bit. Come next year, there are supposed to be more mortgage resets, which may cause another economy/financial hit, more people losing homes, and as a result jobs. After that, hopefully people's eternal optimism will start to kick in, and it is possible we could see a 10K Dow by the end of 2009 or first quarter 2010.

According to Donald Luskin, the bear market will be over when "stocks have rallied at least 20% from any given low point, over at least two calendar months". The pattern in December 2002 almost but not quite made the 20% criteria. It wasn't till after March 2003 did things conform to pattern.

Perhaps 2008/2009 may hold a similar pattern to 2002/2003.

[/Trading/MarketNotes] permanent link

2008 Nov 04 - Tue

Debian Lenney Exim Configuration

I can't recall, but I think by default, on a Debian Lenny install, the email server is configured to send email locally only. In order to get it to send email to other servers, the following command needs to be run to reconfigure Exim:

dpkg-reconfigure exim4-config

This is in response to a Non Delivery Report (NDR) of: Remote Domains Not Supported

[/OpenSource/Debian] permanent link

2008 Nov 03 - Mon

Multiple Switch Interfaces

Acktomic's genDevConfig creates it's Default files with one interface per view. For switches, it would be nice to see all interfaces presented on one page. To do this, I manually create a file to show these interfaces on one page. Here is a sample config:

target --default--
   devicename           = sw35
   directory-desc       = ""
   interface-name       = ""
   long-desc            = %short-desc%
   short-desc           = ""
   target-type          = cisco-interface

target sw35-ports
  targets = "/switches/sw35/FastEthernet0_1;
                /switches/sw35/FastEthernet0_2;
                /switches/sw35/FastEthernet0_3;
                /switches/sw35/FastEthernet0_4;
                /switches/sw35/FastEthernet0_5;
                /switches/sw35/FastEthernet0_6;
                /switches/sw35/FastEthernet0_7;
                /switches/sw35/FastEthernet0_8;
                /switches/sw35/FastEthernet0_9;
                /switches/sw35/FastEthernet0_10;
                /switches/sw35/FastEthernet0_11;
                /switches/sw35/FastEthernet0_12;
                /switches/sw35/FastEthernet0_13;
                /switches/sw35/FastEthernet0_14;
                /switches/sw35/FastEthernet0_15;
                /switches/sw35/FastEthernet0_16;
                /switches/sw35/FastEthernet0_17;
                /switches/sw35/FastEthernet0_18;
                /switches/sw35/FastEthernet0_19;
                /switches/sw35/FastEthernet0_20;
                /switches/sw35/FastEthernet0_21;
                /switches/sw35/FastEthernet0_22;
                /switches/sw35/FastEthernet0_23;
                /switches/sw35/FastEthernet0_24;
                /switches/sw35/GigabitEthernet0_1;
                /switches/sw35/GigabitEthernet0_2"
  short-desc = "Sw35 ports"

[/OpenSource/Debian/Monitoring/Cricket] permanent link

Cricket Summation

I have a number of routers, each with an interface to an upstream provider. I'd like to show a graph with the three providers aggregated. This is a config I did to do so:

target --default--
   devicename           = statistics
   directory-desc       = ""
   interface-name       = ""
   long-desc            = %short-desc%
   short-desc           = ""
   target-type          = standard-interface

target ProviderAggregate
  mtargets  = "/routers/router1/fastethernet2_0;
               /routers/router2/serial1_0;
               /routers/router3/atm2_0.1-aal5_layer"
  mtargets-ops = "sum()"
  short-desc "sum(Prov1, Prov2, Prov3)"

The above is the content of a file located in the /routers subdirectory. This turned out to be easier than I thought. The file needs the target --default-- section to start. Then one or more of the aggregate targets can be present. The 'mtargets' simply needs to know the directory and interface. The basic Default configurations in each subdirectory were created with devConfig tool from Acktomic.

[/OpenSource/Debian/Monitoring/Cricket] permanent link

2008 Nov 02 - Sun

Governmental Capitalism

The article "Why The Mortgage Crisis Happened" goes into some detail regarding the political background of the current financial situation originating in the US and spreading through out the world.

Some might say it was capitalism running rampant. But it looks more like the government trying to do the socialist thing and trying to get home ownership into the hands of those who can't/couldn't afford it. Isn't that what credit reports are for? In the words of Scott Francis: "The Community Reinvestment Act is a freaking joke. Why should a minority have a different set of rules and credit requirements than someone who has good credit?"

It is interesting that John McCain is painted in a positve light as knowing about the situation and attempted to do something about it. Obama, on the other hand, is painted in a bad light as being a perpetrator of the whole situation, and even accepted money to perpetuate the whole fiasco. And guess who it looks like the US will have as it's next president? Unless the undecided's all vote for McCain. Which shows my bias. But, perhaps in some version of the future, the US may field a third political strong enough to bite the hands of both the consumer and big business and make the decisions necessary to reduce the size of government, the debt, and everything else. Yeah, right. Too many self-interested groups.

According the article, business institutions needed to bury the good with the bad. However, it seems that the bad started to infect the good in a larger degree than was thought possible. Then (over-)leverage opened the whole festering wound. Please note that my remark regarding over-leveraged Wall Street places a good chunk of follow on blame on the rocket scientists who attempted to help monetize the government's problem. A commenter named Terry writes that the article fails to "fully examine the role of the conversion of mortgages into mortgage backed securities that were improperly rated by corrupted rating agencies and then sold into the marketplace. This, in combination with the looming problem with credit default swaps, is a much more significant pathogen in this disease process. " Terry indicates that this is an issue of deregulation, something of which the 'conservative commentator' doesn't cover.

According to Foreclosure Myths: Can the Media Handle the Truth?, the media is suggesting that the crisis was started through "Americans overwhelmed by circumstances beyond their control, from job losses to health problems to personal crises like divorce which ultimately cost them their homes." "the foreclosure problems began in mid-2006 when the nation.s unemployment rate was holding steady at a mere 4.6 percent. What triggered the crisis were not layoffs but an end of the rise in home prices." "Starting in mid-2006, foreclosures jumped sharply for both prime and subprime ARMs, but not for fixed-rate mortgages of any kind, including subprime ones." "ARMs draw a different kind of buyer, one who is often intent on selling or refinancing before rates re-set." "... buyers ... made speculative loans or were intent on flipping their homes, and they instead walked away from their mortgages at the first sign of home depreciation." "... purchases of homes for investment purposes that the buyer didn't intend to live in, amounted to a whopping 28 percent of all deals, and 22 percent in 2006."

According to the chart Real Estate Melt Down, making it easier to obtain sub-prime mortgages lead to an increase housing pricing relative to the average family income. Speculation as well as the laws of supply and demand would easily justify such a scenario.

What we end up with is a situation in which the home owners who got in early, have nice properties to their credit. Those in late couldn't ride the gravy train and got tossed overboard. This affects/affected builders, mortgage companies, bankers, and ultimately the general public due to the fact that whole statue of gold was attempting to be supported through feat of clay.

The stock market suffered as a result. Long term investors have felt this most tellingly. However, for those who know how to play the market in both up and down modes, are making huge sums of money through the market volatility. I've done some manual trades on both sides and have seen some appreciation, but I wish I was much better at seeing the possibilities.

Anyway, as a summary to the article regarding risk gone bad, in 2003 the government already knew about the issue, but due to partisan interests across the board, was unable to do anything:

History teaches that even the best minds in financial management cannot entirely eliminate 
risk. This was shown quite clearly by the severe difficulties encountered by Long-Term 
Capital Management several years ago. Nor do the GSE shareholders have the incentive to call 
for eliminating risk. The perception of a government bailout if things go wrong surely 
enhances any firm's willingness to take on risk and enjoy the associated increase in return. 
The savings and loan crisis of the 1980s illustrates the adverse incentive effects that can 
arise as a result of government guarantees. 

In an A Letter to Senator Obama by Tony Batman, he makes a very enlightening remark:

In other words, whatever you tax, you get less of; whatever you subsidize, you get more of.

The implication of this remark is that we need to somehow remove subsidies and come up with more creative mechansims for balancing the perceived inequalities in the market place.

Later in the same article, one possible solution is mentioned:

Increased taxes on the so-called 'rich' high income earners - and their businesses will affect the incomes 
of those who strive to move up from lower and middle classes to become high income earners!

In follow up to my mention of subsidy elimination a few paragraphs ago, another article mentiones that We Need Reagan + Friedman + Keynes. In summary, "during periods of crisis, sometimes you have to be a supply-sider (tax rates), sometimes a monetarist (Fed money supply), and sometimes a Keynesian (federal deficits).", ie, "Choose the best policies as put forth by the great economic philosophers without being too rigid."

[/Personal/Business] permanent link

Why The Mortgage Crisis Happened

Written by M. Jay Wells

[/Personal/Business] permanent link

2008 Nov 01 - Sat

MinHeap: C++ Template Implementation of Minimum Heap Algorithm

I am in the process of developing a trading application based in C++. I've realized that it would be useful to test code and algorithms off line, before connecting to a paper-trading system, or even a live scenario.

For code testing and algorithm testing, I need to send the quotes and trades I've collected through an order fullfilment simualation engine. As the solution I'm developing depends upon multiple symbols, I need a synchronization process to ensure the simulated quotes and trades are processed in the same temporal order as I recieved them in real time.

One option for solving this ordering problem would be to simply merge/sort all the vectors and process the single vector. I didn't like that idea as it wasn't amenable to incorporating time based events as they are generated by the fulfillment algorithm.

I ended up designing a 'carrier' to hold each vector. In addition to a few other house-keeping chores, The carrier manages the index into the vector for the latest datum to be processed. The various carriers are then linked into another vector, with the vector sorted in ascending date/time order based upon the current datum pointed to by the carrier.

My first kick at the can for maintaining the vector of carriers was a brute force sift down approach, with the sifting occuring after each datum was processed. This could turn out to be O(n*m/2) time where n is the number of carriers and m is the number of total datums across all carriers.

After looking at performance figures, I figured I could improve on this somewhat. The MinHeap algorithm looked like a suitable candidate.

The Standard Template Library implementation of the algorithm uses pops and pushes to get carriers into and out of the vector. This creates too much overhead. I basically needed a mechanism to maintain minheap order as the carriers are added to the vector, and then perform an in-place re-minheap when an datum from a carrier was 'consumed'. Once a carrier is exhausted of all datums in its vector, the carrier is moved to the end of the carrier vector with an 'archival' call.

Because, during the process of adding carriers to the vector, they are added at the end and sifted up, once carrier archival starts to occur, no further carriers can be added. A flag in the code is used to check for this condition.

The code is written for Visual Studio, but with a few minor mods, will hopefully run with GCC.

There are two template parameters: C, T. T is the basic type of the carrier, and C is used for determining the type of the operands for the lt operator in the SiftDown and SiftUp functions. I'm sure there is a more elegant way of handling the storage of pointers in the carrier vector and trying to perform comparisons of the class represented by the pointer. I havn't take the time to read up on that yet.

The following is the 'lt' operator of the class C:

static bool lt( CMergeCarrierBase *plhs, CMergeCarrierBase *prhs ) { return plhs->m_dt < prhs->m_dt; };

Here is the rest of the code:

#pragma once

#include <vector>
#include <assert.h>

// http://cis.stvincent.edu/html/tutorials/swd/heaps/heaps.html
// http://en.wikipedia.org/wiki/Binary_heap
// http://en.wikipedia.org/wiki/Heapsort
// http://people.cs.vt.edu/~shaffer/Book/C++2e/progs/minheap.h
// http://www.staroceans.com/minmaxHeap1.htm
// http://www.cppreference.com/wiki/stl/algorithm/is_heap  is_heap()

template<class T, class C> class CMinHeap {
public:
  CMinHeap<T,C>( size_t size );
  CMinHeap<T,C>( void );
  virtual ~CMinHeap( void );
  void Append( T );  // automatic sift up
  T RemoveEnd( void );
  T GetRoot( void ) { assert( 0 != m_cntActiveItems ); return m_vT.front(); };
  void ArchiveRoot( void ); // root item of no further use, swap to end and sift down
  void SiftDown( void ) { SiftDown( 0 ); }; // reorder new root after use and change
  bool Empty( void ) { return m_vT.empty(); };
  size_t Size( void ) { return m_vT.size(); };
protected:
  inline size_t Parent( size_t ix ) { return ( ix - 1 ) / 2; };
  inline size_t RightChild( size_t ix ) { return 2 * ( ix + 1 ); };
  inline size_t LeftChild( size_t ix ) { return ( 2 * ix ) + 1; };
  inline bool isLeaf( size_t ix ) { return LeftChild( ix ) >= m_cntActiveItems; };
  inline bool hasOneLeaf( size_t ix ) { return m_cntActiveItems == RightChild( ix ); };
  inline size_t ixLastItem( void );
  void SiftDown( size_t ix ); // from ix downwards, reordering item
  void SiftUp( size_t ix );   // from ix upwards towards root, when appending new items
  inline void Swap( size_t ix, size_t iy );
private:
  std::vector<T> m_vT;
  size_t m_cntActiveItems;
  bool m_bArchivalStarted; // prevents further Appends
};

template<class T, class C> CMinHeap<T,C>::CMinHeap(size_t size) 
: m_cntActiveItems( 0 ), m_bArchivalStarted( false )
{
  m_vT.reserve( size );
}

template<class T, class C> CMinHeap<T,C>::CMinHeap( void ) 
: m_cntActiveItems( 0 ), m_bArchivalStarted( false )
{
}

template<class T, class C> CMinHeap<T,C>::~CMinHeap() {
}

template<class T, class C> size_t CMinHeap<T,C>::ixLastItem() {
  assert( 0 < m_cntActiveItems );
  return m_cntActiveItems - 1;
}

template<class T, class C> void 
CMinHeap<T,C>::Append( T item ) {
  assert( !m_bArchivalStarted );
  m_vT.push_back( item );
  ++m_cntActiveItems;
  SiftUp( ixLastItem() );
}

template<class T, class C> T CMinHeap<T,C>::RemoveEnd( void ) {
  assert( !m_vT.empty() );
  T item = m_vT.back();
  m_vT.pop_back();
  return item;
}

template<class T, class C> void 
CMinHeap<T,C>::ArchiveRoot() {
  // swap with last item and SiftDown
  assert( 0 < m_cntActiveItems );
  m_bArchivalStarted = true;
  Swap( 0, ixLastItem() );
  --m_cntActiveItems;
  if ( 1 < m_cntActiveItems ) { // sift only if 2 or more items
    SiftDown();
  }
}

template<class T, class C> void 
CMinHeap<T,C>::Swap( size_t ix, size_t iy ) {
  assert( ix < m_cntActiveItems );
  assert( iy < m_cntActiveItems );
  T tmp = m_vT.at( ix );
  m_vT.at( ix ) = m_vT.at( iy );
  m_vT.at( iy ) = tmp;
}

template<class T, class C> void 
CMinHeap<T,C>::SiftUp( size_t ix ) {
  size_t cur = ix;
  size_t parent;
  while ( 0 != cur ) {
    parent = Parent( cur );
    if ( C::lt( m_vT.at( parent ), m_vT.at( cur ) ) ) {
      break; // done sifting
    }
    else { // swap and try again
      Swap( cur, parent );
      cur = parent;
    }
  }
}

template<class T, class C> void 
CMinHeap<T,C>::SiftDown( size_t ix ) {
  size_t cur = ix;
  while ( !isLeaf( cur ) ) {
    if ( hasOneLeaf( cur ) ) {
      size_t left = LeftChild( cur );
      if ( C::lt( m_vT.at( left ), m_vT.at( cur ) ) ) {
        Swap( left, cur );
        cur = left;
      }
      else {
        break;
      }
    }
    else { // has two leaves
      size_t right = RightChild( cur );
      size_t left = LeftChild( cur );
      // right side has shorter distance by default for same or greater
      bool bGoRight = !( C::lt( m_vT.at( left ), m_vT.at( right ) ) );
      if ( bGoRight ) {
        if ( C::lt( m_vT.at( right ), m_vT.at( cur ) ) ) {
          Swap( right, cur );
          cur = right;
        }
        else {
          break;
        }
      }
      else { // test with left
        if ( C::lt( m_vT.at( left ), m_vT.at( cur ) ) ) {
          Swap( left, cur );
          cur = left;
        }
        else {
          break;
        }
      }
    }
  }
}

// if we needed to build a heap from pre-assigned vector:
//  for (int i=n/2-1; i>=0; i--) siftdown(i);

[/OpenSource/Programming] permanent link

2008 Oct 29 - Wed

Choosing a GCC Version

Debian Lenny has a very recent GCC Compiler for those who wish to build things with the latest and greatest, which is v4.3 something or other. On the other hand, the kernel is built with GCC 4.1. Kernel modules, as a result need to be built with GCC 4.1. The VMWare tools are one example of this requirement.

I've finally figured out what the CC environment variable can be used for. By keeping gcc-4.3 as a standard default, one can use 'export CC=gcc-4.1' to make use of the older GCC v4.1 compiler for compiling kernel modules.

[/OpenSource/Debian] permanent link

Root Login Through KDE

It typically isn't a good idea to login into a server as root. A basic Debian install, in non-expert mode, requires an alternate user name to be set up. Only in an expert mode install can one skip that step. If one installs in expert mode, and skips the step of installing a regular user, then trying to log in to KDE after an 'apt-get install kde' is impossible. Well, almost impossible.

An adjustment needs to be made to a file called kdmrc, which can be found in /etc/kde3/kdm. The next version of KDE will probably change the directory in which it is located, so use 'locate kdmrc' to figure out the location if not found where it is expected.

Setting AllowRootLogin to true will allow a root login after a reboot. As I find I do more root stuff than not, this is a nice little trick.

[/OpenSource/Debian] permanent link

2008 Oct 27 - Mon

Interactive Brokers TWS on Linux

Installing Interactive Brokers Traders Workstation on Linux is relatively painless. It is probably best to stick with Java Runtime 1.5 rather than using 1.6. The IB forums mention problems with 1.6, and Think Or Swim does not like 1.6 either: apt-get install jre-java5-jre. This will require non-free and contrib in the /etc/apt/sources.list file. After installation, 'update-alternatives --config java' will get the right version set.

The IB suggested command line has problems with hsqldb.jar. I use the following command line as an alternative:

java -cp \
jts.jar:pluginsupport.jar:jcommon-1.0.12.jar:jfreechart-1.0.9.jar:jhall.jar:other.jar:riskfeed.jar:rss.jar:/usr/share/java/hsqldb.jar \ 
-Xmx256M jclient.LoginFrame . &

[/Trading] permanent link

NVidia/FX1700 on HP xw8600 with Debian/AMD64 Lenny

I recently obtained an HP xw8600 workstation with an E5420 series processor and two NVidia FX1700 dual head DVI video cards. This provides the ability for four monitors, which in this case, are four HP LP2065 1600x1200 pixel 20" LCD monitors.

Getting these monitors to work together in XP was easy.

To do the same in Debian/Linux was, well, less easy. There was much trial and error. I think Gnome and KDE interact with the drivers differently. I think KDE has more flexibility, but ultimately I was able to get the driver/monitor combination working independently of which GUI solution I was using.

The fly in the ointment is that I want my monitors in portrait mode, rather than landscape mode. It was difficult figuring out if I needed Xinerama or xrandr to the job. More on this further down.

There are two basic steps to my installation: get the NVidia drivers installed, then get everything configured.

I used the current Lenny beta of Debian/AMD64 Net Installation (AMD64 is another misnomer, one needs the AMD64 version to run the Intel64 version, wouldn't there be a better name for this?) to get the operating system and graphics installed. I then went to Debian Wiki NvidiaGraphicsDrivers for inspiration on how to get the drivers installed. I would recommend not going there. The decision tree and instruction sets are not very well designed.

Instead, go to Pendrivelinux. The instructions are dead simple and accurate. The only change is that I downloaded the more recent 64 bit NVidia driver: NVidia NVIDIA-Linux-x86_64-177.80-pkg2.run.

Knowing about "apt-get install build-essential linux-headers-$(uname -r)" is good for installing a minimal build system after getting a minimal operating system installed.

Something else good to know is the ctl-alt-backspace to exit the GUI to a command line. All except gdm wants to start back up again. The solution for that is to open a terminal window, do a "ps aux | grep gdm" and then kill the process.

The hard part was getting the driver configured with portrait mode monitors. For configuration, the tool at Applications-> System Tools -> NVIDIA X Server Settings can be used. I had to resort to a few minor manual edits of /etc/X11/xorg.conf to make things work. In any case, some documentation stated that Xinerama wouldn't work for rotated monitors, and that I should use xrandr. In my case, the opposite is true. xrandr doesn't work, and Xinerama does work with a couple of manual xorg.conf tweaks.

When testing, a useful console command to start the GUI is: startx -- -layout Layout0

Here is my final xorg.conf file:

Section "ServerLayout"
    Identifier     "Layout0"
# a little out of order, but that is the order in which I plugged them
    Screen      0  "Screen0" 0 0
    Screen      1  "Screen1" 2400 0
    Screen      2  "Screen2" 1200 0
    Screen      3  "Screen3" 3600 0
    InputDevice    "Keyboard0" "CoreKeyboard"
    InputDevice    "Mouse0" "CorePointer"
EndSection

Section "Files"
EndSection

Section "Module"
    Load           "dbe"
    Load           "extmod"
    Load           "type1"
    Load           "freetype"
    Load           "glx"
EndSection

Section "ServerFlags"
    Option         "Xinerama" "1"
EndSection

Section "InputDevice"
    # generated from default
    Identifier     "Mouse0"
    Driver         "mouse"
    Option         "Protocol" "auto"
    Option         "Device" "/dev/psaux"
    Option         "Emulate3Buttons" "no"
    Option         "ZAxisMapping" "4 5"
EndSection

Section "InputDevice"
    # generated from default
    Identifier     "Keyboard0"
    Driver         "kbd"
EndSection

Section "Monitor"
    Identifier     "Monitor0"
    VendorName     "Unknown"
    ModelName      "HP LP2065"
    HorizSync       30.0 - 92.0
    VertRefresh     48.0 - 85.0
    Option         "DPMS"
EndSection

Section "Monitor"
    Identifier     "Monitor3"
    VendorName     "Unknown"
    ModelName      "HP LP2065"
    HorizSync       30.0 - 92.0
    VertRefresh     48.0 - 85.0
EndSection

Section "Monitor"
    Identifier     "Monitor1"
    VendorName     "Unknown"
    ModelName      "HP LP2065"
    HorizSync       30.0 - 92.0
    VertRefresh     48.0 - 85.0
EndSection

Section "Monitor"
    Identifier     "Monitor2"
    VendorName     "Unknown"
    ModelName      "HP LP2065"
    HorizSync       30.0 - 92.0
    VertRefresh     48.0 - 85.0
EndSection

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "Quadro FX 1700"
    BusID          "PCI:96:0:0"
    Screen          0
EndSection

Section "Device"
    Identifier     "Device1"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "Quadro FX 1700"
    BusID          "PCI:96:0:0"
    Screen          1
EndSection

Section "Device"
    Identifier     "Device2"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "Quadro FX 1700"
    BusID          "PCI:128:0:0"
    Screen          0
EndSection

Section "Device"
    Identifier     "Device3"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "Quadro FX 1700"
    BusID          "PCI:128:0:0"
    Screen          1
EndSection

Section "Screen"
    Identifier     "Screen0"
    Device         "Device0"
    Monitor        "Monitor0"
    DefaultDepth    24
#    Option         "RandRRotation" "on"
#    Option         "Rotate" "right"
    Option         "Rotate" "CW"
    Option         "TwinView" "0"
    Option         "TwinViewXineramaInfoOrder" "DFP-0"
    Option         "metamodes" "DFP-0: 1600x1200 +0+0; DFP-0: nvidia-auto-select +0+0; DFP-0: 1440x900 
+0+0; DFP-0: 1400x1050 +0+0; DFP-0: 1400x1050_70 +
0+0; DFP-0: 1400x1050_60 +0+0; DFP-0: 1280x1024 +0+0; DFP-0: 1280x1024_75 +0+0; DFP-0: 1280x1024_60 +0+0; 
DFP-0: 1280x960 +0+0; DFP-0: 1280x960_60 +0+0;
DFP-0: 1280x800 +0+0; DFP-0: 1280x768 +0+0; DFP-0: 1152x864 +0+0; DFP-0: 1152x864_75 +0+0; DFP-0: 1152x768 
+0+0; DFP-0: 1024x768 +0+0; DFP-0: 1024x768_75
 +0+0; DFP-0: 1024x768_70 +0+0; DFP-0: 1024x768_60 +0+0; DFP-0: 960x720 +0+0; DFP-0: 960x600 +0+0; DFP-0: 
960x600d60 +0+0; DFP-0: 928x696 +0+0; DFP-0: 89
6x672 +0+0; DFP-0: 840x525 +0+0; DFP-0: 832x624 +0+0; DFP-0: 800x600 +0+0; DFP-0: 800x600_85_0 +0+0; 
DFP-0: 800x600_75 +0+0; DFP-0: 800x600_72 +0+0; DFP-
0: 800x600_60 +0+0; DFP-0: 800x600_56 +0+0; DFP-0: 800x600d70 +0+0; DFP-0: 800x600d65 +0+0; DFP-0: 
800x600d60 +0+0"
    SubSection     "Display"
        Depth       24
    EndSubSection
EndSection

Section "Screen"
    Identifier     "Screen3"
    Device         "Device3"
    Monitor        "Monitor3"
    DefaultDepth    24
#    Option         "RandRRotation" "on"
#    Option         "Rotate" "right"
    Option         "Rotate" "CW"
    Option         "TwinView" "0"
    Option         "metamodes" "DFP-1: nvidia-auto-select +0+0"
    SubSection     "Display"
        Depth       24
    EndSubSection
EndSection

Section "Screen"
    Identifier     "Screen1"
    Device         "Device2"
    Monitor        "Monitor1"
    DefaultDepth    24
#    Option         "RandRRotation" "on"
#    Option         "Rotate" "right"
    Option         "Rotate" "CW"
    Option         "TwinView" "0"
    Option         "metamodes" "DFP-0: nvidia-auto-select +0+0"
    SubSection     "Display"
        Depth       24
    EndSubSection
EndSection

Section "Screen"
    Identifier     "Screen2"
    Device         "Device1"
    Monitor        "Monitor2"
    DefaultDepth    24
#    Option         "RandRRotation" "on"
#    Option         "Rotate" "right"
    Option         "Rotate" "CW"
    Option         "TwinView" "0"
    Option         "metamodes" "DFP-1: nvidia-auto-select +0+0"
    SubSection     "Display"
        Depth       24
    EndSubSection
EndSection
 

[/OpenSource/Debian] permanent link

2008 Oct 24 - Fri

Single User Mode

From Single User Mode, I like the following extract for root password recovery:

If you get asked for the root password when entering single user mode use 'init=/bin/bash' on the kernel append line which should 
boot the machine into a bash console where you can get read/write access to your /etc/shadow file. You can then either run `passwd` 
or edit the shadow file directly to put in an empty string. This allows you to reboot the machine into its normal environment and 
login as root leaving the password blank and then run the `passwd` program to set the root password. This should really only be done 
when the machine is detached from any networks.

[/OpenSource/Debian] permanent link

2008 Oct 10 - Fri

Boost Build

Boost C++ Library has a large number of interesting modules. Most of the modules are header only templates readily used on many platforms and many recent version compilers.

A few of the modules, such as Regex and DateTime, need to be compiled into static or dynamic link libraries. To do so, requires the use of Boost's build tool: bjam.

To build all variations of the libraries, single/multithreaded, debug/release, and static/dynamic, the command is simple (example for windows):

bjam --toolset=msvc --build-type=complete stage

Finding out how to build a subset of the libraries is a bit harder. The options I use are:

bjam --toolset=msvc variant=debug,release threading=multi link=static stage

[/OpenSource/Programming] permanent link

2008 Aug 28 - Thu

Console to Cisco Device from FreeBSD

I'm now involved in running an ISP. An existing ISP. One that has been through severval hands already. One that has many strange corners and alleys. Unknown servers. Inconsistent switch configurations. I could go on, but suffice to say, my job is to make it all work.... better. And in the mean time, bring my self up to speed on a bunch more technologies. I've been mostly a Linux hack todate. I get to add FreeBSD and NetBSD to my list. Baby steps first.

The task of connecting a Cisco device to the serial port of a FreeBSD computer and communicating with it turned out to be quite easy. So for my reference (which I found at O'Reilly BSD DevCenter):

cu -l /dev/cuaa0 -s 9600

This uses the first serial port at 9600 bits per second. Connection and remote access is as simple as that. To exit the session:

~.

[/OpenSource/FreeBSD] permanent link

2008 Aug 22 - Fri

DNS Tools

For my reference, a command for looking at ownership of w.x.y.z:

 dig +trace  z.y.x.w.in-addr.arpa.

DNS Operations List

DNS List: NameDropers: IETF DNSEXT working group.

DNS Operatins

[/OpenSource/Linux] permanent link

2008 Aug 07 - Thu

Labour Saving Devices (Software Wise)

Today's Dr. Dobb's Report via email from Jonathan Erickson has a reference to Michael Abrash's book Graphics Programming Black Book. Based upon the book's table of content, the content covers many interesting algorithms in and outside of graphics programming.

However, to download the book, one has to download over 76 individual files. What a pain, especially if one is using Windows.

Actually, it isn't so bad. I manually downloaded the preludes and afterwords. Cygwin came to the rescue for automating the chapter downloads. By starting a Cygwin shell, putting the following content in a file called 'getch', and running it, I was able to automate the download of the 70 individual chapters. For those running Linux or BSD directly, then this little script can be used directly.

for (( i = 1; i <= 70; i++ ))
do
  wget http://www.byte.com/abrash/chapters/gpbb$i.pdf
done

The 'wget' program is a useful tool for downloading web pages without using a browser. And by creating a for loop with a variable substitution, one can iteratively download each of the chapters.

[/OpenSource] permanent link

2008 Aug 06 - Wed

Market Notes for August 6, 2008

This article really isn't about today. Nor is it about yesterday. It is about Monday. But becuase I only read Mark Hulbert's colummn today, which he penned yesterday, which is about Monday's 332 point gain, I've dated my article for today.

Mark's point of the day revolves around another rule of thumb: the day was not a 9-to-1 up day. This type of day is when 90% of the volume of shares that rose or fell in price was in shares that went up. The term was coined by Martin Zweig, who used to publish several investment newsletters. On Monday, for the NYSE, it was only an 8.1-to-1 day.

According to his 1986 book called "Winning on Wall Street", Zweig states "Every bull market in history, and many good intermediate advances, [has] been launched with a buying stampede that included one or more 9-to-1 up days".

In Mark's article of yesterday, he mentions that valuations are still a little high. As such, a bull market may not be in the offing yet. So far July 15 is considered our low of the current bear market.

I think because of additional mortgage resets happening later this year, we may not be completely ready for bullish thinking. Maybe a nice little volatile rally, but we may see some more lows.

[/Trading/MarketNotes] permanent link

2008 Jul 30 - Wed

SmartQuant QuantDeveloper & DataCenter Release

SmartQuant has released a revision to DataCenter and QuantDeveloper. DataCenter and QuantDeveloper are at the following revision levels:

DataCenter
Version 3.0.3 (30-Jul-2008) 

QuantDeveloper Enterprise Edition
Version 3.0.3 (30-Jul-2008)  

QuantDeveloper Source Code
Version 3.0.1 (21-Apr-2008) 
* Recent Versions available through 
  version control 

[/Trading/SmartQuant/Releases] permanent link

2008 Jul 16 - Wed

Market Notes: 2008/07/15
I guess the markets really are bearish about what is happening in the world. The Dow gapped downwards on open and played with 10850 for a bit. It seems that with Bernanke's Congressional testimony later in the morning being hard and to the point, the markets had their edge taken off and rebounded to a little in the positive zone. OPEC indicated that their demand forcasts are being reduced, which caused a $5 dollar drop in Brent crude in the midst of a $10 high low swing for the day as traders took their profits. Usually the Dow has an exact opposite swing, but narry a blip occurred, and actually closed down for the day under 11000.

[/Trading/MarketNotes] permanent link

2008 Jul 12 - Sat

Freddie Mac and Fannie Mae

The markets are not very patiently waiting for all the sub-prime mortgages to reset later this year and next year. I've always thought that it will be during that time period in which we'll find out if we are in a repression, depression, or a recession.

It seems we have lots of near term worries. Various and sundry resets must already be taking place. The financial system seems to be stressing all over the place. I'm curious to know just how leveraged the industry is. After all, in the end, the money has to be some place. If leverage is the problem, someone is going to be left holding the bag, and I guess that is starting to happen now.

This reminds me of the great internet expansion a number of years ago. Every who thought they knew what a fibre optic cable was, starting laying the stuff across land and under water. Most if not all of those companies reorgainized or got sold at pennies on the dollar. The smart guys on the sidelines smiled and spent their pennies on valuable infrastructure. Mean while the original investors and vendors probably didn't fair so well. in the end, after the market settled, we have high capacity bandwidth at reasonable prices (well for North America anyway, we in Bermuda still pay an arm and a leg for the privilege, although that should change with the new consortium laying new fiber later this year).

That story leads me into today's newsletter by John Mauldin. He is saying that sub-prime resets aren't our only problem. Orbourous is eating its tail. Lenders to lenders and lenders to corporate beings are starting to cause problems. A figure like $1600 billion dollars of losses in the international banking system are being bandied about. That is a terrific amount of business failure, residential collapse, and bad business judgement. Is there a figure somewhere that suggests what the equivalent of a world wide Gross Domestic Product might be? (I know that appears to be a contradiction of terms, but with the ever shrinking world, there is an element of realism there). With a WW GDP, this type of loss could be put in to perspective. Grasping at straws, I came up with one form of perspecitive. According to the US Federal Reserve Statistical Release, that is $300 Billion more than the M1 money supply and about 23% of the M2 money stock measure.

Anyway, his previous articles had some concrete examples as to what sort of numbers losses were based upon. In this article, there seems to be a bunch more hand waving going on. Perhaps the Bridgewater Associates report to which he refers offers up some concrete basis for their opinion.

Finding the headwaters of investment sources is what John Mauldin's friend David Kotok specializes in. In a recent newsletter, he is saying that Freddie and Fannie (F&F), between them, hold about $5000 Billion in mortgages. Me, coming from the outback, think that a $150,000 mortgage is big. Having one of that size, and if I've used the correct number of zeros in my calculations, that could mean about 30 million mortgages. To stretch the statistic even further, that would be a mortgage for 1 out of every 10 US residents. That is a lot of cash flow to them and to their holders of sub-paper.

Both of the authors tend to agree that holding paper from F&F is not too risky being that the mortgages that they do hold are reasonably sturdy, and they both agree that holding shares is valueless. So, so long as the cashflow meets payment expectations, things shouldn't be too bad.

However, all this is contrary to what the notable publications such as WSJ are publishing, so no wonder we bounced off the Dow 11,000 level yesterday. I think Mauldin even joked about the 9XXX level not being too far off the mark in his article.

One other thing Mauldin mentioned is that he is doing a survey. As a reward for filling out his survey, he provides a link to speech in which he talks about how the markets might re-arrange themselves. Perhaps this might be similar to what happened with the post fibre-laying companies... will the new credit/debt institutions be valuable because of what they got for pennies on the dollar?

I wrote this article in order to set a baseline of expectations of what is to come. Will we, indeed be seeing more losses, more than what the subprime fiasco has caused directly? In which direction are the markets headed and what will be their prime motivator? Will it be more credit problems? I'll be able to look back here and hopefully see what happened when we start our descent into the 10K category.

[/Trading/MarketNotes] permanent link

Open Source: Sonic Visualizer

This blog entry should actually go under the title of Open Source Package of the Day, but because it solves a different problem for me, it comes under the heading of Lighting.

Back in April, I did lighting for a student Jazz group. As part of the show, I wanted to do my own little ditty. My desire was to do a mini light show set to music. In some shows, I've seen the lighting designer loosely sync the lighting display to the music theme. My goal for this particular display was to ave the light show visibly keep time to the beat of the music.

I considered quite a number of ways to do this. Some more manual than others. A key feature had to do with my inherent 'laziness'. Why should I have to manually redo and retranscribe the beat when I should be able to get that out of the music file itself? Easier said than done. I first turned to Cakewalk's Sonar Producer Edition to do waveform analysis for me. With the waveform editor, it is easy to find 'note starts'. Those note starts don't necessarily carry the rhythm. Another tool within Sonar allows one to filter to certain frequencies. I was interested in the bass beats. That helped, but would still very time consuming to identify and place a note by hand.

I finally cheated. I found a fully functional MIDI file of Pinky and the Brain. All the instruments were nicely laid out, plus it had a kick drum track. This kick drum track was monotone, as a kick drum track should be. This made it dead easy to set off cues from the MIDI input in Light Factory. I ended up with a 20 to 30 second subsection with 157 cues.

The little ditty worked very well and the audience loved it.

Now that I've figured out music timing, light timing, and integrating the two, not to mention that amount of time one can spend on making it look easy, I've been thinking of turning the concept up a notch, and maybe doing a larger production.

Up till now, I figured I'd only be able to this with MIDI tracks, as the timing and instruments are all laid out, and I really didn't/don't want to go back to waveform analysis, at least being limited to what Sonar provides.

Enter Sonic Visualizer. With this, one can take a mucic track and run the standard note start tools on it. The cool feature is the spectrogram views. One can actually see the vibrato of the violin, the beat of the drum, the complex weave of tones of a symphony, .... The program also has a MIMDI annotation feature.

I'll have to manually key in the MIDI notes I want for cue changes, but with the visible segmentation of notes, instruments, and voices, creating and coordinating a lighting show tied to music could be a delight to do. Lots of time will be eaten, but with Sonic Visualizer, Light Factory, and a lighting show visualizer like Capture Console, it could be quite an experience, not only from a design point of view but from an audience pleasure perspective.

Ok, back to daydreaming. In a previous article, I wrote about a light console. Not any ordinary console, but one which is simply a piece of acrylic with infrared sensors on it. This would allow realtime multiple touch live input. By including this in the show, on stage, a light show would become live performance art in itself. I think a live spectogram would be an interesting light show addition in itself. Are there such shows out there?

[/Personal/Lighting] permanent link

2008 Jul 11 - Fri

Alpha Generation Platforms

Sometime ago, SmartQuant sold their QuantDeveloper code to QuantHouse. I now see in an article at Wall Street & Technology, QuantHouse listed as one of five vendors who have Alpha Generation Technology. QuantHouse must have done some additional work on the platform. QuantDeveloper definitely fit the defintion of a workflow platform:

• Data acquisition and preparation
• creation of the initial alpha discovery model
• back testing the model using historical data sets
• analyzing the results of back-testing and fine tuning the model
• simulation with live data
• coding the quant research model into production for the live trading environment

Alpha is defined as "excess risk-adjusted returns measured above a benchmark".

Key attributes of the packages reviewe included items such as:

• seamless integration with data sources and databases for rapid data capture
• transformation and storage for analysis
• ease of use in creating back testing and simulation environment
• detailed documentation of model creation process
• charting, reporting and visualization tools
• ease of integration with leading statistical packages
• offers a straight through processing feature that enables quants to move from idea generation to order generation in a reduced time frame
• offers a codeless environment for rapid strategy development

QuantDeveloper did offer up all those features but that last one. QD is actually a C# development environment disguised as an Alpha Generation Platform.

I'm not promoting or demonting QD here. I did use the package for a couple of years and have since migrated to a custom C++ platform, which I think, in the end, is going offer very similar capabilities. What with QuickFix tested against OpenFix, QuantLib for the math, and a myriad of other integrated abilites, it may just have a chance to be seen in the big leagues.

Yes I do spend much of my time day dreaming. But there is a drop of reality there somewhere.

As a side note, the article threw out a bunch of names. I'll have to follow up on what these do sometime: "real-time high performance databases such as Vhayu, KX and OneTIck. On top of that, analytics and statistical packages are required, such as MATLAB, S+ and R, as well as optimization tools like Northfield, BARRA, Morningstar, Ibbotson, etc., and EMSs/OMSs like Portware, FlexTrade, OrcSoftware, Aegis Software and Tethys, etc.".

[/Trading/SmartQuant/Articles] permanent link

2008 Jul 03 - Thu

Upgrade from Eclipse Europa to Ganymede (with painful Subversion)

Today, I upgraded from Eclipse/CDT Europa to Eclipse/CDT Ganymede. (CDT meaning C++ Developer Tools). The Eclipse upgrade was painless: download the Eclipse/CDT package, expand it, and start eclipse from within the directory. After pointing it to my workspace, everything was there. Nicely simple.

My subversion client was an entirely different story. For the Europa installation, everything came from the tigris site and worked well. For the Ganymede installation, there are now two sites involved, and I'm not sure which is what. I think the tigris site can now be ignored (for the time being). In the installation instructions somewhere, one needs to go to the Polarion site for their client. There are Eclipse update links there.

However, what I assumed to be workable defaults of using the JavaHL library on Debian turned out to be non-workable. The solution was not to use the JavaHL client but use the SVNKit client.

My key problem is that my SVN repository requries an ssh public/private key. The JavaHL library, if or when it would or wouldn't load, I'm not sure what I was seeing, but I could only see the option for user name and password authentication.

It would have been nice if the Subversion/Polarion/Eclipse guys would all get together and make it straight-forward in terms of which libraries from which sites need to be download. If they imply that the JavaHL libraries should be downloaded, please make it painless to get the paths set and ensure the binaries are present. After 15 million lines of code, you'd think that would be a small task to accomplish.

[/OpenSource/Programming] permanent link

2008 Jun 18 - Wed

Concurrency aka MultiThreading

A number of my projects are approaching the phase where some of their feature sets will work better with some form of background processing. In a trading application, plowing through historical data on thousands of symbols looking for patterns would be best left to a background task, rather than rendering the user-interface frozen during the, well, duration. For Radius, with a listener on an accounting port, and one on an authorization port, the two threads need to coordinate access to resources.

Windows has a native threads API, but that isn't necessarily portable between operating systems. My trading application is in Windows, and the Radius application in on Linux. Using the same API on both platforms wuold be cool.

As I already use the Boost tools in their various forms, Boost::Thread would be a good candidate. The Boost documentation isn't exactly overflowing with examples. Dr. Dobb's Portal saves the day with an article dating from May 2002 entitled The Boost.Threads Library. It has good examples covering the basics:

• Thread Creation (boost::thread)
• Mutexes (boost::mutex), which protects one thread from another
• Condition Variables (boost::condition), which are good for getting data into and out of a thread
• Thread Local Storage (boost::thread_specific_ptr), for keeping thread specific storage separate from other threads
• Once Routines (boost::call_once), for making sure statics are initialized once and only once

For mutexes, protecting code regions can be as easy as declaring a mutex:

boost::mutex CodeProtectionMutex;

And then putting a scoped lock in the code encountered by multiple threads:

{ // some scope some where
  // ... some code
boost::mutex::scoped_lock lock(CodeProtectionMutex);
  // .. some more code, which is protected by the lock
} // the scope exit, no unlock is required as the destructor does the work

After reviewing the examples, making use of the Boost documentation should be an easier task. As such, boost::thread documentation should be reviewed anywa as boost::thread has gone through some changes since that article.

Paul Bridger has also written a tutorial on multithreading making use of the boost::thread class. The navigation through the tutorial isn't the greatest, but the content is good.

Making use of boost::thread as a base, Philipp Henkel has written threadpool. It has been brought up to date for use with boost v1.35. It provides a dead easy solution to making use of a limited number of worker threads to carry out tasks:

    pool tp(2);  //create a 2 thread pool
    
    // Add some tasks to the pool.
    tp.schedule(&first_task);
    tp.schedule(&second_task);
    tp.schedule(&third_task);  // this task waits until of the other two completes

In the similar vein to Henkel, Ted Yuan has a boost::thread based C++ Producer-Consumer Concurrency Template Library.

Zoltán Porkoláb has written an article on Distributed Programming and Metaprogramming in C++. It has many examples and goes into some additional examples for boost::thread. His article also introduces bind and tuples, which are good backgrounds to boost::lambda.

Back to boost for a second. You can't find it from the boost home page, but here is a good link to Boost Libraries Listed Alphabetically.

boost::thread is a basic threading library. Going above and beyond multi-threading grunt work, Intel's Thread Building Blocks has higher level constructs for getting multiple threads going. For example, it has a 'for' construct for simultaneously executing multiple elements of the for statement. Good tutorials and background information can be read through Kevin Farnham's Blog.

Building further on the Threading Building Blocks is something of simulating interest: go parallel looks to be chock full of content related to multi-core and multi-threaded programming.

From the theoretical perspective, I came across an HP paper called Foundations of the C++ Concurrency Memory Model, and written by Hans-J. Boehm and Sarita V. Adve. I havn't read it all the way through, but at some time, I think the bibiliography may be a worthy read in itself.

[/OpenSource/Programming] permanent link

2008 Jun 16 - Mon

Keyword Matching (non-text streams)

In a previous blog article, I presented the beginnings of a Keyword Lookup class. This article takes that work, turns it into a template and makes it useful for longest match lookups.

I'll have to compare this lookup library with what a C++ map or unordered_map does for lookup speed. I'm hoping that when a comparison is made, that this routine is indeed faster. I'm thinking it might be because, even though a map will do a binary search through it's map, complete strings are compared at each step. With this library, keyward patterns are added to a rooted tree of characters, possibly reducing the amount of time spent mactching.

I must admit that at each match step, there is a linear search performed through a list of likely character candidates. To imporve the search, I've been thinking that once the pattern tree has been created, it could be sorted so that each step search can be done with a binary search. This will be something for next time.

In the meantime, this routine does work for finding maximum matches. For example when performing a long distance rate lookup, this will find the most specific rate code from a list of various length candidates, something which is difficult to do with a map.

This keyword match algorithm is template based. 'class T' is the type to be returned upon a successful match. It can be an index, a pointer, a number, or anything else suitable. The class constructor requires an initializer... basically a value to be given the equivalent meaning of NULL. On no match, this value is returned. Use 'AddPattern' to add patterns and their associated 'meanings'. Use FindMatch to perform the lookup.

// this is kind of a subset of Aho Corasick algorithm
// only full keyword matching, no text searches
// no on failure coding

#ifndef CKEYWORDMATCH_H_
#define CKEYWORDMATCH_H_

#include <string>
#include <vector>
#include <stdexcept>
#include <iostream>

template<class T> class CKeyWordMatch {
public:
  explicit CKeyWordMatch<T>( T initializer, size_t size );
  virtual ~CKeyWordMatch(void);
  void ClearPatterns( void );
  void AddPattern( const std::string &sPattern, T object );
  T FindMatch( const std::string &sMatch );
  size_t size( void ) { return m_vNodes.size(); };
protected:
    T m_Initializer;
  struct structNode {
    size_t ixLinkToNextLevel;  // next letter of same word
    size_t ixLinkAtSameLevel;  // look for other letters at same location
    T object;  // upon match, (returned when keyword found)
    char chLetter;  // the letter at this node
    explicit structNode( T initializer ) : ixLinkToNextLevel( 0 ), ixLinkAtSameLevel( 0 ), 
      object( initializer ), chLetter( 0 ) {};
  };
  std::vector<structNode> m_vNodes;
private:
};

template<class T> CKeyWordMatch<T>::CKeyWordMatch( T initializer, size_t size )
: m_Initializer( initializer )
{
  m_vNodes.reserve( size );
  ClearPatterns();
}

template<class T> CKeyWordMatch<T>::~CKeyWordMatch(void) {
  m_vNodes.clear();
}

template<class T> void CKeyWordMatch<T>::ClearPatterns() {
  m_vNodes.clear();
  structNode node( m_Initializer );
  m_vNodes.push_back( node ); // root node with nothing
}

template<class T> void CKeyWordMatch<T>::AddPattern( 
              const std::string &sPattern, T object ) {
  std::string::const_iterator iter = sPattern.begin(); 
  if ( sPattern.end() == iter ) {
    throw std::invalid_argument( "zero length pattern" );
  }
  size_t ixNode = 0;
  size_t ix;
  bool bDone = false;
  while ( !bDone ) {
    char ch = *iter;
    ix = m_vNodes[ ixNode ].ixLinkToNextLevel;
    if ( 0 == ix ) { // end of chain, so add letter
      structNode node( m_Initializer );
      node.chLetter = ch;
      m_vNodes.push_back( node );
      ix = m_vNodes.size() - 1;
      m_vNodes[ ixNode ].ixLinkToNextLevel = ix;
      ixNode = ix;
    }
    else { // find letter at this level
      bool bLevelDone = false;
      size_t ixLevel = ix;  // set from above
      while ( !bLevelDone ) {
        if ( ch == m_vNodes[ ixLevel ].chLetter ) { 
          // found matching character
          ixNode = ixLevel;
          bLevelDone = true;
        }
        else {
          // move onto next node at this level to find character
          size_t ixLinkAtNextSameLevel 
            = m_vNodes[ ixLevel ].ixLinkAtSameLevel;
          if ( 0 == ixLinkAtNextSameLevel ) {
            // add a new node at this level
            structNode node( m_Initializer );
            node.chLetter = ch;
            m_vNodes.push_back( node );
            ix = m_vNodes.size() - 1;
            m_vNodes[ ixLevel ].ixLinkAtSameLevel = ix;
            ixNode = ix;
            bLevelDone = true;
          }
          else {
            // check the new node, nothing to do here
            // check next in sequence
            ixLevel = ixLinkAtNextSameLevel;
          }
        }
      }
    }
    ++iter;
    if ( sPattern.end() == iter ) {
      if ( m_Initializer != m_vNodes[ ixNode ].object ) {
        throw std::domain_error( "Pattern already present" );
      }
      m_vNodes[ ixNode ].object = object;  // assign and finish
      bDone = true;
    }
  }
}

template<class T> T CKeyWordMatch<T>::FindMatch( const std::string &sPattern ) {
  // traverse structure looking for matches, object at longest match is returned
  std::string::const_iterator iter = sPattern.begin(); 
  if ( sPattern.end() == iter ) {
    throw std::runtime_error( "zero length pattern" );
  }
  T object = m_Initializer;
  size_t ixNode = 0;
  size_t ix;
  bool bDone = false;
  while ( !bDone ) {
    char ch = *iter;
    ix = m_vNodes[ ixNode ].ixLinkToNextLevel;
    if ( 0 == ix ) {
      bDone = true;  // no more matches to be found so exit
    }
    else {
      // compare characters at this level
      bool bLevelDone = false;
      size_t ixLevel = ix;  // set from above
      while ( !bLevelDone ) {
        if ( ch == m_vNodes[ ixLevel ].chLetter ) {
            if ( m_Initializer != m_vNodes[ ixLevel ].object )
                object = m_vNodes[ ixLevel ].object;
          ixNode = ixLevel;
          bLevelDone = true;
        }
        else {
          ixLevel = m_vNodes[ ixLevel ].ixLinkAtSameLevel;
          if ( 0 == ixLevel ) {  // no match so end
            bLevelDone = true;
            bDone = true;
          }
        }
      }
    }
    ++iter;
    if ( sPattern.end() == iter ) {
      bDone = true;
    }
  }
  return object;
}

#endif /*CKEYWORDMATCH_H_*/

[/OpenSource/Programming] permanent link

Mean Reversion Thoughts

While still putting together the code for a trading solution, I've been thinking about what algorithms to implement for a trading strategy. I have access to live intra-day tick and quote data, so mean-reversion aka contrarian strategies seem like interesting candidates.

In the course of manual trading, I've learned that one needs to keep track of a number of items: current portfolio costs, current holding costs, existing profit/losses, expected market direction, current market location, external influences. This is a lot to do manually. Hence the desire to implment tools to automate, or even semi-automate the process.

A paper by Subramanian Ramamoorthy called A strategy for stock trading based on multiple models and trading rules discusses a state space mechanism for determining how to manage the portfolio composition. Another item he brings to the foreground is a description of the Sharpe Ratio, a ratio which helps one to keep profit consistent rather than widely dynamic.

Using different terminology, the makers of NeoTicker have a blog with an article called Counter-Trend Trading with Simple Range Exhaustion System. The key point, which could be hard to do, is "most counter-trend traders will try to time their entries as close to the extreme reversal points as possible to maximize the profits and minimize the risk exposures". Using multiple time frame charts, and reading the tape, along with some possibly helpful technical analysis tools, it might be possible to home in on the zones of reversal.

Working my way into a little scalping in the futures, an older article at Interactive Brokers explains the birth of the Dow Mini Futures. Some interesting points:

• "try to identify the leader in a group and how its price movement can help us predict movement in others in the group"
• "we start to trade it by hand so we can get a better understanding of the nuances in that particular trade"
• "We have a trader and a programmer trade together for a while and then we start the process of automation. We define our risk parameters and write the rules that we feel give us an opportunity to be profitable."
• "In our back testing we saw that if we were patient it would be profitable for us. The hard part was learning to be patient because our other successful trades were very high frequency. In the mini-sized Dow we may be in and out of 5 to 10 trades in a less than minute."
• hedge the mini dow with the underlying basket of stocks
• "We don't have scalping targets. We generate a theoretical value and make markets based purely on that value If we our pricing is accurate and we should naturally be able to scalp."
• "In the Dow because the bid-ask spread is so tight most of our profits are generated from trading."
• "he dow has a much tighter spread compared to the mini-spu. Also it is much easier to watch the stocks in the underlying basket to ascertain their effect on the future."
• "The Russell tends to be trendier than other indices."

[/Trading/AutomatedTrading] permanent link

2008 Jun 15 - Sun

Adaptive Arrival Price

A keynote lecture at the April 7th Algorithmic Trading Conference in London was by Mr. Julian Lorenz of ETH Zurich. The abstract for his lecture reads as follows:

Electronic trading of equities and other securities makes heavy use of "arrival price" algorithms, that balance the market impact cost of rapid execution against the volatility risk of slow execution. In the standard formulation, mean-variance optimal trading strategies are static: they donot modify the execution speed in response to price motions observed during trading. We show that with a more realistic formulation of the mean-variance tradeoff, with no momentum or mean reversion in the price process, substantial improvements are possible by using dynamic trading strategies. We develop a technique for computing optimal dynamic strategies to any desired degree of precision. The asset price process is observed on a discrete tree with a arbitrary number of levels. We introduce a novel dynamic programming technique in which the control variables are not only the shares traded at each time step, but also the maximum expected cost for the remainder of the program; the value function is the variance ofthe remaining program. The resulting adaptive strategies are"aggressive-in-the-money": they accelerate the execution when the price moves in the trader's favor, spending parts of the trading gains to reduce risk. The improvement is larger for large initial positions.

I think I'll add 'arrival price algorithms' to my key word searches. The above extract was from a search on 'mean reversion trading system algorithms'.

[/Trading/AutomatedTrading] permanent link

2008 Jun 09 - Mon

Stocks & Commodities, 2008/06

In a recent issue of Technical Analysis of Stocks and Commodities, there was an interview with Tom Busby. A number of his comments struck home with some things I've learned. He also introduced a few more things about which I should think.

He noted that trading can be a twenty four hour operation. There is always some market open to trade. The world starts off with the Nikkei and the Hang Seng in the far east. In Europe, primary markets are CAS, FTSE, DAX and the Swiss. I'd say in today's market the IPE, with the Brent Crude Futures, is also important. Here in the west, we have the morning New York market and the afternoon California market.

Busby made mention that 'market open' is an important event. As such, it is important to know the time each of the markets open. I've been working on an algorithm that selects a series of instruments, selects a direction and lets the instruments run. I've been wondering what to set for an exit though. Busby, in the interview, suggests exiting once a third of ATR (Average True Range) has been reached. I'm not sure why he would use ATR (which accounts for any opening gap) rather than just the daily average range. Assuming one gets in sometime in the open, and exits by the end of the day (in order to eliminate what gaps in the wrong direction can do to one's portfolio), then using ATR doesn't seem quite right.

Anyway, To set the tone for a trading day, he suggests some benchmark indexes to be watched. Seven, which he calls the Seven Sisters are:

• S&P
• NASDAQ
• Dow Jones Indexes
• DAX
• Crude Oil
• Long Bonds
• Gold

As for micro-signals, he uses three kinds, with each needing to be in the same direction:

• Volume
• Tick (gainers vs loser)
• Trend

To finish things off, he suggests splitting an entry into three parts:

• Tick Part: the trickiest part of the entry based upon the three variables above
• Trade Part: with confidence building, try to make twice the reward vs risk
• Trend Part: capture the full movement of the day

[/Trading/ReadingMaterial] permanent link

2008 Jun 06 - Fri

SmartQuant QuantDeveloper & DataCenter Release

SmartQuant has released a revision to DataCenter and QuantDeveloper. DataCenter and QuantDeveloper are at the following revision levels:

DataCenter
Version 3.0.2 (06-Jun-2008) 

QuantDeveloper Enterprise Edition
Version 3.0.2 (06-Jun-2008)  

QuantDeveloper Source Code
Version 3.0.1 (21-Apr-2008) 
* Recent Versions available through 
  version control 

[/Trading/SmartQuant/Releases] permanent link

Wt, Some Build Modifications

Back on 2007/10/03, I wrote about installing Wt (a C++ library and application server for developing and deploying web applications) on a Debian server. I've revised things a little bit since thing while building Wt v2.1.3.

In this case, I build with the newly released version of the Boost libraries: 1.35. ASIO is now included in Boost, so some build steps can be removed.

Prerequisites are little changed but for a different library for gd:

apt-get install gcc
apt-get install zlib1g
apt-get install zlib1g-dev
apt-get install libbz2-dev
apt-get install libgd2-noxpm-dev
apt-get install cmake
apt-get install libfcgi-dev
apt-get install libapache2-mod-fastcgi
apt-get install libssl-dev

The web site and repository for Wt have changed, so CVS commands will be a bit different:

cvs -d :pserver:anonymous@cvs.webtoolkit.eu/opt/cvs login
cvs -z3 -d :pserver:anonymous@cvs.webtoolkit.eu/opt/cvs co wt

I've changed the cmake/build a little bit so the results go into /usr/local/wt/include and /usr/local/wt/lib:

cmake -D DEPLOYROOT=/var/www/wt -D WEBUSER=www-data -D WEBGROUP=www-data \
-D BOOST_DIR=/usr/local \
-D BOOST_COMPILER=gcc42 \
-D BOOST_VERSION=1_35 \
-D BOOST_INCLUDE_DIR=/usr/local/include/boost \
-D BOOST_LIB_DIR=/usr/local  \
-D BOOST_DT_LIB_MT=/usr/local/lib \
-D BOOST_DT_LIB=/usr/local/lib \
-D BOOST_FS_LIB=/usr/local/lib \
-D BOOST_FS_LIB_MT=/usr/local/lib \
-D BOOST_PO_LIB_MT=/usr/local/lib \
-D BOOST_REGEX_LIB_MT=/usr/local/lib \
-D BOOST_SIGNALS_LIB_MT=/usr/local/lib \
-D BOOST_THREAD_LIB=/usr/local/lib \
-D BOOST_ASIO_INCLUDE_DIR=/usr/local/include/boost  \
-D SHARED_LIBS=ON \
-D CONNECTOR_FCGI=OFF \
-D CONNECTOR_HTTP=ON \
-D EXAMPLES_CONNECTOR=wthttp \
-D WTHTTP_CONFIGURATION=/etc/wt/wthttpd \
-D CONFIGURATION=/etc/wt/wt_config.xml \
-D CMAKE_INSTALL_PREFIX=/usr/local/wt \
.

During make install, an error regarding CMakeFiles arises. The secret, that I know, is to remove the line which includes CmakeFiles in src/Ext/cmake_install.cmake, and restart 'make install'. The install should complete normally.

The library directory /usr/local/wt/lib will need to be added to /etc/ld.so.conf, and then run ldconfig to update things.

Remember to review the Ext widgets deployment page as there are some additional files to be downloaded and installed from Ext JS.

[/OpenSource/Debian/Development] permanent link

2008 Jun 04 - Wed

PostgreSQL Upgrade 8.2 to 8.3

Back in Febrary, I wrote a longish article on how to upgrade PostgreSQL. That article is outdated. An upgrade can now take place with two lines:

pg_upgradecluster -v 8.3 8.2 main
pg_dropcluster 8.2 main

The first copies the older version 8.2 files to the new 8.3 files directory. It does any modifications necessary. The second line then removes the old stuff.

[/OpenSource/Debian] permanent link

OpenSSH Issues

In light of the not so recent news regarding the vulnerability of openSSH in Debian, many systems have had to be patched and inter-machine keys changed.

Via Steven Rosenberg's Site I learn that a simple 'apt-get update && apt-get dist-upgrade' will update the necessary files on my system. Also in the blog entry is a reference to DRONEBL which is another black list site dealing with root compromised sites. A commenter posts the following interesting remarks about further protecting a server:

If you aren't running fail2ban or denyhosts, you should. Both will detect brute force attempts and deny connections from the attacker for a time. If you feel uncomfortable automatically banning hosts for failed logins, you can weakly configure whichever you choose to allow 20 or more failed attempts before banning. There's no reason any authenticated service should tolerate brute force attempts, in my humble opinion.

Finally, there are services, such as the DroneBL dnsbl, which have honeypot servers set up to detect brute force attempts and add them to a blacklist. You can use the "aclexec" directive in hosts.deny to query this blacklists before allowing clients to connect, to prevent connections from known brute force attackers. See http://headcandy.org/rojo/ for a suitable script to call via aclexec (view the source for the checkdnsbl script for usage instructions), and see the man page for hosts_options for more info.

Running 'ssh-vulnkey -a' showed that there were a couple keys that needed to be deleted and/or redone.

Debian has a WIKI with good information regarding the problem, affected programs, and utilities to help determine where the problems are.

If weak keys have been copied to other non-Debian hosts, the keys need to be removed from those hosts as well.

[/OpenSource/Debian] permanent link

2008 May 31 - Sat

Decision Trees, Automated Trading, Simulations, and Strategies

A paper called Stock Picking via Nonsymmetrically Pruned Binary Decision Trees by Anton V. Andriyashin discusses a method for picking stocks for inclusion in a portfolio. By integrating technical analysis with binary decision trees, the author indicates that "BNS clearly outperforms the traditional approach according to the backtesting results and the Diebold-Mariano test for statistical significance", where BNS is Best Node Strategy. David Aronson of Evidence Based Technical Analysis fame may call the use of some the technical indicators as 'so much snake oil', the paper, at its heart, does describe a methodology for selecting a potentially profitable portfolio if one can use alternate forms of trading signals.

Alternate forms of decision tree based automated trading can be found in two papers by German Creamer and Yoav Freund called Automated Trading with Boosting and Expert Weighting and A Boosting Approach for Automated Trading. These represent algorithms used in the Penn-Lehman Automated Trading Project. Anyway, the two papers get down and dirty with some of the indiators they use in their trading simulation. Their bibliography references a number of good sources of information.

In the PLAT paper, here are a few strategies worthy of further investigation:

• Case-based reasoning applied to the parameters of the SOBI strategy (see text for SOBI description).
• Predictive strategy using money ow (price movement times volume traded) as a trend indicator.
• Market-maker that positions orders in front of the nth orders on both books.
• Mixture of a Dynamically Adjusted Market-Maker which calibrates by recent volatility, and a trendbased predictive strategy.
• Sells on rising prices, buys on falling prices.
• Trades based on relative spreads in the buy and sell books, interpreting small standard deviation as a sign of codence.
• Simple predictive strategy using total volumes in buy and sell books.

Peter Stone's group has done well with the PLAT simulations. His papers, with this one as a example, Two Stock-Trading Agents: Market Making and Technical Analysis have many good implentable ideas for an automated trading strategy. Outside of the world of finance, general algorithmic bidding and optimization strategies are described in The First International Trading Agent Competition: Autonomous Bidding Agents. Another interesting Peter Stone paper called Designing Safe, Profitable Automated Stock Trading Agents Using Evolutionary Algorithms They discuss the concept that common trading rules have weaknesses under various trading conditions. By identifying the conditions, and adaptively switching among rules, trading results can be improved. One more Peter Stone supported effort is the poster: Safe Strategies for Autonomous Financial Trading Agents: A Qualitative Multiple-Model Approach.

Through the use of evolutionary reinforcement on data to which us mere mortals have no access, M.A.H. Dempster has a number of related papers. The bibilographies may be good sources of further inspiration:

• INTRADAY FX TRADING: AN EVOLUTIONARY REINFORCEMENT LEARNING APPROACH
• Adaptive systems for foreign exchange trading
• A real-time adaptive trading system using genetic programming

In a sort-of-related paper, Robert Almgren and Julian Lorenz provide an insight into Adaptive Arrival Price. A couple of extracts from their abstract:

• Electronic trading of equities and other securities makes heavy use of .arrival price. algorithms, that determine optimal trade schedules by balancing the market impact cost of rapid execution against the volatility risk of slow execution.
• We show that with a more realistic formulation of the mean-variance tradeoff, and even with no momentum or mean reversion in the price process, substantial improvements are possible for adaptive strategies that spend trading gains to reduce risk, by accelerating execution when the price moves in the trader.s favor.

Now for a really un-related paper: A market-induced mechanism for stock pinning. The authors suggest that some stock prices can be pinned at strike prices on option expiration dates. As various market participants cover their positions with options and the related underlying securities, some interesting market dynamics unfold.

[/Trading/ReadingMaterial] permanent link

The Joy of Volatility

I initially had this embedded in my follow on article, but I think the information in this paper bears further scrutiny and testing, in regards to what could be classified as what I think is called pairs trading. I guess the secret is in the selection of the pairs.

The paper is by Dempster/Evstigneev/Schenk-Hoppé, and called The Joy of Volatility. They take a coin flipping strategy to picking a couple of assets. They show that the volatility is a positive benefit to portfolio profitability in a dynamic rebalancing strategy versus a buy and hold mentality. A couple of key quotes though:

Poverty is the inevitable fate of the passive investor.

Consider making an investment according to a simple active management style: buying or selling assets so as to always maintain an equal investment in both. On average, wealth will double in 80 periods and grow without limits. This investment style rebalances wealth according to a constant proportions strategy. It succeeds, where buy-and-hold fails, because of the volatility of asset returns.

However, as with any investment advice, a word of caution is in order: Constant proportions strategies do well in the long term but, over short time horizons, their superior performance cannot be guaranteed!

[/Trading/ReadingMaterial] permanent link

2008 May 29 - Thu

Evaluating Inter-Process Communication Frameworks

I'm reposting some comments regarding IPC frameworks that I made to the Boost-Users listserve today. It is in response to someone making unsubstantiated remarks regarding the relative merits of ACE and Boost, and another looking for some substatiated remarks. What follows are some substantiated remarks, based upon my personal experience with it and several other libraries.

I've started working on a number of distributed system projects. As a consequence, I started looking for distributed system libraries. References to ACE were most pervasive. I implemented a number of trial applications with the library. That was after plowing through relevant sections in the three primary ACE reference books. That was a good learning experience, if only to find out the various patterns in distributed architecture definition. I had the inter-process/inter-server communications (which only sent simple stuff) working well within ACE's Acceptor/Connector framework. ACE has a number of other patterns one can use. I was really impressed with the fact that the examples I used from the books worked as advertised, and I was able to bend them to my will.

ACE is based upon an interaction of classes, macros, and templates. One has to spend some time with the environment in order to become proficient with it. It has a large API. A number of lower level API's upon which higher level API's are based. For example the Acceptor/Connector uses constructs described earlier in the books.

Once I had my basic communications going, I realized I needed to get some form concrete messaging infrastructure in place. I had an impression that TAO, which is a layer above ACE, would be quite extravagant to implement, with it being an implementation of the CORBA specification. I wanted something a little lighter (a whole lot lighter actually).

As I worked through that project, I started hearing about ASIO, indirectly through some other libraries I was using. ASIO is now a member of Boost. I read a review somewhere that ASIO is a 'modern' replacement for ACE. If you want to get into real template structures and Boost oriented philosophy, I'd say that is a valid statement. I'd also say that ASIO is 'more to the point' and straight forward than is ACE, at least for the things I want to accomplish. But like ACE, ASIO is the basic communications infrastructure, no real messaging capability, which is what distributed computing is all about. ASIO turned out to be a little harder to get my head wrapped around as it uses a number of advanced C++ and Boost related idioms. For a run-of-the-mill C++ programmer, ACE would be better. For someone steeped in the power and obscurity of advanced C++, and is looking to advance their skill set, ASIO would be better.

I came across RCF - Interprocess communication for C++, which is a messaging framework riding atop of ASIO. Flexible, lightweight, and to the point. I worked through the examples and things worked as advertised. It has the encryption, publisher/subscriber, oneway/twoway idioms, and a few other nifty features.

At the same time I was doing that, seemingly coincidently, I learned a few more interesting facts. Going into this, I realized that I need a message dispatcher/proxy, some decent failover techniques, and some additional event handling for non-IPC related activities.

Someone suggested ICE from www.zeroc.com for an RCF-like solution, but working to a larger scale. I've heard that the library's originator is someone who spent much time on CORBA standards and redid the concept without the 'benefit' of committee involvement. I think the library has all the bases covered in terms of lightweight message handling, dispatching, resiliency, and higher level distributed processing philosophies. The drawback is that it will have a steeper learning curve than would an implementation using RCF. I like RCF, but I think I.m going to have to tilt towards ICE (itself, like RCF, developed and focused towards C++ in a multiple license environment).

On the non-IPC front, Qt's QCoreApplication looks to be a good substrate on which to build event driven daemons.

In the end, I think my solutions are going to involve:

• ZeroC's ICE for primary inter-process communications
• Qt QCoreApplication as a base for daemon development (which has built-in stuff for threads/locks, slots/signals)
• Wt, a C++ based web toolkit for distributed GUI development
• Boost Libraries to fill in all the holes
• a little legacy layer 3/4 ACE in one library I'm using, but with some work, I think I can convert the ACE stuff to ASIO

[/Personal/SoftwareDevelopment/CPP] permanent link

2008 May 28 - Wed

Put Me To Sleep Reading Material

Someone in some data provider's forum was making mention of doing order flow analysis in Excel through Interactive Brokers, and the person felt that they weren't getting enough data. Which is true, Interactive Brokers sends data based upon what is necessary for someone viewing a screen, not based upon some automated data hungry automaton looking to crunch full data feeds.

That got me to thinking and to reading more about order flow analysis. This gets in to market orders, limit orders, bid/ask spreads, order books, market makers, rational traders, uninformed traders, instantaneous impact of variable sized market orders, as well as whole raft of other micro-economic activity that comes with high frequency trading.

Marco Avellaneda and Sasha Stoikov and recently released a paper entitled High-frequency trading in a limit order book, with another version of the same thing here. They develop some interesting equations on determining a bid/ask spread in the midst of a moving market, based upon a market maker's inventory and risk capability. I'm wondering if that is what BATS does for their trading capability.

Karl Ludwig Keiber has a paper called Price Discovery in the Presence of Boundedly Rational Agents. In the paper, he discusses some market maker concepts and what they deal with. Momentum as well as mean reversion are discussed in the context of bid/ask spread and price discovery. There is a minor discussion regarding adverse selection during a transition from momentum to reversal trading on page 25 which may be of some value. The cross over between reversal and momentum is a weakness in my trading.

Bruce Mizrach has a paper called The next tick on Nasdaq. Although a recently published paper, he uses data from 2002. The paper goes into some history of market making, limit books, and how Nasdaq grew up. Some of his interesting observations:

• This paper asks a surprisingly simple but neglected question: does the entire order book help predict the next inside quote revision?
• Lillo and Farmer (2004) find that orders on the London Stock Exchange follow a long memory process.
• Bouchaud et al. (2002), while analysing the Paris Bourse, found a power law for the placement of new limit orders and a hump shape for the depth in the order book.
• Weber and Rosenow (2005) find a log linear relationship between signed market order flows and returns on Island.
• I find, for example, that the number of bids or offers is more important than the quoted depth.
• In general, I find that the bids (offers) away from the inside increase the probability of a down (up) tick.
• The last result I obtain is that this volatility decreases with larger market capitalization and the presence of more market makers.
• Traders call the market makers or ECNs that frequently appear on the inside market the .ax., and they claim that taking note of the ax's activity is informativey.
• for example, the advice from the Daytrading University at http://www.daytrading-university.com/ samplesson4ways.htm. ..Even with the ECN routing that mm.s [market makers] use to hide their order flow, there.s still plenty of profitable trading to be had by correctly: (1) Avoiding buying when a major mm/ax is selling (e.g. if you see MSCO and MLCO both sitting on the inside ask you probably shouldn.t buy if their bid is three levels outside the market) and (2) .Shadowing. the ax.s buying/selling behavior, if you see that all else looks okay, e.g. no suspiciously strong ECN buying/selling on INCA/ISLD...
• The presence of a particular participant does not by itself indicate that they are significant contributors to subsequent quote revisions though.
• Looking more closely at individual participants, there are some interesting results. When ARCA takes the inside bid, the next tick is more likely to be a downtick than an uptick in 65 of 71 cases.
• When ARCA takes the inside ask, there is an uptick in 63 of 73 instances
• The effect of specific participants in the small cap market differs from the large caps. ARCA has a negative impact from the bid in all 41 cases in which it is statistically significant.
• A vector autogression can be inverted into its moving average representation, and one can then compute impulse responses functions. In our model of trades and quotes, these have the interpretation of market impact functions, or the effect on stock returns of an unexpected buy order arriving into the market.
• It can also be explained in an order driven market by what Biais et al. (1995) call the .diagonal effect. in which they observe that a limit order that improves the inside bid (ask) is more likely to be followed by another limit order which increases (decreases) the inside bid (ask). A similar diagonal effect for trades is present as well. The negative serial correlation in the small caps suggest that the quote revision process for that group can be explained without assuming informed traders,
• As in many auction designs, additional buy (sell) side interest makes the next price change more likely to be an uptick (downtick). Biais et al. (1999) observe this behaviour even in an environment in which quotes are only indicative. Similarly, in the period in which quotes are firm, the authors find that additional depth on one side of the book helps predict the appearance of additional liquidity on the same side of the book.
• The number of buyers and sellers, I find, is almost always more important than quoted depth.
• Aggregate depth, either at the inside market, or as a weighted average of the demand curve, is also helpful, and this information is surprisingly persistent. In general, the results are more successful for large cap stocks than small caps.
• Quotes away from the inside are generally not informative. Large numbers of buyers (sellers) at tiers away from the best bid (offer) are more likely to result in a downtick (uptick).
• The model of trades and quotes presented also produces dynamic estimates of market impact. The impact of a buy order can be determined beyond its impact on the current spread. The estimates appear to vary sensibly with standard measures of liquidity.

I wonder if the above snippets could be coded as in an expert system.

In Relation between Bid-Ask Spread, Impact and Volatility in Order-Driven Markets by Wyart/Bouchaud/Kockelkoren/Potters/Vettorazzo, the BATS philosophy of infinitesimal market-making can be expressed in terms of spread and the instantaneous impact of market orders. They indicate that there is an empirical correlation between the spread and the volatility per trade. As mentioned in one of the other papers, they confirm that the main determinant of the bid-ask spread is adverse selection. They also confirm that volatility comes from trade impact. The paper has an extensive bibliography worth looking into. There is an interesting corrolary in the conclusion, namely that "when the volatility per trade is large, the risk of placing limit orders is large and therefore the spread widens until limit orders become favorable."

[/Trading/ReadingMaterial] permanent link

2008 May 25 - Sun

A Keyword Matching Algorithm

There are a number of well known algorithms out there for taking in a set of keywords and matching them against test. Aho and Corasick comes to mind, as does the Wu Manber algorithm (the latter I've implemented, and the code resides elsewhere on this site).

For another project, I didn't need something quite so fancy. Actually two projects come to mind. One is that I have a input comma separated value file which includes stock symbols, a description, and the associated exchange. I wanted to keep statitics on what is read in on an exchange basis. My first kick at the can on this was to implement a string look up table using

In another part of the file, to do some special computing, and in a fit of late night programming, I went so far as to implementing a multi-stage strcmp on each exchange to do something special for each.

I knew from the outset, that would be horribly inefficient, but I didn't have a nice algorithm at hand to do it with.

After being reminded of Aho Corasick in another context, I decided to take another stab at improving efficiency. I ended up implementing half of Aho Corasick's algorithm, the 'go' function. Since I'm always matching from the beginning of the string, and am not stepping through text, the exclusion of the 'fail' function works quite well, and keeps the code smaller.

At some point in time the code can be coverted to something using templates in order to accept various string types, various element matches, and return codes.

Since the 'fail' function isn't implemented, the lookup table can be updated dynamically without being rebuilt from scratch.

The library is quite easy to use. With AddPattern, add new keywords along with some sort of index or object. When using FindMatch with the target keyword, the appropriate object will be returned on a successful match. If no match is found NULL will be returned. A modification to the class would add a default object for use when no matches are found.

Here is the header file.

#pragma once

// Copyright (2008) Ray Burkholder
// Matching against multiple keywords

#include <vector>
#include <string>

class CKeyWordMatch {
public:
  CKeyWordMatch(void);
  virtual ~CKeyWordMatch(void);
  void ClearPatterns( void );
  void AddPattern( const std::string 
&sPattern, void *object );
  void *FindMatch( const std::string 
&sMatch );
protected:
  struct structNode {
    size_t ixLinkToNextLevel;  // next letter of same word
    size_t ixLinkAtSameLevel;  // look for other letters at same location
    void *object;  // upon match, (returned when 
keyword found)
    char chLetter;  // the letter at this node
    structNode() : ixLinkToNextLevel( 0 ), ixLinkAtSameLevel( 0 ), 
      object( NULL ), chLetter( 0 ) {};
  };
  std::vector<structNode> m_vNodes;
private:
};

Here is the code file:

#include "StdAfx.h"
#include "KeyWordMatch.h"

// Copyright (2008) Ray Burkholder
// Matching against multiple keywords

#include <stdexcept>

CKeyWordMatch::CKeyWordMatch(void) {
  ClearPatterns();
}

CKeyWordMatch::CKeyWordMatch(size_t size) {
  m_vNodes.reserve( size );
  ClearPatterns();
}

CKeyWordMatch::~CKeyWordMatch(void) {
  m_vNodes.clear();
}

void CKeyWordMatch::ClearPatterns() {
  m_vNodes.clear();
  structNode node;
  m_vNodes.push_back( node ); // root node with nothing
}

void CKeyWordMatch::AddPattern( 
              const std::string &sPattern, void *object ) {
  std::string::const_iterator iter = sPattern.begin(); 
  if ( sPattern.end() == iter ) {
    throw std::runtime_error( "zero length pattern" );
  }
  size_t ixNode = 0;
  size_t ix;
  bool bDone = false;
  while ( !bDone ) {
    char ch = *iter;
    ix = m_vNodes[ ixNode ].ixLinkToNextLevel;
    if ( 0 == ix ) { // end of chain, so add letter
      structNode node;
      node.chLetter = ch;
      m_vNodes.push_back( node );
      ix = m_vNodes.size() - 1;
      m_vNodes[ ixNode ].ixLinkToNextLevel = ix;
      ixNode = ix;
    }
    else { // find letter at this level
      //ix = m_vNodes[ ixNode ].ixLinkToNextLetter;  // already set
      bool bLevelDone = false;
      size_t ixLevel = ix;  // set from above
      while ( !bLevelDone ) {
        if ( ch == m_vNodes[ ixLevel ].chLetter ) { 
          // found matching character
          ixNode = ixLevel;
          bLevelDone = true;
        }
        else {
          // move onto next node at this level to find character
          size_t ixLinkAtNextSameLevel 
            = m_vNodes[ ixLevel ].ixLinkAtSameLevel;
          if ( 0 == ixLinkAtNextSameLevel ) {
            // add a new node at this level
            structNode node;
            node.chLetter = ch;
            m_vNodes.push_back( node );
            ix = m_vNodes.size() - 1;
            m_vNodes[ ixLevel ].ixLinkAtSameLevel = ix;
            ixNode = ix;
            bLevelDone = true;
          }
          else {
            // check the new node, nothing to do here
            // check next in sequence
            ixLevel = ixLinkAtNextSameLevel;
          }
        }
      }
    }
    ++iter;
    if ( sPattern.end() == iter ) {
      if ( NULL != m_vNodes[ ixNode ].object ) {
        std::runtime_error( "Pattern already present" );
      }
      m_vNodes[ ixNode ].object = object;  // 
assign and finish
      bDone = true;
    }
  }
}

void *CKeyWordMatch::FindMatch( const std::string &sPattern ) {
  // traverse structure looking for matches
  std::string::const_iterator iter = sPattern.begin(); 
  if ( sPattern.end() == iter ) {
    throw std::runtime_error( "zero length pattern" );
  }
  void *object = NULL;
  size_t ixNode = 0;
  size_t ix;
  bool bOpFound = true;
  bool bDone = false;
  while ( !bDone ) {
    char ch = *iter;
    ix = m_vNodes[ ixNode ].ixLinkToNextLevel;
    if ( 0 == ix ) {
      bDone = true;  // no more matches to be found so exit
    }
    else {
      // compare characters at this level
      bool bLevelDone = false;
      size_t ixLevel = ix;  // set from above
      while ( !bLevelDone ) {
        if ( ch == m_vNodes[ ixLevel ].chLetter ) {
          ixNode = ixLevel;
          bLevelDone = true;
        }
        else {
          ixLevel = m_vNodes[ ixLevel ].ixLinkAtSameLevel;
          if ( 0 == ixLevel ) {  // no match so end 
            bLevelDone = true;
            bDone = true;
          }
        }
      }
    }
    ++iter;
    if ( sPattern.end() == iter ) {
      object = m_vNodes[ ixNode ].object;
      bDone = true;
    }
  }
  return object;
}

[/Personal/SoftwareDevelopment/CPP] permanent link

2008 May 23 - Fri

A Half Hearted Day

Last night I got some chart software programming accomplished. I can now see bars, trades and quotes. Over the weekend my task to get some indicators on to them, particularily pivots, Bollinger Bands of two or three different time frames, volume historgrams, and a zig zag indicator. A little further down the road, the zig zag indicator will be used for 'snapping' trend/support/resistance lines in to place to help solidify some chart patterns.

I looked in on COIL again this morning. I got sidetracked watching it and didn't realize the rest of the market had opened. When I did notice what was happening, a lot of things went south. It was all well and good that I didn't do anything. There will always be another trading day, and hopefully for Tuesday I can have my basket trading in place.

That is, I'm hoping to finish off the order entry bit that talks to Interactive Brokers. In doing so, I can then finish the integration my order basket tracking. Each evening, I run three different stock selection filters and come up with a total of about 40 different symbols with associated entry parameters. If all goes well, I can do some semi-automated trading: ie let the computer get my entries in first thing in the morning, then I can monitor the profit curve and start setting stop-loss points to generate automated exits.

[/Trading/Diary/D200805] permanent link

RCF - Interprocess Communications for C++

For a couple of distributed computing projects, I've been trying to come up with a feasible and easy to use method for making applications talk to each other, whether they be on the same machine or across a network.

I started off doing some work with Douglas C. Schmidt's ACE: The ADAPTIVE Communication Environment. I plowed through ACE's three primary programming books to see what would be the best bit of the environment I would need. I ended up implementing a demo with the Acceptor - Connector framework, just to see how things worked.

I then started on thinking on the messaging structure and the event handling structures. ACE's mixture of macros and classes turned out to be a little overwhelming for what I wanted to accomplish.

During my stint with ACE, I started to use ASIO, from the Boost libraries. I was first introduced to ASIO through working with WT: WebToolKit. I used Wt as a frontend to a voip call sign in server.

The next step in the evolution is to present a real time call summary report to authorized management as the calls are authenticated, authorized, and accounted for from a Radius server. This means sending call detail messages from the Radius server to a central dispatch server, and then publish to active web clients (with the clients written with Wt).

As Wt uses ASIO for its underlying network communications, and I had read a remark somewhere that ASIO is the new improved ACE, I started to look into it as the mechanism for my inter-process communications. I even got a good chunk of messaging infrastructure written as was about to get it testing when I found it was all for nought.

I came across RCF - Interprocess Communications for C++. It is a library that has been in development for the last few years by a talented fellow by the name of Jarl Lindrud. The library has implemented all the stuff that I only dreamed about doing: publish/subscribing, stream encryption, payload filtering, and any number of other nifty features.

I had a few painful moments in getting the library built. After a couple of messages back and forth to the author, I realized I was trying to build the whole thing into a static library rather than using an 'include' technique to get the platform specific files built.

The client and server examples built and ran without a hitch. I must admit that I was impressed by the examples in the ACE books as well: they compiled and ran with little or no messing about.

The RCF library is better because it deals with serializing native values back and forth, something that ACE only accomplishes when you get into the TAO and CORBA levels of the environment.

So now with Boost (which includes ASIO), RCF (which uses ASIO), and Wt (which also uses ASIO), I think I have all the interprocess tools I need to make my modules talk to each other. Now I can get on with the meat of my projects.

[/Personal/SoftwareDevelopment/CPP] permanent link

2008 May 22 - Thu

Trading Notes: 2008/05/22

I've been trading most days during the month of May. I've been using Interactive Brokers as a broker, and have been using their BookTrader to execute my trades. Regarding things I've learned while using the BookTrader, I'll leave that for another post.

My trading account (real money) is up by 9.4% since April 28, when I first started manual trading, and so far, knock on wood, I've had all positive days, some more positive than others, some a lot more work than others.

I think it is time to keep track of what I do and what I see so I can ensure I don't do the same mistakes more than once.

Limit orders is what I started with. Using a mostly contrarian strategy, I've been able to find some profit areas. I have been caught a couple of times when the market kept going in the wrong direction, and I was getting in deeper and deeper. Those were the rough days where I had to do tricky trading, and through mostly luck, the symbol recovered enough that I could end positive.

With that said, it is now time to figure out the price levels at which to do reversal orders. I'm setting up some charting to help me with that, and hope to have it done for trading next week.

The news over the last 12 hours has been heavy with the news of the large leap in oil (COIL), traded on IPE. I've been watching the 2008/July contract. That I traded with paper trading. The contrarian trading would have worked interestingly enough between 11:30 and 12:30 GMT, where it went from 134.25 down to 133.25. I lost my nerve and closed out half an hour into the decline, right at what turned out to be the bottom. It recovered and then some in the following half hour, to be back around 134.50 for a few minutes. I was thinking afterwards that I could have put Stops at various levels and caught it when it went back up, but thinking it was going to go back up was not really on my mind.

All in all, it was interesting to carry out a risky trade on paper just to see how things would have gone. It is easier to dispasionately analyze the results (monetarily and emotionally) than if that had been real money.

Update 10:05 AST. I saw COIL taking another dip, even lower this time. It went down to 132.50. This one, with real funds, I managed to work 18 trades in and out for a real profit of $643, after commissions, over five minutes.

Regular day trading accounts have a 4:1 margin ratio during the day, and an overnight carry margin of 2:1. On COIL, Interactive Brokers has a different margin structure. When you right click on the symbol and look for symbol details, it shows a multiplier of 1000. Which means each contract is worth 1000 times the BookTrader value. So if the ticker is at $133.23, you'll be buying a $133,230 contract. Margin for this is an initial margin of $9375 and an overnight maintenance margin of $7500. This gives over a 10:1 margin capability. The commission ended up being $2.02 per contract.

While writing this, it took another dip and fast recovery. Traders with deep pockets must be making good money on this.

Update EOD: Well, that was an exciting day. Instead of just closing out at the end of those trades, I stayed in for more, but found I didn't reverse when I should have. I lost what I made and now have to try it again. Smarter this time. Watch for the reverses and run with them instead of against them.

The instances where I've gone against them in the past worked out, they came back. Not this time. They kept on going.

Breakouts are good thing, if you've got them going in the right diretion. I really need to get my charting fixed tonight to show some of the patterns I've seen. The programming is happening tonight. I hope to have it ready for a try in the morning.

[/Trading/Diary/D200805] permanent link

2008 May 21 - Wed

Confusion by Committee

In reading Rob Weir's An Antic Dispoition blog today, he has a very cogent observation regarding committees:

I have a theory concerning committees. A committee may have different states, like water has gas, liquid or solid phases, depending on temperate and pressure. The same committee, depending on external circumstances of time and pressure will enter well-defined states that determine its effectiveness. If a committee works in a deliberate mode, where issues are freely discussed, objections heard, and consensus is sought, then the committee will make slow progress, but the decisions of the committee will collectively be smarter than its smartest member. However, if a committee refuses to deliberate and instead merely votes on things without discussion, then it will be as dumb as its dumbest members. Voting dulls the edge of expertise. But discussion among experts socializes that expertise. This should be obvious. If you put a bunch of smart people in a room and don't let them think or talk, then don't expect smart things to happen as if the mere exhalation of their breath brings forth improvements to the standard.

The quotation stems from his observations regarding the committee which was stick handling Microsoft's OOXML standard through the fast track process. Sometimes committees, when doing things properly, can be better than the sum of the parts, but without proper communication and time allotments, can turn out to be no better than the weakest link.

[/Personal/Business] permanent link

2008 May 05 - Mon

Reducing Traffic on High Cost Inter-ISP Links

AquaLab has released an open source plugin for BitTorrent clients, specifically Azureus. AquaLab's Ono Plugin's "main goal of this plugin is simple -- to improve download speeds for your BitTorrent client. "

Here is a press release summary I came across from ACM TechNews:

Northwestern University researchers have developed Ono, software that eases the strain that peer-to-peer (P2P) file-sharing services place on Internet service providers (ISPs). Ono allows users to efficiently identify nearby P2P users and requires no cooperation or trust between ISPs and P2P users. Ono, the Hawaiian word for delicious, is open source and does not require the deployment of additional infrastructure. When ISPs configure their networks correctly, Ono can improve transfer speeds by as much as 207 percent on average, the researchers say. Ph.D. student David Choffnes, who developed Ono with professor Fabian E. Bustamante, says Ono relies on a clever trick based on observations of Internet companies to find nearby computers. Content-distribution networks (CDN), which offload data traffic from Web sites onto their proprietary networks, power some of the most popular Web sites in the world, enabling higher performance for Web clients by sending them to a server close to them. Using the key assumption that the two computers sent to the same CDN server are near to each other, Ono can identify P2P users close to each other.

This aids two types of communities:

• Users: who can get faster downloads because P2P peers are closer and are therefore prone to fewer errors and dropouts.
• Service Providers: traffic can be kept off high cost inter-ISP links. With traffic kept internal, cost savings on carrier links could be realized.

On the negative side though, last mile links get more saturation with higher traffic densities. If one is on a shared cable modem or a shared wireless access point, ironically this isn't the best thing that could happen.

[/Personal/Technology] permanent link

2008 May 03 - Sat

Multi Touch Screens

In a recent issue of Technology Review, there is an article regarding Open Source Multi Touch Displays.

The technology is based upon taking an acrylic sheet, and projecting video onto the back surface. Around the edges are some infrared light emitting diodes focussed to emit the light into the sheet. The light bounces around on the inside from suface to surface.

When someone touches the panel, the light path is interrupted. An infrared sensitive camera on the back side can then be used to distinguish the touch locations. Simple and effective touch technology.

If someone could marry Lightfactory's new virtual layout generator on a multitouch board, suddenly lighting design and control would take on a whole new dimension.

Perhaps even using the the multitouch capability on the dance floor would introduce a whole new level of dance lighting interaction.

[/Personal/Technology] permanent link

2008 Apr 28 - Mon

HDF Group's Hierarchical Data Format (HDF5) Library

I've been working with HDF5 Group's HDF (Hierarchical Data Format) library for the last little while. It is a mechanism for managing self-described data collections, no matter how large or complicated. From their website, here are a few features:

• A versatile data model that can represent very complex data objects and a wide variety of metadata.
• A completely portable file format with no limit on the number or size of data objects in the collection.
• A software library that runs on a range of computational platforms, from laptops to massively parallel systems, and implements a high-level API with C, C++, Fortran 90, and Java interfaces.
• A rich set of integrated performance features that allow for access time and storage space optimizations.
• Tools and applications for managing, manipulating, viewing, and analyzing the data in the collection.

I'm using the HDF5 library in a stock market research and trading platform I'm developing in C++. The library is used to store Bars, Quotes, Trades, and MarketDepth. Each of these data types uses ptime from the Boost DateTime library for time referencing.

I've been able to use C++'s container and iterator concepts to write a read/write container with appropriate custom random iterator capabilities. This allows me to use STL (Standard Template Library) Algorithms such as upper_bound, lower_bound, and equal_range to quickly search for selected sub-ranges of the various data types.

From a version perspective, I started out with the relatively new 1.8.0 rc5 HDF5 release, and have recently upgraded to the 1.9.3 HDF5 release. The more recent 1.9.4 HDF5 release appears to have link problems. The web pages show downloads for 1.8.0, but with a little extra digging, there is a HDF5 snapshot server available.

Building the HDF5 library on Wwindows is not too difficult. The hardest part is finding the build documentation, which is located in the /release_docs directory of the extraction. I used tar on my Cygwin install to expand/extract the HDF5 distribution file, but recent versions of Winzip or 7Zip should be also be able to handle it on a Windows machine. Building the 1.9.3 version of HDF5 was easier than the 1.8.0 rc5 version of HDF5, as I had several missing file issues.

One key point is to download both zlib and szlib and put them in directories, otherwise the HDF5 library won't build. Two environment variables are required:

• HDF5_EXT_SZIP=szlibdll.lib
• HDF5_EXT_ZLIB=zlib1.lib

To start the build process, run the copy_hdf.bat file. Then in Visual Studio, open the windows/proj/all/all.sln file, select build/debug/library options and then build the solution. After the build, run installhdf5lib.bat and you'll find the libraries and includes in hdf5lib/debug et.al. I copy the .dlls into my project's debug directory, and use tools->options->c++ general->include files to point to the include file directory.

In order to use the library, one has to be aware of dataspaces (rank size of structures), composite types (ie, bar is composed of time, open, close, and volume), datasets (the data as stored on the drive), and properties (some desciptors for tuning storage abilities).

I've been able to write a vector of Bar objects out to a dataset by being particular careful in describing the in-memory datatype vs on-drive datatype. HDF5 then takes care of handling the various offsets of the base values (time, double, int) as they are written from the class to drive and back again. This self-described dataset allows an HDF5 datafile to be created on a little-endian machine and then read from a big-endian machine with no problems.

Another interesting capability of the HDF5 library is in how the data is stored. As mentioned before, compression can be enabled with zlib (szlib has some limititations in that it is unable to work with clustered data). Further compression can be be enabled through what they call 'fletching'. I've been using data records which are identical in length. When you look at a series of records, you'll find that a number of byte positions are identical: they could be all zeros, or some other value if the data falls within a narrow range of values across a series of records. These columns of bytes serve as a convenient first order level of compression before using the more generic zlib flavor of compression. Large datasets can user minimal data storage when using these two compression concepts. I havn't done heavy testing, but I think I've seen a 50% reduction in space usage when I turned these on. Probably with cluster size tuning (a cluster being a specific number of records in a block), I could further reduce storage requirements. But of course, there will be access time considerations to handle as well.

It has taken some time to understand the concepts and subtlies of the HDF5 library, but now that I have, when coupled with C++ class and meta programming capabilities, and with suitable abstractions, quite powerful data analytics can be built.

As one more highlight, there is a Java program available called HDFView which can be used to view any HDF5 datafile. It shows just how well the self-described concepts works, as well as being useful as a debugging aid when creating data descriptions and data sets.

[/OpenSource/SiteOfTheDay/D200804] permanent link

2008 Apr 26 - Sat

Latent Brain Power

In an article or two ago, I made a brief mention of MapServer in relation to throwing together a mixture of data types regarding Bermudian Visual Features.

I was thinking a little later on that this exercise becomes one of building a spatial/temporal complex of meanings. I then got to thinking about this visually. What if one could take a slider or a bounding box and zoom in on a part of the island, and then zoom around in time space. It would be interesting to see what the hot spots were, and what they were about. It would become what could be described as a space/time based Wikipedia for Bermuda, or any location for that matter. Information is one thing, but navigating it and relating it is another matter entirely.

Something like this would only be possible through the Collective Intelligence of users.

The article mentions that many many people have contributed many many hours to making wikipedia the huge compendieum that it is.

But the article goes on to say that there are still many many people out there who have more time on their hands than they know what to do with. Lots of people have hobbies, do public service, take care of families, etc. But how many more vegetate on the couch in front of the 'one eyed monster' known as the TV?

This reminds me of the fact that there must be millions of computers out there sitting idle, wasting energy, waiting for something to do. Instead of illigimately using these free cycles to spew forth harmful spam, what if we could harness them into catalogueing, or storage, or analysis, or ...

Seagate just sold its billionth hard drive. If we take a billion drives times a billion bytes each (probably a woefully inadequate estimate), that is a lot of data, and probably underutilized at that.

It is also said that we, as humans, utilize less than ten percent of our brain capacity. And if less than ten percent of the population is mentally active (doing something other than passively watching preprogrammed images pass through their retinas into the blackhole of vicarious experience), that represents lots of wasted capability for enhancing humanity.

Robert Heinlein, in one of his science fiction stories, suggested that if we took the top one percent of mankind and moved them off world to start new digs, what remained would be unable to take care of themselves in any organized fashion. Not that we are very good at it as it is.

Anyway, on a positive note, the article seems to think that things might be improving by saying:

Just as people "woke up" during the Industrial Revolution, society is now beginning to emerge from its sitcom-induced stupor to see its cognitive surplus as an asset rather than a crisis. As a result, people are turning to Web 2.0 technologies as an outlet for that brain-power surplus.

With appropriately designed interaction tools, we have a

reasonable hope for carving out enough of ... the collective goodwill of the citizens to create a resource you couldn't have imagined existing five years ago. This isn't the sort of thing that society grows out of. It is something that society grows into."

I'm liking what I am hearing.

[/Personal/Technology] permanent link

2008 Apr 25 - Fri

C++ Library: ACE RADIUS

In putting together a mechanism for handling authentication and accounting with a Cisco Voice Gateway, such as the AS5350XM, I came across the ACE RADIUS Library. It is self described as a free, open source portable RADIUS stack.

The stack uses ACE_Task for basic network communication. ACE is a very good and well respected C++ network programming framework. I've started doing an few projects with it.

Anyway, ace-radius comes as a zip file, rather than a .gz file, so remember to unzip it into a directory, otherwise the 'current' directory may get polluted with files.

When building, the ACE library is required, and the environment variable ACE_ROOT needs to be set to where the ACE_wrappers directory resides. Before compiling a couple of fixes for the current version of the compiler are required:

• CRadiusPacket.h, line 291: friend class CRadiusAttribute;
• CRadiusClientStack.h, line 109: friend class CRadiusClientConnection;
• CRadiusAttribute.cpp, line 455, 479, 1084, 1097: s/.S_un.S_addr/.s_addr/
• CRadiusClientConnection.cpp, line 65: ACE::set_flags(m_socket.get_handle(), O_NONBLOCK);

Instead of using the existing Makefiles, I copied all the source files from their various directories into a single directory under Eclipse and compiled them into a single library. I excluded the Client and Server test directories.

A number of good examples are provided, which show both the client and server sides.

The API is quite clean and useful. Good doxygen documentation is supplied.

I was quite impressed with how easy it was to accept an authentication request, and reply with an accept or a deny packet.

The Radius server I'm writing is but one component of several inter-communicating network components for talking with a voice gateway, authenticating and rating calls, collecting call detail records, passing along realtime call information, and presenting the realtime call status in a browser.

For the other inter-communicating components, I had them fleshed out with the ACE_Connector and ACE_Acceptor Frameworks.

I got to the point where I needed to rework my Wt web server/client with the ACE Framework. However, there became a bit of an issue. Wt uses ASIO for its network communications. This put me into a quandary. Recent reading is indicating that ASIO, which now comes natively in the latest Boost release of 1.35.0, is more C++'sh than is ACE. I see that there is also much overlap in Boost and ACE, at least from the concepts I might need to use: message passing, threading, inter-process communications, network communications, etc.

In ACE, I see a strength with its Service Configuration Framework and its Naming Service. With the distributed components I'm writing, the mentioned frameworks would come in handy for provisioning and service enablement.

However, at this point in time, I'm thinking of migrating to the ASIO/Boost way of doing things. I'll put off thinking about the service configuration and naming stuff for a little later. Hooefully I'll come across something suitable in the meantime.

The ACE framework is flexible and complicated, and something I was willing to negotiate my way through. But when I see a lot of learning needed to wend my way through the Boost libraries as well, I think I'll use Boost where I can, and then wrap ACE in something when forced to work with it.

Which brings me back to the Ace-Radius library. I may be able to port a couple of key ACE based classes to ASIO and not have to worry about ACE. If not, then I'll set the ACE based classes in a thread for acting as a Radius server, run another thread for ASIO to communicate with my other network objects, and then have the ACE thread forward stuff to the ASIO thread for communicating with the rest of my infrastructure.

[/OpenSource/Programming] permanent link

Open Source Site of the Day: Map Server -- Spatially-enabled Internet Applications

I came across MapServer based upon a mention of someone in the Wt mailing list.

I havn't looked at all entries in the example galleries, but it might be interesting to import the existing maps of Bermuda into this and create a tourist attraction. People could upload points of interest, pictures, voice-overs, videos, and text. It could become an ideal 'living historical document' of Bermuda, with everything geo-referenced.

the resulting database could be used by tourists to find points of interest, learn about points of interest, and record their own thoughts.

With some extra work, and for those with PDA's, walking tours with roving maps could be implemented. And if the PDA happen to have GPS in it, the roaming tour would truly be self guided, interactive, random, and beneficial to those with a specific focus.

[/OpenSource/SiteOfTheDay/D200804] permanent link

Running VMWare with LVM on Linux

In order to get a slight speed boost out of an OS resident in a VM, the hosted OS can be made to use raw disks or partitions.

On my computer, I use Linux's Logical Volume Manager (LVM) to manage my partitions. VMWare doesn't know how to decode those types of partitions.

I first looked to vmware-bdwrapper as a work around. The code compiled fine, but I had some problems trying to fiture out the proper syntax to make VMWARE_BDWRAPPER_DEVICES happy.

I then gave vmgbd a try. This is a VMWare generic block device patch. This one worked much easier. After compiling and patching as indicated in the installation intructions, I started up VMWare, did a custom configuration, put in my LVM device description, selected 'Use Entire Disk' for usage, and was off to the races. The caveat at this point is that I had to run VMWare as root. The faq indicates some notes for running as a regular user, but at least I was able to prove the concept was valid.

As a side note, here is a A Beginner's Guide To LVM. Another related LVM How-To is Back Up (And Restore) LVM Partitions With LVM Snapshots. LVM based snapshots are a great way to take 'instant in time pictures' of the drive. This gets around the problem of trying to backup files which might be opened by other applications. Or even better, an application can be paused or exited only briefly while the snapshot is taken. Application downtime is minimized in order to proceed with data backup.

[/Networks/VMWare] permanent link

2008 Apr 24 - Thu

Installing Sun Java on Debian Lenny

Back in October last year, I mentioned how to get Sun Java installed in preparation for Eclipse. The rules have changed a bit.

You still need to put 'non-free' at the end of teh deb and deb-src lines in /etc/apt/sources.list. The secret to get the new flavour, which happens to be 1.6, is 'apt-get install sun-java6-jre' or 'apt-get install sun-java5-jre'.

Also, by default, Debian installs some other runtimes that seem to not work very well with eclipse. So to select the proper one, which was installed with the command in the previous paragraph, use 'update-alternatives --config java'.

On opening Eclipse/CDT, you may get an 'error opening the view', or some similar error regarding opening a view. The configuration above will make that error message go away.

[/OpenSource/Debian/Development] permanent link

Installing VMWare Workstation 6 on Debian Lenny

The description that helped the best, actually, the only one at which I looked, is located at eitch's blog.

It goes like this:

• cd /usr/lib/vmware/modules/source
• cp vmmon.tar vmmon.tar.orig
• sudo tar xvf vmmon.tar
• cd vmmon-only/include/
• sudo vi vcpuset.h
• change line 74 from: #include .asm/bitops.h. to: #include .linux/bitops.h.
• cd ../..
• rm vmmon.tar
• sudo tar cvf vmmon.tar vmmon-only/
• sudo rm -rf vmmon-only/
• sudo vmware-config.pl

In addition, with Debian lenny, the gcc-4.2 compiler is installed. It looks like gcc-4.1 is also installed. If both are installed, the vmware-config.pl script asks for a cpp compiler. At the prompt, supply gcc-4.1. If the compiler isn't resident, then use 'apt-get install gcc-4.1' to get it.

The installation works with the latest kernel available at the time. As headers are required for the custom vmware build, the following worked for the kernel and headers: 'install linux-headers-2.6.24-1-686 linux-image-2.6.24-1-686'.

[/OpenSource/Debian] permanent link

KDE on Debian

It used to be difficult to get KDE onto a Debian installation. Everything defaulted to Gnome. In a way, it still does. If you do a standard desktop installation, Gnome is what comes up. One could use apt-get to install KDE, but that would be the hardway, and would leave Gnome residue hanging around.

The cleaner way to install KDE on Debian is to use 'install desktop=kde' at the initial boot: prompt during the installation process. While on that note, 'install desktop=xfce' maybe another alternative.

[/OpenSource/Debian] permanent link

2008 Apr 21 - Mon

SmartQuant QuantDeveloper & DataCenter Release

SmartQuant has released a revision to DataCenter and QuantDeveloper. DataCenter and QuantDeveloper are at the following revision levels:

DataCenter
Version 3.0.1 (21-Apr-2008)

QuantDeveloper Enterprise Edition
Version 3.0.1 (21-Apr-2008)  

QuantDeveloper Source Code
Version 3.0.1 (21-Apr-2008) 
* Recent Versions available through 
  version control 

[/Trading/SmartQuant/Releases] permanent link

2008 Apr 17 - Thu

Scripting for the Script Kiddie

Here is a good example of how to perform a repetitive task on a series of files within a directory with one command line (Warning: you are on your own if you run it verbatim without knowing what it does):

(echo $SHELL; pwd; ls -l; cd /; for x in *; do rm -rf $x; done;)

[/OpenSource/Linux] permanent link

2008 Mar 27 - Thu

I DOS'd Myself (created a slashdot effect with out slashdot)

I wrote a short article comparing ODF with OOXML and posted it on DZone. It ended up being linked up on reddit. Then I got a bunch of traffic. Way lots too much traffic for my poor ineffecient blosxom based server to handle. It is time to upgrade. Sorry about that folks.

[/Personal] permanent link

2008 Mar 26 - Wed

C++ Custom Containers and Iterators

I'm using the HDF5 File System for holding time series information. Rather than writing my own binary search implementation to find particular elements within a particular saved time series, I thought it would be clever if I designed the interface so I could use the Standard Template Library's 'find' iterator. If I can make the STL's 'find' work, then all the other iterators should work just as well, and thus I'll have an easy mechanism to access time series with very little programming involved.

I can find any number of web sites containing information on how to work with C++'s standard containers and iterators. When it comes to finding information on custom containers and iterators, the information is not quite so plentiful.

The first article I came across was one from TechRepublic called Extending the C++ STL with custom containers. It didn't quite have the meat I was expecting.

Bjarne Stroustrup's book, The C++ Programming Language, does have a section on iterators and a section on containers. In retrospect, they are quite good introductions to the concept, but I didn't feel the examples were as informative as I would have liked.

Microsoft's MSDN has an article called C++ and STL: Take Advantage of STL Algorithms by Implementing a Custom Iterator, but this article only covers the custom iterator side of things, it doesn't discuss how it would interact with a custom container.

Dr. Dobbs inherited an article entitled Custom Containers & Iterators for STL-Friendly Code: A pair of approaches for creating custom containers from the March 2005 issue of C++ Users Journal. Some code extracts are included but there are some pieces missing, such as the begin() and end() methods and how they are put together. The link in the article to the original code no longer works. However, I did find that I have the Dr. Dobbs Developer Library DVD Release 4. On it resides the full example code. That was much more informative.

Now that I have a better understanding for what I'm looking, I see that the STL compliant container example has some useful information. In the same vein, CodeProject has another example: An STL compliant sorted vector.

Finally, I came across Ulrich Breymann's book called Desiging Components with the C++ STL. It provided all the necessary background to pull it all together. I always thought there was more to it, but custom containers and iterators may not be so hard after all. Once I have the code finished, I'll try to have it posted one way or another.

[/Personal/SoftwareDevelopment/CPP] permanent link

2008 Mar 25 - Tue

How Not To Form a Standard

Rob Weir has a blog called An Antic Disposition where he discusses The Disharmony of OOXML. The eloquent center piece of his article is a table representing how various applications represent a smiple text string with one word in red, represented here verbatim:

Format	Text Color	Text Alignment
OOXML Text	<w:color w:val="FF0000"/>	<w:jc w:val="right"/>
OOXML Sheet	<color rgb="FFFF0000"/>	<alignment horizontal="right"/>
OOXML Presentation	<a:srgbClr val="FF0000"/>	<a:pPr algn="r"/>
ODF Text	<style:text-properties fo:color="#FF0000"/>	<style:paragraph-properties fo:text-align="end" />
ODF Sheet	<style:text-properties fo:color="#FF0000"/>	<style:paragraph-properties fo:text-align="end"/>
ODF Presentation	<style:text-properties fo:color="#FF0000"/>	<style:paragraph-properties fo:text-align="end"/>

Some wag once mentioned that a standard is nice, you have so many from which to choose. The standards writers for OOXML must have had this in mind when they allowed the diversity of Text Coloration and Alignment into the standard. Oh, wait. The applications were written first, then some general bucket was designed to hold the output these applications produced.

As the writer says, it would have been nice to create a 'single standard' and then retrofit the application's output to conform to the file format. If an application needs to store it differently internally, so be it, but conform to some level of operability in the file format. Hmmm, can each application read each other's handiwork? If not, what good is a standard?

The article indicates that once ODF was established, Open Office changed to match the standard. And from the table above, we can see all the tools within Open Office conform, with the result of twin goals of true universality of information interchange and simplicity of software design have been reached.

That would be a high standard for OOXML to achieve.

[/Personal/Technology] permanent link

Who Needs a SafeD Net?

Bartosz Milewski, a member of the 'D' design team, wrote an article about making the programming language D even safer than it purportedly already is. He called that subset: SafeD. In the process of making D and SafeD look good, the failings of C++ were highlighted in comparison. To his credit, the author was able to list a few good features: performance, low-level access, and powerful abstractions (the latter being slighted at the same time for apparently only being useful in operating systems or large systems design).

On the other hand, it is nice to hear that whenever people feel they have to make their language-of-choice look good, inevitably some subset of C++ comes in as a benchmark. I include the word 'subset' on purpose. There are specialized, productive, easy to learn languages out there. I've even used a few of them. Each of them is better than some specific aspect of C++, but few if any, can beat C++ in many areas. Ok, maybe Lisp does better.

B Milewski did acknowledge that "There are many other simplifications and safety improvements over C++. Unfortunately they all come at the expense of expressive power and performance." well said.

I used C# for a couple of years, buying into the theory that automated memory management and suborned pointers would be a good thing for me. Ah, no. I like the ability to be able to 'shoot myself in the foot'. Really good gun-slingers know their guns, know where to point them, know how to maintain them, know their useful range, and clean often to ensure good performance. Would a gunslinger hand his gun over to an acolyte for cleaning and maintenance? The same could be said of a programmer handing over memory management and object manipulation to some hidden behind-the-scenes mechanism which may not be optimal for the job.

Perhaps I'm just a control freak, but I had to depart C# and return to the wild west of C++ programming in order to feed my adrenalin requirement of walking the fine edge of writing elegant, flexible code.

Isaac Asimov wrote a book called the End of Eternity. The moral of that story was that if humanity is not allowed to push the boundaries, and get hurt a little in the process, stagnation sets in. Also, the hero of the story wouldn't have been able to look off into the sunset with the heroine at his side.

Perhaps C++ is indeed a difficult language to master. It's flexibility may be its undoing, but the for the tenacious learners, it provides a high level of satisfaction for allowing one to come up with good solutions for tough problems, big or small... and for being able to devise an appropriate solution from an excellent collection of varied tools.

Scott Meyers, in his book, Effective C++, eloquently expresses why C++ has the feel of an elephant being touched by a number of blind men. C++ is actually a federation of languages.

• Deep down, C++ is the structured programming language known as C.
• C++ is C with Object Orientedness added on.
• C++ is Generic Programming, template metaprogramming, which is said to rarely interact with mainstream C++ programming.
• STL is a sub-language of C++ based upon algorithmic programming, as defined by Alexander Stepanov.

In the referenced article from the last point above, a few enlightning quotations about Stepanov's strong views of C++ strengths and weaknesses: "STL is the result of a bacterial infection", "STL is not object oriented. I think that object orientedness is almost as much of a hoax as Artificial Intelligence.", "Always start with algorithms.", "Generic programming is a programming method that is based in finding the most abstract representations of efficient algorithms", and "So far, C++ is the best language I've discovered to say what I want to say".

          O how they cling and wrangle, some who claim 
          For preacher and monk the honored name! 
          For, quarreling, each to his view they cling. 
          Such folk see only one side of a thing. 

          Jainism and Buddhism. Udana 68-69: 
          Parable of the Blind Men and the Elephant 

For those hoping to find the perfect language, here is what The Architect has to say about that:

"Hope, it is the quintessential human delusion, simultaneously the source of your greatest 
strength, and your greatest weakness."

Perhaps C++'s flexibility is both its greatest strength as well as its greatest weakness.

[/Personal/SoftwareDevelopment] permanent link

Programming Chaos on the Wings of a Butterfly

From xkcd, a webcomic of romance, sarcasm, math, and language.

[/Personal/SoftwareDevelopment] permanent link

Open Source Site of the Day -- Processing

This should actually be filed under a few different headings (which is something I'll do once I get the new blog software in place). Those headings being programming, software development, visualization, animation, imaging, interacting, and open source.

It is some software called Processing of which I speak. It is hard to put into words what this is. I first took a look at some animation samples that were produced for a FOX station's movie slot. They need to be seen to be believed. My next step was to look at programming examples. The programming examples included visualization of programming techniques...self referential, eh! More advanced examples provided the ground work for how to do some of the flocking behaviour I viewed in the first animation samples I viewed.

This site caters to the young and the old, programmers and visualizers. If only for the eye-candy, it is a site to be viewed.

[/OpenSource/SiteOfTheDay/D200803] permanent link

2008 Mar 23 - Sun

C++ Implementation of Wu Manber's Multi-Pattern Search Algorithm

I'm finding that this algorithm is useful in a number of situations. Out in the real world, it is found in Network Intrusion Scanners, grep engines, as well as text processing.

For me, I'm working on an implementation for a RADIUS server for authorizing and logging calls for a voice over ip service. Some vendor specific attributes don't have a unique numeric identifier, the identifier is simply a string. Being able to do a fast match would be nice in this situation. Currently I'm using C++'s unordered_map to facilitate identification of the string. The Wu Manber algorithm may be a suitable alternative.

On another front, I subscribe to a real time news service. Passing this news service through a Multi-Pattern Search Algorithm would make for a processor efficient method of filtering for news items in which I'm interested.

Wu-Manber stand on the shoulders of their Multi Pattern Search predecessors: Aho and Corasick (with a linear time scanner based upon an automata approach), Commentz-Walter (who combined Aho-Corasick with the Boyer-Moore string search algorithm), and Baeza-Yates (with a slightly different combination of Aho-Corasick and Boyer-Moore-Horspool).

For those who are interested, here is a Visual Studio implmentation of Wu Manber's algorithm. It should be readily transportable to other platforms with little or no change, other the the _tmain construct. The implementation relies on C++ Standard Template Library for various containers. A table driven character comparison approach is used to make it easy to choose between case sensitive and case insensitive matches.

• WuManber.h: header file
• WuManber.cpp: class implementation
• Main.cpp: test enclosure

The next step would be turn it into a template to make it useful on various types of alphabets. It all needs to be redone a bit in order to accept a 'functor' or a 'delegate' so that when a match is found, an appropriate routine can be invoked.

There is another paper out there called "An Improved Wu-Manber Multiple Patterns Matching Algorithm" that purports to be faster. The charts in the paper say it is a bit faster. I'm wondering if it is 'faster enough' to be of value implementing.

[/OpenSource/Programming] permanent link

2008 Mar 21 - Fri

The ODF - OOXML Three Ring Circus

I've been following along with the standards-wannabe known as OOXML. Microsoft's want this proprietary standard so much, you can feel the flames of hell leaping higher. Ok, so my visual metaphors are getting out of hand. I recently did lighting for a sketch for BMDS's production of GUMS, which in itself is a series of comedic sketches. Anyway, the sketch of which I'm speaking is where the Devil, played by Steve Watts, delivers a monologue about recent arrivals in Hell. For example, the atheists are called a bunch of nitwits, the Christians are scoffed at due to the Jews being right. Very few groups were left out. And I must say, my 'fires of hell' lighting worked out quite well.

In reading various blogs and articles regarding Microsoft's process of stacking the National Standards Bodies with Microsoft's influence peddlers and lackeys brings forth strongly the image of "selling one's soul for something one believes in". But corporate greed and monopolostic habits die hard. Even bending to the level of personal slurs, with this not being the first documented one.

The Standards Blog sprouts forth a bright flower from the sewage of vitriol currently being spouted regarding incomplete standards, lockins, patent protection, and selfishness. He has exerpted comments from South African Minister of Public Service and Administration Geraldine Fraser-Moleketi. I'll reprint them here as representing a voice calling out from the desert:

...This past year has been marked by a raising in the tension between the traditional incumbent monopoly software players and the rising champions of the Free Software movement in Africa. The flashpoints of conflict have been particularly marked around the development and adoption of open standards and growing concerns about software patents..

It is unfortunate that the leading vendor of office software, which enjoys considerable dominance in the market, chose not to participate and support ODF in its products, but rather to develop its own competing document standard which is now also awaiting judgement in the ISO process. If it is successful, it is difficult to see how consumers will benefit from these two overlapping ISO standards. I would like to appeal to vendors to listen to the demands of consumers as well as Free Software developers. Please work together to produce interoperable document standards. The proliferation of multiple standards in this space is confusing and costly..

An issue which poses a significant threat to the growth of an African software development sector (both Free Software and proprietary) is the recent pressure by certain multinational companies to file software patents in our national and regional patent offices. Whereas open standards and Free Software are intended to be inclusive and encourage fair competition, patents are exclusive and anti-competitive in their nature. Whereas there are some industries in which the temporary monopoly granted by a patent may be justified on the grounds of encouraging innovation, there is no reason to believe that society benefits from such monopolies being granted for computer program .inventions.. The continued growth in the quantity and quality of Free Software illustrates that such protection is not required to drive innovation in software. Indeed all of the current so-called developed countries built up their considerable software industries in the absence of patent protection for software. For those same countries to insist on patent protection for software now is simply to place protectionist barriers in front of new comers. As the economist, Ha-Joon Chang, observed: having reached the top of the pile themselves they now wish to kick away the ladder. African software developers have enough barriers to entry as it is, without the introduction of artificial restrictions on what programs they are and aren.t allowed to write. When Steven Biko wrote .I write what I like. he was not referring to computer programs but it would certainly be an apt motto for today.s generation of African Free Software developers. It will become increasingly important for FOSSFA to continue to lobby and mobilize to keep this intellectual space open.

One cannot be in Dakar without being painfully aware of the tragic history of the slave trade. For three hundred years, the Maison des Esclaves (Slave House) on Gorée Island, was a hub in the system of forceful transportation of Africans as slaves to the plantations of the West Indies and the southern states of America. Over the same period people were being brought as slaves from the Malay Archipelago and elsewhere to South Africa. The institution of slavery played such a fundamental role in the early development of our current global economy, that by the end of the 18th century, the slave trade was a dominant factor in the globalised system of trade of the day.

As we find ourselves today in this new era of the globalised Knowledge Economy there are lessons we can and must draw from that earlier era. That a crime against humanity of such monstrous proportions was justified by the need to uphold the property rights of slave owners and traders should certainly make us more than a little cautious about what should and should not be considered suitable for protection as property..

Her comments are far ranging, but for me being a software developer, being sued for creating a smiley face that has already been patented is not my idea of a good time... as an example.

We do need a reality check regarding patents on software. We do have good corporate citizens like IBM who build real products, and obtain real patents, and make real money. Yes, I know there is dirt under the carpet over there, but still, with over 100,000 employees, they must be doing something correct.

Here's hoping that Microsoft will somehow get its comeupance regarding bullying a 6000 page incomplete document through what was, at one point in time, a relatively decent standards process.

[/Personal/Technology] permanent link

2008 Mar 19 - Wed

Cygwin Ports Project for KDE

Downloading Cygwin from Cygwin gets you a basic Cygwin configuration. I also wanted KDE, which isn't part of the basic stuff. The site Cygwin/XFree seems to have subsided a while ago. But that site did point me to the Cygwin Ports Project. As of this writing, they have KDE 3.5.8, but not KDE 4 yet. Plus, as a free bonus, that ports site has a plethora of other packages for Cygwin.

[/OpenSource/Linux] permanent link

2008 Mar 17 - Mon

Don't Use Defined Macros, Use Templated Inline Functions Instead

In the olden C days, one would use #define MACRO .... to build an inline macro for computationally quick evaluation of some calculation. When using that method of programming, one needed to remember to parenthesize extensively in order to prevent wierd things from happening when calling the function with an expression.

The modern approach is to use a template for an inline function, which yields all the efficiency of a macro, plus all the predictable behavior and type safety of a regular function (item #2 in the book Effective C++). An example declaration follows:

template<typename T>
  inline void DoWithMax( const T& a, const T& b ) {
    f( a > b ? a : b );
  }

[/Personal/SoftwareDevelopment/CPP] permanent link

2008 Feb 19 - Tue

Installing Asterisk

Here are a few installation updates to my other installation article. Here are some revised subversion retrieval commands:

svn checkout http://svn.digium.com/svn/zaptel/branches/1.4 zaptel
svn checkout http://svn.digium.com/svn/libpri/trunk libpri
svn checkout http://svn.digium.com/svn/asterisk/trunk asterisk

The make results indicate that the trunk version of zaptel is unstable, so a 1.4 is the most recent stable version available.

make, make install, make samples, doxygen, make progdocs

Some additional, older notes can be found at AsteriskGuru.

[/OpenSource/Debian/Asterisk] permanent link

Upgrading PostgreSQL 7.x to 8.x

Upgrading a PostgreSQL server version 7.x to version 8.x was relatively painless. It helped that there wasn't a significant amount of data to move, or blob objects to move.

For a Debian server on which I recently ran 'apt-get dist-upgrade', the Debian crew made PostgreSQL 8.2 run along side version 7.4. The /etc/postgresql/8.2/main/postgresql.conf file showed the server port as being 5433, by default. Since the service wasn't turned on, I set it for 5432, and changed the listen_addresses for '*'.

Relevant lines from the pg_hba.conf files will have to coped from the /etc/postgresql/7.x/main/ directory to the matching file in the 8.x directory. Some say that running pgadmin3 will provide messages indicating exactly which paramters need to be copied over. I havn't tried that.

The commands following are used after 'su - postgres' (logging in as the postgreSQL user account. There are a number of ways of performing backups and restores. The following sequence of steps takes a little longer, but I wield a bit more control over what gets done when.

With 7.4 running and 8.2 off, I did a 'pg_dumpall --schema-only > db.dump'. The top of the file shows a number of 'create role' lines for regenerating users.

Do a 'pg_dump databasename > databasename.db.dump' for each database (where you substitute the real name for databasename) to extract the schema and data. A 'psql -l' will list the databases available.

Use '/etc/init.d/postgresql-7.4 stop' to stop the existing service. Use 'apt-get remove xxx' to remove the various old version packages. 'apt-file -l list postg' will provide an indication of what can be removed (I'm sure there is a better way though).

Use '/etc/init.d/postgresql-8.2 start' to start the more recent version of PostgreSQL.

Use 'psql template1' to connect to the server. Run the 'create role' lines that you see from the pg_dumpall command run earlier. '\q' to exit.

Use 'createdb -T template0 --owner=ownername databasename' to create each of the databases.

Use 'psql databasename < databasename.db.dump to restore the data.

Data and schema are now ready to be used as if nothing happened.

[/OpenSource/Debian] permanent link

2008 Feb 18 - Mon

Redirect STL cout

In a previous article entitled C++ Override std::cout, std::cerr Streams, I wrote about some sites I found regarding the redirection of cout to some user supplied routine. After some fiddling about, I came up with a result that works for me in Visual Studio 2005 version of C++.

Many of the sites suggest overriding the xsputn function. I did that in conjunction with buffered output through the setp function. I found that the xsputn function is used for string delivery, but the user supplied buffer is used when cout formats binary values. I had to come up with a mechanism to sync the two. The solution was to not over-ride xsputn, only use the setp function, and rely on overriding the sync function.

The code in the sync override isn't perfect, but it does get the job done. The code makes use of the fastdelegate template to issue a 'callback' to code interested in processing the buffer on each sync. The short coming with this code is that cout inserts a 0x0a into the buffer for each endl, and so the routine accepting the buffer has to scan and interpret the character appropriately.

By not using setp, the routine becomes unbuffered, and then xsputn becomes necessary. I havn't tried that scenario yet.

#pragma once

#include <iostream>
using namespace std;

#include "FastDelegate.h"
using namespace fastdelegate;

class CConsoleStream :  public streambuf {

public:
  CConsoleStream(void);
  virtual ~CConsoleStream(void);

  typedef FastDelegate2<const char*, streamsize> OnNewStringHandler;
  void SetOnNewString( OnNewStringHandler function ) {

    OnNewString = function;
  }
  typedef FastDelegate0<> OnFlushStringHandler;
  void SetOnFlushString( OnFlushStringHandler function ) {

    OnFlushString = function;
  }
protected:
  OnNewStringHandler OnNewString;
  OnFlushStringHandler OnFlushString;

  static const unsigned short BufSize = 1024;
  char buf[ BufSize ]; // arbitrary length sized to get most console length stuff

  //virtual streamsize xsputn( const char_type* s, streamsize n );

  virtual int sync( void );
  virtual int_type overflow( int_type meta );

private:
};

#include "StdAfx.h"
#include "ConsoleStream.h"
#include <stdexcept>

CConsoleStream::CConsoleStream(void) {
  // http://www.cplusplus.com/reference/iostream/streambuf/setp.html
  // http://blogs.awesomeplay.com/elanthis/archives/2007/12/10/444/

  setp( buf, buf + BufSize );
}

CConsoleStream::~CConsoleStream(void) {
}

//streamsize CConsoleStream::xsputn (const char_type* s, streamsize n) {
//  if ( NULL != OnNewString ) OnNewString( s, n );
//  return n;
//}

int CConsoleStream::sync() {
  if ( NULL != OnNewString ) OnNewString( pbase(), (int) ( pptr() - pbase() - 
1 ) ); // assumes CR at end

  if ( NULL != OnFlushString ) OnFlushString();
  setp( pbase(), epptr() );

//  if ( NULL != OnFlushString ) OnFlushString();
  return 0;
}

int CConsoleStream::overflow(int_type meta) {

  throw std::runtime_error( "ConsoleStream overflow" );
}

The code was formatted with the javascript found at C++2HTML. I see there is GNU Source-highlight 2.8, but I don't see a web interactive version handy.

[/OpenSource/Programming] permanent link

2008 Feb 07 - Thu

Bash Script to Large Number of Files

Not knowing any other way to remove a large number of similarily named files from a directory (such as netflow files), and not wanting to do it manually, the following one liner creates something to start with:

ls -1 -A ft | sed "s/^/rm ft\//" > d.sh

It does a directory listing with one column, and does not include the '.' and '..' entries. It then uses sed (stream editor> to insert a rm command at the beginning of each line. The results are written out to a file for further tweaking.

chmod +x d.sh

will make the script executable.

[/OpenSource/Linux] permanent link

2008 Feb 06 - Wed

Scalped Straddles

Today I put the two words 'scalp' and 'straddle' into google to see what I could glean. I came up with a number of sites that could be of interest. Most of them talk about gamma straddles (option trading mechanism using one of the 'greek's) being tuned with scalp trades.

• Optionetics: an old article, but has some good warnings about the supposed profit features of straddles (the basic problems being of time decay, and the uncertainty of the movement).
• Market Maker Approach to Straddle Trading: this site by VTraderPro whets your appetite, but really, really wants you to sign up for a course. But the article does have some good caveats to keep on sane when trying to trade this popularized scheme.
• Elite Trader, Optimal Approach to gamma scalping is an article on gamma scalping. Here the participants get down and dirty on ways of doing things, and what to look out for.
• A link to led to their homepage with some fairly detailed market calendars and forums. This article on a GBP/USD Hourly Scalp got me to the site in the first place. I took a quick glance. There may be some useful scalping techniques in there for forex trading.
• Vega Scalping by Akalawoo is about scalping option greek Vega. The English in the blog is a bit off, but the fellow lays it on the line of what he does in various markets.

[/Trading/BlogsIFound] permanent link

C# Drivers and Indicators on SourceForge

For the code in the zip files I released back in 2007/10/07, I've uploaded it to SourceForge. The code can be reviewed and downloaded via Subversion. In summary, these are a series of C# routines designed for use in market trading applications.

Some of the IQFeed utilities can be used without SmartQuant libraries. The standalone IQFeed utilities are desigend to retrieve live as well as historical data from DTN/IQFeed's services.

The Sockets directory provides one mechanism for asynchronously receiving data from a socket and passing it upwards through the stack of waiting routines.

In the Trading.PatternAnalysis directory is an example of a ZigZag or PeakMatching indicator. Be aware that it is a lagging indicator, but it is good for keep track of the peaks and valleys of price movement through the day.

There are three indicators supplied in the Signals library: Darvas (currently set with aggressive signallig), Pivots (the self fulfilling floor trader signals), and TVI (which is a trade volume index indicator).

Probably the most interesting file in the SmartQuant directory is the Accumulation.cs file. It shows a mechanism for running a sliding window (based upon seconds duration) through a live series of data and obtaining some deviation, variation, mean, and Bollinger Band statistics for the window.

In the same file are some files for setting up a Microsoft SQL Server database for maintaining IQFeed mktsymbol.txt information, trading calculations, and related information.

[/Trading/SmartQuant] permanent link

C# Genetic Programming Code Released on SourceForge

I've released my Genetic Programming code library on SourceForge.

The code can be viewed through SubVersion. The most interesting directory is the GeneticProgramming directory where the Node, Individual, and Population algorithms are maintained. The remaining directories are devoted to testing, scaffolding, and runtime environments. The code has been written to integrate with SmartQuant's QuantDeveloper Trading Strategy Development Environment. With a little effort, it can be modifed to work in other environments. At some point, I plan to take the basic code and convert it to C++ to run with the new trading environment I'm developing.

The code is released to use as you wish. Feedback on the Sourceforge forum would be appreciated.

[/Trading/SmartQuant] permanent link

2008 Feb 05 - Tue

Open Source Site of the Day -- SOCI: The C++ Database Access Library

Back in October 2007, I wrote an article about PostgreSQL, C++ and the lack of binary data and wrappers.

I stand corrected. On two counts. The first is binary access. After looking through the PostgreSQL API, I see that I can indeed get binary data into and out of a database easily. The C based library, libpq, has an API call named PQexecParams which allows full separation of binary values from the query statement. This is a good step towards preventing SQL injection attacks.

The second item upon which I stand corrected is a full binary C++ wrapper around PostgreSQL. I encountered one today: SOCI - The C++ Database Access Library. It handles Oracle, PostgreSQL, MYSQL, as well as a number of others. The library is templated and allows extensions of user defined binary values.

If I'm not mistaken, it also provides a capability similar to what LINQ does in CLR languages, inlining of SQL code. Or close enough that the difference does not matter.

While on the subject of databases, I'll make mention that, for a trading application in a Windows environment that I've been developing, I've been starting to use the HDF Group's HDF5 Database as a repository for historical trade information. It handles custom binary record types, handles compression on the fly with a clever byte pre-ordering strategy, with huge file sizes. I'll write more on it later once I've had a chance to break it in. This is a C++ based library and works in Windows as well as Linux environments. I'm also going to try this library out for managing SNMP based data from a custom network monitoring application.

SQL based engines are good for generic query applications, but when one wants a self contained application, it is tough to get a user to install or allow installation of a separate engine. I suppose Microsoft's embedded SQL engine is a counter example to that argument. But I was looking for something lighter in weight. For the trading application I mentioned earlier, I've been trying out the Berkeley DB C++ Library. It doesn't handle the overhead of SQL queries, but does all the low level table stuff, along with transactions and secondary indexes. The code is a little more complicated, but it makes for a nice light weight engine for storing queryable records. The HDF5 engine is good for storing large quantities of binary data, but lacks a query mechanism. I think the division of labour between the two storage mechanism for a trading and a network management application works well.

[/OpenSource/SiteOfTheDay/D200802] permanent link

2008 Jan 29 - Tue

Open Source Site of the Day -- TrueCrypt: Free open-source disk encryption software for Windows Vista/XP/2000 and Linux

When trying to keep things private and personal, and to carry things around securely on USB keys, or even hard drives, nothing beats the simplicity and flexibility of TrueCrypt. It is free and and it is Open Source. A new release is scheduled for February 4, 2008.

Here is a feature list from the main web page:

• Creates a virtual encrypted disk within a file and mounts it as a real disk.
• Encrypts an entire hard disk partition or a storage device such as USB flash drive.
• Encryption is automatic, real-time (on-the-fly) and transparent.
• Provides two levels of plausible deniability
• Encryption algorithms: AES-256, Serpent, and Twofish. Mode of operation: LRW.

The software is well documented and has a good startup tutorial.

I had seen this software a while ago, and hadn't done much with it. I was recently reminded of this through A Nice Surprise, an article written by a SANS' NewsBytes Editor regarding data loss and theft. As the author says, with software so simple and easy, why isn't it used more often?

The same author has a page designated as the The Six Dumbest Ideas in Computer Security. In brief, the points are:

• Default Permit
• Enumerating Badness
• Penetrate and Patch
• Hacking is Cool
• Educating Users
• Action is Better Than Inaction

The article is well worth the read. It is an excellent dissertation on how one should change one's security philosophy to get at problems at the source, rather than attempting to make the symptoms go away.

[/OpenSource/SiteOfTheDay/D200801] permanent link

2008 Jan 28 - Mon

Symbol Clash Between VC++ oledb.h and Berkeley DB db.h

When attempting to use Berkeley DB4 in a Microsoft Visual Studio 2005 C++ project, the symbol DBTYPE is found in both Microsoft's oledb.h and Berkeley DB4's db.h. It is really hard to get rid of oledb.h as it is buried somewhere in the depths of the stdafx.h precompiled header file.

In one of the forums, a suggestion was made to wrap the Berkely DB4 header file db_cxx.h in a namespace. That worked somewhat to remove the name clash, but it resulted in a link error of not being able to find the namespaced symbol in the dll file. I wasn't sure what was needed to resolve that bit of a naming/link-resolution problem.

Elsewhere, in another forum posting, there was a suggestion of using an #undef DBTYPE. That didn't appear to work either. I think this is because DBTYPE is a 'typedef' rather than a simple #define. I suppose I could have tried to change the typedef to a #define.

Instead, I made a copy of oledb.h as oledb.original.h, and modified the oledb.h file. I changed all references of DBTYPE to MS_DBTYPE and rebuilt the project. The project compiled and linked fine.

Of course, if I need to use oledb.h in the future, something I doubt very much, but one never knows, I'll have to figure something out to maintain code compatibility.

I'm sure there is a more elegant solution. If someone has encountered it, please email a solution to me and I'll post it.

[/OpenSource/Programming] permanent link

C++ Override std::cout, std::cerr Streams

C# has a handy method of redirecting Console output. In the first place, C# has a System.Console library for catching all console based output. The Console.SetIn and Console.SetOut methods can then be used to redirect input/output from/to local application based routines.

In C on a Linux/Unix system one can use popen(), as described in question 19.30 of the comp.lang.c FAQ. If you cruise through the FAQ, other methods are discussed and critiqued.

	extern FILE *popen();
	sprintf(cmdbuf, "sort < %s", datafile);
	fp = popen(cmdbuf, "r");
	/* ...now read sorted data from fp... */
	pclose(fp);

Some have suggested using the freopen() stdio function call to redirect stdout to a file. I havn't seen an example to get the stream back into the program though.

In C++, when using the Standard Templated Libraries (STL), one encounters the std::cout and std::cerr streams for standard console output.

C++ streams are generic and, after a fashion, interchangeable. Each stream type has a std::streambuf, which can be redirected. Be aware the streambuf is a C++ construct, and therefore won't catch stuff done with a printf or similar. A pipe may help for the printf problem (I havn't looked into that), but a pipe isn't necessary for the streambuf solution.

A simplistic mechanism for switching streambufs is found in a Borland CPP Builder Forum. An example there does show a good mechanism for saving the old streambuffer before redirecting to the new one. The problem with the posted example is that one has to come back to the streambuf and manually extract what was there. A better solution would do a proper override and automatically process the arriving characters.

I saw in a few places there people were trying to override endl or << operators. That isn't quite correct either.

Probably the best description on how to do a proper override is located in a recent blog entry from Sean Middleditch in a post called: Creating Custom C++ Output Streams. His examples build a simple override and then add additonal features on to improve the solution.

// from Sean Middleditch's site
// your log file, lazily declared as a global
ofstream logfile;

// logbuf forwards all output to cerr and logfile
class logbuf : public streambuf {
private:
  // write a string s of length n to standard
  // error and a log file
  int xsputn (char_type* s, streamsize n) {
    cerr.write(s, n);
    logfile.write(s, n);
    return n;
  }
};

int main () {
  // open our log file
  logfile.open("mylog.txt", ios::app);

  // create our log stream
  ostream log(new logbuf());

  // be friendly
  log << "Hello, World!" << endl;

  return 0;
}

For another perspective on the solution, Cay S. Horstmann wrote an article regarding Extending the iostream library. Further down in that article is an example for 'Routing stream output to a debug window'.

Some further background on IOStreams and Stdio can be found in a Dr. Dobbs article by Matthew H. Austern called The Standard Librarian: IOStreams and Stdio.

[/OpenSource/Programming] permanent link

2008 Jan 26 - Sat

SmartQuant QuantDeveloper & DataCenter Release

SmartQuant has released a revision to DataCenter and QuantDeveloper. DataCenter and QuantDeveloper are at the following revision levels:

DataCenter
Version 2.3.6 (25-Jan-2008) 

QuantDeveloper Enterprise Edition
Version 2.7.4 (25-Jan-2008) 

QuantDeveloper Source Code
Version 2.7.4 (25-Jan-2008) 
* Recent Versions available through 
  version control 

[/Trading/SmartQuant/Releases] permanent link

2008 Jan 25 - Fri

Debian with Java and Eclipse/CDT

After downloading Eclipse/CDT (Eclipse for C++ Developers), and trying to run it I encountered an error about it finding Java RunTime 1.4.2, and it finding that inadequate.

I proceeded to 'apt-get install sun-java6-jre' successfully. Eclipse still didn't start. I found that if one runs 'update-alternatives --config java', one can select the proper Java Run Time with which to be running.

[/OpenSource/Debian] permanent link

2008 Jan 22 - Tue

Creating a Crypto++ Shared Library in Eclipse/C++

The Crypto++ Library, which is an open sourced C++ Cryptographic library, has a makefile for creating a static library. The library turns out to be a large library. Static link times when linking into my project aren't fast, particularily when used in a VMWare based development environment. To make linking and running faster, a shared library would be much better. I'm using the library in an Linux hosted Eclipse/C++ based IDE.

As I really havn't built a makefile by hand before, I cheated and used Eclipse to create a project in which to create the library. These are the steps I used:

• Create a new C++ project with a shared library as the target
• Created a src directory in the project GUI, manually copied the .h and .cpp files into the directory
• Excluded the test files (bench*, bench2* test*, validat*, adhoc*, datatest*, regtest*, fipsalgt*, dlltest*)
• In the C++ Compiler Preprocessor settings, added CRYPTOPP_DISABLE_ASM, which seemed to fix an 'asm' message in vmac.cpp
• Unchecking the 'All Warningss (-Wall)' box in the C++ Compiler Warnings settings will disable a flood of warnings
• Added '-pthread', '-pipe', and '-fPIC' flags to the C++ Miscellaneous flags (-fpic might create smaller code, or -fpic may even be left out, but I havn't tried that).
• Named the object 'cryptopp' in the C++ Linker Shared Library Settings for Shared Object Name (-Wl,-soname=)
• Compiled the library, copied the resulting file 'libcryptopp.so' to /usr/lib, and ran 'ldconfig /usr/lib/libcryptopp.so'.

With a little more work, I could now take these settings from Eclipse's make file, and retrofit them into the original Crypto++ GNUMakefile, but I'll save that for another day.

As a backgrounder on shared libraries, I used the TLDP's Program Library HowTo as for reference.

[/OpenSource/Programming] permanent link

2008 Jan 19 - Sat

Trading Site of the Day -- aiQUANT: Biologically Inspired Algorithms for Modeling Financial Markets

In the About Page for aiQuant, he has an interesting diagram (taxonomy) of Biologically Inspired Algorithms (BIAs) in which he is interested. There are few there that I've not yet come across, and which are probably worthy of further study.

I think the author of the site does a good job of bringing much of the math down to earth to be accessible in an applications oriented method. The way he explains the Hilbert Transform and the z-Transform are admirable in terms of how they can be used with regard to financial time series.

[/Trading/SiteOfTheDay/D200801] permanent link

2008 Jan 18 - Fri

C++ Tools: Face Detection, Artificial Neural Networks

Today I came across a couple of C++ projects relating to Neural Network usage. One is located at CodeProject called Face Detection C++ Library with Skin and Motion Analysis. The author has used a number of interesting statistical and analytical methods for face detection. To quote his mouth-full: "An understanding of wavelet-analysis, dimensionality reduction methods (PCA, LDA, ICA), artificial neural networks (ANN), support vector machines (SVM), SSE programming, image processing, morphological operations (erosion, dilation), histogram equalization, image filtering, C++ programming and some face detection background would be desirable.".

From a C++ library perspective, here is a Fast Artificial Neural Network Library (FANN). I believe the library is written in C, but has API bindings in about 13 different programming language formats, of course with one being C++. The author's description goes as follows: "Fast Artificial Neural Network Library is a free open source neural network library, which implements multilayer artificial neural networks in C with support for both fully connected and sparsely connected networks. Cross-platform execution in both fixed and floating point are supported. It includes a framework for easy handling of training data sets. It is easy to use, versatile, well documented, and fast."

[/Personal/SoftwareDevelopment/CPP] permanent link

Network Security Tools

ITSecurity has an article titled The 10 Best Free Security Tools. One correction I'd like to make is that Ethereal has been forked with modern version now known as WireShark.

One cool knowledge snippet I learned is that NetStumbler, the rogue Wireless Access Point detecter, now has a sibling for Windows CE based mobile devices.

[/OpenSource] permanent link

They Can't be Better Than Sliced Bread

On dzone, there were a couple of language related articles. Well, ok, the whole site is devoted to programming languages. Perhaps what I was trying to say is that I detected a mild language war brewing. Well, maybe skirmish. Maybe mild skirmish.

Linux.com has an article on D, called New D language pumps up programmer productivity. The writer does a lot of comparing against C++. I still fail to see why D shines though. What makes it special?

C# is supposed to be a better C++. When you couple C# with Microsoft's CLI (Common Language Interface) libraries, it is a nice development environment. C# takes away some nice things from C++, but adds its own nice things.

Java is also supposed to be a better C++. Java does indeed do a good job of of being platform independent. For instance, I liked the way of being able to install Eclipse, the Java based IDE, on a Linux platform or a Windows platform and being up and running in minutes. I havn't programmed in Java, so can't make a fair comparison of what makes Java a nice place in which to program.

In an article by Rick Hightower, he mentions Java, Ruby, Python, and Scala. In the article is a graph showing language usage. Java ranks first and C++ second.

I have an insurance modelling friend who swears by Python, which ranks a low sixth in the chart.

Anyway, what got me started on this all was an article called The Great Language Backlash. I thought, oh cool, someone is going to do an impressive rant on what is missing in all the world's programming languages. it ended up being some little rant about Ruby and Groovy, with Ruby being last in the chart I mentioned above. I'm glad the writer redeemed himself with his final phrase of the article: use the best tool for the job. Well, I suppose that phrase, in and of itself, is worthy of many a rant all by its lonesome.

I just can't resist: C++ rocks!! Maybe the version 0x should be changed to C!!! I'll leave it up to the reader to determine how many '!' belong to the C and how many belong to sentence punctuation.

[/Personal/SoftwareDevelopment] permanent link

2008 Jan 17 - Thu

Open Document Format Alliance Refutes the Burton Group Report on ODF

I have to say, starting out, that I do use Open Source productions as much as I use Microsoft products. Each has its merits.

On the other hand, Microsoft would prefer that I use their products exclusively. I don't think so. Interoperability is not really an allowed word in their books. The words extend, embrace, and extinguish do feature prominently.

Much has been said, written, and done with regards to their OOXML format, which they are trying to ratify. As it turns out, the standard they are presenting does not fully represent their formats. Perhaps with some recent releases, that may be resolved. But with a standard, an incomplete one at that, already at 6000 pages, additional information releases just adds more bulk.

On the other hand, why can't they just allow an ODF transalator into their main 'Save As ...' menu? Why make things complicated. Oh. Sorry. I know the answer to that one... to keep the competition at bay.

If Microsoft's shareholders were polled, do you think they would want Microsoft to continue in the vein they are currently, or to inject a certain amount of ethical, healthy market competition into the proceedings? After all, they started to thrive once IBM openly published the specs for the PC. Instead, Microsoft publishes only what they need to, and only at the lash of the whip of the European Union.

The basis for today's rant is an article over at Groklaw entitled Open Document Format Alliance Refutes the Burton Group Report on ODFM. The Burton Group had developed a report which has quite a few false facts. ODFA has refuted them. I just wanted to add my bit in publicising the counter to the fear, uncertainty, and doubt.

And I hope reason prevails when the OOXML vote comes up in February.

[/OpenSource] permanent link

Crypto++

The C++ library, Crypto++, has an amazing array of crypto routines, including stream cyphers, block ciphers, message authentication codes, hash functions, public-key cryptography, elliptic curve cryptopgraphy, as well as hold-over insecure and obsolescent algorithms.

I'm using the library for a relatively simple task of using SHA-1 hashing on user passwords for a variety of software applications I'm in the midst of writing: blog routines, IP addressing documentation, and network management.

In the readme file in the archive is a couple of paragraphs I found as succinct descriptions of what to do in contstructors with C++ references and pointers:

1. If a constructor for A takes a pointer to an object B (except primitive types such as int and char), then A owns B and will delete B at A's destruction. If a constructor for A takes a reference to an object B, then the caller retains ownership of B and should not destroy it until A no longer needs it.

2. Crypto++ is thread safe at the class level. This means you can use Crypto++ safely in a multithreaded application, but you must provide synchronization when multiple threads access a common Crypto++ object.

[/OpenSource/Programming] permanent link

2008 Jan 10 - Thu

Additional Configuration Notes for Wt, a C++ Web Toolkit

I need to append some additional notes to my 2007/10/03 Wt Configuration Guide article.

As mentioned in the Wt Ext Widget deployment notes, the Wt toolkit makes use of libraries from Ext JS. There are notes at the bottom of the Wt page itemizing the files needed from the JS Ext 1.1 .zip file.

As a side note, all things Ajax can be referenced one way or another starting at the ajaxian web site. Maybe not quite the truth. More all things Ajax at AjaxPatterns... if you want to get under the hood.

[/OpenSource/Debian/Development] permanent link

2008 Jan 03 - Thu

Being Stopped Out, Accidently On Purpose

Putting stops on trades is a popular mechanism for protecteding 'the trade'. In some situations, if they are not placed properly, they can be used against the trader in their pursuit of profits.

According the exerpt below, which I gleaned from a thread on Elite Traders, putting stops just outside of common resistance and support lines can cause premature failure of what could have been a profitable trade.

I think the writer's comments are worth repeating as a possible explanation for why some traders may think that the market 'is out to get them'.

From my personal experience that is derived from day trading primarily NASDAQ through a proprietary firm, I suspect that the main running of stops by MMs and by large operators occurs when these professionals are able to move the market to a place where the majority of stops are most likely located. IMO such places are usually just beyond natural support or natural resistance levels, e.g. the high or the low of any of the last three trading days, or perhaps just beyond the boundaries of a congestion. Basically, if the market mover is able to take the market to a place where some stops are likely to be located, unless a significant amount of the outsiders/public joins in on the action and carry the price further, the market is likely to reverse shortly after reaching that point. For example if a MM or a large operator suspects that there are orders sitting just above yesterday.s high, then basically he knows that what he buys below yesterday.s high, he will be able to dispose of at a higher price because once he moves the market just beyond yesterday.s high a number of buy orders will enter the market. At this point, once the operator disposes of his original line he is very likely to sell short a significant amount of shares, and so unless there is enough buying from the outsiders to take care of this supply, the market is likely to reverse. This is just one scenario of many, however, I do think that the logic is peaty similar in most cases.

Many of the breakouts are real, such is the nature of market dynamics, however, when the market is in a consolidation or in a congestion it.s somewhat easy for the big operators to make their profits by running the market back and forth. Actually, if your stops are constantly being hit, it may be a sign that you are in a congestion. Personally, I try to stay out of the market as soon as I suspect that I am in a congestion, and wait for a good indication that the market is beginning to trend again.

To protect myself, I often do not trade the first breakout of the natural support or resistance, or the congestion. Personally, I.d rather wait for a correction or a second time through. Such strategy is much more conservative and will prevent you from taking many traders, however, IMO I think that being patient and waiting for a high-probability setup is essential if one is to achieve consistent success in day-trading.

This trader is still looking for the traditional retail style of trading of looking for the trends and runs. Rather than staying out of the conditions mentioned, I am still researching and developing some trading methods to stay with the market, bracket it, and make money through these congestive periods.

The other moral of the story is that a trader needs to be careful of where stops are placed. Stay out of the usual areas, and place them in the less travelled paths.

[/Trading] permanent link