2009 May 26 - Tue

VMWare Datastore Browser

I'm sure the VMWare people have hidden this on purpose... just so you think you are forced into installing command line utilities or buying licensing for their management products.

Anyway, I have a couple of ESXi 3.5 U4 servers installed. I created a Virtual Machine on one server, then used the SSH scp command to copy the Virtual Machine from one host to the other. That is all well and good, but how do you get it to show in inventory?

The answer to that is to run the VMWare Infrastructure Client. That is no problem. The trick is to click on the Summary tab while in Inventory mode, and right click on the datastore. One can then browse the datastore. And one can right click on a .vmx file to register the Virtual Machine in Inventory. That same menu allows one to upload and download images from a local computer.

I think it would have been more intuitively obvious to have the datastore(s) listed in the left hand tree, but I guess that would make too much sense.

Some random notes on ESXi 3.5 U4:

• One needs to purchase at least the foundation license in order to get the remote command line tools to work
• When in the ESXi console, one can use vmkfstools to create and resize virtual drives. The GUI does not allow the 'thin' command, but the vmkfstools command does. 'thin' is the ability to indicate what the overall size is, but not to preallocate all the space necessary all at once.
• When using an Asterisk based server in VMWare, allocate at least 500MHz to the server in order to maintain non slipping time. More VMWare Timekeeping Best Practices
• Veeam FastSCP: Veeam FastSCP- VMware ESX/ESXi managment tool FastSCP provides a fast, secure and easy way to manage files and bulk copy VMs across your VMware ESX environment.

[/Networks/VMWare] permanent link

2009 May 24 - Sun

VMWare on HP DL360 G6

I recently acquired a couple of decently configured HP DL360 G6 servers. Each boots VMWare directly from an embedded USB Token. Now that is a server that works right out of the box. And it did.

It is an excellent ability to be able to use HP's management tools to view the console remotely. I've not laid hands on the server, but I have almost complete visibility into the unit. There are about 20 different temperature sensors, I can monitor and cap power usage, evaluate processor utilization, and much more. Remote access to CDRoms is also available through a virtual media Java mechanism. I'm using that now to upgrade to U4 of ESXi.

HP has their own special image and after a bunch of searching, I found it at Software Depot Home.

I had tried the U4 version from VMWare's site, but it wouldn't install itself in the correct spot. That is when I figured that HP must have a special version. Don't try to install HP's v8.20 of management tools either. They are frought with installation problems.

[/Networks/VMWare] permanent link

Sun Java 6 on Debian Lenny 5.01

I'd think Debian Linux should get simpler all the time. Maybe not. My tricks from Installing Sun Java on Debian Lenny didn't yield the desired results.

Perhaps if I had performed a standard Debian Lenny Desktop install, I would not have had this problem. Instead, I took the expert/custom route. During the beginning of the install of Debian Lenny, I chose the advanced options where I could install a KDE desktop. I'm not sure if the standard variation would have worked out of the box, but, whatever, this one didn't.

I had to go to Debian Tutorials to find the answer, which was a basic one-liner, a long, but it required one preparation step before hand. The directory /usr/lib/iceweasel/plugins needed to be corrected first. Then the one liner could be performed: ln -s /usr/lib/jvm/java-6-sun-1.6.0.12/jre/plugin/i386/n s7/libjavaplugin_oji.so /usr/lib/iceweasel/plugins/

With that in place, I can now run Java applets in IceWeasel.

[/OpenSource/Debian/Development] permanent link

Enable SSH on VMWare ESXi

VMMWare ESXi is installed and started with SSH disabled. To enable it is an unsupported option, as it allows a user access to the console, operating system and associated file system.

My primary reason for accessing the VMWare ESXi file system (vmfs), is the ease in which one can get ISO images on to the system. When running the VMWare Infrastructure Client, during the creation of a virtual machine, the virtual CD Drive can be attached to an ISO image resident in the DataStore, with the DataStore basially being the vmfs file system.

So to get read/write access to vmfs, one needs to activate SSH on VMWare:

• At the console of ESXi host, press Alt-F1 to access bypass the simple management window and gain access to the console window.
• There is no prompt and no text echo, but type unsupported and hit the enter key.
• Enter the password you've assigned for root.
• A prompt of ~ # will become visible.
• Use vi to edit /etc/inetd.conf.
• Find the line that begins with #ssh and remove the #, and save the file.
• Use ps | grep inetd to find the existing inetd process id.
• Restart the process with kill -HUP id.
• You will now have access via SSH.

After logging in, the default datastore can be found at /vmfs/volumes/DataStore1. I created a sub-directory there named ISO to hold my ISO images. The directory and files are accessible from the VMWare Infrastructure Client when creating a new Virtual Machine. ISO files can be retrieved with the wget command.

I havn't done it yet, but one could add a .ssh directory on /root, do the appropriate magic (covered in another article), and login with an ssh key rather than root password.

Much of the information here was extracted from a couple of web sites, with VM-Help being the primary one. It's forum entries have additional useful information.

[/Networks/VMWare] permanent link

2009 May 16 - Sat

High Performance Messaging

The most mention I hear of low latency trading is from data vendors who say their market data feeds are 'the best' because they are nearest the data source, and that their infrastructures have been designed for high availability and performance.

I've always thought though, that market data source adjacency forms only a portion of the overall delay budget. It seems to me that 'closeness' to the execution side of things is just as important, if not more so. This is confirmed through some articles I've recently seen that discuss some colocation facilities situated to optimally provide this 'betweenness', aka Smart Proximity Hosting.

The third aspect of low-latency trading resides within the compute engine, the engine that receives market data, calculates the trades, performs risk management, sends out the execution requests, and receives the execution confirmations. Copying data from and into packets as well as receiving and transmitting them can be a time consuming processing. Buffer management is a serious consideration in high frequency trading scenarios (the concept of high-frequency trading being intimately intertwinded with the concept of low-latency market data feeds).

I came across Topics in High-Performance Messaging in relation to someone's generic question about how to test throughput on links. Buffer sizing is one of many important topics in optimizing throughput and reducing latency. This paper makes obvious many of the hidden gotchas for the compute engine, the links (how many, what kind, and how they are joined), the feed types, and the supporting L2/L3 infrastructure. Even though I came across it as a generic response to throughput testing, I see it is written by a group that has spent much time on investigating low-latency issues in trading. I see the article as being very usful for optimizing additional milliseconds/microseconds out of the execution cycle time.

Another view on this low-latency issue arises in a blog entry from The Blog of James: Does the need to process volumes of data prohibit lower latency?

There is a news site dedicated to news regarding low latency trading issues: low-latency.com.

[/Trading/AutomatedTrading] permanent link

Martians

In terms of managing addresses on for the public internet, there are a set of address ranges which one should never see... publically. Privately, that is, within someone's local network, they can be seen, are seen, and should be seen.

• 0.0.0.0/8: not seen as an address but as a default route.
• 10.0.0.0/8: a common internal rfc 1918 range.
• 127.0.0.0/8: localhost addresses, ie, loopbacks on individual machines, with 127.0.0.1 the most common. I've used addional addresses for setting proxy forwarding with ssh port forwarding configurations
• 169.254.0.0/16: rfc 3927 for internal networks without dhcp and no addressing structure
• 172.16.0.0/12: a common internal rfc1918 range.
• 192.0.2.0/24: rfc 3330 for documentation and example code
• 192.168.0.0/16: a common internal rfc1918 range.
• 198.18.0.0/15: rfc 2544 network benchmark tests
• 223.0.0.0/8: reserved
• 224.0.0.0/3: multicasting

More information on IPv4 addressing can be found at Wikipedia.

[/Networks] permanent link

2009 May 03 - Sun

Open Source Site of the Day: ModSecurity -- Open Source Web Application Firewall

mod_security is an actively maintained web application firewall. From my reading, it looks like it is a filter for processing web requests before they hit a company's main web server. It performs a series of different check and balances: looks at http headers for correctness, does common checks on field content so as to prevent injection attacks, and through a command language, can perform so complex analysis within a request as well as across requests.

In can be used as an appliance in-line or out-of-line, or can be used as a module right on the web server. The company defines their 'Web Application Firewall' as a reverse proxy with additional security related features.

Is is an adjunct to a firewall, which can only do some basic session state analysis. There is one slide in a presentation on the site which provides a good summary of its capabilities:

• Monitoring: know what happened
• Detection: know when you are being attacked
• Prevention: stop attacks before they succeed
• Assessment: discover problems before the attackers do

It looks like mod_security is a very good tool for helping web developers protect themselves from things they don't know. Web developers focus more on content and less on security. This tool helps rebalance the problem.

SANS is a good place to start learning about security.

[/OpenSource/SiteOfTheDay/D200905] permanent link

Time Series Analysis on RRD Files

Crist Clark, in a posting on the NANOG mailing list, started an interesting thread on analyzing network traffic based upon frequency analysis rather than the traditional time based analysis. He started the thread by asking about Fourier Analysis on network traffic time series. A number of responses indicated that Wavelet Analysis might be the 'more modern' approrach. This type of analysis has been used for Network Traffic Anomoalies Detection. The responses indicate that operating systems can be deduced through analysis of RTD (Round Trip Delay) of ping generated traffic.

The thread started with:

Crist Clark started:

Has anyone found any value in examining network utilization numbers with Fourier analyses? After staring at pretty MRTG graphs for a bit too long today, I'm wondering if there are some interesting periodic characteristics in the data that could be easily teased out beyond, "Well, the diurnal fluctuations are obvious, but looks like we may have some hourly traffic spikes in there too. And maybe some of those are bigger every fourth hour."

Dave Plonka Responded:

Such techniques are used in the are of network anomaly detection. For instance, a search for "network anomaly detection" at scholar.google.com will yield very many results.

Our 2002 paper, "A Signal Analysis of Network Traffic Anomalies" [ACM SIGCOMM Internet Measurement Workshop 2002, Barford, et al.], is one such work. We mention that we use wavelet analysis rather than Fourier analysis because wavelet/framelet analysis is able to localize events both in the frequency and time domains, whereas Fourier analysis would localize the events only in frequency, so an iterative approach (with varying intervals of time) would be necessary. In general, this is the reason why Fourier analysis has not been a common technique used in network anomaly detection.

That work used data stored in RRD files at five minute intervals. Our subsequent work used data stored at one second intervals, again in RRD files.

Anton Kapela had a couple of messages and a link (look for Kapela):

Indeed, there are. Interesting things emerge in frequency (or phase) space - bits/sec, packets/sec, and ave size, etc. - all have new meaning, often revealing subtle details otherwise missed. The UW paper [Barford/Plonka et. al] is one of my favories and often referenced in other publications.

Along similar lines, I presented a lightning talk at nanog that demonstrates using windowed Ft's (mostly Gaussian or Hamming) in three-axis graphs (i.e. 'waterfalls') available in common tools (buadline, sigview, labview, etc) for characterizing round trip times through various network queues and queue states. Unexpectedly, interesting details regarding host IP stacks and OS scheduler behavior became visible.

I want to suggest that time windowed Ft might be a reasonable middle ground, certainly for Crist's case. Naturally, the trade-offs will be in frequency accuracy (ie. longer window) vs. temporal accuracy (ie. short window). Another solution for your needs might be cascaded FIR "bandpass" filters, but again, you're subject to time/frequency error trade-offs as related a filter's bandwidth.

While you're at it, consider processing your time series data into histogram stacks, or nested histograms. I haven't specifically seen a paper covering this, but another UW gent (DW, are you reading this?) used to process their 30 second ifmib data into a raw .ps file, and printed this out weekly/daily. The trends visible here were quite interesting, but I don't think much further work was done to see if anything super-interesting was more/less visible in this form than traditional ones.

... one point - since packets/bits/etc data is more monotonic than not (math wizards, please debate/chime in) and since it's not a 'signal' in the continuous sense, you might find value in differentially filtering the input data *before* FT or wavelet processing. This would serve to remove the weird-looking "DC" offset in the output simply by creating a semi-even distribution of both positive and negative input sample values.

[/OpenSource/Debian/Monitoring] permanent link

Routing Within An ISP

Many ISP's I've seen have had two routing protocols implemented: BGP to talk to the 'internet' with the external /24 and shorter prefixes, and an internal routing protocol such as EIGRP or OSPF to handle the internal /24 and longer prefixes. The internal protocol would be running on all ISP devices and would handle all infrastructure devices and customer links. For a multi-homed ISP, BGP would need to be running on all internal devices that form internal paths from one external link to another. This provides an ability to choose an appropriate exit point for any traffic generated from within an ISP destined for the external network. Some ISP's 'cheat' by generating default routes to the nearest exit and having BGP reside only on edge devices. Some optimum paths will be missed using this simplified arrangement, particularily if an ISP is connected to non-transit neighbors.

Current best practices make expanded use of BGP. BGP, known as IBGP, is used extensively within the ISP to carry customer prefixes. The internal routing protocol such as OSPF or EIGRP is used simply for carrying infrastructure routes such as loopback addresses and link addresses.

With this arrangement, it is then easy to make use of MP-BGP (Multi-Protocol BGP) to handle the various requirements for carrying MPLS links.

One presentation at RIPE shows some basics of BGP Best Practices.

[/Cisco] permanent link

64 Bit Data Models

As we move to 64 bit processors, variable types and their widths change. I had originally thought that there would be a consistent naming convention as one moved from 32 bit programming to 64 bit programming. At a 64 Bit Wiki Entry, I find that such is not the case. Different compilers choose different ways. For example the Microsoft VC compiler will use the LLP64 model which keeps an int as 32 bits. This is something that one needs to keep in mind when re-compiling software created for 32 bit processors in a 64 bit environment.

In the same article, mention is made that it is a good habit to make use of 'ptrdiff_t', which is declared in , when subtracting two pointers and using the result.

[/Personal/SoftwareDevelopment] permanent link