2012 May 09 - Wed

LXC Linux Containers on Debian

As part of the virtualization solutions I'm working on, I wanted something a little lighter weight than a full Qemu/KVM solution, which has a higer overhead than a kernel oriented container mechanism. There are many services that can be run with a common operating system. Containers provides the ability for solid segregation of services at the price of very low overhead.

For a container style of virtualization, I had been looking at OpenVZ (which is used by Parallel's Virtuozzo, I believe), Linux-VServer, and VirtualBox. However, on Debian's Upgrade Path to Wheezy, I see that Debian is deprecating OpenVZ and Linux-VServer.

Independently of this, I had earlier come across lxc Linux Containers. As such, it looks like LXC is the path forward for light weight, container-based virtualization.

[/Virtualization] permanent link

2012 May 04 - Fri

C++ OpenSource IDE

Many moons ago, I wrote about a few opensource IDE's I've encountered for building C++ code. I just wanted to update the record to say that I have now been using NetBeans as my preferred IDE for C++ development.

The interface is multi-window, has good compile time error messages, has some built-in performance monitoring indicators, does debug single-stepping easily, and overall handles itself quite well for modularizing and maintaining code. The make system is also easier to work with.

[/Personal/SoftwareDevelopment] permanent link

Qemu/KVM Headless Console

Most examples for beginners using Qemu/KVM assume that the user is starting up guest operating systems from a desktop running some sort of X11 windowing environment. As I need to host guests on servers with the servers having a minimal installation, this article reflects the very easy experience of running a guest and connecting to it with out the safety net of an X11 installation.

In a previous article, Virtualization on Opensource Platforms, I discussed the steps of installing the Qemu/KVM virtualization tools on Debian Wheezy. I was able to immediately start up with a local image in a local console.

This time, I want to create a new install of a guest Debian Wheezy on my existing Debian Wheezy host. The first step was to wget the latest Debian Wheezy iso and put it into a sub-directory called iso.

An image file needs to be created from which the guest will run. There are various ways of doing this from a thin/thick/sparse/initialized-with-0 perspective, but I'll use the shortest way of getting it going:

qemu-img create images/test.img 4g

This creates a file in called test.img in the sub-directory images, and makes the file 4G in file.

The next step is to boot from the iso and install to the img file. At the same time, I want to access the console remotely. To do this, one uses the vnc protocol. The base tcp port for vnc is 5900. As there may be multiple guests running, each needs a unique instance number. I ran the following command line:

qemu -hda images/test.img -cdrom iso/debian-testing-amd64-netinst.iso  \
 -boot d -m 512 -cpu kvm64 -name test -vnc :2 -no-reboot -enable-kvm

This associates the image file with drive hda, associates the cdrom device with the iso, d says boot from the cdrom, :2 says use display 2 with vnc, which translates into tcp port 5902, and when the install is done, I don't want to reboot automatically.

On another machine, I can connect to the host machine with a suitable vnc viewer, connect to port 5902, and am able to see the console of the installer.

Once the install is complete, I restart the guest with the following command line:

qemu -hda images/test.img   -boot c -m 512 -cpu kvm64 -name test -vnc :2 -enable-kvm

This boots from the drive, and reconnects the vnc server to the same port.

I now have a fully functional guest running in a NAT'd environment on a host. The next article will get bridged networking up and running.

In preparation for more virtualization functionality, some other sites in which i've come across:

• Virtual Square: Discusses the various aspects of VDE, the virtual Distributed Ethernet tool package, which gets guests across different hosts communicating with each other. This, I think, is from where a bunch of network based redundancy and resiliency will arise.
• Hardware virtualization with QEMU: A good intro to Qemu and networking, but unfortunately, rather than discussing KVM, it uses some lesser known hardware virtualizer. Other than that, there is some good discussion about the network side of things.

[/Virtualization] permanent link

2012 Apr 30 - Mon

NexentaStor Root / Expert Mode

Nexenta has a pretty good web interface on their SAN product, and when that isn't good enough, there is always their web console mode. But when even that isn't good enough, and you just have to see what is happening under the hood, there is something called expert mode.

Logging in as admin, provides some good commands when dealing with the various file shares. When I changed into root mode, I've always wondered why it was so lacking. By reading some one else's site, I now know why. Nexenta has disabled most of the root stuff, and you can only access it be going a secret route:

option expert_mode=1
!bash

When finished use exit and then run their resync command to make everything right.

[/Virtualization] permanent link

2012 Apr 26 - Thu

Cloud / Virtualization Management

At the KVM/Qemu command level, there are a number of additional configuration requirements for connecting a network, remote storage, local devices, and other related operating environment devices. I'll write an entry about those later on. In the meantime, I wanted to put together a list of higher level management tools which may eliminate the need for dealing with the lower level infrastructure details. Some of the ones I have encountered are listed below.

For those wondering what is the different between cloud computing and virtualization, NIST has come up with a definition to help clarify the cloud computing side of things at: A NIST Definition of Cloud Computing

• OpenStack: A group of companies organized around Rackspace to come up with a scaleable, resilient virtualization environment.
• cloudStack: Cloud management tools developed primarily through Citrix for the Xen platform. I think Citrix has now joined the Openstack movement now. It has been said (Cloud code merges with Cloudstack) that code from cloudstack will merge into openstack.
• OpenNebula: A seemingly robust, mature cloud management platform widely used by many big name companies. It interacts with Xen, KVM, and VMWare. There appears to be a good set of installation and configuration documents.
• Ganeti: Ganeti is a cluster virtual server management software tool built on top of existing virtualization technologies. It is said that Google uses this to handle their internal cloud/virtualization requirements. There is a good set of top level documents.
• Eucalyptus: One of the first open source cloud platforms I encountered. It has good integration with Amazon's EC2 services. It seems there are partially closed services in Eucalyptus.
• openQRM: openQRM supports and seamlessly manages VMware, Xen, KVM and Citrix XenServer virtual machines. openQRM supports P2V (physical to virtual), V2P (virtual to physical) AND V2V (virtual to virtual) migration. The one difficulty with this is that their web site offers no direct line to downloads or documentation, so I'm not sure just how 'open' it is.

[/Virtualization] permanent link

2012 Apr 24 - Tue

Virtualization on Opensource Platforms

VMWare is a very popular virtualization solution. I first encountered it many years ago when someone introduced me to VMWare Workstation. I was very impressed with the ability to run any operating system and its associated applications fully isolated 'on top' of an existing operating system. It makes for a very effective mechanism for trying out different operating sytems. In addition, with its snapshot capabilities, testing and performing experiments is simplified by having only to roll back to a specific marker in order to start over again with another variation.

Since then, I have progressed to using VMWare in a multi-host environment. In order to make use of the various enterprise grade resilient and redundancy options offered by VMWare, one has to spend enterprise style funds to obtain and maintain those abilities. For smaller organizations, those costs can be prohibitive. When one is used to working with open source software, those costs may also become out of proportion to the costs associated with the hosted operating systems and applications.

The natural inclination is therefore to look for virtualization solutions in the open source world. From a Kernel perspective, the two full virtualization environments I have encountered are Xen and KVM/Qemu. Xen has been supported through the efforts of Citrix.

My interests have been attracted to the KVM/Qemu side of the fence. The letters KVM are an abbreviation for Kernel-based Virtual Machine. The main support web site for this is found at www.linux-kvm.org. Primarily, KVM is a kernel module used for managing the hardware integrated virualization within the host operating system.

Assoicated with KVM is Qemu, which is a machine emulator and virtualizer. It provides related drivers and scaffolding for the guest environment.

When the underlying physical CPU(s) do(es) not have the built-in virtualization capabilities, KVM cannot be used, and Qemu handles the full emulation/virtualization structures for hosting a guest environment.

On current testing version of Debian, starting up a basic virtualization environment is as easy as running a few installation commands:

apt-get install qemu-kvm libvirt-bin
apt-get install qemu

To test the installation, download a test image, and decompress it:

wget http://wiki.qemu.org/download/linux-0.2.img.bz2
bzip2 -d linux-0.2.img.bz2

Running the simplest command may get you and error if you are starting things from the console and do not have X running:

# qemu linux-0.2.img 
Could not initialize SDL(No available video device) - exiting

To get around this when running from the console, change the command slightly:

#qemu -curses linux-0.2.img

That is a quick introduction to the basics of hardware accelerated virtualization on a console based Linux platform.

[/Virtualization] permanent link

2012 Feb 05 - Sun

Inconsistent Option Naming For 'Last Trading Day' and For 'Day of Expiry'

The symbol GLD has options available to it. Option naming doesn't appear to be consistent. For example, for a 2012/02/10 expiry for a GLD put at Interactive Brokers is 'GLD 120210P00167000'. 120210 is a Friday. On the other hand, through the API, I had requested options for 20120518, and I receive in response the options I need, but they are dated 120519, which is a Saturday: 'GLD 120519P00109000'.

Is any one able to shed some light on this? Responses can be sent to ray@oneunified.net and I'll update this post with a summary of what I receive.

[/Trading/Options] permanent link

Tuning VMWare Network Performance

A VMWare Enterprise licensed solution consisting of a number of VMWare hosts, shared storage, plus the VSphere management application is a complex product. Obtaining maximum performance means being able to tune a number of different sub-systems, and to get the various sub-systems working together efficiently.

One of the key sub-systems many people over-look is the network. It appears as though many people think that simply plugging gear into a series of gigibit ethernet ports is all that is necessary for inter-connecting the various VMWare solution components. That may be true for a basic level of functionality, but not for optimal performance.

VMWare has a capability called vMotion. This allows guest operating systems to be migrated live from one host to another. This requires synchronization of sessions between hosting physical servers, which relies on an efficient network connection.

vStorage is a function, very similar in capability to vMotion. In this case it is used to migrate file sets between VMWare DataStores in a live scenario. This requires coordination between shared storage devices, between hosts, and between the host and shared storage devices. If iSCSI is used for accessing shared storage, the network beocmes a doubly critical component of this migration and synchronization.

Here are some ideas for improving the performance of a VMWare solution at the network level.

Speed and Duplex: Sometimes it is easy to overlook the fact that the host server may not always negotiate proper speed and duplex settings with the switch. Both the server and the switch should be checked to ensure that they have both negotiated to 1gbps at full duplex. GigE ports may also perform handshaking. You will want to ensure that the host and the switch are consistent in their settings. Switches with management interfaces will commonly show if there are any duplex mismatches, and will also show if there are errors encountered.

TCP Offload Engine (TOE): Do your Network Interface Cards (NICs) have TCP Offload Engine capability? Has it been enabled? Are the cards compatible with VMWare?

Fault Tolerance: Most modern enterprise servers come with two NICs. This provides for load-balancing and for fault-tolerance abilities. In one scenario, the two NICs can be bundled and connected to one switch for higher overall throughput. The other scenario involves connecting one NIC to one switch and the other NIC to another switch. In this mode, bundling has to be turned off. If one switch becomes unavailable, all traffic will run through the one switch still operating. With this two switch configuration, there are a number of additional optimizations available, which will be described in subsequent points.

Separation of Data and ISCSI Traffic: When hosts use iSCSI for connecting to SAN or NAS devices, the network becomes an integral part of a host/datastore communications. It is commonly recommended that iSCSI traffic should not traverse the same network links as regular host data traffic. Therefore, in a general view, in the two NIC/two switch configuration defined above, iSCSI traffic should be on one NIC and regular data traffic on the other. If you do regular switch access ports for the two types of traffic, the fault tolerance is no longer available, with a solution for this outlined below. Also, all iSCSI preferred ports should be connected to one switch, and all data preferred ports should be connected to another switch.

Use of VLANs: In order to mix traffic types on the NICs, VLANs should be configured on the switches, and the switch ports connecting to the servers should be configured as trunk ports. At this point, at least two VLANs are required: a data VLAN, and an iSCSI VLAN. Typically a third native vlan is supplied, which can be the default vLAN of 1, or some other neutral VLAN. The native VLAN should not be used for any sort of traffic. It is only on 802.1q type VLANs on which QOS can be set. The VLAN configurations should be identical on the two switches, and on each of the two trunk ports connecting to the servers.

Server Separation of Traffic: Once the VLANs have been configured and matched on switch and server sides, the server should be set so that the iSCSI traffic favours one VLAN and the data traffic favours the other VLAN. In the event of a switch failure, both traffic types will use the one link in a slightly degraded state.

Switch Ports: On many switches, each switch port shares bandwidth with other switch ports. This can cause traffic contention, and possibly packet loss. For example, in a Cisco 4500E switch with a Supervisor V, each set of 8 ports on a 48 port blade shares 1gbps of bandwidth to the Supervisor. This is called an over-subscription ratio, and in this case, the ports are over-subscribed in an 8:1 ratio. When working with high instantaneous traffic loads that VMWare hosts can place on their associated iSCSI DataStores, use of over-subscribed ports is not recommended. It is best to use low port count server blades, or higher capacity switches in order to eliminate these issues of bandwidth contention.

Switch Cross-Connects: In a similar vein, when cross connecting two switches, it is best to use non-blocking, non over-subscribed switch ports. Bundling multple ports together to improve inter-switch traffic capacity is also recommended. Just remember that bundling two or more adjacent switch ports on an over-subscribed blade will not yield the desired benefit. Only non-blocking, non-over-subscribed switch ports should be in a bundle.

Switch Spanning Tree: When multiple switches are inter-connected, they should be configured with spanning tree in order to prevent loops in the network. For optimizing traffic patterns in a mixed iSCSI/Data network configured on redundant switches, a common rule of thumb is to keep iSCSI traffic on one switch, and all other data traffic on the other switch. If the host server port connections for iSCSI and data, as explained above, are mixed between switches, then in some cases, one extra switch hop is required, which even at the GigE level, can slow things down. Per-VLAN spanning tree should be implmented. The root for the iSCSI VLAN should be on the iSCSI preferred switch, and the root for the Data VLAN should be on the Data preferred switch. This minimized the amount of cross switch data transfer, therefore optimizing traffic flow.

Switch Port Settings: When devices are turned on while connected to a switch port, or are first connected to a switch port, the switch will typically not allow traffic to flow for a number of seconds while it recalculatese spanning tree. This delay period can be reduced on Cisco switches through the use of three settings having to do with: portfast, bpdu-filter, and bpdu-guard.

Fault Tolerant Routing: This two switch configuration should be supported through redundant layer 3 routing, commonly implemented via HSRP, or VRRP. When setting up with fault tolerant routing scenario, default gateway weighting for the iSCSI VLAN should favour the iSCSI switch, and default gateway weighting for the data VLAN should favour the data preferred switch.

Fault Tolerant VLAN: Some VMWare Fault Tolerant operations require an inter-host heartbeat. It is best to create an additional VLAN for this data and make it available over the trunk ports to the servers. I'd suggest setting preferences of this VLAN to the same NIC as the iSCSI VLAN. This can cause some contention, but can be minimized through the use of QOS, as suggested in a following point.

Quality of Service: When multiple data types from multiple sources attempt to use common links, there is always the opportunity for contention, packet jitter, and subsequent packet loss. I would rank heart beat traffic to be of highest priority (low volume), then iSCI traffic (high volume), then regular traffic (high volume). Suitable QOS settings should be set on the host server side and on the various switch ports to ensure high priority traffic is prioritized and apportioned appropriately.

Management Overhead: Switches and ports may also carry other traffic such as routing protocols, network management data, voice traffic, etc. These other traffic types have to be appropriately analyzed and integrated into the overall VLAN, QOS, and routing architecture.

Jumbo Frames: Common IPv4 traffic relies on packets containing payloads of a maximum of 1500 bytes (1500 byte MTU). Large data streams can be further optimized by adjusting switches and host server NICs to allow larger MTU sizes. Values in the 9000 range are commonly used.

Summary: As you can see, there are many network related optimizations available for obtaining even better performance and reliablity for VMWare based clusters.

[/Networks/VMWare] permanent link