2013 Mar 30 - Sat
GNS3 206-unable to create UDP NIO
GNS3 has it's benefits in terms of being able to test out reasonably complicated scenarios in
terms of routing. But it does have it's quirks and issues.
One issue with the GNS3 0.8.4-RC2 release is in how it deals with its topology file. Well,
now that I think of it, there are two issues I recently encountered with this build. If others
encounter the same thing, I've included some work arounds.
The first issue I encountered is that when I built a topology, there were a few routers
for which I did not create links. When I saved and then reloaded the topology file, the
unlinked routers did not show up on the diagram. So.... whenever adding routers, ensure
there is at least one link provided. This ensures that router shows up next time in the drawing.
The second issue is somewhat more insidious. A reasonably complicated set of routers
and links can be created. The topology can then be saved. When restarting it a subsequent time,
the following error occurs:
*** Warning: Connecting to resulted in:
206-unable to create UDP NIO
*** Error: errors during loading of the topology file, please correct them
The solution to this is to manually edit the topology file and reorder the sections in
ascending order, something like:
127.0.0.1:7200]
... section content
[127.0.0.1:7201]
... section content
... simlar sections
2013 Mar 21 - Thu
GNS3 And OSPF-4-ERRRCV: Received invalid packet: Bad LLS Checksum
GNS3 works quite well for simulating and testing Cisco routed networks. However, when
creating links between routers, there are times when the following message appears on the
console of some interconnected routers:
OSPF-4-ERRRCV: Received invalid packet: Bad LLS Checksum
I am uncertain as to the source of this issue. Some forums recommend applying the command to each interface:
- ip ospf network point-to-point
This did not work for me. Instead I tried the following at the interface level:
- ip ospf authentication message-digest
- ip ospf message-digest-key 1 md5 7
The above md5 digest key settings have stopped the errors from occuring.
2013 Mar 03 - Sun
FOFOA - (Gold) Friend of Friend of Another
FOFOA represents the continuation of
an ongoing interpretation of a very long running
commentary on the shadowy world of gold, oil, and international currencies.
The linked commentaries go back for more than ten years, and provide a whole new
perspective on gold price 'discovery' mechanisms.
One fascinating aspect of reading through some of the 'Thoughts', was that the current
depressed prices of gold producers could be attributed to the fact that world wide aggregate
gold production is declining on a year to year basis. If there are not more inventories,
nothing to sell, and therefore no value for the company. The issues are much
more complex than that.
As well, the gold producers a bit more diversified than the statement suggests.
2012 Dec 07 - Fri
BGP Default-Information Originate
A brief note on the rules for originating a default route into BGP (copied from a posting
made by Mohammed Mahmoud):
- default-information originate + redistribute static (or any dynamic routing protocol having the default route - you may filter only the default route)
- network command but must make sure the default route is present in the routing table
- issuing the neighbor default-originate command. This method does not require the presence of the 0.0.0.0/0 network in the routing table of the advertising router
Additional notes: The configuration of the default-information originate command in
BGP is similar to the configuration of the network (BGP) command.
The default-information originate command, however, requires explicit
redistribution of the route 0.0.0.0. The network command requires only
that the route 0.0.0.0 is present in the Interior Gateway Protocol (IGP)
routing table. For this reason, the network command is preferred.
2012 Oct 23 - Tue
HostDatastoreSystem.QueryVmfsDatastoreCreateOptions
I was having problems with getting a VMWare ESXi 5.1 Host to register properly with vSphere 5.0 (probably something seriously wrong
with that whole concept all by itself). During the install of ESXi 5.1, the local drives were attached and formatted for
vmfs capability.
Since I couldn't fix the registration problem, I rolled the host back to ESXi 4.1. When it came time to reformat and make the
drives useful for vmfs, I encountered an error along the lines of:
VMware: Call "HostDatastoreSystem.QueryVmfsDatastoreCreateOptions"
for object "ha-datastoresystem" on ESXi "SERVERNAME" failed
The comments of VMPros Virtualization
provided the solution (with an enhancement included by someone in the comments):
- I just did a .ls /vmfs/devices/disks/. from the shell to identify what was the vml I had to use.
- To find the vml I just ran the following command .partedUtil get /vmfs/devices/disks/VMLID. and I got an error with the VMLID corresponding to the new drive.
- After that, I just ran your .dd if=/dev/zero of=./vmfs/devices/disks/VMLID. bs=512 count=34 conv=notrunc. and I was able to add the hard drive.
Linux File Tools
In reviewing some Qemu/KVM/VMWare migration tools, I came across some other interesting file
system tools. For my own reference for later:
- dm-crypt: a device-mapper target which provides transparent encryption of block devices using the Linux cryptoapi
- cryptsetup: setup cryptographic volumes for dm-crypt (including LUKS extension)
- Linux LUKS Volumes: Examples
- Portable Encrypted File System on Loop Device
- pv: Monitor the progress of data through a pipe
- dd: using with pv on lvm based devices: dd if=/dev/myvolumegroup/mylogicalvolume bs=4096 | pv | ssh targetmachine dd of=/dev/myvolumegroup/mylogicalvolume bs=4096
- kpartx: Mount raw image files, such as drive images created by dd, also for mounting VM disk images.
And then I ended up at Forensics Wiki with, they say, over 780 pages of content.
2012 Oct 22 - Mon
Migrating a QEMU/KVM Image to VMWare
I recently found out that ISPs and Managed Service Providers are able to obtain a different
licensing schedule from VMWare for hosted services. For some facilities, just starting up,
this may be an attractive alternative to VMWare's mainstream licensing policy. Expecting that
growth will be a primary driving factor, licensing is based upon a monthly useage fee. One has
to become a partner with VMWare in order to take advantage of this program.
With this licensing plan in place, a company is migrating their virtualized guests from QEMU/KVM over
to VMWare.
For Linux and Windows boxes, VMWare has a Converter program. For all other guests, it would,
at first blush, appear to be a hard problem to solve. However, such is not the case. Instead,
it is quite easy to convert and migrate guests from QEMU/KVM over to VMWare.
For Linux guests, while still active on QEMU/KVM, I installed the VMWare tools. This
ensures network connectivity once the guest is migrated. On Wheezy/SID Debian, the following
command gets the tools loaded:
- apt-get install open-vm-tools open-vm-dkms
Then shutdown the guest.
At the QEMU/KVM host level, run the following command on the image file to prepare it for
transfer to VMWare:
- qemu-img convert filename.img -O vmdk filename.vmdk
On the the VMWare vSphere Client, I'll create new guest with a dummy harddrive component.
As I have had issues trying to use VMWare's GUI to upload large files into a datastore.
I have also had timeout issues trying to SCP files to a VMWare Host.
Instead, I login via SSH into the destination VMWare host, and copy files from there via SSH from the
QEMU/KVM Host server. The file gets placed into the newly created guest directory. I then
edit settings to delete the temporary connection and connect in the real harddrive/vmdk file.
The guest can be started and should run as normal.
2012 Jul 20 - Fri
Quagga Installation And Configuration Notes
Even though it hasn't hit a 1.0 release yet, it sounds as though Quagga, the open source "advanced routing software package that provides a suite of TCP/IP based routing protocols",
has become a mature package for use in various parts of a network design. I am looking
to use it in several different scenarios:
- with it configured on two different servers, use it as a BGP Route Reflector pair in a small ISP
- use it in a hosted virtualization environment where it can speak BGP with the hosting facility, and break out smaller subnets to various local virtualized guests
- communicate black-hole prefixes to an up-stream provider for assistance with remediation of Denial Of Service (DoS) attacks (some do it this way rather than using BGP community strings
With Debian Wheezy, it starts with a simple package install:
apt-get install quagga
There are then a number of manual steps to be performed. I am interested in the following routing daemons:
- zebra: which looks after local interface and routing table maintenance. If Quagga is configured
only as a BGP Route Reflector, I don't think Zebra is required. When BGP or OSPF are used locally, then Zebra is required
- ospf: local network routing
- bgp: route reflecting, and inter Autonomous System (AS) routing
Sample configuration files have to be copied over:
root@quagga01:/etc/network# history
cp /usr/share/doc/quagga/examples/bgpd.conf.sample bgpd.conf
cp /usr/share/doc/quagga/examples/ospfd.conf.sample ospfd.conf
cp /usr/share/doc/quagga/examples/vtysh.conf.sample vtysh.conf
cp /usr/share/doc/quagga/examples/zebra.conf.sample zebra.conf
In /etc/quagga/daemons, the appropriate daemons need to be turned on with a 'yes'.
Some privilege and ownership changes are required:
chown quagga.quagga zebra.conf
chmod 640 zebra.conf
chown quagga.quagga bgpd.conf
chmod 640 bgpd.conf
chown quagga.quagga ospfd.conf
chmod 640 ospfd.conf
chown quagga.quaggavty vtysh.conf
chmod 660 vtysh.conf
The other file for content modification is debian.conf. This is a file of command
line parameters for each daemon. Addresses from which management telnet is accepted would be
placed in to this file as well.
The built in 'shell', vtysh, uses an external program to provide listings. It seems to be a
bit intrusive, but everyone seems to deal with it. It requires a 'more' or 'less' external program.
Create a permanent shell variable with:
echo VTYSH_PAGER=less >> /etc/environement
Create a current shell variable with:
export VTYSH_PAGER=less
vtysh appears to bypass any username or password requirements. On the otherhand, vtysh does
facilitate local scripting with the -e execute command line parameter.
vtysh documentation indicates that an integrated configuration file is not on by default. However,
it looks as though the Debian package configuration turns it on. Therefore the initial configuration is
loaded from the individual daemon files, consolidated, and saved in the Quagga.conf file. In
the vtysh.conf file, the paramter 'service integrated-vtysh-config' appears to control this.
With this included in the file, configuration changes can only be made through vtysh. The telnet
modes are only read access and for routing evaluation.
Otherwise, once services have been started, local telnet can be used to
connect to the daemons. Usernames, passwords, and enable are enforced for telnet.
Port numbers are identified in the /etc/services file with:
grep 260 /etc/services
For example:
- telnet localhost 2601 # for zebra
- telnet localhost 2605 # for bgpd
The daemons can be started with:
/etc/init.d/quagga start
Useful tricks and tips for Quagga can be found at
Sourceforge Quagga Tips and Tricks.
2012 Jul 02 - Mon
Open vSwitch Configuration on Debian Sid/Wheezy
In my previous article, I went through the steps of installing
Open vSwitch for use within a Qemu/KVM solution
in substitute of the usual Linux bridge utility.
OVS works differently from normal Linux networking in at least a couple of different ways. The first way is that a
single regular bridge in Linux is designed to handle a single VLAN, ie, one bridge per VLAN.
This is why one reason why I decided to forgo using the standard Linux bridge utilities. With OVS, an OVS bridge takes the
VLAN trunking in stride, and can bridge out trunk ports or regular access ports. A trunk port uses the command of
'ovs-vsctl add-port br1 tap0', whereas an access port connected to VLAN 101 would use 'ovs-vsctl add-port br1 tap0 tag=101'.
Both commands attach the tap0 interface to the bridge br1, the first as a trunk, the second as an access port.
The second difference relates to something which I'll loosely define as non-persisted network configuration versus
persisted network configuration. Using regular Linux bridging tools, the individual commands for bridging are
incorporated into the /etc/network/interfaces stanzas. These stanzas are run on power up and power down, and manually when
changing network interface state at the command line. The state of the network is not remembered from
session to session (ie, through reboots).
With OVS, things work a bit differently. Commands, as entered at the command line, not only update the
appropriate network state, but the requested network configuration is recorded into a database maintained by OVS.
Thus, when rebooting the physical machine, what existed prior to reboot, will be automatically recreated after reboot.
The punch line is that OVS configuration commands should not be entered into the /etc/network/interfaaces file,
as the commands will be redundant, and may even cause problems on reboot if entered there.
For attaching bridge port 1 to eth1, the following commands are used:
ovs-vsctl add-br br1
ovs-vsctl add-port br1 eth1
Then the stanzas for /etc/network/interfaces are:
allow-hotplug eth1
iface eth1 inet manual
pre-up ifconfig $IFACE up
post-down ifconfig $IFACE down
auto br1
iface br1 inet manual
pre-up ifconfig $IFACE up
post-down ifconfig $IFACE down
auto eth1
Using the above, I see eth1 come up automatically, but br1 still needs a 'ifconfig br1 up'. Perhaps it is an affect of
when the OVS network services are started in relationship to when the /etc/network/interfaces file is evaluated.
Open vSwitch VLANs
has a good reference for configuring access ports.
How to Use Open vSwitch with KVM
deals with the special up/down files needed for Qemu/KVM.
In a nutshell, two files are required. The first, /etc/network/ovs-ifup which adds a tap interface to br1:
#!/bin/sh
switch='br1'
ovs-vsctl add-port ${switch} $1
/sbin/ifconfig $1 0.0.0.0 up
The second, /etc/network/ovs-ifdown removes the tap interface:
#!/bin/sh
switch='br1'
/sbin/ifconfig $1 0.0.0.0 down
ovs-vsctl del-port ${switch} $1
Be aware that the above represents two scenario: the physical source port is an access port or a trunk port. If the
physical port is an access port, then the access port traffic will be delivered to the guest, as expected. On the other hand,
if the physical is a trunk port, then all VLAN traffic on the physical port will be replicated to the guest machine's
network port, and the guest machine will need to know how to work with 802.1q tagged frames.
To deliver access ports to the guest from a trunked physical port, the following example represents delivering
VLAN as an access port to a guest machine. /etc/network/ovs-ifup:
#!/bin/sh
switch='br1'
/sbin/ifconfig $1 0.0.0.0 up
ovs-vsctl add-port ${switch} $1 tag=101
For /etc/network/ovs-ifdown:
#!/bin/sh
switch='br1'
/sbin/ifconfig $1 0.0.0.0 down
ovs-vsctl del-port ${switch} $1
Then the -net parameter like the following can be used when starting Qemu/KVM:
-net tap0,script=/etc/network/ovs-ifup,downscript=/etc/network/ovs-ifdown
A few other useful status commands:
ovs-dpctl show br1
ovs-ofctl show br1
2012 Jun 29 - Fri
Installing Open vSwitch on Debian Sid/Wheezy
Open vSwitch appears to be the up and coming mechanism
for connecting virtual guests to the physical network through bridges, vlans, and network cards.
Through my previous postings, I described the processes I went through to get basic networking running with
my virtual guests in a QEMU/KVM environment on Debian Sid/Wheezy. After getting basic, simple bridged connectivity
working with bridge-utils, I realized I wanted to make things a bit more complicated. I want to be able to make
use of 802.1q VLAN trunking ports, and to bridge to individual VLANs from various virtualized guests.
I believe it would have been relatively straightforward to add the Debian VLAN module in, and start bridging that way.
But after a bit more research, it looks like Open vSwitch handles that, plus has a number of other capabilities.
Namely, it can deal with sFlow and Netflow exporting. It has other monitoring features as well. On top of that,
it Open vSwitches on different machines can be linked together via tunnels. I understand it also integrates into
OpenStack.
Therefore, I decided to take the big leap and start into Open vSwitch. With the basic knowledge gained from this tool,
I can start to integrate additional infrastructure into the solution.
The first step is to install Open vSwitch. It isn't as simple as expected.
I first shut down and removed existing bridge configurations from the /etc/network/interfaces file. Then removed
bridge-utils:
apt-get remove bridge-utils
Open vSwitch requires a kernel module to be built and installed. It doesn't come pre-built. And I see the reason. The
kernel in Sid/Wheezy is in a state of flux. I started with 3.2.0-1-amd64, but the current is 3.2.0-2-amd64. When
building the kernel module, linux-headers are needed. I had an error like:
FATAL: Module openvswitch_mod not found.
* Inserting openvswitch module
So the following was needed to sync headers and kernel:
apt-get install linux-headers
apt-get install linux-image-3.2.0-2-amd64
With the kernel and headers consistent, the Open vSwitch kernel module is downloaded and built:
apt-get install openvswitch-datapath-source
module-assistant auto-install openvswitch-datapath
The main Open vSwitch modules can then be installed:
apt-get install openvswitch-common
apt-get install openvswitch-switch
Output similar to the following should be obtained:
[ ok ] ovs-brcompatd is not running.
[ ok ] ovs-vswitchd is not running.
[ ok ] ovsdb-server is not running.
[ ok ] Inserting openvswitch module.
[warn] /etc/openvswitch/conf.db does not exist ... (warning).
[ ok ] Creating empty database /etc/openvswitch/conf.db.
[ ok ] Starting ovsdb-server.
[ ok ] Configuring Open vSwitch system IDs.
[ ok ] Starting ovs-vswitchd.
[ ok ] Enabling gre with iptables.
I can now start to experiment with Open vSwitch.
QEMU/KVM Bridge/Tap Network Configuration
QEMU has a number of different ways of connecting the network to a virtualized guest, or vice-versa,
connecting a virtualized guest to the network. The mechanism providing the most flexibility
involves using a bridge mechanism plus a tap mechanism.
I'll resort to a cattle culling metaphor to explain the basic concept. Each cow represents a packet. The cows
from the field are directed from the fields into a holding pen, which is loosely represented
by the bridge concept. From the holding pen, individual cows are selected and
directed to their destination, which is loosely represented by the tap.
From an implementation point of view, for the purposes of this discussion, will bond together an external physical
interface (such as eth1) with a series of internal interfaces connecting to the virtualized guests (tap0, tap1, ...).
The tap interfaces connect the virtualized guest to the bridge and the bridge works to get those packets
out onto the physical interface, and for externally arriving packets, works to get the packets to the
appropriate tap interface.
On a Linux Debian Wheezy system, things are straight forward to configure. The system, for the purposes
of this example, has two external interfaces: eth0 and eth1. I have eth0 configured with private ip addresses
behind a firewall and is dedicated for physical host management. The other interface, eth1, is connected to the
public interface side of the network. The physical machine has no addresses on this side to help reduce the
risk of compromise of the physical host. For different environments, an ip address could be assigned to the bridge
interface, which would result in the physical machine being reachable on the public side of things.
Even more sophistication can be obtained with VLANs on the external interface. This topic is best left for another
blog entry, and will be covered with a discussion of Open vSwitch.
Before looking at the actual implementation commands, I need to mention two more caveats. The first is that
the following example turns off spanning tree related commands. The configuration is basically a stub off a physical
network. If multiple physical interfaces will belong to the same bridge group, then spanning tree will need to
be enabled. That would be a discussion for one or more separate blog entries.
The second caveat is that the example is based upon QEMU being used with root level privileges. Many other blog
entries use this type of networking in non-root scenarios, and are thus a bit more complicated due to the fact that
sudo is required for bringing up tap connections.
To ensure bridge utilities can be used, the result of the following should be CONFIG_TUN=m or CONFIG_TUN=y:
grep CONFIG_TUN= /boot/config-`uname -r`
Ensure the bridge utilities are installed. The assumption is that qemu/kvm are already installed and operational.
apt-get install bridge-utils
Here is an extract from the /etc/network/interfaces file:
auto eth1
allow-hotplug eth1
iface eth1 inet manual
pre-up ifconfig $IFACE up
post-down ifconfig $IFACE down
auto bvi1
iface bvi1 inet manual
pre-up brctl addbr bvi1
pre-up brctl addif bvi1 eth1
pre-up brctl stp bvi1 off
pre-up brctl setfd bvi1 1
pre-up ifconfig $IFACE up
post-down ifconfig $IFACE down
post-down brctl delif bvi1 eth1
post-down brctl delbr bvi1
The eth1 interface is automatically started. For this example, I have used Cisco terminology for the bridge
interface: bvi. BVI is an acronym for Bridged Virtual Interface. For this example, I have numbered the bvi interface as bvi1 to
match that it is attached to eth1.
Under 'iface bvi1 inet manual' stanza, there are a series of pre-up commands. These are commands which are operable
at the command line as well. The command brctl is used to create the bvi1 interface, add eth1 to the bridge group,
turn spanning tree protocol off, and then set a forwarding delay. The interface is then brought up.
When shutting down the interface, eth1 is removed from the bridge group, and then bvi1 is deleted.
Two more files need to be created with execute privileges. The first is /etc/qemu-ifup. This is a file
which qemu executes in order to connect the virtualized guest network interface to the bridge:
/sbin/ifconfig $1 0.0.0.0 promisc up
/sbin/brctl addif bvi1 $1
The other is /etc/qemu-ifdown which qemu uses when the virtualized guest exits:
/sbin/brctl delif bvi1 $1
I add the following parameters to the qemu startup command:
-net nic,vlan=0,model=virtio -net tap,vlan=0,ifname=tap0
'-net nic,vlan=0,model=virtio' sets the type of network card to emulate. 'qemu -net nic,model=?' can be used to obtain
a list of available devices. 'virtio' is a driver used to optimze the speed of communications between the guest and the host.
'tap,vlan=0,ifname=tap0' creates the tap0 interface, and it is with this parameter, that the /etc/qemu-if? commands
are used for joining the tap interface to the bridge.
Ip addresses can then be assigned statically or via dhcp within the guest, and all network operations operate
as though the guest is directly connected to the network.
A few web sites I encountered with additional background material:
2012 May 22 - Tue
Shrew Soft Inc. Ipsec Vpn Client
Cisco is migrating away from their Ipsec based VPN Client Software, and are pushing
their SSL client traffic encryption solutions. As a consequence, since I want to keep using
Ipsec based VPN connections, it was time to look for an alternate client softare package.
I came across Shrew Soft Inc. Ipsec Vpn Client.
I first installed the
release labelled as 2.1.7 stable on to Windows 7 64. It didn't seem to connect to a Cisco IOS based
VPN server. After troubleshooting the connection for a bit, I upgraded to the version labelled as
2.2.0-beta-2. That version worked successfully.
The Shrew Soft Ipsec Vpn Client starts up quickly, has a number of handy troubleshooting
views, and makes it easy to copy and re-use configuration files on a number of different
client machines.
I was quite impressed that it readily and properly dealt with Ipsec server supplied split tunnel
access-lists, set up proper routes in the workstation routing table, and even offers a pre-signin
vpn connection option.
Jimmy Larsson's
Config example: Static to dynamic IPSec
helped with the configuration on the Cisco IOS Router/Server side of things.
|
