The video is temporarily removed on request from the HP Security team.

1 Virtualisation of Critical Network Services

The first part of this document describes the advantages and disadvantages of virtualisation for given types of services and explains the purpose of building a virtualisation cluster in two geographically distant locations. The second part is devoted to a specific configuration of network devices and to the preparation that needs to be done before connecting individual parts of the virtualisation cluster. It describes actual experience gained in the operation of these clusters in a situation where it was necessary to revise the network topology. The next section of the document explains how to select the appropriate hardware, especially in the choice of storage and hypervisor hardware. It describes the required properties, with an emphasis on diversity, and in particular, how the virtualisation cluster differs from the usual VMware vSphere cluster. This is followed by a section devoted to the practical configuration of a VMware vSphere and a vCenter. The final section presents the results of the measurement of consumption before and after the virtualisation of the critical servers as well as other operating statistics.

The term, virtualisation, is an IT buzzword, referring to technologies that create an abstraction layer between computer hardware and software. This layer creates a transparent, logical structure that masks the physical (real) one. The goals of virtualisation are the simplification of maintenance, easier scalability, higher accessibility, better utilisation of hardware, and improved security. The memory, the processors, the computer network, the storage, the data, or the whole computer system can be virtualised. The virtualisation of a computer enables one physical server to run multiple operating systems. This is called server virtualisation and can be accomplished in different ways. The most frequently used method of server virtualisation is hardwareassisted virtualisation, which requires a special instruction to be set in the CPU (Intel VT-x and AMD-V), but offers the best performance. This method of virtualisation is the focus of this document.

1.1 Resilient virtualisation cluster

To be able to choose the right platform, it is necessary to know and understand the basics of server virtualisation. Everyone who starts with server virtualisation typically installs virtualisation software on a server with large amounts of memory and high capacity drives for all of the virtual servers. This system works well in a test environment. In a production environment, is also necessary to provide protection against various types of accidents that may occur during operation. Current servers have two power supplies, hot swap disk drives in RAID array, multiple CPUs, and many memory modules. However, consideration has to be given to the chance that the server motherboard, the RAID controller, or the power can fail. It is necessary to anticipate infrastructure failures caused by network problems, failure of the cooling system in the server room, or revision of the wiring. All of these cases of failure will result in the unavailability of all of the virtual servers. To counter these potential threats, is necessary to extend the virtualisation system by adding several elements. First, it is necessary to increase the number of hypervisors.

This, in itself, does not contribute substantially, because live migrations of the virtual systems cannot be achieved on two servers alone. Migration of these systems is only possible when all hypervisors have access to shared storage. This storage then becomes a bottleneck, because a failure of the device causes unavailability of the system. Therefore, storage is designed with this in mind: enterprise storage and hard disks. These types of storage have two independent RAID controllers. Each enterprise hard disk has two storage interfaces to connect it to both of the RAID controllers, and this provides resilience in the event of the failure of one of the controllers. The security threats for this type of device are storage power failure, air conditioning failure, or some other disaster. Maintenance of this device is also very complicated, because all actions are performed during run time.

The solution to this problem is to have two storage devices in two, geographically distant locations. All hypervisors have access to both of these storage locations. This makes it possible to move all virtual servers to the first device while maintenance is performed on the second. This ability to move virtual systems to another location is important when there is a planned, structural modification of the wiring, and also in the event of a disaster. A potential bottleneck of this system is an unexpected failure of storage that contains production data. Although this scenario is unlikely, due to the features of enterprise storage, it must be considered. These cases can be solved by restoring the data from a backup server or by using storage facilities with cross-data replication. Unfortunately,these functions are only supported on the top brands of storage hardware, where cost and complexity are much higher than for middle-class storage hardware.

This paper describes a virtualisation cluster, based on two middle-class arrays (without the support of crossreplication) and several hypervisors.

1.2 Selection of the virtualisation platform

There are many virtualisation solutions. Each of them has its advantages/disadvantages and developers always claim that their product is the best. Selection of the best may not be completely obvious. Among other requirements, a virtualisation system must permit the installation of any operating system, live migration of virtual systems between hypervisors, and movement of live virtual systems from one storage subsystem to another (live migration of a Virtual Machine from one storage location to another without downtime). Another important feature is the ability to ensure uninterrupted operation of the virtual systems when the hypervisor fails. The following table compares the characteristics of the best-known virtualisation solutions.

* Storage vMotion is enabled in VMware vSphere Enterprise
** OpenVZ needs a modified kernel for VM
*** Hyper-V online VM storage motion is possible, but with downtime during motion (partially solved in Windows Server 2012)

VMware vSphere is definitely not the most powerful solution, but its flexibility, ease of implementation, and system-support from hardware vendors sets the standard for virtualisation. It provides many features that competitors do not yet offer, such as Distributed Resource Scheduler (DRS), Distributed Power Management (DPM), High Availability (HA), Fault Tolerance (FT), and Network I/O Control. Thus, it is well ahead of its competitors, who are still several steps behind this solution. This dominance entails a disadvantage in terms of price, which is several times higher than the price of other virtualisation platforms.

2 Hardware Selection

Most virtualisation clusters focus on maximum performance, memory size and IOPS. Virtual servers only use the resources related to the services running on them. The performance of most services depends on the speed of CPU (processing video data, simulation, etc.). Other services require maximum IOPS and extra memory (large database systems, web servers). There are also services that never use more than a fraction of CPU and the IOPS-value is not relevant for them because data from storage are rarely loaded. These are also the services necessary for the operation of large computer networks on the campus: DHCP, DNS, VPN, email, radius, and various monitoring tools or web services. The following section explains the selection of hardware suitable for virtualisation of services of this kind.

2.1 Optimal hypervisor hardware

As described in previous sections, the most important hardware parts of the virtualisation cluster are the hypervisors and storage devices. Campus network infrastructure requires their interconnection into one robust VMware vSphere cluster. The selection of hardware must be adapted to the characteristics of the virtualised services, and also to VMware licensing policy. It is necessary to license each physical processor unit and every few gigabytes of memory.

Details about VMware vSphere licensing are available on the website [1]. The best server is equipped with one powerful CPU and up to 64GB memory. This configuration uses all of the resources of one licence of VMware vSphere ESXi 5 Enterprise.

Today’s processors are about ten times more powerful than the CPU’s of five years ago. This allows a processor to replace several older servers. The best price-performance ratio is offered by the Intel Xeon 5600 or the Xeon E5-2600 family of processors.

A VMware vSphere hypervisor requires about 1GB of disk space for installation. Essentially, this storage is only used at system boot. The optimal solution for hypervisor storage is an enterprise flash memory with a capacity of at least 2GB. Other storage systems are not necessary because the data of the virtual servers are stored on a shared iSCSI storage device. Because there is no need to install additional hard drives, it is possible to fit the server hardware into a small server chassis with a height of 1U (Standard Rack Unit).

The parameters described above correspond to many of the servers from different vendors, and campus networks use servers from HP, IBM, Supermicro, Dell, and others. The best operating characteristics are found in Dell servers, which are better than their competitors in design, operating characteristics, warranty, and low failure rate. Their price for academic institutions is very advantageous and for larger purchases, and discounts of 50-60 percent from the list prices can be obtained. The best offer found was the Dell R610 server in the configuration:

• PowerEdge R610;
• Intel Xeon X5690 Processor;
• 48GB Memory for 1CPU;
• Internal SD Module with 2GB SD Card;
• High Output Redundant Power Supply (2 PSU) 717W;
• Intel X520-T2 10GbE Dual Port Server Adapter, Cu, PCIe;
• iDRAC6 Enterprise.

This is a relatively cheap and powerful solution. This server, with a single license of VMware vSphere 5.0 Enterprise with one-year subscription, cost less than €6000 at the end of 2011. It is possible to equip the server with 48GB of DDR3 memory for one CPU socket (6x8GB), and memory size is more important than CPU performance. Therefore, in the case of limited resources, it is better to use a cheaper CPU with more memory. A server, with less memory than 36 gigabytes will, most likely, be rare in the future. If a failure of one of the hypervisors occurs, it becomes necessary to fit all running virtual systems to the memory of the remaining hypervisors. Otherwise, it is impossible to guarantee their uninterrupted operation. The same restriction applies, not only during failure of a hypervisor, but also for its upgrade. For this reason, it is advantageous for the virtualisation cluster to have at least three or four hypervisors.

VMware vCenter Server is the simplest, most efficient way to manage VMware vSphere. It provides unified management of all of the hosts and VMs in the datacentre from a single console and also provides aggregate performance monitoring. A VMware vCenter Server gives administrators deep insight into the status and configuration of clusters, hosts, VMs, storage, the guest OS, and other critical components of a virtual infrastructure – all from one location [2].

VMware vCenter is a software application designed for Windows. Versions for Linux already exist, but do not yet have all the features of the original Windows version. This application is very important, because without it, it is not possible to migrate both of the virtual servers and their virtual disks. The license for this product is determined by the number of hypervisors in the cluster. If the virtualisation cluster includes three hypervisors and this number is not going to increase in the future, it can be very advantageous to choose a VMware vCenter Academic licence, which provides the full version of central management for up to three hypervisors. In other cases, it is necessary to use the VMware vCenter Standard, which can handle an unlimited number of hypervisors. A single license for a VMware vCenter 5.0 Standard with a one-year subscription cost less than €4000 at the end of 2011.

2.2 Fibre vs. iSCSI storage

The heart of each storage unit is a host bus adapter (HBA) that manages the physical disk drives and presents them as logical units. Connection to the network is also provided by the host bus adapter. Powerful storage devices have additional HBA for load-balancing between both controllers, and these can secure the full functionality of the storage system during a failure of one of them. Selection of the HBA depends on the choice of suitable technology. The dominant technologies in the enterprise storage field are Fibre Channel and iSCSI. Both technologies are supported by Vmware vSphere and offer adequate performance characteristics. The cost of HBAs for both technologies is comparable. The advantage of Fibre Channel technology is its throughput and overall performance. The disadvantage is that it is more expensive and requires a complex network infrastructure. ISCSI is cheaper because it can run on almost any switches.

Most critical services in the network do not require storage systems with extra performance. Therefore, it is not absolutely necessary to deploy Fibre Channel, although it is better in many aspects. ISCSI was chosen primarily because of its lower cost, operational characteristics, and the support from VMware. In addition to the type of HBA chosen, it is also necessary to specify other operating parameters, such as device dimensions, type, and number of power supplies, speed of iSCSI connectivity, form factors, and the capacity of the hard drive and software features.

2.3 Storage parameters

Storage can be connected to the SAN using either 1Gb or 1OGb iSCSI HBA. The difference in price between these HBAs is minimal in comparison with the price of the whole storage system. The advantage of 10Gb is in the acceleration of iSCSI traffic, and thereby, in higher read/write performance of the virtual servers. Sometimes, it may be useful to reduce the interface speed to 1Gb. This allows the connection of the storage system to the existing 1Gb SAN infrastructure, and SAN infrastructure devices can still use the same HBA after an upgrade to 10Gb.

This allows the connection of the storage to the existing 1GbE SAN infrastructure and also makes it possible to continue to use it with higher performance, following an upgrade to 10GbE SAN.

The disk array must be maximally resilient to hardware failures. For this reason, the array must be equipped, not only with two HBA, but also with several power sources. The size of the storage quipment is mostly limited by the height that the storage device would take up in the rack, which should not exceed 2U (Standard Rack Unit).

Twelve classical 3.5″ discs or twenty-four 2.5″ discs can be accommodated in the space of 2U. 3.5″ drives offer the greatest output and capacity. The total capacity of twelve of these disks is 48TB for SATA disks, or roughly 10TB for SAS disks. The total capacity for twenty-four 2.5″ SAS disks is 21.6 TB. These disks have a lower tendency to heat, making them more reliable. A greater number of disks provide a higher IOPS and more flexibility with the RAID array [3]. Modern enterprise SSD disks are also 2.5″ in size. The following table compares different disk configurations that can be fitted into the 2U disk array.

* values are actual at the end of 2011; [4]

Therefore, it is best to fill the disk array with a higher number of 2.5″ SAS disks, mostly because they are more reliable and have a higher capacity. In some cases, it is even better to use several 2.5″ SSDs. After considering all the above requirements, the Dell PowerVault MD3620i disk array was chosen as the best, with the following configuration:

• PowerVault MD 3620i;
• 2x HBA 10Gb iSCSI (2x 10GbE port and 1x 1GbE Management port per HBA);
• 24x 600GB 10K RPM SAS 6Gbps 2.5″ Hotplug Hard Drive;
• 2x Redundant Power Supply (2 PSU) 717W (600W peak output);

3 Network Infrastructure

The first step in setting up a virtualisation cluster is to prepare the network infrastructure. This infrastructure provides two primary functions: connection of the hypervisors to the backbone network and to the disk storage.

3.1 Storage interconnection

Switches that connect hypervisors with the disk storage are referred to as SAN infrastructure. A SAN infrastructure could, theoretically, be shared with server access network. This option was tested over a period of several months. This testing determined that it is not the correct choice.

Some network technologies that are used in backbone networks to ensure uninterrupted operation can result in small interruptions in seconds (STP, OSPF, etc.) and cause their convergence time. Another source of potential problems is short-term network peaks, when a link is saturated with a lot of traffic for several seconds (possibly due to DOS attacks or broadcast storms). These problems are characterised by similar behaviour of the virtual servers. A short-term interruption or significant slowing of the SAN network can lead to serious problems with the virtual server. Problems were found on all virtual operating systems, but the most occurred on Linux servers. Their IO system is set up to avoid problems when accessing the hard drive. If the Linux kernel does not get a response from the hard drive in the predefined interval, it connects to the affected part as read-only, making the virtual server entirely dysfunctional. Other systems of connecting to the disk are not read-only, although these too were affected by interruptions. All processes requiring a disk operation at the time had to wait several seconds for the disk operation to be completed. As a result of these problems, it was necessary to set up separate network infrastructures between remote locations to allow direct connection to the SAN switches in both locations.

HP switches are the most used switches on the VUT campus in Brno, which is why the proven HP 2910-24al switch was selected for the SAN infrastructure. This is a standard L2 switch with a management system that allows the connection of four optical transceivers. This switch has sufficient throughput and supports JUMBO packets. Theoretically, the use of JUMBO packets is highly suitable, although the increase in throughput realised is only a few percentage points. A more complex configuration, without diagnostic tools, represents a disadvantage. This is why JUMBO packets are not permitted in the final SAN infrastructure. The following illustrations describe the SAN connection in detail. Two independent L2 segments, ensuring iSCSI connectivity, are highlighted in blue and red. Each of these segments uses a different range of IP addresses: 10.255.3.0/24 and 10.255.4.0/24. All of the hypervisors and the HBA are part of both subnets. This in the only way that resilience against an interruption in one branch can be guaranteed.

3.2 Backbone interconnection

In addition to the SAN infrastructure, the connectivity of hypervisors is also necessary. Each of these must be connected to two backed-up access devices. Through the backbone network, these are then connected to another pair of access devices in the remote location. Hypervisors in all locations must have access to the same VLAN, in such a way so that only one virtual server with a fixed IP address is required to operate any of the hypervisors. This functionality can be achieved using the RSTP protocol for backing up two remote locations, and the VRRP protocol is used to backup the default gateways in each network. The following illustration depicts the resulting topology. The SAN infrastructure is depicted in green. The connections between the active components within the distribution layer are depicted in blue. The backbone links are drawn in red; these provide a high-speed connection to the remote locations. The illustration also includes two separate optical cables, which are crucial to the entire cluster’s full redundancy.

3.3 Network configuration

The previous chapter described important aspects for the proper functioning of a virtual cluster. This primarily requires access to the same VLAN for all hypervisors in all locations, as well as back up of the default gates in each subnet in order to keep it accessible even during outages or router upgrades. This functionality is primarily achieved by the backbone network with the STP and VRRP protocols. The configuration of backbone components is examined in detail in [5] and [6]. The VLAN configuration is most important when configuring access points. Every hypervisor must have access to all networks. The names of network interfaces must be same on every hypervizor. This is the only way that problem-free migration of the virtual servers can be guaranteed between the individual hypervisors in the various locations.

The crucial configuration of active components is shown in the following examples. The purpose of the configuration referred to in these examples is to set up three VLAN for the switches. Two of them are set aside to operate the virtual servers (10.0.1.0/24, 10.0.2.0/24) and one is for management (10.255.1.0/24). Besides these three networks, the backbone network must also include two networks reserved for iSCSI operation (10.255.3.0/24, 10.255.4.0/24).

3.3.1 An example of the configuration of a SAN device

Switches in a SAN infrastructure do not require a complicated configuration. Basic commands suffice for their installation. To confirm the functioning of the SFP modules and the stability of the links, it is easiest to use the command show interface brief or show interface 24, where the number 24 represents the number of the SFP port of the optical module. To confirm the basic connection, the usual ping command suffices.

Each component’s configuration requires the setting up of IP addresses in the management network and several other commands.

hostname "virtual-kou-sw1"
no cdp run
no web-management
vlan 506
	name "mgmt"
	ip address 10.255.1.9 netmask 255.255.255.0
	untagged 1
	exit
snmp-server community "public" operator
snmp-server contact "hostmaster@example.com" location "kou, sal"

Properly set values should be applied to the switches. Otherwise, the logs will not make sense. It is a good idea to store the logs remotely, because the reboot of any equipment usually erases the log file. The ip authorizedmanagers command offers, at least, basic security.

timesync sntp
time timezone 60
time daylight-time-rule Western-Europe
sntp unicast
sntp server priority 1 10.255.1.1
logging 10.255.1.1
logging facility syslog

ip authorized-managers 10.255.1.0 255.255.255.0 access manager
crypto key generate ssh rsa
ip ssh

The previous configuration is usually the same on all equipment, while the most important configuration for SAN switches is the configuration of the VLAN for iSCSI operation and their IP addresses.

vlan 546
	name "vmware-iscsi"
	untagged 2-24
	ip address 10.255.3.109 255.255.255.0
	exit

3.3.2 Example of access-device configuration

The basic configuration has already been described in the previous chapter. Besides these basics, a redundant connection to the backbone network and VLAN configuration for connected equipment is important for component access. Configuration of VLAN management and spanning tree protocol (STP) is a basic component of backbone connectivity.

vlan 506
	name "mgmt"
	tagged 1-4
	ip address 10.255.1.8 netmask 255.255.255.0
	exit
spanning-tree force-version rstp-operation
spanning-tree

Use the show spanning-tree command to verify STP functionality. This provides an abundance of useful information. Usually it suffices to verify the state of both uplinks. One should be set to the “Forwarding state”, while the other to “Blocking”. Furthermore, the “Time Since Last Change” value should be the same for all other components in the same STP domain. Once the STP functions properly, it is possible to set up the user VLANs.

 vlan 3
	name "ant-servers"
	tagged 1-4
	exit
vlan 654
	name "kou-servers"
	tagged 1-4
	exit

3.3.3 Example of backbone-device configuration

The configuration of backbone components is sufficiently dealt with in the GN3 documentation [6]. The most important part of the configuration is to set up the STP, GVRP, VRRP, and OSPF protocols.

 vlan 506
	name "mgmt"
	tagged 1-4
	ip address 10.255.1.3 netmask 255.255.255.0
	exit
spanning-tree force-version rstp-operation
spanning-tree
gvrp

To implement the OSPF protocol, it is first necessary to create a point-to-point connection between neighbouring routers and to activate the OSPF to these interfaces. The show ip ospf neighbour and show ip route commands are the most important for determining the protocol states.

 ip routing
router ospf
area 0.0.0.2
	redistribute connected
	exit
vlan 240
	name "ext240"
	ip address 147.229.240.2 255.255.255.252
	ip ospf 147.229.240.2 area 0.0.0.2
	tagged B21
	exit
vlan 241
	name "ext241"
	ip address 147.229.241.2 255.255.255.252
	ip ospf 147.229.241.2 area 0.0.0.2
	tagged B22
	exit

The last part of the configuration concerns the VRRP protocol. Its configuration must be made on both backbone routers (which perform the backups) at the same time. The configuration should be made for all subnets, whose default gateway should be backed up. The configuration of the primary router can be as follows.

 vlan 3
	name "ant-servers"
	ip address 147.229.3.1 255.255.255.128
	tagged 1-4
	vrrp vrid 1
		owner
		virtual-ip-address 147.229.3.1 255.255.255.128
		enable
		exit
	exit
vlan 654
	name "kou-servers"
	ip address 147.229.3.254 255.255.255.128
	tagged 1-4
	vrrp vrid 2
		backup
		virtual-ip-address 147.229.3.130 255.255.255.128
		enable
		exit
	exit

The configuration of the other router can be as follows.

 vlan 3
	name "ant-servers"
	ip address 147.229.3.126 255.255.255.128
	tagged 1-4
	vrrp vrid 1
		backup
		virtual-ip-address 147.229.3.1 255.255.255.128
		enable
		exit
	exit
vlan 654
	name "kou-servers"
	ip address 147.229.3.130 255.255.255.128
	tagged 1-4
	vrrp vrid 2
		owner
		virtual-ip-address 147.229.3.130 255.255.255.128
		enable
		exit
exit

4 Storage Installation

Preparation of the disk array is another step in the installation process. When mounting into the rack, it is best to install such equipment close to the ground, mostly for stability reasons, because the disk array full of hard drives is usually rather heavy. The temperature surrounding the drives is also important. In some cases, the temperature between the upper and lower sections of the rack can differ by tens of degrees Celsius, depending on the performance of the server room’s air conditioning system. After installation into the rack, the disk array management ports can be connected to the same subnet, and both ports should be connected to different switches, for backup purposes. Within the same subnet, a station running on a Windows operating system with the Powervault Modular Disk Storage Manager must be installed in advance. The most up-to-date version of this software is available from the developer [7].

4.1 Connection to the device

The first step in installing the disk array is to connect it to the management software. For arrays with default settings, it is best to use Automatic Discovery. If the management ports are already configured (if they are already assigned to an IP address), it is faster to connect to this array using the assigned IP address.

After connecting to the disk array, the basic status is displayed. Modification of the management port IP addresses is accomplished via Setup > Configure Ethernet Management Ports. Configuration of the iSCSI ports is closely related to the SAN infrastructure topology (refer to Chapter 3.1) and is accomplished via Setup > Configure iSCSI Host Ports.

4.2 Disk group configuration

The disk array must first be partitioned into disk groups. The size (capacity) of individual disk groups is determined by the number of assigned hard disks, the RAID level, and the set cash of the SSD unit. The total capacity of individual disk groups is then further partitioned into virtual disks.

4.3 Virtual disk mappings

These virtual disks are assigned to individual iSCSI clients within the “Mappings” tab. The virtual disk can be shared among clients, provided the sharing is supported by the file operating system. The individual clients are identified, not only by their IP addresses, but are also identified by an “iSCSI Initiator String”. For iSCSI clients of the VMware vSphere hypervisor, the “iSCSI Initiator String” is generated once the iSCSI protocol is activated. This is further explained in Chapter 5.5. The mapping of the new hypervisor to the virtual disk or a group of virtual disks is depicted in the following illustration. To add a new host, you will need to know its identifier, as described above. If the host had already attempted to connect to the iSCSI array, its hostname and iSCSI Initiator String have already been stored in the unestablished connection table, where the correct host can be chosen with a single click, without the need to manually input the host’s identifier.

5 Hypervisor Installation

A hypervisor’s installation is not significantly different from the installation of other operating systems. When booting from the installation disk, it is sufficient to run the Esxi-5.0.0 Installer from the menu. It is necessary to choose the location where the VMware sSphere system should be installed. This location can be on a hard drive, a RAID disk array, an SSD disk, or a flash drive. In Chapter 2.1, the Dell R610 server was chosen, configured for SD memory, and is the server on which the hypervisor is installed. The last step before initiating the installation is to configure the root password. Once it has been rebooted, the system is now set up.

5.1 Connection to the hypervisor

In the default state, the hypervisor is assigned a dynamic IP address from the DHCP server. For problemfree work with the hypervisor, it is better to set a fixed IP address. This is possible from the system’s console. Press F2 and enter your login details to show the basic configuration panel.

The purpose of each item should be evident. A fixed IP address can be set under “Configure Management Network”. In addition to the IP address, this item menu can also be used to set up the network interface, VLANid, or the DNS parameters. The Management Network should be restarted and tested according to the following steps. A specialised client should be used for complete configuration.

5.2 VMware vSphere client

This is a separate application used to control and configure the VMware vSphere system. The easiest means of obtaining a client is through the hypervisor’s web interface.

5.3 Hypervisor configuration

Configuration of the hypervisor, using the VmWare vSphere Client, requires a preset IP address on the network interface, and login details. All these configuration details are inputted locally on the server (refer to Chapter 5.1).

When logging into the hypervisor, it is advisable to configure the NTP server, the DNS parameters, and the SSH server.

• Time Configuration - Properties - Options - NTP Settings
• DNS and Routing - Properties - DNS Configuration - Look for hosts in the following domains
• DNS and Routing - Properties - Routing - Default gateway
• Security Profile - Services Properties - Options - SSH - Startup Policy - Start and Stop with host
• Security Profile - Services Properties - Options - SSH - Services commands - Start

5.3.1 Network interface configuration

In VmWare, there are two types of network interface: VMKernel and Virtual Machine. VMKernel is a network interface for connecting the following ESXi services: vMotion, iSCSI, NFS, and Host Management. The Virtual Machine interface establishes a connection between the virtual server and the computer network. These interfaces can be configured, either through a VMware vSphere Client or from the command line. The easiest way to configure the network interface is with the VMware vSphere Client. The advantage of this type of configuration is its simplicity; even a beginner can set up the required network interface relatively comfortably. The disadvantage of this method of configuration is the time required. Also, with a larger number of network interfaces, it is more difficult to achieve an identical configuration for all hypervisors (which is required for the proper migration of virtual servers between hypervisors). When configuring multiple hypervisors, it is better to configure the network interfaces using the console. This allows for the configuration of all hypervisor network interfaces using a sequence of commands, which may repeated for all hypervisors that have the same hardware configuration. This method enables an identical configuration of hypervisor network interfaces within a cluster. The following command will show the initial state of network interfaces once the installation of the hypervisor is complete.

 ~ # esxcfg-vswitch -l
Switch Name	Num Ports	Used Ports 	Configured Ports 	MTU 	Uplinks
vSwitch0 	128 		3 		128 			1500 	vmnic0

	PortGroup Name 		VLAN ID 	Used Ports 	Uplinks
	VM Network 		0 		0 		vmnic0
	Management Network 	0 		1 		vmnic0

To understand the virtual network interface in VMware, it is important to understand the following hierarchal concepts: vswitch, vmknic, vmnic, and port group. On the lowest level, physical network interfaces (vmnic) are assigned to individual vswitch objects. These physical network interfaces are used for communication between the hypervisor and the virtual servers with connected systems. More PNIC interfaces in a single vSwitch reveal a higher degree of redundancy (standby or link-aggregation). The lack of a VMNIC interface means that the given vSwitch only handles communication between the virtual servers. Each vSwitch is similar to an L2 switch that supports VLAN. In VMware terminology, “port group” is used instead of VLAN. These port groups are also used to connect virtual servers and for the services of the hypervisor system.

5.3.2 vSwitch configuration

Before configuring vSwitches, you must decide which physical interface should be used to create the vSwitch objects, and which services the vSwitch should stop. In most cases of a VMware cluster with iSCSI storage/array, the correct partition for four physical network cards (vmnic0-4) is as follows:

• vSwitch0 – vmnic0, vmnic1 – back up connection of virtual servers, host management and vMotion;
• vSwitch1 – vmnic2 – iSCSI operation;
• vSwitch2 – vmnic3 – iSCSI operation.

The first step is to configure port groups for host management. At start, a “Management Network” port group is created on the vSwitch0 with vmnic0 uplink. The following commands add a second physical network interface to the virtual switch, set them as standby backup interface, and create port groups for connection to the virtual servers. With the esxcfg-vswitch command, the numbers following the -v parameter represent the VLAN ID value, with which the given port group’s packets are distributed across the vmnic0 and vmnic1 physical interfaces to the backbone network.

esxcfg-vswitch -L vmnic1 vSwitch0
esxcli network vswitch standard policy failover set -s vmnic1 -v vSwitch0

esxcfg-vswitch -A "vpn-server" vSwitch0
esxcfg-vswitch -A "mgmt-ro" vSwitch0
esxcfg-vswitch -A "vlan3" vSwitch0
esxcfg-vswitch -A "vlan3-kou" vSwitch0

esxcfg-vswitch -v 660 -p "vpn-server" vSwitch0
esxcfg-vswitch -v 506 -p "mgmt-ro" vSwitch0
esxcfg-vswitch -v 3 -p "vlan3" vSwitch0
esxcfg-vswitch -v 654 -p "vlan3-kou" vSwitch0

The following steps describe how to create other vSwitch objects and their configuration as iSCSI ports.

 esxcfg-vswitch -a vSwitch1
esxcfg-vswitch -a vSwitch2
esxcfg-vswitch -L vmnic2 vSwitch1
esxcfg-vswitch -L vmnic3 vSwitch2
esxcfg-vswitch -A "iSCSI1" vSwitch1
esxcfg-vswitch -A "iSCSI2" vSwitch2

Each hypervisor has two separate IP address in the subnets, and each address establishes connectivity with the disk array independently.

hypervisor1:

 esxcfg-vmknic -a -i 10.255.3.10 -n 255.255.255.0 iSCSI1
esxcfg-vmknic -a -i 10.255.4.10 -n 255.255.255.0 iSCSI2

hypervisor2:

 esxcfg-vmknic -a -i 10.255.3.11 -n 255.255.255.0 iSCSI1
esxcfg-vmknic -a -i 10.255.4.12 -n 255.255.255.0 iSCSI2

5.3.3 iSCSI configuration

A graphics client is suitable for hypervisor iSCSI configuration.

• Configuration - Storage adapters - Add - Add software iSCSI adapter
  A new software iSCSI adapter will be added to the Storage Adapter list. After it has been added, select the software iSCSI adapter in the list and click on Properties to complete the configuration.
• OK
• iSCSI Software Adapter - vmhba<number> - Properties - Dynamic Discovery / Static Discovery
  Add IP addresses of iSCSI target. These addresses match topology of SAN infrastructure (Chapter 3.1).

  10.255.3.1
  10.255.4.1
  10.255.3.2
  10.255.4.2

• Next
  A rescan of the host bus adapter is recommended for this configuration change. Rescan the adapter?
• Yes
  This step secures a hypervisor-connection attempt to storage, using its IP addresses and iSCSI name. The iSCSI session must be permited on storage. Here, it is important to check that the storage knows the iSCSI name of hypervisor. The next steps are realised in Powervault Modular Disk Storage Manager and build on information obtained in Chapter 4.3.
• Open Powervault Modular Disk Storage Manager
• Mappings - Storage(in left window) - View - Unassociated Host Port Identifiers
• List of unassociated host port identifiers.
• Mappings - Storage - Host Group - Define Host - <Host name> - Add by selecting a know unassociated host port identifier <choose right one> - User Laber <write some good string> - Add <check that Host port Identifier and User Label match hypervisors values> - Next
• Host Type - VMWARE (or linux) - Next - Finish
• Close Powervault Modular Disk Storage Manager
• Go back to VMware vSphere Client
• Configuration - Storage adapters - Rescan all
  Now it is possible to see active devices and paths
  

5.3.3.1 Partitions

The following lines can be skipped if partitions have been created and formatted on the disk array. In other cases, partitions must be created and formatted. Follow these next steps for each newly added partition, according to need.

• Configuration - Storage - Add Storage - Disk/LUN
  
• A partition will be created and used - Next
• <Enter the partition name> - Next
• <Choose optimal Block Size> - Maximum available space - Next
  1024GB, 4MB Block Size should be good for majority
• Finish

5.4 vMotion

The VMware Cluster base is now functional. It has access to the iSCSI array. In order to migrate virtual servers between hypervisors, all hypervisors must be administered centrally by the VMware vCenter Server application. It is also necessary to define the hypervisor’s VMKernel network interface (refer to Chapter 5.3.1), across which the vMotion transmissions will be made.

• Configuration - Networking - vSwitch0 - properties
• <choose right VMKernel or define a new one>
• Port Properties - vMotion (checkbox on)

6 vCenter Server

VMware vCenter Server is a software application for the centralised management of a virtual infrastructure. VCenter vSphere 5 comes in either Windows or Linux versions. However, the Linux version is somewhat behind, since it is not possible to run the Update Manager or some of the plugins. For these reasons, it is better to use the full functionality of the Windows version. This version operates on the MSSQL database. MS SQL 2005 is included in vCenter Server’s installation. This server is fully functional, but without additional software, it is not possible to backup the database, or to perform more advanced management of the database. However, in most cases, it is sufficient. If you require some of the more advanced functions, simply install MS SQL Server Management Studio. Alternatively, you can migrate to MS SQL 2008, which may be used free-of-charge after fulfilling the licensing conditions.

6.1 vCenter installation

Some conditions must be met before installing the vCenter Server. Above all, you will need a 64bit version of Windows installed on a suitable server. The server may be physical or virtual, but should not be located on any of the hypervisors in the created virtualisation cluster. One of the disadvantages to this would be evident with any hypervisor outage on which vCenter Server is running. If this were the case, then the central management would stop running, as would the arbitrator, which would normally be the server that would determine which virtual server had been affected by the hypervisor outage, and which server should run instead on another hypervisor. From this perspective, it is truly better to run vCenter Server on a separate server, ideally as a virtual server in a standalone installation of VMware vSphere. The advantage of this approach, as opposed to a hardware server, is the VCenter Server’s ability to take snapshots, its easy re-installation, and its ability to better manage the resources of the physical servers.

The actual installation of the vCenter Server is trivial and does not require any special effort. The user name used during the installation is the same as the one that will be used to run the server application. However, later users and groups will be added, and their access to virtualisation clusters will be administered. After the server is installed, it is a good idea to also install VMware Update Manager. This is a software application that manages updates of hypervisors and virtual servers. After installing these core applications, you should restart the server and verify, through the services administrator, whether or not the corresponding services have automatically rebooted.

Sometimes, the order in which the services start may cause conflicts, so if the vCenter does not start up properly, both services should have the following Startup Type value: “Automatic (Delayed Start)” for the following services:

• VMware VirtualCenter Management Webservices;
• VMware VirtualCenter Server.

The VMware vCenter should now be fully functional. For login, the VMware vSphere Client uses the same username and password as the system.

6.2 vCenter configuration

In its default state, vCenter does not yet include any objects that could be administered. Such objects must first be created. The basic object types are as follows.

vCenter

Datacenter

Cluster

Host

Virtual Machine

The virtual server (VM) runs on the hypervisor (Host). Hosts are assigned, either to a cluster or directly into the Datacenter. The Datacenter is the basic hierarchical block, which groups clusters with individual hosts. The root of the hierarchy tree is an instance of the VMware vCenter.

The first step in configuring the vCenter is to create a Datacenter object. The importance of this block is that it separates the hardware resources provided by individual hosts and the disk array into functional blocks. Another step is to create a cluster. The last configuration step is to assign the hypervisors to a cluster.

• <Focus on vCenter instance and show context menu> - New Datacenter - Name - Finish
• <Focus on Datacenter instance and show context menu> - New Cluster
  • Name - Turn ON vSphere HA - Next
  • Host Monitoring Status: Enable Host Monitoring
  • Admission Control: Enable
  • Admission Control Policy: Host failures the cluster tolerates: 1
  • Next
  • Cluster Default Settings
  • VM restart priority: Medium
  • Host Isolation response: Leave powered on
  • Next
  • VM Monitoring: Disabled
  • Monitorign sensitivity: High
  • Next
  • <Choose right type of CPU>
  • Enable EVC for Intel Hosts: Intel Sandy Bridge Generation
  • Next
  • Store the swapfile in the same directory as the Virtual Machine
  • Next - Finish
• <Focus on Cluster instance and show context menu> - Add Host
  • Connection: <Enter IP or HOSTNAME of hypervisor>
  • Authorization: <Enter right credentials>
  • Next - Finish

These steps create a tree structure, such as the one that follows.

6.3 Virtual machines

When creating virtual servers, an item from the “Cluster” or “Host” context menu is usually used.

• <Focus on Cluster or Host instance and show context menu> - New Virtual Machine
  • Configuration: Typical - Next
  • Name - Next
  • Choose a specific host within a cluster
  • Select a destination storage - <iSCSI shared storage must be selected to provide redundancy>
  • Next
  • Guest Operating System - Next
  • Number of NICs
  • Network: <Choose network name - VLAN ID>
  • Type of adapter: <Intel E1000 is widely supported network adapter>
  • Next
  • Virtual disk size:
    • 64 GB is good minimal value for WINDOWS 2008 R2/WINDOWS 7 and newer
    • 8 GB is good minimal value for UNIX/LINUX without X system mounted on / Additional virtual disk for special server functionality is necessary.
e. g., 64 GB virtual disk mounted on /var/www for webserver
e. g., 64 GB virtual disk mounted on /var/db for databases
This schema with every partition on extra virtual disk is very useful for resizing. Instead of resizing a virtual disk and partition and file system is possible to add a new bigger virtual disk to the system a copy all data to a new one. This way save a lot of time and is much saver for your data.
  • Thin Provisioning - <little bit slower, but save a lot of disk space, highly recommended>
  • Next - Finish

A virtual server may be installed by running an ISO file from a mapped CV/DVD device, or by PXE. This document does not cover the installation of a virtual operating system.

6.4 Plugins

The following plugins offer advanced functionality of a vSphere Client connected to a vCenter Server. The server part of a plugin is usually installed in the vCenter Server. The vSphere Client displays which plugins are available in the plugin menu

6.4.1 Update Manager

After installing the VMware Update Manager, the Update Manager plugin becomes available.

• Plugins - Manage Plugins - VMware vSphere Update Manager ExtensionDownload and Install in Status column will begin the installation.
• Run - Next - Accept - Next - Install - Finish
• Install this certificate - IgnoreVMware vSphere Update Manager Extension is enabled

The vSphere Client interface now includes an Update Manager menu and an interface for Update Manager Administration. Below is a description of how to upgrade a hypervisor. The first step is to define which patches will be applied to a hypervisor.

• Home - Update Manager - Download patches and upgrades
• Go to Compliance View
• Attach
• Patch Baselines
• Critical Host Patches
• Non-Critical Host Patches
• Attach

There are crucial steps to consider whenever patching a hypervisor.

• Home - Update Manager - Download patches and upgrades
• Go to Compliance View
• Scan

Compliant Host is green. Non-Compilant is red.

• <Focus on Non-Compilant Host>
• <Migrate all VM to another Host>
• <Enter the Maintenance Mode>
• Remediate - Next - Next - Next - Next - FinishInstallation and restart or host will take about 5 minutes.
• <Exit the Maintenance Mode>

6.5 vMotion and Storage vMotion

A final step should include verifying the functional migration of virtual servers between hypervisors.

• <Focus on online VM to migrate> - Migrate
• Change Host - Next
• <Choose different Host> - Next
• Finish

The migration of a virtual server’s data storage may be verified by the following steps.

• <Focus on online VM to migrate> - Migrate
• Change Datastore - Next
• <Choose different Datastore> - Next
• Finish

Both of the above tasks should be possible without a VM outage. The time required to make changes to a hypervisor depend on the operating memory of the virtual server. The time required to make changes to a data storage depends on the size of a virtual server’s hard disk. A data storage change usually takes some time and may take several dozen hours with large data servers.

7 Conclusion

The purpose of this document is to describe all aspects of operating a virtualisation cluster in order to use it as a manual to design other virtualisation clusters. At the moment, VMware vSphere is the best-tuned platform for virtualisation. This best practice document focuses on this most widely-used tool and on the hardware required for its operation. When choosing hardware and software tools, emphasis is placed on an optimal price/performance ratio so that the chosen software would best serve the conditions of the target environment, and not require the purchase of unnecessary virtualisation software licenses. The final result is proposal for the optimal hardware for this type of virtualisation cluster, as described in the second chapter.

As the following table shows, the migration of the original thirty physical servers to a newly-created virtualisation cluster lowers the power required to operate these systems by 77%.

After virtualisation cluster installation and migration of all physical systems into a virtual environment, a basic measurement of consumption yielded the following results.

The performance of the redundant virtualisation cluster significantly surpassed all expectation and clearly demonstrates the advantages of virtualisation, especially for older server systems. In addition to the energy savings, there was a savings of about a 60% in the required rack space. However, the greatest advantage is the resilience of virtual systems against hardware failures and the ability to migrate the virtual systems to other locations. Migration to other locations is important during power or temperature-control failures or due to natural disasters affecting the data centre, because it would be able to transfer the virtual servers with their storage to an unaffected location.

The system described is the result of many years of experience with VMware virtualisation at the Brno VUT campus. The maintenance of such a system is easier than dozens of individual servers and the operation of a virtualisation cluster has avoided problems associated with incompatible hardware servers.

8 References

[1] VMware vSphere 5, Licensing, Pricing and Packaging http://www.vmware.com/files/pdf/vsphere_pricing.pdf

[2] VMware vSphere Documentation http://www.vmware.com/support/pubs/vsphere-esxi-vcenter-server-pubs.html

[3] Tom’s Hardware, 3.5″ Vs. 2.5″ SAS HDDs: In Storage, Size Matters Patrick Schmid, Achim Roos, May 2010 http://www.tomshardware.com/reviews/enterprise-storage-sas-hdd,2612.html

[4] Dell Enterprise HDD Specification, August 2011 http://www.dell.com/downloads/global/products/pvaul/en/enterprise-hdd-sdd-specification.pdf

[5] Configuration of HP Procurve Devices in a Campus Environment, Tomas Podermanski, Vladimir Zahorik, March 2010 (CBPD111, the Czech Republic) http://www.terena.org/activities/campus-bp/pdf/gn3-na3-t4-cbpd111.pdf

[6] Recommended Resilient Campus Network Design, Tomas Podermanski, Vladimir Zahorik, March 2010 (CBPD114, the Czech Republic) http://www.terena.org/activities/campus-bp/pdf/gn3-na3-t4-cbpd114.pdf

[7] Drivers for PowerVault MD3620i, August 2011 http://ftp.euro.dell.com/Pages/Drivers/powervault-md3620i.html

9 List of acronyms

DHCP

Dynamic Host Configuration Protocol

DNS

Domain Name System

DOS

Denial-of-service Attack (DoS Attack)

GVRP

GARP VLAN Registration Protocol

GARP

Generic Attribute Registration Protocol

IOPS

Input/Output Operations Per Second

IP

Internet Protocol

iSCSI

Internet Small Computer System Interface

L2

Layer 2 – Data link layer of OSI model

L3

Layer 3 – Network layer of OSI model

OSPF

Open Shortest Path First

PSU

Power Supply Unit

RSTP

Rapid Spanning Tree Protocol

SFP

Small Form-factor Pluggable Transceiver

SM

fiber Single-mode Optical Fiber

STP

Spanning Tree Protocol

VLAN

Virtual Local Area Network

VPN

Virtual Private Network

VRRP

Virtual Router Redundancy Protocol

Full paper

Presentation used at the conference

Citation: Polčák, L., Hranický, R., Matoušek, P.: Hiding TCP Traffic: Threats and Counter-measures, In: Security and Protection of Information 2013, Brno, CZ, UNOB, 2013, s. 83-96, ISBN 978-80-7231-922-0

Monitoring network

We tested the operating systems on virtual machines in VMware Workstation 7.1., the network cards of the virtual machines were operating in Host-Only mode. You can see the topology on the following image: Additionally, Cisco 3725 router was connected to the topology:

Performed tests

Selection of a new address

1. The router is connected to the network and the tested interface is down on the tested OS
2. The tested interface is enabled and consequently, the OS generates a new address

Does the tested computer reply to DAD-NS (EUI-64, static, random, PE address)?

1. The router, the tested computer and a virtual PC with Ubuntu are present in the tested network.
2. The tested address (EUI-64, static, random, PE) is set on the Ubuntu computer, consequently Ubuntu issues DAD and the tested OS should reply with the NA.

Does the tested computer use duplicate EUI-64 address?

1. The router, the tested computer and a virtual PC with Ubuntu are present in the tested network. The tested interface is down.
2. The tested system is configured to generate EUI-64 if necessary.
3. Ubuntu is set up to use the same MAC address as the tested computer has.
4. The tested interface is enabled, the OS generates the same address as the Ubuntu computer already has. Ubuntu replies with NA.

Does the tested computer use duplicate global EUI-64 address?

1. Only the systems with EUI-64 addresses enabled by default were tested
2. The router, the tested computer and a virtual PC with Ubuntu are present in the tested network. The tested interface is down.
3. Ubuntu is set up to use the same IPv6 address as the tested computer would use as EUI-64 address.
4. The tested interface is enabled, the OS generates the same address as the Ubuntu computer already has. Ubuntu replies with NA.

Does the tested computer use duplicate random address?

1. Only the systems with random addresses enabled by default were tested (newer Windows).
2. The router, the tested computer and a virtual PC with Ubuntu are present in the tested network. The tested interface is down.
3. Ubuntu is set up to use the same IPv6 address as the tested computer would use as random address (the IID is generated once and then it stays constant).
4. The tested interface is enabled, the OS generates the same address as the Ubuntu computer already has. Ubuntu replies with NA.

Does the tested computer use duplicate PE address?

1. PE addresses are enabled on the tested OS.
2. The router, the tested computer and a virtual PC with Ubuntu are present in the tested network. The tested interface is down.
3. A script that replies with NA to all PE addresses is started on the computer with Ubuntu. The script does not reply to EUI-64 (link-local, global) a random addresses used by the tested Windows computers.
4. The tested interface is enabled, the OS generates PE address and the script replies with NA.

Captured PCAP files

During the test, we captured various PCAP files that can be downloaded used for your analysis. We have also analysed the PCAP files, see the following section and the publications below.

• POLČÁK Libor, HOLKOVIČ Martin a MATOUŠEK Petr. A New Approach for Detection of Host Identity in IPv6 Networks. In: Proceedings of the 4th International Conference on Data Communication Networking, 10th International Conference on e-Business and 4th International Conference on Optical Communication Systems. Reykjavík: SciTePress – Science and Technology Publications, 2013, pp. 57-63. ISBN 978-989-8565-72-3.
• POLČÁK Libor, HOLKOVIČ Martin and MATOUŠEK Petr. Host Identity Detection in IPv6 Networks. In: E-Business and Telecommunications. Berlin: Springer Verlag, 2014, pp. 74-89. ISBN 978-3-662-44787-1. ISSN 1865-0929. (This is an extended version of the previous paper.)
• POLČÁK, Libor. Lawful Interception: Identity Detection. Brno, 2017. PhD. Thesis. Brno University of Technology, Faculty of Information Technology. 2017-10-13. Supervisor Švéda Miroslav.

Test results

The tests result are presented in the following table

Notes to the tests

• 1.Random addresses:
  • They are only supported on Windows Vista and newer.
• 2. Privacy extension addresses:
  • PE was not configured
• 3. EUI-64 global addresses – reaction to duplicate address:
  • Tested OS keeps the address but it is marked as duplicate and it is not used by the OS (e.g. DAD, ping)
• 4. Privacy extension addresses – reaction to duplicate address:
  • Only one PE is tried, it is kept after DAD but it is marked as duplicate and it is not used by the OS (e.g. DAD, ping)
• 5. EUI-64 – reaction to DAD:
  • Marked systems ignore DAD from the same MAC address as is used by the system.
• 6. Solaris – static address:
  • Static address was not tested because it was not used by the system.

IPv6 Neighbor Discovery, Router Advertisement, Rogue RA attack, RA Guard.

1. Introduction to NDP

The Neighbor Discovery Protocol (NDP) specified by RFC 4861 can be thought of as an IPv6 analogy to the set of IPv4 protocols, namely Address Resolution Protocol (ARP), ICMP Redirect and ICMP Router Discovery. To be more precise, it provides even more services, but as far as the Rogue RA Attack is concerned, the most important are Router Discovery and SLAAC. The main purpose of these services is to provide hosts with a unique IPv6 address and a default gateway. After a host connects to the network, the process is as follows:

1. A host creates an Interface ID, usually via EUI-64 procedure or in case of Privacy Extensions it uses a (pseudo)random ID.
2. The host creates a link-local IPv6 address by appending an Interface ID to the FE80::/10 prefix.
3. The host discovers routers by sending Router Solicitation (ICMPv6 message type 133).
4. IF there are any routers on the link, they announce themselves by replying with Router Advertisement (ICMPv6 message type 134), including:
   - Advertised prefix
   - Router Lifetime – a positive value indicates the remaining time a router is willing to behave as a default gateway
   - Additional parameters such as MTU, etc.
5. The host receives a Router Advertisement, creates an IPv6 Globally Unique Address by appending its Interface ID to the advertised prefix.
6. IF the Router Lifetime value is non-zero, it inserts the IPv6 address of a router to its Default Router List.

The following figure illustrates the whole process.

2. Rogue RA Attack

The main issue of the NDP protocol is that its messages are unsecured. Nodes receiving unsecured NDP messages cannot distinguish between valid and “bogus” NDP messages, so there are only two possible ways of treating them. Either to ignore (which is not acceptable) or to process all of them. This is a huge security issue, because an attacker can pretend to be a valid router and provide hosts with malicious information in order to disrupt their connectivity (Denial of Service) or worse, capture their data (Man-in-the-Middle). The outline of a typical MitM Rogue RA Attack can be as follows.

1. An attacker captures a valid Router Advertisement.
2. An attacker changes a Router Lifetime value to zero and sends such crafted message in order to force the victim to remove the current valid router from its Default Router List.
3. An attacker then creates another Router Advertisement, but this time announcing himself as a valid router, which effectively redirects the traffic through an attacker’s station. In fact, the traffic flowing back to the victim will be forwarded directly to the victim. An attacker can, however, perform a Neighbor Cache Poisoning Attack to change this, but this is beyond the scope of this article.

As mentioned above, currently there are few mitigation techniques available against this type of attack, but there is probably only one of them that is usable — RA Snooping.

3. RA Guard

Most of the networks have a common attribute that their nodes are on a link layer bonded by some intermediary device (usually switch), rather than directly. Such scenarios provide an opportunity to make use of a mechanism called RA Snooping. The idea of so-called snooping has been present in computer networks for long time, in the form of, for instance, IGMP Snooping or DHCP Snooping. It is well-known that switches use only information provided by link-layer header of received frames to forward them through the right interface. The concept of snooping exceeds this behavior in that the switch inspects also upper-layer information, based on which it takes beforehand defined actions. These actions have either optimizing (IGMP) or security (DHCP, RA) character.

RA Guard is a prevention mechanism against Rogue RA attack that utilizes RA Snooping. The key condition that must be met prior to RA Guard deployment is that there must be some intermediary device in a network that all traffic passes through. Typical example of such device is an Ethernet switch. RA Guard is implemented precisely on this device. The core functionality of RA Guard is that it inspects Router Advertisements and based on the information provided it decides whether to drop or forward them. Note that RA Guard is not any particular protocol or strictly defined mechanism. It is just common term describing a set of recommendations and general description of prevention mechanisms that were implemented by various vendors and appeared under various names. For example, while Cisco retains the name RA Guard, H3C uses the different label — ND Detection.

3.1 Configuring RA Guard on Cisco device

This section briefly describes, the configuration of RA Guard on Cisco devices. More details can be found in official Cisco documentation [4]

Switch(config)# ipv6 nd raguard policy POLICY-NAME   ! Defines the RA Guard policy name
Switch(config-ra-guard)# device-role {host | router} ! Specifies the role of the device attached to the port.

The RA Guard policy can have several other optional options checking the hop limit, router preference etc. After the policy is configured, it has to be applied to appropriate interfaces.

Switch(config)# interface INTERFACE
Switch(config-if)# ipv6 nd raguard attach-policy POLICY-NAME ! Applies the RA Guard policy

If a policy is configured with device-role host, a switch will drop any RA messages received on an interface where the policy is applied, thus eliminating the RA attack. Interfaces towards to a router should be configured with a policy where device-role is set to router.

3.2 Configuring RA Guard on H3C device

H3C devices incorporate RA Guard into more complex ND attack defense tool called ND Detection. Similarly to Cisco RA Guard implementation, also ND Detection function differentiates between trusted and untrusted ports. RA messages are not checked for correctness at trusted ports and are always forwarded.

[Switch] ipv6                                         # Enable IPv6, as it is disabled by default.
[Switch] vlan 1
[Switch-vlan1] ipv6 nd detection enable               # Enable ND Detection functionality on selected VLAN.
[Switch-vlan1] quit
[Switch] interface GigabitEthernet 1/0/1
[Switch-GigabitEthernet1/0/1] ipv6 nd detection trust # Set interface connected to router as trusted.

4. Bypassing RA Guard

Against the simplest form of this attack (i.e. IPv6 header and Rogue ICMPv6 message are the only content of an IPv6 packet) are both RA Guard and ND Detection effective. As it turns out, if an attacker modifies the Router Advertisement message in an appropriate way, RA Guard may not recognize such messages as bogus. For that purpose, an attacker can make use of the concept of IPv6 Extension Headers and/or IPv6 packet fragmentation.

4.1 RA Guard Bypass Using Extension Headers

Similarly to IPv4, also IPv6 provides the option to extend the protocol. IPv4 has the Options field in IPv4 packet header, which can contain an information of arbitrary length (with respect to the maximum length of an IPv4 packet and IHL maximum value) as specified by RFC 791. Some intermediary nodes, processing IPv4 packets need an access to upper-layer information (e.g. firewalls), but as the Options field is variable in size, the first octet of an upper-layer header must be calculated dynamically according to Internet Header Length (IHL) value of an IPv4 packet.

Also the specification of IPv6 (RFC 2460) provides a room for protocol extensions, but the realization is slightly different as with its predecessor. The size of an IPv6 header is now fixed, in order to speed up the forwarding process. The Options field is not present anymore, but its role was substituted by the Next Header field. Apart from the fixed-size main header of an IPv6 packet, there may be a number of supplementary Extension Headers. These Extension Headers are concatenated in a row, each containing the Next Header field in addition to data contained. Each Extension Header is identified by its identifier, which is contained in the Next Header field of a previous header. The semantics of the last Extension Header is the same as the Protocol field of an IPv4 packet (i.e. it identifies the payload of an IPv6 packet).

In most cases, ICMPv6 messages are transmitted without any Extension Header. Some of the implementations of RA Guard may reckon on this, so they inspect only the Next Header field of an IPv6 header. If it does not contain a value of 58, the packet is considered as non-ICMPv6, thus RA Guard does not prevent it from forwarding. This is, however, false premise, as an attacker can easily craft packets with additional Extension Header preceding the ICMPv6 message itself.

To build upon this idea, an attacker can concatenate more than one extension header in order to bypass RA Guard. In order to improve network performance, NDP messages inspection is usually implemented in hardware, which has limited resources. If an attacker creates a packet with absurdly long header chain and succeeds in exhausting all available resources of a switch, the device fails to identify spurious RA message and the defense fails. This may not be true for all devices, but there is a high probability that it may pose a significant challenge for low-end devices.

Further testing revealed that even RA Guard implementation of tested devices fails to inspect the whole chain of Extension Headers and already seven empty Destination Options Extension Headers are sufficient enough to bypass the RA Guard on tested Cisco devices. H3C device has even lower limit — three Extension Headers (see figure below). Tests have shown that if an attacker includes more than 16 extension headers (All of the conducted tests used empty Destination Options Extension Headers), the Cisco RA Guard Implementation is suddenly able to properly recognize bogus Router Advertisements. Based on the information provided by Cisco, hardware forwarding path is preffered to software forwarding. This means that if an IPv6 traffic filtering is in use, also Extension Headers must be inspected in hardware. In case of the header chain too large to fit into the resources allocated in hardware, the packet is punted to the software path. Based on this information, it is reasonable to assume that if a packet has an Extension Header chain short enough to fit into available resources, but consisting of sufficient amount of headers, so that they would not be inspected entirely, such packet would most probably not be dropped (Since this inspection is implemented in hardware, the number of headers that could actually be inspected must be fixed). This is probably only a flaw in implementation, but the fact remains that these devices are vulnerable from this kind of attack.

4.2 RA Guard Bypass Using Packet Fragmentation

The purpose of IP fragmentation is to transfer packets larger than the MTU of used link-layer technology. This is achieved by splitting the original packet into multiple fragments, each of the size less than given MTU and transferring each part separately, so that they can be reassembled by their receiver. As with IPv6, the idea remains the same as in IPv4, but there are some differencies. In IPv4, any device is allowed to fragment the transiting packet in case, when the MTU of a link that a packet is supposed to be sent to is smaller than the size of a packet itself. On the contrary, fragmentation of IPv6 datagrams is allowed only at the sender’s node. In case, when a datagram cannot be forwarded by some intermediary device because of too small MTU of the particular link, the device sends an ICMPv6 Packet Too Big message including the MTU that caused the failure to the sender. This behavior is a part of the Path MTU Discovery defined by RFC 1981, which is used to determine the minimum MTU size of all the links spanning from the sender to the receiver, so that IPv6 datagrams would not need to be fragmented. What is more, IPv6 specification requires that every link in the Internet have an MTU of 1280 octets or greater. Another difference from IPv4 is that in IPv4, all information needed for IP fragments assembly at receiving node are present directly in IPv4 header. On the other side, IPv6 excluded this information from an IPv6 header and introduced separate Extension Header for that purpose, because its aim is to avoid packet fragmentation at all.

The concept of IPv6 packet fragmentation can be exploited to bypass RA Guard. The idea is as follows. An attacker inserts additional Extension Header (e.g. Destination Options), splits the Rogue RA message into more fragments, and sends them separately. Therefore, also a device with RA Guard configured will treat these fragments separately. Note, that most of the IP networks ensure high availability by means of path redundacy and therefore, if there are multiple paths available between the sender and the receiver, each fragment can take another path. Following figure illustrates the original Rogue RA message, prior to fragmentation, and the two resulting fragments which are subsequently sent as a part of the attack.

In this case, a device processing the second fragment is unable to locate the ICMPv6 header contained in this fragment. The reason for this is following. To locate the first byte of an ICMPv6 header a device needs to know the length of a previous header, which is, however, present in the first fragment. Thus, by leveraging the use of a Fragment Header together with another Extension Header (in our case Destination Options), the attacker is able to conceal the content of an ICMPv6 message including its type. The transit device with RA Guard configured has only two options. Either it blocks all such fragmented messages including those that are valid, or it forwards all of them, including the malicious ones. The reasonable choice is the second one, as it is unacceptable to drop any valid traffic.

If an attacker uses the approach mentioned above, a snooping device can at least detect that the original packet was an ICMPv6 message, since the Next Header field of the last Extension Header of the first fragment is set to 58. This idea can be pushed further, so that it is impossible for the intermediary device to even detect that it is an ICMPv6 message, actually. This can be achieved with an ICMPv6 message preceded with two or more Extension Headers that are splitted into fragments as illustrated in the following figure.

The challenge that the second fragment presents is the same as in the previous case, but this time, a device processing the first fragment is not even able to detect that ICMPv6 message is being transmitted. This is because the Next Header field of the last Extension Header of the first fragment has the value of 60, which indicates the Destination Options as the following Extension Header.

All of the tested devices are vulnerable from Rogue RA attack with packet fragmentation, as expected. Current specification of RA Guard actually does not take into account issues associated with packet fragmentation and therefore, practically all of the current implementations of RA Guard are vulnerable from this form of an attack.

There is, however, some work in progress [1] that recommends to drop all IPv6 packets that are first fragments (i.e., Fragment Offset is set to 0) and fail to include the entire header chain. Non-first fragments may be safely forwarded, because their destination would not be able to properly assemble all of the fragments, as the first one would be missing. At the time of this writing, such behavior of RA Guard has not been implemented yet and it is questionable, if it ever will be. There is, however, a possibility to achieve such behavior on Cisco devices by making use of an ACL deny ip any any undetermined-transport, but unfortunately, none of the tested devices supports such ACL. The main obstacle preventing this solution from being deployed is that it may drop unusual but valid packets.

Even if such restriction would be acceptable, there is still a chance to bypass RA Guard [3]. An attacker can create two overlapping fragments in a way so that the first fragment would contain some regular message (e.g. ICMPv6 Echo Request), so that it will not be dropped by RA Guard and the second fragment would contain malicious RA message with an offset being set in such a way that when these fragments will be assembled at their destination, RA message overwrites the message included in the first fragment. Following figure illustrates the situation.

Since different operating systems handled assembly of fragments in different ways and overlapping fragments represented critical security issue, RFC 5722 forbade their usage. According to research conducted by SI6 Networks [2], all of the current operating systems are converging towards the implementation of RFC 5722, so overlapping fragments should not pose a threat in most cases.

5. Conclusion

The presented attack was tested on following devices with RA Guard (resp. ND Detection) enabled:

• Cisco Catalyst 3750-X Series Switch with IOS version 15.0(2)SE1
• Cisco Catalyst 2960-S Series Switch with IOS version 15.0(2)SE2
• H3C A5800 with OS version 5.20

Presentation: Download presentation (ppt)

URL: https://events.nordu.net/display/ndn2012web/Security+challenges+in+IPv6+from+the+campus+perspective

Videos used in presentation:

• IPv6 RA flood DoS attack in Windows 8
• The Man-in-the-middle Attack Using IPv6

IPv6, security, campus network, transition

1. Introduction

The idea of IPv6 deployment proposed by RFC 5211 [1] expected that the Internet would be fully migrated to IPv6 in these days. Unfortunately, it seems that all deployment strategies defined in either political [2],[3] or technical documents were too optimistic. IANA IPv4 address pool is already depleted, more than 10% autonomous systems (AS) announces IPv6 prefix in global BGP table [4] and most of operating systems support IPv6. However, the content providers still indicate that less than 0.5% [5] of clients is being able to use IPv6 connectivity. What is worse, there are statistics showing that IPv6 connectivity in some networks might be broken or less stable comparing to IPv4 [6]. As the result, most of content providers are afraid to announce IPv6 for their services in fear of losing customers.

Observing the global IPv6 infrastructure and deployment, we believe there are no significant problems to have a good IPv6 connectivity for servers in data centers. Many transport networks can deliver IPv6 traffic today as well. We believe that the key problem of the low IPv6 penetration is on the side of ISPs providing last mile to customers. ISP must ensure a positive user experience, thus if the IPv6 is deployed, it is important that the deployment is secure and a service quality is the same as in IPv4 network. Unfortunately, there are not many devices able to implement IPv6 first hop security that is equivalent to IPv4 security. Ordinary user does not care about protocol used for connection, but does care if the connection is broken or unstable, which is sometimes a problem with IPv6 connection.

This article comprises experience with deploying IPv6 at the campus network at the Brno University of Technology (BUT) which is one of the biggest universities in the Czech Republic. Currently, the core of the network has fully enabled support for the dual-stack and the IPv6 network completely follows topology of IPv4 network. University also has own provider independent (PI) IPv6 address space to be able to use multihomed IPv6 connections in future. The university campus network connects more than 2,500 staff users and more than 23,000 students. The top utilization is present at student dormitories where more than 6,000 students are connected via 100 Mb/s and l Gb/s links.

In the following sections we will describe the most significant issues that we have been faced with during the deployment of IPv6 protocol in the campus network. The deployment started at the university in 2002. At the beginning, the experimental network on dedicated devices and links was created. Currently, the IPv6 and IPv4 share the same infrastructure that is operated as a dualstack network. Most of the network services support both protocols. However, the process of transition to IPv6 at the university has not been finished yet, and it would take a lot of time and effort to move all services to IPv6 with the equivalent stability and reliability as we have in IPv4.

2. Addressing issues

One of the main issues is address assignment for clients. The mixture of various OSs in a network requires automatic address assignment that is supported by most of the systems. Assigning addresses with a DHCP server became de-facto standard for IPv4. However, DHCPv6 protocol is different.

DHCPv6 features two basic modes. In practice, the first mode, stateless DHCPv6, is a layer on top of the autoconfiguration mechanism (SLAAC) and is used to provide recursive DNS server addresses. Two special flags are used for this purpose in the Router advertisement (RA) message: M – managed, O – other. These flags tell the client that it should ask a DHCPv6 server for more information related to the connection parameters. If the M flag is set, statefull DHCPv6 is used. If the O flag is set, SLAAC will be combined with stateless DHCPv6. The strong binding between SLAAC and DHCPv6 brings several problems.

• It is not possible to pass all necessary configuration options (e.g. option for default route) via DHCPv6 server. Authors of DHCPv6 protocol stated that because SLAAC has to be used anyway, default route is not necessary – client learns a default route from RA message. However, this forces to use both autoconfiguration mechanisms together and increases the complexity.
• When a client sees an RA with M flag on, a client sends a DHCPv6 Solicit message looking for a DHCPv6 server. A DHCPv6 server responds with appropriate configuration for the client. However, if the client has a DHCPv6 derived address, and receives an RA with M flag off, the client will release that DHCPv6 derived address. Unfortunately, RA messages can be easily spoofed so the attacker can force all clients in a local network to release their IPv6 addresses just with one packet. This can be solved by proper filtering on access layer; however this is sometimes a problem. The filtering possibilities are discussed later in the paper.

Using DHCPv6, we do not get the same results as with DHCPv4 server (MAC to IPv6 address binding).
DHCPv6 does not use a MAC address to identify the client; instead, it uses a specially created unique identifier called a DUID (DHCP Unique Identifier). The main idea behind this identifier is to release the clients from dependence on hardware and on a specific network interface. The advantage is that a change of a network adapter or a connection through another interface (such as WiFi instead of Ethernet) would mean that the user always obtain same IPv6 address. Unfortunately, there are several issues connected with the DUID identifier.

• DUID is controlled by software, thus it is not as stable as it should. E.g., if the client has dual boot, every OS will have different DUID.
• DUID is changed, after OS is reinstalled.
• If the administrator clones an OS image and copy it to another computer, two computers will have the same DUID.
• It is impossible to tie DUID with the host identification that is used in DHCP(v4) – host’s MAC address which complicates assigning address especially in a dualstack environment. Solution is either to extend existing systems for address management to support DUIDs or to use workarounds like MAC address option specified in RFC 6221. Unfortunately, vendors have not included the support for the RFC 6221 yet.

Moreover, statefull autoconfiguration using DHCPv6 is very difficult to be used today because of lack of support on many platforms including Windows XP, which is still very widespread OS. Most of mobile devices has not implemented support for DHCPv6 yet and some OSs (e.g. MAC OSX) must be updated to the latest version, which is a problem if the devices are not managed by an ISP.

The operational experience shows that according to these issues, the DHCPv6 protocol and its implementations are still not mature enough to be used in a production network and moreover, it is not feasible to use it for address assignment in networks where the hosts’ identification is required.

Another choice for address assignment available in IPv6 is to use the stateless utoconfiguration (SLAAC). Unfortunately, the stateless autoconfiguration in some OSs turns on privacy extensions. This means that devices generate a random end user identifier (EUI) – temporary IPv6 Address. This is a brand new IPv6 feature that allows a node to automatically generate a random IPv6 address on its own without the control of a network administrator.

However, this contradicts the need to identify a malevolent user. Private, temporary addresses hinder the unique identification of users/hosts connecting to a service. This prevents logging and tracking users based on IPv6 address. However, the knowledge of relation between an address and a device (or user) that has been used is necessary for solving security incidents and is required by law in several countries.
The Table 1 summarizes the autoconfiguration techniques in IPv6 protocol.

If a security policy requires better control, either fixed IPv6 addresses must be centrally assigned and logged, which is not a feasible option for a large network, or statefull configuration using DHCPv6 has to be deployed. Before the deployment of IPv6 in a local network, a network administrator must decide whether:

• Addresses will be assigned by DHCPv6 – the advantage is better control over hosts with, but many devices will not be able to use IPv6.
• Hosts will create own address based on SLAAC – all hosts will be able to use IPv6, but an administrator either gives up to have control over user identification in the network or a new mechanism must be deployed to identify the users [21]. Example of our implementation is described later on.

The Table 2 summarizes the IPv6 autoconfiguration support among the OSs.

First hop security in IPv4

IP address autoconfiguration process might be perceived as a honey pot for a hacker. If the hacker is able to interfere the configuration process, the whole user’s traffic can be rerouted to the attacker’s PC. In many cases it does not need to be a targeted attack, but simply an accident, where a user connects a Wi-Fi router with a preconfigured DHCP server to the network and causes a network malfunction for other users. This problem, as the other problems with first hop security, is known in IPv4 world for quite long time. For this reason some mechanisms were created in the IPv4 world which would prevent or at least complicate some of these attacks. The best place to implement protection for end users is on the end-user switch access port that the user is connected to. Different vendors use slightly different terminology for individual types of protection but generally we can meet the following ones:

DHCP Snooping: Some ports are explicitly defined in the switch configuration so port is able to receive DHCP responses from DHCP (so called trusted port). It is assumed that somewhere behind the trusted port is a DHCP server. If a reply from a DHCP server arrives to a port not defined as trusted, the response is discarded. Any DHCP server running on the client system (whether intentionally or by accident) does not threaten other clients on the network because the answers will not reach further than the access port for which this protection has been activated. DHCP snooping is usually prerequisite for other protection mechanisms such as IP lockdown or ARP protection as described below.

Dynamic ARP protection, ARP inspection: DHCP snooping database contains MAC address – IP address – switch port combination. This database is then used on untrusted ports to inspect ARP packets. Other MAC addresses not recorded in the database are discarded. This eliminates attacks focused on creating fake records in the ARP table (poisoned ARP cache). Another often-appreciated feature of this mechanism is the fact that the client cannot communicate over the network unless an IP address from the DHCP server is obtained. That forces
user to use DHCP instead of configuring static address.

Dynamic IP Lockdown, IP source guard: Another degree of protection is achieved by inspecting source MAC and IPv4 address on untrusted ports for all packets entering the port. This eliminates spoofing a source IPv4 or MAC address.

The Table 3 describes different attacks and techniques available for mitigating the attacks in IPv4 network.

3. First hop security in IPv6

The above described solutions for mitigating attacks in IPv4 networks are implemented in various access switches on the market. As we wrote previously, the autoconfiguration techniques are different in IPv6 so new solutions are necessary. Security mitigation techniques in IPv6 networks are described below.

Source Address Validation Improvements (SAVI): The set of techniques that complement ingress filtering with finer-grained, standardized IP source address validation [11]. Framework has option for DHCP servers and tries to solve a mechanism similar to the one we described with DHCP snooping for IPv4. It is limited to DHCPv4 and DHCPv6 and does not deal with the problems of rogue Router Advertisement messages. SAVI is mainly supported in devices produced by Hewlett Packard – A series. Very similar technique called ND
inspection is implemented in Cisco devices.

Secure Network Discovery (SEND): This method tries to deal with autoconfiguration problem in a totally different way. SEND is based on signing packets with cryptographic methods [15]. Apart from a router it does not require support on the access switches. The validity verification itself through message certificate takes place at the end-user system. IPv6 address of the end-user system is a result of a cryptographic function (see, we have another auto configuration method). Using SEND directly excludes using static, EUI 64 and Privacy Extensions Address. However SEND provides great advantage – it not only solves the autoconfiguration problem but also other safety problems of the Network Discovery protocol (RFC 2461). Another advantage is independent infrastructure; hence it can also be used in the same way in the either wired or Wi-Fi networks.

The main shortcoming of SEND is the requirement for public key infrastructure according to X.509. To make the SEND works properly, a certificate of the certification authority have to be installed on each client that wants to use SEND. The certificate has to be issued for each router as well. So it grows the costs of the management of the network – certificates have to be reissued or replaced before they expire.

SEND is a patented Cisco technology (US patent number 20080307), therefore no one would be surprised that it is implemented especially on some devices of this company. Presence of the patent raised several discussions especially with SEND and RA Guard integration. According to Cisco statement, Cisco will however not assert any patents against any party that implements the standard [17].

Considering the SEND deployment, a network administrator will face the problem that the protocol is not supported on any operation system yet. There are only some basic Linux and Windows implementations; however thay cannot be used in a production environment. Windows OS, as the most widespread system, nor Mac OS do not support the SEND protocol even in the most recent versions. SEND protocol could potentially solve security problems of NDP protocol; however, it cannot be deployed and there are no indications that this should change in the near future.

Router Advertisement Guard (RA Guard): Another alternative, which unfortunately deals only with the issue of fake router advertisements, is IPv6 Router Advertisement Guard [9]. It is similar technique as DHCP snooping, but determined for Router Advertisement packets. It tries to block fake router advertisements on an access user port. Apart from tools that should ease the initial switch configuration (learning mode), it opens the
path to integration with SEND. In this mode the switch works as a so called node-in-the-middle, where the switch with activated RA Guard uses information from SEND to verify packet validity and for the connected end-user system it appears as normal Router Advertisement packet. As you could guess from the title RA Guard does not solve DHCP or DHCPv6 issues in any way. We can already find an implementation in some Cisco devices.

Access Lists on the switch (ACL/PACL): If a network device supports ACL or PACL (Port access list) it is possible to configure an ACL blocking the malevolent traffic. A network administrator can configure ACL that will block all ICMPv6 messages type 134 (RA messages) and also block traffic to the UDP target port 546 (dhcpv6-c1ient). The rules are subsequently applied to the inputs of ports to which the c1ients are connected. This can eliminate instances of rogue routers and DHCPv6 servers. A required condition to use this mechanism is IPv6 ACL support on the relevant switch. The problem of this solution is that very few access switches
supports creation of IPv6 access-lists today. Also price of that switches is usually two or three times higher comparing to the switches where IPv6 PACL and other IPv6 security features are not implemented.

The solution with RA-Guard and ACL/PACL has however big disadvantage. It works very well for accidently announced RA messages but can be easily avoided using combination of fragmented packets and extension header in the RA message [18]. Existing protection for the targeted IPv6 attack does not exist today. There are some proposed solutions how to solve the problem [19],[20], but all of them are only drafts and it will take a long time before support will be added to operating systems and network devices.

The most undesired option is blocking or disabling whole IPv6 traffic. It is obvious that this step protect a host against all attack related to IPv6, however it is against the idea of deploying IPv6. That solution can be used only as a short-term solution when the other possibilities are not available. IPv6 protocol can be suppressed on the client by disabling o uninstalling IPv6 protocol or blocking the whole IPv6 traffic on the access port by filtering out all Ethernet packets identified with Ether Type 0x86DD (Ether type for IPv6 protocol).

The Table 4 summarizes first hop security threads and protection in IPv6 network.

4. First hop security in IPv6 – current situation

Thinking about IPv6 deployment and security, network administrators will face the fact, that all of the above mentioned techniques can be used rather theoretically today. Either they are not implemented on client’s side or device support is missing. Another issue is that if the protective devices are to be truly purposeful they must be placed as close to the end-user system as possible. This could often mean a complete replacement of network infrastructure that is a job that few will want afford just to implement IPv6.

To summarize our experience, after two years of operating the IPv6 network in production environment and several more years of testing the IPv6 protocol, we did not observe a targeted attack to our IPv6 network. The main problems, we have to solve are above described problems with autoconfiguration, bogus router advertisements together with missing monitoring tools for IPv6 network.

Many rogue advertises are generated by Windows computers. This is a serious issue because computers propagate their own interface as a default gateway. Unfortunately this behaviour can be in some conditions caused by properly used Internet connection sharing service. The Figure 1 shows the number of rogue advertises on the network with approximately 2000 of connected hosts.

As we mentioned earlier, because of missing implementation of mitigation techniques on switches, more affordable solution that would at least alleviate efforts to paralyze the IPv6 autoconfiguration mechanism is detection of fake Router Advertisements. This will not protect the network from a well-crafted and targeted attack but it can at least detect incorrectly configured c1ients. For many networks it would be the only usable solutions for a long time. All tools for detection of rogue Router Advertisements work based on the same principle. They connect to the FF02::1 multicast group where the router advertises messages are sent and thus are
able to monitor all RA messages appearing on the network. The monitoring tool can then inform the
administrator about the undesirable status, call an automated action (Ndpmon [22], Ramond [23]), or even send a message cancelling the validity of fake Router Advertisements (rafixd [24]).

5. User tracking, monitoring and accounting

Long-term network monitoring, accounting and backtracking of security incidents is often achieved in IPv4 networks using NetFlow probes and collectors. This can be a problem if IPv6 is deployed and privacy extensions are allowed in the network. The same user can than communicate with different addresses. That means that address cannot be used as a unique identifier anymore. As the part of deploying IPv6 we tried to develop extension to existing monitoring systems to allow easier tracking users in an IPv6 network.

The main idea of the extension is collecting and putting together data obtained from differed parts of the network. A neighbour cache database on routers and forwarding databases on switches can provide information about relation between an IPv6 address port on a switch and a MAC address used by a user. In the next step the MAC address can be used for identifying a user in a database provided by radius server.

These pieces of information, together, provide a complex view of the network and can help to identify a host. A tuple (lPv6 address, MAC address, Login name) is sufficient to identify a host/user. In practice, an extended tuple is built: (Timestamp, IPv6 address, MAC address, Switch port, Login) as depicted in the following Figure 2.

Timestamp is added to provide a history of communication. Switch port is necessary if the user is blocked or if an unregistered MAC address is used on a port. In addition to these values, the VLAN number and interface statistics are stored; however, these data are not necessary for host identification.

It is important to note that this data structure is not created at once, but it is filled in when data are available. For instance, NetFlow data are taken from the NetFlow probe when they are sent to the collector. However, there is no information about MAC addresses yet. The address is downloaded later from the switch’s ND cache. Login data from RADIUS can also be added. However, RADIUS data are not available for every user – only for those who are connected using 802.1x authentication. For other users, only the IPv6 address and the switch port number and MAC address are used for identification.

Data are collected using the SNMP protocol and stored in the central database where the network administrator can search data using the IPv6, IPv4 or MAC addresses as keys. Useful tool for polling and storing information from switches and routers is Network Administration Visualized (NAV) [12]. SNMP polls the data from switches every fifteen minutes. The mapping between the IPv6 address and its corresponding MAC address is downloaded from the router’s neighbour cache. Port, VLAN number and other information comes from the switch’s FDB (Forwarding Database) table. Traffic statistics are obtained from NetFlow. NetFflow records alone
are not sufficient for user surveillance and activity tracking because of the temporary IPv6 addresses as described in previous sections.

The time dependency of gathering different data is crucial when accessing the ND Cache. This temporary memory at the router stores information needed to build the link between the IPv6 address and the MAC address. Because IPv6 addresses change in time and have limited validity, if the ND entry is lost, there is no way to link the IPv6 address and the user/host. To ensure that all information is stored properly in the monitoring system, the SNMP polling interval has to be shorter than the expiration timeout of the ND Cache. Otherwise, some entries in the ND Cache could expire without being downloaded into the central system. Typical timeouts for collecting
SNMP and RADIUS data are fifteen minutes. The ND Cache expiration timeout is usually set to more than one hour.

6. Conclusion

This paper presents security and addressing issues in IPv4 and IPv6 protocol environment and solutions how to solve them. Nowadays, the IPv6 traffic volume is low, but this is caused by the lack of IPv6 sources (web pages, servers) on the Internet. Also widespread operation systems such as Windows XP support IPv6 but IPv6 is not enabled by default.

Next generation Windows systems together with Linux, Mac OS and Unix systems have however IPv6 protocol enabled by default and penetration of these systems is growing every day. Security and addressing issues discussed in this paper present the overview of problems we encountered when we deployed IPv6 protocol in BUT campus network. Addressing issues and problems with user tracking in IPv6 protocol introduce the necessity for a new monitoring system that is able to overcome the specific problems in IPv6 address assignment. Solutions, how to solve these issues, are proposed. We discussed possibilities, how we are able to limit the impact of security problems in IPv6 network together with monitoring and tracking system that is able to identify and track a host in IPv4 and IPv6 network.

References

[1] J. Curran: An Internet Transition Plan, [online], url: http://tools.ietf.org/html/rfc5211

[2] Commission of the European Communities: Action Plan for the deployment of Internet Protocol version
6 (IPv6) in Europe, [online], url: http://ec.europa.eu/information_society/policy/ipv6/action_plan/

[3] Transition Planning for Internet Protocol Version 6 (IPv6), [online],
url: http://georgewbush-whitehouse.archives.gov/omb/memoranda/fy2005/m05-22.pdf

[4] IPv6 CIDR REPORT, [online],
url: http://www.cidr-report.org/v6/as2.0/, http://bgp.potaroo.net/index-bgp.html

[5] Lorenzo Colitti, Steinar H. Gunderson, Erik Kline, Tiziana Refice, Evaluating IPv6 Adoption in the
Internet, [online], url: http://www.google.com/intl/en/ipv6/statistics/

[6] Geoff Huston: Flailing IPv6, [online], url: http://www.potaroo.net/ispcol/2010-12/6to4fail.html

[7] S.Thomson, T.Narten, and T.Jinmei. IPv6 Stateless Address Autoconfiguration. RFC 4862, September
2007, [online], url: http://tools.ietf.org/html/rfc4862

[8] J. Curran: An Internet Transition Pian, RFC 5211, July 2008, url: http://tools.ietf.org/html/rfc5211

[9] E. Levy-Abegnoli, G. Van de Velde, C. Popoviciu, J. Mohacsi: IPv6 Router Advertisement Guard, RFC
6105, February 2011, [online], url: http://tools.ietf.org/html/rfc6105

[10] S. Frankel, R. Graveman, and J. Pearce. Guidelines for the Secure Deployment of IPv6. Technical
Report 800-119, National Institute of Standards and Technology, 2010, [online],
url: http://csrc.nist.gov/publications/nistpubs/800-119/sp800-119.pdf

[11] J. Bi, J. Wu, G. Yao, F. Baker: SAVI Solution for DHCP (work in progress) July 2011, [online], url: http://tools.ietf.org/html/draft-ietf-savidhcp-10

[12] UNINETT and Norwegian University of Science and Technology: NAV, [online], 2011-03-15,
[online], url: http://metanav.uninett.no/

[13] J. Bi, G. Yao, J. Wu, F. Baker.: Savi solution for Stateless Address – work in progress, April 2010,
[online], url: http://tools.ietf.org/html/draft-bi-savi-stateless-00

[14] E. Levy-Abegnoli, G. Van de Velde, C. Popoviciu, J. Mohacsi.: IPv6 Router Advertisement Guard.
RFC 6105, February 2011. [online], url: http://tools.ietf.org/html/rfc6105

[15] J. Arkko, Ed., J. Kempf, B. Zill, P. Nikander: SEcure Neighbor Discovery (SEND). RFC 3971,
Febuary 2011, [online], url: http://tools.ietf.org/html/rfc3971

[16] Wu, J., Bi, J., Bagnulo, M., Baker, F., and C. Vogl: Source Address Validation Improvement
Framework”, draft-ietf-saviframework-04 (work in progress), March 2011.

[17] R. Albright, Cisco System’s Statement of IPR related to draft-ietf-v60ps-ra-guard-02, April 2009,
[online], url: http://www.ietf.org/ietfftpIlPR/cisco-ipr-draft-ietf-v60ps-ra-guard-02.txt

[18] F. Gont, IPv6 Router Advertisement Guard (RA-Guard) Evasion, December 2011, [online], url: http://tools.ietf.org/html/draft-gont-v6ops-ra-guard-evasion-01

[19] F. Gont, Implementation Advice for IPv6 Router Advertisement Guard (RA-Guard), Febuary 2012,
[online], url: http://tools.ietf.org/html/draft-gont-v6ops-ra-guard-implementation-01

[20] F. Gont, Security Implications of the Use of IPv6 Extension Headers with IPv6, January 2012,
[online], url: http://tools.ietf.org/html/draft-gont-6man-nd-extension-headers-02

[21] Grégr, M., Podermański, T., Šoltés, M., Žádník, M.: Design of Data Retention System in IPv6 network,
December 2011, [online],http://www.fit.vutbr.cz/~igregr/pubs.php?id=9840

[22] Frederic Beck, Ndpmon, August 2009, [online], url: http://ndpmon.sourceforge.net/

[23] Ramond, [online], url: http://ramond.sourceforge.net/

[24] Rafixd, [online], url: https://github.com/strattg/rafixd

[25] Tony Cheneau, Ndprotector, March 2012, [online], http://amnesiak.org/NDprotector/

[26] Meinel Ch.: Winsend, March 2012, [online],
url: http://www.hpi.uni-potsdam.de/meinel/forschung/security_engineering/ipv6_security/winsend.html

Traditional firewall can’t detect traffic inside the tunnel if the DPI of every UDP packet is not performed. Unfortunately, firewalls in current network equipment (Cisco, Juniper, HP …) do not support this functionality. To make things worse, these firewalls are often used as border firewalls in enterprise networks. The presentation focuses on our monitoring solution of IPv6 transition techniques. The probe monitors network traffic and generates NetFlow statistics. The type of transition technique is encoded in NetFlow data. We support AYIYA, 6to4, 6in4, Teredo and ISATAP.