One of the things I have been meaning to try with virtual machines are jumbo frames. It would be interesting to see what they do for performance but my networks at work are not ready for them. They will be eventually, but not at the moment. This post is simply to log information when we are ready to configure the network to use them.

NOTE: The post will be a work in progress. I started it with the intention of setting them up on my network but as with any job, schedules change and this will be done at a later time. For now it’s meant to hold information I have found and offer something to discuss.  I will update it it from time to time.

Why even use Jumbo Frames?

The simplest explanation is it will send less frames across a network which in turn means less CPU cycles. When it’s time to send data; a unit is assembled and it will involve having it’s TCP/IP headers read by the sending and receiving devices. Each device touched along the way will read the frame and will add to the frames and packets to get them to the final receiver. This process consumes cpu cycles and bandwidth.

Using jumbo frames means less frames are sent across the network. A single jumbo frame is 9 kilobytes.  The default frame is 1.5 kilobytes. A jumbo frame will eliminate the need for six default frames. Less frames means less CPU cycles and bandwidth.

To fill a gigabit Ethernet pipe, you would need over 80000 default standard frames per second. With jumbo frames; you only need 14000 or more. This is a significant reduction of CPU cycles and overhead. As such network performance increases.

What is needed for Jumbo Frames

Before you attempt to enable them; you have to verify your network is able to handle them. First and foremost, you have to be running gigabit Ethernet. NICs and switches have to support them as well. You will have to do work or coordinate the effort as you simply can’t enable them on your hosts and VMs while the Network switches are not configured to use them.

Another issue is how the vendor configured them. Are they the same size? Are all the devices in the middle using the same size? As you can read, this is not a task to take lightly.

Things to consider:

  • You need gigabit Ethernet.
  • Measure your devices on the Network as your frames will be sized by the device with the smallest capacity.
  • Fragmentation will only happen at Layer 3.
  • Layer 2 switches will not fragment frames. They either forward or will drop them.
  • Size adjustments will only happen at TCP level.
  • Know your applications. Low latency applications can be hurt by jumbo frames.
  • Know your devices. Jumbo frames will require computing power.

Check for jumbo frame capability

I don’t have access to my network devices. My network people report they are all capable. I would verify this first because if you switches can’t handle jumbo frames, your whole effort ends as it means new equipment and network gear can be rather expensive.

Virtual Machine Host

Your virual machines may be able to run them but the host has to as well.

VMware ESXI:

The host NIC can only be checked by using the SSH connection to the host.

login as: root
root@host's password:

You have activated Tech Support Mode.
The time and date of this activation have been sent to the system logs.

VMware offers supported, powerful system administration tools.  Please
see www.vmware.com/go/sysadmintools for details.

Tech Support Mode may be disabled by an administrative user.
Please consult the ESXi Configuration Guide for additional
important information.

~ # esxcfg-vswitch -l
Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch0         64          9           64                1500    vmnic0,vmnic1

  PortGroup Name        VLAN ID  Used Ports  Uplinks
  VM Network            0        4           vmnic0,vmnic1
  VMkernel              0        1           vmnic0,vmnic1
  Management Network    0        1           vmnic0,vmnic1
~ #

As you can see the MTU is set to the basic size of 1500. To increase the size you can do the following:

esxcfg-vswitch -m 9000 vSwitch0

This will set the MTU to 9000. Verify the change by issuing:

~ # esxcfg-vswitch -l

 

XenServer:

Not had a chance to set this yet.

 

Setting Jumbo Frames on a VM

The virtual machine will need to have jumbo frames set. They have a default of 1500 MTU.

I do have an question of if this should be set to jumbo frames as well. Could it be the host being set will serve the virtual machines best? I do not know at this point as I have not been able to test.

Windows 7

The resolution is to set the default MTU of 1500. You can use the netsh utility to change this. First, lets see what exists by entering:

netsh interface ipv4 show subinterfaces

Of course there is the assumption you have not enabled ipv6. The output will look something like:

C:\>netsh interface ipv4 show subinterfaces

   MTU  MediaSenseState   Bytes In  Bytes Out  Interface
------  ---------------  ---------  ---------  -------------
4294967295                1          0   26497571  Loopback Pseudo-Interface 1
  1500                5          0     854768  Local Area Connection 2

If jumbo frames are set on your switches and host, you can set them on the host (using the example above) by entering:

netsh interface ipv4 set subinterface "Local Area Connection 2" mtu=9000 store=persistent

You can verify the change by using the show subinterfaces command again.

If all is in order, your network connection will still be available.

TEST THE SETUP!

Jumbo frames are not something to enable and “walk away.” You should test your setup and verify they work and they are giving you the desired increase in performance. If they are not working, you may have to disable them and return to the standard MTU of 1500 at the VM level.

Information Links

A nice small explanation of Jumbo Frames

VMWare entry for enabling Jumbo Frames on Host

Advertisements