The purpose of this document is to capture the most common network questions and answers.
You can edit this document and add your own questions, or you can add a question in the Comments below, or post a question in Re: audio hardware issues in guest o/s.
There are 5 questions so far
- Why is some IP traffic not being forwarded - I'm using IP Hash
- Is there a way to force the choice of a pNIC in a virtual switch?
- Can I mix pNIC types in a vSwitch; what happens if they have different features?
- How can I tell what features my physical NICs (pNICs) have?
- What is the best way to allocate my pNICs between the service console, vmotion, production network, DMZ and possibly storage?
| 1. Why is some IP traffic not being forwarded - I'm using IP Hash |
|---|
Some IP traffic is not being forwarded by the vSwitch. Each pNIC goes to a different physical switch. Load balancing algorithm is "IP Hash".
IP Hash hashes the source and destination IP address of each frame together and takes that modulo the number of pNICs in the team to choose a pNIC. MAC hash takes the LSB of the source MAC in each frame modulo the number of NICs to choose a pNIC.In this case you may not use IP based hash since it will confuse the switches which are unable to do link aggregation across multiple chassis. If they are connected to the same switch then the switch ports must be set to 802.3ad static mode with no LACP or PAGP enabled.
| 2. Is there a way to force the choice of pNIC in a virtual switch? |
|---|
You can create policies wihta single active pNIC and the rest standby.
Only the active will be used until a failure occurs and then we will pull in a standby. Since you can have different teaming policies associated with different portgroups you can configure something like the following:
portgroupA:
- pNIC1 active
- pNIC2 standby
- pNIC3 standby
portgroupB:
- pNIC1 standby
- pNIC2 active
- pNIC3 standby
portgroupC:
- pNIC1 standby
- pNIC2 standby
- pNIC3 active
This will result in Vms attached to portgroupA always using pNIC1 unless it fails, portgroupB always using pNIC2 unless it fails etc. But in the even of failure any of the 3 pNICs can be used as a backup for any of the 3 portgroups.
| 3. Can I mix pNIC types in a vSwitch; what happens if they have different features? |
|---|
Yes you can mix different pNIC types in a vSwitch - the vmkernel handles each pNIC differently in a vSwitch and they don't have to be all the same brand/vendor, or have all the same features (except Jumbo Frames).
What does this mean?
- This means that the same vmkernel driver is not required for all pNICs in a vSwitch.
the vmkernel will use TSO on that pNIC, and perform segmentation in software before forwarding packets to others.
- If one pNIC has a feature that another pNIC doesn't, the vmkernel will emulate that feature on behalf of the pNIC that doesn't have the feature which will increase the CPU workload by some unspecified measure. TSO is an example of one such feature, where the vmkernel will perform segmentation on behalf of a pNIC that doesn't have a TSO feature.
- Jumbo Frames are an exception to this rule - all pNICs must have the Jumbo Frame feature in a vSwitch or it isn't used.
- The fact that one pNIC has a feature that the others in the vSwitch don't, does not mean that the unique feature is switched off; all pNICs are not reduced to the "lowest common denominator" feature list. The vmkernel will in effect "bring up" all pNICs by emulating the features they don't have, like TSO but not Jumbo Frames.
NOTE This last point is new for VI3, and is a change from previous ESX2.5 versions
people/rodos highlighted a quote from the VMware Networking Team blog:
In reality it should not matter what low level capabilities
your NIC has or which ones are enabled and used by the
VMKernel. Most capabilities that are not available on your
particular NIC, the VMKernel will handle in software. As such
you can have a vSwitch configured with two NICs, one capable of
TSO and the other not. VMKernel will use the TSO capabilities
of the NIC that has it, if a failover occurs to the NIC without
TSO capabilities, the VMKernel will just handle segmentation in
software before delivering data to the NIC.
http://blogs.vmware.com/networking/2009/03/deciphering-nic-capabilities-from-log-entries.html
| 4. How can I tell what features my physical NICs (pNICs) have? |
|---|
From Kamau Wangahu and Guy Brunsdon over n the VMware Networking Blog
You may have seen messages like those below in your logs and wondered what neat capabilities are being enabled on your network card (NIC).
ESXsrv01 vmkernel: 22:04:29:43.583 cpu1:1198)Uplink: 2491: Setting capabilities 0x0 for device vmnic4
ESXsrv01 vmkernel: 22:04:29:43.583 cpu1:1198)Uplink: 2491: Setting capabilities 0x0 for device vmnic5
ESXsrv01 vmkernel: 22:04:59:11.410 cpu0:1235)Uplink: 2491: Setting capabilities 0x300 for device vmnic2 ESXsrv01 vmkernel: 22:04:59:11.410 cpu0:1235)Uplink: 2491: Setting capabilities 0x300 for device vmnic3 ESXsrv01 vmkernel: 22:04:59:11.410 cpu0:1235)Uplink: 2491: Setting capabilities 0x2b for device vmnic2
ESXsrv01 vmkernel: 22:04:59:11.410 cpu0:1235)Uplink: 2491: Setting capabilities 0x2b for device vmnic3
What you need is The Magic Decoder Ring!
The Magic Decoder Ring is the table below which states the HEX codes we use to decode the “Setting Capabilities” for the above style log messages are as follows:
0x0001 Capable of scatter-gather transmits
0x0002 can checksum only TCP/UDP over IPv4
0x0008 Can use DMA to high memory locations
0x0020 Supports TCP Segmentation offload, TSO
0x0100 Can do VLAN tagging in HW
0x0200 Can do VLAN untagging in HW
How to map the log messages to the Magic Decoder Ring
So now off to the fun stuff of decoding our log file output….
The first two log entries set the capabilities for vmnic4 and vmnic5 to 0x0. This is logged when the last virtual port of a portgroup that the uplink is assigned to is disconnected. In other words, the hardware capability is no longer needed, as no traffic will be sent over these uplinks (vmnic4 and vmnic5).
The next capability that is set is 0x300 for vmnic2 and vmnic3. From the table above we can get 0x300 by adding 0x0200 and 0x0100. This means that we are enabling:
0x0100 VLAN tagging in hardware.
0x0200 VLAN untagging in hardware
The next capability that is set is 0x2b. From this we can tell we are enabling:
0x0001 scatter-gather transmits
0x0002 checksum only TCP/UDP over IPv4
0x0008 DMA to high memory
0x0020 TCP Segmentation offload, TSO
as
0x0001 + 0x0002 + 0x0008 + 0x0020 + = 0x002b
And that is all there is to translating the NIC “Setting Capabilities” log entries!
| 5. What is the best way to allocate my pNICs between the service console, vmotion, production network, DMZ and possibly storage? |
|---|
Details of many combinations of pNICs and VMware requirements can be found at http://www.networkworld.com/community/blog/7119 (Edward Haletky). The list includes how to best layout the follow:
- DMZ w/5 Physical NICs with VMware ESX
- DMZ w/4 Physical NICs with VMware ESX
- DMZ w/3 Physical NICs with VMware ESX
- DMZ w/2 Physical NICs with VMware ESX
- 8 Physical NICs with VMware ESX
- 6 Physical NICs with VMware ESX
- 5 Physical NICs with VMware ESX
- 4 Physical NICs with VMware ESX
- 3 Physical NICs with VMware ESX
- 2 Physical NICs with VMware ESX