Recently another TAM requested some infos on this topic in VMware’s slack, finally giving me the push to publish as I had this post for ages in my drafts. Given, it is somewhat of an edge case for only a hand full of customers and the problem itself is not new but here is my take. As always please do your own research before you put this untested into production.
Problem statement#
If you have a high number of vCenters there is chance that you might end up with the same MAC addresses across different vCenter instances.
But the TL, DR: Don’t panic if you are not a big enterprise or service provider. And even if you are one, think twice and in terms of effort versus gains first.
Background#
The default behavior for MAC address assignment to virtual machines (or rather network adapters) in vSphere is very well documented and I think everyone has come to know the famous prefix:
VMware Organizationally Unique Identifier (OUI) allocation assigns MAC addresses based on the default VMware OUI 00:50:56 and the vCenter Server ID.
Source: VMware vSphere 6.7 documentation
vCenter Server ID#
Let’s dig into this by starting with the vCenter Server ID. With the initial setup each vCenter gets a randomly generated ID that allows for 64 different values. You can find the vCenter Server ID of your current installation in the general settings of the vCenter
VMware Organizationally Unique Identifier (OUI)#
The OUI covers 22 bits in a MAC address, in general identifying the vendor (it is actually a bit more complicated than that – but let’s leave it here for now). A complete list of all OUI is available from the IEEE (see further down). Taken from the same documentation page as before:
According to the VMware OUI allocation scheme, a MAC address has the format 00:50:56:XX:YY:ZZ where 00:50:56 represents the VMware OUI, XX is calculated as (80 + vCenter Server ID), and YY and ZZ are random two-digit hexadecimal numbers. The addresses created through the VMware OUI allocation are in the range 00:50:56:80:YY:ZZ – 00:50:56:BF:YY:ZZ.
Source: VMware vSphere 6.7 documentation
In my case, all my virtual machines get a MAC from the 00:50:56:AB:YY:ZZ range. Counterintuitively, the specified offset of 80 is a hex number and the shown vCenter ID 43 is a decimal. Converting these number delivers us 80 hex == 123 dec and 43 dec == 2B hex which makes it a total of 123 decimal or AB in hex.
On a side node: About the number of VMs and available MAC addresses#
The second thing people notice in the documentation is the limit of 64000 unique addresses per vCenter. Personally, I don’t think that is not an issue if you look at the scale limits listed in the VMware ConfigMax. As an example the supported numbers for vCenter 6.7:
vCenter Server Scalability Powered-on virtual machines per vCenter Server: 25000 Registered virtual machines per vCenter Server: 35000
Locally administered addresses (LAA) – the alternative to VMware provided MAC addresses#
Before we continue let’s revisit the the MAC address guidelines from IEEE (little did I know about this before this post). For context, an Extended Unique Identifier (EUI) with 48 bits (= 6 bytes) is a the base for an Ethernet MAC:
When an EUI is used as a MAC address (for example, an IEEE 802 network address), the two least significant bits of the initial octet (Octet 0) are used for special purposes. The least significant bit of Octet 0 (the I/G bit) indicates either an individual address (I/G=0) or group address (I/G=1), and the second least significant bit of Octet 0 (the U/L bit) indicates universal (U/L=0) or local (U/L=1) administration of the address. A universally administered address is intended to be a globally unique address.
Source: IEEE – Guidelines for Use of EUI, OUI, and CID
This means we have an officially sanctioned way to provide our own MAC addresses and the IEEE provides a bulky tutorial on how to handle these kind of things. In essence, you must ensure that the second last bit of the first byte is set to “1” – as seen above the official docs call it the second least significant bit (X bit) of Octet 0. In a visual representation:
MAC address:
1st byte | 2nd byte | 3rd byte | 4th byte | 5th byte | 6th byte |
000000*1*0 | ...
^
This bit must be set to 1 if you want to have a valid LAA.
No rule without an exception#
Approximately 18 organizational identifiers assigned to early Ethernet implementers (some of the BlockID assignments made prior to approval of IEEE Std 802.3-1985) have the X bit equal to 1.
Source: IEEE – Guidelines for Use of EUI, OUI, and CID
Make sure you check your selected MAC address range against all officially listed assignments on the IEEE page. A un-official but comprehensive overview is also available from the wireshark team.
Implementation#
With the infos above we can select a suitable, private MAC address range like:
02:xx:xx:xx:xx:xx (the first byte is: 000000 >1 0)
06:xx:xx:xx:xx:xx (the first byte is: 000001 >1 0)
0A:xx:xx:xx:xx:xx (the first byte is: 000010 >1 0)
0E:xx:xx:xx:xx:xx (the first byte is: 000011 >1 0)
Taking the 02:xx:xx:xx:xx:xx-namespace as an example: In theory there are 256^5 addresses available (02:00:00:00:00:00 – 02:FF:FF:FF:FF:FF), compared to the 256^2 addresses normally provided by a vCenter (two remaining bytes). With regard to the vSphere documentation you can select between the prefixed- or range-based allocation to assign the addresses. The major difference is basically how granular you want to specify the range.
For my example I will use the prefixed-based assignment for which two parameters in the vCenter advanced configuration need to be modified:
config.vpxd.macAllocScheme.prefixScheme.prefix
config.vpxd.macAllocScheme.prefixScheme.prefixLength
You can look up the details in the documentation but in short:
The parameter config.vpxd.macAllocScheme.prefixScheme.prefix defines a fixed prefix for an address, much like a IP-prefix (e.g. 10.x.x.x.x) while config.vpxd.macAllocScheme.prefixScheme.prefixLength defines which part of the prefix is variable, much like a subnet mask for an IP (e.g. /16)
Example#
Let’s generate a scheme of
<LAA prefix>:<CLOUD>:<REGION>:<VC number>:xx:xx
which allows a setup with up to 256 different cloud environments with up to 256 regions each and up to 256 vCenter instances in each region. We use these values as an example:
LAA prefix: 02
Cloud = EE
Region = DE
VC Number = 01
The resulting advanced settings for this specific vCenter would be:
config.vpxd.macAllocScheme.prefixScheme.prefix = 02eede01
config.vpxd.macAllocScheme.prefixScheme.prefixLength = 32
This would assign virtual machines MAC addresses from 02:EE:DE:01:00:00 to 02:EE:DE:01:FF:FF. Please note that this will not work virtual machines that have been created beforehand.
Summary and gotchas#
Setting a new MAC address prefix by changing two advanced options is not hard but every design decision has drawbacks. Let’s discuss a few that come to my mind:
It all starts with the address space, since you have the freedom to manage the addresses by yourself – you are also responsible for implementing a scheme where you ensure that no overlapping or conflicting ranges are defined. This all boils down to people, process and technology – or as the IEEE puts it:
Local addresses are not globally unique, and a network administrator is responsible for assuring that any local addresses assigned are unique within the span of use.
But not only the vSphere side needs to be considered in this address scheming, note that nearly all vendors use LAA for their virtual IP implementations. This means a company wide solutions is needed since you don’t want to find out the hard way which physical networking devices like load balancers use that range (as one example).
The second consideration is: Do you need it at all?
Apart from some ill-written software which has license restrictions based on the MAC address of a system (god, I hate these!) your MAC address should only be relevant for layer 2 communications, basically in up to the next router. Do you have more than 64 vCenters on the same campus communicating to the same set of switches before a router so that a duplicate MAC address might matter at all?
Overlay networking may change things a bit as people may think of it as a viable solution to extend layer2 between data centers, but that’s another set of problems.
Comments