How EVPN and VXLAN fabrics actually work

A fabric is two networks in the same hardware. Almost every fabric fault is one of them being reasoned about as though it were the other.

Traditional data centre networks stretched VLANs across the building and relied on spanning tree to stop loops, which meant blocking links you had paid for. A leaf-spine fabric replaces that with routing, and puts the layer 2 back on top as tunnels.

The underlay

A plain routed IP network. Every leaf connects to every spine, nothing else connects to anything, and every path between two leaves is the same length — two hops through a spine.

Its only job is to get each leaf's VTEP address to every other leaf, with equal-cost paths so traffic spreads across the spines. It knows nothing about tenants, VLANs or VNIs. Adding a customer changes nothing in the underlay at all.

The overlay

A mesh of VXLAN tunnels between those VTEPs. A tenant frame is wrapped in UDP and IP, sent across the underlay to the far leaf, and unwrapped. As far as tenant traffic is concerned, every leaf is directly adjacent to every other one.

EVPN is the control plane. It is BGP carrying which MAC addresses and which prefixes live behind which VTEP, so a leaf knows where to send a frame instead of flooding it everywhere and learning from the replies.

VNIs

A VXLAN Network Identifier is 24 bits, which is why VXLAN is usually introduced as "VLANs but sixteen million of them". Two kinds do different jobs:

• An L2VNI carries a bridge domain — one VLAN stretched across the fabric.
• An L3VNI carries routed traffic for one VRF.

They share one number space, so a value cannot be both.

Anycast gateways

Every leaf answers for the same gateway address, with the same MAC. A virtual machine that moves from one rack to another keeps its default gateway and does not notice.

The MAC matters more than the address. If two leaves use the same gateway address but different MACs, a moved host keeps the old MAC in its ARP cache and cannot reach its gateway until the entry ages out — an outage of minutes that resolves itself, which is among the hardest kinds of fault to catch.

The faults worth knowing

One VLAN mapped to two VNIs. Hosts in the same VLAN on different leaves land in different segments and cannot reach each other. Nothing is logged. This is the quietest serious fault a fabric has.

A leaf-to-leaf link. It looks like helpful redundancy and breaks the equal-cost assumption the design rests on, giving some traffic a shorter path than the fabric expects.

An underlay MTU left at 1500. VXLAN adds 50 bytes, so a full-sized tenant packet does not fit. The symptom is small exchanges working and large transfers hanging, which looks like an application problem.

Frequently asked questions

What is the difference between VXLAN and EVPN?

VXLAN is the encapsulation — how a tenant frame is wrapped to cross the underlay. EVPN is the control plane, BGP carrying which MACs and prefixes live behind which VTEP so frames do not have to be flooded.

Why is the underlay MTU a problem?

VXLAN adds 50 bytes, so a 1500-byte underlay cannot carry a 1500-byte tenant packet. The failure looks like an application fault rather than a network one.

What happens if a VLAN maps to different VNIs on different leaves?

Hosts in that VLAN land in different segments and cannot reach each other, with nothing logged. It is the quietest serious fault in a fabric.

Why must the anycast gateway MAC match everywhere?

So a host that moves between racks keeps working. A different MAC leaves the moved host with a stale ARP entry and no gateway until it ages out.

Should leaves ever connect directly to each other?

No. Leaves reach each other through the spines, and every path being the same length is what the design depends on. A direct link gives some traffic a shortcut the fabric did not plan for.