MTU size on ERS vs VOSS switches

[XTRMUser]

Hi all,

We have MTU size of 9216 (Jumbo frames max) on our ERS switches. Many of these switches are SMLT'd to VOSS core using Fabric Attach (using fiber connections (10Gb) with transceivers). The VOSS switches all have MTU of 1950. 

This seems to work ok, but we are noticing some odd behaviors, and wondering if the different MTU sizes could be causing some issues. On some links we are seeing a high number of discarded packets, and wondering if this might be a (significant) cause.

We can bump the VOSS switches to 9600. Is there any gotcha's if we do?

Thanks for any advice/hints/ you may have.

[hkhctslue]

Good explanation about MTU behavior between ERS and VOSS. The limits on VOSS can definitely cause issues when mixing hardware — sticking to a common MTU across all devices is usually the safest call. I’ve spent hours fine-tuning similar setups, and honestly, using online fasting tracker https://wellness-app.com/ helped me remember to take breaks instead of staring at configs all day. Keeps your focus sharper for troubleshooting.

[XTRMUser]

Thanks so much FredrikB-NN. This is a great explanation, and thanks for taking the time to do this.

[FredrikB-NN · Accepted solution]

Start by understanding that MTU can have at least two meanings: Ethernet frame size and IP packet size. I like to call them L2 MTU and L3 MTU respectively. In switches that do not do any routing, the L2 MTU is what you need to care about. Jumbo frames are an L2 MTU setting, effectively telling the switch how big Ethernet frames are allowed to be when they are received on a switch port. It is always safe to enable jumbo frames and increase the L2 MTU. This will not affect routing (unless the OS links the L2 and L3 MTU sizes to each other implicitly).

The L3 MTU setting (often configured on a VLAN, not a port) tells the routing function how to handle IP packets. If you have an L3 MTU of 9000 bytes on one VLAN and route big packets that enter this VLAN to a VLAN that has only 1500 bytes MTU, the router either needs to discard this packet or fragment it (split it in several smaller packets).

If your servers and other end devices only have an MTU of 1500 bytes, you shouldn't see any Ethernet frames larger than 1518 bytes (or 1522 with one VLAN tag). A router will never mess with a TCP session and suddenly take two 1500 byte packets and assemble them to a 3000 byte packet. If nobody sends larger IP packets than 1500 bytes, you should be safe. If you need/want to use a higher MTU in your hosts, perhaps because you see vast improvements such as reduced overhead in the servers, all your switches and routers need to be configured to handle that.

An IP packet is almost never transmitted as it is. In our world, we normally encapsulate the IP packets in an Ethernet frame.

So, if you see oversized frames, this means that some host is sending Ethernet frames that are larger than your configured L2 MTU. This in turn means that the host has an L3 (IP ) MTU that is larger than normal. There is not necessarily something wrong with varying MTU settings in your switches. The most important thing is to have a switch MTU that is at least the size of the largest IP packet plus the Ethernet overhead.