Start by understanding that MTU can have at least two meanings: Ethernet frame size and IP packet size. I like to call them L2 MTU and L3 MTU respectively. In switches that do not do any routing, the L2 MTU is what you need to care about. Jumbo frames are an L2 MTU setting, effectively telling the switch how big Ethernet frames are allowed to be when they are received on a switch port. It is always safe to enable jumbo frames and increase the L2 MTU. This will not affect routing (unless the OS links the L2 and L3 MTU sizes to each other implicitly).
The L3 MTU setting (often configured on a VLAN, not a port) tells the routing function how to handle IP packets. If you have an L3 MTU of 9000 bytes on one VLAN and route big packets that enter this VLAN to a VLAN that has only 1500 bytes MTU, the router either needs to discard this packet or fragment it (split it in several smaller packets).
If your servers and other end devices only have an MTU of 1500 bytes, you shouldn't see any Ethernet frames larger than 1518 bytes (or 1522 with one VLAN tag). A router will never mess with a TCP session and suddenly take two 1500 byte packets and assemble them to a 3000 byte packet. If nobody sends larger IP packets than 1500 bytes, you should be safe. If you need/want to use a higher MTU in your hosts, perhaps because you see vast improvements such as reduced overhead in the servers, all your switches and routers need to be configured to handle that.
An IP packet is almost never transmitted as it is. In our world, we normally encapsulate the IP packets in an Ethernet frame.
So, if you see oversized frames, this means that some host is sending Ethernet frames that are larger than your configured L2 MTU. This in turn means that the host has an L3 (IP ) MTU that is larger than normal. There is not necessarily something wrong with varying MTU settings in your switches. The most important thing is to have a switch MTU that is at least the size of the largest IP packet plus the Ethernet overhead.
