System Description: "Extreme Networks, Inc. Bonded S4 Chassis Rev 08.4\
2.02.0012H3 03/14/2016--15:50 ofc"
I am not sure if this can be configured directly on an S-Series.
But you can try to enable PMTUD with the command set mtu enable and use a network that send ICMP Packet Too Big messages. But since PMTUD is default on the S-Series this probably does not help, and the network either does not generate Packet Too Big or it is filtered somewhere.
As far as I know the IP MTU cannot be controlled directly on the S-Series, the port MTU cannot be reduced below 1518 B (for Ethernet), and the S-Series will not create IP packets larger than 1500 B (but transport them if jumbo frames are enabled, including layer 3 forwarding).
Direct info from vendor below if anyone wants to research and understand this better.
""Let me elaborate. xxxx ISD has purchased a DDoS protection solution that is a hybrid of an on premise appliance and a cloud scrubbing service for volumetric attacks. The mechanism to gain attack traffic to the cloud for scrubbing is to divert the customer traffic over to our scrubbing center via BGP announcements (shortest path) and return clean traffic back to xxxx over GRE tunnels. It must be returned over GRE since the scrubbing center is advertising xxxx AS there is no way to dynamically route back to them. While under diversion xxxx traffic will be asymmetrical with outbound traffic passing out the normal path and inbound through our scrubbing center and over GRE back to customer
Due to the nature of GRE (the extra overhead due to the encapsulation) any packet that is already at 1500MTU will need to be fragmented once the GRE overhead is added to keep it from being a jumbo frame. The problem is when an application has the DF (don’t frag) flag set in the TCP header. Since it cannot be fragmented it gets dropped as a jumbo. The workaround to this is to insure the max size of a TCP packet is under 1500 (we recommend 1360) to allow for GRE overhead without fragmenting. This is normally accomplished in the customers internet router. Depending on the make of the router it could be called MSS clamping or TCP adjust. Any packet that contains an initial TCP header flowing through your router will be examined against the MSS in the router. The MSS in the header will be lowered to this amount if the setting is lower than what is in the header. If the header value is already lower, it will flow through unmodified. The end hosts will use the lower setting of the two hosts. For example, in a Cisco router the command will look like "ip tcp adjust-mss 1360".The goal is to insure TCP sessions between two hosts negotiate to 1360MTU so that there will be no jumbo frames once GRE encapsulation is added.""
I am not sure I understand your question completely. :-(
As I understood your question you asked for a way to change the MSS used by the TCP connection the BGP peering is based upon. But that is not what a DDoS scrubbing service would need.
As long as no cloud scrubbing is happening, no encapsulation overhead is added by the DDoS mitigation service. That was the simple part. ;-)
For cloud based scrubbing, the data is diverted to the cloud, scrubbed, and the clean traffic sent GRE encapsulated to the customer. As I understand it that should affect traffic through the BGP router, but not (all) traffic to the BGP router, especially the BGP sessions.
Thus it is needed to reduce the IP MTU, not just the TCP MSS, for all through traffic (there is still UDP traffic that can fill 1500B, e.g. DNSSEC or QUIC or DTLS, that might need to pass through the scrubber).
As written above I do not know of a way to implement this with the S-Series. As far as I know this is not possible with EXOS either (configure ip-mtu accepts values of 1500 and bigger only).
To my knowledge adjusting the IP MTU (or even the TCP MSS) to lower values is not usually found in multi-layer switches, but rather in routers and firewalls.
As you hinted at, the Brocade router platforms might provide this feature, but I do not know this.