Hi EtherMAN,
I am not sure I understand your question completely. 😞
As I understood your question you asked for a way to change the MSS used by the TCP connection the BGP peering is based upon. But that is not what a DDoS scrubbing service would need.
As long as no cloud scrubbing is happening, no encapsulation overhead is added by the DDoS mitigation service. That was the simple part. 😉
For cloud based scrubbing, the data is diverted to the cloud, scrubbed, and the clean traffic sent GRE encapsulated to the customer. As I understand it that should affect traffic through the BGP router, but not (all) traffic to the BGP router, especially the BGP sessions.
Thus it is needed to reduce the IP MTU, not just the TCP MSS, for all through traffic (there is still UDP traffic that can fill 1500B, e.g. DNSSEC or QUIC or DTLS, that might need to pass through the scrubber).
As written above I do not know of a way to implement this with the S-Series. As far as I know this is not possible with EXOS either (configure ip-mtu accepts values of 1500 and bigger only).
To my knowledge adjusting the IP MTU (or even the TCP MSS) to lower values is not usually found in multi-layer switches, but rather in routers and firewalls.
As you hinted at, the Brocade router platforms might provide this feature, but I do not know this.
Thanks,
Erik