G/C5/C3/B5/B3-Series Interface IP for Host/Management Functions has Intermittently Slow Response

[FAQ_User]

Article ID: 14754

Products
G-Series, firmware 6.42.01.0046 through 6.61.08.0013
C5-Series, firmware 6.41.00.0022 through 6.61.08.0013, 6.71.01.0067 through 6.71.02.0008
C3-Series, firmware 6.42.01.0046 through 6.61.08.001
B5-Series, firmware 6.41.00.0022 through 6.61.08.0013, 6.71.01.0067 through 6.71.02.0008
B3-Series, firmware 6.42.01.0046 through 6.61.08.001

Changes
VLAN-tagged host management traffic, tagged with a Priority of 2 or less; is addressed to a local L3 Interface IP address but destined for the local host, or is a certain broadcast type expected to be relayed by the local host.
Also; presence in any local VLAN of a "significant" (not necessarily large) amount of flooded broadcast, non-scoped multicast, or unknown unicast traffic.

Symptoms
Performance of host management (e.g. SSH, Telnet, TACACS, TFTP, SNTP/NTP, SNMP, PING/ICMP) functions is adversely affected.
Performance of subnet relay (e.g. Wake-on-LAN, DNS, Bootp/DHCP) via the forward-protocol function (11980) is adversely affected.
Intermittent loss of management traffic.
Slow management response.

Cause
Under the stated circumstances; this traffic will, at best, share a transmit queue with the generic broadcast/multicast/unicast traffic being flooded to the host CPU.
Further, low priority internal queues to the host are ratelimited so may silently drop such traffic.

Solution/Workaround
Upgrade to 6.61 firmware 6.61.09.0012 or higher.
Release notes document the treatment change, in the 'Changes and Enhancements in 6.61.09.0012' section:

G-Series:
16073 Adjusted the priority of packets destined to the IPv4 address of loopback interface 1 (if configured), to increase the ability to maintain management when there is large volumes for traffic trapped to the host CPU.

C5-Series:
16073 Adjusted the priority of packets destined to IPv4 primary and loopback interface addresses, to increase the ability to maintain management, when there is large volumes for traffic trapped to the host CPU.

C3-Series:
16073 Adjusted the priority of packets destined to IPv4 loopback interface addresses, to increase the ability to maintain management, when there is large volumes for traffic trapped to the host CPU.

B5-Series:
16073 Adjusted the priority of packets destined to primary IPv4 and loopback interface addresses, to increase the ability to maintain management, when there is large volumes for traffic trapped to the host CPU.

For the C5/B5, you may also upgrade to firmware 6.71.03.0025 or higher.
Release notes state, in the 'Changes and Enhancements in 6.71.03.0025' section:
16073 Adjusted the priority of packets destined to IPv4 primary and loopback interface addresses, to increase the ability to maintain management, when there is large volumes for traffic trapped to the host CPU.

Pre-upgrade workaround: Adjust the priority of legitimate host-destined or host-relayed traffic, somewhere between its transmission and local host receipt.

One prioritization option is to set the CoS/Priority of management traffic on the (untagged first hop) edge so that the Priority has a value of 3 or higher when forwarded within the VLAN tag of the last hop. There are several methods to do this; including Policy, Access Control Lists (ACLs) using the assign-queue option (f/w 6.51.01.0018 and higher), and Port Priority ('set cos state disable', 'set port priority <port#> 3'). The most granular method is to use Policy.

These sample Policy configurations may be used upon ingress on the first or intermediate hop, or on the tagged last hop. They may be applied to any policy-compliant switch (including the affected switches under discussion) within the data path, to identify host-destined traffic and increase its priority to 3 for assignment to transmit queue 2, which is the minimum queue# necessary to promote good performance in the presence of flooded traffic within transmit queue 1 (7584).
The suggested rules may be tailored as desired; to best accommodate network requirements in consideration of which devices need to connect to management, where they are located, and which protocols they need to use. The reprioritization action is triggered when any of the underlying rules encounter a match while examining a packet.

Example 1 (generic)
set policy profile 1 name manager [Creates a role/profile #1, named "manager"]
set policy rule 1 ipdestsocket x.x.x.x mask 32 cos 3 [Matches the Interface IP destination]Example 2 (focused)
set policy profile 1 name manager [Creates a role/profile #1, named "manager"]
set policy rule 1 ipdestsocket x.x.x.x:22 mask 48 cos 3 [Matches SSH to the Interface IP]
set policy rule 1 ipdestsocket x.x.x.x:23 mask 48 cos 3 [Matches Telnet to the Interface IP]
set policy rule 1 ipdestsocket x.x.x.x:49 mask 48 cos 3 [Matches TACACS to the Interface IP]
set policy rule 1 ipdestsocket x.x.x.x:69 mask 48 cos 3 [Matches TFTP to the Interface IP]
set policy rule 1 ipdestsocket x.x.x.x:123 mask 48 cos 3 [Matches SNTP/NTP to the Interface IP]
set policy rule 1 ipdestsocket x.x.x.x:161 mask 48 cos 3 [Matches SNMP to the Interface IP]
set policy rule 1 ipdestsocket 255.255.255.255:0 mask 48 cos 3 [Matches WoL for relay]
set policy rule 1 ipdestsocket 255.255.255.255:7 mask 48 cos 3 [Matches WoL for relay]
set policy rule 1 ipdestsocket 255.255.255.255:9 mask 48 cos 3 [Matches WoL for relay]
set policy rule 1 ipdestsocket 255.255.255.255:53 mask 48 cos 3 [Matches DNS for relay]
set policy rule 1 ipdestsocket 255.255.255.255:67 mask 48 cos 3 [Matches Bootp/DHCP for relay]
set policy rule 1 ipdestsocket 255.255.255.255:68 mask 48 cos 3 [Matches Bootp client for relay]
set policy rule 1 ipproto 1 cos 3 [Matches PING/ICMP, to host or pass-through]To statically apply the policy to all ingress port(s):

set policy port ge.*.* 1

Note that this command is supported for standalone ports only, not LAG aggregator ports.

To instead dynamically apply the policy to the authenticated management user, use authentication (e.g. 10283).

If NetSight Policy Manager has been deployed, then it should be used to apply these configurations, as any manual policy configuration will be overwritten by PM's policy enforcement.