Link Problems, Flapping, Flood Rate Limit Activated, clear eee stats Feature unavailable?

[Anonymous]

Have created an ACL that is meant to be blocking MDNS multicast addresses and an additional address used my Microsoft.

Have written the ACL to every port on Ingress so that I can see hits per port.

Problem is I'm not seeing the counters incrementing and aside from a packet trace I am confident there is this traffic on the network. I know this because we are trying to resolve an issue with a stack of X440's that keep rebooting because the CPU seems to be getting overwhelmed with packets from these address - as diagnosed by GTAC.

Policies at Policy Server:
Policy: Block_MDNS_Ingress
entry Block_1_MDNS_Ingress {
if match all {
source-address 224.0.0.251/32 ;
}
then {
deny ;
packet-count Block_251_MDNS_Ingress ;
}
}
entry Block_2_MDNS_Ingress {
if match all {
source-address 224.0.0.252/32 ;
}
then {
deny ;
packet-count Block_252_MDNS_Ingress ;
}
}
entry Block_3_MDNS_Ingress {
if match all {
source-address 239.255.255.250/32 ;
}
then {
deny ;
packet-count Block_250_MDNS_Ingress ;
}
}
Number of clients bound to policy: 1
Client: acl bound once

System Type: X440-48p (Stack)

SysHealth check: Enabled (Normal)
Recovery Mode: All
System Watchdog: Enabled

Current Time: Sat Sep 12 16:28:48 2015
Timezone: [Auto DST Disabled] GMT Offset: 0 minutes, name is UTC.
Boot Time: Fri Aug 28 00:37:38 2015
Boot Count: 135
Next Reboot: None scheduled
System UpTime: 15 days 15 hours 51 minutes 9 seconds

Slot: Slot-1 * Slot-2
------------------------ ------------------------
Current State: MASTER BACKUP (In Sync)

Image Selected: secondary secondary
Image Booted: secondary secondary
Primary ver: 15.3.1.4 15.3.1.4
Secondary ver: 15.5.4.2 15.5.4.2
patch1-5 patch1-5

Config Selected: primary.cfg
Config Booted: Factory Default

primary.cfg Created by ExtremeXOS version 15.5.4.2
2246563 bytes saved on Fri Sep 11 07:54:18 2015

Many thanks in advance.

[Prashanth_KG]

Martin,

Thanks for the reply.

Please share the following outputs to investiagate the eee counters error.

show conf | include eee
show slot

Regarding the flow-control, on the ports where the congestion is seen, please try disabling the rx pause with the following command:

disable flow-control rx-pause port

This would ensure that the traffic is not stopped transmitting even if a pause frame is received.

Monitor and let us know if that helps!!

[Anonymous]

Thanks for posting back.

Regarding eee this is not configured on this switch. Port 3:46 is currently not active and 2:46 shows a Dell device attached, and it a copper port:

Port: 2:46
Virtual-router: VR-Default
Type: UTP
Redundant Type: NONE
Random Early drop: Unsupported
Admin state: Enabled
Copper Medium Configuration: auto-speed sensing auto-duplex auto-polarity on
Fiber Medium Configuration: auto-speed sensing auto-duplex
Link State: Active, 1Gbps, full-duplex
Link Ups: 2 Last: Mon Sep 14 06:37:07 2015
Link Downs: 1 Last: Mon Sep 14 06:37:04 2015

With regards to broadcast control what would your recommendation be, either as a value for 100/1gb links, and if to be done by ACL metering what the best default value / configuration might be?

Is there anything in EXOS that is equivalent to configuring Link Flap as have been struggling to find anything?

We are still currently experiencing packet loss, although this is only on ports on the first switch / master switch in the stack which consists of 4 x 440x's, also interesting is the values are all pretty much the same?

1:26 A 548181
1:27 A 547201
1:32 A 548213
1:36 A 548182
1:38 A 548182
1:39 A 548204
1:41 A 548206

These ports are all 100mb and possibly connected to IP Phones, maybe there is a PC piggy backing them (need to investigate), negotiation issue, contention issue with 1gb PC port etc. The master switch seems to be having congestion problems as a whole if you look at the output below, yet port utilisation is next to nil.

Stack 1.19 # show stacking
Stack Topology is a Ring
Active Topology is a Ring
Node MAC Address Slot Stack State Role Flags
------------------ ---- ----------- ------- ---
*00:04:96:82:46:c1 1 Active Master CA-
00:04:96:82:46:ec 2 Active Backup CA-
00:04:96:82:10:07 3 Active Standby CA-
00:04:96:82:44:34 4 Active Standby CA-

Stack 1.24 # show stacking configuration
Stack MAC in use: 02:04:96:82:46:c1
Node Slot Alternate Alternate
MAC Address Cfg Cur Prio Mgmt IP / Mask Gateway Flags Lic
------------------ --- --- ---- ------------------ --------------- --------- ---
*00:04:96:82:46:c1 1 1 50 CcEeMm-Nn --
00:04:96:82:46:ec 2 2 45 CcEeMm-Nn --
00:04:96:82:10:07 3 3 1 --EeMm-Nn --
00:04:96:82:44:34 4 4 Auto --EeMm-Nn --

Stack 1.18 # debug hal show congestion
Congestion information for slot 1 type X440-48p since last query
Switch fabric congestion present: 2559080

Congestion information for slot 2 type X440-48p since last query
No switch fabric and CPU congestion present

Congestion information for slot 3 type X440-48p since last query
No switch fabric and CPU congestion present

Congestion information for slot 4 type X440-48p since last query
Switch fabric congestion present: 504

Before correcting the QoSprofiles it also seems that the CPU was experiencing congestion too:

CPU congestion present: 237

Here is the port information for 1:26 that is experiencing packet loss:

Port: 1:26
Virtual-router: VR-Default
Type: UTP
Random Early drop: Unsupported
Admin state: Enabled with auto-speed sensing auto-duplex
Link State: Active, 100Mbps, full-duplex
Link Ups: 0 Last: Sun Sep 13 10:13:19 2015
Link Downs: 0 Last: Sun Sep 13 10:13:10 2015

VLAN cfg:
Name: HH-Data, Internal Tag = 100, MAC-limit = No-limit, Virtual router: VR-Default
Name: HH-Voice, 802.1Q Tag = 200, MAC-limit = No-limit, Virtual router: VR-Default
Port-specific VLAN ID: 200
STP cfg:
s0(enable), Tag=(none), Mode=802.1D, State=FORWARDING
s1(enable), Tag=(none), Mode=802.1D, State=FORWARDING

Protocol:
Name: HH-Data Protocol: ANY Match all protocols.
Trunking: Load sharing is not enabled.

EDP: Enabled

EEE: Disabled
ELSM: Disabled
Ethernet OAM: Disabled
Learning: Enabled
Unicast Flooding: Enabled
Multicast Flooding: Enabled
Broadcast Flooding: Enabled
Jumbo: Disabled
Flow Control: Rx-Pause: Enabled Tx-Pause: Enabled
Priority Flow Control: Disabled
Reflective Relay: Disabled
Link up/down SNMP trap filter setting: Enabled
Egress Port Rate: No-limit
Broadcast Rate: 30 packets-per-second
Multicast Rate: No-limit
Unknown Dest Mac Rate: No-limit
QoS Profile: None configured
Ingress Rate Shaping : Unsupported
Ingress IPTOS Examination: Enabled
Ingress 802.1p Examination: Enabled
Ingress 802.1p Inner Exam: Disabled
Egress IPTOS Replacement: Disabled
Egress 802.1p Replacement: Disabled
NetLogin: Disabled
NetLogin port mode: Port based VLANs
Smart redundancy: Enabled
Software redundant port: Disabled
IPFIX: Disabled Metering: Ingress, All Packets, All Traffic
IPv4 Flow Key Mask: SIP: 255.255.255.255 DIP: 255.255.255.255
IPv6 Flow Key Mask: SIP: ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff
DIP: ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff

auto-polarity: Enabled
Shared packet buffer: 100%
VMAN CEP egress filtering: Disabled
Isolation: Off
PTP Configured: Disabled
Time-Stamping Mode: None
Synchronous Ethernet: Unsupported
Dynamic VLAN Uplink: Disabled
VM Tracking Dynamic VLANs: Disabled

As for flow control RX and TX has been configured for ALL ports, not sure why and in my view doesn't seem necessary on this site, but interested on your view of when it should and shouldn't be enabled in either direction.

Here is the output of port utilisation:

show port utilization bandwidth
Port Link Link Rx Peak Rx Tx Peak Tx
State Speed % bandwidth % bandwidth % bandwidth % bandwidth
================================================================================
1:1 A 100 0.00 0.00 0.40 0.40
1:2 R 0 0.00 0.00 0.00 0.00
1:3 A 1000 0.00 0.00 0.00 0.00
1:4 R 0 0.00 0.00 0.00 0.00
1:5 A 100 0.00 0.00 0.40 0.40
1:6 A 1000 0.00 0.00 0.00 0.00
1:7 A 100 0.00 0.00 0.40 0.40
1:8 R 0 0.00 0.00 0.00 0.00
1:9 R 0 0.00 0.00 0.00 0.00
1:10 A 1000 0.00 0.00 0.00 0.00
1:11 A 1000 0.00 0.00 0.00 0.00
1:12 A 1000 0.00 0.00 0.00 0.00
1:13 A 100 0.00 0.00 0.40 0.40
1:14 A 1000 0.00 0.00 0.00 0.00
1:15 A 1000 0.00 0.00 0.00 0.00
1:16 R 0 0.00 0.00 0.00 0.00
1:17 R 0 0.00 0.00 0.00 0.00
1:18 A 1000 0.00 0.00 0.00 0.00
1:19 A 1000 0.00 0.00 0.00 0.00
1:20 R 0 0.00 0.00 0.00 0.00
1:21 A 100 0.00 0.00 0.40 0.40
1:22 R 0 0.00 0.00 0.00 0.00
1:23 R 0 0.00 0.00 0.00 0.00
1:24 A 10 0.00 0.00 4.02 4.02
1:25 A 1000 0.00 0.00 0.00 0.00
1:26 A 100 0.00 0.00 0.43 0.43
1:27 A 100 0.00 0.00 0.43 0.43
1:28 A 1000 0.00 0.00 0.00 0.00
1:29 R 0 0.00 0.00 0.00 0.00
1:30 R 0 0.00 0.00 0.00 0.00
1:31 A 1000 0.00 0.00 0.00 0.00
1:32 A 100 0.00 0.00 0.43 0.43
1:33 A 1000 0.00 0.00 0.00 0.00
1:34 A 1000 0.00 0.00 0.05 0.05
1:35 R 0 0.00 0.00 0.00 0.00
1:36 A 100 0.00 0.00 0.43 0.43
1:37 R 0 0.00 0.00 0.00 0.00
1:38 A 100 0.00 0.00 0.43 0.43
1:39 A 100 0.00 0.00 0.43 0.43
1:40 A 100 0.00 0.00 4.92 4.92
1:41 A 100 0.00 0.00 0.43 0.43
1:42 R 0 0.00 0.00 0.00 0.00
1:43 A 1000 0.00 0.00 0.05 0.05
1:44 R 0 0.00 0.00 0.00 0.00
1:45 R 0 0.00 0.00 0.00 0.00
1:46 R 0 0.00 0.00 0.00 0.00
1:47 R 0 0.00 0.00 0.00 0.00
CoreUpli> A 1000 1.39 1.39 0.02 0.02
2:1 A 100 0.00 0.00 0.40 0.40
2:2 R 0 0.00 0.00 0.00 0.00
2:3 A 100 0.00 0.00 0.40 0.40
2:4 A 1000 0.00 0.00 0.04 0.04
2:5 A 100 0.00 0.00 0.40 0.40
2:6 A 100 0.01 0.01 0.41 0.41
2:7 R 0 0.00 0.00 0.00 0.00
2:8 R 0 0.00 0.00 0.00 0.00
2:9 R 0 0.00 0.00 0.00 0.00
2:10 R 0 0.00 0.00 0.00

[Prashanth_KG]

Hi Martin,

As Stephane asked, we need to know if flow-control is necessary for your environment. Because, when pause frames are received by the switch, it will stop transmitting the packets for the specified time frame. During this time, the egress buffer of the port would be full and it could result in the packet drops which you are noticing on the ports. Similarly, with tx-pause frames enabled on the switch port, the switch could send out the pause frames to the connected switch.

Also, please check if you have the eee support enabled on the ports.

command:
configure port <> eee on

This command is to enable EEE on the switch. Specifies that the port advertises to its linkpartner that it is EEE capable at certain speeds. If both sides, during auto-negotiation, determine that
they both have EEE on and are compatible speed wise, they will determine other parameters (how long
it takes to come out of sleep time, how long it takes to wake up) and the link comes up. During periods
of non-activity, the link will shut down parts of the port to save energy. This is called LPI for low power idle. When one side sees it must send something, it wakes up the remote and then transmits.

You could find more details in the command reference guide!
The below error could be because of the fact that ports 3:46 and 2:46 do not support eee feature.

09/12/2015 22:57:18.26 Slot-1: could not clear eee stats Feature unavailable for slot 3: port 4609/12/2015 22:57:17.61 Slot-1: could not clear eee stats Feature unavailable for slot 2: port 46

Please let us know what switches are slots 2 and 3. And if port 46 is a copper of a fiber port.

Hope this helps!

[Stephane_Grosj1]

Hi,

Do you have Flow Control enabled? If yes, how and why?

From the logs, port 4:17 is appearing very often, and keeps flapping between 10Mbps HD and 1G FD. What's behind it? 10Mbps HD may be the result of wrong autoneg (usually only one side has it configured).

As for the flood-rate, I believe 30 / 300 are very low values. The HW is not based on a 1s monitoring, but works at a much more higher frequency, and it extrapolates your config values into some other values (I don't remember exactly off the top of my head the exact math behind it, see with GTAC for a deeper analysis and explanation). A small burst can trigger the threshold even though on a 1s basis the traffic is under your setting. If you need some accurate threshold I would recommend using ACL with meters instead of the flood-rate knob.

My 2cents.

[Anonymous]

Just as an update I noticed the switch had qosprofiles 2-6 configured, although only QP6 was being used. This therefore had port buffer space being reserved unnecessarily. After removing the unused qosprofiles the packet drops seem to have got a lot better.

Still a lot to do and appreciate any feedback on the above. Thanks