X460 and X450 Flapping Ports

[Martin_Harrow]

I have installed 15 new X460 switch stacks, and upgraded the Firmware on some old X450s.
I notice that many of the Ethernet Ports are "Flapping". There is no stable connection, and devices constantly disconnect/reconnect at 10M/100M/1000M. I note that the problem is worse with some Canon Printers, but other devices have problems. I can make the ports more stable by forcing the speed to 10M.

Here are some typical syslogs:

May 6 10:21:31 degp-s02 Port 5:2 link down
May 6 10:21:34 degp-s02 Port 5:2 link UP at speed 1 Gbps and full-duplex
May 6 10:22:25 degp-s02 Port 5:2 link down
May 6 10:22:27 degp-s02 Port 5:2 link UP at speed 10 Mbps and half-duplex
May 6 10:22:43 degp-s02 Port 4:24 link down
May 6 10:22:45 degp-s02 Port 4:24 link UP at speed 10 Mbps and half-duplex
May 6 10:22:47 degp-s02 Port 1:10 link down
May 6 10:22:50 degp-s02 Port 1:10 link UP at speed 1 Gbps and full-duplex
May 6 10:22:50 degp-s02 Port 5:2 link down
May 6 10:22:53 degp-s02 Port 5:2 link UP at speed 1 Gbps and full-duplex
May 6 10:23:14 degp-s02 Port 1:10 link down
May 6 10:23:16 degp-s02 Port 1:10 link UP at speed 1 Gbps and full-duplex
May 6 10:25:24 degp-s02 Port 5:2 link down
May 6 10:25:27 degp-s02 Port 5:2 link UP at speed 10 Mbps and half-duplex
May 6 10:25:39 degp-s02 Port 3:42 link down
May 6 10:25:41 degp-s02 Port 3:42 link UP at speed 10 Mbps and half-duplex
May 6 10:25:43 degp-s02 Port 3:42 link down
May 6 10:25:45 degp-s02 Port 3:42 link UP at speed 10 Mbps and half-duplex
May 6 10:26:29 degp-s02 Port 4:24 link down
May 6 10:26:32 degp-s02 Port 4:24 link UP at speed 10 Mbps and half-duplex
May 6 10:27:44 degp-s02 Port 3:38 link down
May 6 10:27:47 degp-s02 Port 3:38 link UP at speed 10 Mbps and half-duplex
May 6 10:27:51 degp-s02 Port 5:2 link down
May 6 10:27:53 degp-s02 Port 5:2 link UP at speed 1 Gbps and full-duplex
May 6 10:28:06 degp-s09 Port 1:16 link down
May 6 10:28:08 degp-s09 Port 1:16 link UP at speed 10 Mbps and half-duplex
May 6 10:30:05 degp-s03 Port 1:1 link down
May 6 10:30:06 degp-s09 Port 1:16 link down
May 6 10:30:08 degp-s03 Port 1:1 link UP at speed 10 Mbps and full-duplex
May 6 10:30:09 degp-s09 Port 1:16 link UP at speed 1 Gbps and full-duplex
May 6 10:30:20 degp-s03 Port 1:1 link down
May 6 10:30:23 degp-s03 Port 1:1 link UP at speed 10 Mbps and full-duplex
May 6 10:30:57 degp-s03 Port 1:19 link down
May 6 10:30:59 degp-s03 Port 1:19 link UP at speed 10 Mbps and full-duplex
May 6 10:31:12 degp-s03 Port 1:19 link down
May 6 10:31:14 degp-s03 Port 1:19 link UP at speed 10 Mbps and full-duplex

How can I fix this? I have new(ish) f/w (X460 = v1541b3-patch1-1 and X450 = v1265b2-patch1-3)

Most of the cabling is brand new, could this be a problem?

Thanks,
Martin

[MUNIR_ENG]

thank for this information , but when i doing this commands i got on many values ! how can i know which the better value?

[Martin_Harrow]

I was using AUTO on all ports. for the flapping ports, I have set the connection to fixed 10M/half.
I think I must open a case.
What are the best commands to check the switch?
You suggested TOP? any others?

[Sumit_Tokle]

How long you have seen this issue? Have you done any changes which could trigger this issue?

Does auto negotiation is on,on all the port which are connected to the end devices? if yes, then you might need to open case with TAC for further investigation.

[Martin_Harrow]

hi

Tha cables are described: MegaLine-Patch 6AEA-RJ45SL
I have a L3 core switch (x460-24x) connected to 10 edge switches (x460-48p) and some older X450-48P), in a star network.

The problems are between the edge switches and the devices. Many Canon Printers iR-ADV C5235 (MAC 88:87:17:7e:82:2d) have the problem - see logs below.
Some PCs also have a problem.
The users are starting to complain about the new network.
The message occurs only on edge ports, not between switches.
There is no loop in the network ( I checked ELRP )
The messages appear every few minutes (see below)

* Slot-1 DEGP-S09.11 # show cpu-monitoring
(sorted output)
Card Process 5 10 30 1 5 30 1 Max Total
Slot-1 fdb 0.7 0.9 1.0 1.0 0.9 0.8 0.8 3.4 16,772.9 16,121.2
Slot-1 exsshd 2.4 2.0 1.6 1.5 1.6 1.1 0.9 6.8 3,707.4 20,690.4
Slot-1 snmpSubagent 0.0 0.0 0.0 0.0 0.0 0.1 0.0 21.9 2,935.9 1,951.0
Slot-1 nodemgr 0.2 0.1 0.1 0.1 0.1 0.1 0.1 0.5 2,509.3 2,651.9
Slot-1 snmpMaster 0.0 0.0 0.0 0.0 0.0 0.1 0.0 22.1 2,451.3 1,914.6
Slot-1 hal 0.8 0.5 0.5 0.4 0.4 0.4 0.4 66.6 2,365.6 14,125.2
Slot-1 epm 0.1 0.0 0.0 0.1 0.1 0.1 0.1 7.1 1,594.7 684.4
Slot-1 acl 0.2 0.2 0.1 0.1 0.1 0.1 0.1 0.8 1,493.1 4,160.6
Slot-1 cfgmgr 0.0 0.0 0.0 0.0 0.0 0.1 0.0 9.6 992.0 546.5
Slot-1 etmon 0.0 0.0 0.0 0.1 0.1 0.1 0.1 7.8 950.2 964.9
Slot-1 bfd 0.1 0.0 0.0 0.1 0.1 0.1 0.1 0.6 677.3 813.0
Slot-1 rip 0.1 0.0 0.0 0.1 0.1 0.1 0.1 0.8 599.5 1,227.9
Slot-1 r.png 0.0 0.0 0.0 0.1 0.1 0.1 0.1 0.6 578.6 679.5
Slot-1 thttpd 0.0 0.0 0.0 0.0 0.0 0.1 0.1 14.2 509.7 71.1
Slot-1 esrp 0.0 0.0 0.0 0.2 0.1 0.1 0.1 6.5 441.9 542.1
Slot-1 msdp 0.0 0.0 0.0 0.1 0.1 0.1 0.1 0.6 382.7 464.2
Slot-1 eaps 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.8 241.0 76.3
Slot-1 vlan 0.0 0.0 0.0 0.0 0.0 0.0 0.0 7.2 214.0 58.4
Slot-1 System 8.3 7.4 8.4 7.6 8.0 7.8 7.8 15.6 196.6 278,076.4

* Slot-1 DEGP-S09.13 # top
Mem: 402768K used, 107736K free, 0K shrd, 144268K buff, 91192K cached
CPU: 1.6% usr 12.7% sys 0.0% nic 81.3% idle 0.0% io 3.7% irq 0.4% sirq
Load average: 7.26 7.33 7.34 4/185 13639
PID PPID USER STAT RSS %MEM CPU %CPU COMMAND
1217 2 root RW< 0 0.0 0 4.2 [bcmLINK.0]
1219 2 root SW< 0 0.0 1 3.7 [bcmLINK.1]
13626 1695 root S 1324 0.2 1 2.1 ./exsshd
1439 1 root S 3212 0.6 1 0.9 ./fdb
13639 13638 root R 820 0.1 0 0.9 top -d 3

I also see on some switches (below is the core switch):
05/06/2014 14:40:44.09 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 13:40:33.31 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 12:40:32.72 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 11:40:29.57 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 10:40:20.42 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 09:40:00.96 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 08:39:54.83 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 07:39:47.14 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 06:39:43.40 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 05:39:41.97 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 04:39:40.86 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 03:39:39.95 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 02:39:39.16 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 01:39:37.87 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 00:39:36.93 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 23:39:35.91 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 22:39:35.10 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 21:39:33.98 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 20:39:32.66 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 19:39:31.67 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 18:39:18.65 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 17:39:09.66 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 16:39:08.45 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 15:38:55.07 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 14:38:53.81 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 13:38:53.23 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 12:38:49.74 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 11:38:41.42 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 10:38:33.90 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 09:38:32.96 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 08:38:30.29 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 07:38:21.25 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.

05/06/2014 15:08:40.90 Slot-1: Port 1:16 link UP at speed 1 Gbps and full-duplex
05/06/2014 15:08:38.38 Slot-1: Port 1:16 link down
05/06/2014 15:08:34.09 Slot-1: Port 1:16 link UP at speed 10 Mbps and half-duplex
05/06/2014 15:08:31.36 Slot-1: Port 1:16 link down
05/06/2014 15:01:02.33 Slot-1: Port 1:15 link UP at speed 100 Mbps and full-duplex
05/06/2014 15:00:59.91 Slot-1: Port 1:15 link down
05/06/2014 14:58:23.57 Slot-1: Port 1:16 link UP at speed 1 Gbps and full-duplex
05/06/2014 14:58:21.19 Slot-1: Port 1:16 link down
05/06/2014 14:56:56.98 Slot-1: Port 1:8 link UP at speed 100 Mbps and full-duplex
05/06/2014 14:56:55.25 Slot-1: Port 1:8 link down
05/06/2014 14:53:49.54 Slot-1: Port 1:16 link UP at speed 10 Mbps and half-duplex
05/06/2014 14:53:47.21 Slot-1: Port 1:16 link down
05/06/2014 14:51:26.24 Slot-1: Port 1:16 link UP at speed 1 Gbps and full-duplex
05/06/2014 14:51:23.67 Slot-1: Port 1:16 link down
05/06/2014 14:48:55.47 Slot-1: Port 1:16 link UP at speed 10 Mbps and half-duplex
05/06/2014 14:48:52.90 Slot-1: Port 1:16 link down
05/06/2014 14:38:36.24 Slot-1: Port 1:16 link UP at speed 1 Gbps and full-duplex
05/06/2014 14:38:33.79 Slot-1: Port 1:16 link down
05/06/2014 14:38:31.71 Slot-1: Port 1:16 link UP at speed 10 Mbps and half-duplex

I need to find a solution for this, it cannot be normal.

I think the flapping is blocking the CPU, because sometimes the Ridgeline cannot get an SNMP response - I suspect CPU overload.

Thx, Martin

[Ronald_Dvorak]

please refer to the "ExtremeXOS Command Reference Guide" to get information about the different commands...

# top

Description:
Displays real-time CPU utilization information by process.