topic RE: X460 and X450 Flapping Ports in ExtremeSwitching (EXOS/Switch Engine)

X460 and X450 Flapping Ports

Martin_Harrow — Tue, 06 May 2014 13:53:00 GMT

I have installed 15 new X460 switch stacks, and upgraded the Firmware on some old X450s.
I notice that many of the Ethernet Ports are "Flapping". There is no stable connection, and devices constantly disconnect/reconnect at 10M/100M/1000M. I note that the problem is worse with some Canon Printers, but other devices have problems. I can make the ports more stable by forcing the speed to 10M.

Here are some typical syslogs:

May 6 10:21:31 degp-s02 Port 5:2 link down
May 6 10:21:34 degp-s02 Port 5:2 link UP at speed 1 Gbps and full-duplex
May 6 10:22:25 degp-s02 Port 5:2 link down
May 6 10:22:27 degp-s02 Port 5:2 link UP at speed 10 Mbps and half-duplex
May 6 10:22:43 degp-s02 Port 4:24 link down
May 6 10:22:45 degp-s02 Port 4:24 link UP at speed 10 Mbps and half-duplex
May 6 10:22:47 degp-s02 Port 1:10 link down
May 6 10:22:50 degp-s02 Port 1:10 link UP at speed 1 Gbps and full-duplex
May 6 10:22:50 degp-s02 Port 5:2 link down
May 6 10:22:53 degp-s02 Port 5:2 link UP at speed 1 Gbps and full-duplex
May 6 10:23:14 degp-s02 Port 1:10 link down
May 6 10:23:16 degp-s02 Port 1:10 link UP at speed 1 Gbps and full-duplex
May 6 10:25:24 degp-s02 Port 5:2 link down
May 6 10:25:27 degp-s02 Port 5:2 link UP at speed 10 Mbps and half-duplex
May 6 10:25:39 degp-s02 Port 3:42 link down
May 6 10:25:41 degp-s02 Port 3:42 link UP at speed 10 Mbps and half-duplex
May 6 10:25:43 degp-s02 Port 3:42 link down
May 6 10:25:45 degp-s02 Port 3:42 link UP at speed 10 Mbps and half-duplex
May 6 10:26:29 degp-s02 Port 4:24 link down
May 6 10:26:32 degp-s02 Port 4:24 link UP at speed 10 Mbps and half-duplex
May 6 10:27:44 degp-s02 Port 3:38 link down
May 6 10:27:47 degp-s02 Port 3:38 link UP at speed 10 Mbps and half-duplex
May 6 10:27:51 degp-s02 Port 5:2 link down
May 6 10:27:53 degp-s02 Port 5:2 link UP at speed 1 Gbps and full-duplex
May 6 10:28:06 degp-s09 Port 1:16 link down
May 6 10:28:08 degp-s09 Port 1:16 link UP at speed 10 Mbps and half-duplex
May 6 10:30:05 degp-s03 Port 1:1 link down
May 6 10:30:06 degp-s09 Port 1:16 link down
May 6 10:30:08 degp-s03 Port 1:1 link UP at speed 10 Mbps and full-duplex
May 6 10:30:09 degp-s09 Port 1:16 link UP at speed 1 Gbps and full-duplex
May 6 10:30:20 degp-s03 Port 1:1 link down
May 6 10:30:23 degp-s03 Port 1:1 link UP at speed 10 Mbps and full-duplex
May 6 10:30:57 degp-s03 Port 1:19 link down
May 6 10:30:59 degp-s03 Port 1:19 link UP at speed 10 Mbps and full-duplex
May 6 10:31:12 degp-s03 Port 1:19 link down
May 6 10:31:14 degp-s03 Port 1:19 link UP at speed 10 Mbps and full-duplex

How can I fix this? I have new(ish) f/w (X460 = v1541b3-patch1-1 and X450 = v1265b2-patch1-3)

Most of the cabling is brand new, could this be a problem?

Thanks,
Martin

RE: X460 and X450 Flapping Ports

Sumit_Tokle — Tue, 06 May 2014 17:09:00 GMT

What type of cable are you using to connect two switches?
Is there any L2 switch present in between the X460 and X450.
Does same type of log messages present on both the side of switch port during the same time?
How frequently the ports are getting flapped?
Is there any loop in the network?
Do you see any process going high in "top" command output?

RE: X460 and X450 Flapping Ports

Martin_Harrow — Tue, 06 May 2014 18:40:00 GMT

hi

Tha cables are described: MegaLine-Patch 6AEA-RJ45SL
I have a L3 core switch (x460-24x) connected to 10 edge switches (x460-48p) and some older X450-48P), in a star network.

The problems are between the edge switches and the devices. Many Canon Printers iR-ADV C5235 (MAC 88:87:17:7e:82:2d) have the problem - see logs below.
Some PCs also have a problem.
The users are starting to complain about the new network.
The message occurs only on edge ports, not between switches.
There is no loop in the network ( I checked ELRP )
The messages appear every few minutes (see below)

* Slot-1 DEGP-S09.11 # show cpu-monitoring
(sorted output)
Card Process 5 10 30 1 5 30 1 Max Total
Slot-1 fdb 0.7 0.9 1.0 1.0 0.9 0.8 0.8 3.4 16,772.9 16,121.2
Slot-1 exsshd 2.4 2.0 1.6 1.5 1.6 1.1 0.9 6.8 3,707.4 20,690.4
Slot-1 snmpSubagent 0.0 0.0 0.0 0.0 0.0 0.1 0.0 21.9 2,935.9 1,951.0
Slot-1 nodemgr 0.2 0.1 0.1 0.1 0.1 0.1 0.1 0.5 2,509.3 2,651.9
Slot-1 snmpMaster 0.0 0.0 0.0 0.0 0.0 0.1 0.0 22.1 2,451.3 1,914.6
Slot-1 hal 0.8 0.5 0.5 0.4 0.4 0.4 0.4 66.6 2,365.6 14,125.2
Slot-1 epm 0.1 0.0 0.0 0.1 0.1 0.1 0.1 7.1 1,594.7 684.4
Slot-1 acl 0.2 0.2 0.1 0.1 0.1 0.1 0.1 0.8 1,493.1 4,160.6
Slot-1 cfgmgr 0.0 0.0 0.0 0.0 0.0 0.1 0.0 9.6 992.0 546.5
Slot-1 etmon 0.0 0.0 0.0 0.1 0.1 0.1 0.1 7.8 950.2 964.9
Slot-1 bfd 0.1 0.0 0.0 0.1 0.1 0.1 0.1 0.6 677.3 813.0
Slot-1 rip 0.1 0.0 0.0 0.1 0.1 0.1 0.1 0.8 599.5 1,227.9
Slot-1 r.png 0.0 0.0 0.0 0.1 0.1 0.1 0.1 0.6 578.6 679.5
Slot-1 thttpd 0.0 0.0 0.0 0.0 0.0 0.1 0.1 14.2 509.7 71.1
Slot-1 esrp 0.0 0.0 0.0 0.2 0.1 0.1 0.1 6.5 441.9 542.1
Slot-1 msdp 0.0 0.0 0.0 0.1 0.1 0.1 0.1 0.6 382.7 464.2
Slot-1 eaps 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.8 241.0 76.3
Slot-1 vlan 0.0 0.0 0.0 0.0 0.0 0.0 0.0 7.2 214.0 58.4
Slot-1 System 8.3 7.4 8.4 7.6 8.0 7.8 7.8 15.6 196.6 278,076.4

* Slot-1 DEGP-S09.13 # top
Mem: 402768K used, 107736K free, 0K shrd, 144268K buff, 91192K cached
CPU: 1.6% usr 12.7% sys 0.0% nic 81.3% idle 0.0% io 3.7% irq 0.4% sirq
Load average: 7.26 7.33 7.34 4/185 13639
PID PPID USER STAT RSS %MEM CPU %CPU COMMAND
1217 2 root RW< 0 0.0 0 4.2 [bcmLINK.0]
1219 2 root SW< 0 0.0 1 3.7 [bcmLINK.1]
13626 1695 root S 1324 0.2 1 2.1 ./exsshd
1439 1 root S 3212 0.6 1 0.9 ./fdb
13639 13638 root R 820 0.1 0 0.9 top -d 3

I also see on some switches (below is the core switch):
05/06/2014 14:40:44.09 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 13:40:33.31 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 12:40:32.72 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 11:40:29.57 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 10:40:20.42 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 09:40:00.96 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 08:39:54.83 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 07:39:47.14 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 06:39:43.40 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 05:39:41.97 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 04:39:40.86 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 03:39:39.95 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 02:39:39.16 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 01:39:37.87 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/06/2014 00:39:36.93 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 23:39:35.91 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 22:39:35.10 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 21:39:33.98 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 20:39:32.66 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 19:39:31.67 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 18:39:18.65 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 17:39:09.66 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 16:39:08.45 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 15:38:55.07 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 14:38:53.81 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 13:38:53.23 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 12:38:49.74 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 11:38:41.42 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 10:38:33.90 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 09:38:32.96 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 08:38:30.29 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.
05/05/2014 07:38:21.25 Slot-2: IPv4 multicast entry not added. Hardware L3 Table full.

05/06/2014 15:08:40.90 Slot-1: Port 1:16 link UP at speed 1 Gbps and full-duplex
05/06/2014 15:08:38.38 Slot-1: Port 1:16 link down
05/06/2014 15:08:34.09 Slot-1: Port 1:16 link UP at speed 10 Mbps and half-duplex
05/06/2014 15:08:31.36 Slot-1: Port 1:16 link down
05/06/2014 15:01:02.33 Slot-1: Port 1:15 link UP at speed 100 Mbps and full-duplex
05/06/2014 15:00:59.91 Slot-1: Port 1:15 link down
05/06/2014 14:58:23.57 Slot-1: Port 1:16 link UP at speed 1 Gbps and full-duplex
05/06/2014 14:58:21.19 Slot-1: Port 1:16 link down
05/06/2014 14:56:56.98 Slot-1: Port 1:8 link UP at speed 100 Mbps and full-duplex
05/06/2014 14:56:55.25 Slot-1: Port 1:8 link down
05/06/2014 14:53:49.54 Slot-1: Port 1:16 link UP at speed 10 Mbps and half-duplex
05/06/2014 14:53:47.21 Slot-1: Port 1:16 link down
05/06/2014 14:51:26.24 Slot-1: Port 1:16 link UP at speed 1 Gbps and full-duplex
05/06/2014 14:51:23.67 Slot-1: Port 1:16 link down
05/06/2014 14:48:55.47 Slot-1: Port 1:16 link UP at speed 10 Mbps and half-duplex
05/06/2014 14:48:52.90 Slot-1: Port 1:16 link down
05/06/2014 14:38:36.24 Slot-1: Port 1:16 link UP at speed 1 Gbps and full-duplex
05/06/2014 14:38:33.79 Slot-1: Port 1:16 link down
05/06/2014 14:38:31.71 Slot-1: Port 1:16 link UP at speed 10 Mbps and half-duplex

I need to find a solution for this, it cannot be normal.

I think the flapping is blocking the CPU, because sometimes the Ridgeline cannot get an SNMP response - I suspect CPU overload.

Thx, Martin

RE: X460 and X450 Flapping Ports

MUNIR_ENG — Tue, 06 May 2014 18:40:00 GMT

what is mean
switch#top .?
for what it using ?

RE: X460 and X450 Flapping Ports

Ronald_Dvorak — Tue, 06 May 2014 18:40:00 GMT

please refer to the "ExtremeXOS Command Reference Guide" to get information about the different commands...

# top

Description:
Displays real-time CPU utilization information by process.

RE: X460 and X450 Flapping Ports

Sumit_Tokle — Tue, 06 May 2014 22:57:00 GMT

How long you have seen this issue? Have you done any changes which could trigger this issue?

Does auto negotiation is on,on all the port which are connected to the end devices? if yes, then you might need to open case with TAC for further investigation.

RE: X460 and X450 Flapping Ports

Martin_Harrow — Wed, 07 May 2014 12:45:00 GMT

I was using AUTO on all ports. for the flapping ports, I have set the connection to fixed 10M/half.
I think I must open a case.
What are the best commands to check the switch?
You suggested TOP? any others?

RE: X460 and X450 Flapping Ports

Sumit_Tokle — Wed, 07 May 2014 12:53:00 GMT

Yes,

You can share the below information to TAC when you open case. It would help to get resolution as soon as possible.

Network topology diagram
show forwarding configuration
debug hal show forwarding distributions
debug hal show ipv4Mc
show iproute reserved-entries statistics
show tech
show port information detail
show port configuration
debug hal show optic-info
debug hal show cpu-queue

Thanks.

RE: X460 and X450 Flapping Ports

MUNIR_ENG — Wed, 07 May 2014 12:53:00 GMT

thank for this information , but when i doing this commands i got on many values ! how can i know which the better value?

RE: X460 and X450 Flapping Ports

Edson_Yuji — Wed, 07 May 2014 22:31:00 GMT

I had this same problem in Extreme switches. This happened when I fixed the port speed to 100Mbps Full duplex for example, workstation and IP station and printers were auto-negotiation happened flapping port.

RE: X460 and X450 Flapping Ports

Martin_Harrow — Thu, 08 May 2014 01:30:00 GMT

Hi Edson, Thanks for the input. Most of my problems occur with Canon printers, so as a work around I have been setting the speed to 10M fixed. I'm just annoyed that we have "upgraded" from 10 year old Cisco switches (which had no problems) to brand new Extreme switches that have these problems.

RE: X460 and X450 Flapping Ports

rbrt — Tue, 20 May 2014 17:32:00 GMT

Hi Martin,

I suspect that your old switches were Fast Ethernet switches? In this case you should check your passive infrastructure. I've seen issues with Gigabit in one of our offices as well, turned out that the - very old - cabling wasn't able to handle Gigabit Ethernet. If cabling turns out to be the issue you've basically got two options:

* Replace the cabling.
* configure ports auto on speed 100 duplex full

The first option can be quite budget intense, but is the better solution.

-Robert

RE: X460 and X450 Flapping Ports

Martin_Harrow — Mon, 26 May 2014 23:13:00 GMT

Robert, thanks for the feedback. I have tried this, it helps a bit. 😉

RE: X460 and X450 Flapping Ports

Boon_Nyian_Jap — Wed, 21 Jan 2015 16:44:00 GMT

hi martin, did you managed to resolve the issue? I having the same issue with x460 now

RE: X460 and X450 Flapping Ports

PARTHIBAN_CHINN — Wed, 21 Jan 2015 17:04:00 GMT

Hi Boon,
Are you seeing this problem in X460 XGM module 10g ports?

could you copy paste the link flap logs from x460 and its peer device?

RE: X460 and X450 Flapping Ports

Boon_Nyian_Jap — Tue, 27 Jan 2015 09:36:00 GMT

Hi there,

after we upgraded the switch firmware and applied debounce time 150 to stack ports, the stack ports flapping issue was resolved. However, I did observed some flapping for data ports. No sure if anyone noticed this issue.

RE: X460 and X450 Flapping Ports

PARTHIBAN_CHINN — Tue, 27 Jan 2015 12:16:00 GMT

It applies to data ports as well Please apply denounce timer settings Summit , bd8k and bdx8 10 g ports are impacted

RE: X460 and X450 Flapping Ports

Boon_Nyian_Jap — Tue, 27 Jan 2015 12:34:00 GMT

should we applied debounce 150 to all data ports too? Extreme support advise that it should be applied to port that are flapping only. is that right?

RE: X460 and X450 Flapping Ports

PARTHIBAN_CHINN — Tue, 27 Jan 2015 12:58:00 GMT

If the switch is connected to another 10 g port
(Switch) then yes. '
All 10g switch to switch connectivity ports must have debounce timer to 150 .

Note: 3rd party vendors have default debounce
Timer as 150

RE: X460 and X450 Flapping Ports

Stephane_Grosj1 — Tue, 27 Jan 2015 18:31:00 GMT

The latest release of EXOS (15.5.3) should have a fix that remove the debounce timer. Please check the RN.