https://community.extremenetworks.com/t5/end-of-service-products/meaning-of-watchdog-line-numbers/m-p/23324#M368 Have an interesting one here. Site with approximately 70 Extreme, along with a mix of Brocades and few Cisco. <BR /> <BR /> On some extremes out of the blue they started rebooting due to<BR /> <BR /> 01/13/2017 12:48:35.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/13/2017 11:55:33.98 <KERN> [1] System rebooted because watchdog timer expired! (0x80a66f2c)<BR /> 01/13/2017 11:48:36.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb350)<BR /> 01/13/2017 11:40:33.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb338)<BR /> 01/13/2017 11:10:55.98 <KERN> [1] System rebooted because watchdog timer expired! (0x8050a75c)<BR /> 01/13/2017 10:55:42.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/13/2017 10:48:46.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb29c)<BR /> 01/13/2017 10:40:35.98 <KERN> [1] System rebooted because watchdog timer expired! (0x80a80590)<BR /> <BR /> Logs show nothing decisive. For the sys log I am unsure configuring the default filters what to include. I do a show logs counters and the items incrementing are just your normal ones one would expect. Odd part also its not affecting all switches...most have cookie cutter configs, only items changed are hn, management ip, and snmp names. And its not affecting any distro switches, only access layer, which are all 200-48s running 7.8 <BR /> <BR /> So my question is what is the number at the end of the log lines above represent? I started to think memory addresses? <BR /> <BR /></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN> Sat, 14 Jan 2017 04:29:00 GMT Jimmy_Sands 2017-01-14T04:29:00Z https://community.extremenetworks.com/t5/end-of-service-products/meaning-of-watchdog-line-numbers/m-p/23324#M368 Have an interesting one here. Site with approximately 70 Extreme, along with a mix of Brocades and few Cisco. <BR /> <BR /> On some extremes out of the blue they started rebooting due to<BR /> <BR /> 01/13/2017 12:48:35.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/13/2017 11:55:33.98 <KERN> [1] System rebooted because watchdog timer expired! (0x80a66f2c)<BR /> 01/13/2017 11:48:36.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb350)<BR /> 01/13/2017 11:40:33.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb338)<BR /> 01/13/2017 11:10:55.98 <KERN> [1] System rebooted because watchdog timer expired! (0x8050a75c)<BR /> 01/13/2017 10:55:42.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/13/2017 10:48:46.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb29c)<BR /> 01/13/2017 10:40:35.98 <KERN> [1] System rebooted because watchdog timer expired! (0x80a80590)<BR /> <BR /> Logs show nothing decisive. For the sys log I am unsure configuring the default filters what to include. I do a show logs counters and the items incrementing are just your normal ones one would expect. Odd part also its not affecting all switches...most have cookie cutter configs, only items changed are hn, management ip, and snmp names. And its not affecting any distro switches, only access layer, which are all 200-48s running 7.8 <BR /> <BR /> So my question is what is the number at the end of the log lines above represent? I started to think memory addresses? <BR /> <BR /></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN> Sat, 14 Jan 2017 04:29:00 GMT https://community.extremenetworks.com/t5/end-of-service-products/meaning-of-watchdog-line-numbers/m-p/23324#M368 Jimmy_Sands 2017-01-14T04:29:00Z https://community.extremenetworks.com/t5/end-of-service-products/meaning-of-watchdog-line-numbers/m-p/23325#M369 Please find below in more details for the above subjected log messages:<BR /> <BR /> <A href="https://gtacknowledge.extremenetworks.com/articles/Q_A/error-Log-analytics/?q=Meaning+of+&amp;#60;Crit%3AKern.Alert&amp;#62;+MSM-A%3A+CPU+0%3A+soft+watchdog+expiration+warning+EPC+803f6e14%28_spin_un&amp;#38;l=en_US&amp;#38;fs=Search&amp;#38;pn=1" target="_blank" rel="nofollow noreferrer noopener">https://gtacknowledge.extremenetworks.com/articles/Q_A/error-Log-analytics/?q=Meaning+of+<CRIT><BR /> <BR /></CRIT></A> Sun, 15 Jan 2017 08:19:00 GMT https://community.extremenetworks.com/t5/end-of-service-products/meaning-of-watchdog-line-numbers/m-p/23325#M369 Tripathy__Priya 2017-01-15T08:19:00Z https://community.extremenetworks.com/t5/end-of-service-products/meaning-of-watchdog-line-numbers/m-p/23326#M370 Hi Jimmy,<BR /> <BR /> The HEX string at the end of the log line is the location of the last instruction executing when the switch watchdog rebooted.<BR /> There is no information on what instruction was being executed available in the switch logs.<BR /> <BR /> Sun, 15 Jan 2017 08:53:00 GMT https://community.extremenetworks.com/t5/end-of-service-products/meaning-of-watchdog-line-numbers/m-p/23326#M370 Ariyakudi_Srini 2017-01-15T08:53:00Z https://community.extremenetworks.com/t5/end-of-service-products/meaning-of-watchdog-line-numbers/m-p/23327#M371 So how to troubleshot a watchdog timer expiration? I saw in the log the process TNettask is causing the timer to expire. From what I know no hw or sw updates have been introduced. No image upgrades. No major configuration changes on the LAN rtr (x460) or other devices that I know of. Site has about 10 .ICX6430-24-HPOE,, 5 .ICX6610-24F, about 60 200-48s, and 3 a X450a-24x, and a 200-24fx,<BR /> <BR /> How do I figure out why the TnetTask is hogging the CPU? I tried a packet capture on a trunk port to see if I could see any network traffic causing the issue ; the capture was running (on a trunk port) as sw's were rebooting because of it, but I could not detect network traffic in the packet capture that was out of the ordinary <BR /> <BR /> We have about 20 out of 70 sw's rebooting while staff is onsite. So tell me user introduced, knowingly or not. Outside of the packet capture, what other tools are there to assist? I know the details I provided are sparse, I have another commitment to tend to, will update later tonight.<BR /> <BR /> 01/17/2017 12:47:47.98 <KERN> [1] System rebooted because watchdog timer expired! (0x80a93120)<BR /> 01/17/2017 12:39:27.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/17/2017 12:18:11.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/17/2017 11:48:18.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb274)<BR /> 01/17/2017 11:18:20.98 <KERN> [1] System rebooted because watchdog timer expired! (0x80a92d40)<BR /> 01/17/2017 10:48:25.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/17/2017 10:18:35.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/17/2017 09:48:17.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/17/2017 09:18:44.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb27c)<BR /> 01/17/2017 08:48:19.98 <KERN> [1] System rebooted because watchdog timer expired! (0x80a92a70)<BR /> 01/17/2017 08:18:15.98 <KERN> [1] System rebooted because watchdog timer expired! (0x80a66b78)<BR /> 01/17/2017 07:48:09.98 <KERN> [1] System rebooted because watchdog timer expired! (0x80a87160)<BR /> 01/17/2017 07:18:17.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/13/2017 13:18:13.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb278)<BR /> 01/13/2017 12:48:39.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/13/2017 11:55:27.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/13/2017 11:40:47.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb274)<BR /> 01/13/2017 11:25:31.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807545dc)<BR /> 01/13/2017 11:10:52.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/13/2017 10:55:31.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb274)<BR /> 01/13/2017 10:40:44.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/13/2017 10:25:18.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/13/2017 10:17:52.44 <SYST> task tNetTask cpu utilization is 87% PC: 80c317d4<BR /> 01/13/2017 10:10:36.98 <KERN> [1] System rebooted because watchdog timer expired! (0x80a67fb4)<BR /> 01/13/2017 09:55:19.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/13/2017 09:40:47.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/13/2017 09:25:52.98 <KERN> [1] System rebooted because watchdog timer expired! (0x8050aed0)<BR /> 01/13/2017 09:10:59.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/13/2017 08:55:32.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb274)<BR /> 01/13/2017 08:48:42.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb2ac)<BR /> 01/13/2017 08:40:39.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb284)<BR /> 01/13/2017 08:25:31.98 <KERN> [1] System rebooted because watchdog timer expired! (0x80752140)<BR /> 01/13/2017 08:10:46.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807d86f4)<BR /> 01/13/2017 07:55:31.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb284)<BR /> 01/13/2017 07:40:51.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/13/2017 07:25:28.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb29c)<BR /> 01/13/2017 07:18:43.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/13/2017 07:10:32.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807d0640)<BR /> 01/12/2017 13:06:46.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/12/2017 13:05:25.44 <SYST> task tNetTask cpu utilization is 88% PC: 80c317d4<BR /> 01/12/2017 12:55:07.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/12/2017 12:36:11.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/12/2017 12:25:18.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/12/2017 12:06:32.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/12/2017 11:55:20.98 <KERN> [1] System rebooted because watchdog timer expired! (0x80a02d18)<BR /> 01/12/2017 11:36:15.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/12/2017 11:25:45.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/12/2017 11:06:13.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb338)<BR /> 01/12/2017 10:55:19.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb338)<BR /> 01/12/2017 10:36:52.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/12/2017 10:35:32.44 <SYST> task tNetTask cpu utilization is 87% PC: 80c317d4<BR /> 01/12/2017 10:25:14.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb29c)<BR /> 01/12/2017 10:06:16.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb28c)<BR /> 01/12/2017 09:55:13.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/12/2017 09:36:51.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb27c)<BR /> 01/12/2017 09:35:29.44 <SYST> task tNetTask cpu utilization is 93% PC: 80c317d4<BR /> 01/12/2017 09:25:10.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/12/2017 09:06:26.98 <KERN> [1] System rebooted because watchdog timer expired! (0x806af67c)<BR /> 01/12/2017 08:54:57.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb274)<BR /> 01/12/2017 08:41:52.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb27c)<BR /> 01/12/2017 08:11:48.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb340)<BR /> 01/12/2017 07:41:50.98 <KERN> [1] System rebooted because watchdog timer expired! (0x80753684)<BR /> 01/12/2017 07:11:03.44 <SYST> task tNetTask cpu utilization is 88% PC: 80c317d4<BR /> 01/11/2017 13:20:31.98 <KERN> [1] System rebooted because watchdog timer expired! (0x80a67e30)<BR /> 01/11/2017 13:07:07.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/11/2017 12:50:52.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> 01/11/2017 12:37:04.98 <KERN> [1] System rebooted because watchdog timer expired! (0x807cb294)<BR /> <BR /></KERN></KERN></KERN></KERN></SYST></KERN></KERN></KERN></KERN></KERN></KERN></SYST></KERN></KERN></KERN></KERN></SYST></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN></SYST></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN></SYST></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN></KERN> Wed, 18 Jan 2017 13:45:00 GMT https://community.extremenetworks.com/t5/end-of-service-products/meaning-of-watchdog-line-numbers/m-p/23327#M371 Jimmy_Sands 2017-01-18T13:45:00Z