ERPS subring problem.

  • 0
  • 1
  • Problem
  • Updated 2 years ago
  • Solved
  • (Edited)
Hi Extreme Community.We have some experience with EAPS, now trying to make some lab test with ERPS and various vendors to push ERPS in to production.

Currently we are observing following problem:
When main ring broken, subring RPL owner does not unblocks RPL port.




Here is our lab.

SW1 config:
#VLANS
create vlan "MNG"
configure vlan MNG tag 9
configure vlan MNG add ports 1-3 tagged
configure vlan MNG ipaddress 10.10.10.111 255.255.255.0
create vlan "control_main"configure vlan control_main tag 305
configure vlan control_main add ports 2-3 tagged
configure vlan control_sub tag 304configure vlan control_sub add ports 1-3 tagged

#ERPS MAIN RING
create erps test
configure erps test subring-mode virtualChannel
configure erps test add control vlan control_main
configure erps test ring-port east 3
configure erps test ring-port west 2
configure erps test protection-port 2
configure erps test revert enable
configure erps test timer wait-to-restore 60000
configure erps test add protected vlan MNG
configure erps test add protected vlan control_sub
configure erps test add sub-ring test-sub
enable erps test

#ERPS SUB-RING
create erps test-sub
configure erps test-sub subring-mode virtualChannel
configure erps test-sub add control vlan control_sub
configure erps test-sub ring-port east 1
configure erps test-sub revert enable
configure erps test-sub timer wait-to-restore 60000
configure erps test-sub add protected vlan MNG

#SW2 config:

#MST configuration (protected vlans, mst disabled)
spanning-tree mst configuration
 instance 0 vlan 1-8;10-303;305-4094
 instance 1 vlan 9;304

#ERPS
erps-ring test
 open-ring
 raps-virtual-channel with
 erps-instance 1
  wtr-timer 1
  raps-mel 0
  rpl port1 owner
  protected-instance 1
  control-vlan 304

#PORTS
Interface Ethernet1/0/25
 switchport mode trunk
 erps-ring test port0
!
Interface Ethernet1/0/26
 switchport mode trunk
 erps-ring test port1



SW3 config:
#VLANS
create vlan "MNG"configure vlan MNG tag 9
configure vlan MNG add ports 31-33 tagged
configure vlan MNG ipaddress 10.10.10.114 255.255.255.0
create vlan "control_main"
configure vlan control_main tag 305
create vlan "control_sub"
configure vlan control_sub tag 304
configure vlan control_main add ports 32-33 tagged
configure vlan control_sub add ports 31-33 tagged

#ERPS MAIN RING
create erps testconfigure erps test subring-mode virtualChannel
configure erps test add control vlan control_main
configure erps test ring-port east 32
configure erps test ring-port west 33
configure erps test revert enable
configure erps test neighbor-port 32
configure erps test timer wait-to-restore 60000
configure erps test add protected vlan MNG
configure erps test add protected vlan control_sub
configure erps test add sub-ring test-sub
enable erps test

#ERPS SUB-RING
create erps test-sub
configure erps test-sub subring-mode virtualChannel
configure erps test-sub add control vlan control_sub
configure erps test-sub ring-port east 31
configure erps test-sub revert enable
configure erps test-sub neighbor-port 31
configure erps test-sub timer wait-to-restore 60000
configure erps test-sub add protected vlan MNG
enable erps test-sub



Ok, all configs are done, let's see what is going on:

#SW1 main ERPS:
 sw1# sh erps test
  Name: test
  Operational State: Idle        Node Type: RPL Owner,  Revertive
  Configured State : Enabled

  East Ring Port :  3   MepId: ---- Remote MepId: ----  Status: Unblocked
  West Ring Port : +2   MepId: ---- Remote MepId: ----  Status: Blocked

  Periodic timer interval:  5000 millisec (Enabled)
  Hold-off timer interval:  0    millisec (Enabled)
  Guard timer interval   :  500  millisec (Enabled)
  WTB timer interval     :  5500 millisec (Enabled)
  WTR timer interval     :  60000 millisec (Enabled)

  Ring MD Level          :  ----
  Notify Topology Change :  -------
  Subring Mode           :  Virtual Channel

  ERPS Control Vlan: control_main  VID:305
  Topology Change Propogation List: None
  Topology Change Propogation  : Disabled
  ERPS Major ring has following Sub-Ring(s):
    Sub-Ring Name        :  test-sub
  ERPS Ring has following Protected Vlan(s):
    Vlan Name                    VID
    MNG                          9
    control_sub                  304
  Number of Protected Vlans: 2
  (+) RPL Protection Port, (^) RPL Neighbor Port
  (f) Force Switch Port, (m) Manual Switch Port

#SW1 subring erps:
 sh erps "test-sub"
  Name: test-sub
  Operational State: Idle        Node Type: InterConnection Node
  Configured State : Enabled

  East Ring Port : 1   MepId: ---- Remote MepId: ----   Status: Unblocked
  West Ring Port : ----- MepId: ---- Remote MepId: ---- Status: ------

  Periodic timer interval:  5000 millisec (Enabled)
  Hold-off timer interval:  0    millisec (Enabled)
  Guard timer interval   :  500  millisec (Enabled)
  WTB timer interval     :  5500 millisec (Enabled)
  WTR timer interval     :  60000 millisec (Enabled)

  Ring MD Level          :  ----
  Notify Topology Change :  -------
  Subring Mode           :  Virtual Channel

  ERPS Control Vlan: control_sub   VID:304
  Topology Change Propogation List: None
  Topology Change Propogation  : Disabled
  ERPS Ring's Main-Ring  : test
  ERPS Ring has following Protected Vlan(s):
    Vlan Name                    VID
    MNG                          9
  Number of Protected Vlans: 1
  (+) RPL Protection Port, (^) RPL Neighbor Port
  (f) Force Switch Port, (m) Manual Switch Port


#On sw2:
#sh erps instance
ERPS Ring: test
  Instance: 1
  Description: -
  Protected Instance: 1
  Revertive mode: revertive
  R-APS MEL: 0
  R-APS Virtual-Channel: with
  Control Vlan: 304
  Ring ID: 1
  Guard Timer(10ms): 50
  Holdoff Timer(seconds): 0
  WTR Timer(min): 1
  -------------------------------------------
  Port      Role             Port-Status
  -------------------------------------------
  Port0     common           forwarding
  Port1     RPL owner        blocked

sh erps ring test
R: RPL Owner
N: RPL Neighbour
C: Common Node
-----------------------------------------
Version: ITU-T G.8032v2
R-APS ring topology: open-ring
R-APS Virtual-Channel: with
Port0: Ethernet1/0/25
Failure-detect type: physical-link
Port1: Ethernet1/0/26
Failure-detect type: physical-link
--------------------------------------------------------------------------------------------------------------------------------
Instance  Contral  Protected                                               WTR_Timer  Guard_Timer  Holdoff_Timer
ID        Vlan     Instance                                                (min)      (10ms)       (second)       Port0  Port1
--------------------------------------------------------------------------------------------------------------------------------
1         304      1                                                       1          50           0              C      R


#on SW3:
#MAIN ERPS:
sh erps test
  Name: test
  Operational State: Idle        Node Type: RPL Neighbor
  Configured State : Enabled

  East Ring Port : ^32  MepId: ---- Remote MepId: ----  Status: Blocked
  West Ring Port :  33  MepId: ---- Remote MepId: ----  Status: Unblocked

  Periodic timer interval:  5000 millisec (Enabled)
  Hold-off timer interval:  0    millisec (Enabled)
  Guard timer interval   :  500  millisec (Enabled)
  WTB timer interval     :  5500 millisec (Enabled)
  WTR timer interval     :  60000 millisec (Enabled)

  Ring MD Level          :  ----
  Notify Topology Change :  -------
  Subring Mode           :  Virtual Channel

  ERPS Control Vlan: control_main  VID:305
  Topology Change Propogation List: None
  Topology Change Propogation  : Disabled
  ERPS Major ring has following Sub-Ring(s):
    Sub-Ring Name        :  test-sub
  ERPS Ring has following Protected Vlan(s):
    Vlan Name                    VID
    MNG                          9
    control_sub                  304
  Number of Protected Vlans: 2
  (+) RPL Protection Port, (^) RPL Neighbor Port
  (f) Force Switch Port, (m) Manual Switch Port

#subring ERPS
 sh erps "test-sub"
  Name: test-sub
  Operational State: Idle        Node Type: RPL Neighbor
  Configured State : Enabled

  East Ring Port : ^31  MepId: ---- Remote MepId: ----  Status: Blocked
  West Ring Port : ----- MepId: ---- Remote MepId: ---- Status: ------

  Periodic timer interval:  5000 millisec (Enabled)
  Hold-off timer interval:  0    millisec (Enabled)
  Guard timer interval   :  500  millisec (Enabled)
  WTB timer interval     :  5500 millisec (Enabled)
  WTR timer interval     :  60000 millisec (Enabled)

  Ring MD Level          :  ----
  Notify Topology Change :  -------
  Subring Mode           :  Virtual Channel

  ERPS Control Vlan: control_sub   VID:304
  Topology Change Propogation List: None
  Topology Change Propogation  : Disabled
  ERPS Ring's Main-Ring  : test
  ERPS Ring has following Protected Vlan(s):
    Vlan Name                    VID
    MNG                          9
  Number of Protected Vlans: 1
  (+) RPL Protection Port, (^) RPL Neighbor Port
  (f) Force Switch Port, (m) Manual Switch Port


And now, we got a problem. When both links in main ERPS ring fails, RPL owner in subring does not opens RPL link.

#SW1
 sh erps "test-sub"
  Name: test-sub
  Operational State: Idle        Node Type: InterConnection Node
  Configured State : Enabled

  East Ring Port : 1   MepId: ---- Remote MepId: ----   Status: Unblocked
  West Ring Port : ----- MepId: ---- Remote MepId: ---- Status: ------

  Periodic timer interval:  5000 millisec (Enabled)
  Hold-off timer interval:  0    millisec (Enabled)
  Guard timer interval   :  500  millisec (Enabled)
  WTB timer interval     :  5500 millisec (Enabled)
  WTR timer interval     :  60000 millisec (Enabled)

  Ring MD Level          :  ----
  Notify Topology Change :  -------
  Subring Mode           :  Virtual Channel

  ERPS Control Vlan: control_sub   VID:304
  Topology Change Propogation List: None
  Topology Change Propogation  : Disabled
  ERPS Ring's Main-Ring  : test
  ERPS Ring has following Protected Vlan(s):
    Vlan Name                    VID
    MNG                          9
  Number of Protected Vlans: 1
  (+) RPL Protection Port, (^) RPL Neighbor Port
  (f) Force Switch Port, (m) Manual Switch Port

#SW2
sh erps instanceERPS Ring: test
  Instance: 1
  Description: -
  Protected Instance: 1
  Revertive mode: revertive
  R-APS MEL: 0
  R-APS Virtual-Channel: with
  Control Vlan: 304
  Ring ID: 1
  Guard Timer(10ms): 50
  Holdoff Timer(seconds): 0
  WTR Timer(min): 1
  -------------------------------------------
  Port      Role             Port-Status
  -------------------------------------------
  Port0     common           forwarding
  Port1     RPL owner        blocked


#SW3
sh erps "test-sub"
  Name: test-sub
  Operational State: Idle        Node Type: RPL Neighbor
  Configured State : Enabled

  East Ring Port : ^31  MepId: ---- Remote MepId: ----  Status: Blocked
  West Ring Port : ----- MepId: ---- Remote MepId: ---- Status: ------

  Periodic timer interval:  5000 millisec (Enabled)
  Hold-off timer interval:  0    millisec (Enabled)
  Guard timer interval   :  500  millisec (Enabled)
  WTB timer interval     :  5500 millisec (Enabled)
  WTR timer interval     :  60000 millisec (Enabled)

  Ring MD Level          :  ----
  Notify Topology Change :  -------
  Subring Mode           :  Virtual Channel

  ERPS Control Vlan: control_sub   VID:304
  Topology Change Propogation List: None
  Topology Change Propogation  : Disabled
  ERPS Ring's Main-Ring  : test
  ERPS Ring has following Protected Vlan(s):
    Vlan Name                    VID
    MNG                          9
  Number of Protected Vlans: 1
  (+) RPL Protection Port, (^) RPL Neighbor Port
  (f) Force Switch Port, (m) Manual Switch Port


Any help will be highly appreciated.
Photo of Nikolay Chernyaev

Nikolay Chernyaev

  • 180 Points 100 badge 2x thumb

Posted 3 years ago

  • 0
  • 1
Photo of Nikolay Chernyaev

Nikolay Chernyaev

  • 180 Points 100 badge 2x thumb
Config is similar to:
https://gtacknowledge.extremenetworks.com/articles/How_To/How-to-configure-a-ERPS-ring-with-a-sub-ring-attached/

We have also tested this config in  UNetLab with extreme summit switches only, with same simptoms.
(Edited)
Photo of Anton Nekhoroshikh

Anton Nekhoroshikh

  • 192 Points 100 badge 2x thumb
Anybody has no ideas?!
Photo of Daniel Flouret

Daniel Flouret, Employee

  • 7,470 Points 5k badge 2x thumb
Nikolai, Anton,

From the EXOS User Guide v16.1:
G.8032 Version 2 (pg. 1086)
The concept of sub-rings is introduced to add multiple rings to the main ring. A sub-ring is an incomplete ring that completes its path through the main ring or other sub-rings. The control path for the sub-ring completes either through the implementation of a virtual channel, or by changing the flow of control packets in the sub-rings. Virtual channels are supported through the use of the sub-rings control channel being configured as a data VLAN in the main ring.


Multiple Failure Protection using CFM (pg. 1087)
You can use CFM UP-MEP support in the sub-ring to provide protection against multiple failures in the main ring. Configure an UP-MEP on the interconnected nodes, where a segmentation of the main ring results in the UP-MEP notifying the sub-ring of a failure. This causes the sub-ring to open its RPL and place the interconnected node in manual switch. This is done to avoid a super-loop during recovery of the main ring. This implementation is as directed in Appendix X.3 of the G.8032 standard.

Let me "translate" this.

When a failure occurs in an ERPS ring, the nodes that own the ERPS ports involved in the failure will send CFM R-‐APS messages to inform of the failure, which makes all nodes in the ring to flush their tables and the owner of the RPL port to close the RPL port. In your diagram, Nikolai, if the left link between SW1 and SW3 failed SW3 would inform of the failure through the main ring's control VLAN. Upon receiving the notification, SW1 would  close the RPL port and reestablish connectivity to SW3.

What happens to the sub-ring when the main ring fails? Nothing. Since the failure does not involve any of its ports or links, no message is transmitted through the sub-ring's control vlan and no action is taken by the sub-ring nodes. The main ring "heals" itself and maintains the connectivity of SW2 with SW3, as indicated in the first paragraph quoted from the User Guide.

Similarly, if the link between SW1 and SW2 failed, SW1would inform SW2 of the failure through the sub-ring's control VLAN (which happens to go through the main ring). Upon receiving the notification, SW2 would  close the RPL port and reestablish connectivity to SW1 and SW3.

Now, the problem comes when we have MULTIPLE failures in the main ring, making it unable to maintain the connectivity between the nodes of the sub-ring. The sub-ring will not detect the failures, as explained previously, because they don't involve ports belonging to the sub-ring. That is why you need to configure CFM, as indicated in the second paragraph quoted from the User Guide.

Now, if you're familiar with EAPS, you might wonder why you need to configure CFM. Well, EAPS includes a mechanism where the Master node send a "health check" message through the control vlan to check that there's connectivity in the ring. This is slow but, in case a failure goes undetected by the other nodes in the ring, the Master will eventually heal the ring. In ERPS the RPL owner doesn't do that. ERPS relies on the configuration of CFM Connectivity Check Messages (CCM) to detect failures not reported by the ring nodes.

To make your configuration work you will need to configure CFM Maintenance End Points (MEPs). There's an sample configuration in the EXOS User Guide v16.1(page 1094), which is almost identical to yours, except that it has 4 switches, instead of 3.

Here's a How-To Guide that explains the basics of CFM R-APS messaging and error detection and notification in a single ERPS ring:
https://www.dropbox.com/s/l5m3fuirlsbmrru/SVC%20Tech%20MB%20Tech%20Guide%20-%20ERPSv1%20%28G.8032v1%...
Photo of Nikolay Chernyaev

Nikolay Chernyaev

  • 180 Points 100 badge 2x thumb
Thank you Daniel, yours answer is very helpfull.
If you have and sample config for UP meps similar to our config, it will be great to see them here.
(Edited)
Photo of Daniel Flouret

Daniel Flouret, Employee

  • 7,470 Points 5k badge 2x thumb
Nikolai,

I've just done a "quick and dirty" config using EXOS-VM switches, and here's my config:

# Configuration for SW 1
# CFM Up MEP Configuration for ERPS sub-ring
create cfm domain string sr1 md-level 3
configure cfm domain sr1 add association string MDlevel1 vlan cfmVlan
configure cfm domain sr1 association MDlevel1 ports 1 add end-point up 100
configure cfm domain sr1 association MDlevel1 ports 1 end-point up add group S2
configure cfm group S2 add rmep 200
configure erps sub_ring cfm protection primary add group S2

# Configuration for SW 2
# CFM Up MEP configuration for ERPS sub-ring
create cfm domain string sr1 md-level 3
configure cfm domain sr1 add association string MDlevel1 vlan cfmVlan
configure cfm domain sr1 association MDlevel1 ports 31 add end-point up 200
configure cfm domain sr1 association MDlevel1 ports 31 end-point up add group S1
configure cfm group S1 add rmep 100
configure erps sub_ring cfm protection primary add group S1

cfmVlan is the vlan through which the CFM messages will be sent. This vlan CAN'T BE a ring control vlan.

The following log messages were generated in SW1 when both links between SW1 and SW2 failed:

12/01/2015 21:15:46.85 <Info:ERPS.RingStateChg> : ERPS ring sub_ring state changed from old state Idle to new state Protection
12/01/2015 21:15:46.85 <Warn:CFM.rMEPExp> : Remote End-Point 200/00:01:30:DF:05:02 expired on up end-point, domain "sr1", ma "MDlevel1", port 3
12/01/2015 21:15:44.15 <Info:vlan.msgs.portLinkStateDown> : Port 2 link down
12/01/2015 21:15:42.16 <Info:vlan.msgs.portLinkStateDown> : Port 1 link down
12/01/2015 21:15:42.06 <Info:ERPS.RingStateChg> : ERPS ring main_ring state changed from old state Idle to new state Protection

These appeared in SW2:

12/01/2015 21:15:45.00 <Warn:CFM.MEPExp> : Maintenance End-Point 200 Error Not recv CCM from RMEP occurred on up end-point, domain "sr1", ma "MDlevel1", port 3
12/01/2015 21:15:42.50 <Info:ERPS.RingStateChg> : ERPS ring sub_ring state changed from old state Idle to new state Protection
12/01/2015 21:15:42.50 <Warn:CFM.rMEPExp> : Remote End-Point 100/00:01:30:DF:05:01 expired on up end-point, domain "sr1", ma "MDlevel1", port 3
12/01/2015 21:15:39.45 <Info:vlan.msgs.portLinkStateDown> : Port 2 link down
12/01/2015 21:15:37.43 <Info:ERPS.RingStateChg> : ERPS ring main_ring state changed from old state Idle to new state Protection
12/01/2015 21:15:37.43 <Info:vlan.msgs.portLinkStateDown> : Port 1 link down

And these in SW3:

12/01/2015 21:15:09.35 <Info:ERPS.RingStateChg> : ERPS ring sub_ring state changed from old state Idle to new state Protection

Photo of Anton Nekhoroshikh

Anton Nekhoroshikh

  • 192 Points 100 badge 2x thumb
Daniel, below the same config don't work, may be you have other version ExtremeXOS? our version is 15.4.1.3 v1541b3-patch1-13
Something can another?
Photo of Nikolay Chernyaev

Nikolay Chernyaev

  • 180 Points 100 badge 2x thumb
We have updated to recomended verison few days ago - ExtremeXOS version 15.6.3.1 v1563b1-patch1-5, with same result.All output posted in this thread is from 15.6.3.1.
(Edited)
Photo of Nikolay Chernyaev

Nikolay Chernyaev

  • 180 Points 100 badge 2x thumb
I have reviewed RFC appendix and configured UP meps on subring ports on interconnected nodes.
Now, SW2 get signalled when multiply failure on main ring occurs, and opens RPL, but ports on interconnected nodes facing sub-ring goes to blocked state.

added configuration:

#SW1 VLAN
create vlan "subring-mep"
configure vlan subring-mep tag 306
configure vlan subring-mep add ports 1-3 tagged

#SW1 CFM:
create cfm domain string "test" md-level 6configure cfm domain "test" add association string "level6" vlan "subring-mep"
configure cfm domain "test" association "level6" ports 1 add end-point up 601
configure cfm domain "test" association "level6" ports 1 end-point up add group "sw1-sw3"
configure cfm group "sw1-sw3" add rmep 501
#SW1 ERPS
configure erps test-sub cfm protection primary add group sw1-sw3
configure erps test add protected vlan subring-mep

#SW3 VLAN
create vlan "subring-mep"
configure vlan subring-mep tag 306
configure vlan subring-mep add ports 31-33 tagged

#SW3 CFM:
create cfm domain string "test" md-level 6configure cfm domain "test" add association string "level6" vlan "subring-mep"
configure cfm domain "test" association "level6" ports 31 add end-point up 501
configure cfm domain "test" association "level6" ports 31 end-point up add group "sw3-sw1"
configure cfm group "sw3-sw1" add rmep 601

#SW3 ERPS
configure erps test-sub cfm protection secondary add group sw3-sw1
configure erps test add protected vlan subring-mep


STATES:

#SW1 CFM state when all ports are enabled:
sh cfmDomain: "test", MD Level: 6
  Association: "level6", Destination MAC Type: Multicast, VLAN "subring-mep" with 3 cfm ports
  Transmit Interval: 1000 ms, Type : IEEE 802.1ag Maintenance Association
    port 1; Up End Point,  mepid: 601, transmit-interval: 1000 ms (from association)
         MEP State: Enabled, CCM Message: Enabled, Send SenderId TLV: Disabled

         Faulting State              : No
         Last Faulting State Change  : Wed Dec  2 19:21:28 2015

         MEP Error Defects           : None
         Port Status                 : Up    port 2; Intermediate Point ( Dynamic )
    port 3; Intermediate Point ( Dynamic )

Total Number of Domain             : 1
Total Number of Association        : 1
Total Number of Up MEP             : 1
Total Number of Down MEP           : 0
Total Number of MIP                : 2
Total Number of CFM port           : 3
Total Number of SW MEP             : 1
Total Number of HW MEP             : 0
Total Number of VPLS MIP(Static/Up): 0 / 0

================================================================================
MEP Error Defect Types:
   Remote : Not receiving CCMs from Remote MEP
   Error  : Erroneous CCM received
   XCON   : Cross-connect CCM received
   RDI    : Remote Defect Indication sent by some MEP

#SW1 MAIN ring:
sh erps test
  Name: test
  Operational State: Idle        Node Type: RPL Owner,  Revertive
  Configured State : Enabled

  East Ring Port :  3   MepId: ---- Remote MepId: ----  Status: Unblocked
  West Ring Port : +2   MepId: ---- Remote MepId: ----  Status: Blocked

  Periodic timer interval:  5000 millisec (Enabled)
  Hold-off timer interval:  0    millisec (Enabled)
  Guard timer interval   :  500  millisec (Enabled)
  WTB timer interval     :  5500 millisec (Enabled)
  WTR timer interval     :  60000 millisec (Enabled)

  Ring MD Level          :  ----
  Notify Topology Change :  -------
  Subring Mode           :  Virtual Channel

  ERPS Control Vlan: control_main  VID:305
  Topology Change Propogation List: None
  Topology Change Propogation  : Disabled
  ERPS Major ring has following Sub-Ring(s):
    Sub-Ring Name        :  test-sub
  ERPS Ring has following Protected Vlan(s):
    Vlan Name                    VID
    MNG                          9
    control_sub                  304
    subring-mep                  306
  Number of Protected Vlans: 3
  (+) RPL Protection Port, (^) RPL Neighbor Port
  (f) Force Switch Port, (m) Manual Switch Port


#SW1 ERPS SUB-RING
sh erps "test-sub"
  Name: test-sub
  Operational State: Idle        Node Type: InterConnection Node
  Configured State : Enabled

  East Ring Port : 1   MepId: ---- Remote MepId: ----   Status: Unblocked
  West Ring Port : ----- MepId: ---- Remote MepId: ---- Status: ------

  Subring Protection Group: sw1-sw3

  Periodic timer interval:  5000 millisec (Enabled)
  Hold-off timer interval:  0    millisec (Enabled)
  Guard timer interval   :  500  millisec (Enabled)
  WTB timer interval     :  5500 millisec (Enabled)
  WTR timer interval     :  60000 millisec (Enabled)

  Ring MD Level          :  ----
  Notify Topology Change :  -------
  Subring Mode           :  Virtual Channel

  ERPS Control Vlan: control_sub   VID:304
  Topology Change Propogation List: None
  Topology Change Propogation  : Disabled
  ERPS Ring's Main-Ring  : test
  ERPS Ring has following Protected Vlan(s):
    Vlan Name                    VID
    MNG                          9
  Number of Protected Vlans: 1
  (+) RPL Protection Port, (^) RPL Neighbor Port
  (f) Force Switch Port, (m) Manual Switch Port


#SW2 ERPS
sh erps instanceERPS Ring: test
  Instance: 1
  Description: -
  Protected Instance: 1
  Revertive mode: revertive
  R-APS MEL: 0
  R-APS Virtual-Channel: with
  Control Vlan: 304
  Ring ID: 1
  Guard Timer(10ms): 50
  Holdoff Timer(seconds): 0
  WTR Timer(min): 1
  -------------------------------------------
  Port      Role             Port-Status
  -------------------------------------------
  Port0     common           forwarding
  Port1     RPL owner        blocked


#SW3 CFM
sh cfmDomain: "test", MD Level: 6
  Association: "level6", Destination MAC Type: Multicast, VLAN "subring-mep" with 3 cfm ports
  Transmit Interval: 1000 ms, Type : IEEE 802.1ag Maintenance Association
    port 31; Up End Point,  mepid: 501, transmit-interval: 1000 ms (from association)
         MEP State: Enabled, CCM Message: Enabled, Send SenderId TLV: Disabled

         Faulting State              : No
         Last Faulting State Change  : Wed Dec  2 16:20:51 2015

         MEP Error Defects           : None
         Port Status                 : Up    port 32; Intermediate Point ( Dynamic )
    port 33; Intermediate Point ( Dynamic )

Total Number of Domain             : 1
Total Number of Association        : 1
Total Number of Up MEP             : 1
Total Number of Down MEP           : 0
Total Number of MIP                : 2
Total Number of CFM port           : 3
Total Number of SW MEP             : 1
Total Number of HW MEP             : 0
Total Number of VPLS MIP(Static/Up): 0 / 0

================================================================================
MEP Error Defect Types:
   Remote : Not receiving CCMs from Remote MEP
   Error  : Erroneous CCM received
   XCON   : Cross-connect CCM received
   RDI    : Remote Defect Indication sent by some MEP


#SW3 ERPS MAIN
sh erps test
  Name: test
  Operational State: Idle        Node Type: RPL Neighbor
  Configured State : Enabled

  East Ring Port : ^32  MepId: ---- Remote MepId: ----  Status: Blocked
  West Ring Port :  33  MepId: ---- Remote MepId: ----  Status: Unblocked

  Periodic timer interval:  5000 millisec (Enabled)
  Hold-off timer interval:  0    millisec (Enabled)
  Guard timer interval   :  500  millisec (Enabled)
  WTB timer interval     :  5500 millisec (Enabled)
  WTR timer interval     :  60000 millisec (Enabled)

  Ring MD Level          :  ----
  Notify Topology Change :  -------
  Subring Mode           :  Virtual Channel

  ERPS Control Vlan: control_main  VID:305
  Topology Change Propogation List: None
  Topology Change Propogation  : Disabled
  ERPS Major ring has following Sub-Ring(s):
    Sub-Ring Name        :  test-sub
  ERPS Ring has following Protected Vlan(s):
    Vlan Name                    VID
    MNG                          9
    control_sub                  304
    subring-mep                  306
  Number of Protected Vlans: 3
  (+) RPL Protection Port, (^) RPL Neighbor Port
  (f) Force Switch Port, (m) Manual Switch Port


#SW3 ERPS SUB-RING
sh erps "test-sub"
  Name: test-sub
  Operational State: Idle        Node Type: RPL Neighbor
  Configured State : Enabled

  East Ring Port : ^31  MepId: ---- Remote MepId: ----  Status: Blocked
  West Ring Port : ----- MepId: ---- Remote MepId: ---- Status: ------

  Subring Protection Group: sw3-sw1

  Periodic timer interval:  5000 millisec (Enabled)
  Hold-off timer interval:  0    millisec (Enabled)
  Guard timer interval   :  500  millisec (Enabled)
  WTB timer interval     :  5500 millisec (Enabled)
  WTR timer interval     :  60000 millisec (Enabled)

  Ring MD Level          :  ----
  Notify Topology Change :  -------
  Subring Mode           :  Virtual Channel

  ERPS Control Vlan: control_sub   VID:304
  Topology Change Propogation List: None
  Topology Change Propogation  : Disabled
  ERPS Ring's Main-Ring  : test
  ERPS Ring has following Protected Vlan(s):
    Vlan Name                    VID
    MNG                          9
  Number of Protected Vlans: 1
  (+) RPL Protection Port, (^) RPL Neighbor Port
  (f) Force Switch Port, (m) Manual Switch Port



##############################
Now, if we disable both ports in main ring, we can see following:

#SW1 CFM
sh cfmDomain: "test", MD Level: 6
  Association: "level6", Destination MAC Type: Multicast, VLAN "subring-mep" with 3 cfm ports
  Transmit Interval: 1000 ms, Type : IEEE 802.1ag Maintenance Association
    port 1; Up End Point,  mepid: 601, transmit-interval: 1000 ms (from association)
         MEP State: Enabled, CCM Message: Enabled, Send SenderId TLV: Disabled

         Faulting State              : Yes
         Last Faulting State Change  : Wed Dec  2 19:25:52 2015

         MEP Error Defects           : Remote
         Port Status                 : Up    port 2; Intermediate Point ( Dynamic )
    port 3; Intermediate Point ( Dynamic )

Total Number of Domain             : 1
Total Number of Association        : 1
Total Number of Up MEP             : 1
Total Number of Down MEP           : 0
Total Number of MIP                : 2
Total Number of CFM port           : 3
Total Number of SW MEP             : 1
Total Number of HW MEP             : 0
Total Number of VPLS MIP(Static/Up): 0 / 0

================================================================================
MEP Error Defect Types:
   Remote : Not receiving CCMs from Remote MEP
   Error  : Erroneous CCM received
   XCON   : Cross-connect CCM received
   RDI    : Remote Defect Indication sent by some MEP


#SW1 ERPS MAIN
sh erps test
  Name: test
  Operational State: Protection        Node Type: RPL Owner,  Revertive
  Configured State : Enabled

  East Ring Port :  3   MepId: ---- Remote MepId: ----  Status: Blocked
  West Ring Port : +2   MepId: ---- Remote MepId: ----  Status: Blocked

  Periodic timer interval:  5000 millisec (Enabled)
  Hold-off timer interval:  0    millisec (Enabled)
  Guard timer interval   :  500  millisec (Enabled)
  WTB timer interval     :  5500 millisec (Enabled)
  WTR timer interval     :  60000 millisec (Enabled)

  Ring MD Level          :  ----
  Notify Topology Change :  -------
  Subring Mode           :  Virtual Channel

  ERPS Control Vlan: control_main  VID:305
  Topology Change Propogation List: None
  Topology Change Propogation  : Disabled
  ERPS Major ring has following Sub-Ring(s):
    Sub-Ring Name        :  test-sub
  ERPS Ring has following Protected Vlan(s):
    Vlan Name                    VID
    MNG                          9
    control_sub                  304
    subring-mep                  306
  Number of Protected Vlans: 3
  (+) RPL Protection Port, (^) RPL Neighbor Port
  (f) Force Switch Port, (m) Manual Switch Port


#SW1 ERPS SUB-RING
sh erps "test-sub"
  Name: test-sub
  Operational State: Protection        Node Type: InterConnection Node
  Configured State : Enabled

  East Ring Port : 1   MepId: ---- Remote MepId: ----   Status: Blocked
  West Ring Port : ----- MepId: ---- Remote MepId: ---- Status: ------

  Subring Protection Group: sw1-sw3

  Periodic timer interval:  5000 millisec (Enabled)
  Hold-off timer interval:  0    millisec (Enabled)
  Guard timer interval   :  500  millisec (Enabled)
  WTB timer interval     :  5500 millisec (Enabled)
  WTR timer interval     :  60000 millisec (Enabled)

  Ring MD Level          :  ----
  Notify Topology Change :  -------
  Subring Mode           :  Virtual Channel

  ERPS Control Vlan: control_sub   VID:304
  Topology Change Propogation List: None
  Topology Change Propogation  : Disabled
  ERPS Ring's Main-Ring  : test
  ERPS Ring has following Protected Vlan(s):
    Vlan Name                    VID
    MNG                          9
  Number of Protected Vlans: 1
  (+) RPL Protection Port, (^) RPL Neighbor Port
  (f) Force Switch Port, (m) Manual Switch Port


#SW2 
sh erps instanceERPS Ring: test
  Instance: 1
  Description: -
  Protected Instance: 1
  Revertive mode: revertive
  R-APS MEL: 0
  R-APS Virtual-Channel: with
  Control Vlan: 304
  Ring ID: 1
  Guard Timer(10ms): 50
  Holdoff Timer(seconds): 0
  WTR Timer(min): 1
  -------------------------------------------
  Port      Role             Port-Status
  -------------------------------------------
  Port0     common           forwarding
  Port1     RPL owner        forwarding


#SW3 CFM
sh cfmDomain: "test", MD Level: 6
  Association: "level6", Destination MAC Type: Multicast, VLAN "subring-mep" with 3 cfm ports
  Transmit Interval: 1000 ms, Type : IEEE 802.1ag Maintenance Association
    port 31; Up End Point,  mepid: 501, transmit-interval: 1000 ms (from association)
         MEP State: Enabled, CCM Message: Enabled, Send SenderId TLV: Disabled

         Faulting State              : Yes
         Last Faulting State Change  : Wed Dec  2 16:25:14 2015

         MEP Error Defects           : Remote
         Port Status                 : Up    port 32; Intermediate Point ( Dynamic )
    port 33; Intermediate Point ( Dynamic )

Total Number of Domain             : 1
Total Number of Association        : 1
Total Number of Up MEP             : 1
Total Number of Down MEP           : 0
Total Number of MIP                : 2
Total Number of CFM port           : 3
Total Number of SW MEP             : 1
Total Number of HW MEP             : 0
Total Number of VPLS MIP(Static/Up): 0 / 0

================================================================================
MEP Error Defect Types:
   Remote : Not receiving CCMs from Remote MEP
   Error  : Erroneous CCM received
   XCON   : Cross-connect CCM received
   RDI    : Remote Defect Indication sent by some MEP


#SW3 ERPS MAIN
sh erps test
  Name: test
  Operational State: Protection        Node Type: RPL Neighbor
  Configured State : Enabled

  East Ring Port : ^32  MepId: ---- Remote MepId: ----  Status: Blocked
  West Ring Port :  33  MepId: ---- Remote MepId: ----  Status: Blocked

  Periodic timer interval:  5000 millisec (Enabled)
  Hold-off timer interval:  0    millisec (Enabled)
  Guard timer interval   :  500  millisec (Enabled)
  WTB timer interval     :  5500 millisec (Enabled)
  WTR timer interval     :  60000 millisec (Enabled)

  Ring MD Level          :  ----
  Notify Topology Change :  -------
  Subring Mode           :  Virtual Channel

  ERPS Control Vlan: control_main  VID:305
  Topology Change Propogation List: None
  Topology Change Propogation  : Disabled
  ERPS Major ring has following Sub-Ring(s):
    Sub-Ring Name        :  test-sub
  ERPS Ring has following Protected Vlan(s):
    Vlan Name                    VID
    MNG                          9
    control_sub                  304
    subring-mep                  306
  Number of Protected Vlans: 3
  (+) RPL Protection Port, (^) RPL Neighbor Port
  (f) Force Switch Port, (m) Manual Switch Port


#SW3 SUB-RING
sh erps "test-sub"
  Name: test-sub
  Operational State: Protection        Node Type: RPL Neighbor
  Configured State : Enabled

  East Ring Port : ^31  MepId: ---- Remote MepId: ----  Status: Blocked
  West Ring Port : ----- MepId: ---- Remote MepId: ---- Status: ------

  Subring Protection Group: sw3-sw1

  Periodic timer interval:  5000 millisec (Enabled)
  Hold-off timer interval:  0    millisec (Enabled)
  Guard timer interval   :  500  millisec (Enabled)
  WTB timer interval     :  5500 millisec (Enabled)
  WTR timer interval     :  60000 millisec (Enabled)

  Ring MD Level          :  ----
  Notify Topology Change :  -------
  Subring Mode           :  Virtual Channel

  ERPS Control Vlan: control_sub   VID:304
  Topology Change Propogation List: None
  Topology Change Propogation  : Disabled
  ERPS Ring's Main-Ring  : test
  ERPS Ring has following Protected Vlan(s):
    Vlan Name                    VID
    MNG                          9
  Number of Protected Vlans: 1
  (+) RPL Protection Port, (^) RPL Neighbor Port
  (f) Force Switch Port, (m) Manual Switch Port

So, both ports on interconnected nodes facing subring are blocked.I have tried setting primary and secondary protection path on both nodes with no luck.
Looks like i just misunderstanding something here.

Any help will be appreciated, thanks in advance.
Photo of Nikolay Chernyaev

Nikolay Chernyaev

  • 180 Points 100 badge 2x thumb
Daniel, looks like we wrote our posts simultaneously.I have done all this configuration today, but got problems on interconnected nodes - ports facing sub-ring blocked.
All output you can check in my previous post.
The only difference in our configurations - i have added cfm-vlan to protected vlan in main ring to avoid loop.
Also, you wrote configuration for SW2, in my diagram it should be SW3 according to RFC appendix.
Any  thoughts will be helpful.
(Edited)
Photo of Nikolay Chernyaev

Nikolay Chernyaev

  • 180 Points 100 badge 2x thumb
I have replaced Qtech 8200 with Juniper EX4500 with same result.
Photo of Daniel Flouret

Daniel Flouret, Employee

  • 7,470 Points 5k badge 2x thumb
Nikolay, I suggest that you open a case with GTAC so they can take a look at what's going on here.
Photo of Nikolay Chernyaev

Nikolay Chernyaev

  • 180 Points 100 badge 2x thumb
Just for the record.
We have discovered an unexpected behavior with ERPS, this is a bug i think.
If any port participating in ERPS formed as a sharing, and config master disabled while other ports in sharing are in service, deleting any ERPS protecting vlan or removing vlan from ring ports causing ERPS to switch to protection.
Photo of Luca Spinacci

Luca Spinacci

  • 216 Points 100 badge 2x thumb

Hi Nikolay,

I have the same problem: when multiple failures on main ring occur the ports on interconnected nodes facing sub-ring go to blocked state.

Have you done any advancement on this?

Thank you for the help.