ERPS subring problem.

[Nikolay_Chernya]

Hi Extreme Community.We have some experience with EAPS, now trying to make some lab test with ERPS and various vendors to push ERPS in to production.

Currently we are observing following problem:
When main ring broken, subring RPL owner does not unblocks RPL port.

Here is our lab.

SW1 config:
#VLANS
create vlan "MNG"
configure vlan MNG tag 9
configure vlan MNG add ports 1-3 tagged
configure vlan MNG ipaddress 10.10.10.111 255.255.255.0
create vlan "control_main"configure vlan control_main tag 305
configure vlan control_main add ports 2-3 tagged
configure vlan control_sub tag 304configure vlan control_sub add ports 1-3 tagged

#ERPS MAIN RING
create erps test
configure erps test subring-mode virtualChannel
configure erps test add control vlan control_main
configure erps test ring-port east 3
configure erps test ring-port west 2
configure erps test protection-port 2
configure erps test revert enable
configure erps test timer wait-to-restore 60000
configure erps test add protected vlan MNG
configure erps test add protected vlan control_sub
configure erps test add sub-ring test-sub
enable erps test

#ERPS SUB-RING
create erps test-sub
configure erps test-sub subring-mode virtualChannel
configure erps test-sub add control vlan control_sub
configure erps test-sub ring-port east 1
configure erps test-sub revert enable
configure erps test-sub timer wait-to-restore 60000
configure erps test-sub add protected vlan MNG

#SW2 config:

#MST configuration (protected vlans, mst disabled)
spanning-tree mst configuration
instance 0 vlan 1-8;10-303;305-4094
instance 1 vlan 9;304

#ERPS
erps-ring test
open-ring
raps-virtual-channel with
erps-instance 1
wtr-timer 1
raps-mel 0
rpl port1 owner
protected-instance 1
control-vlan 304

#PORTS
Interface Ethernet1/0/25
switchport mode trunk
erps-ring test port0
!
Interface Ethernet1/0/26
switchport mode trunk
erps-ring test port1

SW3 config:
#VLANS
create vlan "MNG"configure vlan MNG tag 9
configure vlan MNG add ports 31-33 tagged
configure vlan MNG ipaddress 10.10.10.114 255.255.255.0
create vlan "control_main"
configure vlan control_main tag 305
create vlan "control_sub"
configure vlan control_sub tag 304
configure vlan control_main add ports 32-33 tagged
configure vlan control_sub add ports 31-33 tagged

#ERPS MAIN RING
create erps testconfigure erps test subring-mode virtualChannel
configure erps test add control vlan control_main
configure erps test ring-port east 32
configure erps test ring-port west 33
configure erps test revert enable
configure erps test neighbor-port 32
configure erps test timer wait-to-restore 60000
configure erps test add protected vlan MNG
configure erps test add protected vlan control_sub
configure erps test add sub-ring test-sub
enable erps test

#ERPS SUB-RING
create erps test-sub
configure erps test-sub subring-mode virtualChannel
configure erps test-sub add control vlan control_sub
configure erps test-sub ring-port east 31
configure erps test-sub revert enable
configure erps test-sub neighbor-port 31
configure erps test-sub timer wait-to-restore 60000
configure erps test-sub add protected vlan MNG
enable erps test-sub

Ok, all configs are done, let's see what is going on:

#SW1 main ERPS:
sw1# sh erps test
Name: test
Operational State: Idle Node Type: RPL Owner, Revertive
Configured State : Enabled

East Ring Port : 3 MepId: ---- Remote MepId: ---- Status: Unblocked
West Ring Port : +2 MepId: ---- Remote MepId: ---- Status: Blocked

Periodic timer interval: 5000 millisec (Enabled)
Hold-off timer interval: 0 millisec (Enabled)
Guard timer interval : 500 millisec (Enabled)
WTB timer interval : 5500 millisec (Enabled)
WTR timer interval : 60000 millisec (Enabled)

Ring MD Level : ----
Notify Topology Change : -------
Subring Mode : Virtual Channel

ERPS Control Vlan: control_main VID:305
Topology Change Propogation List: None
Topology Change Propogation : Disabled
ERPS Major ring has following Sub-Ring(s):
Sub-Ring Name : test-sub
ERPS Ring has following Protected Vlan(s):
Vlan Name VID
MNG 9
control_sub 304
Number of Protected Vlans: 2
(+) RPL Protection Port, (^) RPL Neighbor Port
(f) Force Switch Port, (m) Manual Switch Port

#SW1 subring erps:
sh erps "test-sub"
Name: test-sub
Operational State: Idle Node Type: InterConnection Node
Configured State : Enabled

East Ring Port : 1 MepId: ---- Remote MepId: ---- Status: Unblocked
West Ring Port : ----- MepId: ---- Remote MepId: ---- Status: ------

Periodic timer interval: 5000 millisec (Enabled)
Hold-off timer interval: 0 millisec (Enabled)
Guard timer interval : 500 millisec (Enabled)
WTB timer interval : 5500 millisec (Enabled)
WTR timer interval : 60000 millisec (Enabled)

Ring MD Level : ----
Notify Topology Change : -------
Subring Mode : Virtual Channel

ERPS Control Vlan: control_sub VID:304
Topology Change Propogation List: None
Topology Change Propogation : Disabled
ERPS Ring's Main-Ring : test
ERPS Ring has following Protected Vlan(s):
Vlan Name VID
MNG 9
Number of Protected Vlans: 1
(+) RPL Protection Port, (^) RPL Neighbor Port
(f) Force Switch Port, (m) Manual Switch Port

#On sw2:
#sh erps instance
ERPS Ring: test
Instance: 1
Description: -
Protected Instance: 1
Revertive mode: revertive
R-APS MEL: 0
R-APS Virtual-Channel: with
Control Vlan: 304
Ring ID: 1
Guard Timer(10ms): 50
Holdoff Timer(seconds): 0
WTR Timer(min): 1
-------------------------------------------
Port Role Port-Status
-------------------------------------------
Port0 common forwarding
Port1 RPL owner blocked

sh erps ring test
R: RPL Owner
N: RPL Neighbour
C: Common Node
-----------------------------------------
Version: ITU-T G.8032v2
R-APS ring topology: open-ring
R-APS Virtual-Channel: with
Port0: Ethernet1/0/25
Failure-detect type: physical-link
Port1: Ethernet1/0/26
Failure-detect type: physical-link
--------------------------------------------------------------------------------------------------------------------------------
Instance Contral Protected WTR_Timer Guard_Timer Holdoff_Timer
ID Vlan Instance (min) (10ms) (second) Port0 Port1
--------------------------------------------------------------------------------------------------------------------------------
1 304 1 1 50 0 C R

#on SW3:
#MAIN ERPS:
sh erps test
Name: test
Operational State: Idle Node Type: RPL Neighbor
Configured State : Enabled

East Ring Port : ^32 MepId: ---- Remote MepId: ---- Status: Blocked
West Ring Port : 33 MepId: ---- Remote MepId: ---- Status: Unblocked

Periodic timer interval: 5000 millisec (Enabled)
Hold-off timer interval: 0 millisec (Enabled)
Guard timer interval : 500 millisec (Enabled)
WTB timer interval : 5500 millisec (Enabled)
WTR timer interval : 60000 millisec (Enabled)

Ring MD Level : ----
Notify Topology Change : -------
Subring Mode : Virtual Channel

ERPS Control Vlan: control_main VID:305
Topology Change Propogation List: None
Topology Change Propogation : Disabled
ERPS Major ring has following Sub-Ring(s):
Sub-Ring Name : test-sub
ERPS Ring has following Protected Vlan(s):
Vlan Name VID
MNG 9
control_sub 304
Number of Protected Vlans: 2
(+) RPL Protection Port, (^) RPL Neighbor Port
(f) Force Switch Port, (m) Manual Switch Port

#subring ERPS
sh erps "test-sub"
Name: test-sub
Operational State: Idle Node Type: RPL Neighbor
Configured State : Enabled

East Ring Port : ^31 MepId: ---- Remote MepId: ---- Status: Blocked
West Ring Port : ----- MepId: ---- Remote MepId: ---- Status: ------

Periodic timer interval: 5000 millisec (Enabled)
Hold-off timer interval: 0 millisec (Enabled)
Guard timer interval : 500 millisec (Enabled)
WTB timer interval : 5500 millisec (Enabled)
WTR timer interval : 60000 millisec (Enabled)

Ring MD Level : ----
Notify Topology Change : -------
Subring Mode : Virtual Channel

ERPS Control Vlan: control_sub VID:304
Topology Change Propogation List: None
Topology Change Propogation : Disabled
ERPS Ring's Main-Ring : test
ERPS Ring has following Protected Vlan(s):
Vlan Name VID
MNG 9
Number of Protected Vlans: 1
(+) RPL Protection Port, (^) RPL Neighbor Port
(f) Force Switch Port, (m) Manual Switch Port

And now, we got a problem. When both links in main ERPS ring fails, RPL owner in subring does not opens RPL link.

#SW1
sh erps "test-sub"
Name: test-sub
Operational State: Idle Node Type: InterConnection Node
Configured State : Enabled

East Ring Port : 1 MepId: ---- Remote MepId: ---- Status: Unblocked
West Ring Port : ----- MepId: ---- Remote MepId: ---- Status: ------

Periodic timer interval: 5000 millisec (Enabled)
Hold-off timer interval: 0 millisec (Enabled)
Guard timer interval : 500 millisec (Enabled)
WTB timer interval : 5500 millisec (Enabled)
WTR timer interval : 60000 millisec (Enabled)

Ring MD Level : ----
Notify Topology Change : -------
Subring Mode : Virtual Channel

ERPS Control Vlan: control_sub VID:304
Topology Change Propogation List: None
Topology Change Propogation : Disabled
ERPS Ring's Main-Ring : test
ERPS Ring has following Protected Vlan(s):
Vlan Name VID
MNG 9
Number of Protected Vlans: 1
(+) RPL Protection Port, (^) RPL Neighbor Port
(f) Force Switch Port, (m) Manual Switch Port

#SW2
sh erps instanceERPS Ring: test
Instance: 1
Description: -
Protected Instance: 1
Revertive mode: revertive
R-APS MEL: 0
R-APS Virtual-Channel: with
Control Vlan: 304
Ring ID: 1
Guard Timer(10ms): 50
Holdoff Timer(seconds): 0
WTR Timer(min): 1
-------------------------------------------
Port Role Port-Status
-------------------------------------------
Port0 common forwarding
Port1 RPL owner blocked

#SW3
sh erps "test-sub"
Name: test-sub
Operational State: Idle Node Type: RPL Neighbor
Configured State : Enabled

East Ring Port : ^31 MepId: ---- Remote MepId: ---- Status: Blocked
West Ring Port : ----- MepId: ---- Remote MepId: ---- Status: ------

Periodic timer interval: 5000 millisec (Enabled)
Hold-off timer interval: 0 millisec (Enabled)
Guard timer interval : 500 millisec (Enabled)
WTB timer interval : 5500 millisec (Enabled)
WTR timer interval : 60000 millisec (Enabled)

Ring MD Level : ----
Notify Topology Change : -------
Subring Mode : Virtual Channel

ERPS Control Vlan: control_sub VID:304
Topology Change Propogation List: None
Topology Change Propogation : Disabled
ERPS Ring's Main-Ring : test
ERPS Ring has following Protected Vlan(s):
Vlan Name VID
MNG 9
Number of Protected Vlans: 1
(+) RPL Protection Port, (^) RPL Neighbor Port
(f) Force Switch Port, (m) Manual Switch Port

Any help will be highly appreciated.

[Anton_Nekhorosh]

Daniel, below the same config don't work, may be you have other version ExtremeXOS? our version is 15.4.1.3 v1541b3-patch1-13
Something can another?

[Nikolay_Chernya]

Thank you Daniel, yours answer is very helpfull.
If you have and sample config for UP meps similar to our config, it will be great to see them here.

[dflouret]

Nikolai, Anton,

From the EXOS User Guide v16.1:
G.8032 Version 2 (pg. 1086)
The concept of sub-rings is introduced to add multiple rings to the main ring. A sub-ring is an incomplete ring that completes its path through the main ring or other sub-rings. The control path for the sub-ring completes either through the implementation of a virtual channel, or by changing the flow of control packets in the sub-rings. Virtual channels are supported through the use of the sub-rings control channel being configured as a data VLAN in the main ring.

Multiple Failure Protection using CFM (pg. 1087)
You can use CFM UP-MEP support in the sub-ring to provide protection against multiple failures in the main ring. Configure an UP-MEP on the interconnected nodes, where a segmentation of the main ring results in the UP-MEP notifying the sub-ring of a failure. This causes the sub-ring to open its RPL and place the interconnected node in manual switch. This is done to avoid a super-loop during recovery of the main ring. This implementation is as directed in Appendix X.3 of the G.8032 standard.

Let me "translate" this.

When a failure occurs in an ERPS ring, the nodes that own the ERPS ports involved in the failure will send CFM R-‐APS messages to inform of the failure, which makes all nodes in the ring to flush their tables and the owner of the RPL port to close the RPL port. In your diagram, Nikolai, if the left link between SW1 and SW3 failed SW3 would inform of the failure through the main ring's control VLAN. Upon receiving the notification, SW1 would close the RPL port and reestablish connectivity to SW3.

What happens to the sub-ring when the main ring fails? Nothing. Since the failure does not involve any of its ports or links, no message is transmitted through the sub-ring's control vlan and no action is taken by the sub-ring nodes. The main ring "heals" itself and maintains the connectivity of SW2 with SW3, as indicated in the first paragraph quoted from the User Guide.

Similarly, if the link between SW1 and SW2 failed, SW1would inform SW2 of the failure through the sub-ring's control VLAN (which happens to go through the main ring). Upon receiving the notification, SW2 would close the RPL port and reestablish connectivity to SW1 and SW3.

Now, the problem comes when we have MULTIPLE failures in the main ring, making it unable to maintain the connectivity between the nodes of the sub-ring. The sub-ring will not detect the failures, as explained previously, because they don't involve ports belonging to the sub-ring. That is why you need to configure CFM, as indicated in the second paragraph quoted from the User Guide.

Now, if you're familiar with EAPS, you might wonder why you need to configure CFM. Well, EAPS includes a mechanism where the Master node send a "health check" message through the control vlan to check that there's connectivity in the ring. This is slow but, in case a failure goes undetected by the other nodes in the ring, the Master will eventually heal the ring. In ERPS the RPL owner doesn't do that. ERPS relies on the configuration of CFM Connectivity Check Messages (CCM) to detect failures not reported by the ring nodes.

To make your configuration work you will need to configure CFM Maintenance End Points (MEPs). There's an sample configuration in the EXOS User Guide v16.1(page 1094), which is almost identical to yours, except that it has 4 switches, instead of 3.

Here's a How-To Guide that explains the basics of CFM R-APS messaging and error detection and notification in a single ERPS ring:
https://www.dropbox.com/s/l5m3fuirlsbmrru/SVC%20Tech%20MB%20Tech%20Guide%20-%20ERPSv1%20%28G.8032v1%...

[Anton_Nekhorosh]

Anybody has no ideas?!

[Nikolay_Chernya]

Config is similar to:
https://gtacknowledge.extremenetworks.com/articles/How_To/How-to-configure-a-ERPS-ring-with-a-sub-ri...

We have also tested this config in UNetLab with extreme summit switches only, with same simptoms.