We run what can essentially be thought of as a WAN service. We have a core switch, with many edge switches off it. Off those edge switches are 3rd party switches we have no control over.
Core ---> Edge ---> 3rd Party Switch
I am trying to correctly configure STP to block loops potentially created by the 3rd parties. i.e.
Core ---> Edge1 ---> 3rd party ---> 3rd party ---> edge2 ---> Core (so the 3rd party could potentially bridge 2 of our edge ports therefore create a loop).
I thought setting the ports between the core and edge switches to "point-to-point" would achieve this, however I have set it up in a lab and the ports continue to say "forwarding". If i set the ports to stp edge mode it sets the port to blocking as i would expect.
Given I'm configuring STP between 2 extreme switches, I thought Point-to-point was the correct method and "edge" mode was for endpoints.
Could someone please advise how to correctly set up STP to achieve what I want - to protect against a 3rd party bridging our network?
uncontrolled switches connected to a layer 2 domain are problematic...
Anyway, you are correct to use point-to-point mode for the links inside your own network (core & edge), and edge mode on the third-party facing ports. You should consider using edge-safeguard on the edge ports.
Edge mode is for end points, that is devices not participating in the spanning tree protocol.
The 3rd party switches might drop BPDUs, thus you should consider using ELRP in addition to STP.
Additionally, you should consider using rate-limiters for broadcast, unknown unicast and possibly multicast frames to mitigate the effects of a bridging loop.
spanning tree blocks one side of a link only. On EXOS, spanning tree blocks only VLANs associated to that spanning tree domain (stpd) (I am assuming you are using EXOS switches, please correct me if I'm wrong).
Spanning tree blocks ports (resp. VLANs on a port) inside the spanning tree domain only. You can create loops outside the STP domain that cannot be detected by the spanning tree protocol, e.g. by connecting to edge ports using a switch that drops the spanning tree BPDUs.
If you connect two of your switches with two cables and configure spanning tree on those ports & VLANs, you will see that one of the switches blocks its port / VLAN(s).
The reason for point-to-point mode is to decrease convergence time on topology changes. The two switches on each end of the link negotiate their respective views on the topology and decide if one of them needs to block the port. This cannot be done on shared media (e.g. a hub), in which case time-outs are used instead of negotiation. Edge mode is used to not even try STP negotiations, because the connected device is not supposed to speak STP. This is where edge-safeguard comes in, blocking a port / VLAN if receiving a BPDU. BPDUs need to be sent out on edge ports to detect e.g. a cable connecting two edge-ports.
Spanning tree configuration on EXOS can be quite involved, but it generally works if configured correctly. EXOS supports many different spanning tree versions in many combinations, which complicates configuration.
When does P-2-P actually block a port? In my lab I cannot get it to set a port in blocking mode, I have deliberately created a loop and can see the effects on performance / cpu on the switches, but the ports are still Forwarding.
The user guide / concepts guide very much glosses over P-2-P mode and focuses mostly on edge w/edge safeguard, and there seems to be nothing specific in the knowledge base.