Added troubleshooting
Also fixed an invalid doc reference
Change-Id: I46cde400d03f71e9660a4b99a35401c62a378cd2
diff --git a/configuration/bridging-unicast.rst b/configuration/bridging-unicast.rst
index 1de2a35..2dc8c92 100644
--- a/configuration/bridging-unicast.rst
+++ b/configuration/bridging-unicast.rst
@@ -2,7 +2,7 @@
============================
.. attention::
- - VLAN **4090** is reserved for :doc:`pseudowire <configuration/pseudowire>` transport
+ - VLAN **4090** is reserved for :doc:`pseudowire <pseudowire>` transport
- VLAN **4094** is reserved for unconfigured ports (e.g. spine facing ports)
Access Ports
diff --git a/troubleshooting.rst b/troubleshooting.rst
index c53f676..6930dd0 100644
--- a/troubleshooting.rst
+++ b/troubleshooting.rst
@@ -1,5 +1,645 @@
Troubleshooting Guide
*********************
-.. note::
- We are updating this page. More coming soon.
\ No newline at end of file
+Overview
+========
+
+The **Trellis Troubleshooting Tools (T3)** is an ONOS application intended to **help you troubleshoot traffic-related problems** in your network by **tracing installed flows and groups**.
+T3 works both on simulated OVS-based topologies but also on real HW, basing it's internal decisions on the driver assigned to the device.
+T3 also currently supports dual links between leaf and spine and properly handles the difference between deferred and immediate rules.
+Dual Homed hosts are also supported in T3.
+
+To activate this tool on your ONOS environment, you need to run:
+
+.. code-block:: console
+
+ onos> app activate t3
+
+T3 supports tracing of following packet types:
+
+- L2 Unicast
+- L2 Broadcast
+- L3 Unicast (both IPv4 and IPv6)
+- L3 Multicast (both IPv4 and IPv6)
+- LLDP and BDDP
+- ARP
+- DHCP
+
+
+t3-troubleshoot
+===============
+As of right now the tools provides a CLI command (troubleshoot).
+As arguments to the CLI command you have to pass arguments describing the packet you want to trace through the network.
+
+t3-troubleshoot Help
+--------------------
+
+.. code-block:: console
+
+ onos> t3-troubleshoot --help
+ DESCRIPTION
+ onos:troubleshoot
+
+ troubleshoots flows and groups between source and destination
+
+ SYNTAX
+ onos:troubleshoot [options]
+
+ OPTIONS
+ -sm, --srcMac
+ Source MAC
+ -j, --json
+ Output JSON
+ --help
+ Display this help message
+ -v, --verbose
+ Outputs complete path
+ -ipp, --ipProto
+ IP Proto
+ -s, --srcIp
+ Source IP
+ -dm, --dstMac
+ Destination MAC
+ -ml, --mplsLabel
+ Mpls label of incoming packet
+ -vv, --veryverbose
+ Outputs flows and groups for every device
+ -udps, --udpSrc
+ UDP Source
+ -mb, --mplsBos
+ MPLS BOS
+ -et, --ethType
+ ETH Type
+ (defaults to ipv4)
+ -sp, --srcPort
+ Source Port
+ -dtp, --dstTcpPort
+ destination TCP Port
+ -udpd, --udpDst
+ UDP Destination
+ -stp, --srcTcpPort
+ Source TCP Port
+ -d, --dstIp
+ Destination IP
+ -vid, --vlanId
+ Vlan of incoming packet
+ (defaults to None)
+
+Default Verbosity Level
+-----------------------
+Let's look at an example.
+
+.. code-block:: console
+
+ onos> t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:00:00:00:02:04 -vid None
+
+with this command we are asking the t3 tool to trace a packet with
+
+- Source IP of ``10.0.2.1/32``
+- Input port ``of:0000000000000204/3``
+- Source MAC ``00:AA:00:00:00:01``
+- ethType ``IPv4``
+- Source TCP Port ``12345``
+- Destination IP ``10.0.3.1/32``
+- Destination MAC ``00:AA:00:00:00:01``
+- Destination TCP Port ``80`` and
+- VLAN ID ``None``
+
+This command has a verbosity level of 0 and will output just the paths that the packet might take inside the network.
+In the example deployment taken from trellis.py [#trellis.py]_, the output will be:
+
+.. code-block:: console
+
+ Tracing packet: [IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.3.1/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true]
+ Paths
+ [of:0000000000000204/3, of:0000000000000204/2, of:0000000000000227/1, of:0000000000000227/2, of:0000000000000205/2, of:0000000000000205/3]
+ [of:0000000000000204/3, of:0000000000000204/1, of:0000000000000226/1, of:0000000000000226/2, of:0000000000000205/1, of:0000000000000205/3]
+ Result:
+ Reached required destination Host
+ Reached required destination Host
+
+Middle Verbosity Level
+----------------------
+There are also two other verbosity levels.
+
+.. code-block:: console
+
+ onos> t3-troubleshoot -v -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:00:00:00:02:04 -vid None
+
+prints flows selectors and groupId that are met along the path
+
+.. code-block:: console
+
+ Tracing packet: [IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.3.1/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true]
+ Path [of:0000000000000204/3, of:0000000000000204/2, of:0000000000000227/1, of:0000000000000227/2, of:0000000000000205/2, of:0000000000000205/3]
+ Device of:0000000000000204
+ Input from of:0000000000000204/3
+ Flows
+ flowId=a20000bd57cb49, selector=[]
+ flowId=900000d4173ff7, selector=[IN_PORT:3, VLAN_VID:None]
+ flowId=9000004b3d1965, selector=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:20]
+ flowId=900000499769f5, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.0/24]
+ flowId=a2000038b487e7, selector=[]
+ Groups
+ groupId=GroupId{id=0x70000010}
+ groupId=GroupId{id=0x9200000d}
+ groupId=GroupId{id=0x9000000c}
+ groupId=GroupId{id=0xffe0002}
+ groupId=GroupId{id=0x9200000f}
+ groupId=GroupId{id=0x9000000e}
+ groupId=GroupId{id=0xffe0001}
+ Output through of:0000000000000204/2
+ Device of:0000000000000227
+ Input from of:0000000000000227/1
+ Flows
+ flowId=a200005577bcd6, selector=[]
+ flowId=900000498457b2, selector=[IN_PORT:1, VLAN_VID:None]
+ flowId=900000f87746c9, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:27, ETH_TYPE:mpls_unicast, VLAN_VID:4094]
+ flowId=a200005bfd6ed7, selector=[]
+ flowId=9000009f3d554f, selector=[ETH_TYPE:mpls_unicast, MPLS_LABEL:205, MPLS_BOS:true]
+ flowId=a20000e601fc35, selector=[]
+ Groups
+ groupId=GroupId{id=0x70000017}
+ groupId=GroupId{id=0x20000016}
+ groupId=GroupId{id=0xffe0002}
+ Output through of:0000000000000227/2
+ Device of:0000000000000205
+ Input from of:0000000000000205/2
+ Flows
+ flowId=a200008d542817, selector=[]
+ flowId=900000be3179a0, selector=[IN_PORT:1, VLAN_VID:None]
+ flowId=900000766be164, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:05, ETH_TYPE:ipv4, VLAN_VID:4094]
+ flowId=900000ec1d5779, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.1/32]
+ flowId=a2000042d8e6b5, selector=[]
+ Groups
+ groupId=GroupId{id=0x2000002a}
+ groupId=GroupId{id=0x1e0003}
+ groupId=GroupId{id=0x2000002a}
+ groupId=GroupId{id=0x1e0003}
+ Output through of:0000000000000205/3
+ Path [of:0000000000000204/3, of:0000000000000204/1, of:0000000000000226/1, of:0000000000000226/2, of:0000000000000205/1, of:0000000000000205/3]
+ Device of:0000000000000204
+ Input from of:0000000000000204/3
+ Flows
+ flowId=a20000bd57cb49, selector=[]
+ flowId=900000d4173ff7, selector=[IN_PORT:3, VLAN_VID:None]
+ flowId=9000004b3d1965, selector=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:20]
+ flowId=900000499769f5, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.0/24]
+ flowId=a2000038b487e7, selector=[]
+ Groups
+ groupId=GroupId{id=0x70000010}
+ groupId=GroupId{id=0x9200000d}
+ groupId=GroupId{id=0x9000000c}
+ groupId=GroupId{id=0xffe0002}
+ groupId=GroupId{id=0x9200000f}
+ groupId=GroupId{id=0x9000000e}
+ groupId=GroupId{id=0xffe0001}
+ Output through of:0000000000000204/1
+ Device of:0000000000000226
+ Input from of:0000000000000226/1
+ Flows
+ flowId=a20000e9b27d24, selector=[]
+ flowId=9000004bd76105, selector=[IN_PORT:1, VLAN_VID:None]
+ flowId=90000069f274d1, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:26, ETH_TYPE:mpls_unicast, VLAN_VID:4094]
+ flowId=a200000ec7529e, selector=[]
+ flowId=90000050f9a350, selector=[ETH_TYPE:mpls_unicast, MPLS_LABEL:205, MPLS_BOS:true]
+ flowId=a20000b15cada6, selector=[]
+ Groups
+ groupId=GroupId{id=0x7000000b}
+ groupId=GroupId{id=0x20000009}
+ groupId=GroupId{id=0xffe0002}
+ Output through of:0000000000000226/2
+ Device of:0000000000000205
+ Input from of:0000000000000205/1
+ Flows
+ flowId=a200008d542817, selector=[]
+ flowId=900000be3179a0, selector=[IN_PORT:1, VLAN_VID:None]
+ flowId=900000766be164, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:05, ETH_TYPE:ipv4, VLAN_VID:4094]
+ flowId=900000ec1d5779, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.1/32]
+ flowId=a2000042d8e6b5, selector=[]
+ Groups
+ groupId=GroupId{id=0x2000002a}
+ groupId=GroupId{id=0x1e0003}
+ groupId=GroupId{id=0x2000002a}
+ groupId=GroupId{id=0x1e0003}
+ Output through of:0000000000000205/3
+ Result:
+ Reached required destination Host
+ Reached required destination Host
+
+High Verbosity Level
+--------------------
+The most verbose output you can get is
+
+.. code-block:: console
+
+ onos> t3-troubleshoot -vv -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:00:00:00:02:04 -vid None
+
+This verbosity level prints all the flows and groups that are met along the path
+
+.. code-block:: console
+
+ Tracing packet: [IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.3.1/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true]
+ Path [of:0000000000000204/3, of:0000000000000204/2, of:0000000000000227/1, of:0000000000000227/2, of:0000000000000205/2, of:0000000000000205/3]
+ Device of:0000000000000204
+ Input from of:0000000000000204/3
+ Flows
+ ADDED, bytes=733506, packets=8691, table=0, priority=0, selector=[], treatment=[transition=TABLE:10]
+ ADDED, bytes=11110, packets=61, table=10, priority=32768, selector=[IN_PORT:3, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:20], transition=TABLE:20]
+ ADDED, bytes=8698, packets=29, table=20, priority=32768, selector=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:20], treatment=[transition=TABLE:30]
+ ADDED, bytes=16612, packets=50, table=30, priority=48010, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.0/24], treatment=[deferred=[GROUP:0x70000010], transition=TABLE:60]
+ ADDED, bytes=34400, packets=112, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]]
+ Groups
+ id=0x70000010, state=ADDED, type=SELECT, bytes=16906, packets=53, appId=org.onosproject.segmentrouting, referenceCount=3
+ id=0x70000010, bucket=1, bytes=490, packets=5, actions=[GROUP:0x9200000d]
+ id=0x70000010, bucket=2, bytes=16416, packets=48, actions=[GROUP:0x9200000f]
+ id=0x9200000d, state=ADDED, type=INDIRECT, bytes=490, packets=5, appId=org.onosproject.segmentrouting, referenceCount=0
+ id=0x9200000d, bucket=1, bytes=490, packets=5, actions=[VLAN_POP, MPLS_PUSH:mpls_unicast, MPLS_LABEL:205, GROUP:0x9000000c, VLAN_PUSH:vlan, VLAN_ID:4095]
+ id=0x9000000c, state=ADDED, type=INDIRECT, bytes=490, packets=5, appId=org.onosproject.segmentrouting, referenceCount=0
+ id=0x9000000c, bucket=1, bytes=490, packets=5, actions=[ETH_DST:00:00:00:00:02:27, ETH_SRC:00:00:00:00:02:04, VLAN_ID:4094, GROUP:0xffe0002]
+ id=0xffe0002, state=ADDED, type=INDIRECT, bytes=726, packets=7, appId=org.onosproject.segmentrouting, referenceCount=0
+ id=0xffe0002, bucket=1, bytes=726, packets=7, actions=[VLAN_POP, OUTPUT:2]
+ id=0x9200000f, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
+ id=0x9200000f, bucket=1, bytes=16416, packets=48, actions=[VLAN_POP, MPLS_PUSH:mpls_unicast, MPLS_LABEL:205, GROUP:0x9000000e, VLAN_PUSH:vlan, VLAN_ID:4095]
+ id=0x9000000e, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
+ id=0x9000000e, bucket=1, bytes=16416, packets=48, actions=[ETH_DST:00:00:00:00:02:26, ETH_SRC:00:00:00:00:02:04, VLAN_ID:4094, GROUP:0xffe0001]
+ id=0xffe0001, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
+ id=0xffe0001, bucket=1, bytes=16416, packets=48, actions=[VLAN_POP, OUTPUT:1]
+ Output through of:0000000000000204/2
+ Device of:0000000000000227
+ Input from of:0000000000000227/1
+ Flows
+ ADDED, bytes=702318, packets=8511, table=0, priority=0, selector=[], treatment=[transition=TABLE:10]
+ ADDED, bytes=343130, packets=4234, table=10, priority=32768, selector=[IN_PORT:1, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:4094], transition=TABLE:20]
+ ADDED, bytes=754, packets=7, table=20, priority=32768, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:27, ETH_TYPE:mpls_unicast, VLAN_VID:4094], treatment=[transition=TABLE:23]
+ ADDED, bytes=17566, packets=57, table=23, priority=0, selector=[], treatment=[transition=TABLE:24]
+ ADDED, bytes=510, packets=5, table=24, priority=100, selector=[ETH_TYPE:mpls_unicast, MPLS_LABEL:205, MPLS_BOS:true], treatment=[deferred=[MPLS_POP:ipv4, DEC_TTL, GROUP:0x70000017], transition=TABLE:60]
+ ADDED, bytes=17566, packets=57, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]]
+ Groups
+ id=0x70000017, state=ADDED, type=SELECT, bytes=754, packets=7, appId=org.onosproject.segmentrouting, referenceCount=2
+ id=0x70000017, bucket=1, bytes=754, packets=7, actions=[GROUP:0x20000016]
+ id=0x20000016, state=ADDED, type=INDIRECT, bytes=754, packets=7, appId=org.onosproject.segmentrouting, referenceCount=0
+ id=0x20000016, bucket=1, bytes=754, packets=7, actions=[ETH_DST:00:00:00:00:02:05, ETH_SRC:00:00:00:00:02:27, VLAN_ID:4094, GROUP:0xffe0002]
+ id=0xffe0002, state=ADDED, type=INDIRECT, bytes=754, packets=7, appId=org.onosproject.segmentrouting, referenceCount=0
+ id=0xffe0002, bucket=1, bytes=754, packets=7, actions=[VLAN_POP, OUTPUT:2]
+ Output through of:0000000000000227/2
+ Device of:0000000000000205
+ Input from of:0000000000000205/2
+ Flows
+ ADDED, bytes=2105886, packets=24084, table=0, priority=0, selector=[], treatment=[transition=TABLE:10]
+ ADDED, bytes=358792, packets=4275, table=10, priority=32768, selector=[IN_PORT:1, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:4094], transition=TABLE:20]
+ ADDED, bytes=16416, packets=48, table=20, priority=32768, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:05, ETH_TYPE:ipv4, VLAN_VID:4094], treatment=[transition=TABLE:30]
+ ADDED, bytes=196, packets=2, table=30, priority=64010, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.1/32], treatment=[deferred=[GROUP:0x2000002a], transition=TABLE:60]
+ ADDED, bytes=1370884, packets=15163, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]]
+ Groups
+ id=0x2000002a, state=ADDED, type=INDIRECT, bytes=196, packets=2, appId=org.onosproject.segmentrouting, referenceCount=1
+ id=0x2000002a, bucket=1, bytes=196, packets=2, actions=[ETH_DST:00:AA:00:00:00:03, ETH_SRC:00:00:00:00:02:05, VLAN_ID:30, GROUP:0x1e0003]
+ id=0x1e0003, state=ADDED, type=INDIRECT, bytes=12960, packets=126, appId=org.onosproject.segmentrouting, referenceCount=1
+ id=0x1e0003, bucket=1, bytes=12960, packets=126, actions=[VLAN_POP, OUTPUT:3]
+ id=0x2000002a, state=ADDED, type=INDIRECT, bytes=196, packets=2, appId=org.onosproject.segmentrouting, referenceCount=1
+ id=0x2000002a, bucket=1, bytes=196, packets=2, actions=[ETH_DST:00:AA:00:00:00:03, ETH_SRC:00:00:00:00:02:05, VLAN_ID:30, GROUP:0x1e0003]
+ id=0x1e0003, state=ADDED, type=INDIRECT, bytes=12960, packets=126, appId=org.onosproject.segmentrouting, referenceCount=1
+ id=0x1e0003, bucket=1, bytes=12960, packets=126, actions=[VLAN_POP, OUTPUT:3]
+ Output through of:0000000000000205/3
+ Path [of:0000000000000204/3, of:0000000000000204/1, of:0000000000000226/1, of:0000000000000226/2, of:0000000000000205/1, of:0000000000000205/3]
+ Device of:0000000000000204
+ Input from of:0000000000000204/3
+ Flows
+ ADDED, bytes=733506, packets=8691, table=0, priority=0, selector=[], treatment=[transition=TABLE:10]
+ ADDED, bytes=11110, packets=61, table=10, priority=32768, selector=[IN_PORT:3, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:20], transition=TABLE:20]
+ ADDED, bytes=8698, packets=29, table=20, priority=32768, selector=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:20], treatment=[transition=TABLE:30]
+ ADDED, bytes=16612, packets=50, table=30, priority=48010, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.0/24], treatment=[deferred=[GROUP:0x70000010], transition=TABLE:60]
+ ADDED, bytes=34400, packets=112, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]]
+ Groups
+ id=0x70000010, state=ADDED, type=SELECT, bytes=16906, packets=53, appId=org.onosproject.segmentrouting, referenceCount=3
+ id=0x70000010, bucket=1, bytes=490, packets=5, actions=[GROUP:0x9200000d]
+ id=0x70000010, bucket=2, bytes=16416, packets=48, actions=[GROUP:0x9200000f]
+ id=0x9200000d, state=ADDED, type=INDIRECT, bytes=490, packets=5, appId=org.onosproject.segmentrouting, referenceCount=0
+ id=0x9200000d, bucket=1, bytes=490, packets=5, actions=[VLAN_POP, MPLS_PUSH:mpls_unicast, MPLS_LABEL:205, GROUP:0x9000000c, VLAN_PUSH:vlan, VLAN_ID:4095]
+ id=0x9000000c, state=ADDED, type=INDIRECT, bytes=490, packets=5, appId=org.onosproject.segmentrouting, referenceCount=0
+ id=0x9000000c, bucket=1, bytes=490, packets=5, actions=[ETH_DST:00:00:00:00:02:27, ETH_SRC:00:00:00:00:02:04, VLAN_ID:4094, GROUP:0xffe0002]
+ id=0xffe0002, state=ADDED, type=INDIRECT, bytes=726, packets=7, appId=org.onosproject.segmentrouting, referenceCount=0
+ id=0xffe0002, bucket=1, bytes=726, packets=7, actions=[VLAN_POP, OUTPUT:2]
+ id=0x9200000f, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
+ id=0x9200000f, bucket=1, bytes=16416, packets=48, actions=[VLAN_POP, MPLS_PUSH:mpls_unicast, MPLS_LABEL:205, GROUP:0x9000000e, VLAN_PUSH:vlan, VLAN_ID:4095]
+ id=0x9000000e, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
+ id=0x9000000e, bucket=1, bytes=16416, packets=48, actions=[ETH_DST:00:00:00:00:02:26, ETH_SRC:00:00:00:00:02:04, VLAN_ID:4094, GROUP:0xffe0001]
+ id=0xffe0001, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
+ id=0xffe0001, bucket=1, bytes=16416, packets=48, actions=[VLAN_POP, OUTPUT:1]
+ Output through of:0000000000000204/1
+ Device of:0000000000000226
+ Input from of:0000000000000226/1
+ Flows
+ ADDED, bytes=701910, packets=8507, table=0, priority=0, selector=[], treatment=[transition=TABLE:10]
+ ADDED, bytes=358984, packets=4275, table=10, priority=32768, selector=[IN_PORT:1, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:4094], transition=TABLE:20]
+ ADDED, bytes=16608, packets=48, table=20, priority=32768, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:26, ETH_TYPE:mpls_unicast, VLAN_VID:4094], treatment=[transition=TABLE:23]
+ ADDED, bytes=17158, packets=53, table=23, priority=0, selector=[], treatment=[transition=TABLE:24]
+ ADDED, bytes=16608, packets=48, table=24, priority=100, selector=[ETH_TYPE:mpls_unicast, MPLS_LABEL:205, MPLS_BOS:true], treatment=[deferred=[MPLS_POP:ipv4, DEC_TTL, GROUP:0x7000000b], transition=TABLE:60]
+ ADDED, bytes=17158, packets=53, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]]
+ Groups
+ id=0x7000000b, state=ADDED, type=SELECT, bytes=16608, packets=48, appId=org.onosproject.segmentrouting, referenceCount=2
+ id=0x7000000b, bucket=1, bytes=16608, packets=48, actions=[GROUP:0x20000009]
+ id=0x20000009, state=ADDED, type=INDIRECT, bytes=16608, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
+ id=0x20000009, bucket=1, bytes=16608, packets=48, actions=[ETH_DST:00:00:00:00:02:05, ETH_SRC:00:00:00:00:02:26, VLAN_ID:4094, GROUP:0xffe0002]
+ id=0xffe0002, state=ADDED, type=INDIRECT, bytes=16608, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
+ id=0xffe0002, bucket=1, bytes=16608, packets=48, actions=[VLAN_POP, OUTPUT:2]
+ Output through of:0000000000000226/2
+ Device of:0000000000000205
+ Input from of:0000000000000205/1
+ Flows
+ ADDED, bytes=2105886, packets=24084, table=0, priority=0, selector=[], treatment=[transition=TABLE:10]
+ ADDED, bytes=358792, packets=4275, table=10, priority=32768, selector=[IN_PORT:1, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:4094], transition=TABLE:20]
+ ADDED, bytes=16416, packets=48, table=20, priority=32768, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:05, ETH_TYPE:ipv4, VLAN_VID:4094], treatment=[transition=TABLE:30]
+ ADDED, bytes=196, packets=2, table=30, priority=64010, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.1/32], treatment=[deferred=[GROUP:0x2000002a], transition=TABLE:60]
+ ADDED, bytes=1370884, packets=15163, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]]
+ Groups
+ id=0x2000002a, state=ADDED, type=INDIRECT, bytes=196, packets=2, appId=org.onosproject.segmentrouting, referenceCount=1
+ id=0x2000002a, bucket=1, bytes=196, packets=2, actions=[ETH_DST:00:AA:00:00:00:03, ETH_SRC:00:00:00:00:02:05, VLAN_ID:30, GROUP:0x1e0003]
+ id=0x1e0003, state=ADDED, type=INDIRECT, bytes=12960, packets=126, appId=org.onosproject.segmentrouting, referenceCount=1
+ id=0x1e0003, bucket=1, bytes=12960, packets=126, actions=[VLAN_POP, OUTPUT:3]
+ id=0x2000002a, state=ADDED, type=INDIRECT, bytes=196, packets=2, appId=org.onosproject.segmentrouting, referenceCount=1
+ id=0x2000002a, bucket=1, bytes=196, packets=2, actions=[ETH_DST:00:AA:00:00:00:03, ETH_SRC:00:00:00:00:02:05, VLAN_ID:30, GROUP:0x1e0003]
+ id=0x1e0003, state=ADDED, type=INDIRECT, bytes=12960, packets=126, appId=org.onosproject.segmentrouting, referenceCount=1
+ id=0x1e0003, bucket=1, bytes=12960, packets=126, actions=[VLAN_POP, OUTPUT:3]
+ Output through of:0000000000000205/3
+ Result:
+ Reached required destination Host
+ Reached required destination Host
+
+Successful Example
+------------------
+In this example we trace a packet that should be bridged between two hosts in the same subnet
+
+.. code-block:: console
+
+ onos> t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:AA:00:00:00:02 -vid None
+ Tracing packet: [IN_PORT:3, ETH_DST:00:AA:00:00:00:02, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.2.2/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true]
+ Paths
+ [of:0000000000000204/3, of:0000000000000204/4]
+ Result:
+ Reached required destination Host
+
+Failed Example
+--------------
+Obviously failures can happen along one of the paths that the packet can take.
+In our sample we removed a flow in ``of:0000000000000226`` thus disabling that possible path for our initial packet.
+The output of our trace would then look something like
+
+.. code-block:: console
+
+ onos> t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:00:00:00:02:04 -vid None
+ Tracing packet: [IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.3.1/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true]
+ Paths
+ [of:0000000000000204/3, of:0000000000000204/2, of:0000000000000227/1, of:0000000000000227/2, of:0000000000000205/2, of:0000000000000205/3]
+ [of:0000000000000204/3, of:0000000000000204/1, of:0000000000000226/1]
+ Result:
+ Reached required destination Host
+ No output out of device of:0000000000000226 Packet is dropped
+
+Notice that on the second path that was previously going through ``of:0000000000000226`` now the packet is dropped due to the rule we removed.
+
+Example of Various Packet Types
+-------------------------------
+These sample commands refer to the Mininet environment found in trellis.py [#trellis.py]_
+
+- L2 Unicast IPv4
+
+ .. code-block:: console
+
+ t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -d 10.0.2.2 -dm 00:AA:00:00:00:02
+
+- L2 Unicast IPv6
+
+ .. code-block:: console
+
+ t3-troubleshoot -vv -s 2000::301 -sp of:0000000000000205/5 -sm 00:BB:00:00:00:03 -d 2000::302 -dm 00:BB:00:00:00:04 -et ipv6
+
+- L3 Unicast IPv4
+
+ .. code-block:: console
+
+ t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -d 10.0.3.1 -dm 00:00:00:00:02:04
+
+- L3 Unicast IPv6
+
+ .. code-block:: console
+
+ t3-troubleshoot -vv -s 2000::301 -sp of:0000000000000205/5 -sm 00:BB:00:00:00:03 -d 2000::202 -dm 00:00:00:00:02:05 -et ipv6
+
+- ARP
+
+ .. code-block:: console
+
+ t3-troubleshoot -sp of:0000000000000204/3 -et ARP
+
+- DHCP
+
+ .. code-block:: console
+
+ t3-troubleshoot -vv -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -d 10.0.3.253 -dm 00:00:00:00:02:04 -ipp 17 -udps 67 -udpd 67
+
+- LLDP
+
+ .. code-block:: console
+
+ t3-troubleshoot -sp of:0000000000000204/3 -et lldp
+
+- BDDP
+
+ .. code-block:: console
+
+ t3-troubleshoot -sp of:0000000000000204/3 -et bddp
+
+- LLDP from CONTROLLER
+
+ T3 is also capable of tracing packets coming in from the CONTROLLER.
+ We use the same input port with the possibility to give CONTROLLER as logical port number.
+ This is a mere convenience to carry the information inside T3 and do the proper flooding to all the ports of the given device.
+ This case is not treated like all the rest and the input port here is a logical entity that has no match on any physical element on the device.
+
+ .. code-block:: console
+
+ t3-troubleshoot -sp of:0000000000000204/CONTROLLER -et lldp
+
+- Multicast
+
+ T3 is also capable of tracing multicast packets.
+
+ .. code-block:: console
+
+ t3-troubleshoot -vv -sp of:0000000000000205/9 -et ipv4 -d 224.0.0.1 -dm 01:00:5e:00:00:01
+
+
+t3-troubleshoot-simple
+======================
+T3 also offers a simplified version of the command to be able to ping between two given hosts without having to specify all the information.
+This commands bases itself on top of some configuration present in ONOS through netcfg, namely the VLAN interface configuration and the segment routing leaf router MAC for L3 Unicast. If that is not present the user needs to input this information manually recurring to the the previously shown command. Same verbosity levels and output as the troubleshoot command are possible.
+
+t3-troubleshoot-simple Help
+---------------------------
+.. code-block:: console
+
+ onos> t3-troubleshoot-simple --help
+ DESCRIPTION
+ onos:troubleshoot-simple
+
+ Given two hosts troubleshoots flows and groups between them, in case of segment routing
+
+ SYNTAX
+ onos:troubleshoot-simple [options] one two
+
+ ARGUMENTS
+ one
+ One host ID
+ two
+ Another host ID
+
+ OPTIONS
+ -vv, --veryverbose
+ Outputs flows and groups for every device
+ -j, --json
+ Output JSON
+ --help
+ Display this help message
+ -et, --ethType
+ ETH Type
+ (defaults to ipv4)
+ -v, --verbose
+ Outputs complete path
+
+t3-troubleshoot-simple Example
+------------------------------
+Examples are given for the same trellis.py [#trellis.py]_ script.
+
+- L2 Unicast
+
+ Hosts are under the same leaf, only interface configuration is needed.
+
+ .. code-block:: console
+
+ t3-troubleshoot-simple -v -et IPV4 00:AA:00:00:00:01/None 00:AA:00:00:00:02/None
+
+- L3 Multicast
+
+ Hosts are under different leafs, also segment routing configuration config is needed for the leaf's routerMac.
+
+ .. code-block:: console
+
+ t3-troubleshoot-simple -et IPV4 00:AA:00:00:00:01/None 00:AA:00:00:00:03/None
+
+
+t3-troubleshoot-pingall
+=======================
+T3 also offers a ping-all command to be able to test ping between any given hosts of an EthType.
+This commands bases itself on top of the simplified Host to Host command. It has two verbosity levels,
+one just with Host and Destination with the result message of the Trace and one that shows also the packet and the paths.
+
+t3-troubleshoot-pingall Help
+----------------------------
+.. code-block:: console
+
+ onos> t3-troubleshoot-pingall --help
+ DESCRIPTION
+ onos:t3-troubleshoot-pingall
+
+ Traces a ping between all hosts in the system of a given ETH type
+
+ SYNTAX
+ onos:t3-troubleshoot-pingall [options]
+
+ OPTIONS
+ -j, --json
+ Output JSON
+ --help
+ Display this help message
+ -et, --ethType
+ ETH Type
+ (defaults to ipv4)
+ -v, --verbose
+ Outputs trace for each host to host combination
+
+t3-troubleshoot-pingall Examples
+--------------------------------
+- IPv4 hosts
+
+ .. code-block:: console
+
+ t3-troubleshoot-pingall
+
+- IPv6 hosts
+
+ .. code-block:: console
+
+ t3-troubleshoot-pingall -et IPV6
+
+
+t3-troubleshoot-mcast
+=====================
+T3 also offers a mcast command queries the system for all the multicast routes given a certain vlanId.
+It prints all the traces given a certain group, source and source connect point. It has two "standard" verbosity levels.
+
+t3-troubleshoot-mcast Help
+--------------------------
+.. code-block:: console
+
+ onos> t3-troubleshoot-mcast --help
+ DESCRIPTION
+ onos:t3-troubleshoot-mcast
+
+ Traces all the mcast routes present in the system
+
+ SYNTAX
+ onos:t3-troubleshoot-mcast [options]
+
+ OPTIONS
+ -j, --json
+ Output JSON
+ --help
+ Display this help message
+ -vv, --veryverbose
+ Outputs details of every trace
+ -vid, --vlanId
+ Vlan of incoming packet
+ (defaults to None)
+ -v, --verbose
+ Outputs trace for each host to host combination
+
+REST API
+========
+T3 now also offers a way to execute the troubleshoot through REST API endpoints.
+You can find the description and test the API at the ONOS docs endpoint here:
+
+.. code-block:: text
+
+ <ONOS_IP>:8181/onos/v1/docs/#/t3
+
+The rest API offered are a non verbose and a verbose version of the troubleshoot simple command.
+It bases itself on top of some configuration present in ONOS through netcfg, namely the VLAN interface configuration and the segment routing leaf router MAC for L3 Unicast.
+The use need to specify just the Source Host, the destination host and the ETH type.
+
+.. code-block:: console
+
+ curl -X GET --header 'Accept: application/json' 'http://localhost:8181/onos/v1/t3/t3/simple/<host_src>/<host_dst>/<eht_type>'
+
+.. code-block:: console
+
+ curl -X GET --header 'Accept: application/json' 'http://localhost:8181/onos/v1/t3/t3/simple/<host_src>/<host_dst>/<eht_type>/verbose
+
+
+Last Resort
+===========
+
+In the case where you can't figure out what is going wrong even with help of T3, you can seek help on Trellis developer mailing list ``trellis-dev@opennetworking.org``.
+There are a few things we would like you to attach:
+
+- **Issue description**
+- **Environment description**, such as ONOS version, switch model and OF-DPA version
+- **Steps of reproduction**, as detail as possible
+- **Diagnostics**.
+ We have built a tool ``onos-diagnostics`` (see `ONOS Remote Admin Tools <https://wiki.onosproject.org/display/ONOS/ONOS+Remote+Admin+Tools>`_) to help you easily collect and package ONOS diagnostics
+
+
+Reference
+=========
+
+.. [#trellis.py] `<https://github.com/opennetworkinglab/routing/blob/master/trellis/trellis.py>`_