Troubleshooting Guide
*********************

Overview
========

The **Trellis Troubleshooting Tools (T3)** is an ONOS application intended to
**help you troubleshoot traffic-related problems** in your network by **tracing
installed flows and groups**.

T3 works both on simulated OVS-based topologies but also on real HW, basing
it's internal decisions on the driver assigned to the device.

T3 also currently supports dual links between leaf and spine and properly
handles the difference between deferred and immediate rules.  Dual Homed hosts
are also supported in T3.

To activate this tool on your ONOS environment, you need to run:

.. code-block:: console

    onos> app activate t3

T3 supports tracing of following packet types:

- L2 Unicast
- L2 Broadcast
- L3 Unicast (both IPv4 and IPv6)
- L3 Multicast (both IPv4 and IPv6)
- LLDP and BDDP
- ARP
- DHCP


Network Information Base (NIB)
==============================
T3 analyzes network states that are stored in the NIB.
A NIB is a collection of snapshots captured from various ONOS stores such as flows, groups, links...etc.
The NIB, acting as a cache, allows T3 troubleshooting logic to avoid massive interaction with live ONOS subsystems,
enabling near-runtime or postmortem analysis in a production environment.

There are two methods of loading the network snapshots into the NIB:
(1) from a live running system or (2) from a onos-diagnostics file dump.
Users can choose one load command before troubleshooting commands,
or can run troubleshooting commands alone (then the NIB is automatically loaded via (1)).


t3-load-file
=========================
T3 offers this command to fill the NIB with files that represent snapshots of the network states in a certain time.

t3-load-file Help
------------------------------

.. code-block:: console

    onos> t3-load-file --help
    DESCRIPTION
            onos:t3-load-file

            Load the NIB with onos-diagnostics dump files

    SYNTAX
            onos:t3-load-file [options] rootDir

    ARGUMENTS
            rootDir
                    Specify the location of the directory where the previous onos-diagnostics command has extracted its dump files (e.g. /tmp/onos-diags/127.0.0.1)

    OPTIONS
            --help
                    Display this help message
            -j, --json
                    Output JSON

t3-load-file Example
---------------------------------
User is expected to run the tool ``onos-diagnostics`` [#onos-diagnostics_src]_ [#onos-diagnostics_man]_
to generate target files first.

.. code-block:: console

    user$ onos-diagnostics -xj [IP_ONOS_INSTANCE_1 IP_ONOS_INSTANCE_2 ...]
    Collecting diagnostics on [IP_ONOS_INSTANCE_1]...logs netcfg.json devices.json device-drivers.json links.json hosts.json ports.json edge-ports.json flows.json groups.json masters.json routes.json mcast-host-show.json  Done.
    Collecting diagnostics on [IP_ONOS_INSTANCE_2]...logs netcfg.json devices.json device-drivers.json links.json hosts.json ports.json edge-ports.json flows.json groups.json masters.json routes.json mcast-host-show.json  Done.

Then, execute this command with the path of the location where those JSON files are extracted.
User can check the creation time and a summary of the NIB contents.

.. code-block:: console

    onos> t3-load-file /tmp/onos-diags/[IP_ONOS_INSTANCE_TARGET]
    Load target files in: /tmp/onos-diags/[IP_ONOS_INSTANCE_TARGET]/
    FlowNib created 03-06-2020 01:30 from FILE
    GroupNib created 03-06-2020 01:30 from FILE
    LinkNib created 03-06-2020 01:30 from FILE
    HostNib created 03-06-2020 01:30 from FILE
    DeviceNib created 03-06-2020 01:30 from FILE
    DriverNib created 03-06-2020 01:30 from FILE
    MastershipNib created 03-06-2020 01:30 from FILE
    EdgePortNib created 03-06-2020 01:30 from FILE
    RouteNib created 03-06-2020 01:30 from FILE
    NetworkConfigNib created 03-06-2020 01:30 from FILE
    MulticastRouteNib created 03-06-2020 01:30 from FILE

Alternatively, it is possible to use ``onos-diagnostics-k8s`` [#onos-diagnostics-k8s_src]_ [#onos-diagnostics_man]_
in Kubernetes enabled environments.

t3-load-snapshot
=============================
T3 offers this command to fill the NIB with snapshots of the network states at the execution time.

t3-load-snapshot Help
----------------------------------

.. code-block:: console

    onos> t3-load-snapshot --help
    DESCRIPTION
            onos:t3-load-snapshot

            Load the NIB with the network states stored in the ONOS instance where the T3 is running

    SYNTAX
            onos:t3-load-snapshot [options]

    OPTIONS
            --help
                    Display this help message
            -j, --json
                    Output JSON

t3-load-snapshot Example
-------------------------------------
User can check the creation time and a summary of the NIB contents.
**SNAPSHOT** indicates that the source of the current NIB is the ONOS instance where the T3 is running.

.. code-block:: console

    onos> t3-load-snapshot
    Load current network states from ONOS stores
    FlowNib created 03-06-2020 01:40 from SNAPSHOT
    GroupNib created 03-06-2020 01:40 from SNAPSHOT
    LinkNib created 03-06-2020 01:40 from SNAPSHOT
    HostNib created 03-06-2020 01:40 from SNAPSHOT
    DeviceNib created 03-06-2020 01:40 from SNAPSHOT
    DriverNib created 03-06-2020 01:40 from SNAPSHOT
    MastershipNib created 03-06-2020 01:40 from SNAPSHOT
    EdgePortNib created 03-06-2020 01:40 from SNAPSHOT
    RouteNib created 03-06-2020 01:40 from SNAPSHOT
    NetworkConfigNib created 03-06-2020 01:40 from SNAPSHOT
    MulticastRouteNib created 03-06-2020 01:40 from SNAPSHOT

If one of T3 commands below is used before loading the NIB from **SNAPSHOT** or **FILE** (so the NIB is empty),
the command internally executes **t3-load-snapshot** to proceed with auto-filled NIB.

.. code-block:: console

    onos> t3-troubleshoot-simple 00:AA:00:00:00:01/None 00:AA:00:00:00:02/None
    *** NIB is invalid. Snapshots for the NIB have been auto-filled: ***
    Load current network states from ONOS stores
    FlowNib created 03-06-2020 01:44 from SNAPSHOT
    GroupNib created 03-06-2020 01:44 from SNAPSHOT
    LinkNib created 03-06-2020 01:44 from SNAPSHOT
    ...
    Tracing between: 00:AA:00:00:00:01/None and 00:AA:00:00:00:02/None
    Tracing Packet: DefaultTrafficSelector{criteria=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.2.2/32]}
    Paths
    [of:0000000000000204/3, of:0000000000000204/4]
    Result:
    Reached required destination Host of:0000000000000204/4


t3-troubleshoot
===============

As of right now the tools provides a CLI command (troubleshoot).

As arguments to the CLI command you have to pass arguments describing the
packet you want to trace through the network.

t3-troubleshoot Help
--------------------

.. code-block:: console

    onos> t3-troubleshoot --help
    DESCRIPTION
            onos:troubleshoot

        troubleshoots flows and groups between source and destination

    SYNTAX
            onos:troubleshoot [options]

    OPTIONS
            -sm, --srcMac
                    Source MAC
            -j, --json
                    Output JSON
            --help
                    Display this help message
            -v, --verbose
                    Outputs complete path
            -ipp, --ipProto
                    IP Proto
            -s, --srcIp
                    Source IP
            -dm, --dstMac
                    Destination MAC
            -ml, --mplsLabel
                    Mpls label of incoming packet
            -vv, --veryverbose
                    Outputs flows and groups for every device
            -udps, --udpSrc
                    UDP Source
            -mb, --mplsBos
                    MPLS BOS
            -et, --ethType
                    ETH Type
                    (defaults to ipv4)
            -sp, --srcPort
                    Source Port
            -dtp, --dstTcpPort
                    destination TCP Port
            -udpd, --udpDst
                    UDP Destination
            -stp, --srcTcpPort
                    Source TCP Port
            -d, --dstIp
                    Destination IP
            -vid, --vlanId
                    Vlan of incoming packet
                    (defaults to None)

Default Verbosity Level
-----------------------
Let's look at an example.

.. code-block:: console

    onos> t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:00:00:00:02:04 -vid None

with this command we are asking the t3 tool to trace a packet with

- Source IP of ``10.0.2.1/32``
- Input port ``of:0000000000000204/3``
- Source MAC ``00:AA:00:00:00:01``
- ethType ``IPv4``
- Source TCP Port ``12345``
- Destination IP ``10.0.3.1/32``
- Destination MAC ``00:AA:00:00:00:01``
- Destination TCP Port ``80`` and
- VLAN ID ``None``

This command has a verbosity level of 0 and will output just the paths that the packet might take inside the network.
In the example deployment taken from trellis.py [#trellis.py]_, the output will be:

.. code-block:: console

    Tracing packet: [IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.3.1/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true]
    Paths
    [of:0000000000000204/3, of:0000000000000204/2, of:0000000000000227/1, of:0000000000000227/2, of:0000000000000205/2, of:0000000000000205/3]
    [of:0000000000000204/3, of:0000000000000204/1, of:0000000000000226/1, of:0000000000000226/2, of:0000000000000205/1, of:0000000000000205/3]
    Result:
    Reached required destination Host
    Reached required destination Host

Middle Verbosity Level
----------------------
There are also two other verbosity levels.

.. code-block:: console

    onos> t3-troubleshoot -v -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:00:00:00:02:04 -vid None

prints flows selectors and groupId that are met along the path

.. code-block:: console

    Tracing packet: [IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.3.1/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true]
    Path [of:0000000000000204/3, of:0000000000000204/2, of:0000000000000227/1, of:0000000000000227/2, of:0000000000000205/2, of:0000000000000205/3]
    Device of:0000000000000204
    Input from of:0000000000000204/3
    Flows
    flowId=a20000bd57cb49, selector=[]
    flowId=900000d4173ff7, selector=[IN_PORT:3, VLAN_VID:None]
    flowId=9000004b3d1965, selector=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:20]
    flowId=900000499769f5, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.0/24]
    flowId=a2000038b487e7, selector=[]
    Groups
    groupId=GroupId{id=0x70000010}
    groupId=GroupId{id=0x9200000d}
    groupId=GroupId{id=0x9000000c}
    groupId=GroupId{id=0xffe0002}
    groupId=GroupId{id=0x9200000f}
    groupId=GroupId{id=0x9000000e}
    groupId=GroupId{id=0xffe0001}
    Output through of:0000000000000204/2
    Device of:0000000000000227
    Input from of:0000000000000227/1
    Flows
    flowId=a200005577bcd6, selector=[]
    flowId=900000498457b2, selector=[IN_PORT:1, VLAN_VID:None]
    flowId=900000f87746c9, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:27, ETH_TYPE:mpls_unicast, VLAN_VID:4094]
    flowId=a200005bfd6ed7, selector=[]
    flowId=9000009f3d554f, selector=[ETH_TYPE:mpls_unicast, MPLS_LABEL:205, MPLS_BOS:true]
    flowId=a20000e601fc35, selector=[]
    Groups
    groupId=GroupId{id=0x70000017}
    groupId=GroupId{id=0x20000016}
    groupId=GroupId{id=0xffe0002}
    Output through of:0000000000000227/2
    Device of:0000000000000205
    Input from of:0000000000000205/2
    Flows
    flowId=a200008d542817, selector=[]
    flowId=900000be3179a0, selector=[IN_PORT:1, VLAN_VID:None]
    flowId=900000766be164, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:05, ETH_TYPE:ipv4, VLAN_VID:4094]
    flowId=900000ec1d5779, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.1/32]
    flowId=a2000042d8e6b5, selector=[]
    Groups
    groupId=GroupId{id=0x2000002a}
    groupId=GroupId{id=0x1e0003}
    groupId=GroupId{id=0x2000002a}
    groupId=GroupId{id=0x1e0003}
    Output through of:0000000000000205/3
    Path [of:0000000000000204/3, of:0000000000000204/1, of:0000000000000226/1, of:0000000000000226/2, of:0000000000000205/1, of:0000000000000205/3]
    Device of:0000000000000204
    Input from of:0000000000000204/3
    Flows
    flowId=a20000bd57cb49, selector=[]
    flowId=900000d4173ff7, selector=[IN_PORT:3, VLAN_VID:None]
    flowId=9000004b3d1965, selector=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:20]
    flowId=900000499769f5, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.0/24]
    flowId=a2000038b487e7, selector=[]
    Groups
    groupId=GroupId{id=0x70000010}
    groupId=GroupId{id=0x9200000d}
    groupId=GroupId{id=0x9000000c}
    groupId=GroupId{id=0xffe0002}
    groupId=GroupId{id=0x9200000f}
    groupId=GroupId{id=0x9000000e}
    groupId=GroupId{id=0xffe0001}
    Output through of:0000000000000204/1
    Device of:0000000000000226
    Input from of:0000000000000226/1
    Flows
    flowId=a20000e9b27d24, selector=[]
    flowId=9000004bd76105, selector=[IN_PORT:1, VLAN_VID:None]
    flowId=90000069f274d1, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:26, ETH_TYPE:mpls_unicast, VLAN_VID:4094]
    flowId=a200000ec7529e, selector=[]
    flowId=90000050f9a350, selector=[ETH_TYPE:mpls_unicast, MPLS_LABEL:205, MPLS_BOS:true]
    flowId=a20000b15cada6, selector=[]
    Groups
    groupId=GroupId{id=0x7000000b}
    groupId=GroupId{id=0x20000009}
    groupId=GroupId{id=0xffe0002}
    Output through of:0000000000000226/2
    Device of:0000000000000205
    Input from of:0000000000000205/1
    Flows
    flowId=a200008d542817, selector=[]
    flowId=900000be3179a0, selector=[IN_PORT:1, VLAN_VID:None]
    flowId=900000766be164, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:05, ETH_TYPE:ipv4, VLAN_VID:4094]
    flowId=900000ec1d5779, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.1/32]
    flowId=a2000042d8e6b5, selector=[]
    Groups
    groupId=GroupId{id=0x2000002a}
    groupId=GroupId{id=0x1e0003}
    groupId=GroupId{id=0x2000002a}
    groupId=GroupId{id=0x1e0003}
    Output through of:0000000000000205/3
    Result:
    Reached required destination Host
    Reached required destination Host

High Verbosity Level
--------------------
The most verbose output you can get is

.. code-block:: console

    onos> t3-troubleshoot -vv -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:00:00:00:02:04 -vid None

This verbosity level prints all the flows and groups that are met along the
path:

.. code-block:: console

    Tracing packet: [IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.3.1/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true]
    Path [of:0000000000000204/3, of:0000000000000204/2, of:0000000000000227/1, of:0000000000000227/2, of:0000000000000205/2, of:0000000000000205/3]
    Device of:0000000000000204
    Input from of:0000000000000204/3
    Flows
    ADDED, bytes=733506, packets=8691, table=0, priority=0, selector=[], treatment=[transition=TABLE:10]
    ADDED, bytes=11110, packets=61, table=10, priority=32768, selector=[IN_PORT:3, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:20], transition=TABLE:20]
    ADDED, bytes=8698, packets=29, table=20, priority=32768, selector=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:20], treatment=[transition=TABLE:30]
    ADDED, bytes=16612, packets=50, table=30, priority=48010, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.0/24], treatment=[deferred=[GROUP:0x70000010], transition=TABLE:60]
    ADDED, bytes=34400, packets=112, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]]
    Groups
    id=0x70000010, state=ADDED, type=SELECT, bytes=16906, packets=53, appId=org.onosproject.segmentrouting, referenceCount=3
    id=0x70000010, bucket=1, bytes=490, packets=5, actions=[GROUP:0x9200000d]
    id=0x70000010, bucket=2, bytes=16416, packets=48, actions=[GROUP:0x9200000f]
    id=0x9200000d, state=ADDED, type=INDIRECT, bytes=490, packets=5, appId=org.onosproject.segmentrouting, referenceCount=0
    id=0x9200000d, bucket=1, bytes=490, packets=5, actions=[VLAN_POP, MPLS_PUSH:mpls_unicast, MPLS_LABEL:205, GROUP:0x9000000c, VLAN_PUSH:vlan, VLAN_ID:4095]
    id=0x9000000c, state=ADDED, type=INDIRECT, bytes=490, packets=5, appId=org.onosproject.segmentrouting, referenceCount=0
    id=0x9000000c, bucket=1, bytes=490, packets=5, actions=[ETH_DST:00:00:00:00:02:27, ETH_SRC:00:00:00:00:02:04, VLAN_ID:4094, GROUP:0xffe0002]
    id=0xffe0002, state=ADDED, type=INDIRECT, bytes=726, packets=7, appId=org.onosproject.segmentrouting, referenceCount=0
    id=0xffe0002, bucket=1, bytes=726, packets=7, actions=[VLAN_POP, OUTPUT:2]
    id=0x9200000f, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
    id=0x9200000f, bucket=1, bytes=16416, packets=48, actions=[VLAN_POP, MPLS_PUSH:mpls_unicast, MPLS_LABEL:205, GROUP:0x9000000e, VLAN_PUSH:vlan, VLAN_ID:4095]
    id=0x9000000e, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
    id=0x9000000e, bucket=1, bytes=16416, packets=48, actions=[ETH_DST:00:00:00:00:02:26, ETH_SRC:00:00:00:00:02:04, VLAN_ID:4094, GROUP:0xffe0001]
    id=0xffe0001, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
    id=0xffe0001, bucket=1, bytes=16416, packets=48, actions=[VLAN_POP, OUTPUT:1]
    Output through of:0000000000000204/2
    Device of:0000000000000227
    Input from of:0000000000000227/1
    Flows
    ADDED, bytes=702318, packets=8511, table=0, priority=0, selector=[], treatment=[transition=TABLE:10]
    ADDED, bytes=343130, packets=4234, table=10, priority=32768, selector=[IN_PORT:1, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:4094], transition=TABLE:20]
    ADDED, bytes=754, packets=7, table=20, priority=32768, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:27, ETH_TYPE:mpls_unicast, VLAN_VID:4094], treatment=[transition=TABLE:23]
    ADDED, bytes=17566, packets=57, table=23, priority=0, selector=[], treatment=[transition=TABLE:24]
    ADDED, bytes=510, packets=5, table=24, priority=100, selector=[ETH_TYPE:mpls_unicast, MPLS_LABEL:205, MPLS_BOS:true], treatment=[deferred=[MPLS_POP:ipv4, DEC_TTL, GROUP:0x70000017], transition=TABLE:60]
    ADDED, bytes=17566, packets=57, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]]
    Groups
    id=0x70000017, state=ADDED, type=SELECT, bytes=754, packets=7, appId=org.onosproject.segmentrouting, referenceCount=2
    id=0x70000017, bucket=1, bytes=754, packets=7, actions=[GROUP:0x20000016]
    id=0x20000016, state=ADDED, type=INDIRECT, bytes=754, packets=7, appId=org.onosproject.segmentrouting, referenceCount=0
    id=0x20000016, bucket=1, bytes=754, packets=7, actions=[ETH_DST:00:00:00:00:02:05, ETH_SRC:00:00:00:00:02:27, VLAN_ID:4094, GROUP:0xffe0002]
    id=0xffe0002, state=ADDED, type=INDIRECT, bytes=754, packets=7, appId=org.onosproject.segmentrouting, referenceCount=0
    id=0xffe0002, bucket=1, bytes=754, packets=7, actions=[VLAN_POP, OUTPUT:2]
    Output through of:0000000000000227/2
    Device of:0000000000000205
    Input from of:0000000000000205/2
    Flows
    ADDED, bytes=2105886, packets=24084, table=0, priority=0, selector=[], treatment=[transition=TABLE:10]
    ADDED, bytes=358792, packets=4275, table=10, priority=32768, selector=[IN_PORT:1, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:4094], transition=TABLE:20]
    ADDED, bytes=16416, packets=48, table=20, priority=32768, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:05, ETH_TYPE:ipv4, VLAN_VID:4094], treatment=[transition=TABLE:30]
    ADDED, bytes=196, packets=2, table=30, priority=64010, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.1/32], treatment=[deferred=[GROUP:0x2000002a], transition=TABLE:60]
    ADDED, bytes=1370884, packets=15163, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]]
    Groups
    id=0x2000002a, state=ADDED, type=INDIRECT, bytes=196, packets=2, appId=org.onosproject.segmentrouting, referenceCount=1
    id=0x2000002a, bucket=1, bytes=196, packets=2, actions=[ETH_DST:00:AA:00:00:00:03, ETH_SRC:00:00:00:00:02:05, VLAN_ID:30, GROUP:0x1e0003]
    id=0x1e0003, state=ADDED, type=INDIRECT, bytes=12960, packets=126, appId=org.onosproject.segmentrouting, referenceCount=1
    id=0x1e0003, bucket=1, bytes=12960, packets=126, actions=[VLAN_POP, OUTPUT:3]
    id=0x2000002a, state=ADDED, type=INDIRECT, bytes=196, packets=2, appId=org.onosproject.segmentrouting, referenceCount=1
    id=0x2000002a, bucket=1, bytes=196, packets=2, actions=[ETH_DST:00:AA:00:00:00:03, ETH_SRC:00:00:00:00:02:05, VLAN_ID:30, GROUP:0x1e0003]
    id=0x1e0003, state=ADDED, type=INDIRECT, bytes=12960, packets=126, appId=org.onosproject.segmentrouting, referenceCount=1
    id=0x1e0003, bucket=1, bytes=12960, packets=126, actions=[VLAN_POP, OUTPUT:3]
    Output through of:0000000000000205/3
    Path [of:0000000000000204/3, of:0000000000000204/1, of:0000000000000226/1, of:0000000000000226/2, of:0000000000000205/1, of:0000000000000205/3]
    Device of:0000000000000204
    Input from of:0000000000000204/3
    Flows
    ADDED, bytes=733506, packets=8691, table=0, priority=0, selector=[], treatment=[transition=TABLE:10]
    ADDED, bytes=11110, packets=61, table=10, priority=32768, selector=[IN_PORT:3, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:20], transition=TABLE:20]
    ADDED, bytes=8698, packets=29, table=20, priority=32768, selector=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:20], treatment=[transition=TABLE:30]
    ADDED, bytes=16612, packets=50, table=30, priority=48010, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.0/24], treatment=[deferred=[GROUP:0x70000010], transition=TABLE:60]
    ADDED, bytes=34400, packets=112, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]]
    Groups
    id=0x70000010, state=ADDED, type=SELECT, bytes=16906, packets=53, appId=org.onosproject.segmentrouting, referenceCount=3
    id=0x70000010, bucket=1, bytes=490, packets=5, actions=[GROUP:0x9200000d]
    id=0x70000010, bucket=2, bytes=16416, packets=48, actions=[GROUP:0x9200000f]
    id=0x9200000d, state=ADDED, type=INDIRECT, bytes=490, packets=5, appId=org.onosproject.segmentrouting, referenceCount=0
    id=0x9200000d, bucket=1, bytes=490, packets=5, actions=[VLAN_POP, MPLS_PUSH:mpls_unicast, MPLS_LABEL:205, GROUP:0x9000000c, VLAN_PUSH:vlan, VLAN_ID:4095]
    id=0x9000000c, state=ADDED, type=INDIRECT, bytes=490, packets=5, appId=org.onosproject.segmentrouting, referenceCount=0
    id=0x9000000c, bucket=1, bytes=490, packets=5, actions=[ETH_DST:00:00:00:00:02:27, ETH_SRC:00:00:00:00:02:04, VLAN_ID:4094, GROUP:0xffe0002]
    id=0xffe0002, state=ADDED, type=INDIRECT, bytes=726, packets=7, appId=org.onosproject.segmentrouting, referenceCount=0
    id=0xffe0002, bucket=1, bytes=726, packets=7, actions=[VLAN_POP, OUTPUT:2]
    id=0x9200000f, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
    id=0x9200000f, bucket=1, bytes=16416, packets=48, actions=[VLAN_POP, MPLS_PUSH:mpls_unicast, MPLS_LABEL:205, GROUP:0x9000000e, VLAN_PUSH:vlan, VLAN_ID:4095]
    id=0x9000000e, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
    id=0x9000000e, bucket=1, bytes=16416, packets=48, actions=[ETH_DST:00:00:00:00:02:26, ETH_SRC:00:00:00:00:02:04, VLAN_ID:4094, GROUP:0xffe0001]
    id=0xffe0001, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
    id=0xffe0001, bucket=1, bytes=16416, packets=48, actions=[VLAN_POP, OUTPUT:1]
    Output through of:0000000000000204/1
    Device of:0000000000000226
    Input from of:0000000000000226/1
    Flows
    ADDED, bytes=701910, packets=8507, table=0, priority=0, selector=[], treatment=[transition=TABLE:10]
    ADDED, bytes=358984, packets=4275, table=10, priority=32768, selector=[IN_PORT:1, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:4094], transition=TABLE:20]
    ADDED, bytes=16608, packets=48, table=20, priority=32768, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:26, ETH_TYPE:mpls_unicast, VLAN_VID:4094], treatment=[transition=TABLE:23]
    ADDED, bytes=17158, packets=53, table=23, priority=0, selector=[], treatment=[transition=TABLE:24]
    ADDED, bytes=16608, packets=48, table=24, priority=100, selector=[ETH_TYPE:mpls_unicast, MPLS_LABEL:205, MPLS_BOS:true], treatment=[deferred=[MPLS_POP:ipv4, DEC_TTL, GROUP:0x7000000b], transition=TABLE:60]
    ADDED, bytes=17158, packets=53, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]]
    Groups
    id=0x7000000b, state=ADDED, type=SELECT, bytes=16608, packets=48, appId=org.onosproject.segmentrouting, referenceCount=2
    id=0x7000000b, bucket=1, bytes=16608, packets=48, actions=[GROUP:0x20000009]
    id=0x20000009, state=ADDED, type=INDIRECT, bytes=16608, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
    id=0x20000009, bucket=1, bytes=16608, packets=48, actions=[ETH_DST:00:00:00:00:02:05, ETH_SRC:00:00:00:00:02:26, VLAN_ID:4094, GROUP:0xffe0002]
    id=0xffe0002, state=ADDED, type=INDIRECT, bytes=16608, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0
    id=0xffe0002, bucket=1, bytes=16608, packets=48, actions=[VLAN_POP, OUTPUT:2]
    Output through of:0000000000000226/2
    Device of:0000000000000205
    Input from of:0000000000000205/1
    Flows
    ADDED, bytes=2105886, packets=24084, table=0, priority=0, selector=[], treatment=[transition=TABLE:10]
    ADDED, bytes=358792, packets=4275, table=10, priority=32768, selector=[IN_PORT:1, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:4094], transition=TABLE:20]
    ADDED, bytes=16416, packets=48, table=20, priority=32768, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:05, ETH_TYPE:ipv4, VLAN_VID:4094], treatment=[transition=TABLE:30]
    ADDED, bytes=196, packets=2, table=30, priority=64010, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.1/32], treatment=[deferred=[GROUP:0x2000002a], transition=TABLE:60]
    ADDED, bytes=1370884, packets=15163, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]]
    Groups
    id=0x2000002a, state=ADDED, type=INDIRECT, bytes=196, packets=2, appId=org.onosproject.segmentrouting, referenceCount=1
    id=0x2000002a, bucket=1, bytes=196, packets=2, actions=[ETH_DST:00:AA:00:00:00:03, ETH_SRC:00:00:00:00:02:05, VLAN_ID:30, GROUP:0x1e0003]
    id=0x1e0003, state=ADDED, type=INDIRECT, bytes=12960, packets=126, appId=org.onosproject.segmentrouting, referenceCount=1
    id=0x1e0003, bucket=1, bytes=12960, packets=126, actions=[VLAN_POP, OUTPUT:3]
    id=0x2000002a, state=ADDED, type=INDIRECT, bytes=196, packets=2, appId=org.onosproject.segmentrouting, referenceCount=1
    id=0x2000002a, bucket=1, bytes=196, packets=2, actions=[ETH_DST:00:AA:00:00:00:03, ETH_SRC:00:00:00:00:02:05, VLAN_ID:30, GROUP:0x1e0003]
    id=0x1e0003, state=ADDED, type=INDIRECT, bytes=12960, packets=126, appId=org.onosproject.segmentrouting, referenceCount=1
    id=0x1e0003, bucket=1, bytes=12960, packets=126, actions=[VLAN_POP, OUTPUT:3]
    Output through of:0000000000000205/3
    Result:
    Reached required destination Host
    Reached required destination Host

Successful Example
------------------

In this example we trace a packet that should be bridged between two hosts in
the same subnet

.. code-block:: console

    onos> t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:AA:00:00:00:02 -vid None
    Tracing packet: [IN_PORT:3, ETH_DST:00:AA:00:00:00:02, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.2.2/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true]
    Paths
    [of:0000000000000204/3, of:0000000000000204/4]
    Result:
    Reached required destination Host

Failed Example
--------------

Obviously failures can happen along one of the paths that the packet can take.

In our sample we removed a flow in ``of:0000000000000226`` thus disabling that
possible path for our initial packet.

The output of our trace would then look something like:

.. code-block:: console

    onos> t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:00:00:00:02:04 -vid None
    Tracing packet: [IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.3.1/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true]
    Paths
    [of:0000000000000204/3, of:0000000000000204/2, of:0000000000000227/1, of:0000000000000227/2, of:0000000000000205/2, of:0000000000000205/3]
    [of:0000000000000204/3, of:0000000000000204/1, of:0000000000000226/1]
    Result:
    Reached required destination Host
    No output out of device of:0000000000000226 Packet is dropped

Notice that on the second path that was previously going through
``of:0000000000000226`` now the packet is dropped due to the rule we removed.

Example of Various Packet Types
-------------------------------
These sample commands refer to the Mininet environment found in trellis.py [#trellis.py]_

- L2 Unicast IPv4

  .. code-block:: console

      t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -d 10.0.2.2 -dm 00:AA:00:00:00:02

- L2 Unicast IPv6

  .. code-block:: console

      t3-troubleshoot -vv -s 2000::301  -sp of:0000000000000205/5 -sm 00:BB:00:00:00:03 -d 2000::302 -dm 00:BB:00:00:00:04 -et ipv6

- L3 Unicast IPv4

  .. code-block:: console

      t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -d 10.0.3.1 -dm 00:00:00:00:02:04

- L3 Unicast IPv6

  .. code-block:: console

      t3-troubleshoot -vv -s 2000::301  -sp of:0000000000000205/5 -sm 00:BB:00:00:00:03 -d 2000::202 -dm 00:00:00:00:02:05 -et ipv6

- ARP

  .. code-block:: console

      t3-troubleshoot -sp of:0000000000000204/3 -et ARP

- DHCP

  .. code-block:: console

      t3-troubleshoot -vv -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -d 10.0.3.253 -dm 00:00:00:00:02:04 -ipp 17 -udps 67 -udpd 67

- LLDP

  .. code-block:: console

      t3-troubleshoot -sp of:0000000000000204/3 -et lldp

- BDDP

  .. code-block:: console

      t3-troubleshoot -sp of:0000000000000204/3 -et bddp

- LLDP from CONTROLLER

  T3 is also capable of tracing packets coming in from the CONTROLLER.

  We use the same input port with the possibility to give CONTROLLER as logical
  port number.

  This is a mere convenience to carry the information inside T3 and do the
  proper flooding to all the ports of the given device.

  This case is not treated like all the rest and the input port here is a
  logical entity that has no match on any physical element on the device.

  .. code-block:: console

      t3-troubleshoot -sp of:0000000000000204/CONTROLLER -et lldp

- Multicast

  T3 is also capable of tracing multicast packets.

  .. code-block:: console

      t3-troubleshoot -vv -sp of:0000000000000205/9 -et ipv4 -d 224.0.0.1 -dm 01:00:5e:00:00:01


t3-troubleshoot-simple
======================

T3 also offers a simplified version of the command to be able to ping between
two given hosts without having to specify all the information.

This commands bases itself on top of some configuration present in ONOS through
netcfg, namely the VLAN interface configuration and the segment routing leaf
router MAC for L3 Unicast. If that is not present the user needs to input this
information manually recurring to the the previously shown command. Same
verbosity levels and output as the troubleshoot command are possible.

t3-troubleshoot-simple Help
---------------------------
.. code-block:: console

    onos> t3-troubleshoot-simple --help
    DESCRIPTION
            onos:troubleshoot-simple

        Given two hosts troubleshoots flows and groups between them, in case of segment routing

    SYNTAX
            onos:troubleshoot-simple [options] one two

    ARGUMENTS
            one
                    One host ID
            two
                    Another host ID

    OPTIONS
            -vv, --veryverbose
                    Outputs flows and groups for every device
            -j, --json
                    Output JSON
            --help
                    Display this help message
            -et, --ethType
                    ETH Type
                    (defaults to ipv4)
            -v, --verbose
                    Outputs complete path

t3-troubleshoot-simple Example
------------------------------
Examples are given for the same trellis.py [#trellis.py]_ script.

- L2 Unicast

  Hosts are under the same leaf, only interface configuration is needed.

  .. code-block:: console

      t3-troubleshoot-simple -v -et IPV4 00:AA:00:00:00:01/None 00:AA:00:00:00:02/None

- L3 Multicast

  Hosts are under different leafs, also segment routing configuration config is needed for the leaf's routerMac.

  .. code-block:: console

      t3-troubleshoot-simple -et IPV4 00:AA:00:00:00:01/None 00:AA:00:00:00:03/None


t3-troubleshoot-pingall
=======================
T3 also offers a ping-all command to be able to test ping between any given hosts of an EthType.
This commands bases itself on top of the simplified Host to Host command. It has two verbosity levels,
one just with Host and Destination with the result message of the Trace and one that shows also the packet and the paths.

t3-troubleshoot-pingall Help
----------------------------
.. code-block:: console

    onos> t3-troubleshoot-pingall --help
    DESCRIPTION
            onos:t3-troubleshoot-pingall

        Traces a ping between all hosts in the system of a given ETH type

    SYNTAX
            onos:t3-troubleshoot-pingall [options]

    OPTIONS
            -j, --json
                    Output JSON
            --help
                    Display this help message
            -et, --ethType
                    ETH Type
                    (defaults to ipv4)
            -v, --verbose
                    Outputs trace for each host to host combination

t3-troubleshoot-pingall Examples
--------------------------------
- IPv4 hosts

  .. code-block:: console

      t3-troubleshoot-pingall

- IPv6 hosts

  .. code-block:: console

      t3-troubleshoot-pingall -et IPV6


t3-troubleshoot-mcast
=====================

T3 also offers a mcast command queries the system for all the multicast routes
given a certain vlanId.  It prints all the traces given a certain group, source
and source connect point. It has two "standard" verbosity levels.

t3-troubleshoot-mcast Help
--------------------------
.. code-block:: console

    onos> t3-troubleshoot-mcast --help
    DESCRIPTION
            onos:t3-troubleshoot-mcast

        Traces all the mcast routes present in the system

    SYNTAX
            onos:t3-troubleshoot-mcast [options]

    OPTIONS
            -j, --json
                    Output JSON
            --help
                    Display this help message
            -vv, --veryverbose
                    Outputs details of every trace
            -vid, --vlanId
                    Vlan of incoming packet
                    (defaults to None)
            -v, --verbose
                    Outputs trace for each host to host combination

REST API
========
T3 now also offers a way to execute the troubleshoot through REST API endpoints.
You can find the description and test the API at the ONOS docs endpoint here:

.. code-block:: text

    <ONOS_IP>:8181/onos/v1/docs/#/t3

The rest API offered are a non verbose and a verbose version of the troubleshoot simple command.

It bases itself on top of some configuration present in ONOS through netcfg,
namely the VLAN interface configuration and the segment routing leaf router MAC
for L3 Unicast.

The use need to specify just the Source Host, the destination host and the ETH
type.

.. code-block:: console

    curl -X GET --header 'Accept: application/json' 'http://localhost:8181/onos/v1/t3/t3/simple/<host_src>/<host_dst>/<eht_type>'

.. code-block:: console

    curl -X GET --header 'Accept: application/json' 'http://localhost:8181/onos/v1/t3/t3/simple/<host_src>/<host_dst>/<eht_type>/verbose


Last Resort
===========

In the case where you can't figure out what is going wrong even with help of
T3, you can seek help on Trellis developer mailing list
``trellis-dev@opennetworking.org``.

There are a few things we would like you to attach:

- **Issue description**

- **Environment description**, such as ONOS version, switch model and OF-DPA
  version

- **Steps of reproduction**, as detail as possible

- **Diagnostics**.

We have built a tool ``onos-diagnostics`` [#onos-diagnostics_src]_ [#onos-diagnostics_man]_
to help you easily collect and package ONOS diagnostics.

Alternatively, you can use ``onos-diagnostics-k8s`` [#onos-diagnostics-k8s_src]_ [#onos-diagnostics_man]_.

Reference
=========

.. [#onos-diagnostics_src] `<https://github.com/opennetworkinglab/onos/blob/master/tools/package/runtime/bin/onos-diagnostics>`_
.. [#onos-diagnostics_man] `<https://wiki.onosproject.org/display/ONOS/ONOS+Remote+Admin+Tools>`_
.. [#onos-diagnostics-k8s_src] `<https://github.com/opennetworkinglab/onos/blob/master/tools/package/runtime/bin/onos-diagnostics-k8s>`_
.. [#trellis.py] `<https://github.com/opennetworkinglab/routing/blob/master/trellis/trellis.py>`_