| Troubleshooting Guide |
| ********************* |
| |
| Overview |
| ======== |
| |
| The **Trellis Troubleshooting Tools (T3)** is an ONOS application intended to |
| **help you troubleshoot traffic-related problems** in your network by **tracing |
| installed flows and groups**. |
| |
| T3 works both on simulated OVS-based topologies but also on real HW, basing |
| it's internal decisions on the driver assigned to the device. |
| |
| T3 also currently supports dual links between leaf and spine and properly |
| handles the difference between deferred and immediate rules. Dual Homed hosts |
| are also supported in T3. |
| |
| To activate this tool on your ONOS environment, you need to run: |
| |
| .. code-block:: console |
| |
| onos> app activate t3 |
| |
| T3 supports tracing of following packet types: |
| |
| - L2 Unicast |
| - L2 Broadcast |
| - L3 Unicast (both IPv4 and IPv6) |
| - L3 Multicast (both IPv4 and IPv6) |
| - LLDP and BDDP |
| - ARP |
| - DHCP |
| |
| |
| Network Information Base (NIB) |
| ============================== |
| T3 analyzes network states that are stored in the NIB. |
| A NIB is a collection of snapshots captured from various ONOS stores such as flows, groups, links...etc. |
| The NIB, acting as a cache, allows T3 troubleshooting logic to avoid massive interaction with live ONOS subsystems, |
| enabling near-runtime or postmortem analysis in a production environment. |
| |
| There are two methods of loading the network snapshots into the NIB: |
| (1) from a live running system or (2) from a onos-diagnostics file dump. |
| Users can choose one load command before troubleshooting commands, |
| or can run troubleshooting commands alone (then the NIB is automatically loaded via (1)). |
| |
| |
| t3-load-file |
| ========================= |
| T3 offers this command to fill the NIB with files that represent snapshots of the network states in a certain time. |
| |
| t3-load-file Help |
| ------------------------------ |
| |
| .. code-block:: console |
| |
| onos> t3-load-file --help |
| DESCRIPTION |
| onos:t3-load-file |
| |
| Load the NIB with onos-diagnostics dump files |
| |
| SYNTAX |
| onos:t3-load-file [options] rootDir |
| |
| ARGUMENTS |
| rootDir |
| Specify the location of the directory where the previous onos-diagnostics command has extracted its dump files (e.g. /tmp/onos-diags/127.0.0.1) |
| |
| OPTIONS |
| --help |
| Display this help message |
| -j, --json |
| Output JSON |
| |
| t3-load-file Example |
| --------------------------------- |
| User is expected to run the tool ``onos-diagnostics`` [#onos-diagnostics_src]_ [#onos-diagnostics_man]_ |
| to generate target files first. |
| |
| .. code-block:: console |
| |
| user$ onos-diagnostics -xj [IP_ONOS_INSTANCE_1 IP_ONOS_INSTANCE_2 ...] |
| Collecting diagnostics on [IP_ONOS_INSTANCE_1]...logs netcfg.json devices.json device-drivers.json links.json hosts.json ports.json edge-ports.json flows.json groups.json masters.json routes.json mcast-host-show.json Done. |
| Collecting diagnostics on [IP_ONOS_INSTANCE_2]...logs netcfg.json devices.json device-drivers.json links.json hosts.json ports.json edge-ports.json flows.json groups.json masters.json routes.json mcast-host-show.json Done. |
| |
| Then, execute this command with the path of the location where those JSON files are extracted. |
| User can check the creation time and a summary of the NIB contents. |
| |
| .. code-block:: console |
| |
| onos> t3-load-file /tmp/onos-diags/[IP_ONOS_INSTANCE_TARGET] |
| Load target files in: /tmp/onos-diags/[IP_ONOS_INSTANCE_TARGET]/ |
| FlowNib created 03-06-2020 01:30 from FILE |
| GroupNib created 03-06-2020 01:30 from FILE |
| LinkNib created 03-06-2020 01:30 from FILE |
| HostNib created 03-06-2020 01:30 from FILE |
| DeviceNib created 03-06-2020 01:30 from FILE |
| DriverNib created 03-06-2020 01:30 from FILE |
| MastershipNib created 03-06-2020 01:30 from FILE |
| EdgePortNib created 03-06-2020 01:30 from FILE |
| RouteNib created 03-06-2020 01:30 from FILE |
| NetworkConfigNib created 03-06-2020 01:30 from FILE |
| MulticastRouteNib created 03-06-2020 01:30 from FILE |
| |
| Alternatively, it is possible to use ``onos-diagnostics-k8s`` [#onos-diagnostics-k8s_src]_ [#onos-diagnostics_man]_ |
| in Kubernetes enabled environments. |
| |
| t3-load-snapshot |
| ============================= |
| T3 offers this command to fill the NIB with snapshots of the network states at the execution time. |
| |
| t3-load-snapshot Help |
| ---------------------------------- |
| |
| .. code-block:: console |
| |
| onos> t3-load-snapshot --help |
| DESCRIPTION |
| onos:t3-load-snapshot |
| |
| Load the NIB with the network states stored in the ONOS instance where the T3 is running |
| |
| SYNTAX |
| onos:t3-load-snapshot [options] |
| |
| OPTIONS |
| --help |
| Display this help message |
| -j, --json |
| Output JSON |
| |
| t3-load-snapshot Example |
| ------------------------------------- |
| User can check the creation time and a summary of the NIB contents. |
| **SNAPSHOT** indicates that the source of the current NIB is the ONOS instance where the T3 is running. |
| |
| .. code-block:: console |
| |
| onos> t3-load-snapshot |
| Load current network states from ONOS stores |
| FlowNib created 03-06-2020 01:40 from SNAPSHOT |
| GroupNib created 03-06-2020 01:40 from SNAPSHOT |
| LinkNib created 03-06-2020 01:40 from SNAPSHOT |
| HostNib created 03-06-2020 01:40 from SNAPSHOT |
| DeviceNib created 03-06-2020 01:40 from SNAPSHOT |
| DriverNib created 03-06-2020 01:40 from SNAPSHOT |
| MastershipNib created 03-06-2020 01:40 from SNAPSHOT |
| EdgePortNib created 03-06-2020 01:40 from SNAPSHOT |
| RouteNib created 03-06-2020 01:40 from SNAPSHOT |
| NetworkConfigNib created 03-06-2020 01:40 from SNAPSHOT |
| MulticastRouteNib created 03-06-2020 01:40 from SNAPSHOT |
| |
| If one of T3 commands below is used before loading the NIB from **SNAPSHOT** or **FILE** (so the NIB is empty), |
| the command internally executes **t3-load-snapshot** to proceed with auto-filled NIB. |
| |
| .. code-block:: console |
| |
| onos> t3-troubleshoot-simple 00:AA:00:00:00:01/None 00:AA:00:00:00:02/None |
| *** NIB is invalid. Snapshots for the NIB have been auto-filled: *** |
| Load current network states from ONOS stores |
| FlowNib created 03-06-2020 01:44 from SNAPSHOT |
| GroupNib created 03-06-2020 01:44 from SNAPSHOT |
| LinkNib created 03-06-2020 01:44 from SNAPSHOT |
| ... |
| Tracing between: 00:AA:00:00:00:01/None and 00:AA:00:00:00:02/None |
| Tracing Packet: DefaultTrafficSelector{criteria=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.2.2/32]} |
| Paths |
| [of:0000000000000204/3, of:0000000000000204/4] |
| Result: |
| Reached required destination Host of:0000000000000204/4 |
| |
| |
| t3-troubleshoot |
| =============== |
| |
| As of right now the tools provides a CLI command (troubleshoot). |
| |
| As arguments to the CLI command you have to pass arguments describing the |
| packet you want to trace through the network. |
| |
| t3-troubleshoot Help |
| -------------------- |
| |
| .. code-block:: console |
| |
| onos> t3-troubleshoot --help |
| DESCRIPTION |
| onos:troubleshoot |
| |
| troubleshoots flows and groups between source and destination |
| |
| SYNTAX |
| onos:troubleshoot [options] |
| |
| OPTIONS |
| -sm, --srcMac |
| Source MAC |
| -j, --json |
| Output JSON |
| --help |
| Display this help message |
| -v, --verbose |
| Outputs complete path |
| -ipp, --ipProto |
| IP Proto |
| -s, --srcIp |
| Source IP |
| -dm, --dstMac |
| Destination MAC |
| -ml, --mplsLabel |
| Mpls label of incoming packet |
| -vv, --veryverbose |
| Outputs flows and groups for every device |
| -udps, --udpSrc |
| UDP Source |
| -mb, --mplsBos |
| MPLS BOS |
| -et, --ethType |
| ETH Type |
| (defaults to ipv4) |
| -sp, --srcPort |
| Source Port |
| -dtp, --dstTcpPort |
| destination TCP Port |
| -udpd, --udpDst |
| UDP Destination |
| -stp, --srcTcpPort |
| Source TCP Port |
| -d, --dstIp |
| Destination IP |
| -vid, --vlanId |
| Vlan of incoming packet |
| (defaults to None) |
| |
| Default Verbosity Level |
| ----------------------- |
| Let's look at an example. |
| |
| .. code-block:: console |
| |
| onos> t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:00:00:00:02:04 -vid None |
| |
| with this command we are asking the t3 tool to trace a packet with |
| |
| - Source IP of ``10.0.2.1/32`` |
| - Input port ``of:0000000000000204/3`` |
| - Source MAC ``00:AA:00:00:00:01`` |
| - ethType ``IPv4`` |
| - Source TCP Port ``12345`` |
| - Destination IP ``10.0.3.1/32`` |
| - Destination MAC ``00:AA:00:00:00:01`` |
| - Destination TCP Port ``80`` and |
| - VLAN ID ``None`` |
| |
| This command has a verbosity level of 0 and will output just the paths that the packet might take inside the network. |
| In the example deployment taken from trellis.py [#trellis.py]_, the output will be: |
| |
| .. code-block:: console |
| |
| Tracing packet: [IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.3.1/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true] |
| Paths |
| [of:0000000000000204/3, of:0000000000000204/2, of:0000000000000227/1, of:0000000000000227/2, of:0000000000000205/2, of:0000000000000205/3] |
| [of:0000000000000204/3, of:0000000000000204/1, of:0000000000000226/1, of:0000000000000226/2, of:0000000000000205/1, of:0000000000000205/3] |
| Result: |
| Reached required destination Host |
| Reached required destination Host |
| |
| Middle Verbosity Level |
| ---------------------- |
| There are also two other verbosity levels. |
| |
| .. code-block:: console |
| |
| onos> t3-troubleshoot -v -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:00:00:00:02:04 -vid None |
| |
| prints flows selectors and groupId that are met along the path |
| |
| .. code-block:: console |
| |
| Tracing packet: [IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.3.1/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true] |
| Path [of:0000000000000204/3, of:0000000000000204/2, of:0000000000000227/1, of:0000000000000227/2, of:0000000000000205/2, of:0000000000000205/3] |
| Device of:0000000000000204 |
| Input from of:0000000000000204/3 |
| Flows |
| flowId=a20000bd57cb49, selector=[] |
| flowId=900000d4173ff7, selector=[IN_PORT:3, VLAN_VID:None] |
| flowId=9000004b3d1965, selector=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:20] |
| flowId=900000499769f5, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.0/24] |
| flowId=a2000038b487e7, selector=[] |
| Groups |
| groupId=GroupId{id=0x70000010} |
| groupId=GroupId{id=0x9200000d} |
| groupId=GroupId{id=0x9000000c} |
| groupId=GroupId{id=0xffe0002} |
| groupId=GroupId{id=0x9200000f} |
| groupId=GroupId{id=0x9000000e} |
| groupId=GroupId{id=0xffe0001} |
| Output through of:0000000000000204/2 |
| Device of:0000000000000227 |
| Input from of:0000000000000227/1 |
| Flows |
| flowId=a200005577bcd6, selector=[] |
| flowId=900000498457b2, selector=[IN_PORT:1, VLAN_VID:None] |
| flowId=900000f87746c9, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:27, ETH_TYPE:mpls_unicast, VLAN_VID:4094] |
| flowId=a200005bfd6ed7, selector=[] |
| flowId=9000009f3d554f, selector=[ETH_TYPE:mpls_unicast, MPLS_LABEL:205, MPLS_BOS:true] |
| flowId=a20000e601fc35, selector=[] |
| Groups |
| groupId=GroupId{id=0x70000017} |
| groupId=GroupId{id=0x20000016} |
| groupId=GroupId{id=0xffe0002} |
| Output through of:0000000000000227/2 |
| Device of:0000000000000205 |
| Input from of:0000000000000205/2 |
| Flows |
| flowId=a200008d542817, selector=[] |
| flowId=900000be3179a0, selector=[IN_PORT:1, VLAN_VID:None] |
| flowId=900000766be164, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:05, ETH_TYPE:ipv4, VLAN_VID:4094] |
| flowId=900000ec1d5779, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.1/32] |
| flowId=a2000042d8e6b5, selector=[] |
| Groups |
| groupId=GroupId{id=0x2000002a} |
| groupId=GroupId{id=0x1e0003} |
| groupId=GroupId{id=0x2000002a} |
| groupId=GroupId{id=0x1e0003} |
| Output through of:0000000000000205/3 |
| Path [of:0000000000000204/3, of:0000000000000204/1, of:0000000000000226/1, of:0000000000000226/2, of:0000000000000205/1, of:0000000000000205/3] |
| Device of:0000000000000204 |
| Input from of:0000000000000204/3 |
| Flows |
| flowId=a20000bd57cb49, selector=[] |
| flowId=900000d4173ff7, selector=[IN_PORT:3, VLAN_VID:None] |
| flowId=9000004b3d1965, selector=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:20] |
| flowId=900000499769f5, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.0/24] |
| flowId=a2000038b487e7, selector=[] |
| Groups |
| groupId=GroupId{id=0x70000010} |
| groupId=GroupId{id=0x9200000d} |
| groupId=GroupId{id=0x9000000c} |
| groupId=GroupId{id=0xffe0002} |
| groupId=GroupId{id=0x9200000f} |
| groupId=GroupId{id=0x9000000e} |
| groupId=GroupId{id=0xffe0001} |
| Output through of:0000000000000204/1 |
| Device of:0000000000000226 |
| Input from of:0000000000000226/1 |
| Flows |
| flowId=a20000e9b27d24, selector=[] |
| flowId=9000004bd76105, selector=[IN_PORT:1, VLAN_VID:None] |
| flowId=90000069f274d1, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:26, ETH_TYPE:mpls_unicast, VLAN_VID:4094] |
| flowId=a200000ec7529e, selector=[] |
| flowId=90000050f9a350, selector=[ETH_TYPE:mpls_unicast, MPLS_LABEL:205, MPLS_BOS:true] |
| flowId=a20000b15cada6, selector=[] |
| Groups |
| groupId=GroupId{id=0x7000000b} |
| groupId=GroupId{id=0x20000009} |
| groupId=GroupId{id=0xffe0002} |
| Output through of:0000000000000226/2 |
| Device of:0000000000000205 |
| Input from of:0000000000000205/1 |
| Flows |
| flowId=a200008d542817, selector=[] |
| flowId=900000be3179a0, selector=[IN_PORT:1, VLAN_VID:None] |
| flowId=900000766be164, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:05, ETH_TYPE:ipv4, VLAN_VID:4094] |
| flowId=900000ec1d5779, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.1/32] |
| flowId=a2000042d8e6b5, selector=[] |
| Groups |
| groupId=GroupId{id=0x2000002a} |
| groupId=GroupId{id=0x1e0003} |
| groupId=GroupId{id=0x2000002a} |
| groupId=GroupId{id=0x1e0003} |
| Output through of:0000000000000205/3 |
| Result: |
| Reached required destination Host |
| Reached required destination Host |
| |
| High Verbosity Level |
| -------------------- |
| The most verbose output you can get is |
| |
| .. code-block:: console |
| |
| onos> t3-troubleshoot -vv -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:00:00:00:02:04 -vid None |
| |
| This verbosity level prints all the flows and groups that are met along the |
| path: |
| |
| .. code-block:: console |
| |
| Tracing packet: [IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.3.1/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true] |
| Path [of:0000000000000204/3, of:0000000000000204/2, of:0000000000000227/1, of:0000000000000227/2, of:0000000000000205/2, of:0000000000000205/3] |
| Device of:0000000000000204 |
| Input from of:0000000000000204/3 |
| Flows |
| ADDED, bytes=733506, packets=8691, table=0, priority=0, selector=[], treatment=[transition=TABLE:10] |
| ADDED, bytes=11110, packets=61, table=10, priority=32768, selector=[IN_PORT:3, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:20], transition=TABLE:20] |
| ADDED, bytes=8698, packets=29, table=20, priority=32768, selector=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:20], treatment=[transition=TABLE:30] |
| ADDED, bytes=16612, packets=50, table=30, priority=48010, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.0/24], treatment=[deferred=[GROUP:0x70000010], transition=TABLE:60] |
| ADDED, bytes=34400, packets=112, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]] |
| Groups |
| id=0x70000010, state=ADDED, type=SELECT, bytes=16906, packets=53, appId=org.onosproject.segmentrouting, referenceCount=3 |
| id=0x70000010, bucket=1, bytes=490, packets=5, actions=[GROUP:0x9200000d] |
| id=0x70000010, bucket=2, bytes=16416, packets=48, actions=[GROUP:0x9200000f] |
| id=0x9200000d, state=ADDED, type=INDIRECT, bytes=490, packets=5, appId=org.onosproject.segmentrouting, referenceCount=0 |
| id=0x9200000d, bucket=1, bytes=490, packets=5, actions=[VLAN_POP, MPLS_PUSH:mpls_unicast, MPLS_LABEL:205, GROUP:0x9000000c, VLAN_PUSH:vlan, VLAN_ID:4095] |
| id=0x9000000c, state=ADDED, type=INDIRECT, bytes=490, packets=5, appId=org.onosproject.segmentrouting, referenceCount=0 |
| id=0x9000000c, bucket=1, bytes=490, packets=5, actions=[ETH_DST:00:00:00:00:02:27, ETH_SRC:00:00:00:00:02:04, VLAN_ID:4094, GROUP:0xffe0002] |
| id=0xffe0002, state=ADDED, type=INDIRECT, bytes=726, packets=7, appId=org.onosproject.segmentrouting, referenceCount=0 |
| id=0xffe0002, bucket=1, bytes=726, packets=7, actions=[VLAN_POP, OUTPUT:2] |
| id=0x9200000f, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0 |
| id=0x9200000f, bucket=1, bytes=16416, packets=48, actions=[VLAN_POP, MPLS_PUSH:mpls_unicast, MPLS_LABEL:205, GROUP:0x9000000e, VLAN_PUSH:vlan, VLAN_ID:4095] |
| id=0x9000000e, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0 |
| id=0x9000000e, bucket=1, bytes=16416, packets=48, actions=[ETH_DST:00:00:00:00:02:26, ETH_SRC:00:00:00:00:02:04, VLAN_ID:4094, GROUP:0xffe0001] |
| id=0xffe0001, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0 |
| id=0xffe0001, bucket=1, bytes=16416, packets=48, actions=[VLAN_POP, OUTPUT:1] |
| Output through of:0000000000000204/2 |
| Device of:0000000000000227 |
| Input from of:0000000000000227/1 |
| Flows |
| ADDED, bytes=702318, packets=8511, table=0, priority=0, selector=[], treatment=[transition=TABLE:10] |
| ADDED, bytes=343130, packets=4234, table=10, priority=32768, selector=[IN_PORT:1, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:4094], transition=TABLE:20] |
| ADDED, bytes=754, packets=7, table=20, priority=32768, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:27, ETH_TYPE:mpls_unicast, VLAN_VID:4094], treatment=[transition=TABLE:23] |
| ADDED, bytes=17566, packets=57, table=23, priority=0, selector=[], treatment=[transition=TABLE:24] |
| ADDED, bytes=510, packets=5, table=24, priority=100, selector=[ETH_TYPE:mpls_unicast, MPLS_LABEL:205, MPLS_BOS:true], treatment=[deferred=[MPLS_POP:ipv4, DEC_TTL, GROUP:0x70000017], transition=TABLE:60] |
| ADDED, bytes=17566, packets=57, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]] |
| Groups |
| id=0x70000017, state=ADDED, type=SELECT, bytes=754, packets=7, appId=org.onosproject.segmentrouting, referenceCount=2 |
| id=0x70000017, bucket=1, bytes=754, packets=7, actions=[GROUP:0x20000016] |
| id=0x20000016, state=ADDED, type=INDIRECT, bytes=754, packets=7, appId=org.onosproject.segmentrouting, referenceCount=0 |
| id=0x20000016, bucket=1, bytes=754, packets=7, actions=[ETH_DST:00:00:00:00:02:05, ETH_SRC:00:00:00:00:02:27, VLAN_ID:4094, GROUP:0xffe0002] |
| id=0xffe0002, state=ADDED, type=INDIRECT, bytes=754, packets=7, appId=org.onosproject.segmentrouting, referenceCount=0 |
| id=0xffe0002, bucket=1, bytes=754, packets=7, actions=[VLAN_POP, OUTPUT:2] |
| Output through of:0000000000000227/2 |
| Device of:0000000000000205 |
| Input from of:0000000000000205/2 |
| Flows |
| ADDED, bytes=2105886, packets=24084, table=0, priority=0, selector=[], treatment=[transition=TABLE:10] |
| ADDED, bytes=358792, packets=4275, table=10, priority=32768, selector=[IN_PORT:1, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:4094], transition=TABLE:20] |
| ADDED, bytes=16416, packets=48, table=20, priority=32768, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:05, ETH_TYPE:ipv4, VLAN_VID:4094], treatment=[transition=TABLE:30] |
| ADDED, bytes=196, packets=2, table=30, priority=64010, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.1/32], treatment=[deferred=[GROUP:0x2000002a], transition=TABLE:60] |
| ADDED, bytes=1370884, packets=15163, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]] |
| Groups |
| id=0x2000002a, state=ADDED, type=INDIRECT, bytes=196, packets=2, appId=org.onosproject.segmentrouting, referenceCount=1 |
| id=0x2000002a, bucket=1, bytes=196, packets=2, actions=[ETH_DST:00:AA:00:00:00:03, ETH_SRC:00:00:00:00:02:05, VLAN_ID:30, GROUP:0x1e0003] |
| id=0x1e0003, state=ADDED, type=INDIRECT, bytes=12960, packets=126, appId=org.onosproject.segmentrouting, referenceCount=1 |
| id=0x1e0003, bucket=1, bytes=12960, packets=126, actions=[VLAN_POP, OUTPUT:3] |
| id=0x2000002a, state=ADDED, type=INDIRECT, bytes=196, packets=2, appId=org.onosproject.segmentrouting, referenceCount=1 |
| id=0x2000002a, bucket=1, bytes=196, packets=2, actions=[ETH_DST:00:AA:00:00:00:03, ETH_SRC:00:00:00:00:02:05, VLAN_ID:30, GROUP:0x1e0003] |
| id=0x1e0003, state=ADDED, type=INDIRECT, bytes=12960, packets=126, appId=org.onosproject.segmentrouting, referenceCount=1 |
| id=0x1e0003, bucket=1, bytes=12960, packets=126, actions=[VLAN_POP, OUTPUT:3] |
| Output through of:0000000000000205/3 |
| Path [of:0000000000000204/3, of:0000000000000204/1, of:0000000000000226/1, of:0000000000000226/2, of:0000000000000205/1, of:0000000000000205/3] |
| Device of:0000000000000204 |
| Input from of:0000000000000204/3 |
| Flows |
| ADDED, bytes=733506, packets=8691, table=0, priority=0, selector=[], treatment=[transition=TABLE:10] |
| ADDED, bytes=11110, packets=61, table=10, priority=32768, selector=[IN_PORT:3, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:20], transition=TABLE:20] |
| ADDED, bytes=8698, packets=29, table=20, priority=32768, selector=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:20], treatment=[transition=TABLE:30] |
| ADDED, bytes=16612, packets=50, table=30, priority=48010, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.0/24], treatment=[deferred=[GROUP:0x70000010], transition=TABLE:60] |
| ADDED, bytes=34400, packets=112, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]] |
| Groups |
| id=0x70000010, state=ADDED, type=SELECT, bytes=16906, packets=53, appId=org.onosproject.segmentrouting, referenceCount=3 |
| id=0x70000010, bucket=1, bytes=490, packets=5, actions=[GROUP:0x9200000d] |
| id=0x70000010, bucket=2, bytes=16416, packets=48, actions=[GROUP:0x9200000f] |
| id=0x9200000d, state=ADDED, type=INDIRECT, bytes=490, packets=5, appId=org.onosproject.segmentrouting, referenceCount=0 |
| id=0x9200000d, bucket=1, bytes=490, packets=5, actions=[VLAN_POP, MPLS_PUSH:mpls_unicast, MPLS_LABEL:205, GROUP:0x9000000c, VLAN_PUSH:vlan, VLAN_ID:4095] |
| id=0x9000000c, state=ADDED, type=INDIRECT, bytes=490, packets=5, appId=org.onosproject.segmentrouting, referenceCount=0 |
| id=0x9000000c, bucket=1, bytes=490, packets=5, actions=[ETH_DST:00:00:00:00:02:27, ETH_SRC:00:00:00:00:02:04, VLAN_ID:4094, GROUP:0xffe0002] |
| id=0xffe0002, state=ADDED, type=INDIRECT, bytes=726, packets=7, appId=org.onosproject.segmentrouting, referenceCount=0 |
| id=0xffe0002, bucket=1, bytes=726, packets=7, actions=[VLAN_POP, OUTPUT:2] |
| id=0x9200000f, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0 |
| id=0x9200000f, bucket=1, bytes=16416, packets=48, actions=[VLAN_POP, MPLS_PUSH:mpls_unicast, MPLS_LABEL:205, GROUP:0x9000000e, VLAN_PUSH:vlan, VLAN_ID:4095] |
| id=0x9000000e, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0 |
| id=0x9000000e, bucket=1, bytes=16416, packets=48, actions=[ETH_DST:00:00:00:00:02:26, ETH_SRC:00:00:00:00:02:04, VLAN_ID:4094, GROUP:0xffe0001] |
| id=0xffe0001, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0 |
| id=0xffe0001, bucket=1, bytes=16416, packets=48, actions=[VLAN_POP, OUTPUT:1] |
| Output through of:0000000000000204/1 |
| Device of:0000000000000226 |
| Input from of:0000000000000226/1 |
| Flows |
| ADDED, bytes=701910, packets=8507, table=0, priority=0, selector=[], treatment=[transition=TABLE:10] |
| ADDED, bytes=358984, packets=4275, table=10, priority=32768, selector=[IN_PORT:1, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:4094], transition=TABLE:20] |
| ADDED, bytes=16608, packets=48, table=20, priority=32768, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:26, ETH_TYPE:mpls_unicast, VLAN_VID:4094], treatment=[transition=TABLE:23] |
| ADDED, bytes=17158, packets=53, table=23, priority=0, selector=[], treatment=[transition=TABLE:24] |
| ADDED, bytes=16608, packets=48, table=24, priority=100, selector=[ETH_TYPE:mpls_unicast, MPLS_LABEL:205, MPLS_BOS:true], treatment=[deferred=[MPLS_POP:ipv4, DEC_TTL, GROUP:0x7000000b], transition=TABLE:60] |
| ADDED, bytes=17158, packets=53, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]] |
| Groups |
| id=0x7000000b, state=ADDED, type=SELECT, bytes=16608, packets=48, appId=org.onosproject.segmentrouting, referenceCount=2 |
| id=0x7000000b, bucket=1, bytes=16608, packets=48, actions=[GROUP:0x20000009] |
| id=0x20000009, state=ADDED, type=INDIRECT, bytes=16608, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0 |
| id=0x20000009, bucket=1, bytes=16608, packets=48, actions=[ETH_DST:00:00:00:00:02:05, ETH_SRC:00:00:00:00:02:26, VLAN_ID:4094, GROUP:0xffe0002] |
| id=0xffe0002, state=ADDED, type=INDIRECT, bytes=16608, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0 |
| id=0xffe0002, bucket=1, bytes=16608, packets=48, actions=[VLAN_POP, OUTPUT:2] |
| Output through of:0000000000000226/2 |
| Device of:0000000000000205 |
| Input from of:0000000000000205/1 |
| Flows |
| ADDED, bytes=2105886, packets=24084, table=0, priority=0, selector=[], treatment=[transition=TABLE:10] |
| ADDED, bytes=358792, packets=4275, table=10, priority=32768, selector=[IN_PORT:1, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:4094], transition=TABLE:20] |
| ADDED, bytes=16416, packets=48, table=20, priority=32768, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:05, ETH_TYPE:ipv4, VLAN_VID:4094], treatment=[transition=TABLE:30] |
| ADDED, bytes=196, packets=2, table=30, priority=64010, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.1/32], treatment=[deferred=[GROUP:0x2000002a], transition=TABLE:60] |
| ADDED, bytes=1370884, packets=15163, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]] |
| Groups |
| id=0x2000002a, state=ADDED, type=INDIRECT, bytes=196, packets=2, appId=org.onosproject.segmentrouting, referenceCount=1 |
| id=0x2000002a, bucket=1, bytes=196, packets=2, actions=[ETH_DST:00:AA:00:00:00:03, ETH_SRC:00:00:00:00:02:05, VLAN_ID:30, GROUP:0x1e0003] |
| id=0x1e0003, state=ADDED, type=INDIRECT, bytes=12960, packets=126, appId=org.onosproject.segmentrouting, referenceCount=1 |
| id=0x1e0003, bucket=1, bytes=12960, packets=126, actions=[VLAN_POP, OUTPUT:3] |
| id=0x2000002a, state=ADDED, type=INDIRECT, bytes=196, packets=2, appId=org.onosproject.segmentrouting, referenceCount=1 |
| id=0x2000002a, bucket=1, bytes=196, packets=2, actions=[ETH_DST:00:AA:00:00:00:03, ETH_SRC:00:00:00:00:02:05, VLAN_ID:30, GROUP:0x1e0003] |
| id=0x1e0003, state=ADDED, type=INDIRECT, bytes=12960, packets=126, appId=org.onosproject.segmentrouting, referenceCount=1 |
| id=0x1e0003, bucket=1, bytes=12960, packets=126, actions=[VLAN_POP, OUTPUT:3] |
| Output through of:0000000000000205/3 |
| Result: |
| Reached required destination Host |
| Reached required destination Host |
| |
| Successful Example |
| ------------------ |
| |
| In this example we trace a packet that should be bridged between two hosts in |
| the same subnet |
| |
| .. code-block:: console |
| |
| onos> t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:AA:00:00:00:02 -vid None |
| Tracing packet: [IN_PORT:3, ETH_DST:00:AA:00:00:00:02, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.2.2/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true] |
| Paths |
| [of:0000000000000204/3, of:0000000000000204/4] |
| Result: |
| Reached required destination Host |
| |
| Failed Example |
| -------------- |
| |
| Obviously failures can happen along one of the paths that the packet can take. |
| |
| In our sample we removed a flow in ``of:0000000000000226`` thus disabling that |
| possible path for our initial packet. |
| |
| The output of our trace would then look something like: |
| |
| .. code-block:: console |
| |
| onos> t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:00:00:00:02:04 -vid None |
| Tracing packet: [IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.3.1/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true] |
| Paths |
| [of:0000000000000204/3, of:0000000000000204/2, of:0000000000000227/1, of:0000000000000227/2, of:0000000000000205/2, of:0000000000000205/3] |
| [of:0000000000000204/3, of:0000000000000204/1, of:0000000000000226/1] |
| Result: |
| Reached required destination Host |
| No output out of device of:0000000000000226 Packet is dropped |
| |
| Notice that on the second path that was previously going through |
| ``of:0000000000000226`` now the packet is dropped due to the rule we removed. |
| |
| Example of Various Packet Types |
| ------------------------------- |
| These sample commands refer to the Mininet environment found in trellis.py [#trellis.py]_ |
| |
| - L2 Unicast IPv4 |
| |
| .. code-block:: console |
| |
| t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -d 10.0.2.2 -dm 00:AA:00:00:00:02 |
| |
| - L2 Unicast IPv6 |
| |
| .. code-block:: console |
| |
| t3-troubleshoot -vv -s 2000::301 -sp of:0000000000000205/5 -sm 00:BB:00:00:00:03 -d 2000::302 -dm 00:BB:00:00:00:04 -et ipv6 |
| |
| - L3 Unicast IPv4 |
| |
| .. code-block:: console |
| |
| t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -d 10.0.3.1 -dm 00:00:00:00:02:04 |
| |
| - L3 Unicast IPv6 |
| |
| .. code-block:: console |
| |
| t3-troubleshoot -vv -s 2000::301 -sp of:0000000000000205/5 -sm 00:BB:00:00:00:03 -d 2000::202 -dm 00:00:00:00:02:05 -et ipv6 |
| |
| - ARP |
| |
| .. code-block:: console |
| |
| t3-troubleshoot -sp of:0000000000000204/3 -et ARP |
| |
| - DHCP |
| |
| .. code-block:: console |
| |
| t3-troubleshoot -vv -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -d 10.0.3.253 -dm 00:00:00:00:02:04 -ipp 17 -udps 67 -udpd 67 |
| |
| - LLDP |
| |
| .. code-block:: console |
| |
| t3-troubleshoot -sp of:0000000000000204/3 -et lldp |
| |
| - BDDP |
| |
| .. code-block:: console |
| |
| t3-troubleshoot -sp of:0000000000000204/3 -et bddp |
| |
| - LLDP from CONTROLLER |
| |
| T3 is also capable of tracing packets coming in from the CONTROLLER. |
| |
| We use the same input port with the possibility to give CONTROLLER as logical |
| port number. |
| |
| This is a mere convenience to carry the information inside T3 and do the |
| proper flooding to all the ports of the given device. |
| |
| This case is not treated like all the rest and the input port here is a |
| logical entity that has no match on any physical element on the device. |
| |
| .. code-block:: console |
| |
| t3-troubleshoot -sp of:0000000000000204/CONTROLLER -et lldp |
| |
| - Multicast |
| |
| T3 is also capable of tracing multicast packets. |
| |
| .. code-block:: console |
| |
| t3-troubleshoot -vv -sp of:0000000000000205/9 -et ipv4 -d 224.0.0.1 -dm 01:00:5e:00:00:01 |
| |
| |
| t3-troubleshoot-simple |
| ====================== |
| |
| T3 also offers a simplified version of the command to be able to ping between |
| two given hosts without having to specify all the information. |
| |
| This commands bases itself on top of some configuration present in ONOS through |
| netcfg, namely the VLAN interface configuration and the segment routing leaf |
| router MAC for L3 Unicast. If that is not present the user needs to input this |
| information manually recurring to the the previously shown command. Same |
| verbosity levels and output as the troubleshoot command are possible. |
| |
| t3-troubleshoot-simple Help |
| --------------------------- |
| .. code-block:: console |
| |
| onos> t3-troubleshoot-simple --help |
| DESCRIPTION |
| onos:troubleshoot-simple |
| |
| Given two hosts troubleshoots flows and groups between them, in case of segment routing |
| |
| SYNTAX |
| onos:troubleshoot-simple [options] one two |
| |
| ARGUMENTS |
| one |
| One host ID |
| two |
| Another host ID |
| |
| OPTIONS |
| -vv, --veryverbose |
| Outputs flows and groups for every device |
| -j, --json |
| Output JSON |
| --help |
| Display this help message |
| -et, --ethType |
| ETH Type |
| (defaults to ipv4) |
| -v, --verbose |
| Outputs complete path |
| |
| t3-troubleshoot-simple Example |
| ------------------------------ |
| Examples are given for the same trellis.py [#trellis.py]_ script. |
| |
| - L2 Unicast |
| |
| Hosts are under the same leaf, only interface configuration is needed. |
| |
| .. code-block:: console |
| |
| t3-troubleshoot-simple -v -et IPV4 00:AA:00:00:00:01/None 00:AA:00:00:00:02/None |
| |
| - L3 Multicast |
| |
| Hosts are under different leafs, also segment routing configuration config is needed for the leaf's routerMac. |
| |
| .. code-block:: console |
| |
| t3-troubleshoot-simple -et IPV4 00:AA:00:00:00:01/None 00:AA:00:00:00:03/None |
| |
| |
| t3-troubleshoot-pingall |
| ======================= |
| T3 also offers a ping-all command to be able to test ping between any given hosts of an EthType. |
| This commands bases itself on top of the simplified Host to Host command. It has two verbosity levels, |
| one just with Host and Destination with the result message of the Trace and one that shows also the packet and the paths. |
| |
| t3-troubleshoot-pingall Help |
| ---------------------------- |
| .. code-block:: console |
| |
| onos> t3-troubleshoot-pingall --help |
| DESCRIPTION |
| onos:t3-troubleshoot-pingall |
| |
| Traces a ping between all hosts in the system of a given ETH type |
| |
| SYNTAX |
| onos:t3-troubleshoot-pingall [options] |
| |
| OPTIONS |
| -j, --json |
| Output JSON |
| --help |
| Display this help message |
| -et, --ethType |
| ETH Type |
| (defaults to ipv4) |
| -v, --verbose |
| Outputs trace for each host to host combination |
| |
| t3-troubleshoot-pingall Examples |
| -------------------------------- |
| - IPv4 hosts |
| |
| .. code-block:: console |
| |
| t3-troubleshoot-pingall |
| |
| - IPv6 hosts |
| |
| .. code-block:: console |
| |
| t3-troubleshoot-pingall -et IPV6 |
| |
| |
| t3-troubleshoot-mcast |
| ===================== |
| |
| T3 also offers a mcast command queries the system for all the multicast routes |
| given a certain vlanId. It prints all the traces given a certain group, source |
| and source connect point. It has two "standard" verbosity levels. |
| |
| t3-troubleshoot-mcast Help |
| -------------------------- |
| .. code-block:: console |
| |
| onos> t3-troubleshoot-mcast --help |
| DESCRIPTION |
| onos:t3-troubleshoot-mcast |
| |
| Traces all the mcast routes present in the system |
| |
| SYNTAX |
| onos:t3-troubleshoot-mcast [options] |
| |
| OPTIONS |
| -j, --json |
| Output JSON |
| --help |
| Display this help message |
| -vv, --veryverbose |
| Outputs details of every trace |
| -vid, --vlanId |
| Vlan of incoming packet |
| (defaults to None) |
| -v, --verbose |
| Outputs trace for each host to host combination |
| |
| REST API |
| ======== |
| T3 now also offers a way to execute the troubleshoot through REST API endpoints. |
| You can find the description and test the API at the ONOS docs endpoint here: |
| |
| .. code-block:: text |
| |
| <ONOS_IP>:8181/onos/v1/docs/#/t3 |
| |
| The rest API offered are a non verbose and a verbose version of the troubleshoot simple command. |
| |
| It bases itself on top of some configuration present in ONOS through netcfg, |
| namely the VLAN interface configuration and the segment routing leaf router MAC |
| for L3 Unicast. |
| |
| The use need to specify just the Source Host, the destination host and the ETH |
| type. |
| |
| .. code-block:: console |
| |
| curl -X GET --header 'Accept: application/json' 'http://localhost:8181/onos/v1/t3/t3/simple/<host_src>/<host_dst>/<eht_type>' |
| |
| .. code-block:: console |
| |
| curl -X GET --header 'Accept: application/json' 'http://localhost:8181/onos/v1/t3/t3/simple/<host_src>/<host_dst>/<eht_type>/verbose |
| |
| |
| Last Resort |
| =========== |
| |
| In the case where you can't figure out what is going wrong even with help of |
| T3, you can seek help on Trellis developer mailing list |
| ``trellis-dev@opennetworking.org``. |
| |
| There are a few things we would like you to attach: |
| |
| - **Issue description** |
| |
| - **Environment description**, such as ONOS version, switch model and OF-DPA |
| version |
| |
| - **Steps of reproduction**, as detail as possible |
| |
| - **Diagnostics**. |
| |
| We have built a tool ``onos-diagnostics`` [#onos-diagnostics_src]_ [#onos-diagnostics_man]_ |
| to help you easily collect and package ONOS diagnostics. |
| |
| Alternatively, you can use ``onos-diagnostics-k8s`` [#onos-diagnostics-k8s_src]_ [#onos-diagnostics_man]_. |
| |
| Reference |
| ========= |
| |
| .. [#onos-diagnostics_src] `<https://github.com/opennetworkinglab/onos/blob/master/tools/package/runtime/bin/onos-diagnostics>`_ |
| .. [#onos-diagnostics_man] `<https://wiki.onosproject.org/display/ONOS/ONOS+Remote+Admin+Tools>`_ |
| .. [#onos-diagnostics-k8s_src] `<https://github.com/opennetworkinglab/onos/blob/master/tools/package/runtime/bin/onos-diagnostics-k8s>`_ |
| .. [#trellis.py] `<https://github.com/opennetworkinglab/routing/blob/master/trellis/trellis.py>`_ |