Charles Chan | 78a737f | 2019-09-12 13:55:22 -0700 | [diff] [blame] | 1 | Troubleshooting Guide |
| 2 | ********************* |
| 3 | |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 4 | Overview |
| 5 | ======== |
| 6 | |
Zack Williams | d63d35b | 2020-06-23 14:12:46 -0700 | [diff] [blame] | 7 | The **Trellis Troubleshooting Tools (T3)** is an ONOS application intended to |
| 8 | **help you troubleshoot traffic-related problems** in your network by **tracing |
| 9 | installed flows and groups**. |
| 10 | |
| 11 | T3 works both on simulated OVS-based topologies but also on real HW, basing |
| 12 | it's internal decisions on the driver assigned to the device. |
| 13 | |
| 14 | T3 also currently supports dual links between leaf and spine and properly |
| 15 | handles the difference between deferred and immediate rules. Dual Homed hosts |
| 16 | are also supported in T3. |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 17 | |
| 18 | To activate this tool on your ONOS environment, you need to run: |
| 19 | |
| 20 | .. code-block:: console |
| 21 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 22 | onos> app activate t3 |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 23 | |
| 24 | T3 supports tracing of following packet types: |
| 25 | |
| 26 | - L2 Unicast |
| 27 | - L2 Broadcast |
| 28 | - L3 Unicast (both IPv4 and IPv6) |
| 29 | - L3 Multicast (both IPv4 and IPv6) |
| 30 | - LLDP and BDDP |
| 31 | - ARP |
| 32 | - DHCP |
| 33 | |
| 34 | |
Seyeon Jeong | 50d3c45 | 2020-05-29 14:07:57 -0700 | [diff] [blame^] | 35 | Network Information Base (NIB) |
| 36 | ============================== |
| 37 | T3 analyzes network states that are stored in the NIB. |
| 38 | A NIB is a collection of snapshots captured from various ONOS stores such as flows, groups, links...etc. |
| 39 | The NIB, acting as a cache, allows T3 troubleshooting logic to avoid massive interaction with live ONOS subsystems, |
| 40 | enabling near-runtime or postmortem analysis in a production environment. |
| 41 | |
| 42 | There are two methods of loading the network snapshots into the NIB: |
| 43 | (1) from a live running system or (2) from a onos-diagnostics file dump. |
| 44 | Users can choose one load command before troubleshooting commands, |
| 45 | or can run troubleshooting commands alone (then the NIB is automatically loaded via (1)). |
| 46 | |
| 47 | |
| 48 | t3-load-file |
| 49 | ========================= |
| 50 | T3 offers this command to fill the NIB with files that represent snapshots of the network states in a certain time. |
| 51 | |
| 52 | t3-load-file Help |
| 53 | ------------------------------ |
| 54 | |
| 55 | .. code-block:: console |
| 56 | |
| 57 | onos> t3-load-file --help |
| 58 | DESCRIPTION |
| 59 | onos:t3-load-file |
| 60 | |
| 61 | Load the NIB with onos-diagnostics dump files |
| 62 | |
| 63 | SYNTAX |
| 64 | onos:t3-load-file [options] rootDir |
| 65 | |
| 66 | ARGUMENTS |
| 67 | rootDir |
| 68 | Specify the location of the directory where the previous onos-diagnostics command has extracted its dump files (e.g. /tmp/onos-diags/127.0.0.1) |
| 69 | |
| 70 | OPTIONS |
| 71 | --help |
| 72 | Display this help message |
| 73 | -j, --json |
| 74 | Output JSON |
| 75 | |
| 76 | t3-load-file Example |
| 77 | --------------------------------- |
| 78 | User is expected to run the tool ``onos-diagnostics`` [#onos-diagnostics_src]_ [#onos-diagnostics_man]_ |
| 79 | to generate target files first. |
| 80 | |
| 81 | .. code-block:: console |
| 82 | |
| 83 | user$ onos-diagnostics -xj [IP_ONOS_INSTANCE_1 IP_ONOS_INSTANCE_2 ...] |
| 84 | Collecting diagnostics on [IP_ONOS_INSTANCE_1]...logs netcfg.json devices.json device-drivers.json links.json hosts.json ports.json edge-ports.json flows.json groups.json masters.json routes.json mcast-host-show.json Done. |
| 85 | Collecting diagnostics on [IP_ONOS_INSTANCE_2]...logs netcfg.json devices.json device-drivers.json links.json hosts.json ports.json edge-ports.json flows.json groups.json masters.json routes.json mcast-host-show.json Done. |
| 86 | |
| 87 | Then, execute this command with the path of the location where those JSON files are extracted. |
| 88 | User can check the creation time and a summary of the NIB contents. |
| 89 | |
| 90 | .. code-block:: console |
| 91 | |
| 92 | onos> t3-load-file /tmp/onos-diags/[IP_ONOS_INSTANCE_TARGET] |
| 93 | Load target files in: /tmp/onos-diags/[IP_ONOS_INSTANCE_TARGET]/ |
| 94 | FlowNib created 03-06-2020 01:30 from FILE |
| 95 | GroupNib created 03-06-2020 01:30 from FILE |
| 96 | LinkNib created 03-06-2020 01:30 from FILE |
| 97 | HostNib created 03-06-2020 01:30 from FILE |
| 98 | DeviceNib created 03-06-2020 01:30 from FILE |
| 99 | DriverNib created 03-06-2020 01:30 from FILE |
| 100 | MastershipNib created 03-06-2020 01:30 from FILE |
| 101 | EdgePortNib created 03-06-2020 01:30 from FILE |
| 102 | RouteNib created 03-06-2020 01:30 from FILE |
| 103 | NetworkConfigNib created 03-06-2020 01:30 from FILE |
| 104 | MulticastRouteNib created 03-06-2020 01:30 from FILE |
| 105 | |
| 106 | t3-load-snapshot |
| 107 | ============================= |
| 108 | T3 offers this command to fill the NIB with snapshots of the network states at the execution time. |
| 109 | |
| 110 | t3-load-snapshot Help |
| 111 | ---------------------------------- |
| 112 | |
| 113 | .. code-block:: console |
| 114 | |
| 115 | onos> t3-load-snapshot --help |
| 116 | DESCRIPTION |
| 117 | onos:t3-load-snapshot |
| 118 | |
| 119 | Load the NIB with the network states stored in the ONOS instance where the T3 is running |
| 120 | |
| 121 | SYNTAX |
| 122 | onos:t3-load-snapshot [options] |
| 123 | |
| 124 | OPTIONS |
| 125 | --help |
| 126 | Display this help message |
| 127 | -j, --json |
| 128 | Output JSON |
| 129 | |
| 130 | t3-load-snapshot Example |
| 131 | ------------------------------------- |
| 132 | User can check the creation time and a summary of the NIB contents. |
| 133 | **SNAPSHOT** indicates that the source of the current NIB is the ONOS instance where the T3 is running. |
| 134 | |
| 135 | .. code-block:: console |
| 136 | |
| 137 | onos> t3-load-snapshot |
| 138 | Load current network states from ONOS stores |
| 139 | FlowNib created 03-06-2020 01:40 from SNAPSHOT |
| 140 | GroupNib created 03-06-2020 01:40 from SNAPSHOT |
| 141 | LinkNib created 03-06-2020 01:40 from SNAPSHOT |
| 142 | HostNib created 03-06-2020 01:40 from SNAPSHOT |
| 143 | DeviceNib created 03-06-2020 01:40 from SNAPSHOT |
| 144 | DriverNib created 03-06-2020 01:40 from SNAPSHOT |
| 145 | MastershipNib created 03-06-2020 01:40 from SNAPSHOT |
| 146 | EdgePortNib created 03-06-2020 01:40 from SNAPSHOT |
| 147 | RouteNib created 03-06-2020 01:40 from SNAPSHOT |
| 148 | NetworkConfigNib created 03-06-2020 01:40 from SNAPSHOT |
| 149 | MulticastRouteNib created 03-06-2020 01:40 from SNAPSHOT |
| 150 | |
| 151 | If one of T3 commands below is used before loading the NIB from **SNAPSHOT** or **FILE** (so the NIB is empty), |
| 152 | the command internally executes **t3-load-snapshot** to proceed with auto-filled NIB. |
| 153 | |
| 154 | .. code-block:: console |
| 155 | |
| 156 | onos> t3-troubleshoot-simple 00:AA:00:00:00:01/None 00:AA:00:00:00:02/None |
| 157 | *** NIB is invalid. Snapshots for the NIB have been auto-filled: *** |
| 158 | Load current network states from ONOS stores |
| 159 | FlowNib created 03-06-2020 01:44 from SNAPSHOT |
| 160 | GroupNib created 03-06-2020 01:44 from SNAPSHOT |
| 161 | LinkNib created 03-06-2020 01:44 from SNAPSHOT |
| 162 | ... |
| 163 | Tracing between: 00:AA:00:00:00:01/None and 00:AA:00:00:00:02/None |
| 164 | Tracing Packet: DefaultTrafficSelector{criteria=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.2.2/32]} |
| 165 | Paths |
| 166 | [of:0000000000000204/3, of:0000000000000204/4] |
| 167 | Result: |
| 168 | Reached required destination Host of:0000000000000204/4 |
| 169 | |
| 170 | |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 171 | t3-troubleshoot |
| 172 | =============== |
Zack Williams | d63d35b | 2020-06-23 14:12:46 -0700 | [diff] [blame] | 173 | |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 174 | As of right now the tools provides a CLI command (troubleshoot). |
Zack Williams | d63d35b | 2020-06-23 14:12:46 -0700 | [diff] [blame] | 175 | |
| 176 | As arguments to the CLI command you have to pass arguments describing the |
| 177 | packet you want to trace through the network. |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 178 | |
| 179 | t3-troubleshoot Help |
| 180 | -------------------- |
| 181 | |
| 182 | .. code-block:: console |
| 183 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 184 | onos> t3-troubleshoot --help |
| 185 | DESCRIPTION |
| 186 | onos:troubleshoot |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 187 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 188 | troubleshoots flows and groups between source and destination |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 189 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 190 | SYNTAX |
| 191 | onos:troubleshoot [options] |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 192 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 193 | OPTIONS |
| 194 | -sm, --srcMac |
| 195 | Source MAC |
| 196 | -j, --json |
| 197 | Output JSON |
| 198 | --help |
| 199 | Display this help message |
| 200 | -v, --verbose |
| 201 | Outputs complete path |
| 202 | -ipp, --ipProto |
| 203 | IP Proto |
| 204 | -s, --srcIp |
| 205 | Source IP |
| 206 | -dm, --dstMac |
| 207 | Destination MAC |
| 208 | -ml, --mplsLabel |
| 209 | Mpls label of incoming packet |
| 210 | -vv, --veryverbose |
| 211 | Outputs flows and groups for every device |
| 212 | -udps, --udpSrc |
| 213 | UDP Source |
| 214 | -mb, --mplsBos |
| 215 | MPLS BOS |
| 216 | -et, --ethType |
| 217 | ETH Type |
| 218 | (defaults to ipv4) |
| 219 | -sp, --srcPort |
| 220 | Source Port |
| 221 | -dtp, --dstTcpPort |
| 222 | destination TCP Port |
| 223 | -udpd, --udpDst |
| 224 | UDP Destination |
| 225 | -stp, --srcTcpPort |
| 226 | Source TCP Port |
| 227 | -d, --dstIp |
| 228 | Destination IP |
| 229 | -vid, --vlanId |
| 230 | Vlan of incoming packet |
| 231 | (defaults to None) |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 232 | |
| 233 | Default Verbosity Level |
| 234 | ----------------------- |
| 235 | Let's look at an example. |
| 236 | |
| 237 | .. code-block:: console |
| 238 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 239 | onos> t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:00:00:00:02:04 -vid None |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 240 | |
| 241 | with this command we are asking the t3 tool to trace a packet with |
| 242 | |
| 243 | - Source IP of ``10.0.2.1/32`` |
| 244 | - Input port ``of:0000000000000204/3`` |
| 245 | - Source MAC ``00:AA:00:00:00:01`` |
| 246 | - ethType ``IPv4`` |
| 247 | - Source TCP Port ``12345`` |
| 248 | - Destination IP ``10.0.3.1/32`` |
| 249 | - Destination MAC ``00:AA:00:00:00:01`` |
| 250 | - Destination TCP Port ``80`` and |
| 251 | - VLAN ID ``None`` |
| 252 | |
| 253 | This command has a verbosity level of 0 and will output just the paths that the packet might take inside the network. |
| 254 | In the example deployment taken from trellis.py [#trellis.py]_, the output will be: |
| 255 | |
| 256 | .. code-block:: console |
| 257 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 258 | Tracing packet: [IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.3.1/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true] |
| 259 | Paths |
| 260 | [of:0000000000000204/3, of:0000000000000204/2, of:0000000000000227/1, of:0000000000000227/2, of:0000000000000205/2, of:0000000000000205/3] |
| 261 | [of:0000000000000204/3, of:0000000000000204/1, of:0000000000000226/1, of:0000000000000226/2, of:0000000000000205/1, of:0000000000000205/3] |
| 262 | Result: |
| 263 | Reached required destination Host |
| 264 | Reached required destination Host |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 265 | |
| 266 | Middle Verbosity Level |
| 267 | ---------------------- |
| 268 | There are also two other verbosity levels. |
| 269 | |
| 270 | .. code-block:: console |
| 271 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 272 | onos> t3-troubleshoot -v -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:00:00:00:02:04 -vid None |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 273 | |
| 274 | prints flows selectors and groupId that are met along the path |
| 275 | |
| 276 | .. code-block:: console |
| 277 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 278 | Tracing packet: [IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.3.1/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true] |
| 279 | Path [of:0000000000000204/3, of:0000000000000204/2, of:0000000000000227/1, of:0000000000000227/2, of:0000000000000205/2, of:0000000000000205/3] |
| 280 | Device of:0000000000000204 |
| 281 | Input from of:0000000000000204/3 |
| 282 | Flows |
| 283 | flowId=a20000bd57cb49, selector=[] |
| 284 | flowId=900000d4173ff7, selector=[IN_PORT:3, VLAN_VID:None] |
| 285 | flowId=9000004b3d1965, selector=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:20] |
| 286 | flowId=900000499769f5, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.0/24] |
| 287 | flowId=a2000038b487e7, selector=[] |
| 288 | Groups |
| 289 | groupId=GroupId{id=0x70000010} |
| 290 | groupId=GroupId{id=0x9200000d} |
| 291 | groupId=GroupId{id=0x9000000c} |
| 292 | groupId=GroupId{id=0xffe0002} |
| 293 | groupId=GroupId{id=0x9200000f} |
| 294 | groupId=GroupId{id=0x9000000e} |
| 295 | groupId=GroupId{id=0xffe0001} |
| 296 | Output through of:0000000000000204/2 |
| 297 | Device of:0000000000000227 |
| 298 | Input from of:0000000000000227/1 |
| 299 | Flows |
| 300 | flowId=a200005577bcd6, selector=[] |
| 301 | flowId=900000498457b2, selector=[IN_PORT:1, VLAN_VID:None] |
| 302 | flowId=900000f87746c9, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:27, ETH_TYPE:mpls_unicast, VLAN_VID:4094] |
| 303 | flowId=a200005bfd6ed7, selector=[] |
| 304 | flowId=9000009f3d554f, selector=[ETH_TYPE:mpls_unicast, MPLS_LABEL:205, MPLS_BOS:true] |
| 305 | flowId=a20000e601fc35, selector=[] |
| 306 | Groups |
| 307 | groupId=GroupId{id=0x70000017} |
| 308 | groupId=GroupId{id=0x20000016} |
| 309 | groupId=GroupId{id=0xffe0002} |
| 310 | Output through of:0000000000000227/2 |
| 311 | Device of:0000000000000205 |
| 312 | Input from of:0000000000000205/2 |
| 313 | Flows |
| 314 | flowId=a200008d542817, selector=[] |
| 315 | flowId=900000be3179a0, selector=[IN_PORT:1, VLAN_VID:None] |
| 316 | flowId=900000766be164, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:05, ETH_TYPE:ipv4, VLAN_VID:4094] |
| 317 | flowId=900000ec1d5779, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.1/32] |
| 318 | flowId=a2000042d8e6b5, selector=[] |
| 319 | Groups |
| 320 | groupId=GroupId{id=0x2000002a} |
| 321 | groupId=GroupId{id=0x1e0003} |
| 322 | groupId=GroupId{id=0x2000002a} |
| 323 | groupId=GroupId{id=0x1e0003} |
| 324 | Output through of:0000000000000205/3 |
| 325 | Path [of:0000000000000204/3, of:0000000000000204/1, of:0000000000000226/1, of:0000000000000226/2, of:0000000000000205/1, of:0000000000000205/3] |
| 326 | Device of:0000000000000204 |
| 327 | Input from of:0000000000000204/3 |
| 328 | Flows |
| 329 | flowId=a20000bd57cb49, selector=[] |
| 330 | flowId=900000d4173ff7, selector=[IN_PORT:3, VLAN_VID:None] |
| 331 | flowId=9000004b3d1965, selector=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:20] |
| 332 | flowId=900000499769f5, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.0/24] |
| 333 | flowId=a2000038b487e7, selector=[] |
| 334 | Groups |
| 335 | groupId=GroupId{id=0x70000010} |
| 336 | groupId=GroupId{id=0x9200000d} |
| 337 | groupId=GroupId{id=0x9000000c} |
| 338 | groupId=GroupId{id=0xffe0002} |
| 339 | groupId=GroupId{id=0x9200000f} |
| 340 | groupId=GroupId{id=0x9000000e} |
| 341 | groupId=GroupId{id=0xffe0001} |
| 342 | Output through of:0000000000000204/1 |
| 343 | Device of:0000000000000226 |
| 344 | Input from of:0000000000000226/1 |
| 345 | Flows |
| 346 | flowId=a20000e9b27d24, selector=[] |
| 347 | flowId=9000004bd76105, selector=[IN_PORT:1, VLAN_VID:None] |
| 348 | flowId=90000069f274d1, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:26, ETH_TYPE:mpls_unicast, VLAN_VID:4094] |
| 349 | flowId=a200000ec7529e, selector=[] |
| 350 | flowId=90000050f9a350, selector=[ETH_TYPE:mpls_unicast, MPLS_LABEL:205, MPLS_BOS:true] |
| 351 | flowId=a20000b15cada6, selector=[] |
| 352 | Groups |
| 353 | groupId=GroupId{id=0x7000000b} |
| 354 | groupId=GroupId{id=0x20000009} |
| 355 | groupId=GroupId{id=0xffe0002} |
| 356 | Output through of:0000000000000226/2 |
| 357 | Device of:0000000000000205 |
| 358 | Input from of:0000000000000205/1 |
| 359 | Flows |
| 360 | flowId=a200008d542817, selector=[] |
| 361 | flowId=900000be3179a0, selector=[IN_PORT:1, VLAN_VID:None] |
| 362 | flowId=900000766be164, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:05, ETH_TYPE:ipv4, VLAN_VID:4094] |
| 363 | flowId=900000ec1d5779, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.1/32] |
| 364 | flowId=a2000042d8e6b5, selector=[] |
| 365 | Groups |
| 366 | groupId=GroupId{id=0x2000002a} |
| 367 | groupId=GroupId{id=0x1e0003} |
| 368 | groupId=GroupId{id=0x2000002a} |
| 369 | groupId=GroupId{id=0x1e0003} |
| 370 | Output through of:0000000000000205/3 |
| 371 | Result: |
| 372 | Reached required destination Host |
| 373 | Reached required destination Host |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 374 | |
| 375 | High Verbosity Level |
| 376 | -------------------- |
| 377 | The most verbose output you can get is |
| 378 | |
| 379 | .. code-block:: console |
| 380 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 381 | onos> t3-troubleshoot -vv -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:00:00:00:02:04 -vid None |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 382 | |
Zack Williams | d63d35b | 2020-06-23 14:12:46 -0700 | [diff] [blame] | 383 | This verbosity level prints all the flows and groups that are met along the |
| 384 | path: |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 385 | |
| 386 | .. code-block:: console |
| 387 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 388 | Tracing packet: [IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.3.1/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true] |
| 389 | Path [of:0000000000000204/3, of:0000000000000204/2, of:0000000000000227/1, of:0000000000000227/2, of:0000000000000205/2, of:0000000000000205/3] |
| 390 | Device of:0000000000000204 |
| 391 | Input from of:0000000000000204/3 |
| 392 | Flows |
| 393 | ADDED, bytes=733506, packets=8691, table=0, priority=0, selector=[], treatment=[transition=TABLE:10] |
| 394 | ADDED, bytes=11110, packets=61, table=10, priority=32768, selector=[IN_PORT:3, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:20], transition=TABLE:20] |
| 395 | ADDED, bytes=8698, packets=29, table=20, priority=32768, selector=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:20], treatment=[transition=TABLE:30] |
| 396 | ADDED, bytes=16612, packets=50, table=30, priority=48010, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.0/24], treatment=[deferred=[GROUP:0x70000010], transition=TABLE:60] |
| 397 | ADDED, bytes=34400, packets=112, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]] |
| 398 | Groups |
| 399 | id=0x70000010, state=ADDED, type=SELECT, bytes=16906, packets=53, appId=org.onosproject.segmentrouting, referenceCount=3 |
| 400 | id=0x70000010, bucket=1, bytes=490, packets=5, actions=[GROUP:0x9200000d] |
| 401 | id=0x70000010, bucket=2, bytes=16416, packets=48, actions=[GROUP:0x9200000f] |
| 402 | id=0x9200000d, state=ADDED, type=INDIRECT, bytes=490, packets=5, appId=org.onosproject.segmentrouting, referenceCount=0 |
| 403 | id=0x9200000d, bucket=1, bytes=490, packets=5, actions=[VLAN_POP, MPLS_PUSH:mpls_unicast, MPLS_LABEL:205, GROUP:0x9000000c, VLAN_PUSH:vlan, VLAN_ID:4095] |
| 404 | id=0x9000000c, state=ADDED, type=INDIRECT, bytes=490, packets=5, appId=org.onosproject.segmentrouting, referenceCount=0 |
| 405 | id=0x9000000c, bucket=1, bytes=490, packets=5, actions=[ETH_DST:00:00:00:00:02:27, ETH_SRC:00:00:00:00:02:04, VLAN_ID:4094, GROUP:0xffe0002] |
| 406 | id=0xffe0002, state=ADDED, type=INDIRECT, bytes=726, packets=7, appId=org.onosproject.segmentrouting, referenceCount=0 |
| 407 | id=0xffe0002, bucket=1, bytes=726, packets=7, actions=[VLAN_POP, OUTPUT:2] |
| 408 | id=0x9200000f, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0 |
| 409 | id=0x9200000f, bucket=1, bytes=16416, packets=48, actions=[VLAN_POP, MPLS_PUSH:mpls_unicast, MPLS_LABEL:205, GROUP:0x9000000e, VLAN_PUSH:vlan, VLAN_ID:4095] |
| 410 | id=0x9000000e, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0 |
| 411 | id=0x9000000e, bucket=1, bytes=16416, packets=48, actions=[ETH_DST:00:00:00:00:02:26, ETH_SRC:00:00:00:00:02:04, VLAN_ID:4094, GROUP:0xffe0001] |
| 412 | id=0xffe0001, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0 |
| 413 | id=0xffe0001, bucket=1, bytes=16416, packets=48, actions=[VLAN_POP, OUTPUT:1] |
| 414 | Output through of:0000000000000204/2 |
| 415 | Device of:0000000000000227 |
| 416 | Input from of:0000000000000227/1 |
| 417 | Flows |
| 418 | ADDED, bytes=702318, packets=8511, table=0, priority=0, selector=[], treatment=[transition=TABLE:10] |
| 419 | ADDED, bytes=343130, packets=4234, table=10, priority=32768, selector=[IN_PORT:1, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:4094], transition=TABLE:20] |
| 420 | ADDED, bytes=754, packets=7, table=20, priority=32768, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:27, ETH_TYPE:mpls_unicast, VLAN_VID:4094], treatment=[transition=TABLE:23] |
| 421 | ADDED, bytes=17566, packets=57, table=23, priority=0, selector=[], treatment=[transition=TABLE:24] |
| 422 | ADDED, bytes=510, packets=5, table=24, priority=100, selector=[ETH_TYPE:mpls_unicast, MPLS_LABEL:205, MPLS_BOS:true], treatment=[deferred=[MPLS_POP:ipv4, DEC_TTL, GROUP:0x70000017], transition=TABLE:60] |
| 423 | ADDED, bytes=17566, packets=57, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]] |
| 424 | Groups |
| 425 | id=0x70000017, state=ADDED, type=SELECT, bytes=754, packets=7, appId=org.onosproject.segmentrouting, referenceCount=2 |
| 426 | id=0x70000017, bucket=1, bytes=754, packets=7, actions=[GROUP:0x20000016] |
| 427 | id=0x20000016, state=ADDED, type=INDIRECT, bytes=754, packets=7, appId=org.onosproject.segmentrouting, referenceCount=0 |
| 428 | id=0x20000016, bucket=1, bytes=754, packets=7, actions=[ETH_DST:00:00:00:00:02:05, ETH_SRC:00:00:00:00:02:27, VLAN_ID:4094, GROUP:0xffe0002] |
| 429 | id=0xffe0002, state=ADDED, type=INDIRECT, bytes=754, packets=7, appId=org.onosproject.segmentrouting, referenceCount=0 |
| 430 | id=0xffe0002, bucket=1, bytes=754, packets=7, actions=[VLAN_POP, OUTPUT:2] |
| 431 | Output through of:0000000000000227/2 |
| 432 | Device of:0000000000000205 |
| 433 | Input from of:0000000000000205/2 |
| 434 | Flows |
| 435 | ADDED, bytes=2105886, packets=24084, table=0, priority=0, selector=[], treatment=[transition=TABLE:10] |
| 436 | ADDED, bytes=358792, packets=4275, table=10, priority=32768, selector=[IN_PORT:1, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:4094], transition=TABLE:20] |
| 437 | ADDED, bytes=16416, packets=48, table=20, priority=32768, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:05, ETH_TYPE:ipv4, VLAN_VID:4094], treatment=[transition=TABLE:30] |
| 438 | ADDED, bytes=196, packets=2, table=30, priority=64010, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.1/32], treatment=[deferred=[GROUP:0x2000002a], transition=TABLE:60] |
| 439 | ADDED, bytes=1370884, packets=15163, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]] |
| 440 | Groups |
| 441 | id=0x2000002a, state=ADDED, type=INDIRECT, bytes=196, packets=2, appId=org.onosproject.segmentrouting, referenceCount=1 |
| 442 | id=0x2000002a, bucket=1, bytes=196, packets=2, actions=[ETH_DST:00:AA:00:00:00:03, ETH_SRC:00:00:00:00:02:05, VLAN_ID:30, GROUP:0x1e0003] |
| 443 | id=0x1e0003, state=ADDED, type=INDIRECT, bytes=12960, packets=126, appId=org.onosproject.segmentrouting, referenceCount=1 |
| 444 | id=0x1e0003, bucket=1, bytes=12960, packets=126, actions=[VLAN_POP, OUTPUT:3] |
| 445 | id=0x2000002a, state=ADDED, type=INDIRECT, bytes=196, packets=2, appId=org.onosproject.segmentrouting, referenceCount=1 |
| 446 | id=0x2000002a, bucket=1, bytes=196, packets=2, actions=[ETH_DST:00:AA:00:00:00:03, ETH_SRC:00:00:00:00:02:05, VLAN_ID:30, GROUP:0x1e0003] |
| 447 | id=0x1e0003, state=ADDED, type=INDIRECT, bytes=12960, packets=126, appId=org.onosproject.segmentrouting, referenceCount=1 |
| 448 | id=0x1e0003, bucket=1, bytes=12960, packets=126, actions=[VLAN_POP, OUTPUT:3] |
| 449 | Output through of:0000000000000205/3 |
| 450 | Path [of:0000000000000204/3, of:0000000000000204/1, of:0000000000000226/1, of:0000000000000226/2, of:0000000000000205/1, of:0000000000000205/3] |
| 451 | Device of:0000000000000204 |
| 452 | Input from of:0000000000000204/3 |
| 453 | Flows |
| 454 | ADDED, bytes=733506, packets=8691, table=0, priority=0, selector=[], treatment=[transition=TABLE:10] |
| 455 | ADDED, bytes=11110, packets=61, table=10, priority=32768, selector=[IN_PORT:3, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:20], transition=TABLE:20] |
| 456 | ADDED, bytes=8698, packets=29, table=20, priority=32768, selector=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:20], treatment=[transition=TABLE:30] |
| 457 | ADDED, bytes=16612, packets=50, table=30, priority=48010, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.0/24], treatment=[deferred=[GROUP:0x70000010], transition=TABLE:60] |
| 458 | ADDED, bytes=34400, packets=112, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]] |
| 459 | Groups |
| 460 | id=0x70000010, state=ADDED, type=SELECT, bytes=16906, packets=53, appId=org.onosproject.segmentrouting, referenceCount=3 |
| 461 | id=0x70000010, bucket=1, bytes=490, packets=5, actions=[GROUP:0x9200000d] |
| 462 | id=0x70000010, bucket=2, bytes=16416, packets=48, actions=[GROUP:0x9200000f] |
| 463 | id=0x9200000d, state=ADDED, type=INDIRECT, bytes=490, packets=5, appId=org.onosproject.segmentrouting, referenceCount=0 |
| 464 | id=0x9200000d, bucket=1, bytes=490, packets=5, actions=[VLAN_POP, MPLS_PUSH:mpls_unicast, MPLS_LABEL:205, GROUP:0x9000000c, VLAN_PUSH:vlan, VLAN_ID:4095] |
| 465 | id=0x9000000c, state=ADDED, type=INDIRECT, bytes=490, packets=5, appId=org.onosproject.segmentrouting, referenceCount=0 |
| 466 | id=0x9000000c, bucket=1, bytes=490, packets=5, actions=[ETH_DST:00:00:00:00:02:27, ETH_SRC:00:00:00:00:02:04, VLAN_ID:4094, GROUP:0xffe0002] |
| 467 | id=0xffe0002, state=ADDED, type=INDIRECT, bytes=726, packets=7, appId=org.onosproject.segmentrouting, referenceCount=0 |
| 468 | id=0xffe0002, bucket=1, bytes=726, packets=7, actions=[VLAN_POP, OUTPUT:2] |
| 469 | id=0x9200000f, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0 |
| 470 | id=0x9200000f, bucket=1, bytes=16416, packets=48, actions=[VLAN_POP, MPLS_PUSH:mpls_unicast, MPLS_LABEL:205, GROUP:0x9000000e, VLAN_PUSH:vlan, VLAN_ID:4095] |
| 471 | id=0x9000000e, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0 |
| 472 | id=0x9000000e, bucket=1, bytes=16416, packets=48, actions=[ETH_DST:00:00:00:00:02:26, ETH_SRC:00:00:00:00:02:04, VLAN_ID:4094, GROUP:0xffe0001] |
| 473 | id=0xffe0001, state=ADDED, type=INDIRECT, bytes=16416, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0 |
| 474 | id=0xffe0001, bucket=1, bytes=16416, packets=48, actions=[VLAN_POP, OUTPUT:1] |
| 475 | Output through of:0000000000000204/1 |
| 476 | Device of:0000000000000226 |
| 477 | Input from of:0000000000000226/1 |
| 478 | Flows |
| 479 | ADDED, bytes=701910, packets=8507, table=0, priority=0, selector=[], treatment=[transition=TABLE:10] |
| 480 | ADDED, bytes=358984, packets=4275, table=10, priority=32768, selector=[IN_PORT:1, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:4094], transition=TABLE:20] |
| 481 | ADDED, bytes=16608, packets=48, table=20, priority=32768, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:26, ETH_TYPE:mpls_unicast, VLAN_VID:4094], treatment=[transition=TABLE:23] |
| 482 | ADDED, bytes=17158, packets=53, table=23, priority=0, selector=[], treatment=[transition=TABLE:24] |
| 483 | ADDED, bytes=16608, packets=48, table=24, priority=100, selector=[ETH_TYPE:mpls_unicast, MPLS_LABEL:205, MPLS_BOS:true], treatment=[deferred=[MPLS_POP:ipv4, DEC_TTL, GROUP:0x7000000b], transition=TABLE:60] |
| 484 | ADDED, bytes=17158, packets=53, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]] |
| 485 | Groups |
| 486 | id=0x7000000b, state=ADDED, type=SELECT, bytes=16608, packets=48, appId=org.onosproject.segmentrouting, referenceCount=2 |
| 487 | id=0x7000000b, bucket=1, bytes=16608, packets=48, actions=[GROUP:0x20000009] |
| 488 | id=0x20000009, state=ADDED, type=INDIRECT, bytes=16608, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0 |
| 489 | id=0x20000009, bucket=1, bytes=16608, packets=48, actions=[ETH_DST:00:00:00:00:02:05, ETH_SRC:00:00:00:00:02:26, VLAN_ID:4094, GROUP:0xffe0002] |
| 490 | id=0xffe0002, state=ADDED, type=INDIRECT, bytes=16608, packets=48, appId=org.onosproject.segmentrouting, referenceCount=0 |
| 491 | id=0xffe0002, bucket=1, bytes=16608, packets=48, actions=[VLAN_POP, OUTPUT:2] |
| 492 | Output through of:0000000000000226/2 |
| 493 | Device of:0000000000000205 |
| 494 | Input from of:0000000000000205/1 |
| 495 | Flows |
| 496 | ADDED, bytes=2105886, packets=24084, table=0, priority=0, selector=[], treatment=[transition=TABLE:10] |
| 497 | ADDED, bytes=358792, packets=4275, table=10, priority=32768, selector=[IN_PORT:1, VLAN_VID:None], treatment=[immediate=[VLAN_PUSH:vlan, VLAN_ID:4094], transition=TABLE:20] |
| 498 | ADDED, bytes=16416, packets=48, table=20, priority=32768, selector=[IN_PORT:1, ETH_DST:00:00:00:00:02:05, ETH_TYPE:ipv4, VLAN_VID:4094], treatment=[transition=TABLE:30] |
| 499 | ADDED, bytes=196, packets=2, table=30, priority=64010, selector=[ETH_TYPE:ipv4, IPV4_DST:10.0.3.1/32], treatment=[deferred=[GROUP:0x2000002a], transition=TABLE:60] |
| 500 | ADDED, bytes=1370884, packets=15163, table=60, priority=0, selector=[], treatment=[immediate=[NOACTION]] |
| 501 | Groups |
| 502 | id=0x2000002a, state=ADDED, type=INDIRECT, bytes=196, packets=2, appId=org.onosproject.segmentrouting, referenceCount=1 |
| 503 | id=0x2000002a, bucket=1, bytes=196, packets=2, actions=[ETH_DST:00:AA:00:00:00:03, ETH_SRC:00:00:00:00:02:05, VLAN_ID:30, GROUP:0x1e0003] |
| 504 | id=0x1e0003, state=ADDED, type=INDIRECT, bytes=12960, packets=126, appId=org.onosproject.segmentrouting, referenceCount=1 |
| 505 | id=0x1e0003, bucket=1, bytes=12960, packets=126, actions=[VLAN_POP, OUTPUT:3] |
| 506 | id=0x2000002a, state=ADDED, type=INDIRECT, bytes=196, packets=2, appId=org.onosproject.segmentrouting, referenceCount=1 |
| 507 | id=0x2000002a, bucket=1, bytes=196, packets=2, actions=[ETH_DST:00:AA:00:00:00:03, ETH_SRC:00:00:00:00:02:05, VLAN_ID:30, GROUP:0x1e0003] |
| 508 | id=0x1e0003, state=ADDED, type=INDIRECT, bytes=12960, packets=126, appId=org.onosproject.segmentrouting, referenceCount=1 |
| 509 | id=0x1e0003, bucket=1, bytes=12960, packets=126, actions=[VLAN_POP, OUTPUT:3] |
| 510 | Output through of:0000000000000205/3 |
| 511 | Result: |
| 512 | Reached required destination Host |
| 513 | Reached required destination Host |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 514 | |
| 515 | Successful Example |
| 516 | ------------------ |
Zack Williams | d63d35b | 2020-06-23 14:12:46 -0700 | [diff] [blame] | 517 | |
| 518 | In this example we trace a packet that should be bridged between two hosts in |
| 519 | the same subnet |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 520 | |
| 521 | .. code-block:: console |
| 522 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 523 | onos> t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:AA:00:00:00:02 -vid None |
| 524 | Tracing packet: [IN_PORT:3, ETH_DST:00:AA:00:00:00:02, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.2.2/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true] |
| 525 | Paths |
| 526 | [of:0000000000000204/3, of:0000000000000204/4] |
| 527 | Result: |
| 528 | Reached required destination Host |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 529 | |
| 530 | Failed Example |
| 531 | -------------- |
Zack Williams | d63d35b | 2020-06-23 14:12:46 -0700 | [diff] [blame] | 532 | |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 533 | Obviously failures can happen along one of the paths that the packet can take. |
Zack Williams | d63d35b | 2020-06-23 14:12:46 -0700 | [diff] [blame] | 534 | |
| 535 | In our sample we removed a flow in ``of:0000000000000226`` thus disabling that |
| 536 | possible path for our initial packet. |
| 537 | |
| 538 | The output of our trace would then look something like: |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 539 | |
| 540 | .. code-block:: console |
| 541 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 542 | onos> t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:00:00:00:02:04 -vid None |
| 543 | Tracing packet: [IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_SRC:00:AA:00:00:00:01, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.3.1/32, TCP_SRC:12345, TCP_DST:80, MPLS_BOS:true] |
| 544 | Paths |
| 545 | [of:0000000000000204/3, of:0000000000000204/2, of:0000000000000227/1, of:0000000000000227/2, of:0000000000000205/2, of:0000000000000205/3] |
| 546 | [of:0000000000000204/3, of:0000000000000204/1, of:0000000000000226/1] |
| 547 | Result: |
| 548 | Reached required destination Host |
| 549 | No output out of device of:0000000000000226 Packet is dropped |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 550 | |
Zack Williams | d63d35b | 2020-06-23 14:12:46 -0700 | [diff] [blame] | 551 | Notice that on the second path that was previously going through |
| 552 | ``of:0000000000000226`` now the packet is dropped due to the rule we removed. |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 553 | |
| 554 | Example of Various Packet Types |
| 555 | ------------------------------- |
| 556 | These sample commands refer to the Mininet environment found in trellis.py [#trellis.py]_ |
| 557 | |
| 558 | - L2 Unicast IPv4 |
| 559 | |
| 560 | .. code-block:: console |
| 561 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 562 | t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -d 10.0.2.2 -dm 00:AA:00:00:00:02 |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 563 | |
| 564 | - L2 Unicast IPv6 |
| 565 | |
| 566 | .. code-block:: console |
| 567 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 568 | t3-troubleshoot -vv -s 2000::301 -sp of:0000000000000205/5 -sm 00:BB:00:00:00:03 -d 2000::302 -dm 00:BB:00:00:00:04 -et ipv6 |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 569 | |
| 570 | - L3 Unicast IPv4 |
| 571 | |
| 572 | .. code-block:: console |
| 573 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 574 | t3-troubleshoot -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -d 10.0.3.1 -dm 00:00:00:00:02:04 |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 575 | |
| 576 | - L3 Unicast IPv6 |
| 577 | |
| 578 | .. code-block:: console |
| 579 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 580 | t3-troubleshoot -vv -s 2000::301 -sp of:0000000000000205/5 -sm 00:BB:00:00:00:03 -d 2000::202 -dm 00:00:00:00:02:05 -et ipv6 |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 581 | |
| 582 | - ARP |
| 583 | |
| 584 | .. code-block:: console |
| 585 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 586 | t3-troubleshoot -sp of:0000000000000204/3 -et ARP |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 587 | |
| 588 | - DHCP |
| 589 | |
| 590 | .. code-block:: console |
| 591 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 592 | t3-troubleshoot -vv -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -d 10.0.3.253 -dm 00:00:00:00:02:04 -ipp 17 -udps 67 -udpd 67 |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 593 | |
| 594 | - LLDP |
| 595 | |
| 596 | .. code-block:: console |
| 597 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 598 | t3-troubleshoot -sp of:0000000000000204/3 -et lldp |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 599 | |
| 600 | - BDDP |
| 601 | |
| 602 | .. code-block:: console |
| 603 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 604 | t3-troubleshoot -sp of:0000000000000204/3 -et bddp |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 605 | |
| 606 | - LLDP from CONTROLLER |
| 607 | |
| 608 | T3 is also capable of tracing packets coming in from the CONTROLLER. |
Zack Williams | d63d35b | 2020-06-23 14:12:46 -0700 | [diff] [blame] | 609 | |
| 610 | We use the same input port with the possibility to give CONTROLLER as logical |
| 611 | port number. |
| 612 | |
| 613 | This is a mere convenience to carry the information inside T3 and do the |
| 614 | proper flooding to all the ports of the given device. |
| 615 | |
| 616 | This case is not treated like all the rest and the input port here is a |
| 617 | logical entity that has no match on any physical element on the device. |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 618 | |
| 619 | .. code-block:: console |
| 620 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 621 | t3-troubleshoot -sp of:0000000000000204/CONTROLLER -et lldp |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 622 | |
| 623 | - Multicast |
| 624 | |
| 625 | T3 is also capable of tracing multicast packets. |
| 626 | |
| 627 | .. code-block:: console |
| 628 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 629 | t3-troubleshoot -vv -sp of:0000000000000205/9 -et ipv4 -d 224.0.0.1 -dm 01:00:5e:00:00:01 |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 630 | |
| 631 | |
| 632 | t3-troubleshoot-simple |
| 633 | ====================== |
Zack Williams | d63d35b | 2020-06-23 14:12:46 -0700 | [diff] [blame] | 634 | |
| 635 | T3 also offers a simplified version of the command to be able to ping between |
| 636 | two given hosts without having to specify all the information. |
| 637 | |
| 638 | This commands bases itself on top of some configuration present in ONOS through |
| 639 | netcfg, namely the VLAN interface configuration and the segment routing leaf |
| 640 | router MAC for L3 Unicast. If that is not present the user needs to input this |
| 641 | information manually recurring to the the previously shown command. Same |
| 642 | verbosity levels and output as the troubleshoot command are possible. |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 643 | |
| 644 | t3-troubleshoot-simple Help |
| 645 | --------------------------- |
| 646 | .. code-block:: console |
| 647 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 648 | onos> t3-troubleshoot-simple --help |
| 649 | DESCRIPTION |
| 650 | onos:troubleshoot-simple |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 651 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 652 | Given two hosts troubleshoots flows and groups between them, in case of segment routing |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 653 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 654 | SYNTAX |
| 655 | onos:troubleshoot-simple [options] one two |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 656 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 657 | ARGUMENTS |
| 658 | one |
| 659 | One host ID |
| 660 | two |
| 661 | Another host ID |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 662 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 663 | OPTIONS |
| 664 | -vv, --veryverbose |
| 665 | Outputs flows and groups for every device |
| 666 | -j, --json |
| 667 | Output JSON |
| 668 | --help |
| 669 | Display this help message |
| 670 | -et, --ethType |
| 671 | ETH Type |
| 672 | (defaults to ipv4) |
| 673 | -v, --verbose |
| 674 | Outputs complete path |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 675 | |
| 676 | t3-troubleshoot-simple Example |
| 677 | ------------------------------ |
| 678 | Examples are given for the same trellis.py [#trellis.py]_ script. |
| 679 | |
| 680 | - L2 Unicast |
| 681 | |
| 682 | Hosts are under the same leaf, only interface configuration is needed. |
| 683 | |
| 684 | .. code-block:: console |
| 685 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 686 | t3-troubleshoot-simple -v -et IPV4 00:AA:00:00:00:01/None 00:AA:00:00:00:02/None |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 687 | |
| 688 | - L3 Multicast |
| 689 | |
| 690 | Hosts are under different leafs, also segment routing configuration config is needed for the leaf's routerMac. |
| 691 | |
| 692 | .. code-block:: console |
| 693 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 694 | t3-troubleshoot-simple -et IPV4 00:AA:00:00:00:01/None 00:AA:00:00:00:03/None |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 695 | |
| 696 | |
| 697 | t3-troubleshoot-pingall |
| 698 | ======================= |
| 699 | T3 also offers a ping-all command to be able to test ping between any given hosts of an EthType. |
| 700 | This commands bases itself on top of the simplified Host to Host command. It has two verbosity levels, |
| 701 | one just with Host and Destination with the result message of the Trace and one that shows also the packet and the paths. |
| 702 | |
| 703 | t3-troubleshoot-pingall Help |
| 704 | ---------------------------- |
| 705 | .. code-block:: console |
| 706 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 707 | onos> t3-troubleshoot-pingall --help |
| 708 | DESCRIPTION |
| 709 | onos:t3-troubleshoot-pingall |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 710 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 711 | Traces a ping between all hosts in the system of a given ETH type |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 712 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 713 | SYNTAX |
| 714 | onos:t3-troubleshoot-pingall [options] |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 715 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 716 | OPTIONS |
| 717 | -j, --json |
| 718 | Output JSON |
| 719 | --help |
| 720 | Display this help message |
| 721 | -et, --ethType |
| 722 | ETH Type |
| 723 | (defaults to ipv4) |
| 724 | -v, --verbose |
| 725 | Outputs trace for each host to host combination |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 726 | |
| 727 | t3-troubleshoot-pingall Examples |
| 728 | -------------------------------- |
| 729 | - IPv4 hosts |
| 730 | |
| 731 | .. code-block:: console |
| 732 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 733 | t3-troubleshoot-pingall |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 734 | |
| 735 | - IPv6 hosts |
| 736 | |
| 737 | .. code-block:: console |
| 738 | |
| 739 | t3-troubleshoot-pingall -et IPV6 |
| 740 | |
| 741 | |
| 742 | t3-troubleshoot-mcast |
| 743 | ===================== |
Zack Williams | d63d35b | 2020-06-23 14:12:46 -0700 | [diff] [blame] | 744 | |
| 745 | T3 also offers a mcast command queries the system for all the multicast routes |
| 746 | given a certain vlanId. It prints all the traces given a certain group, source |
| 747 | and source connect point. It has two "standard" verbosity levels. |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 748 | |
| 749 | t3-troubleshoot-mcast Help |
| 750 | -------------------------- |
| 751 | .. code-block:: console |
| 752 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 753 | onos> t3-troubleshoot-mcast --help |
| 754 | DESCRIPTION |
| 755 | onos:t3-troubleshoot-mcast |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 756 | |
Seyeon Jeong | 50d3c45 | 2020-05-29 14:07:57 -0700 | [diff] [blame^] | 757 | Traces all the mcast routes present in the system |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 758 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 759 | SYNTAX |
| 760 | onos:t3-troubleshoot-mcast [options] |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 761 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 762 | OPTIONS |
| 763 | -j, --json |
| 764 | Output JSON |
| 765 | --help |
| 766 | Display this help message |
| 767 | -vv, --veryverbose |
| 768 | Outputs details of every trace |
| 769 | -vid, --vlanId |
| 770 | Vlan of incoming packet |
| 771 | (defaults to None) |
| 772 | -v, --verbose |
| 773 | Outputs trace for each host to host combination |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 774 | |
| 775 | REST API |
| 776 | ======== |
| 777 | T3 now also offers a way to execute the troubleshoot through REST API endpoints. |
| 778 | You can find the description and test the API at the ONOS docs endpoint here: |
| 779 | |
| 780 | .. code-block:: text |
| 781 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 782 | <ONOS_IP>:8181/onos/v1/docs/#/t3 |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 783 | |
| 784 | The rest API offered are a non verbose and a verbose version of the troubleshoot simple command. |
Zack Williams | d63d35b | 2020-06-23 14:12:46 -0700 | [diff] [blame] | 785 | |
| 786 | It bases itself on top of some configuration present in ONOS through netcfg, |
| 787 | namely the VLAN interface configuration and the segment routing leaf router MAC |
| 788 | for L3 Unicast. |
| 789 | |
| 790 | The use need to specify just the Source Host, the destination host and the ETH |
| 791 | type. |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 792 | |
| 793 | .. code-block:: console |
| 794 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 795 | curl -X GET --header 'Accept: application/json' 'http://localhost:8181/onos/v1/t3/t3/simple/<host_src>/<host_dst>/<eht_type>' |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 796 | |
| 797 | .. code-block:: console |
| 798 | |
Charles Chan | 8299f07 | 2019-09-23 11:32:26 -0700 | [diff] [blame] | 799 | curl -X GET --header 'Accept: application/json' 'http://localhost:8181/onos/v1/t3/t3/simple/<host_src>/<host_dst>/<eht_type>/verbose |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 800 | |
| 801 | |
| 802 | Last Resort |
| 803 | =========== |
| 804 | |
Zack Williams | d63d35b | 2020-06-23 14:12:46 -0700 | [diff] [blame] | 805 | In the case where you can't figure out what is going wrong even with help of |
| 806 | T3, you can seek help on Trellis developer mailing list |
| 807 | ``trellis-dev@opennetworking.org``. |
| 808 | |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 809 | There are a few things we would like you to attach: |
| 810 | |
| 811 | - **Issue description** |
Zack Williams | d63d35b | 2020-06-23 14:12:46 -0700 | [diff] [blame] | 812 | |
| 813 | - **Environment description**, such as ONOS version, switch model and OF-DPA |
| 814 | version |
| 815 | |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 816 | - **Steps of reproduction**, as detail as possible |
Zack Williams | d63d35b | 2020-06-23 14:12:46 -0700 | [diff] [blame] | 817 | |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 818 | - **Diagnostics**. |
Zack Williams | d63d35b | 2020-06-23 14:12:46 -0700 | [diff] [blame] | 819 | |
Seyeon Jeong | 50d3c45 | 2020-05-29 14:07:57 -0700 | [diff] [blame^] | 820 | We have built a tool ``onos-diagnostics`` [#onos-diagnostics_src]_ [#onos-diagnostics_man]_ |
| 821 | to help you easily collect and package ONOS diagnostics |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 822 | |
| 823 | |
| 824 | Reference |
| 825 | ========= |
| 826 | |
Seyeon Jeong | 50d3c45 | 2020-05-29 14:07:57 -0700 | [diff] [blame^] | 827 | .. [#onos-diagnostics_src] `<https://github.com/opennetworkinglab/onos/blob/master/tools/package/runtime/bin/onos-diagnostics>`_ |
| 828 | .. [#onos-diagnostics_man] `<https://wiki.onosproject.org/display/ONOS/ONOS+Remote+Admin+Tools>`_ |
Charles Chan | 4ade88d | 2019-09-16 16:30:31 -0700 | [diff] [blame] | 829 | .. [#trellis.py] `<https://github.com/opennetworkinglab/routing/blob/master/trellis/trellis.py>`_ |