[AETHER-76] Complete T3 offline mode
Regarding the T3 offline mode support, Troubleshooting Guide in the
trellis doc is updated (https://docs.trellisfabric.org/troubleshooting.html).
- Explanations on Network Information Base (NIB) and purposes of the offline mode support
- Usage instruction for the t3-load-file CLI command
- Usage instruction for the t3-load-snapshot CLI command
Change-Id: Iab8340ee50e539c40bf925ea1eec3a410357ff56
diff --git a/troubleshooting.rst b/troubleshooting.rst
index 0a50212..797c5e2 100644
--- a/troubleshooting.rst
+++ b/troubleshooting.rst
@@ -32,6 +32,142 @@
- DHCP
+Network Information Base (NIB)
+==============================
+T3 analyzes network states that are stored in the NIB.
+A NIB is a collection of snapshots captured from various ONOS stores such as flows, groups, links...etc.
+The NIB, acting as a cache, allows T3 troubleshooting logic to avoid massive interaction with live ONOS subsystems,
+enabling near-runtime or postmortem analysis in a production environment.
+
+There are two methods of loading the network snapshots into the NIB:
+(1) from a live running system or (2) from a onos-diagnostics file dump.
+Users can choose one load command before troubleshooting commands,
+or can run troubleshooting commands alone (then the NIB is automatically loaded via (1)).
+
+
+t3-load-file
+=========================
+T3 offers this command to fill the NIB with files that represent snapshots of the network states in a certain time.
+
+t3-load-file Help
+------------------------------
+
+.. code-block:: console
+
+ onos> t3-load-file --help
+ DESCRIPTION
+ onos:t3-load-file
+
+ Load the NIB with onos-diagnostics dump files
+
+ SYNTAX
+ onos:t3-load-file [options] rootDir
+
+ ARGUMENTS
+ rootDir
+ Specify the location of the directory where the previous onos-diagnostics command has extracted its dump files (e.g. /tmp/onos-diags/127.0.0.1)
+
+ OPTIONS
+ --help
+ Display this help message
+ -j, --json
+ Output JSON
+
+t3-load-file Example
+---------------------------------
+User is expected to run the tool ``onos-diagnostics`` [#onos-diagnostics_src]_ [#onos-diagnostics_man]_
+to generate target files first.
+
+.. code-block:: console
+
+ user$ onos-diagnostics -xj [IP_ONOS_INSTANCE_1 IP_ONOS_INSTANCE_2 ...]
+ Collecting diagnostics on [IP_ONOS_INSTANCE_1]...logs netcfg.json devices.json device-drivers.json links.json hosts.json ports.json edge-ports.json flows.json groups.json masters.json routes.json mcast-host-show.json Done.
+ Collecting diagnostics on [IP_ONOS_INSTANCE_2]...logs netcfg.json devices.json device-drivers.json links.json hosts.json ports.json edge-ports.json flows.json groups.json masters.json routes.json mcast-host-show.json Done.
+
+Then, execute this command with the path of the location where those JSON files are extracted.
+User can check the creation time and a summary of the NIB contents.
+
+.. code-block:: console
+
+ onos> t3-load-file /tmp/onos-diags/[IP_ONOS_INSTANCE_TARGET]
+ Load target files in: /tmp/onos-diags/[IP_ONOS_INSTANCE_TARGET]/
+ FlowNib created 03-06-2020 01:30 from FILE
+ GroupNib created 03-06-2020 01:30 from FILE
+ LinkNib created 03-06-2020 01:30 from FILE
+ HostNib created 03-06-2020 01:30 from FILE
+ DeviceNib created 03-06-2020 01:30 from FILE
+ DriverNib created 03-06-2020 01:30 from FILE
+ MastershipNib created 03-06-2020 01:30 from FILE
+ EdgePortNib created 03-06-2020 01:30 from FILE
+ RouteNib created 03-06-2020 01:30 from FILE
+ NetworkConfigNib created 03-06-2020 01:30 from FILE
+ MulticastRouteNib created 03-06-2020 01:30 from FILE
+
+t3-load-snapshot
+=============================
+T3 offers this command to fill the NIB with snapshots of the network states at the execution time.
+
+t3-load-snapshot Help
+----------------------------------
+
+.. code-block:: console
+
+ onos> t3-load-snapshot --help
+ DESCRIPTION
+ onos:t3-load-snapshot
+
+ Load the NIB with the network states stored in the ONOS instance where the T3 is running
+
+ SYNTAX
+ onos:t3-load-snapshot [options]
+
+ OPTIONS
+ --help
+ Display this help message
+ -j, --json
+ Output JSON
+
+t3-load-snapshot Example
+-------------------------------------
+User can check the creation time and a summary of the NIB contents.
+**SNAPSHOT** indicates that the source of the current NIB is the ONOS instance where the T3 is running.
+
+.. code-block:: console
+
+ onos> t3-load-snapshot
+ Load current network states from ONOS stores
+ FlowNib created 03-06-2020 01:40 from SNAPSHOT
+ GroupNib created 03-06-2020 01:40 from SNAPSHOT
+ LinkNib created 03-06-2020 01:40 from SNAPSHOT
+ HostNib created 03-06-2020 01:40 from SNAPSHOT
+ DeviceNib created 03-06-2020 01:40 from SNAPSHOT
+ DriverNib created 03-06-2020 01:40 from SNAPSHOT
+ MastershipNib created 03-06-2020 01:40 from SNAPSHOT
+ EdgePortNib created 03-06-2020 01:40 from SNAPSHOT
+ RouteNib created 03-06-2020 01:40 from SNAPSHOT
+ NetworkConfigNib created 03-06-2020 01:40 from SNAPSHOT
+ MulticastRouteNib created 03-06-2020 01:40 from SNAPSHOT
+
+If one of T3 commands below is used before loading the NIB from **SNAPSHOT** or **FILE** (so the NIB is empty),
+the command internally executes **t3-load-snapshot** to proceed with auto-filled NIB.
+
+.. code-block:: console
+
+ onos> t3-troubleshoot-simple 00:AA:00:00:00:01/None 00:AA:00:00:00:02/None
+ *** NIB is invalid. Snapshots for the NIB have been auto-filled: ***
+ Load current network states from ONOS stores
+ FlowNib created 03-06-2020 01:44 from SNAPSHOT
+ GroupNib created 03-06-2020 01:44 from SNAPSHOT
+ LinkNib created 03-06-2020 01:44 from SNAPSHOT
+ ...
+ Tracing between: 00:AA:00:00:00:01/None and 00:AA:00:00:00:02/None
+ Tracing Packet: DefaultTrafficSelector{criteria=[IN_PORT:3, ETH_DST:00:00:00:00:02:04, ETH_TYPE:ipv4, VLAN_VID:None, IPV4_SRC:10.0.2.1/32, IPV4_DST:10.0.2.2/32]}
+ Paths
+ [of:0000000000000204/3, of:0000000000000204/4]
+ Result:
+ Reached required destination Host of:0000000000000204/4
+
+
t3-troubleshoot
===============
@@ -618,7 +754,7 @@
DESCRIPTION
onos:t3-troubleshoot-mcast
- Traces all the mcast routes present in the system
+ Traces all the mcast routes present in the system
SYNTAX
onos:t3-troubleshoot-mcast [options]
@@ -681,12 +817,13 @@
- **Diagnostics**.
- We have built a tool ``onos-diagnostics`` (see `ONOS Remote Admin Tools
- <https://wiki.onosproject.org/display/ONOS/ONOS+Remote+Admin+Tools>`_) to
- help you easily collect and package ONOS diagnostics
+We have built a tool ``onos-diagnostics`` [#onos-diagnostics_src]_ [#onos-diagnostics_man]_
+to help you easily collect and package ONOS diagnostics
Reference
=========
+.. [#onos-diagnostics_src] `<https://github.com/opennetworkinglab/onos/blob/master/tools/package/runtime/bin/onos-diagnostics>`_
+.. [#onos-diagnostics_man] `<https://wiki.onosproject.org/display/ONOS/ONOS+Remote+Admin+Tools>`_
.. [#trellis.py] `<https://github.com/opennetworkinglab/routing/blob/master/trellis/trellis.py>`_