Update Sphinx version and versioning process
- Updated to newer Sphinx version
- Changed to use doc8 to check .rst files
- Added a dictionary (dict.txt) and spellchecking
- Reformatted and fixed issues that were found in the content
Change-Id: If7b35e01ee8be25dbbd1ecd0e67b264aa6cc2a94
diff --git a/Makefile b/Makefile
index 724d01c..d1160da 100644
--- a/Makefile
+++ b/Makefile
@@ -10,46 +10,51 @@
SOURCEDIR ?= .
BUILDDIR ?= _build
+VENV_NAME := doc_venv
+
# Put it first so that "make" without argument is like "make help".
-help: doc_venv
- source ./doc_venv/bin/activate ;\
+help: $(VENV_NAME)
+ source ./$(VENV_NAME)/bin/activate ;\
$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help lint reload Makefile test
# Create the virtualenv with all the tools installed
-doc_venv:
- virtualenv -p python3 doc_venv ;\
- source ./doc_venv/bin/activate ;\
- pip install livereload ;\
+$(VENV_NAME):
+ virtualenv -p python3 $(VENV_NAME) ;\
+ source ./$(VENV_NAME)/bin/activate ;\
pip install -r requirements.txt
# automatically reload changes in browser as they're made
-reload: doc_venv
- source ./doc_venv/bin/activate ;\
- set -u ;\
+reload: $(VENV_NAME)
+ source $</bin/activate ; set -u ;\
sphinx-reload $(SOURCEDIR)
+# build multiple versions
+multiversion: $(VENV_NAME) Makefile
+ source $</bin/activate ; set -u ;\
+ sphinx-multiversion "$(SOURCEDIR)" "$(BUILDDIR)/multiversion" $(SPHINXOPTS)
+ cp "$(SOURCEDIR)/_templates/meta_refresh.html" "$(BUILDDIR)/multiversion/index.html"
+
# lint and link verification. linkcheck is built into sphinx
-test: lint linkcheck
+test: lint spelling linkcheck
-lint: rst-lint
+lint: doc8
-rst-lint: doc_venv
- source ./doc_venv/bin/activate ;\
- set -u ;\
- rstcheck -r $$(find . -name \*.rst ! -path "*doc_venv*")
+doc8: $(VENV_NAME) | $(OTHER_REPO_DOCS)
+ source $</bin/activate ; set -u ;\
+ doc8 --max-line-length 119 \
+ $$(find . -name \*.rst ! -path "*venv*" ! -path "*vendor*" ! -path "*repos*" )
# clean up
clean:
rm -rf $(BUILDDIR)
clean-all: clean
- rm -rf doc_venv
+ rm -rf $(VENV_NAME)
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
-%: Makefile doc_venv | $(OTHER_REPO_DOCS)
- source ./doc_venv/bin/activate ;\
- set -u ;\
+%: $(VENV_NAME) | $(OTHER_REPO_DOCS)
+ source $</bin/activate ; set -u ;\
$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
diff --git a/_templates/meta_refresh.html b/_templates/meta_refresh.html
new file mode 100644
index 0000000..f836173
--- /dev/null
+++ b/_templates/meta_refresh.html
@@ -0,0 +1,8 @@
+<!--
+SPDX-FileCopyrightText: © 2020 Open Networking Foundation <support@opennetworking.org>
+SPDX-License-Identifier: Apache-2.0
+--!>
+<html><head>
+ <meta charset="UTF-8">
+ <meta http-equiv="refresh" content="0;url=1.12/index.html" />
+</head></html>
diff --git a/_templates/versions.html b/_templates/versions.html
new file mode 100644
index 0000000..8ed5e31
--- /dev/null
+++ b/_templates/versions.html
@@ -0,0 +1,31 @@
+<!--
+SPDX-FileCopyrightText: © 2020 Open Networking Foundation <support@opennetworking.org>
+SPDX-License-Identifier: Apache-2.0
+--!>
+{%- if current_version %}
+<div class="rst-versions" data-toggle="rst-versions" role="note" aria-label="versions">
+ <span class="rst-current-version" data-toggle="rst-current-version">
+ <span class="fa fa-book"> Other Versions</span>
+ v: {{ current_version.name }}
+ <span class="fa fa-caret-down"></span>
+ </span>
+ <div class="rst-other-versions">
+ {%- if versions.tags %}
+ <dl>
+ <dt>Tags</dt>
+ {%- for item in versions.tags %}
+ <dd><a href="{{ item.url }}">{{ item.name }}</a></dd>
+ {%- endfor %}
+ </dl>
+ {%- endif %}
+ {%- if versions.branches %}
+ <dl>
+ <dt>Branches</dt>
+ {%- for item in versions.branches %}
+ <dd><a href="{{ item.url }}">{{ item.name }}</a></dd>
+ {%- endfor %}
+ </dl>
+ {%- endif %}
+ </div>
+</div>
+{%- endif %}
diff --git a/architecture.rst b/architecture.rst
index 7a1f8a6..5bd6a40 100644
--- a/architecture.rst
+++ b/architecture.rst
@@ -9,25 +9,34 @@
Trellis operates as a **hybrid L2/L3 fabric**.
-As a **pure (or classic) SDN** solution, Trellis **does not use any of the traditional control protocols** typically found in networking, a non-exhaustive list of which includes:
-STP, MSTP, RSTP, LACP, MLAG, PIM, IGMP, OSPF, IS-IS, Trill, RSVP, LDP and BGP.
+As a **pure (or classic) SDN** solution, Trellis **does not use any of the
+traditional control protocols** typically found in networking, a non-exhaustive
+list of which includes: STP, MSTP, RSTP, LACP, MLAG, PIM, IGMP, OSPF, IS-IS,
+Trill, RSVP, LDP and BGP.
-Instead, Trellis **uses an SDN Controller (ONOS) decoupled from the data-plane hardware to directly program ASIC forwarding tables using OpenFlow and with OF-DPA**,
-an open-API from Broadcom running on the switches.
+Instead, Trellis **uses an SDN Controller (ONOS) decoupled from the data-plane
+hardware to directly program ASIC forwarding tables using OpenFlow and with
+OF-DPA**, an open-API from Broadcom running on the switches.
-In this design, a set of applications running on ONOS implement all the fabric functionality and features,
-such as **Ethernet switching**, **IP routing**, **multicast**, **DHCP Relay**, **pseudowires** and more.
+In this design, a set of applications running on ONOS implement all the fabric
+functionality and features, such as **Ethernet switching**, **IP routing**,
+**multicast**, **DHCP Relay**, **pseudowires** and more.
.. note::
- You can learn more about Trellis features and design concepts by visiting the `Project Website <https://opennetworking.org/trellis>`_ and
- reading the `Platform Brief <https://www.opennetworking.org/wp-content/uploads/2019/09/TrellisPlatformBrief.pdf>`_.
+ You can learn more about Trellis features and design concepts by visiting
+ the `Project Website <https://opennetworking.org/trellis>`_ and reading the
+ `Platform Brief
+ <https://www.opennetworking.org/wp-content/uploads/2019/09/TrellisPlatformBrief.pdf>`_.
Introduction to OF-DPA Pipeline
-------------------------------
-In this design note, we are going to explain the design choices we have made and how we got around OF-DPA (OpenFlow Data Plane Abstraction) pipeline restrictions to implement the features we need.
-We will start from explaining the OFDPA flow tables and group tables we use.
+In this design note, we are going to explain the design choices we have made
+and how we got around OF-DPA (OpenFlow Data Plane Abstraction) pipeline
+restrictions to implement the features we need. We will start from explaining
+the OFDPA flow tables and group tables we use.
+
Fig. 1 shows the simplified OFDPA pipeline overview.
.. image:: images/arch-ofdpa.png
@@ -35,118 +44,168 @@
Fig. 1 Simplified OF-DPA pipeline overview
-
Flow Tables
-----------
VLAN Table
^^^^^^^^^^
.. note::
- The **VLAN Flow Table (id=10)** is used for IEEE 801.Q VLAN assignment and filtering to specify how VLANs are to be handled on a particular port.
- **All packets must have an associated VLAN id in order to be processed by subsequent tables**.
+ The **VLAN Flow Table (id=10)** is used for IEEE 801.Q VLAN assignment and
+ filtering to specify how VLANs are to be handled on a particular port.
+ **All packets must have an associated VLAN id in order to be processed by
+ subsequent tables**.
**Table miss**: goto **ACL table**.
According to OFDPA spec, we need to assign a VLAN ID even for untagged packets.
-Each untagged packet will be tagged with an **internal VLAN** when being handled by VLAN table.
-The internal VLAN will be popped when the packet is sent to a output port or controller.
-The internal VLAN is assigned according to the subnet configuration of the input port.
-Packets coming from ports that do not have subnet configured (e.g. the spine facing ports) will be tagged with VLAN ID **4094**.
+Each untagged packet will be tagged with an **internal VLAN** when being
+handled by VLAN table. The internal VLAN will be popped when the packet is
+sent to a output port or controller. The internal VLAN is assigned according
+to the subnet configuration of the input port. Packets coming from ports that
+do not have subnet configured (e.g. the spine facing ports) will be tagged with
+VLAN ID **4094**.
-The internal VLAN is also used to determine the subnet when a packet needs to be flooded to all ports in the same subnet. (See L2 Broadcast section for detail.)
-
+The internal VLAN is also used to determine the subnet when a packet needs to
+be flooded to all ports in the same subnet. (See L2 Broadcast section for
+detail.)
Termination MAC Table
^^^^^^^^^^^^^^^^^^^^^
.. note::
- The **Termination MAC (TMAC) Flow Table (id=20)** determines whether to do bridging or routing on a packet.
+ The **Termination MAC (TMAC) Flow Table (id=20)** determines whether to do
+ bridging or routing on a packet.
+
It identifies routed packets their destination MAC, VLAN, and Ethertype.
- Routed packet rule types use a Goto-Table instruction to indicate that the next table is one of the routing tables.
+
+ Routed packet rule types use a Goto-Table instruction to indicate that the
+ next table is one of the routing tables.
**Table miss**: goto **Bridging table**.
-In this table, we determine which table the packet should go to by checking the destination MAC address and the Ethernet type of the packet.
+In this table, we determine which table the packet should go to by checking the
+destination MAC address and the Ethernet type of the packet.
- if dst_mac = router MAC and eth_type = ip, goto **unicast routing** table
-- if dst_mac = router MAC and eth_type = mpls, goto **MPLS table**
-- if dst_mac = multicast MAC (01:00:5F:00:00:00/FF:FF:FF:80:00:00), goto **multicast routing** table
-- none of above, goto **bridging table**
+- if dst_mac = router MAC and eth_type = mpls, goto **MPLS table**
+
+- if dst_mac = multicast MAC (01:00:5F:00:00:00/FF:FF:FF:80:00:00), goto
+ **multicast routing** table
+
+- none of above, goto **bridging table**
MPLS Tables
^^^^^^^^^^^
.. note::
- The MPLS pipeline can support three **MPLS Flow Tables, MPLS Table 0 (id=23), MPLS Table 1 (id=24) and MPLS Table 2 (id=25)**.
- An MPLS Flow Table lookup matches the label in the outermost MPLS shim header in the packets.
+ The MPLS pipeline can support three **MPLS Flow Tables, MPLS Table 0
+ (id=23), MPLS Table 1 (id=24) and MPLS Table 2 (id=25)**.
- - MPLS Table 0 is only used to pop a protection label on platforms that support this table, or to detect an MPLS- TP Section OAM PDU.
+ An MPLS Flow Table lookup matches the label in the outermost MPLS shim
+ header in the packets.
+
+ - MPLS Table 0 is only used to pop a protection label on platforms that
+ support this table, or to detect an MPLS- TP Section OAM PDU.
+
- MPLS Table 1 and MPLS Table 2 can be used for all label operations.
- - MPLS Table 1 and MPLS Table 2 are synchronized flow tables and updating one updates the other
+
+ - MPLS Table 1 and MPLS Table 2 are synchronized flow tables and updating
+ one updates the other
**Table miss**: goto **ACL table**.
We only use MPLS Table 1 (id=24) in current design.
+
MPLS packets are matched by the MPLS label.
-The packet will go to **L3 interface group** with MPLS label being popped and further go to destination leaf switch.
+
+The packet will go to **L3 interface group** with MPLS label being popped and
+further go to destination leaf switch.
Unicast Routing Table
^^^^^^^^^^^^^^^^^^^^^
.. note::
- The **Unicast Routing Flow Table (id=30)** supports routing for potentially large numbers of IPv4 and IPv6 flow entries using the hardware L3 tables.
+ The **Unicast Routing Flow Table (id=30)** supports routing for potentially
+ large numbers of IPv4 and IPv6 flow entries using the hardware L3 tables.
**Table miss**: goto **ACL table**.
-In this table, we determine where to output a packet by checking its **destination IP (unicast)** address.
+In this table, we determine where to output a packet by checking its
+**destination IP (unicast)** address.
-- if dst_ip locates at a **remote switch**, the packet will go to an **L3 ECMP group**, be tagged with MPLS label, and further go to a spine switch
-- if dst_ip locates at the **same switch**, the packet will go to an **L3 interface group** and further go to a host
+- if dst_ip locates at a **remote switch**, the packet will go to an **L3 ECMP
+ group**, be tagged with MPLS label, and further go to a spine switch
-Note that the priority of flow entries in this table is sorted by prefix length.
+- if dst_ip locates at the **same switch**, the packet will go to an **L3
+ interface group** and further go to a host
+
+Note that the priority of flow entries in this table is sorted by prefix
+length.
+
Longer prefix (/32) will have higher priority than shorter prefix (/0).
Multicast Routing Table
^^^^^^^^^^^^^^^^^^^^^^^
.. note::
- The **Multicast Routing Flow Table (id=40)** supports routing for IPv4 and IPv6 multicast packets.
+ The **Multicast Routing Flow Table (id=40)** supports routing for IPv4 and
+ IPv6 multicast packets.
**Table miss**: goto **ACL table**.
Flow entries in this table always match the **destination IP (multicast)**.
-Matched packets will go to an **L3 multicast group** and further go to the next switch or host.
+
+Matched packets will go to an **L3 multicast group** and further go to the next
+switch or host.
Bridging Table
^^^^^^^^^^^^^^
.. note::
- The **Bridging Flow Table (id=50)** supports Ethernet packet switching for potentially large numbers of flow entries using the hardware L2 tables.
- The Bridging Flow Table forwards either based on VLAN (normal switched packets) or Tunnel id (isolated forwarding domain packets),
- with the Tunnel id metadata field used to distinguish different flow table entry types by range assignment.
+ The **Bridging Flow Table (id=50)** supports Ethernet packet switching for
+ potentially large numbers of flow entries using the hardware L2 tables.
+
+ The Bridging Flow Table forwards either based on VLAN (normal switched
+ packets) or Tunnel id (isolated forwarding domain packets), with the Tunnel
+ id metadata field used to distinguish different flow table entry types by
+ range assignment.
**Table miss**: goto **ACL table**.
-In this table, we match the **VLAN ID** and the **destination MAC address** and determine where the packet should be forwarded to.
+In this table, we match the **VLAN ID** and the **destination MAC address** and
+determine where the packet should be forwarded to.
-- if the destination MAC can be matched, the packet will go to the **L2 interface group** and further sent to the destination host.
-- if the destination MAC can not be matched, the packet will go to the **L2 flood group** and further flooded to the same subnet.
- Since we cannot match IP in bridging table, we use the VLAN ID to determine which subnet this packet should be flooded to.
- The VLAN ID can be either (1) the internal VLAN assigned to untagged packets in VLAN table or (2) the VLAN ID that comes with tagged packets.
+- if the destination MAC can be matched, the packet will go to the **L2
+ interface group** and further sent to the destination host.
+
+- if the destination MAC can not be matched, the packet will go to the **L2
+ flood group** and further flooded to the same subnet.
+
+ Since we cannot match IP in bridging table, we use the VLAN ID to determine
+ which subnet this packet should be flooded to.
+
+ The VLAN ID can be either (1) the internal VLAN assigned to untagged packets
+ in VLAN table or (2) the VLAN ID that comes with tagged packets.
Policy ACL Table
^^^^^^^^^^^^^^^^
.. note::
The Policy ACL Flow Table supports wide, multi-field matching.
- Most fields can be wildcard matched, and relative priority must be specified in all flow entry modification API calls.
- This is the preferred table for matching BPDU and ARP packets. It also provides the Metering instruction.
+
+ Most fields can be wildcard matched, and relative priority must be
+ specified in all flow entry modification API calls.
+
+ This is the preferred table for matching BPDU and ARP packets. It also
+ provides the Metering instruction.
**Table miss**: **do nothing**.
- The packet will be forwarded using the output or group in the action set, if any.
+ The packet will be forwarded using the output or group in the action set,
+ if any.
+
If the action set does not have a group or output action the packet is dropped.
-In ACL table we trap **ARP**, **LLDP**, **BDDP**, **DHCP** and send those packets to the **controller**.
-
+In ACL table we trap **ARP**, **LLDP**, **BDDP**, **DHCP** and send those
+packets to the **controller**.
Group Tables
------------
@@ -155,92 +214,136 @@
^^^^^^^^^^^^^
.. note::
OF-DPA L3 ECMP group entries are of OpenFlow type **SELECT**.
- For IP routing the action buckets reference the OF-DPA **L3 Unicast group** entries that are members of the multipath group for ECMP forwarding.
+
+ For IP routing the action buckets reference the OF-DPA **L3 Unicast group**
+ entries that are members of the multipath group for ECMP forwarding.
An OF-DPA L3 ECMP Group entry can also be used in a Provider Edge Router.
- In this packet flow it can chain to either an **MPLS L3 Label** group entry or to an **MPLS Fast Failover** group entry.
- An OF-DPA L3 ECMP Group entry can be specified as a routing target instead of an OF-DPA L3 Unicast Group entry. Selection of an action bucket for forwarding a particular packet is hardware-specific.
+ In this packet flow it can chain to either an **MPLS L3 Label** group entry
+ or to an **MPLS Fast Failover** group entry.
+ An OF-DPA L3 ECMP Group entry can be specified as a routing target instead
+ of an OF-DPA L3 Unicast Group entry. Selection of an action bucket for
+ forwarding a particular packet is hardware-specific.
MPLS Label Group
^^^^^^^^^^^^^^^^
.. note::
MPLS Label Group entries are of OpenFlow **INDIRECT** type.
- There are four MPLS label Group entry subtypes, all with similar structure.
- These can be used in different configurations to **push up to three labels** for tunnel initiation or LSR swap.
+ There are four MPLS label Group entry subtypes, all with similar structure.
+
+ These can be used in different configurations to **push up to three
+ labels** for tunnel initiation or LSR swap.
MPLS Interface Group
^^^^^^^^^^^^^^^^^^^^
.. note::
MPLS Interface Group Entry is of OpenFlow type **INDIRECT**.
- It is used to **set the outgoing L2 header** to reach the next hop label switch router or provider edge router.
-We use **L3 ECMP** group to randomly pick one spine switch when we need to route a packet from leaves to spines.
+ It is used to **set the outgoing L2 header** to reach the next hop label
+ switch router or provider edge router.
-We point each bucket to an **MPLS Label** Group in which the MPLS labels are pushed to the packets to realize Segment Routing mechanism.
-(More specifically, we use the subtype 2 **MPLS L3 VPN Label**).
+We use **L3 ECMP** group to randomly pick one spine switch when we need to
+route a packet from leaves to spines.
-We then point an MPLS Label Group points to an **MPLS Interface** Group in which the destination MAC is set to the next hop (spine router).
+We point each bucket to an **MPLS Label** Group in which the MPLS labels are
+pushed to the packets to realize Segment Routing mechanism. (More
+specifically, we use the subtype 2 **MPLS L3 VPN Label**).
-Finally, the packet will goto an **L2 Interface** Group and being sent to the output port that goes to the spine router.
-Detail of how segment routing is implemented will be explained in the L3 unicast section below.
+We then point an MPLS Label Group points to an **MPLS Interface** Group in
+which the destination MAC is set to the next hop (spine router).
+Finally, the packet will goto an **L2 Interface** Group and being sent to the
+output port that goes to the spine router.
+
+Detail of how segment routing is implemented will be explained in the L3
+unicast section below.
L3 Unicast Group
^^^^^^^^^^^^^^^^
.. note::
OF-DPA L3 Unicast group entries are of OpenFlow **INDIRECT** type.
- L3 Unicast group entries are used to supply the routing next hop and output interface for packet forwarding.
- To properly route a packet from either the Routing Flow Table or the Policy ACL Flow Table, the forwarding flow entry must reference an L3 Unicast Group entry.
+
+ L3 Unicast group entries are used to supply the routing next hop and output
+ interface for packet forwarding.
+
+ To properly route a packet from either the Routing Flow Table or the Policy
+ ACL Flow Table, the forwarding flow entry must reference an L3 Unicast
+ Group entry.
All packets must have a VLAN tag.
- **A chained L2 Interface group entry must be in the same VLAN as assigned by the L3 Unicast Group** entry.
-We use L3 Unicast Group to rewrite the **source MAC**, **destination MAC** and **VLAN ID** when routing is needed.
+ **A chained L2 Interface group entry must be in the same VLAN as assigned
+ by the L3 Unicast Group** entry.
+We use L3 Unicast Group to rewrite the **source MAC**, **destination MAC** and
+**VLAN ID** when routing is needed.
L3 Multicast Group
^^^^^^^^^^^^^^^^^^
.. note::
OF-DPA L3 Multicast group entries are of OpenFlow **ALL** type.
- The action buckets describe the interfaces to which multicast packet replicas are forwarded.
+
+ The action buckets describe the interfaces to which multicast packet
+ replicas are forwarded.
+
Note that:
- - Chained OF-DPA **L2 Interface** Group entries must be in the **same VLAN** as the OF-DPA **L3 Multicast** group entry. However,
+ - Chained OF-DPA **L2 Interface** Group entries must be in the **same
+ VLAN** as the OF-DPA **L3 Multicast** group entry. However,
- - Chained OF-DPA **L3 Interface** Group entries must be in **different VLANs** from the OF-DPA **L3 Multicast** Group entry, **and from each other**.
+ - Chained OF-DPA **L3 Interface** Group entries must be in **different
+ VLANs** from the OF-DPA **L3 Multicast** Group entry, **and from each
+ other**.
We use L3 multicast group to replicate multicast packets when necessary.
-It is also possible that L3 multicast group consists of only one bucket when replication is not needed.
-Detail of how multicast is implemented will be explained in the L3 multicast section below.
+It is also possible that L3 multicast group consists of only one bucket when
+replication is not needed.
+
+Detail of how multicast is implemented will be explained in the L3 multicast
+section below.
L2 Interface Group
^^^^^^^^^^^^^^^^^^
.. note::
- L2 Interface Group entries are of OpenFlow **INDIRECT** type, with a single action bucket.
- OF-DPA L2 Interface group entries are used for egress VLAN filtering and tagging.
- If a specific set of VLANs is allowed on a port, appropriate group entries must be defined for the VLAN and port combinations.
+ L2 Interface Group entries are of OpenFlow **INDIRECT** type, with a single
+ action bucket.
- Note: OF-DPA uses the L2 Interface group declaration to configure the port VLAN filtering behavior.
- This approach was taken since OpenFlow does not support configuring VLANs on physical ports.
+ OF-DPA L2 Interface group entries are used for egress VLAN filtering and
+ tagging.
+ If a specific set of VLANs is allowed on a port, appropriate group entries
+ must be defined for the VLAN and port combinations.
+
+ Note: OF-DPA uses the L2 Interface group declaration to configure the port
+ VLAN filtering behavior.
+
+ This approach was taken since OpenFlow does not support configuring VLANs
+ on physical ports.
L2 Flood Group
^^^^^^^^^^^^^^
.. note::
- L2 Flood Group entries are used by VLAN Flow Table wildcard (destination location forwarding, or DLF) rules.
- Like OF-DPA L2 Multicast group entry types they are of OpenFlow **ALL** type.
+ L2 Flood Group entries are used by VLAN Flow Table wildcard (destination
+ location forwarding, or DLF) rules.
+
+ Like OF-DPA L2 Multicast group entry types they are of OpenFlow **ALL**
+ type.
+
The action buckets each encode an output port.
- Each OF-DPA L2 Flood Group entry bucket forwards a replica to an output port, except for packet IN_PORT.
- All of the OF-DPA L2 Interface Group entries referenced by the OF-DPA Flood Group entry,
- and the OF- DPA Flood Group entry itself, must be in the **same VLAN**.
+ Each OF-DPA L2 Flood Group entry bucket forwards a replica to an output
+ port, except for packet IN_PORT.
- Note: There can only be **one OF-DPA L2 Flood Group** entry defined **per VLAN**.
+ All of the OF-DPA L2 Interface Group entries referenced by the OF-DPA Flood
+ Group entry, and the OF- DPA Flood Group entry itself, must be in the
+ **same VLAN**.
+ Note: There can only be **one OF-DPA L2 Flood Group** entry defined **per
+ VLAN**.
L2 Unicast
----------
@@ -255,16 +358,28 @@
Fig. 3: Simplified L2 unicast pipeline
-The L2 unicast mechanism is designed to support **intra-rack (intra-subnet)** communication when the destination host is **known**.
+The L2 unicast mechanism is designed to support **intra-rack (intra-subnet)**
+communication when the destination host is **known**.
Pipeline Walkthrough - L2 Unicast
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-- **VLAN Table**: An untagged packet will be assigned an internal VLAN ID according to the input port and the subnet configured on the input port. Packets of the same subnet will have the same internal VLAN ID.
-- **TMAC Table**: Since the destination MAC of a L2 unicast packet is not the MAC of leaf router, the packet will miss the TMAC table and goes to the bridging table.
-- **Bridging Table**: If the destination MAC is learnt, there will be a flow entry matching that destination MAC and pointing to an L2 interface group.
-- **ACL Table**: IP packets will miss the ACL table and the L2 interface group will be executed.
- L2 Interface Group: The internal assigned VLAN will be popped before the packet is sent to the output port.
+- **VLAN Table**: An untagged packet will be assigned an internal VLAN ID
+ according to the input port and the subnet configured on the input port.
+ Packets of the same subnet will have the same internal VLAN ID.
+
+- **TMAC Table**: Since the destination MAC of a L2 unicast packet is not the
+ MAC of leaf router, the packet will miss the TMAC table and goes to the
+ bridging table.
+
+- **Bridging Table**: If the destination MAC is learnt, there will be a flow
+ entry matching that destination MAC and pointing to an L2 interface group.
+
+- **ACL Table**: IP packets will miss the ACL table and the L2 interface group
+ will be executed.
+
+ L2 Interface Group: The internal assigned VLAN will be popped before the
+ packet is sent to the output port.
L2 Broadcast
------------
@@ -281,15 +396,25 @@
Pipeline Walkthrough - L2 Broadcast
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-- **VLAN Table**: (same as L2 unicast)
-- **TMAC Table**: (same as L2 unicast)
-- **Bridging Table**: If the destination MAC is not learnt,
- there will NOT be a flow entry matching that destination MAC.
- It will then fallback to a lower priority entry that matches the VLAN (subnet) and point to an L2 flood group.
-- **ACL Table**: IP packets will miss the ACL table and the L2 flood group will be executed.
-- **L2 Flood Group**: Consists of all L2 interface groups related to this VLAN (subnet).
-- **L2 Interface Group**: The internal assigned VLAN will be popped before the packet is sent to the output port.
+- **VLAN Table**: (same as L2 unicast)
+
+- **TMAC Table**: (same as L2 unicast)
+
+- **Bridging Table**: If the destination MAC is not learnt, there will NOT be a
+ flow entry matching that destination MAC.
+
+ It will then fallback to a lower priority entry that matches the VLAN
+ (subnet) and point to an L2 flood group.
+
+- **ACL Table**: IP packets will miss the ACL table and the L2 flood group will
+ be executed.
+
+- **L2 Flood Group**: Consists of all L2 interface groups related to this VLAN
+ (subnet).
+
+- **L2 Interface Group**: The internal assigned VLAN will be popped before the
+ packet is sent to the output port.
ARP
---
@@ -300,15 +425,20 @@
Fig. 6: Simplified ARP pipeline
All ARP packets will be forwarded according to the bridging pipeline.
+
In addition, a **copy of the ARP packet will be sent to the controller**.
-- Controller will use the ARP packets for **learning purpose and update host store** accordingly.
-- Controller only **replies** an ARP request if the request is trying to **resolve an interface address configured on the switch edge port**.
+- Controller will use the ARP packets for **learning purpose and update host
+ store** accordingly.
+- Controller only **replies** an ARP request if the request is trying to
+ **resolve an interface address configured on the switch edge port**.
Pipeline Walkthrough - ARP
^^^^^^^^^^^^^^^^^^^^^^^^^^
-It is similar to L2 broadcast. Except ARP packets will be matched by a special ACL table entry and being copied to the controller.
+
+It is similar to L2 broadcast. Except ARP packets will be matched by a special
+ACL table entry and being copied to the controller.
L3 Unicast
@@ -317,7 +447,7 @@
.. image:: images/arch-l3u.png
:width: 800px
-Fig. 7: L3 unicxast
+Fig. 7: L3 unicast
.. image:: images/arch-l3u-src-pipeline.png
:width: 1000px
@@ -326,14 +456,32 @@
Pipeline Walkthrough - Source Leaf Switch
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-- **VLAN Table**: An untagged packet will be assigned an internal VLAN ID according to the input port and the subnet configured on the input port. Packets of the same subnet will have the same internal VLAN ID.
-- **TMAC Table**: Since the destination MAC of a L3 unicast packet is the MAC of leaf router and the ethernet type is IPv4, the packet will match the TMAC table and go to the unicast routing table.
-- **Unicast Routing Table**: In this table we will lookup the destination IP of the packet and point the packet to corresponding L3 ECMP group
-- **ACL Table**: IP packets will miss the ACL table and the L3 ECMP group will be executed.
-- **L3 ECMP Group**: Hashes on 5 tuple to pick a spine switch and goto the MPLS Label Group.
-- **MPLS Label Group**: Push the MPLS label corresponding to the destination leaf switch and goto the MPLS Interface Group.
-- **MPLS Interface Group**: Set source MAC address, destination MAC address, VLAN ID and goto the L2 Interface Group.
-- **L2 Interface Group**: The internal assigned VLAN will be popped before the packet is sent to the output port that goes to the spine.
+
+- **VLAN Table**: An untagged packet will be assigned an internal VLAN ID
+ according to the input port and the subnet configured on the input port.
+ Packets of the same subnet will have the same internal VLAN ID.
+
+- **TMAC Table**: Since the destination MAC of a L3 unicast packet is the MAC
+ of leaf router and the ethernet type is IPv4, the packet will match the TMAC
+ table and go to the unicast routing table.
+
+- **Unicast Routing Table**: In this table we will lookup the destination IP
+ of the packet and point the packet to corresponding L3 ECMP group
+
+- **ACL Table**: IP packets will miss the ACL table and the L3 ECMP group will
+ be executed.
+
+- **L3 ECMP Group**: Hashes on 5 tuple to pick a spine switch and goto the
+ MPLS Label Group.
+
+- **MPLS Label Group**: Push the MPLS label corresponding to the destination
+ leaf switch and goto the MPLS Interface Group.
+
+- **MPLS Interface Group**: Set source MAC address, destination MAC address,
+ VLAN ID and goto the L2 Interface Group.
+
+- **L2 Interface Group**: The internal assigned VLAN will be popped before the
+ packet is sent to the output port that goes to the spine.
.. image:: images/arch-l3u-transit-pipeline.png
:width: 1000px
@@ -342,13 +490,30 @@
Pipeline Walkthrough - Spine Switch
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-- **VLAN Table**: An untagged packet will be assigned an internal VLAN ID according to the input port and the subnet configured on the input port. Packets of the same subnet will have the same internal VLAN ID.
-- **TMAC Table**: Since the destination MAC of a L3 unicast packet is the MAC of spine router and the ethernet type is MPLS, the packet will match the TMAC table and go to the MPLS table.
-- **MPLS Table**: In this table we will lookup the MPLS label of the packet, figure out the destination leaf switch, pop the MPLS label and point to L3 ECMP Group.
-- **ACL Table**: IP packets will miss the ACL table and the MPLS interface group will be executed.
-- **L3 ECMP Group**: Hash to pick a link (if there are multiple links) to the destination leaf and goto the L3 Interface Group.
-- **MPLS Interface Group**: Set source MAC address, destination MAC address, VLAN ID and goto the L2 Interface Group.
-- **L2 Interface Group**: The internal assigned VLAN will be popped before the packet is sent to the output port that goes to the destination leaf switch.
+
+- **VLAN Table**: An untagged packet will be assigned an internal VLAN ID
+ according to the input port and the subnet configured on the input port.
+ Packets of the same subnet will have the same internal VLAN ID.
+
+- **TMAC Table**: Since the destination MAC of a L3 unicast packet is the MAC
+ of spine router and the ethernet type is MPLS, the packet will match the TMAC
+ table and go to the MPLS table.
+
+- **MPLS Table**: In this table we will lookup the MPLS label of the packet,
+ figure out the destination leaf switch, pop the MPLS label and point to L3
+ ECMP Group.
+
+- **ACL Table**: IP packets will miss the ACL table and the MPLS interface
+ group will be executed.
+
+- **L3 ECMP Group**: Hash to pick a link (if there are multiple links) to the
+ destination leaf and goto the L3 Interface Group.
+
+- **MPLS Interface Group**: Set source MAC address, destination MAC address,
+ VLAN ID and goto the L2 Interface Group.
+
+- **L2 Interface Group**: The internal assigned VLAN will be popped before the
+ packet is sent to the output port that goes to the destination leaf switch.
.. image:: images/arch-l3u-dst-pipeline.png
:width: 1000px
@@ -357,18 +522,34 @@
Pipeline Walkthrough - Destination Leaf Switch
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-- **VLAN Table**: An untagged packet will be assigned an internal VLAN ID according to the input port and the subnet configured on the input port. Packets of the same subnet will have the same internal VLAN ID.
-- **TMAC Table**: Since the destination MAC of a L3 unicast packet is the MAC of leaf router and the ethernet type is IPv4, the packet will match the TMAC table and go to the unicast routing table.
-- **Unicast Routing Table**: In this table we will lookup the destination IP of the packet and point the packet to corresponding L3 Unicast Group.
-- **ACL Table**: IP packets will miss the ACL table and the L3 Unicast Group will be executed.
-- **L3 Unicast Group**: Set source MAC address, destination MAC address, VLAN ID and goto the L2 Interface Group.
-- **L2 Interface Group**: The internal assigned VLAN will be popped before the packet is sent to the output port that goes to the destination leaf switch.
+
+- **VLAN Table**: An untagged packet will be assigned an internal VLAN ID
+ according to the input port and the subnet configured on the input port.
+ Packets of the same subnet will have the same internal VLAN ID.
+
+- **TMAC Table**: Since the destination MAC of a L3 unicast packet is the MAC
+ of leaf router and the ethernet type is IPv4, the packet will match the TMAC
+ table and go to the unicast routing table.
+
+- **Unicast Routing Table**: In this table we will lookup the destination IP
+ of the packet and point the packet to corresponding L3 Unicast Group.
+
+- **ACL Table**: IP packets will miss the ACL table and the L3 Unicast Group
+ will be executed.
+
+- **L3 Unicast Group**: Set source MAC address, destination MAC address, VLAN
+ ID and goto the L2 Interface Group.
+
+- **L2 Interface Group**: The internal assigned VLAN will be popped before the
+ packet is sent to the output port that goes to the destination leaf switch.
-The L3 unicast mechanism is designed to support inter-rack(inter-subnet) untagged communication when the destination host is known.
+The L3 unicast mechanism is designed to support inter-rack(inter-subnet)
+untagged communication when the destination host is known.
Path Calculation and Failover - Unicast
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
Coming soon...
@@ -385,10 +566,14 @@
Fig.12 Simplified L3 multicast pipeline
-The L3 multicast mechanism is designed to support use cases such as IPTV. The multicast traffic comes in from the upstream router, replicated by the leaf-spine switches, send to multiple OLTs and eventually get to the subscribers.
+The L3 multicast mechanism is designed to support use cases such as IPTV. The
+multicast traffic comes in from the upstream router, replicated by the
+leaf-spine switches, send to multiple OLTs and eventually get to the
+subscribers.
.. note::
- We would like to support different combinations of ingress/egress VLAN, including
+ We would like to support different combinations of ingress/egress VLAN,
+ including
- untagged in -> untagged out
- untagged in -> tagged out
@@ -398,14 +583,24 @@
However, due to the above-mentioned OFDPA restrictions,
- - It is NOT possible to chain L3 multicast group to L2 interface group directly if we want to change the VLAN ID
- - It is NOT possible to change VLAN ID by chaining L3 multicast group to L3 interface group since all output ports should have the same VLAN
- but the spec requires chained L3 interface group to have different VLAN ID from each other.
+ - It is NOT possible to chain L3 multicast group to L2 interface group
+ directly if we want to change the VLAN ID
- That means, if we need to change VLAN ID, we need to change it before the packets get into the multicast routing table.
+ - It is NOT possible to change VLAN ID by chaining L3 multicast group to L3
+ interface group since all output ports should have the same VLAN but the
+ spec requires chained L3 interface group to have different VLAN ID from
+ each other.
+
+ That means, if we need to change VLAN ID, we need to change it before the
+ packets get into the multicast routing table.
+
The only viable solution is changing the VLAN ID in the VLAN table.
- We change the VLAN tag on the ingress switch (i.e. the switch that connects to the upstream router) when necessary.
- On transit (spine) and egress (destination leaf) switches, output VLAN tag will remain the same as input VLAN tag.
+
+ We change the VLAN tag on the ingress switch (i.e. the switch that connects
+ to the upstream router) when necessary.
+
+ On transit (spine) and egress (destination leaf) switches, output VLAN tag
+ will remain the same as input VLAN tag.
Pipeline Walkthrough - Ingress Leaf Switch
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -416,19 +611,36 @@
:header-rows: 1
.. note::
- In the presence of ``vlan-untagged`` configuration on the ingress port of the ingress switch, the ``vlan-untagged`` will be used instead of 4094.
- The reason is that we cannot distinguish unicast and multicast traffic in that case, and therefore must assign the same VLAN to the packet.
+ In the presence of ``vlan-untagged`` configuration on the ingress port of
+ the ingress switch, the ``vlan-untagged`` will be used instead of 4094.
+
+ The reason is that we cannot distinguish unicast and multicast traffic in
+ that case, and therefore must assign the same VLAN to the packet.
+
The VLAN will anyway get popped in L2IG in this case.
-Table 1 shows all possible VLAN combinations on the ingress switches and how the packet is processed through the pipeline.
-We take the second case **untagged -> tagged 200** as an example to explain more details.
+Table 1 shows all possible VLAN combinations on the ingress switches and how
+the packet is processed through the pipeline. We take the second case
+**untagged -> tagged 200** as an example to explain more details.
- **VLAN Table**: An untagged packet will be assigned the **egress VLAN ID**.
-- **TMAC Table**: Since the destination MAC of a L2 unicast packet is a multicast MAC address, the packet will match the TMAC table and goes to the multicast routing table.
-- **Multicast Routing Table**: In this table we will lookup the multicast group (destination multicast IP) and point the packet to the corresponding L3 multicast group.
-- **ACL Table**: Multicast packets will miss the ACL table and the L3 multicast group will be executed.
-- **L3 Multicast Group**: The packet will be matched by **egress VLAN ID** and forwarded to multiple L2 interface groups that map to output ports.
-- **L2 Interface Group**: The egress VLAN will be kept in this case and the packet will be sent to the output port that goes to the transit spine switch.
+
+- **TMAC Table**: Since the destination MAC of a L2 unicast packet is a
+ multicast MAC address, the packet will match the TMAC table and goes to the
+ multicast routing table.
+
+- **Multicast Routing Table**: In this table we will lookup the multicast group
+ (destination multicast IP) and point the packet to the corresponding L3
+ multicast group.
+
+- **ACL Table**: Multicast packets will miss the ACL table and the L3 multicast
+ group will be executed.
+
+- **L3 Multicast Group**: The packet will be matched by **egress VLAN ID** and
+ forwarded to multiple L2 interface groups that map to output ports.
+
+- **L2 Interface Group**: The egress VLAN will be kept in this case and the
+ packet will be sent to the output port that goes to the transit spine switch.
Pipeline Walkthrough - Transit Spine Switch and Egress Leaf Switch
@@ -439,22 +651,37 @@
:widths: 2, 5, 5, 10, 10, 5
:header-rows: 1
-Table 2 shows all possible VLAN combinations on the transit/egress switches and how the packet is processed through the pipeline.
-Note that we have already changed the VLAN tag to the desired egress VLAN on the ingress switch.
-Therefore, there are only two cases on the transit/egress switches - either keep it untagged or keep it tagged. We take the first case **untagged -> untagged** as an example to explain more details.
+Table 2 shows all possible VLAN combinations on the transit/egress switches and
+how the packet is processed through the pipeline.
-- **VLAN Table**: An untagged packet will be assigned an **internal VLAN ID** according to the input port and the subnet configured on the input port. Packets of the same subnet will have the same internal VLAN ID.
+Note that we have already changed the VLAN tag to the desired egress VLAN on
+the ingress switch.
+
+Therefore, there are only two cases on the transit/egress switches - either
+keep it untagged or keep it tagged. We take the first case **untagged ->
+untagged** as an example to explain more details.
+
+
+- **VLAN Table**: An untagged packet will be assigned an **internal VLAN ID**
+ according to the input port and the subnet configured on the input port.
+ Packets of the same subnet will have the same internal VLAN ID.
+
- **TMAC Table**: (same as ingress switch)
+
- **Multicast Routing Table**: (same as ingress switch)
+
- **ACL Table**: (same as ingress switch)
-- **L3 Multicast Group**: The packet will be matched by **internal VLAN ID** and forwarded to multiple L2 interface groups that map to output ports.
-- **L2 Interface Group**: The egress VLAN will be popped in this case and the packet will be sent to the output port that goes to the egress leaf switch.
+
+- **L3 Multicast Group**: The packet will be matched by **internal VLAN ID**
+ and forwarded to multiple L2 interface groups that map to output ports.
+
+- **L2 Interface Group**: The egress VLAN will be popped in this case and the
+ packet will be sent to the output port that goes to the egress leaf switch.
Path Calculation and Failover - Multicast
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Coming soon...
-
VLAN Cross Connect
------------------
@@ -468,30 +695,46 @@
Fig. 14 Simplified VLAN cross connect pipeline
-VLAN Cross Connect is originally designed to support Q-in-Q packets between OLTs and BNGs.
+VLAN Cross Connect is originally designed to support Q-in-Q packets between
+OLTs and BNGs.
+
The cross connect pair consists of two output ports.
-Whatever packet comes in on one port with specific VLAN tag will be sent to the other port.
+
+Whatever packet comes in on one port with specific VLAN tag will be sent to the
+other port.
.. note::
It can only cross connects **two ports on the same switch**.
- :doc:`Pseudowire <configuration/pseudowire>` is required to connect ports across different switches.
+ :doc:`Pseudowire <configuration/pseudowire>` is required to connect ports
+ across different switches.
We use L2 Flood Group to implement VLAN Cross Connect.
+
The L2 Flood Group for cross connect only consists of two ports.
-The input port will be removed before flooding according to the spec and thus create exactly the desire behavior of cross connect.
+
+The input port will be removed before flooding according to the spec and thus
+create exactly the desire behavior of cross connect.
Pipeline Walkthrough - Cross Connect
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-- **VLAN Table**: When a tagged packet comes in, we no longer need to assign the internal VLAN.
- The original VLAN will be carried through the entire pipeline.
-- **TMAC Table**: Since the VLAN will not match any internal VLAN assigned to untagged packets,
- the packet will miss the TMAC table and goes to the bridging table.
-- **Bridging Table**: The packet will hit the flow rule that match the cross connect VLAN ID and
- being sent to corresponding L2 Flood Group.
-- **ACL Table**: IP packets will miss the ACL table and the L2 flood group will be executed.
-- **L2 Flood Group**: Consists of two L2 interface groups related to this cross connect VLAN.
- L2 Interface Group: The original VLAN will NOT be popped before the packet is sent to the output port.
+- **VLAN Table**: When a tagged packet comes in, we no longer need to assign
+ the internal VLAN. The original VLAN will be carried through the entire
+ pipeline.
+
+- **TMAC Table**: Since the VLAN will not match any internal VLAN assigned to
+ untagged packets, the packet will miss the TMAC table and goes to the
+ bridging table.
+
+- **Bridging Table**: The packet will hit the flow rule that match the cross
+ connect VLAN ID and being sent to corresponding L2 Flood Group.
+
+- **ACL Table**: IP packets will miss the ACL table and the L2 flood group will
+ be executed.
+
+- **L2 Flood Group**: Consists of two L2 interface groups related to this cross
+ connect VLAN. L2 Interface Group: The original VLAN will NOT be popped
+ before the packet is sent to the output port.
vRouter
-------
@@ -501,14 +744,16 @@
Fig. 15 vRouter
-The Trellis fabric needs to be connected to the external world via the vRouter functionality.
-**In the networking industry, the term vRouter implies a "router in a VM". This is not the case in Trellis**.
-Trellis vRouter is NOT a software router.
+The Trellis fabric needs to be connected to the external world via the vRouter
+functionality. **In the networking industry, the term vRouter implies a
+"router in a VM". This is not the case in Trellis**. Trellis vRouter is NOT a
+software router.
+
**Only the control plane of the router, i.e routing protocols, runs in a VM**.
We use the Quagga routing protocol suite as the control plane for vRouter.
-The **vRouter data plane is entirely in hardware**.
-Essentially the entire hardware fabric serves as the (distributed) data plane for vRouter.
+The **vRouter data plane is entirely in hardware**. Essentially the entire
+hardware fabric serves as the (distributed) data plane for vRouter.
The **external router views the entire Trellis fabric as a single router**.
@@ -517,15 +762,20 @@
.. image:: images/arch-vr-logical.png
.. note::
- Dual external routers is also supported for redundancy. Visit :doc:`External Connectivity <configuration/dual-homing>` for details.
+ Dual external routers is also supported for redundancy. Visit
+ :doc:`External Connectivity <configuration/dual-homing>` for details.
Pipeline Walkthrough - vRouter
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-The pipeline is exactly as same as L3 unicast. We just install additional flow rules in the unicast routing table on each leaf routers.
+
+The pipeline is exactly as same as L3 unicast. We just install additional flow
+rules in the unicast routing table on each leaf routers.
Learn More
----------
.. tip::
- Most of our design discussion and meeting notes are kept in `Google Drive <https://drive.google.com/drive/folders/0Bz9dNKPVvtgsR0M5R0hWSHlfZ0U>`_.
- If you are wondering why features are designed and implemented in a certain way, you may find the answers there.
+ Most of our design discussion and meeting notes are kept in `Google Drive
+ <https://drive.google.com/drive/folders/0Bz9dNKPVvtgsR0M5R0hWSHlfZ0U>`_.
+ If you are wondering why features are designed and implemented in a certain
+ way, you may find the answers there.
diff --git a/conf.py b/conf.py
index 5437ef2..8692aca 100644
--- a/conf.py
+++ b/conf.py
@@ -72,11 +72,22 @@
'sphinxcontrib.packetdiag',
'sphinxcontrib.rackdiag',
'sphinxcontrib.seqdiag',
+ 'sphinxcontrib.spelling',
+ 'sphinx_multiversion',
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
+# Text files with lists of words that shouldn't fail the spellchecker:
+spelling_word_list_filename=['dict.txt', ]
+
+# inlcude only the branches matching master and 1.x
+smv_branch_whitelist = r'^(master|1.*)$'
+
+# include all remote branches
+smv_remote_whitelist = r'^.*$'
+
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
diff --git a/configuration.rst b/configuration.rst
index c4f0b1a..e88f4bf 100644
--- a/configuration.rst
+++ b/configuration.rst
@@ -2,9 +2,12 @@
*******************
.. tip::
- Trellis configurations are done through ONOS **network configuration** service (sometime referred as **netcfg**), **CLI** or **REST API**.
- We encourage you to `learn more about ONOS network configuration service <https://wiki.onosproject.org/display/ONOS/The+Network+Configuration+Service>`_ before continue reading the following contents.
+ Trellis configurations are done through ONOS **network configuration**
+ service (sometime referred as **netcfg**), **CLI** or **REST API**.
+ We encourage you to `learn more about ONOS network configuration service
+ <https://wiki.onosproject.org/display/ONOS/The+Network+Configuration+Service>`_
+ before continue reading the following contents.
.. toctree::
:maxdepth: 1
@@ -17,4 +20,4 @@
configuration/multicast
configuration/xconnect
configuration/pseudowire
- configuration/aaa
\ No newline at end of file
+ configuration/aaa
diff --git a/configuration/aaa.rst b/configuration/aaa.rst
index a2da53d..8d839f2 100644
--- a/configuration/aaa.rst
+++ b/configuration/aaa.rst
@@ -3,8 +3,9 @@
Introduction
------------
-In this section, we will explain how to use Trellis with AAA service, which can be used to authenticate a client host.
-We will explain how this works with a simple **single switch** topology.
+In this section, we will explain how to use Trellis with AAA service, which can
+be used to authenticate a client host. We will explain how this works with a
+simple **single switch** topology.
.. image:: ../images/config-aaa.png
@@ -13,8 +14,9 @@
Activate AAA app
^^^^^^^^^^^^^^^^
-We need to install and activate AAA app separately since it is located in a separate (CORD) repository.
-There are multiple methods to install and activate a pre-compiled app. Let's use CLI now.
+We need to install and activate AAA app separately since it is located in a
+separate (CORD) repository. There are multiple methods to install and activate
+a pre-compiled app. Let's use CLI now.
.. code-block::console
@@ -23,10 +25,10 @@
Provide network configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-We need to provide AAA configuration in the apps section of network configuration.
+We need to provide AAA configuration in the apps section of network
+configuration.
.. code-block:: json
- :linenos:
{
"apps": {
@@ -42,8 +44,12 @@
- ``radiusIp``: The IP address of the Radius server
-- ``radiusServerPort``: The UDP port of the Radius server. (Optional -- ONOS will use port 1812 by default).
-- ``radiusSecret``: The Radius secret. This needs to be consistent with the Radius server configuration
+
+- ``radiusServerPort``: The UDP port of the Radius server. (Optional -- ONOS
+ will use port 1812 by default).
+
+- ``radiusSecret``: The Radius secret. This needs to be consistent with the
+ Radius server configuration
Then push the JSON to ONOS:
@@ -57,10 +63,10 @@
Install FreeRadius
^^^^^^^^^^^^^^^^^^
-Technically all Radius server should work.
-However, the way to configure them are probably different case to case.
-Here we use FreeRadius on Ubuntu as an example.
-To install the Radius server, simply run:
+
+Technically, any Radius server should work. However, the way to configure them
+are probably different case to case. Here we use FreeRadius on Ubuntu as an
+example. To install the Radius server, simply run:
.. code-block:: console
@@ -71,9 +77,11 @@
Add a user
""""""""""
-We usually connect Radius server to a database where we store the user information.
-In this section, we statically configure a user to simplify the setup.
-To add a user ``admin`` with password ``cord_test``, edit ``/etc/freeradius/users`` and add following lines:
+
+We usually connect Radius server to a database where we store the user
+information. In this section, we statically configure a user to simplify the
+setup. To add a user ``admin`` with password ``cord_test``, edit
+``/etc/freeradius/users`` and add following lines:
.. code-block:: text
@@ -109,14 +117,16 @@
+ private_key_password = onos_test
.. note::
- The key and certificates required by TLS will locate under ``/etc/freeradius/certs`` by default.
- There will be three symbolic links link to ``ca.pem``, ``server.key``, ``server.pem``.
- We only need to change the symbolic links after we generates the keys and certificates.
+ The key and certificates required by TLS will locate under
+ ``/etc/freeradius/certs`` by default. There will be three symbolic links
+ link to ``ca.pem``, ``server.key``, ``server.pem``. We only need to change
+ the symbolic links after we generates the keys and certificates.
Therefore, we don't need to change the path in ``/etc/freeradius/eap.conf``
.. note::
- Both server certificate and client certificate need to be signed by the same CA certificate.
- Also note that each key we generate below needs a unique Common Name.
+ Both server certificate and client certificate need to be signed by the
+ same CA certificate. Also note that each key we generate below needs a
+ unique Common Name.
Generate CA certificate (ca.pem) and private key (privkey.pem)
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
@@ -146,14 +156,17 @@
Deploy keys and certificates
""""""""""""""""""""""""""""
-On the server side, please link **/etc/freeradius/{ca.pem, server.key, server.pem}** to the files we just generated.
-Also copy **ca.pem, client.key, client.pem** to the client side through a secured channel.
-They will later be used when testing the Radius authentication.
+On the server side, please link **/etc/freeradius/{ca.pem, server.key,
+server.pem}** to the files we just generated. Also copy **ca.pem, client.key,
+client.pem** to the client side through a secured channel. They will later be
+used when testing the Radius authentication.
Testing
-------
-We can use the ``wpa_supplicant`` as the test client. In case ``wpa_supplicant`` has not been installed, you can run ``sudo apt-get install wpasupplicant``
+We can use the ``wpa_supplicant`` as the test client. In case
+``wpa_supplicant`` has not been installed, you can run ``sudo apt-get install
+wpasupplicant``
Compose wpa_supplicant.conf
^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -178,8 +191,10 @@
Run the test client
^^^^^^^^^^^^^^^^^^^
+
.. tip::
- If you are using a Linux VM behind a bridge to send out this authentication message, make sure the Linux kernel of your host machine is 3.2 or above.
+ If you are using a Linux VM behind a bridge to send out this authentication
+ message, make sure the Linux kernel of your host machine is 3.2 or above.
Otherwise the EAPOL messages won't go through the bridge.
.. code-block:: console
diff --git a/configuration/bridging-unicast.rst b/configuration/bridging-unicast.rst
index 2dc8c92..7a590f9 100644
--- a/configuration/bridging-unicast.rst
+++ b/configuration/bridging-unicast.rst
@@ -7,10 +7,10 @@
Access Ports
------------
+
The necessary but minimum configuration for an access port is simply a VLAN.
.. code-block:: json
- :linenos:
{
"ports" : {
@@ -29,17 +29,23 @@
}
}
-The example above shows two ports (12 and 16) on switch of:204 that have been assigned to VLAN 10 using the ``vlan-untagged`` keyword.
+The example above shows two ports (12 and 16) on switch ``of:204`` that have
+been assigned to VLAN 10 using the ``vlan-untagged`` keyword.
+
It simply means that packets come in and leave out of these switches untagged,
-but internally they are assigned VLAN 10 and they belong to the bridging domain defined for VLAN 10.
+but internally they are assigned VLAN 10 and they belong to the bridging domain
+defined for VLAN 10.
-``name`` is used to associate the interface with a globally unique, user friendly name. It can be omitted.
+``name`` is used to associate the interface with a globally unique, user
+friendly name. It can be omitted.
-With the configuration shown above, the packets will always be bridged, but they cannot be routed out of the VLAN (e.g. to other subnets).
-To add the capability to route out of VLAN 10, we need to add a subnet/gateway IP (similar to `interface-vlans or SVIs in traditional networks <https://www.youtube.com/watch?v=bUXpmiJpGb0>`_).
+With the configuration shown above, the packets will always be bridged, but
+they cannot be routed out of the VLAN (e.g. to other subnets). To add the
+capability to route out of VLAN 10, we need to add a subnet/gateway IP (similar
+to `interface-vlans or SVIs in traditional networks
+<https://www.youtube.com/watch?v=bUXpmiJpGb0>`_).
.. code-block:: json
- :linenos:
{
"ports" : {
@@ -60,15 +66,24 @@
}
}
-In this example, VLAN 10 is associated with subnet ``10.0.1.0/24``, and the gateway IP for hosts in this subnet is ``10.0.1.254/32``.
-When the desire is to route out of a VLAN, this assignment is currently necessary on all ports configured in the same VLAN.
+In this example, VLAN 10 is associated with subnet ``10.0.1.0/24``, and the
+gateway IP for hosts in this subnet is ``10.0.1.254/32``.
+
+When the desire is to route out of a VLAN, this assignment is currently
+necessary on all ports configured in the same VLAN.
.. note::
- Typically we only expect a single subnet for a VLAN. Similar to traditional networks, for us, a subnet == VLAN. Different VLANs should be configured in different subnets.
- In certain use-cases, it may be necessary to configure multiple subnets in the same VLAN. This is possible by adding more subnet/gateway IPs in the ``ips`` array.
+ Typically we only expect a single subnet for a VLAN. Similar to traditional
+ networks, for us, a subnet == VLAN. Different VLANs should be configured in
+ different subnets.
+
+ In certain use-cases, it may be necessary to configure multiple subnets in
+ the same VLAN. This is possible by adding more subnet/gateway IPs in the
+ ``ips`` array.
.. tip::
One subnet cannot be configured on multiple leaf switches.
+
We usually configure one subnet for all the ports on the same leaf switch.
Tagged Ports
@@ -76,7 +91,6 @@
Tagged port configuration is similar.
.. code-block:: json
- :linenos:
{
"ports" : {
@@ -90,18 +104,26 @@
}
}
-The configuration above for port 24 on switch of:204 shows two VLANs 20 and 40 configured on that port, with corresponding subnets and gateway IPs.
-Note that there is no specific ordering required in the ``ips`` or ``vlan-tagged`` arrays to correlate the VLANs to their corresponding subnets.
-In a future release, we will correlate VLAN and subnets configuration in a more readable way.
+The configuration above for port 24 on switch of:204 shows two VLANs 20 and 40
+configured on that port, with corresponding subnets and gateway IPs.
+Note that there is no specific ordering required in the ``ips`` or
+``vlan-tagged`` arrays to correlate the VLANs to their corresponding subnets.
+
+In a future release, we will correlate VLAN and subnets configuration in a more
+readable way.
Native VLAN on Tagged Ports
---------------------------
-An additional configuration ``vlan-native`` possible on tagged ports includes the ability to specify a VLAN (and thus a bridging domain) for incoming untagged packets.
-Typically, such configuration in trunk ports in traditional networks is referred to a native VLAN.
+
+An additional configuration ``vlan-native`` possible on tagged ports includes
+the ability to specify a VLAN (and thus a bridging domain) for incoming
+untagged packets.
+
+Typically, such configuration in trunk ports in traditional networks is
+referred to a native VLAN.
.. code-block:: json
- :linenos:
{
"ports" : {
@@ -116,15 +138,16 @@
}
}
-Note that it is also necessary to configure the subnet/gateway IP corresponding to the native VLAN if you wish to route out of that VLAN.
-
+Note that it is also necessary to configure the subnet/gateway IP corresponding
+to the native VLAN if you wish to route out of that VLAN.
Configuring interface for IPv6
------------------------------
-It is similar to configure IPv6 routing. Simply replace the addresses in ``ips`` with IPv6 addresses. For example,
+
+It is similar to configure IPv6 routing. Simply replace the addresses in
+``ips`` with IPv6 addresses. For example:
.. code-block:: json
- :linenos:
{
"ports" : {
@@ -144,36 +167,63 @@
Router Advertisement overview
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Router advertisement application is for enabling **Router Advertisement** and **Router Solicitation** functionalities supported by IPv6 routers.
+
+Router advertisement application is for enabling **Router Advertisement** and
+**Router Solicitation** functionalities supported by IPv6 routers.
+
More details are available in `RFC 4861 <https://tools.ietf.org/html/rfc4861>`_.
-Application identifies which IPv6 interfaces are currently configured in the system and it will try to send out **unsolicited Router Advertisement** (RA) messages from these interfaces.
-Each such RA message will have two mandatory options named **Source link-layer address** and **MTU**.
-Additional RA option **prefix** can be enabled using component configuration **raGlobalPrefixConfStatus**.
+Application identifies which IPv6 interfaces are currently configured in the
+system and it will try to send out **unsolicited Router Advertisement** (RA)
+messages from these interfaces.
-Application also processes **Router Solicitations** (RS) sent from hosts. Upon receiving RS on a particular interface application stops RA transmission in that interface and immediately sends RA targeted to the solicited host. After that application continues unsolicited RA transmission on that interface.
+Each such RA message will have two mandatory options named **Source link-layer
+address** and **MTU**.
+
+Additional RA option **prefix** can be enabled using component configuration
+**raGlobalPrefixConfStatus**.
+
+Application also processes **Router Solicitations** (RS) sent from hosts. Upon
+receiving RS on a particular interface application stops RA transmission in
+that interface and immediately sends RA targeted to the solicited host. After
+that application continues unsolicited RA transmission on that interface.
Activate and configure RA
^^^^^^^^^^^^^^^^^^^^^^^^^
+
RA application can be activated from CLI by running
.. code-block:: console
onos> app activate routeradvertisement
-Behavior of RA application is controlled by ONOS component configuration subsystem and following are possible configuration options.
+Behavior of RA application is controlled by ONOS component configuration
+subsystem and following are possible configuration options.
- ``raThreadDelay``: Delay between consecutive RA transmissions
+
- ``raPoolSize``: Capacity of thread pool to be used for RA transmissions
-- ``raFlagMbitStatus``: RA flag “Managed address configuration” enabled/disabled
+
+- ``raFlagMbitStatus``: RA flag “Managed address configuration”
+ enabled/disabled
+
- ``raFlagObitStatus``: RA flag “Other configuration” enabled/disabled
-- ``raOptionPrefixStatus``: RA Option “prefix” is enabled/disabled. Router prefixes will be available in RA only if this flag is “true”
-- ``raGlobalPrefixConfStatus``: Enable switch level global prefix configuration.
- Once “raGlobalPrefixConfStatus” is enabled, RA prefix option is generated from port configuration of device, see for more details.
+
+- ``raOptionPrefixStatus``: RA Option “prefix” is enabled/disabled. Router
+ prefixes will be available in RA only if this flag is “true”
+
+- ``raGlobalPrefixConfStatus``: Enable switch level global prefix
+ configuration.
+
+ Once “raGlobalPrefixConfStatus” is enabled, RA prefix option is generated
+ from port configuration of device, see for more details.
Prefix details are picked up from network interface configuration.
+
RA app will filter out link-local IPs while preparing prefixes.
-For example, in following configuration, Prefix will include only **2001:0558:FF10:04C9::2:1ff/120**.
+
+For example, in following configuration, Prefix will include only
+**2001:0558:FF10:04C9::2:1ff/120**.
.. code-block:: json
@@ -192,8 +242,12 @@
Global prefix configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^
-In some cases, users may want to have a set of global prefix **advertised on all edge interfaces**.
-Such prefixes can be configured in **devices** section of network configuration in the following way.
+
+In some cases, users may want to have a set of global prefix **advertised on
+all edge interfaces**.
+
+Such prefixes can be configured in **devices** section of network configuration
+in the following way.
.. code-block:: json
@@ -208,23 +262,42 @@
}
.. note::
- When global prefix is configured, RA app will ignore any prefixes configured on switch interfaces.
+ When global prefix is configured, RA app will ignore any prefixes
+ configured on switch interfaces.
Notes about interface config
----------------------------
-There is no need to configure ports on switches that are meant to connect to other switches.
-The VLAN (untagged or tagged) configuration is only meant for ports that are connected to hosts (edge ports).
+
+There is no need to configure ports on switches that are meant to connect to
+other switches.
+
+The VLAN (untagged or tagged) configuration is only meant for ports that are
+connected to hosts (edge ports).
.. image:: ../images/config-vlan.png
-Furthermore, note that the same VLAN can be configured on multiple ToRs - e.g. vlan 20 in the figure above.
-However this does not mean that the ports are in the same bridging domain, because in the fabric, the communication between ToRs is through a routed network. '
-In other words, a host on VLAN 20 (untagged or tagged) connected to one ToR can communicate with another host on VLAN 20 (untagged or tagged) connected to a different ToR,
-but the MAC addresses will change as the traffic goes through a routed network.
+Furthermore, note that the same VLAN can be configured on multiple ToRs - e.g.
+vlan 20 in the figure above.
-Please do not use this feature to connect switches in unsupported topologies as shown in the example below.
-The fabric is not designed to be one big Ethernet fabric. The bridging domain is restricted to within one ToR.
-If the bridging domain is extended across two ToRs directly linked to each other, there is a chance of loops.
-In other words, the ToRs/Leafs are not standalone 802.1Q bridges, and should not be used as such.
+However this does not mean that the ports are in the same bridging domain,
+because in the fabric, the communication between ToRs is through a routed
+network.
+
+In other words, a host on VLAN 20 (untagged or tagged) connected to one ToR can
+communicate with another host on VLAN 20 (untagged or tagged) connected to a
+different ToR, but the MAC addresses will change as the traffic goes through a
+routed network.
+
+Please do not use this feature to connect switches in unsupported topologies as
+shown in the example below.
+
+The fabric is not designed to be one big Ethernet fabric. The bridging domain
+is restricted to within one ToR.
+
+If the bridging domain is extended across two ToRs directly linked to each
+other, there is a chance of loops.
+
+In other words, the ToRs/Leafs are not standalone 802.1Q bridges, and should
+not be used as such.
.. image:: ../images/config-vlan-invalid.png
diff --git a/configuration/device-config.rst b/configuration/device-config.rst
index 6da9acd..9a55895 100644
--- a/configuration/device-config.rst
+++ b/configuration/device-config.rst
@@ -3,7 +3,6 @@
Each switch in Trellis requires a device config.
.. code-block:: json
- :linenos:
{
"devices" : {
@@ -26,30 +25,55 @@
}
- ``of:0000000000000001``: DPID of the device.
-- ``ipv4NodeSid``: IPv4 node segment ID, which is used as an MPLS label in forwarding IPv4 traffic. Can be arbitrary and should be globally unique.
-- ``ipv4Loopback``: IPv4 loopback address. Can be arbitrary, should be globally unique and should not be part of the same subnet(s) defined on the data plane ports (see port config).
-- ``ipv6NodeSid``: IPv6 node segment ID, which is used as an MPLS label in forwarding IPv6 traffic. Can be arbitrary and should be globally unique. Only required when using IPv6.
-- ``ipv6Loopback``: IPv6 loopback address. Can be arbitrary, should be globally unique and should not be part of the same subnet(s) defined on the data plane ports (see port config).
- Only required when using IPv6.
-- ``routerMac``: Router MAC address. Can be arbitrary and should be globally unique.
- This MAC address will be used to reply the ARP request for the loopback IP or the Interface IP that will be introduced later.
- (We recommend using the MAC address of the device's management interface as the router MAC.)
+
+- ``ipv4NodeSid``: IPv4 node segment ID, which is used as an MPLS label in
+ forwarding IPv4 traffic. Can be arbitrary and should be globally unique.
+
+- ``ipv4Loopback``: IPv4 loopback address. Can be arbitrary, should be globally
+ unique and should not be part of the same subnet(s) defined on the data plane
+ ports (see port config).
+
+- ``ipv6NodeSid``: IPv6 node segment ID, which is used as an MPLS label in
+ forwarding IPv6 traffic. Can be arbitrary and should be globally unique. Only
+ required when using IPv6.
+
+- ``ipv6Loopback``: IPv6 loopback address. Can be arbitrary, should be globally
+ unique and should not be part of the same subnet(s) defined on the data plane
+ ports (see port config). Only required when using IPv6.
+
+- ``routerMac``: Router MAC address. Can be arbitrary and should be globally
+ unique. This MAC address will be used to reply the ARP request for the
+ loopback IP or the Interface IP that will be introduced later. (We recommend
+ using the MAC address of the device's management interface as the router
+ MAC.)
+
- ``isEdgeRouter``: True for leaf switches. False for spine switches.
+
- ``adjacencySids``: Deprecated. Just put an empty array for now.
+
- ``name``: Name of the device. It is an arbitrary name to identify the device easily.
-- ``driver``: This tells ONOS which OpenFlow driver should be loaded for this device. It overrides the driver selected by ONOS automatically when the device connects.
+
+- ``driver``: This tells ONOS which OpenFlow driver should be loaded for this
+ device. It overrides the driver selected by ONOS automatically when the
+ device connects.
- When using OpenvSwitch, set this to ``ovs-ofdpa``
- - When using hardware switches, set this to ``ofdpa3`` or eliminate ``driver`` entirely as the correct driver configuration will be set automatically by ONOS
+
+ - When using hardware switches, set this to ``ofdpa3`` or eliminate
+ ``driver`` entirely as the correct driver configuration will be set
+ automatically by ONOS
.. caution::
- We should avoid using reserved MPLS labels for ``ipv4NodeSid`` and ``ipv6NodeSid``.
- Please check here for the reserved values: http://www.iana.org/assignments/mpls-label-values/mpls-label-values.xhtml
+ We should avoid using reserved MPLS labels for ``ipv4NodeSid`` and
+ ``ipv6NodeSid``. Please check here for the reserved values:
+ http://www.iana.org/assignments/mpls-label-values/mpls-label-values.xhtml
.. note::
Most of the Trellis configurations support dynamic configuration updates.
- Unfortunately, Trellis currently **do not support dynamic device configuration updates**.
- You will have to restart the device when if corresponding device configuration changes.
+ Unfortunately, Trellis currently **do not support dynamic device
+ configuration updates**. You will have to restart the device when if
+ corresponding device configuration changes.
Having said that, when introducing a completely new device in the network,
- the device configurations pushed before the device's connection should apply correctly.
+ the device configurations pushed before the device's connection should
+ apply correctly.
diff --git a/configuration/dhcp-relay.rst b/configuration/dhcp-relay.rst
index efebaa9..c9e68ff 100644
--- a/configuration/dhcp-relay.rst
+++ b/configuration/dhcp-relay.rst
@@ -2,20 +2,29 @@
==========
.. tip::
- We strongly recommend you to setup DHCP relay and configure the hosts to **obtain address via DHCP**.
- See `Alternative: Configure static IP`_ if you want to statically configure IP address on each host.
+ We strongly recommend you to setup DHCP relay and configure the hosts to
+ **obtain address via DHCP**.
+ See `Alternative: Configure static IP`_ if you want to statically configure
+ IP address on each host.
Overview
--------
The DHCP relay app used in Trellis is an L3 relay.
-That is, it support relaying DHCP packets from/to a server that's not in the same subnet of the client.
+
+That is, it support relaying DHCP packets from/to a server that's not in the
+same subnet of the client.
Here's a list of features supported:
- DHCPv4 and DHCPv6
-- DHCP server directly attached to fabric leaves, or indirectly connected via upstream router
-- DHCP client directly attached to fabric leaves, or indirectly connected via `LDRA (Light-weight DHCP Relay Agent) <https://tools.ietf.org/html/rfc6221>`_
+
+- DHCP server directly attached to fabric leaves, or indirectly connected via
+ upstream router
+
+- DHCP client directly attached to fabric leaves, or indirectly connected via
+ `LDRA (Light-weight DHCP Relay Agent) <https://tools.ietf.org/html/rfc6221>`_
+
- Multiple DHCP servers for HA
.. note::
@@ -30,8 +39,11 @@
.. image:: ../images/config-dhcp.png
-In this case, the configuration involves first configuring the switch interface with the vlan/subnet the DHCP service is part of.
-For example, if I have a switch ``of:205`` with a DHCP server on port 24 on vlan 20, the port config looks like:
+In this case, the configuration involves first configuring the switch interface
+with the vlan/subnet the DHCP service is part of.
+
+For example, if I have a switch ``of:205`` with a DHCP server on port 24 on
+vlan 20, the port config looks like:
.. code-block:: json
@@ -47,7 +59,8 @@
}
}
-A second part of the configuration for the DHCP relay app requires a json configuration under the key apps:
+A second part of the configuration for the DHCP relay app requires a json
+configuration under the key apps:
.. code-block:: json
@@ -64,16 +77,24 @@
}
}
-Note that the dhcprelay app is configured with location of the DHCP server (the switch port to which it is connected to the fabric).
-It is also configured with the DHCP server IP, but it is no longer necessary to configure the MAC address of the server.
+Note that the dhcprelay app is configured with location of the DHCP server (the
+switch port to which it is connected to the fabric).
+
+It is also configured with the DHCP server IP, but it is no longer necessary to
+configure the MAC address of the server.
+
ONOS will automatically learn the MAC and VLAN corresponding to the serverIP.
Server reachable via external router
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-In this case, it is actually the external router that is directly connected to the fabric.
-This external router is already configured in the ports section of network-config (for vRouter functionality).
-For example, if the external router is connected to switch of:205 on port 1
+In this case, it is actually the external router that is directly connected to
+the fabric.
+
+This external router is already configured in the ports section of
+network-config (for vRouter functionality).
+
+For example, if the external router is connected to switch ``of:205`` on port 1
.. code-block:: json
@@ -89,8 +110,12 @@
}
}
-As before the ``ips`` and ``mac`` configured on port 1, actually correspond to the addresses configured in Quagga.
-The app config in this case, includes an additional field necessary to inform the dhcp-relay app of the gatewayIP through which the DHCP server can be reached.
+As before the ``ips`` and ``mac`` configured on port 1, actually correspond to
+the addresses configured in Quagga.
+
+The app config in this case, includes an additional field necessary to inform
+the dhcp-relay app of the gatewayIP through which the DHCP server can be
+reached.
.. code-block:: json
@@ -109,13 +134,15 @@
}
.. note::
- Note that the dhcpserverConnectPoint should now be the switchport to which the external router is connected to the fabric.
+ Note that the ``dhcpserverConnectPoint`` should now be the switchport to
+ which the external router is connected to the fabric.
Setup DHCP server
-----------------
Install DHCP server
^^^^^^^^^^^^^^^^^^^
+
Modern DHCP servers should support relayed DHCP request.
However, the way to configure them are probably different case to case.
Here we use **isc-dhcp-server** on Ubuntu as an example.
@@ -128,6 +155,7 @@
Configure DHCP Server
^^^^^^^^^^^^^^^^^^^^^
+
Two configuration files are required by DHCP server.
First, we need to specify which network interface the DHCP server should listen on.
@@ -141,7 +169,9 @@
To do that, we need to modify ``/etc/dhcp/dhcpd.conf`` and add the following lines.
Note that the subnet of ``eth1`` needs to be included.
-Otherwise, the DHCP server will not listen to the interface even though we have specified that in ``/etc/default/isc-dhcp-server``.
+
+Otherwise, the DHCP server will not listen to the interface even though we have
+specified that in ``/etc/default/isc-dhcp-server``.
.. code-block:: text
@@ -188,6 +218,7 @@
Testing
-------
+
The host should be able to obtain an IP address from the pool we specified.
Try to run ``dhclient`` and see if the host can get an IP address.
@@ -206,19 +237,28 @@
sudo dhclient -6 -P -r eth1 # for releasing prefix
-If something goes wrong, check ``/var/log/syslog`` for DHCP server log and run ``tcpdump`` on DHCP server to see if the DHCP packets from the host reach the server correctly.
-
+If something goes wrong, check ``/var/log/syslog`` for DHCP server log and run
+``tcpdump`` on DHCP server to see if the DHCP packets from the host reach the
+server correctly.
Additional Features
-------------------
DHCP Relay store
^^^^^^^^^^^^^^^^
-DHCP relay application stores information from DHCP packet which processed by the app, administrator can use CLI command ``dhcp-relay`` to query these information.
+
+DHCP relay application stores information from DHCP packet which processed by
+the app, administrator can use CLI command ``dhcp-relay`` to query these
+information.
+
The store provides these functionality:
-- Latest state of DHCP client (e.g. client location, last seen time, DHCP type...), for debugging purpose
-- For direct host, ONOS can find location and vlan from relay agent option, however, for indirect host, ONOS need to query last state from the store to find correct destination.
+- Latest state of DHCP client (e.g. client location, last seen time, DHCP
+ type...), for debugging purpose
+
+- For direct host, ONOS can find location and vlan from relay agent option,
+ however, for indirect host, ONOS need to query last state from the store to
+ find correct destination.
DHCPv6 Relay counter
@@ -226,7 +266,9 @@
There are two DHCPv6 packet counters which are Host basis counters and Global counters.
Host basis counters count and record DHCPv6 packets received on this host.
-It can be displayed by ``dhcp-relay counter``. These counters can be reset by typing ``dhcp-relay counter reset``.
+
+It can be displayed by ``dhcp-relay counter``. These counters can be reset by
+typing ``dhcp-relay counter reset``.
.. code-block:: console
@@ -249,7 +291,9 @@
onos> dhcp-relay counter reset
Global counters counts and records all DHCPv6 packets received in ONOS.
-It can be displayed by ``dhcp-relay-agg-counters``. These counters can be reset by typing ``dhcp-relay-agg-counters reset``.
+
+It can be displayed by ``dhcp-relay-agg-counters``. These counters can be reset
+by typing ``dhcp-relay-agg-counters reset``.
.. code-block:: console
@@ -269,26 +313,47 @@
Indirect client support
^^^^^^^^^^^^^^^^^^^^^^^
DHCP relay can support hosts which do not directly connect to Trellis fabric.
-These hosts usually connected to another LDRA, the LDRA will forward DHCP packet to/from Trellis network.
-For **DHCPv4**, packets from the LDRA includes a valid DHCP relay agent option (option 82).
-DHCP Relay application checks relay agent option and determine the DHCP packet comes from direct or indirect host.
+These hosts usually connected to another LDRA, the LDRA will forward DHCP
+packet to/from Trellis network.
+
+For **DHCPv4**, packets from the LDRA includes a valid DHCP relay agent option
+(option 82).
+
+DHCP Relay application checks relay agent option and determine the DHCP packet
+comes from direct or indirect host.
.. image:: ../images/config-dhcp-indirect.jpg
-ONOS uses circuit id option in relay agent option with specific format if DHCP packet comes without relay agent option, the format of circuit will be: ``ConnectPoint:VlanId``
-For example, the DHCP request/discover packet comes from ``of:000000000000001/1`` with ``VLAN 100``, the circuit ONOS put will be ``of:000000000000001/1:100`` and send DHCP packet to DHCP server.
-Indirect host won't put into host store. DHCP relay app will put IP address of indirect host to the route store, and use IP address of relay agent as next hop.
+ONOS uses circuit id option in relay agent option with specific format if DHCP
+packet comes without relay agent option, the format of circuit will be:
+``ConnectPoint:VlanId``
+
+For example, the DHCP request/discover packet comes from
+``of:000000000000001/1`` with ``VLAN 100``, the circuit ONOS put will be
+``of:000000000000001/1:100`` and send DHCP packet to DHCP server.
+
+Indirect host won't put into host store. DHCP relay app will put IP address of
+indirect host to the route store, and use IP address of relay agent as next
+hop.
**DHCPv6** clients will be handled similar to DHCPv4.
-One major difference is that DHCPv6 supports ``RELAY-FORWARD`` message type and ``InterfaceId`` option natively, so we utilize those fields to encode information.
+One major difference is that DHCPv6 supports ``RELAY-FORWARD`` message type and
+``InterfaceId`` option natively, so we utilize those fields to encode
+information.
Overwrite relay agent IP
^^^^^^^^^^^^^^^^^^^^^^^^
-The DHCP relay can overwrite the relay agent address (``giaddr`` in **DHCPv4**, ``link-addr`` in **DHCPv6**) in DHCP message for different device.
-If ``relayAgentIps`` is configured, the app will overwrite ``giaddr`` or ``link-addr`` before it forward the DHCP message to the server.
+
+The DHCP relay can overwrite the relay agent address (``giaddr`` in **DHCPv4**,
+``link-addr`` in **DHCPv6**) in DHCP message for different device.
+
+If ``relayAgentIps`` is configured, the app will overwrite ``giaddr`` or
+``link-addr`` before it forward the DHCP message to the server.
+
Otherwise, it will retain the original relay agent IP.
+
An example configuration is shown below:
.. code-block:: json
@@ -318,8 +383,13 @@
Configure multiple servers
^^^^^^^^^^^^^^^^^^^^^^^^^^
-DHCP server HA can be achieved by specifying additional server configuration objects.
-Client initiated packets like ``SOLICIT`` or ``REBIND`` shall be replicated and sent to all server objects.
+
+DHCP server HA can be achieved by specifying additional server configuration
+objects.
+
+Client initiated packets like ``SOLICIT`` or ``REBIND`` shall be replicated and
+sent to all server objects.
+
Below is an example of multiple server configuration:
.. code-block:: json
@@ -402,20 +472,28 @@
}
- ``dhcpServerConnectPoint``: represent the location of DHCP server
+
- ``serverIps``: IP address of the DHCP server, contains at least one IP address of DHCP server.
IP address can be IPv4 or IPv6 for different version of DHCP.
Will use first address if multiple IPv4 or IPv6 address configured.
-- ``gatewayIps``: Optional. Should be configured if the DHCP server is not directly connected to the Trellis network
- . It tells which gateway we need to send to reach the server.
+
+- ``gatewayIps``: Optional. Should be configured if the DHCP server is not
+ directly connected to the Trellis network. It tells which gateway we need to
+ send to reach the server.
.. note::
- - If ``indirect`` server configuration is not configured, the app will use ``default`` configuration for all cases.
+ - If ``indirect`` server configuration is not configured, the app will use
+ ``default`` configuration for all cases.
Ignoring DHCP relay on a particular VLAN
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-In some cases, it may be necessary to avoid punting DHCP packets to the controller, and letting them be forwarded normally through the data plane.
-In such cases, the DHCP relay application can be configured to avoid punting DHCP packets on a particular VLAN on a particular switch.
+
+In some cases, it may be necessary to avoid punting DHCP packets to the
+controller, and letting them be forwarded normally through the data plane.
+
+In such cases, the DHCP relay application can be configured to avoid punting
+DHCP packets on a particular VLAN on a particular switch.
.. code-block:: json
@@ -430,8 +508,8 @@
}
}
-In the example shown above, DHCP packets on vlan 24 are not punted to the controller from switches of:205 and of:206
-
+In the example shown above, DHCP packets on vlan 24 are not punted to the
+controller from switches of:205 and of:206
DHCPv6 Prefix Delegation (PD) Pushing
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -439,8 +517,12 @@
.. note::
This feature requires both ``dhcprelay`` and ``fpm`` apps to be activated
-PD pushing allows IPv6 prefixes from DhcpRelay to be sent over the FPM connection to Quagga where they will be configured as a static route.
-Prior to PD Pushing, the FPM connection was only used by Quagga in one direction to push routes to FPM. PD pushing is disabled by default in DHCP Relay and FPM.
+PD pushing allows IPv6 prefixes from DhcpRelay to be sent over the FPM
+connection to Quagga where they will be configured as a static route.
+
+Prior to PD Pushing, the FPM connection was only used by Quagga in one
+direction to push routes to FPM. PD pushing is disabled by default in DHCP
+Relay and FPM.
To enable in DHCP relay:
@@ -454,9 +536,16 @@
onos> dhcp-fpm-routes
-When PD pushing is enabled in FPM, by default the next-hop to be used for all prefixes pushed to Quagga will be retrieved from the first interface with ``RUR`` in the name in ONOS.
-Next-hop may also be configured using FPM component config. This will override a ``RUR`` interface if present.
-If there is no interface with ``RUR`` in the name and the next-hop is not configured, no prefixes can be pushed to Quagga even if PD pushing is enabled. For DhcpRelay, only the IPv6 next-hop is needed.
+When PD pushing is enabled in FPM, by default the next-hop to be used for all
+prefixes pushed to Quagga will be retrieved from the first interface with
+``RUR`` in the name in ONOS.
+
+Next-hop may also be configured using FPM component config. This will override
+a ``RUR`` interface if present.
+
+If there is no interface with ``RUR`` in the name and the next-hop is not
+configured, no prefixes can be pushed to Quagga even if PD pushing is enabled.
+For DhcpRelay, only the IPv6 next-hop is needed.
To enable in FPM:
@@ -482,12 +571,17 @@
.. note::
Quagga requires a patch to be able to receive Netlink Messages from FPM.
-
Clean up expired address and PD prefix
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-DHCPv6 relay cleans up stale IP address and pd prefix based on timer whose default interval is 24 hours (24 * 3600 secs = 86400 secs).
-If the preferred life time of ip address or pd prefix exceeds 1/2 of poll interval, they will be removed from ONOS.
-The poll interval can be modified by ``cfg set org.onosproject.dhcprelay.DhcpRelayManager dhcpPollInterval <newVal>``
+
+DHCPv6 relay cleans up stale IP address and pd prefix based on timer whose
+default interval is 24 hours (24 * 3600 secs = 86400 secs).
+
+If the preferred life time of ip address or pd prefix exceeds 1/2 of poll
+interval, they will be removed from ONOS.
+
+The poll interval can be modified by ``cfg set
+org.onosproject.dhcprelay.DhcpRelayManager dhcpPollInterval <newVal>``
.. code-block:: console
@@ -504,13 +598,15 @@
Alternative: Configure static IP
--------------------------------
-Although we strongly recommend to use `DHCP Relay`_ for IP assignment,
-it is also possible to statically configure the IP address and route on the host.
+
+Although we strongly recommend to use `DHCP Relay`_ for IP assignment, it is
+also possible to statically configure the IP address and route on the host.
1. **Configure the IP address and subnet mask**
- Make sure the IP address and the subnet mask on the fabric network interface of the host is consistent with
- the information in the Network Configuration section. For example, you can run
+ Make sure the IP address and the subnet mask on the fabric network interface
+ of the host is consistent with the information in the Network Configuration
+ section. For example, you can run
.. code-block:: console
@@ -518,8 +614,8 @@
2. **Configure the default route**
- Make sure you change the default route of the host to the interface IP of the leaf switch it connects to.
- For example, you can run
+ Make sure you change the default route of the host to the interface IP of
+ the leaf switch it connects to. For example, you can run
.. code-block:: console
@@ -531,9 +627,13 @@
3. **Trigger host learning**
- We need to let ONOS learn the host in order to program corresponding flows and groups.
+ We need to let ONOS learn the host in order to program corresponding flows
+ and groups.
+
This is automatically done as part of the DHCP process.
- However, we need to manually triggers it by sending an ARP or ND packet if the host is configured to use static IP.
+
+ However, we need to manually triggers it by sending an ARP or ND packet if
+ the host is configured to use static IP.
.. code-block:: console
@@ -543,7 +643,6 @@
# ndsend ${HOST_IP} ${INTF}
-
Reference
---------
- https://www.cisco.com/c/en/us/support/docs/security/adaptive-security-appliance-asa-software/116265-configure-product-00.html
diff --git a/configuration/dual-homing.rst b/configuration/dual-homing.rst
index 3e5f729..318dc02 100644
--- a/configuration/dual-homing.rst
+++ b/configuration/dual-homing.rst
@@ -8,31 +8,36 @@
The dual-homing feature includes several sub components
-- **Use of "paired" ToRs**: Each rack of compute nodes have exactly two Top-of-Rack switches (ToRs),
- that are linked to each other via a single link - such a link is referred to as a **pair link**.
- This pairing should NOT be omitted.
- Currently there is support for only a single link between paired ToRs.
- In future releases, we may include dual pair links.
- Note that the pair link is only used in failure scenarios, and not in normal operation.
+- **Use of "paired" ToRs**: Each rack of compute nodes have exactly two
+ Top-of-Rack switches (ToRs), that are linked to each other via a single link
+ - such a link is referred to as a **pair link**. This pairing should NOT be
+ omitted.
-- **Dual-homed servers (compute-nodes)**: Each server is connected to both ToRs.
- The links to the paired ToRs are (Linux) bonded
+ Currently there is support for only a single link between paired ToRs. In
+ future releases, we may include dual pair links. Note that the pair link is
+ only used in failure scenarios, and not in normal operation.
-- **Dual-homed upstream routers**: The upstream routers MUST be connected to the two ToRs that are part of a leaf-pair.
- You cannot connect them to leafs that are not paired. This feature also requires two Quagga instances.
+- **Dual-homed servers (compute-nodes)**: Each server is connected to both
+ ToRs. The links to the paired ToRs are (Linux) bonded
+
+- **Dual-homed upstream routers**: The upstream routers MUST be connected to
+ the two ToRs that are part of a leaf-pair. You cannot connect them to leafs
+ that are not paired. This feature also requires two Quagga instances.
- **Dual-homed access devices**. This component will be added in the future.
Paired ToRs
-----------
-The reasoning behind two ToR (leaf) switches is simple.
-If you only have a single ToR switch, and you lose it, the entire rack goes down.
-Using two ToR switches increases your odds for continued connectivity for dual homed servers.
-The reasoning behind pairing the two ToR switches is more involved, as is explained in the Usage section below.
+The reasoning behind two ToR (leaf) switches is simple. If you only have a
+single ToR switch, and you lose it, the entire rack goes down. Using two ToR
+switches increases your odds for continued connectivity for dual homed servers.
+The reasoning behind pairing the two ToR switches is more involved, as is
+explained in the Usage section below.
Configure pair ToRs
^^^^^^^^^^^^^^^^^^^
-Configuring paired-ToRs involves device configuration. Assume switches of:205 and of:206 are paired ToRs.
+Configuring paired-ToRs involves device configuration. Assume switches of:205
+and of:206 are paired ToRs.
.. code-block:: json
@@ -71,15 +76,21 @@
There are two new pieces of device configuration.
-Each device in the ToR pair needs to specify the **deviceId of the leaf it is paired to**, in the ``pairDeviceId`` field.
-For example, in of:205 configuration the pairDeviceid is specified as of:206, and similarly in of:206 configuration the pairDeviceId is of:205
-Each device in the ToR pair needs to specify the **port on the device used for the pair link** in the ``pairLocalPort`` field.
-For example, the pair link in the config above show that port 20 on of:205 is connected to port 30 on of:206.
+Each device in the ToR pair needs to specify the **deviceId of the leaf it is
+paired to**, in the ``pairDeviceId`` field. For example, in ``of:205``
+configuration the ``pairDeviceId`` is specified as ``of:206``, and similarly in ``of:206``
+configuration the ``pairDeviceId`` is ``of:205``. Each device in the ToR pair needs to
+specify the **port on the device used for the pair link** in the
+``pairLocalPort`` field. For example, the pair link in the config above show
+that port 20 on of:205 is connected to port 30 on of:206.
-In addition, there is one crucial piece of config that needs to **match for both ToRs** – the ``routerMac`` address.
-The paired-ToRs MUST have the same routerMac - in the example above, they both have identical 00:00:02:05:00:01 routerMacs.
+In addition, there is one crucial piece of config that needs to **match for
+both ToRs** – the ``routerMac`` address. The paired-ToRs MUST have the same
+routerMac - in the example above, they both have identical 00:00:02:05:00:01
+routerMacs.
-All other fields are the same as before, as explained in :doc:`Device Configuration <device-config>` section.
+All other fields are the same as before, as explained in :doc:`Device
+Configuration <device-config>` section.
Usage of pair link
@@ -90,19 +101,26 @@
Dual-Homed Servers
------------------
+
There are a number of things to note when connecting dual-homed servers to paired-ToRs.
-- The switch ports on the two ToRs have to be configured the same way, when connecting a dual-homed server to the two ToRs.
+- The switch ports on the two ToRs have to be configured the same way, when
+ connecting a dual-homed server to the two ToRs.
+
- The server ports have to be Linux-bonded in a particular mode.
Configure Switch Ports
^^^^^^^^^^^^^^^^^^^^^^
-The way to configure ports are similar as described in :doc:`Bridging and Unicast <bridging-unicast>`.
-However, there are a couple of things to note.
-**First**, dual-homed servers should have the **identical configuration on each switch port they connect to on the ToR pairs**.
-The example below shows that the ``vlans`` and ``ips`` configured are the same on both switch ports ``of:205/12`` and ``of:206/29``.
-They are both configured to be access ports in ``VLAN 20``, the subnet ``10.0.2.0/24`` is assigned to these ports, and the gateway-IP is ``10.0.2.254/32``.
+The way to configure ports are similar as described in :doc:`Bridging and
+Unicast <bridging-unicast>`. However, there are a couple of things to note.
+
+**First**, dual-homed servers should have the **identical configuration on each
+switch port they connect to on the ToR pairs**. The example below shows that
+the ``vlans`` and ``ips`` configured are the same on both switch ports
+``of:205/12`` and ``of:206/29``. They are both configured to be access ports
+in ``VLAN 20``, the subnet ``10.0.2.0/24`` is assigned to these ports, and the
+gateway-IP is ``10.0.2.254/32``.
.. code-block:: json
@@ -125,17 +143,24 @@
}
}
-It is worth noting the meaning behind the configuration above from a routing perspective.
-Simply put, by configuring the same subnets on these switch ports, the fabric now believes that the entire subnet ``10.0.2.0/24`` is reachable by BOTH ToR switches ``of:205`` and ``of:206``.
+It is worth noting the meaning behind the configuration above from a routing
+perspective. Simply put, by configuring the same subnets on these switch
+ports, the fabric now believes that the entire subnet ``10.0.2.0/24`` is
+reachable by BOTH ToR switches ``of:205`` and ``of:206``.
.. caution::
- Configuring different VLANs, or different subnets, or mismatches like "vlan-untagged" in one switch port and "vlan-tagged" in the corresponding switch port facing the dual-homed server, will result in incorrect behavior.
+ Configuring different VLANs, or different subnets, or mismatches like
+ "vlan-untagged" in one switch port and "vlan-tagged" in the corresponding
+ switch port facing the dual-homed server, will result in incorrect
+ behavior.
-**Second**, we need to configure the **pair link ports on both ToR switches to be trunk (vlan-tagged) ports that contains all dual-homed VLANs and subnets**.
-This is an extra piece of configuration, the need for which will be removed in future releases.
-In the example above, a dual-homed server connects to the ToR pair on port 12 on of:205 and port 29 on of:206.
-Assume that the pair link between the two ToRs is connected to port 5 of both of:205 and of:206.
-The config for these switch ports is shown below:
+**Second**, we need to configure the **pair link ports on both ToR switches to
+be trunk (vlan-tagged) ports that contains all dual-homed VLANs and subnets**.
+This is an extra piece of configuration, the need for which will be removed in
+future releases. In the example above, a dual-homed server connects to the ToR
+pair on port 12 on of:205 and port 29 on of:206. Assume that the pair link
+between the two ToRs is connected to port 5 of both of:205 and of:206. The
+config for these switch ports is shown below:
.. code-block:: json
@@ -159,13 +184,18 @@
}
.. note::
- Even though the ports ``of:205/12`` and ``of:206/`` facing the dual-homed server are configured as ``vlan-untagged``,
- the same vlan MUST be configured as ``vlan-tagged`` on the pair-ports.
- If additional subnets and VLANs are configured facing other dual-homed servers, they need to be similarly added to the ``ips`` and ``vlan-tagged`` arrays in the pair port config.
+ Even though the ports ``of:205/12`` and ``of:206/`` facing the dual-homed
+ server are configured as ``vlan-untagged``, the same vlan MUST be
+ configured as ``vlan-tagged`` on the pair-ports.
+
+ If additional subnets and VLANs are configured facing other dual-homed
+ servers, they need to be similarly added to the ``ips`` and ``vlan-tagged``
+ arrays in the pair port config.
Configure Servers
^^^^^^^^^^^^^^^^^
+
Assuming the interfaces we are going to use for bonding are ``eth1`` and ``eth2``.
- Bring down interfaces
@@ -235,19 +265,24 @@
.. caution::
**Dual-homed host should not be statically configured.**
- Currently in ONOS, configured hosts are not updated when the connectPoint is lost.
- This is not a problem with single-homed hosts because there is no other way to reach them anyway if their connectPoint goes down.
- But in dual-homed scenarios, the controller should take corrective action if one of the connectPoints go down –
- the trigger for this event does not happen when the dual-homed host's connect points are configured (not discovered).
+ Currently in ONOS, configured hosts are not updated when the connectPoint
+ is lost. This is not a problem with single-homed hosts because there is no
+ other way to reach them anyway if their connectPoint goes down. But in
+ dual-homed scenarios, the controller should take corrective action if one
+ of the connectPoints go down – the trigger for this event does not happen
+ when the dual-homed host's connect points are configured (not discovered).
.. note::
- We also support static routes with dual-homed next hop.
- The way to configure it is exactly the same as regular single-homed next hop, as described in :doc:`External Connectivity <external-connectivity>`.
+ We also support static routes with dual-homed next hop. The way to
+ configure it is exactly the same as regular single-homed next hop, as
+ described in :doc:`External Connectivity <external-connectivity>`.
- ONOS will automatically recognize when the next-hop IP resolves to a dual-homed host and program both switches (the host connects to) accordingly.
+ ONOS will automatically recognize when the next-hop IP resolves to a
+ dual-homed host and program both switches (the host connects to)
+ accordingly.
- The failure recovery mechanism for dual-homed hosts also applies to static routes that point to the host as their next hop.
-
+ The failure recovery mechanism for dual-homed hosts also applies to static
+ routes that point to the host as their next hop.
Dual External Routers
---------------------
@@ -257,22 +292,25 @@
.. image:: ../images/config-dh-vr-logical.png
:width: 200px
-In addition to what we describe in :doc:`External Connectivity <external-connectivity>`,
-Trellis also supports dual external routers, which view the Trellis fabric as 2 individual routers, as shown above.
+In addition to what we describe in :doc:`External Connectivity
+<external-connectivity>`, Trellis also supports dual external routers, which
+view the Trellis fabric as 2 individual routers, as shown above.
As before the vRouter control plane is implemented as a combination of Quagga,
-which peers with the upstream routers, and ONOS which listens to Quagga (via FPM) and programs the underlying fabric.
-**In dual-router scenarios, there are two instances of Quagga required**.
+which peers with the upstream routers, and ONOS which listens to Quagga (via
+FPM) and programs the underlying fabric. **In dual-router scenarios, there are
+two instances of Quagga required**.
-As before the hardware fabric serves as the data-plane of vRouter.
-In dual-router scenarios, the **external routers MUST be connected to paired-ToRs**.
-
+As before the hardware fabric serves as the data-plane of vRouter. In
+dual-router scenarios, the **external routers MUST be connected to
+paired-ToRs**.
ToR connects to one upstream
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Lets consider the simpler case where the external routers are each connected to a single leaf in a ToR pair.
-The figure on the left below shows the logical view. The figure on the right shows the physical connectivity.
+Lets consider the simpler case where the external routers are each connected to
+a single leaf in a ToR pair. The figure on the left below shows the logical
+view. The figure on the right shows the physical connectivity.
.. image:: ../images/config-dh-vr-logical-simple.png
:width: 200px
@@ -280,19 +318,22 @@
.. image:: ../images/config-dh-vr-physical-simple.png
:width: 400px
-One of the upstream routers is connected to ``of:205`` and the other is connected to ``of:206``.
-Note that ``of:205`` and ``of:206`` are paired ToRs.
+One of the upstream routers is connected to ``of:205`` and the other is
+connected to ``of:206``. Note that ``of:205`` and ``of:206`` are paired ToRs.
-The ToRs are connected via a physical port to separate Quagga VMs or containers.
-These Quagga instances can be placed in any compute node. They do not need to be in the same server, and are only shown to be co-located for simplicity.
+The ToRs are connected via a physical port to separate Quagga VMs or
+containers. These Quagga instances can be placed in any compute node. They do
+not need to be in the same server, and are only shown to be co-located for
+simplicity.
-The two Quagga instances do NOT talk to each other.
-
+The two Quagga instances do NOT talk to each other.
Switch port configuration
"""""""""""""""""""""""""
-The ToRs follow the same rules as single router case described in :doc:`External Connectivity <external-connectivity>`.
-In the example shown above, the switch port config would look like this:
+
+The ToRs follow the same rules as single router case described in
+:doc:`External Connectivity <external-connectivity>`. In the example shown
+above, the switch port config would look like this:
.. code-block:: json
@@ -333,35 +374,51 @@
}
.. note::
- In the example shown above, switch ``of:205`` uses ``VLAN 100`` for bridging the peering session between Quagga1 and ExtRouter1,
- while switch ``of:205`` uses ``VLAN 200`` to do the same for the other peering session.
- But since these vlans and bridging domains are defined on different switches, the VLAN ids could have been the same.
+ In the example shown above, switch ``of:205`` uses ``VLAN 100`` for
+ bridging the peering session between Quagga1 and ExtRouter1, while switch
+ ``of:205`` uses ``VLAN 200`` to do the same for the other peering session.
+ But since these vlans and bridging domains are defined on different
+ switches, the VLAN ids could have been the same.
- This philosophy is consistent with the fabric use of :doc:`bridging <bridging-unicast>`.
+ This philosophy is consistent with the fabric use of :doc:`bridging
+ <bridging-unicast>`.
Quagga configuration
""""""""""""""""""""
-Configuring Quagga for dual external routers are similar to what we described in :doc:`External Connectivity <external-connectivity>`. However, it is worth noting that:
+Configuring Quagga for dual external routers are similar to what we described
+in :doc:`External Connectivity <external-connectivity>`. However, it is worth
+noting that:
-- The two Zebra instances **should point to two different ONOS instances** for their FPM connections.
- For example Zebra in Quagga1 could point to ONOS instance with ``fpm connection ip 10.6.0.1 port 2620``,
- while the other Zebra should point to a different ONOS instance with ``fpm connection ip 10.6.0.2 port 2620``.
- It does not matter which ONOS instances they point to as long as they are different.
-- The two Quagga BGP sessions should appear to come from different routers but still use the same AS number –
- i.e. the two Quaggas' belong to the same AS, the one used to represent the entire Trellis infrastructure.
-- The two upstream routers can belong to the same or different AS,
- but these AS numbers should be different from the one used to represent the Trellis AS.
+- The two Zebra instances **should point to two different ONOS instances** for
+ their FPM connections. For example Zebra in Quagga1 could point to ONOS
+ instance with ``fpm connection ip 10.6.0.1 port 2620``, while the other Zebra
+ should point to a different ONOS instance with ``fpm connection ip 10.6.0.2
+ port 2620``. It does not matter which ONOS instances they point to as long
+ as they are different.
+
+- The two Quagga BGP sessions should appear to come from different routers but
+ still use the same AS number – i.e. the two Quaggas' belong to the same AS,
+ the one used to represent the entire Trellis infrastructure.
+
+- The two upstream routers can belong to the same or different AS, but these AS
+ numbers should be different from the one used to represent the Trellis AS.
+
- Typically both Quagga instances advertise the same routes to the upstream.
- These prefixes belonging to various infrastructure nodes in the deployment should be reachable from either of the leaf switches connected to the upstream routers.
-- The upstream routers may or may not advertise the same routes.
- Trellis will ensure that traffic directed to a route reachable only one upstream router is directed to the appropriate leaf.
+ These prefixes belonging to various infrastructure nodes in the deployment
+ should be reachable from either of the leaf switches connected to the
+ upstream routers.
+- The upstream routers may or may not advertise the same routes. Trellis will
+ ensure that traffic directed to a route reachable only one upstream router is
+ directed to the appropriate leaf.
ToR connects to both upstream
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Now lets consider the **more-complicated but more fault-tolerant** case of each Quagga instance peering with BOTH external routers.
-Again the logical view is shown on the left and the physical view on the right.
+
+Now lets consider the **more-complicated but more fault-tolerant** case of each
+Quagga instance peering with BOTH external routers. Again the logical view is
+shown on the left and the physical view on the right.
.. image:: ../images/config-dh-vr-logical.png
:width: 200px
@@ -379,19 +436,24 @@
- Quagga instance 2 peers with external router R1 via port 2 on switch of:206
- Quagga instance 2 peers with external router R2 via port 1 on switch of:206
-To distinguish between the two peering sessions in the same physical switch, say of:205,
-the physical ports 1 and 2 need to be configured in **different VLANs and subnets**.
-For example, port 1 on of:205 is (untagged) in VLAN 100, while port 2 is in VLAN 101.
+To distinguish between the two peering sessions in the same physical switch,
+say of:205, the physical ports 1 and 2 need to be configured in **different
+VLANs and subnets**. For example, port 1 on of:205 is (untagged) in VLAN 100,
+while port 2 is in VLAN 101.
-Note that peering for **Quagga1 and R1** happens with IPs in the ``10.0.100.0/29`` subnet,
-and for **Quagga 1 and R2** in the **10.0.101.0/29** subnet.
+Note that peering for **Quagga1 and R1** happens with IPs in the
+``10.0.100.0/29`` subnet, and for **Quagga 1 and R2** in the **10.0.101.0/29**
+subnet.
-Furthermore, **pair link** (port 48) on of:205 carries both peering sessions to Quagga1.
-Thus port 48 should now be configured as a **trunk port (vlan-tagged) with both VLANs and both subnets**.
+Furthermore, **pair link** (port 48) on of:205 carries both peering sessions to
+Quagga1. Thus port 48 should now be configured as a **trunk port (vlan-tagged)
+with both VLANs and both subnets**.
-Finally the **Quagga interface** on the VM now needs **sub-interface configuration for each VLAN ID**.
+Finally the **Quagga interface** on the VM now needs **sub-interface
+configuration for each VLAN ID**.
-Similar configuration concepts apply to IPv6 as well. Here is a look at the switch port config in ONOS for of:205
+Similar configuration concepts apply to IPv6 as well. Here is a look at the
+switch port config in ONOS for of:205
.. code-block:: json
diff --git a/configuration/external-connectivity.rst b/configuration/external-connectivity.rst
index 65fbcf9..abc3884 100644
--- a/configuration/external-connectivity.rst
+++ b/configuration/external-connectivity.rst
@@ -6,23 +6,34 @@
Physical Connectivity
^^^^^^^^^^^^^^^^^^^^^
-External routers must be physically connected to one of the fabric leaf switches.
-Currently there is a limitation that the **external/upstream router and the Quagga instance must be connected to the same fabric leaf switch**.
-Therefore it is necessary to use an additional front panel port on the leaf-switch (or at least an additional vlan) to connect to the compute node hosting Quagga.
+External routers must be physically connected to one of the fabric leaf
+switches.
+
+Currently there is a limitation that the **external/upstream router and the
+Quagga instance must be connected to the same fabric leaf switch**.
+
+Therefore it is necessary to use an additional front panel port on the
+leaf-switch (or at least an additional vlan) to connect to the compute node
+hosting Quagga.
.. image:: ../images/config-vr-physical.png
Configure vRouter
^^^^^^^^^^^^^^^^^
-The operator will need to configure a subnet between the Leaf-switch, the external/upstream router and the Quagga instance.
-There are 3 IP addresses we need to allocate - 1 on the switch port, 1 in Quagga, and 1 on the upstream router.
-This means the peering subnet **cannot be smaller than a /29**.
-BGP peering happens between the IP addresses configured on the interfaces in Quagga and the external router.
+The operator will need to configure a subnet between the Leaf-switch, the
+external/upstream router and the Quagga instance. There are 3 IP addresses we
+need to allocate - 1 on the switch port, 1 in Quagga, and 1 on the upstream
+router. This means the peering subnet **cannot be smaller than a /29**.
-Routes are advertised by Quagga to the upstream with the next-hop set to the switch port IP address.
-This means that when traffic comes to the fabric leaf switch from outside, the switch is able to distinguish peering traffic from data traffic and treat each appropriately.
+BGP peering happens between the IP addresses configured on the interfaces in
+Quagga and the external router.
+
+Routes are advertised by Quagga to the upstream with the next-hop set to the
+switch port IP address. This means that when traffic comes to the fabric leaf
+switch from outside, the switch is able to distinguish peering traffic from
+data traffic and treat each appropriately.
The following shows an ONOS interface configuration example:
@@ -52,66 +63,88 @@
}
- ``name``: An arbitrary name string for the interface. Optional.
-- ``ips``: Configure the peering subnet (10.0.1.0/24) and the switch port IP (10.0.1.2).
- Note that we use the same IP address on both the quagga and upstream interfaces.
-- ``vlan-untagged``: Configure the same VLAN ID on both interfaces.
- It doesn't matter exactly what the VLAN ID is, but it must be the same on both the Quagga-facing and upstream-facing interfaces.
+
+- ``ips``: Configure the peering subnet (10.0.1.0/24) and the switch port IP
+ (10.0.1.2). Note that we use the same IP address on both the quagga and
+ upstream interfaces.
+
+- ``vlan-untagged``: Configure the same VLAN ID on both interfaces. It doesn't
+ matter exactly what the VLAN ID is, but it must be the same on both the
+ Quagga-facing and upstream-facing interfaces.
In this case the peering subnet is ``10.0.1.0/24``.
The upstream router is using the ``10.0.1.1`` address.
Quagga is assigned ``10.0.1.3``, which is the address used for peering.
-The upstream router needs to be configured with ``10.0.1.3`` as its BGP neighbor, and the BGP peering will be established between ``10.0.1.1`` and ``10.0.1.3``. The ``10.0.1.2`` address is used by the fabric switch and for the next-hop for routes advertised by Quagga.
-Of course you are not obliged to use ``10.0.1.0/24``, you should use a subnet that makes sense for your peering environment.
+The upstream router needs to be configured with ``10.0.1.3`` as its BGP
+neighbor, and the BGP peering will be established between ``10.0.1.1`` and
+``10.0.1.3``. The ``10.0.1.2`` address is used by the fabric switch and for the
+next-hop for routes advertised by Quagga.
+
+Of course you are not obliged to use ``10.0.1.0/24``, you should use a subnet
+that makes sense for your peering environment.
.. note::
- This configuration will set up an L2 link between the two fabric switch ports, over which the Quagga and external router can communicate.
- Both Quagga and the upstream router will receive untagged packets
- (i.e they will never see packets with vlanId 4000, which is used inside the leaf switch to establish a bridging domain).
+ This configuration will set up an L2 link between the two fabric switch
+ ports, over which the Quagga and external router can communicate.
- If you need a vlan-tag in the compute node to distinguish the traffic going to Quagga, you can change the vlan assignment on the switch port "of:0000000000000001/2" to be vlan-tagged instead of vlan-untagged.
+ Both Quagga and the upstream router will receive untagged packets (i.e they
+ will never see packets with vlanId 4000, which is used inside the leaf
+ switch to establish a bridging domain).
+
+ If you need a vlan-tag in the compute node to distinguish the traffic going
+ to Quagga, you can change the vlan assignment on the switch port
+ "of:0000000000000001/2" to be vlan-tagged instead of vlan-untagged.
Deploy the Quagga Docker Image
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Trellis uses a slightly modified version of Quagga, so the easiest way to deploy this is to use the provided docker image.
+
+Trellis uses a slightly modified version of Quagga, so the easiest way to
+deploy this is to use the provided docker image.
.. code-block:: console
$ docker pull opencord/quagga
-We also need to download the **pipework** tool which will be used to connect the docker image to the physical interface that we set aside earlier.
+We also need to download the **pipework** tool which will be used to connect
+the docker image to the physical interface that we set aside earlier.
.. code-block:: console
$ wget https://raw.githubusercontent.com/jpetazzo/pipework/master/pipework
$ chmod +x pipework
-Create a directory for your Quagga configuration files, and create a bgpd.conf and zebra.conf in there.
-This folder is going to be mounted into the Quagga container.
-More on configuring Quagga later.
+Create a directory for your Quagga configuration files, and create a bgpd.conf
+and zebra.conf in there. This folder is going to be mounted into the Quagga
+container. More on configuring Quagga later.
.. code-block:: console
$ mkdir configs
$ touch zebra.conf bgpd.conf
-Now run the docker image (make sure the path the config directory matches what is on your system):
+Now run the docker image (make sure the path the config directory matches what
+is on your system):
.. code-block:: console
$ sudo docker run --privileged -d -v configs:/etc/quagga -n quagga opencord/quagga
-Finally, we can use the pipework tool to add the physical interface into the container so that Quagga can talk out over the fabric:
+Finally, we can use the pipework tool to add the physical interface into the
+container so that Quagga can talk out over the fabric:
.. code-block:: console
$ sudo ./pipework mlx1 -i eth1 quagga 10.0.1.3/24
-This will add host interface ``mlx1`` to the container with name ``quagga`` with interface name ``eth1`` inside the container.
-The newly added interface will have the IP ``10.0.1.3``.
-This IP address should be the peering subnet address that you want to assign to Quagga.
+This will add host interface ``mlx1`` to the container with name ``quagga``
+with interface name ``eth1`` inside the container. The newly added interface
+will have the IP ``10.0.1.3``. This IP address should be the peering subnet
+address that you want to assign to Quagga.
-If you need to change anything about the container (for example if you change the Quagga configuration) you can remove the original container and run a new one:
+If you need to change anything about the container (for example if you change
+the Quagga configuration) you can remove the original container and run a new
+one:
.. code-block:: console
@@ -120,19 +153,28 @@
Configure Quagga
^^^^^^^^^^^^^^^^
-At this point Quagga should have IP connectivity to the external routers, and it should be able to ping them on the peering subnet.
+
+At this point Quagga should have IP connectivity to the external routers, and
+it should be able to ping them on the peering subnet.
Now Quagga and the upstream routers can be configured to peer with one another.
-This configuration of Quagga is going to be highly dependent on the configuration of the upstream network, so it won't be possible to give comprehensive configuration examples here.
-It is recommended to consult the Quagga documentation for exhaustive information on Quagga's capabilities and configuration.
-Here I will attempt to provide a few basic examples of Quagga configuration to get you started.
-You'll have to enhance these with the features and functions that are needed in your network.
+This configuration of Quagga is going to be highly dependent on the
+configuration of the upstream network, so it won't be possible to give
+comprehensive configuration examples here.
+
+It is recommended to consult the Quagga documentation for exhaustive
+information on Quagga's capabilities and configuration. Here I will attempt to
+provide a few basic examples of Quagga configuration to get you started.
+You'll have to enhance these with the features and functions that are needed in
+your network.
Zebra configuration
"""""""""""""""""""
-Regardless of which routing protocols you are using in your network,
-it is important to configure Zebra's FPM connection to send routes to the FPM app running on ONOS.
-This feature was enabled by the patch that was applied earlier when we installed Quagga.
+
+Regardless of which routing protocols you are using in your network, it is
+important to configure Zebra's FPM connection to send routes to the FPM app
+running on ONOS. This feature was enabled by the patch that was applied
+earlier when we installed Quagga.
A minimal Zebra configuration might look like this:
@@ -145,12 +187,15 @@
fpm connection ip 10.6.0.1 port 2620
!
-The FPM connection IP address is the IP address of **one of the onos cluster instances** - does not matter which one.
-If you have other configuration that needs to go in zebra.conf you should add that here as well.
+The FPM connection IP address is the IP address of **one of the onos cluster
+instances** - does not matter which one. If you have other configuration that
+needs to go in zebra.conf you should add that here as well.
BGP configuration
"""""""""""""""""
-An example simple BGP configuration for peering with one BGP peer might look like this:
+
+An example simple BGP configuration for peering with one BGP peer might look
+like this:
.. code-block:: text
@@ -173,16 +218,22 @@
neighbor 10.0.1.1 route-map NEXTHOP out
!
-This configuration peers with one upstream router ``10.0.1.1`` and advertises one route ``192.168.0.0/16``.
-Note that Quagga (and as a result Trellis) is in a different AS ``65535`` from the upstream router AS ``65540``,
-as we are using E-BGP for this connectivity.
+This configuration peers with one upstream router ``10.0.1.1`` and advertises
+one route ``192.168.0.0/16``. Note that Quagga (and as a result Trellis) is in
+a different AS ``65535`` from the upstream router AS ``65540``, as we are using
+E-BGP for this connectivity.
.. note::
- Pay attention to the configuration to rewrite the next hop of routes that are advertised to the upstream router.
- A ``route-map`` is used to set the next hop of advertised routes to ``10.0.1.2``,
- which is **different from the address that Quagga is using to peer with the external router**.
- As mentioned above, it is important that this rewriting is done correctly so that the fabric switch is able to **distinguish data plane and control plane** traffic.
+ Pay attention to the configuration to rewrite the next hop of routes that
+ are advertised to the upstream router.
+ A ``route-map`` is used to set the next hop of advertised routes to
+ ``10.0.1.2``, which is **different from the address that Quagga is using to
+ peer with the external router**.
+
+ As mentioned above, it is important that this rewriting is done correctly
+ so that the fabric switch is able to **distinguish data plane and control
+ plane** traffic.
Route service and static route
------------------------------
@@ -192,8 +243,11 @@
View routes
"""""""""""
+
This will show routes from all sources, including static and dynamic routes.
-The example below shows routes learned from the upstream router (Source: FPM) and routes configured manually (Source: STATIC)
+
+The example below shows routes learned from the upstream router (Source: FPM)
+and routes configured manually (Source: STATIC)
.. code-block:: text
@@ -287,7 +341,8 @@
Verify routes
^^^^^^^^^^^^^
-Check the leaf switches that the route (e.g. 1.1.0.0/18) has been programmed in the routing table (table 30).
+Check the leaf switches that the route (e.g. 1.1.0.0/18) has been programmed in
+the routing table (table 30).
.. code-block:: console
@@ -299,9 +354,10 @@
Notes about next hops
^^^^^^^^^^^^^^^^^^^^^
-The next hop of a route should be resolvable to a MAC address that is known to ONOS.
-Typically the next hop is a server interface that is known to ONOS as a host learned via ARP or DHCP.
-If you are not sure, check the ``hosts`` command on the ONOS CLI.
+The next hop of a route should be resolvable to a MAC address that is known to
+ONOS. Typically the next hop is a server interface that is known to ONOS as a
+host learned via ARP or DHCP. If you are not sure, check the ``hosts`` command
+on the ONOS CLI.
.. code-block:: console
@@ -311,17 +367,24 @@
id=B2:A4:E2:72:D1:91/None, mac=B2:A4:E2:72:D1:91, location=of:0000000000000204/16, vlan=None, ip(s)=[10.0.1.20], configured=false
id=EE:22:F7:BE:86:50/None, mac=EE:22:F7:BE:86:50, location=of:0000000000000205/16, vlan=None, ip(s)=[10.0.2.15], configured=false
-If the next hop has not been resolved for any reason, it would be necessary to configure the next hop as a host (/32 prefix) together with MAC address and location.
-Learn more about how to configure a host using `Network Config Host Provider <https://wiki.onosproject.org/display/ONOS/Network+Config+Host+Provider>`_
+If the next hop has not been resolved for any reason, it would be necessary to
+configure the next hop as a host (/32 prefix) together with MAC address and
+location.
-Finally note that if you are configuring routes manually/statically and they are publicly routable IPs that should be reachable from “outside”, you would need to configure Quagga to advertise them upstream.
+Learn more about how to configure a host using `Network Config Host Provider
+<https://wiki.onosproject.org/display/ONOS/Network+Config+Host+Provider>`_
+
+Finally note that if you are configuring routes manually/statically and they
+are publicly routable IPs that should be reachable from “outside”, you would
+need to configure Quagga to advertise them upstream.
Route blackhole
---------------
-The blackhole consists of a rule on table 30 on every edge device on the fabric.
-The Table 30 rule matches on a given IP address and mask and has nothing but a clearDeferred action, practically dropping the packet.
-Every IP we want to blackhole will have it's own rule in every edge switch.
+The blackhole consists of a rule on table 30 on every edge device on the
+fabric. The Table 30 rule matches on a given IP address and mask and has
+nothing but a clearDeferred action, practically dropping the packet. Every IP
+we want to blackhole will have it's own rule in every edge switch.
An example of such rule is:
@@ -347,8 +410,10 @@
Ignore certain FPM peer
-----------------------
-The ``FpmConnectionInfo`` consists a new flag ``acceptRoutes``, indicating whether we want to accept or discard the routes advertised by certain FPM peer.
-Per current requirement, we always have the ``acceptRoutes`` flag set to ``true`` by default, meaning that we will accept routes from all peers.
+The ``FpmConnectionInfo`` consists a new flag ``acceptRoutes``, indicating
+whether we want to accept or discard the routes advertised by certain FPM peer.
+Per current requirement, we always have the ``acceptRoutes`` flag set to
+``true`` by default, meaning that we will accept routes from all peers.
We can updated the flag using REST API and CLI command as below
@@ -360,9 +425,6 @@
.. image:: ../images/config-fpm-rest.png
:width: 900px
-
-
-
CLI
^^^
diff --git a/configuration/multicast.rst b/configuration/multicast.rst
index a33fe3c..1d435b0 100644
--- a/configuration/multicast.rst
+++ b/configuration/multicast.rst
@@ -2,8 +2,14 @@
=========
Trellis supports IP multicast in an classic SDN way.
-**None of the common IP multicast protocols (e.g. IGMP, PIM) is used when calculating the multicast tree internally**.
-Instead, benefiting from SDN, Trellis calculates the multicast tree in a centralized way and program the switches accordingly. Having said that, it doesn't prevent us from using common IP multicast protocols to communicate with externals.
+
+**None of the common IP multicast protocols (e.g. IGMP, PIM) is used when
+calculating the multicast tree internally**.
+
+Instead, benefiting from SDN, Trellis calculates the multicast tree in a
+centralized way and program the switches accordingly. Having said that, it
+doesn't prevent us from using common IP multicast protocols to communicate with
+externals.
Here's a summary of multicast features Trellis supports.
@@ -16,19 +22,30 @@
:width: 1000px
.. note::
- Dual-homed sinks and multiple sources support comes with major changes in the Multicast Routing subsystem.
- **Sink and sources are now identified by an ONOS HostId (MAC/VLAN)** and has associated a number of ConnectPoint while in the past we identified the sinks and the sources just using one ConnectPoint.
- Multicast subsystem listen for Host events and according to them modify properly the ConnectPoint associated to the HostId.
- In particular, it leverages the Host Location which is a collection of ONOS ConnectPoint where the Host is attached.
- This information is maintained updated by ONOS thanks to several discovery mechanisms implemented in the Host subsystem.
- Following image try to summarize the work flow of the Multicast Routing subsystem after the aforementioned changes:
+ Dual-homed sinks and multiple sources support comes with major changes in
+ the Multicast Routing subsystem.
+
+ **Sink and sources are now identified by an ONOS HostId (MAC/VLAN)** and
+ has associated a number of ConnectPoint while in the past we identified the
+ sinks and the sources just using one ConnectPoint.
+
+ Multicast subsystem listen for Host events and according to them modify
+ properly the ConnectPoint associated to the HostId.
+
+ In particular, it leverages the Host Location which is a collection of ONOS
+ ConnectPoint where the Host is attached. This information is maintained
+ updated by ONOS thanks to several discovery mechanisms implemented in the
+ Host subsystem. Following image try to summarize the work flow of the
+ Multicast Routing subsystem after the aforementioned changes:
.. image:: ../images/config-mcast-internal.png
:width: 600px
:align: center
.. caution::
- We should avoid using reserved Multicast groups. Please check here for the reserved values: http://www.iana.org/assignments/multicast-addresses/multicast-addresses.xhtml
+ We should avoid using reserved Multicast groups. Please check here for the
+ reserved values:
+ http://www.iana.org/assignments/multicast-addresses/multicast-addresses.xhtml
Activate Multicast
@@ -39,7 +56,8 @@
onos> app activate mcast
-You can also add ``mcast`` into ``$ONOS_APPS`` environment variable such that it gets started automatically every time.
+You can also add ``mcast`` into ``$ONOS_APPS`` environment variable such that
+it gets started automatically every time.
CLI commands
@@ -47,25 +65,35 @@
Creating a route and adding sinks
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-In order to add a new Multicast route, we have to use ``mcast-host-join`` command and we need to provide following information:
+In order to add a new Multicast route, we have to use ``mcast-host-join``
+command and we need to provide following information:
+
1) **source address**; 2) **group address**; 3) **sources** ; 4) **sinks**.
+
Here's an example:
.. code-block:: console
onos> mcast-host-join -sAddr * -gAddr 224.0.0.1 -srcs 00:AA:00:00:00:01/None -srcs 00:AA:00:00:00:05/None -sinks 00:AA:00:00:00:03/None -sinks 00:CC:00:00:00:01/None
-- ``-sAddr``: we can provide **\* for ASM** or an **IP address for SSM**.
- At the time being we do not enforce the matching of the -sAddr on the data-plane;
-- ``-gAddr`` specifies Multicast group address
-- ``-srcs`` identifies the source we changed from ConnectPoint to HostId in this release.
-- ``-sinks`` specifies the sinks of the group, we changed from ConnectPoint to HostId in this release.
+- ``-sAddr``: we can provide **\* for ASM** or an **IP address for SSM**
+ At this time being we do not enforce the matching of the ``-sAddr`` on the
+ data plane.
+- ``-gAddr`` specifies Multicast group address
+
+- ``-srcs`` identifies the source we changed from ConnectPoint to HostId in
+ this release.
+
+- ``-sinks`` specifies the sinks of the group, we changed from ConnectPoint to
+ HostId in this release.
Removing a sink
^^^^^^^^^^^^^^^
-A sink can be removed through ``mcast-sink-delete`` command.
-The major difference is that the sinks to be removed is provided through ``-h`` option.
+
+A sink can be removed through ``mcast-sink-delete`` command. The major
+difference is that the sinks to be removed is provided through ``-h`` option.
+
If ``-h`` is not provided the command will issue a route removal.
.. code-block:: console
@@ -75,9 +103,11 @@
Modifying a source
^^^^^^^^^^^^^^^^^^
-We also provide a command to modify the sources of a Multicast group.
-In particular ``mcast-source-delete`` allows to modify the source connect points of a source.
-If ``-src`` is not provided, the command will issue a route removal
+
+We also provide a command to modify the sources of a Multicast group. In
+particular ``mcast-source-delete`` allows to modify the source connect points
+of a source. If ``-src`` is not provided, the command will issue a route
+removal
.. code-block:: console
@@ -109,14 +139,20 @@
REST APIs
---------
+
You can find the REST API documentation at ``http://<ONOS-IP>:8181/onos/v1/docs``.
+
Please click the drop down list and select Multicast API.
.. image:: ../images/config-mcast-rest.png
We configure and implement multicast routes through the dedicated REST API.
-Some of the APIs are outlined here, you can find complete list in the ONOS docs page.
-The referenced files in these examples can be found in ``$ONOS_ROOT/apps/mcast/web/src/main/resources/jsonExamples``
+
+Some of the APIs are outlined here, you can find complete list in the ONOS docs
+page.
+
+The referenced files in these examples can be found in
+``$ONOS_ROOT/apps/mcast/web/src/main/resources/jsonExamples``
GET operations
^^^^^^^^^^^^^^
@@ -246,8 +282,14 @@
}
- ``group``: IP address of the Multicast Group
-- ``source``: IP address of the Multicast source. If specified (e.g. 10.0.1.1) we will treat this route as SSM (Single Source Multicast), if instead * is used the route will be ASM (Any Source Multicast)
-- ``sources``: Array containing the connect points to which the source is connected to.
-- ``sinks``: Array containing a set of Hosts that we want as sinks of the multicast stream. ONOS will automatically handle the connect points of these hosts if they are single or dual homed.
+- ``source``: IP address of the Multicast source. If specified (e.g. 10.0.1.1)
+ we will treat this route as SSM (Single Source Multicast), if instead * is
+ used the route will be ASM (Any Source Multicast)
+- ``sources``: Array containing the connect points to which the source is
+ connected to.
+
+- ``sinks``: Array containing a set of Hosts that we want as sinks of the
+ multicast stream. ONOS will automatically handle the connect points of these
+ hosts if they are single or dual homed.
diff --git a/configuration/pseudowire.rst b/configuration/pseudowire.rst
index 60d1ca0..09b3917 100644
--- a/configuration/pseudowire.rst
+++ b/configuration/pseudowire.rst
@@ -1,19 +1,28 @@
Pseudowire
==========
-In Trellis, we assign different subnets to hosts that attach to different edge (leaf) switches.
-These hosts communicate over IP when traversing the spine switches, meaning that the layer-2 header is not preserved across transportation.
-However, in some use cases it is beneficial for hosts that connect to different leaves to be able to talk directly over L2 and also belong to the same IP subnet.
+In Trellis, we assign different subnets to hosts that attach to different edge
+(leaf) switches. These hosts communicate over IP when traversing the spine
+switches, meaning that the layer-2 header is not preserved across
+transportation. However, in some use cases it is beneficial for hosts that
+connect to different leaves to be able to talk directly over L2 and also belong
+to the same IP subnet.
+
We implemented Pseudowire (PW) in Trellis to fulfill this requirement.
-Pseudowires create tunnels between different connection points that transport L2 traffic between them.
-Pseudowires can handle single-tagged/double-tagged traffic and are able to modify the VLAN tags in some cases.
+Pseudowires create tunnels between different connection points that transport
+L2 traffic between them.
+
+Pseudowires can handle single-tagged/double-tagged traffic and are able to
+modify the VLAN tags in some cases.
.. note::
Pseudowires are only supported on OF-DPA hardware switches.
Managing pseudowires via REST
-----------------------------
+
You can find the REST API documentation at ``http://${ONOS-IP}:8181/onos/v1/docs``.
+
Please click the drop down list and select Segment Routing Server.
.. image:: ../images/config-pw-rest.png
@@ -27,9 +36,9 @@
$ curl --user onos:rocks -X GET -H 'Content-Type:application/json' http://${ONOS-IP}:8181/onos/segmentrouting/pseudowire/ -d@pseudowire.json
$ curl --user onos:rocks -X DELETE -H 'Content-Type:application/json' http://${ONOS-IP}:8181/onos/segmentrouting/pseudowire/ -d@pseudowire.json
-Below is an example configuration for a single pseudowire,
-which transfers L2 double tagged traffic from cp1 to cp2 and also changes the outer VLAN tag of the packets.
-
+Below is an example configuration for a single pseudowire, which transfers L2
+double tagged traffic from cp1 to cp2 and also changes the outer VLAN tag of
+the packets.
.. code-block:: json
@@ -47,13 +56,21 @@
}
- ``cP1``: First connection point of PW
+
- ``cP2``: Second connection point of PW
-- ``cP1InnerTag / cp1OuterTag``: The expected inner and outer VLAN tags for ``cp1``. If you only want to use a single incoming tag,
- use the innerTag, and use ``None`` for the outerTag.
+
+- ``cP1InnerTag / cp1OuterTag``: The expected inner and outer VLAN tags for
+ ``cp1``. If you only want to use a single incoming tag, use the innerTag, and
+ use ``None`` for the outerTag.
+
- ``cP2InnerTag / cp2OuterTag``: The expected inner and outer VLAN tags for ``cp2``.
+
- ``mode``: Pseudowire mode - RAW and TAGGED. Only RAW mode is supported in Trellis.
+
- ``sdTag``: Service delimiting tag, only meaningful when in TAGGED mode.
+
- ``pwLabel``: Unique pseudowire label.
+
- ``pwId``: ID of the pseudowire.
The REST API also supports bulk addition and removal of pseudowire.
@@ -131,50 +148,76 @@
Type of pseudowires
-------------------
+
- **Leaf - Leaf** : As explained above we support leaf - leaf pseudowires.
-- **Spine - Leaf** : We further support leaf to spine pseudowires,
- where one termination point resides in a spine switch rather than a leaf switch.
- Please note that the encapsulated traffic for these types of pseudowires is carried **tagged** between leaf and spines,
- the reason for this is due to some hardware limitation. The **reserved transport VLAN is 4093**.
+- **Spine - Leaf** : We further support leaf to spine pseudowires, where one
+ termination point resides in a spine switch rather than a leaf switch.
+ Please note that the encapsulated traffic for these types of pseudowires is
+ carried **tagged** between leaf and spines, the reason for this is due to
+ some hardware limitation. The **reserved transport VLAN is 4093**.
- Multi-stage
- - **Leaf - Spine-1 - Spine-2**: Where Spine-1 and Spine-2 are interconnected between them and are part of distinct topologies.
- - **Leaf_1 - Spine_1 - Spine_2 - Leaf_2**: Where leaf-1 and leaf-2 are part of different fabric topologies which are interconnected between them between two spines.
+ - **Leaf - Spine-1 - Spine-2**: Where Spine-1 and Spine-2 are interconnected
+ between them and are part of distinct topologies.
+
+ - **Leaf_1 - Spine_1 - Spine_2 - Leaf_2**: Where leaf-1 and leaf-2 are part
+ of different fabric topologies which are interconnected between them
+ between two spines.
Common errors
-------------
-If your pseudowires do not work make sure you have done the following in the host side :
-- Disabled VLAN offloading in the VM interfaces and also in the compute node interfaces.
+If your pseudowires do not work make sure you have done the following in the
+host side:
+
+- Disabled VLAN offloading in the VM interfaces and also in the compute node
+ interfaces.
+
- Set the compute node interfaces and the VM interfaces at promiscuous mode.
-- In general, do the above for all the interfaces and bridges participated in the topology (either in a VM or physical server).
-- Before programming pseudowires make sure that the topology is completely up and running and also the configuration is added to the controller and the fabric is
- in an operational state.
+
+- In general, do the above for all the interfaces and bridges participated in
+ the topology (either in a VM or physical server).
+
+- Before programming pseudowires make sure that the topology is completely up
+ and running and also the configuration is added to the controller and the
+ fabric is in an operational state.
Restrictions
------------
-We list now some restrictions of pseudowire support, some of them stem from the implementation while others from switch hardware constraints.
+
+We list now some restrictions of pseudowire support, some of them stem from the
+implementation while others from switch hardware constraints.
- We only support pseudowires with the same type of tagged traffic at each end.
- For example, if cP1 handles single tagged traffic then cP2 must handle single tagged traffic (same applies for single tagged and double tagged traffic).
- However, for pseudowires with single tagged traffic we support changing the tag,
- and for pseudowires with double tagged traffic we support changing the outer tag.
- We would like to support more use cases, however we are limited by hardware pipeline restrictions.
- In the future, and if needed, we might address this by using the egress tables of OF-DPA.
-- As of now, we only support RAW mode. We will support also TAGGED mode in the future if needed by any use case.
- Since, we only support RAW mode the sdTag must always be empty or None.
+ For example, if cP1 handles single tagged traffic then cP2 must handle single
+ tagged traffic (same applies for single tagged and double tagged traffic).
+ However, for pseudowires with single tagged traffic we support changing the
+ tag, and for pseudowires with double tagged traffic we support changing the
+ outer tag. We would like to support more use cases, however we are limited
+ by hardware pipeline restrictions. In the future, and if needed, we might
+ address this by using the egress tables of OF-DPA.
+- As of now, we only support RAW mode. We will support also TAGGED mode in the
+ future if needed by any use case. Since, we only support RAW mode the
+ ``sdTag`` must always be empty or None.
Implementation limits
---------------------
+
- **Link failures**: We partially support link failures.
- Specifically, our PW implementation does not properly handle link failures when the link involved is the one used for pseudowire traffic at the spine switches towards a leaf switch.
- This is due to an OF-DPA limitation which does not allow the use of MPLS ECMP groups in the spine.
+ Specifically, our PW implementation does not properly handle link failures
+ when the link involved is the one used for pseudowire traffic at the spine
+ switches towards a leaf switch.
+
+ This is due to an OF-DPA limitation which does not allow the use of MPLS ECMP
+ groups in the spine.
+
We plan to address this in the next release.
-- **Device failures**: We do not handle device failures currently, on a device failure please remove and re-install the pseudowire.
- We will address this soon.
+- **Device failures**: We do not handle device failures currently, on a device
+ failure please remove and re-install the pseudowire. We will address this
+ soon.
diff --git a/configuration/xconnect.rst b/configuration/xconnect.rst
index 6486d96..a33d2b2 100644
--- a/configuration/xconnect.rst
+++ b/configuration/xconnect.rst
@@ -1,11 +1,16 @@
VLAN Cross Connect
==================
-VLAN cross connect creates a L2 bridge between two given ports on the same device.
-Once configured, every packets arriving at one of the port with specific VLAN tag will be sent to another port directly.
-Current implementation only matches on the outermost VLAN tag. If the packet is double tagged, the S-tag will be matched and both the S-tag and C-tag will be persisted.
-VLAN cross connect only works on the same device.
-If you are looking for transporting L2 traffic across devices, please refer to :doc:`pseoduwire <pseudowire>`
+VLAN cross connect creates a L2 bridge between two given ports on the same
+device. Once configured, every packets arriving at one of the port with
+specific VLAN tag will be sent to another port directly. Current
+implementation only matches on the outermost VLAN tag. If the packet is double
+tagged, the S-tag will be matched and both the S-tag and C-tag will be
+persisted.
+
+VLAN cross connect only works on the same device. If you are looking for
+transporting L2 traffic across devices, please refer to :doc:`pseoduwire
+<pseudowire>`
View cross connect via CLI
--------------------------
@@ -70,4 +75,4 @@
}' 'http://${ONOS-IP}:8181/onos/segmentrouting/xconnect'
- ``deviceId``: device ID, e.g. **of:0000000000000201**
-- ``vlanId``: VLAN ID, e.g. **94**
\ No newline at end of file
+- ``vlanId``: VLAN ID, e.g. **94**
diff --git a/dict.txt b/dict.txt
new file mode 100644
index 0000000..59ef6e7
--- /dev/null
+++ b/dict.txt
@@ -0,0 +1,36 @@
+Accton
+Edgecore
+Inventec
+Tofino
+Whitebox
+addr
+bgpd
+blackholing
+decap
+dhcp
+dhcprelay
+dst
+encap
+eth
+ethType
+gw
+inexternal
+mcast
+mpls
+multipath
+netcfg
+ofagent
+ofagentd
+ofdpa
+patchset
+pingall
+privkey
+rst
+uboot
+unconfigured
+virtualenv
+vlan
+vlanId
+vlans
+vtysh
+withvlan
diff --git a/index.rst b/index.rst
index 8bcf3a3..da381f9 100644
--- a/index.rst
+++ b/index.rst
@@ -7,44 +7,70 @@
========
Trellis is an **open-source multi-purpose L2/L3 leaf-spine switching fabric**.
-The development of Trellis over the last four years has been influenced by three core trends in the networking industry:
+
+The development of Trellis over the last four years has been influenced by
+three core trends in the networking industry:
- First, **Trellis is built using bare-metal switches with merchant-silicon ASICs**.
Instead of using OEM networking hardware, Trellis uses hardware directly from ODMs.
- The trend of using bare-metal (white-box) switches is unmistakable in the networking industry today, spurred by the massive bandwidth-density and growing sophistication of merchant silicon ASICs.
- Production quality Trellis today is based on EdgeCore switches with Broadcom Trident2, Tomahawk and Qumran switch ASICs.
- The Trellis team continues to work towards including more ODMs and merchant silicon vendors.
-- Second, **Trellis is based on SDN principles**, to provide simpler, more flexible and easily customizable networks.
- By externalizing the network’s control, management functions and policy decisions in the ONOS SDN controller,
- Trellis provides network operators with a number of SDN benefits compared to traditional box-embedded network control.
- These include centralized configuration, automation, operation and troubleshooting.
+ The trend of using bare-metal (white-box) switches is unmistakable in the
+ networking industry today, spurred by the massive bandwidth-density and
+ growing sophistication of merchant silicon ASICs.
+
+ Production quality Trellis today is based on EdgeCore switches with Broadcom
+ Trident2, Tomahawk and Qumran switch ASICs. The Trellis team continues to
+ work towards including more ODMs and merchant silicon vendors.
+
+- Second, **Trellis is based on SDN principles**, to provide simpler, more
+ flexible and easily customizable networks. By externalizing the network’s
+ control, management functions and policy decisions in the ONOS SDN
+ controller, Trellis provides network operators with a number of SDN benefits
+ compared to traditional box-embedded network control. These include
+ centralized configuration, automation, operation and troubleshooting.
- Third, **Trellis is open-source**.
- The networking industry has seen an explosion of open source projects, and network operators have been eager to embrace open-source solutions.
- Trellis allows operators unparalleled ability to customize Trellis for their application, integrate with the rest of their systems,
- add features and APIs themselves and not be beholden to a traditional vendor’s timelines and prices.
- An absence of commercial licenses lowers the bar for anyone to try out Trellis.
+ The networking industry has seen an explosion of open source projects, and
+ network operators have been eager to embrace open-source solutions. Trellis
+ allows operators unparalleled ability to customize Trellis for their
+ application, integrate with the rest of their systems, add features and APIs
+ themselves and not be beholden to a traditional vendor’s timelines and
+ prices. An absence of commercial licenses lowers the bar for anyone to try
+ out Trellis.
-Together, all three attributes of Trellis considerably lower the Total Cost of Ownership (TCO) for operators who plan to run it in production.
+Together, all three attributes of Trellis considerably lower the Total Cost of
+Ownership (TCO) for operators who plan to run it in production.
.. note::
- You can learn more about Trellis features and design concepts by visiting the `Project Website <https://opennetworking.org/trellis>`_ and
- reading the `Platform Brief <https://www.opennetworking.org/wp-content/uploads/2019/09/TrellisPlatformBrief.pdf>`_.
+ You can learn more about Trellis features and design concepts by visiting
+ the `Project Website <https://opennetworking.org/trellis>`_ and reading the
+ `Platform Brief
+ <https://www.opennetworking.org/wp-content/uploads/2019/09/TrellisPlatformBrief.pdf>`_.
Highlights
==========
-- Classic-SDN Control with ONOS to directly program ASIC forwarding tables in bare metal switches with merchant silicon
-- L2 forwarding (Bridging) within server-racks and L3 forwarding (Routing) across racks
-- MPLS Segment routing for better scale and reduced programming
-- Control plane functionality with Trellis vRouter for external connectivity
-- N-way redundancy and tier-1 telecom operator scale
-- Support for different use cases ranging from Central Office to backhaul in operator networks
-- Single-pane-of-glass monitoring and troubleshooting with ONOS
-- CapEx and OpEx savings with white-box hardware and open source software
-- Future Proof: P4 and Stratum integration to unlock advanced capabilities
+- Classic-SDN Control with ONOS to directly program ASIC forwarding tables in
+ bare metal switches with merchant silicon
+
+- L2 forwarding (Bridging) within server-racks and L3 forwarding (Routing)
+ across racks
+
+- MPLS Segment routing for better scale and reduced programming
+
+- Control plane functionality with Trellis vRouter for external connectivity
+
+- N-way redundancy and tier-1 telecom operator scale
+
+- Support for different use cases ranging from Central Office to backhaul in
+ operator networks
+
+- Single-pane-of-glass monitoring and troubleshooting with ONOS
+
+- CapEx and OpEx savings with white-box hardware and open source software
+
+- Future Proof: P4 and Stratum integration to unlock advanced capabilities
Use Cases
=========
@@ -73,11 +99,17 @@
Get Started
===========
-- Read the `Quick Start <quickstart.html>`_ guide and get some hands on experiences.
-- Stay in touch by joining
- - `Trellis Developer Mailing List <https://groups.google.com/a/opennetworking.org/forum/#!forum/trellis-dev>`_
- - \#trellis channel in `ONOS Slack <https://onosproject.slack.com>`_ (`register <https://slackin.onosproject.org/>`_)
+- Read the `Quick Start <quickstart.html>`_ guide and get some hands on
+ experiences.
+
+- Stay in touch by joining:
+
+ - `Trellis Developer Mailing List
+ <https://groups.google.com/a/opennetworking.org/forum/#!forum/trellis-dev>`_
+
+ - \#trellis channel in `ONOS Slack <https://onosproject.slack.com>`_
+ (`register <https://slackin.onosproject.org/>`_)
.. toctree::
:maxdepth: 1
diff --git a/installation.rst b/installation.rst
index 9e4be7e..f96483d 100644
--- a/installation.rst
+++ b/installation.rst
@@ -10,30 +10,48 @@
Install Controller - ONOS
=========================
-You could run Trellis with a `single instance of ONOS <https://wiki.onosproject.org/display/ONOS/Local+workflow>`_. But it is recommended to run ONOS as a cluster.
-The idea is to have a "build machine", where you host and build ONOS source code. But to launch it in operation you use other "target machines" (VMs, containers or servers).
-Typically we use `STC <https://wiki.onosproject.org/pages/viewpage.action?pageId=12421025>`_ to launch ONOS ins 3 target machines which form a ONOS cluster.
+You could run Trellis with a `single instance of ONOS
+<https://wiki.onosproject.org/display/ONOS/Local+workflow>`_. But it is
+recommended to run ONOS as a cluster.
+
+The idea is to have a "build machine", where you host and build ONOS source
+code. But to launch it in operation you use other "target machines" (VMs,
+containers or servers).
+
+Typically we use `STC
+<https://wiki.onosproject.org/pages/viewpage.action?pageId=12421025>`_ to
+launch ONOS ins 3 target machines which form a ONOS cluster.
Download ONOS |onos_version|
---------------------------------
-Trellis currently is released as part of ONOS and therefore it follows ONOS version number.
-This document is written based on ONOS |onos_version|.
-You can find more information about how to setup build environment and fetch ONOS source code from `Development Environment Setup <https://wiki.onosproject.org/display/ONOS/Development+Environment+Setup>`_.
+Trellis currently is released as part of ONOS and therefore it follows ONOS
+version number. This document is written based on ONOS |onos_version|.
+
+You can find more information about how to setup build environment and fetch
+ONOS source code from `Development Environment Setup
+<https://wiki.onosproject.org/display/ONOS/Development+Environment+Setup>`_.
Prepare Your Target Machine
---------------------------
-Please refer to `target machines requirements <https://wiki.onosproject.org/display/ONOS/Cells+and+ONOS+test+scripts#CellsandONOStestscripts-Targetmachinesrequirements>`_ for details (you only have to do this once).
-You do not need to follow the directions for the "Mininet Target Machine" if you are using hardware switches.
+Please refer to `target machines requirements
+<https://wiki.onosproject.org/x/twAQ>`_ for details (you only have to do this
+once).
+
+You do not need to follow the directions for the "Mininet Target Machine" if
+you are using hardware switches.
Create a Cell File
--------------------
-We need to compose a cell file in order to give the information about target machines to the build machine. Please refer to `test cells <https://wiki.onosproject.org/display/ONOS/Cells+and+ONOS+test+scripts#CellsandONOStestscripts-TestCells>`_ for details. For example:
+
+We need to compose a cell file in order to give the information about target
+machines to the build machine. Please refer to `test cells
+<https://wiki.onosproject.org/x/twAQ>`_ for details. For example:
.. code-block:: bash
@@ -48,21 +66,39 @@
export ONOS_WEB_USER=onos
export ONOS_WEB_PASS=rocks
-- ``OC1, OC2, OC3`` are the IP addresses of the 3 target machines where the ONOS cluster will be deployed.
+- ``OC1, OC2, OC3`` are the IP addresses of the 3 target machines where the
+ ONOS cluster will be deployed.
+
- ``ONOS_APPS`` are the apps you want to automatically deploy at launch.
- - ``drivers`` includes drivers for various devices and pipelines. Always required.
- - ``gui`` enables graphic user interface. Highly recommended.
- - ``openflow`` is a meta app that loads ``openflow-base``, ``lldpprovider`` and ``hostprovider``. Always required.
- - ``segmentrouting`` controls forwarding in the fabric. Always required.
- - ``fpm`` (Forwarding Plane Manager) exchanges forwarding information with Quagga. Required if connecting to external router.
- - ``dhcprelay`` relays DHCP packets between clients and servers. Required if using DHCP to configure IP addresses for hosts
- - ``routeradvertisement`` periodically sends IPv6 router advertisement packets on configured interfaces. Required if using IPv6
- - ``hostprobingprovider`` probes and verifies locations of dual-homed hosts. Required if using dual-homing feature with paired switches
- - ``t3`` (Trellis Troubleshooting Tool) is very useful for debugging. Highly recommended.
+ - ``drivers`` includes drivers for various devices and pipelines. Always
+ required.
-- ``ONOS_USER`` is used to login to the target machines,
- and the password is not required as you have setup the target machines for password-less access over ssh.
+ - ``gui`` enables graphic user interface. Highly recommended.
+
+ - ``openflow`` is a meta app that loads ``openflow-base``, ``lldpprovider``
+ and ``hostprovider``. Always required.
+
+ - ``segmentrouting`` controls forwarding in the fabric. Always required.
+
+ - ``fpm`` (Forwarding Plane Manager) exchanges forwarding information with
+ Quagga. Required if connecting to external router.
+
+ - ``dhcprelay`` relays DHCP packets between clients and servers. Required
+ if using DHCP to configure IP addresses for hosts
+
+ - ``routeradvertisement`` periodically sends IPv6 router advertisement
+ packets on configured interfaces. Required if using IPv6
+
+ - ``hostprobingprovider`` probes and verifies locations of dual-homed
+ hosts. Required if using dual-homing feature with paired switches
+
+ - ``t3`` (Trellis Troubleshooting Tool) is very useful for debugging.
+ Highly recommended.
+
+- ``ONOS_USER`` is used to login to the target machines, and the password is
+ not required as you have setup the target machines for password-less access
+ over ssh.
Once that's done, you can use the ``cell`` command to load the cell file.
@@ -74,6 +110,7 @@
Check Your Target Environment
-----------------------------
+
Check if your environment is OK with ``stc prerequisites`` (you only have to do this once).
.. code-block:: console
@@ -168,7 +205,9 @@
Useful Utilities
----------------
-- Push network configuration to one particular instance of ONOS (all instances will automatically get the same configuration)
+
+- Push network configuration to one particular instance of ONOS (all instances
+ will automatically get the same configuration)
.. code-block:: console
@@ -189,10 +228,14 @@
Install Switch OS - ONL
=======================
-The switches listed in the :doc:`Supported Hardware <supported-hardware>` are shipped with Open Networking Install Environment (ONIE) boot loader.
-After booting up, we should see the ONIE prompt from console.
-Here we assume that the management port on the switch already has Internet access. (via DHCP)
+The switches listed in the :doc:`Supported Hardware <supported-hardware>` are
+shipped with Open Networking Install Environment (ONIE) boot loader.
+
+After booting up, we should see the ONIE prompt from console.
+
+Here we assume that the management port on the switch already has Internet
+access. (via DHCP)
Enter ONIE
----------
@@ -200,18 +243,28 @@
Has no ONL or Has ONL 1.x previously installed
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
The way to install ONL 2.x is the same as ONL 1.x.
-However, if you have no ONL installed or have older version of ONL, you might find it tricky to (re)install a newer version.
+
+However, if you have no ONL installed or have older version of ONL, you might
+find it tricky to (re)install a newer version.
+
Here's the instruction:
1. Plug in the console cable and reboot the switch
- 2. (If your boot loader is grub) When you see the boot menu, select **ONIE -> ONIE: Rescue**
- 3. (If your boot loader is uboot) When you see **Hit any key to stop autoboot** instead of the boot menu,
- press any key and then enter ``run onie_rescue`` to enter ONIE rescue mode.
+
+ 2. (If your boot loader is grub) When you see the boot menu, select **ONIE
+ -> ONIE: Rescue**
+
+ 3. (If your boot loader is uboot) When you see **Hit any key to stop
+ autoboot** instead of the boot menu, press any key and then enter
+ ``run onie_rescue`` to enter ONIE rescue mode.
Has ONL 2.x previously installed
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-It would be much more easier to reboot into ONIE if you have a previously installed ONL 2. Simply run:
+
+It would be much more easier to reboot into ONIE if you have a previously
+installed ONL 2. Simply run:
.. code-block:: console
@@ -225,9 +278,12 @@
Install ONL
-----------
-At the ONIE prompt, we need to download and install ONL.
-To be able to dowload ONL the management interface needs to have an IP.
-ONIE will try to obtain IP address from DHCP. In the case where DHCP service is not available, please configure it manually:
+
+At the ONIE prompt, we need to download and install ONL. To be able to
+download ONL the management interface needs to have an IP. ONIE will try to
+obtain IP address from DHCP. In the case where DHCP service is not available,
+please configure it manually:
+
.. code-block:: console
ip addr add <IP>/<Netmask> dev <interface>
@@ -243,10 +299,14 @@
**Checksum**: *sha256:2db316ea83f5dc761b9b11cc8542f153f092f3b49d82ffc0a36a2c41290f5421*
-The switch will automatically reboot into ONL after installation. Default login credential of ONL is: ``root/onl``
+The switch will automatically reboot into ONL after installation. Default login
+credential of ONL is: ``root/onl``
-We might want to configure a fixed IP address for the management interface in ONL.
-First edit ``/etc/network/interfaces`` and configure ``ma1``, which is the management interface.
+We might want to configure a fixed IP address for the management interface in
+ONL.
+
+First edit ``/etc/network/interfaces`` and configure ``ma1``, which is the
+management interface.
.. code-block:: text
@@ -265,15 +325,24 @@
**Community vs. Premium Version**
The OF-DPA image we distribute for free with Trellis is a community version.
+
This documentation is also written based on the community version.
- The **community version** has most of the features available and therefore it is **good for small scale deployments** such as lab trials.
- However, we highly recommend you to get the **premium version** from Broadcom if you are aiming for **production deployments at scale**.
+
+ The **community version** has most of the features available and therefore
+ it is **good for small scale deployments** such as lab trials.
+
+ However, we highly recommend you to get the **premium version** from
+ Broadcom if you are aiming for **production deployments at scale**.
Install OF-DPA
--------------
-We need to use different OF-DPA images depending on the switch model you are using.
-Please find the installer URL corresponding to each switch model in the `Vendor Specific Information`_ section.
+
+We need to use different OF-DPA images depending on the switch model you are
+using.
+
+Please find the installer URL corresponding to each switch model in the `Vendor
+Specific Information`_ section.
Copy the image to the switch and start the installation process by running:
@@ -282,7 +351,9 @@
$ scp ${OFDPA_DEB} ${SWITCH_IP}:/root
$ dpkg -i --force-overwrite /root/${OFDPA_DEB}
-For example, assuming the OF-DPA image is ``ofdpa_3.0.5.5+accton1.7-1_amd64.deb`` and the switch management IP is ``10.128.0.201``, you should run:
+For example, assuming the OF-DPA image is
+``ofdpa_3.0.5.5+accton1.7-1_amd64.deb`` and the switch management IP is
+``10.128.0.201``, you should run:
.. code-block:: console
@@ -294,17 +365,24 @@
============================
.. caution::
- We are going to describe two different ways to start OF-DPA and OpenFlow agent in this section.
- Some vendors use ``ofagentd`` service while others use old ``launcher`` app to start the agent.
- Please check the `Vendor Specific Information`_ to understand which method you should use before continuing this section.
+ We are going to describe two different ways to start OF-DPA and OpenFlow
+ agent in this section.
+
+ Some vendors use ``ofagentd`` service while others use old ``launcher`` app
+ to start the agent.
+
+ Please check the `Vendor Specific Information`_ to understand which method
+ you should use before continuing this section.
Use ofagentd service
--------------------
Launch with ofagentd
^^^^^^^^^^^^^^^^^^^^
-The OFDPA software and the Indigo agent are now a single process launched by Linux service. First of all, we need to configure **/etc/ofagent/ofagent.conf**.
-Uncomment and edit the following line. Make sure all other lines stay commented.
+The OFDPA software and the Indigo agent are now a single process launched by
+Linux service. First of all, we need to configure
+**/etc/ofagent/ofagent.conf**. Uncomment and edit the following line. Make
+sure all other lines stay commented.
.. code-block:: bash
@@ -317,20 +395,27 @@
OPT_ARGS="-d 2 -c 2 -c 4 -t <controller_ip_1> -t <controller_ip_2> -t <controller_ip_3> -i <dpid>"
- To **start** the ofagent, run ``service ofagentd start``
+
- To **stop** the agent, run ``service ofagentd stop``
+
- More ``ofagentapp`` options can be found by running ``ofagentapp --help``
Launch in listen mode with ofagentd
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-An optional ``-l`` parameter can be added for switch to listen on specific **IP:port**.
-This allows us to use tools like **ovs-ofctl** or **dpctl** to connect and control the switch.
+
+An optional ``-l`` parameter can be added for switch to listen on specific
+**IP:port**.
+
+This allows us to use tools like **ovs-ofctl** or **dpctl** to connect and
+control the switch.
.. code-block:: bash
OPT_ARGS="-t <controller_ip_1> -t <controller_ip_2> -t <controller_ip_3> -i <dpid> -l <ip>:<port>"
-Please refer to `Connect to Switch in Listen Mode`_ to learn more about how to access switches in listen mode.
+Please refer to `Connect to Switch in Listen Mode`_ to learn more about how to
+access switches in listen mode.
Use launcher app
----------------
@@ -357,14 +442,19 @@
Launch in listen mode with launcher
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-An optional ``-l`` parameter can be added for switch to listen on specific **IP:port**.
-This allows us to use tools like **ovs-ofctl** or **dpctl** to connect and control the switch.
+
+An optional ``-l`` parameter can be added for switch to listen on specific
+**IP:port**.
+
+This allows us to use tools like **ovs-ofctl** or **dpctl** to connect and
+control the switch.
.. code-block:: console
# ./launcher ofagentapp <controller_ip_1> -t <controller_ip_2> -t <controller_ip_3> -i <dpid> -l <ip>:<port>
-Please refer to `Connect to Switch in Listen Mode`_ to learn more about how to access switches in listen mode.
+Please refer to `Connect to Switch in Listen Mode`_ to learn more about how to
+access switches in listen mode.
Vendor Specific Information
@@ -375,13 +465,19 @@
--------
- **OF-DPA Image**
- - `EdgeCore 5712-54X / 5812-54X / 6712-32X / 7712-32X <https://raw.githubusercontent.com/onfsdn/atrium-docs/master/16A/ONOS/builds/ofdpa_3.0.5.5%2Baccton1.7-1_amd64.deb>`_ - **Checksum**: *sha256:db228b6e79fb15f77497b59689235606b60abc157e72fc3356071bcc8dc4c01f*
+
+ - `EdgeCore 5712-54X / 5812-54X / 6712-32X / 7712-32X
+ <https://raw.githubusercontent.com/onfsdn/atrium-docs/master/16A/ONOS/builds/ofdpa_3.0.5.5%2Baccton1.7-1_amd64.deb>`_
+ **Checksum**: *sha256:db228b6e79fb15f77497b59689235606b60abc157e72fc3356071bcc8dc4c01f*
- **Start OF-DPA and OpenFlow agent**
+
- Please refer to `Use ofagentd service`_
- **Configure Port Speed and Breakout**
- - By default all the switch ports are running at maximum speed. The port speed can be modified in **/etc/accton/ofdpa.conf**
+
+ - By default all the switch ports are running at maximum speed. The port
+ speed can be modified in **/etc/accton/ofdpa.conf**
.. code-block:: text
@@ -395,20 +491,28 @@
OF-DPA service needs to be restarted after modifying this configuration.
-
QCT
---
- OF-DPA Image
- - `QuantaMesh T3048-LY8 <https://raw.githubusercontent.com/onfsdn/atrium-docs/master/16A/ONOS/builds/ofdpa-ag7648_0.3.0.5.6_amd64.deb>`_ - **Checksum**: *sha256:f8201530b1452145c1a0956ea1d3c0402c3568d090553d0d7b3c91a79137da9e*
- - `QuantaMesh T7032-IX1 / IX1B <https://raw.githubusercontent.com/onfsdn/atrium-docs/master/16A/ONOS/builds/ofdpa-ix1_0.3.0.5.0-EA5-qct-01.00_amd64.deb>`_ **Checksum**: *sha256:278b8ffed8a8fc705a1b60d16f8e70377e78342a27a11568a1d80b1efd706a46*
+
+ - `QuantaMesh T3048-LY8
+ <https://raw.githubusercontent.com/onfsdn/atrium-docs/master/16A/ONOS/builds/ofdpa-ag7648_0.3.0.5.6_amd64.deb>`_
+ **Checksum**: *sha256:f8201530b1452145c1a0956ea1d3c0402c3568d090553d0d7b3c91a79137da9e*
+
+ - `QuantaMesh T7032-IX1 / IX1B
+ <https://raw.githubusercontent.com/onfsdn/atrium-docs/master/16A/ONOS/builds/ofdpa-ix1_0.3.0.5.0-EA5-qct-01.00_amd64.deb>`_
+ **Checksum**: *sha256:278b8ffed8a8fc705a1b60d16f8e70377e78342a27a11568a1d80b1efd706a46*
- **Start OF-DPA and OpenFlow agent**
- Please refer to `Use launcher app`_
- Configure Port Speed and Breakout
+
The command ``client_drivshell`` can be used to configure the port speed.
- Use the parameter ``ps`` to list all ports or ``ps <port number>`` to list one specific port.
+
+ Use the parameter ``ps`` to list all ports or ``ps <port number>`` to list
+ one specific port.
.. code-block:: console
@@ -420,7 +524,8 @@
Returned from ofdpaBcmCommand rpc with rc = 0.
- Then use the parameter ``port <port number> sp=<port speed>`` to change the port speed.
+ Then use the parameter ``port <port number> sp=<port speed>`` to change the
+ port speed.
.. code-block:: console
@@ -441,20 +546,27 @@
-----
- OF-DPA Image
- - `Delta AG7648 <https://raw.githubusercontent.com/onfsdn/atrium-docs/master/16A/ONOS/builds/ofdpa-ly8_0.3.0.5.0-EA5-qct-01.01_amd64.deb>`_ - **Checksum**: *sha256:ddfc13cb98ca47291dce5e6938b1d65f0b99bbe77f0585e36ac0007017397f23*
+
+ - `Delta AG7648
+ <https://raw.githubusercontent.com/onfsdn/atrium-docs/master/16A/ONOS/builds/ofdpa-ly8_0.3.0.5.0-EA5-qct-01.01_amd64.deb>`_
+ **Checksum**: *sha256:ddfc13cb98ca47291dce5e6938b1d65f0b99bbe77f0585e36ac0007017397f23*
- **Start OF-DPA and OpenFlow agent**
+
- Please refer to `Use launcher app`_
- Special instructions to install ONL
- Make sure ``/etc/machine.conf`` looks like the following in ONIE before running ONL installation script:
+
+ Make sure ``/etc/machine.conf`` looks like the following in ONIE before
+ running ONL installation script:
.. code-block:: text
onie_platform=x86_64-delta_<platform name>-r0
onie_machine=delta_<platform name>
- After the installation of ONL, if you don't see '/usr/bin' in your PATH variable, please run the following command:
+ After the installation of ONL, if you don't see '/usr/bin' in your PATH
+ variable, please run the following command:
.. code-block:: console
@@ -489,13 +601,20 @@
-------------
OFDPA offers two boot mode: **INSTALLED** and **SWI**.
-- In **INSTALLED** mode, ONL mounts the root filesystem from /dev/sdb7 which is a flash drive that persists everything even during a power cycle.
-- In **SWI** mode, ONL will load the root filesystem from a read-only image (\*.swi) and put the modified files in an copy-on-write overlay filesystem.
- The change will not be persisted during a power cycle.
- The pros and cons are obvious. INSTALLED mode is more convenient while SWI mode is more error-safe.
+- In **INSTALLED** mode, ONL mounts the root filesystem from ``/dev/sdb7``
+ which is a flash drive that persists everything even during a power cycle.
-The boot mode will be determined by which image we used install ONL. We use the **INSTALLED** mode in this instruction for simplicity.
-But we can still change it after the installation. Here's the instruction:
+- In **SWI** mode, ONL will load the root filesystem from a read-only image
+ (``*.swi``) and put the modified files in an copy-on-write overlay filesystem.
+
+ The change will not be persisted during a power cycle.
+
+ The pros and cons are obvious. INSTALLED mode is more convenient while SWI
+ mode is more error-safe.
+
+The boot mode will be determined by which image we used install ONL. We use the
+**INSTALLED** mode in this instruction for simplicity. But we can still change
+it after the installation. Here's the instruction:
.. code-block:: console
@@ -506,15 +625,23 @@
Persistence Mechanism
---------------------
-INSTALLED mode already persists everything for you. This section is mainly for SWI mode.
-The mechanism to persist files on the switch is different from ONL 1.x to ONL 2.x.
-Following are the steps required to persist a file (e.g. OFDPA package) in ONL 2.x.
+INSTALLED mode already persists everything for you. This section is mainly for
+SWI mode.
+
+The mechanism to persist files on the switch is different from ONL 1.x to ONL
+2.x. Following are the steps required to persist a file (e.g. OFDPA package)
+in ONL 2.x.
- Files put in **/mnt/onl/data** will be persisted automatically.
-- To install a deb package during start-up, put the deb file under **/mnt/onl/data/install-debs** folder and
- create a plain text file **/mnt/onl/data/install-debs/list** with all the filename of the deb (e.g. ofdpa_3.0.5.5+accton1.7-1_amd64.deb)
-- To execute some command during start-up, put the command in **/mnt/onl/data/rc.boot**
+
+- To install a deb package during start-up, put the deb file under
+ **/mnt/onl/data/install-debs** folder and create a plain text file
+ **/mnt/onl/data/install-debs/list** with all the filename of the deb (e.g.
+ ``ofdpa_3.0.5.5+accton1.7-1_amd64.deb``)
+
+- To execute some command during start-up, put the command in
+ **/mnt/onl/data/rc.boot**
.. code-block:: bash
@@ -524,26 +651,44 @@
# grant executable permission
chmod a+x /mnt/onl/data/rc.boot
-See `PersistWorkflow.md <https://github.com/opencomputeproject/OpenNetworkLinux/blob/master/docs/PersistWorkflow.md>`_ for more detail.
+See `PersistWorkflow.md
+<https://github.com/opencomputeproject/OpenNetworkLinux/blob/master/docs/PersistWorkflow.md>`_
+for more detail.
Build a Customized ONL Image
----------------------------
-Sometimes we need to build our own ONL image from source to include some extra files. (The most common case is a modified /etc/network/interfaces).
-The instruction of build process has already been well-documented in ONL repo `Building.md <https://github.com/opencomputeproject/OpenNetworkLinux/blob/master/docs/Building.md>`_.
+
+Sometimes we need to build our own ONL image from source to include some extra
+files. (The most common case is a modified /etc/network/interfaces).
+
+The instruction of build process has already been well-documented in ONL repo
+`Building.md
+<https://github.com/opencomputeproject/OpenNetworkLinux/blob/master/docs/Building.md>`_.
+
But you might find it a little bit tricky to inject some extra files.
-To do that, you need to put the files under **$ONL_ROOT/builds/any/rootfs/jessie/common/overlay/** and then run the make command again.
+
+To do that, you need to put the files under
+**$ONL_ROOT/builds/any/rootfs/jessie/common/overlay/** and then run the make
+command again.
Recovery from a Faulty ONL Install
----------------------------------
-If, for some reason, the ONL install process fails, you may be brought to the grub rescue prompt upon reboot.
-You may or may not find system files, or even basic grub rescue commands (i.e. 'help').
+If, for some reason, the ONL install process fails, you may be brought to the
+grub rescue prompt upon reboot.
+
+You may or may not find system files, or even basic grub rescue commands (i.e.
+'help').
+
There are two options for returning to a (known working) ONIE prompt.
-1. **Reinstaller** - You should be able to acquire an ISO image for resetting the firmware to factory default from your switch vendor.
- Once you have the ISO, you can either use PXE or a live USB to boot the switch into the image.
-2. **Manually boot to ONIE** - If the files are there, you can manually configure GRUB to load the necessary modules to boot back into ONIE.
+1. **Reinstaller** - You should be able to acquire an ISO image for resetting
+ the firmware to factory default from your switch vendor. Once you have the
+ ISO, you can either use PXE or a live USB to boot the switch into the image.
+
+2. **Manually boot to ONIE** - If the files are there, you can manually
+ configure GRUB to load the necessary modules to boot back into ONIE.
.. code-block:: console
@@ -567,12 +712,15 @@
# The system will not boot into ONIE and you can recover or re-install from there.
grub rescue> boot
-Once back at the ONIE prompt, you can try to install ONL by following the same steps again.
+Once back at the ONIE prompt, you can try to install ONL by following the same
+steps again.
Connect to Switch in Listen Mode
--------------------------------
-To connect to the switch in Listen Mode using **ovs-ofctl**, you can use commands such as
+
+To connect to the switch in Listen Mode using **ovs-ofctl**, you can use
+commands such as:
.. code-block:: console
@@ -581,10 +729,16 @@
$ ovs-ofctl -O OpenFlow13 dump-groups tcp:<ip>:<port>
$ ovs-ofctl -O OpenFlow13 add-flow tcp:<ip>:<port> table=60,priority=40000,eth_type=0x0800,ip_dst=55.55.55.55,actions=controller
-For more command, please see ``ovs-ofctl --help`` or the post `OpenvSwitch ovs-ofctl and OF-DPA <https://blog.pichuang.com.tw/20170106-ovs-ofctl-and-ofdpa/>`_.
+For more command, please see ``ovs-ofctl --help`` or the post `OpenvSwitch
+ovs-ofctl and OF-DPA
+<https://blog.pichuang.com.tw/20170106-ovs-ofctl-and-ofdpa/>`_.
Reference
---------
-- `ONL/OF-DPA installation guide by Edgecore Networks <https://www.youtube.com/watch?v=mCUtkIc_hVI>`_
-- `ONL/OF-DPA cheat sheet by Phil Huang <https://blog.pichuang.com.tw/20170104-ofdpa-with-onl-cheat-sheet/>`_
+
+- `ONL/OF-DPA installation guide by Edgecore Networks
+ <https://www.youtube.com/watch?v=mCUtkIc_hVI>`_
+
+- `ONL/OF-DPA cheat sheet by Phil Huang
+ <https://blog.pichuang.com.tw/20170104-ofdpa-with-onl-cheat-sheet/>`_
diff --git a/quickstart.rst b/quickstart.rst
index 77938a0..a7fd97c 100644
--- a/quickstart.rst
+++ b/quickstart.rst
@@ -3,17 +3,31 @@
Trellis in a Box
----------------
-`Trellis in a Box (TiaB) <https://github.com/opennetworkinglab/routing/tree/master/trellis>`_ is a software emulated environment that allows people to run Trellis without hardware switches.
-TiaB comes with a few Mininet scripts that emulate various network topologies from a **simple 2x2 leaf-spine** to a **full multi-stage** fabric.
-The configurations corresponding to each topology are also packaged together such that you can start the environment right away.
+`Trellis in a Box (TiaB)
+<https://github.com/opennetworkinglab/routing/tree/master/trellis>`_ is a
+software emulated environment that allows people to run Trellis without
+hardware switches.
-Currently TiaB only emulates the data plane, meaning that you will have to run ONOS separately.
+TiaB comes with a few Mininet scripts that emulate various network topologies
+from a **simple 2x2 leaf-spine** to a **full multi-stage** fabric.
+
+The configurations corresponding to each topology are also packaged together
+such that you can start the environment right away.
+
+Currently TiaB only emulates the data plane, meaning that you will have to run
+ONOS separately.
+
We plan to integrate the control plane (ONOS) in the future.
-
Trellis with Stratum
--------------------
-There is another tutorial of `How to Use Trellis with Stratum Switches <https://github.com/stratum/stratum/tree/master/tools/mininet/examples/trellis>`_.
-This is also a software emulated environment similar to TiaB. But unlike TiaB, this tutorial uses ``stratum-bmv2`` to emulate the data plane switches instead of ``ovs``.
-This tutorial focuses more on Trellis and Stratum integration and therefore only demonstrates a **simple 2x2 leaf-spine** topology.
+There is another tutorial of `How to Use Trellis with Stratum Switches
+<https://github.com/stratum/stratum/tree/master/tools/mininet/examples/trellis>`_.
+
+This is also a software emulated environment similar to TiaB. But unlike TiaB,
+this tutorial uses ``stratum-bmv2`` to emulate the data plane switches instead
+of ``ovs``.
+
+This tutorial focuses more on Trellis and Stratum integration and therefore
+only demonstrates a **simple 2x2 leaf-spine** topology.
diff --git a/requirements.txt b/requirements.txt
index e4a344f..a65fef2 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,15 +1,15 @@
-Sphinx~=1.8.5
-actdiag~=0.5.4
+Sphinx~=3.1.1
+actdiag~=2.0.0
blockdiag~=1.5.4
-livereload~=2.6.1
-nwdiag~=1.0.4
+doc8~=0.8.0
+nwdiag~=2.0.0
recommonmark~=0.6.0
-rstcheck~=3.3.1
-seqdiag~=0.9.6
-sphinx-reload~=0.2.0
+seqdiag~=2.0.0
sphinx-rtd-theme~=0.4.3
-sphinxcontrib-actdiag~=0.8.5
-sphinxcontrib-blockdiag~=1.5.5
-sphinxcontrib-nwdiag~=0.9.5
-sphinxcontrib-seqdiag~=0.8.5
-git+https://github.com/leokoppel/sphinxcontrib-versioning
+sphinxcontrib-actdiag~=2.0.0
+sphinxcontrib-blockdiag~=2.0.0
+sphinxcontrib-nwdiag~=2.0.0
+sphinxcontrib-openapi~=0.6.0
+sphinxcontrib-seqdiag~=2.0.0
+sphinxcontrib-spelling~=4.3.0
+git+https://github.com/zdw/sphinx-multiversion@ec7c01cdaf2f9241245e84483dfb9bc0d0dffc62
diff --git a/specification.rst b/specification.rst
index cde9dd2..f221380 100644
--- a/specification.rst
+++ b/specification.rst
@@ -4,13 +4,14 @@
SDN Features
============
+
- ONOS cluster of all-active N instances affording N-way redundancy and scale, where N = 3 or N = 5.
- Unified operations interface (GUI/REST/CLI)
- Centralized configuration – all configuration is done on controller instead of each individual switch
- Centralized role-based access control (RBAC)
- Automatic host (end-point) discovery – attached hosts, access-devices, appliances (PNFs), routers, etc.
- based on ARP, DHCP, NDP, etc.
- - Automatic switch, link and topology discovery and maintenance (keep-alives, failure recovery)
+ - Automatic switch, link and topology discovery and maintenance (keepalive, failure recovery)
L2 Features
===========
@@ -145,6 +146,10 @@
Whitebox Switch Software
========================
- Open source ONL, ONIE and Indigo OF client
+
- (in production) OF-DPA software commercial version – contact Broadcom
- - (in labs/trials) OF-DPA software community version available from ONF (for switch models based on Trident and Tomahawk, not Qumran)
- - (in labs/trails) Stratum available from ONF
\ No newline at end of file
+
+ - (in labs/trials) OF-DPA software community version available from ONF (for
+ switch models based on Trident and Tomahawk, not Qumran)
+
+ - (in labs/trails) Stratum available from ONF
diff --git a/supported-hardware.rst b/supported-hardware.rst
index e9d2b34..3f49976 100644
--- a/supported-hardware.rst
+++ b/supported-hardware.rst
@@ -1,48 +1,62 @@
Supported Hardware
==================
-Following is a list of hardware that people from the ONF community have tested over time in lab trials.
+
+Following is a list of hardware that people from the ONF community have tested
+over time in lab trials.
Supported Switches
------------------
1G/10G models (with 40G uplinks)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- - OCP Accepted™ EdgeCore AS5712-54X
- - OCP Accepted™ EdgeCore AS5812-54X
- - QCT QuantaMesh T3048-LY8
- - Delta AG7648
- - Inventec D6254 (verified by Inventec)
+
+- OCP Accepted™ EdgeCore AS5712-54X
+- OCP Accepted™ EdgeCore AS5812-54X
+- QCT QuantaMesh T3048-LY8
+- Delta AG7648
+- Inventec D6254 (verified by Inventec)
25G models (with 100G uplinks)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- - QuantaMesh BMS T7032-IX1/IX1B (with 25G breakout cable)
- - Inventec D7054Q28B (verified by Inventec)
+
+- QuantaMesh BMS T7032-IX1/IX1B (with 25G breakout cable)
+- Inventec D7054Q28B (verified by Inventec)
40G models
^^^^^^^^^^
- - OCP Accepted™ EdgeCore AS6712-32X
+
+- OCP Accepted™ EdgeCore AS6712-32X
100G models
^^^^^^^^^^^
- - OCP Accepted™ EdgeCore AS7712-32X
- - QCT QuantaMesh BMS T7032-IX1/IX1B
- - OCP Accepted™ Inventec D7032Q28B (verified by Inventec)
+
+- OCP Accepted™ EdgeCore AS7712-32X
+- QCT QuantaMesh BMS T7032-IX1/IX1B
+- OCP Accepted™ Inventec D7032Q28B (verified by Inventec)
Commercially supported models
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Models listed here are not supported by ONF community. You should contact switch vendor for commercial support.
- - OCP Accepted™ EdgeCore AS5912-54X
+
+Models listed here are not supported by ONF community. You should contact
+switch vendor for commercial support.
+
+- OCP Accepted™ EdgeCore AS5912-54X
Supported Servers
-----------------
-ONOS can be run on any x86 servers, with the following notes on CPU and memory allocation **per ONOS instance**:
+ONOS can be run on any x86 servers, with the following notes on CPU and memory
+allocation **per ONOS instance**:
- - CPU (Recommended: **32 cores**)
- There are not much requirement on CPU.
- In practice, we avoid running other CPU intensive processes/containers/VMs on the same physical machine where ONOS runs.
+- CPU (Recommended: **32 cores**)
+ There are not much requirement on CPU.
- - Memory (Recommended: **65G RAM**)
- The memory requirement highly depends on the deployment scale.
- In general, we suggest that each ONOS instance should get **at least 16G RAM**.
- For production deployment at scale, we will definitely need not only more memory but also some fine tuning on JVM garbage collection mechanism.
+ In practice, we avoid running other CPU intensive processes/containers/VMs
+ on the same physical machine where ONOS runs.
+
+- Memory (Recommended: **65G RAM**)
+ The memory requirement highly depends on the deployment scale.
+ In general, we suggest that each ONOS instance should get **at least 16G RAM**.
+
+ For production deployment at scale, we will definitely need not only more
+ memory but also some fine tuning on JVM garbage collection mechanism.
diff --git a/supported-topology.rst b/supported-topology.rst
index c7632ab..b482d00 100644
--- a/supported-topology.rst
+++ b/supported-topology.rst
@@ -1,20 +1,25 @@
Supported Topology
******************
-Trellis supports various topologies, from a single switch to a full multi-stage leaf-spine fabric.
+Trellis supports various topologies, from a single switch to a full multi-stage
+leaf-spine fabric.
Simple
======
Single switch
-------------
-This is the minimum Trellis setup. In this setup, all servers are connected to a single switch.
+
+This is the minimum Trellis setup. In this setup, all servers are connected to
+a single switch.
.. image:: images/install-single.png
Paired switches
---------------
-Compared to single switch, it provides more redundancy in terms of server NIC failure and link failure.
+
+Compared to single switch, it provides more redundancy in terms of server NIC
+failure and link failure.
.. image:: images/install-pair.png
@@ -23,20 +28,27 @@
Without paired switches
-----------------------
-Compared to single switch, it offers more redundancy in terms of switch failure and provides better scalability.
+
+Compared to single switch, it offers more redundancy in terms of switch failure
+and provides better scalability.
.. image:: images/install-2x2.png
With paired switches
--------------------
+
It supports all the redundancy and scalability features mentioned above.
.. image:: images/install-2x4.png
Multi-Stage Leaf-Spine
======================
+
Multi-stage is specifically designed for telco service providers.
-The first stage can be installed in the central office, while the second stage can be installed in a field office that is closer to the subscribers.
+
+The first stage can be installed in the central office, while the second stage
+can be installed in a field office that is closer to the subscribers.
+
Two stages are typically connected via long distance optical transport.
- .. image:: images/install-full.png
\ No newline at end of file
+ .. image:: images/install-full.png
diff --git a/testing.rst b/testing.rst
index 6213b74..ccca1d9 100644
--- a/testing.rst
+++ b/testing.rst
@@ -4,19 +4,30 @@
We use two open source test tools to test Trellis behavior.
- **Data Plane Testing - OFTest**
- We use **OFTest** to verify the **behavior of a hardware switch** is the same as described in the OF-DPA specification.
+
+ We use **OFTest** to verify the **behavior of a hardware switch** is the
+ same as described in the OF-DPA specification.
+
This tool was originally developed by Big Switch and later modified by Accton.
+
The test cases for verifying OF-DPA behavior were written by ONF.
- To learn more about OFTest and how we use it for OF-DPA verification, please visit :doc:`Data Plane Testing <testing/data-plane-testing>`.
+
+ To learn more about OFTest and how we use it for OF-DPA verification,
+ please visit :doc:`Data Plane Testing <testing/data-plane-testing>`.
- **Control Plane Testing - TestON**
- We use **TestON** to verify the **Trellis control application behavior** when used for a leaf-spine fabric.
+
+ We use **TestON** to verify the **Trellis control application behavior**
+ when used for a leaf-spine fabric.
+
These tests are merged into ONOS test repository and run nightly.
- To learn more about TestON, please visit :doc:`Control Plane Testing <testing/control-plane-testing>`.
+
+ To learn more about TestON, please visit :doc:`Control Plane Testing
+ <testing/control-plane-testing>`.
.. toctree::
:maxdepth: 1
:hidden:
testing/data-plane-testing
- testing/control-plane-testing
\ No newline at end of file
+ testing/control-plane-testing
diff --git a/testing/control-plane-testing.rst b/testing/control-plane-testing.rst
index 7499296..4e6a8ac 100644
--- a/testing/control-plane-testing.rst
+++ b/testing/control-plane-testing.rst
@@ -6,11 +6,13 @@
Setup ONOS Cluster
------------------
-Follow instructions in `Install ONOS Controller <../installation.html#install-controller-onos>`_ to setup and run ONOS cluster
+Follow instructions in `Install ONOS Controller
+<../installation.html#install-controller-onos>`_ to setup and run ONOS cluster
Setup Mininet Environment
-------------------------
+
- Clone Mininet and run the install script
.. code-block:: console
@@ -23,39 +25,51 @@
$ cd util
$ sudo ./install.sh -3fvn # Install OpenFlow 1.3, Open Vswitch and Mininet dependencies
-- Follow `Trellis in a Box <https://github.com/opennetworkinglab/routing/tree/master/trellis>`_ to install Trellis dependencies, such as DHCP server and Quagga.
+- Follow `Trellis in a Box
+ <https://github.com/opennetworkinglab/routing/tree/master/trellis>`_ to
+ install Trellis dependencies, such as DHCP server and Quagga.
.. note::
- If DHCP server does not properly run because of a permission error, please see `this <https://help.ubuntu.com/community/ isc-dhcp-server#Permission_issues_with_ISC-DHCP_server>`_ to solve the problem.
+ If DHCP server does not properly run because of a permission error,
+ please see `this
+ <https://help.ubuntu.com/community/isc-dhcp-server#Permission_issues_with_ISC-DHCP_server>`_
+ to solve the problem.
-- Set environment variables as described in `Create a Cell File <../installation.html#create-a-cell-file>`_
+- Set environment variables as described in `Create a Cell File
+ <../installation.html#create-a-cell-file>`_
Setup Test Station
------------------
-- Follow `TestON installation guide <https://wiki.onosproject.org/display/ONOS/Installation>`_ to install TestON
+
+- Follow `TestON installation guide
+ <https://wiki.onosproject.org/display/ONOS/Installation>`_ to install TestON
+
- (Optional) Add ``OnosSystemTest/TestON/bin`` to ``$PATH`` for convenience
Run Tests
=========
-Before you run any test, it is recommended to run cleanup script to kill any TestON, ssh, and Mininet sessions that are running.
+Before you run any test, it is recommended to run cleanup script to kill any
+TestON, ssh, and Mininet sessions that are running.
.. code-block:: console
cd ~/OnosSystemTest/TestON/bin
./cleanup.sh
-Fabric-related tests are located in ``OnosSystemTest/TestON/tests/USECASE/SegmentRouting/``.
-Each individual test can be run by executing ``cli.py`` script with test name. For example:
+Fabric-related tests are located in
+``OnosSystemTest/TestON/tests/USECASE/SegmentRouting/``.
+
+Each individual test can be run by executing ``cli.py`` script with test name.
+For example:
.. code-block:: console
cd ~/OnosSystemTest/TestON/bin
./cli.py run SRSanity
-
Test Plan
=========
@@ -70,9 +84,10 @@
- L1+L2 and L3+L4 are paired switches.
- All spine links with L1-4 are double links.
- - Hosts take ips with dhcp and there exist 2 external routers.
+ - Hosts take IPs with dhcp and there exist 2 external routers.
- Hosts can use ipv4 or ipv6.
- - We configure untagged/tagged single/dual home hosts in order to test every possible combination.
+ - We configure untagged/tagged single/dual home hosts in order to test every
+ possible combination.
- Most of the hosts reside in different ip subnets.
Tests Steps
@@ -84,8 +99,8 @@
- pingall
- (Optional) ``ping 10.0.99.2`` to check external IPv4 connectivity
- (Optional) ``ping6 2000:9902`` to check external IPv6 connectivity
-- Additional actions and tests defined in each individual test case
+- Additional actions and tests defined in each individual test case
Test Cases
----------
@@ -97,4 +112,7 @@
Test Results
============
ONF runs all Mininet based Trellis tests nightly.
-Test results are automatically published `here <https://wiki.onosproject.org/display/ONOS/1.12-Segment+Routing>`_
\ No newline at end of file
+
+Test results are automatically published `here
+<https://wiki.onosproject.org/display/ONOS/1.12-Segment+Routing>`_
+
diff --git a/testing/data-plane-testing.rst b/testing/data-plane-testing.rst
index 9c194db..75c5a14 100644
--- a/testing/data-plane-testing.rst
+++ b/testing/data-plane-testing.rst
@@ -4,12 +4,19 @@
OFTest Overview
===============
-The oftest framework consists of a test server connected to a target switch through an OpenFlow channel and directly to the data plane ports of the OF switch.
-After programming the switch, oftest inserts packets in the data plane and verifies if the output is as expected.
+The oftest framework consists of a test server connected to a target switch
+through an OpenFlow channel and directly to the data plane ports of the OF
+switch.
+
+After programming the switch, oftest inserts packets in the data plane and
+verifies if the output is as expected.
.. image:: ../images/testing-oftest.png
-We recommend using a server connected to a management network, a switch connected to the same management network, and then a **minimum of 2 ports directly connected between the server and the switch**.
+We recommend using a server connected to a management network, a switch
+connected to the same management network, and then a **minimum of 2 ports
+directly connected between the server and the switch**.
+
A few tests will fail if you have less than 3 ports.
OFTest Setup
@@ -27,10 +34,15 @@
$ git clone https://gerrit.opencord.org/fabric-oftest
Now let's configure the switch and point it to the test server.
-It can be done in the same way as connecting the switch to any controller.
-See `Connect Switch to Controller <../installation.html#connect-switch-to-controller>`_ for more details.
-Assuming that you plugged the ports ``12`` and ``24`` of the switch to interfaces ``eth1`` and ``eth2`` of the server, respectively, then the following command can tell you if everything is working.
+It can be done in the same way as connecting the switch to any controller.
+
+See `Connect Switch to Controller
+<../installation.html#connect-switch-to-controller>`_ for more details.
+
+Assuming that you plugged the ports ``12`` and ``24`` of the switch to
+interfaces ``eth1`` and ``eth2`` of the server, respectively, then the
+following command can tell you if everything is working.
.. code-block:: console
@@ -69,15 +81,16 @@
flows.PacketInSrcMacMiss ... FAIL
flows.PacketInUDP ... ok
-For example, in this case, all the multicast failed because we don't have three ports, the test L3VPN failed as well, and that is a bug of the current build.
+For example, in this case, all the multicast failed because we don't have three
+ports, the test L3VPN failed as well, and that is a bug of the current build.
The **PacketInSrcMac** Miss failed because Mac learning is not enabled.
.. note::
Expected test results can be found in https://github.com/opencord/fabric-oftest/blob/master/README.md
.. tip::
- OFTest can't successfully delete all groups using an OF message, because of that some tests interfere with others.
- If the results seem suspicious, **erase the flow tables and group tables** and run the test alone.
+ OFTest can't successfully delete all groups using an OF message, because of
+ that some tests interfere with others.
-
-
+ If the results seem suspicious, **erase the flow tables and group tables**
+ and run the test alone.
diff --git a/troubleshooting.rst b/troubleshooting.rst
index 4412a28..0a50212 100644
--- a/troubleshooting.rst
+++ b/troubleshooting.rst
@@ -4,10 +4,16 @@
Overview
========
-The **Trellis Troubleshooting Tools (T3)** is an ONOS application intended to **help you troubleshoot traffic-related problems** in your network by **tracing installed flows and groups**.
-T3 works both on simulated OVS-based topologies but also on real HW, basing it's internal decisions on the driver assigned to the device.
-T3 also currently supports dual links between leaf and spine and properly handles the difference between deferred and immediate rules.
-Dual Homed hosts are also supported in T3.
+The **Trellis Troubleshooting Tools (T3)** is an ONOS application intended to
+**help you troubleshoot traffic-related problems** in your network by **tracing
+installed flows and groups**.
+
+T3 works both on simulated OVS-based topologies but also on real HW, basing
+it's internal decisions on the driver assigned to the device.
+
+T3 also currently supports dual links between leaf and spine and properly
+handles the difference between deferred and immediate rules. Dual Homed hosts
+are also supported in T3.
To activate this tool on your ONOS environment, you need to run:
@@ -28,8 +34,11 @@
t3-troubleshoot
===============
+
As of right now the tools provides a CLI command (troubleshoot).
-As arguments to the CLI command you have to pass arguments describing the packet you want to trace through the network.
+
+As arguments to the CLI command you have to pass arguments describing the
+packet you want to trace through the network.
t3-troubleshoot Help
--------------------
@@ -235,7 +244,8 @@
onos> t3-troubleshoot -vv -s 10.0.2.1 -sp of:0000000000000204/3 -sm 00:AA:00:00:00:01 -et IPV4 -d 10.0.3.1 -dm 00:00:00:00:02:04 -vid None
-This verbosity level prints all the flows and groups that are met along the path
+This verbosity level prints all the flows and groups that are met along the
+path:
.. code-block:: console
@@ -368,7 +378,9 @@
Successful Example
------------------
-In this example we trace a packet that should be bridged between two hosts in the same subnet
+
+In this example we trace a packet that should be bridged between two hosts in
+the same subnet
.. code-block:: console
@@ -381,9 +393,13 @@
Failed Example
--------------
+
Obviously failures can happen along one of the paths that the packet can take.
-In our sample we removed a flow in ``of:0000000000000226`` thus disabling that possible path for our initial packet.
-The output of our trace would then look something like
+
+In our sample we removed a flow in ``of:0000000000000226`` thus disabling that
+possible path for our initial packet.
+
+The output of our trace would then look something like:
.. code-block:: console
@@ -396,7 +412,8 @@
Reached required destination Host
No output out of device of:0000000000000226 Packet is dropped
-Notice that on the second path that was previously going through ``of:0000000000000226`` now the packet is dropped due to the rule we removed.
+Notice that on the second path that was previously going through
+``of:0000000000000226`` now the packet is dropped due to the rule we removed.
Example of Various Packet Types
-------------------------------
@@ -453,9 +470,15 @@
- LLDP from CONTROLLER
T3 is also capable of tracing packets coming in from the CONTROLLER.
- We use the same input port with the possibility to give CONTROLLER as logical port number.
- This is a mere convenience to carry the information inside T3 and do the proper flooding to all the ports of the given device.
- This case is not treated like all the rest and the input port here is a logical entity that has no match on any physical element on the device.
+
+ We use the same input port with the possibility to give CONTROLLER as logical
+ port number.
+
+ This is a mere convenience to carry the information inside T3 and do the
+ proper flooding to all the ports of the given device.
+
+ This case is not treated like all the rest and the input port here is a
+ logical entity that has no match on any physical element on the device.
.. code-block:: console
@@ -472,8 +495,15 @@
t3-troubleshoot-simple
======================
-T3 also offers a simplified version of the command to be able to ping between two given hosts without having to specify all the information.
-This commands bases itself on top of some configuration present in ONOS through netcfg, namely the VLAN interface configuration and the segment routing leaf router MAC for L3 Unicast. If that is not present the user needs to input this information manually recurring to the the previously shown command. Same verbosity levels and output as the troubleshoot command are possible.
+
+T3 also offers a simplified version of the command to be able to ping between
+two given hosts without having to specify all the information.
+
+This commands bases itself on top of some configuration present in ONOS through
+netcfg, namely the VLAN interface configuration and the segment routing leaf
+router MAC for L3 Unicast. If that is not present the user needs to input this
+information manually recurring to the the previously shown command. Same
+verbosity levels and output as the troubleshoot command are possible.
t3-troubleshoot-simple Help
---------------------------
@@ -575,8 +605,10 @@
t3-troubleshoot-mcast
=====================
-T3 also offers a mcast command queries the system for all the multicast routes given a certain vlanId.
-It prints all the traces given a certain group, source and source connect point. It has two "standard" verbosity levels.
+
+T3 also offers a mcast command queries the system for all the multicast routes
+given a certain vlanId. It prints all the traces given a certain group, source
+and source connect point. It has two "standard" verbosity levels.
t3-troubleshoot-mcast Help
--------------------------
@@ -614,8 +646,13 @@
<ONOS_IP>:8181/onos/v1/docs/#/t3
The rest API offered are a non verbose and a verbose version of the troubleshoot simple command.
-It bases itself on top of some configuration present in ONOS through netcfg, namely the VLAN interface configuration and the segment routing leaf router MAC for L3 Unicast.
-The use need to specify just the Source Host, the destination host and the ETH type.
+
+It bases itself on top of some configuration present in ONOS through netcfg,
+namely the VLAN interface configuration and the segment routing leaf router MAC
+for L3 Unicast.
+
+The use need to specify just the Source Host, the destination host and the ETH
+type.
.. code-block:: console
@@ -629,14 +666,24 @@
Last Resort
===========
-In the case where you can't figure out what is going wrong even with help of T3, you can seek help on Trellis developer mailing list ``trellis-dev@opennetworking.org``.
+In the case where you can't figure out what is going wrong even with help of
+T3, you can seek help on Trellis developer mailing list
+``trellis-dev@opennetworking.org``.
+
There are a few things we would like you to attach:
- **Issue description**
-- **Environment description**, such as ONOS version, switch model and OF-DPA version
+
+- **Environment description**, such as ONOS version, switch model and OF-DPA
+ version
+
- **Steps of reproduction**, as detail as possible
+
- **Diagnostics**.
- We have built a tool ``onos-diagnostics`` (see `ONOS Remote Admin Tools <https://wiki.onosproject.org/display/ONOS/ONOS+Remote+Admin+Tools>`_) to help you easily collect and package ONOS diagnostics
+
+ We have built a tool ``onos-diagnostics`` (see `ONOS Remote Admin Tools
+ <https://wiki.onosproject.org/display/ONOS/ONOS+Remote+Admin+Tools>`_) to
+ help you easily collect and package ONOS diagnostics
Reference