Update Sphinx version and versioning process
- Updated to newer Sphinx version
- Changed to use doc8 to check .rst files
- Added a dictionary (dict.txt) and spellchecking
- Reformatted and fixed issues that were found in the content
Change-Id: If7b35e01ee8be25dbbd1ecd0e67b264aa6cc2a94
diff --git a/configuration/dual-homing.rst b/configuration/dual-homing.rst
index 3e5f729..318dc02 100644
--- a/configuration/dual-homing.rst
+++ b/configuration/dual-homing.rst
@@ -8,31 +8,36 @@
The dual-homing feature includes several sub components
-- **Use of "paired" ToRs**: Each rack of compute nodes have exactly two Top-of-Rack switches (ToRs),
- that are linked to each other via a single link - such a link is referred to as a **pair link**.
- This pairing should NOT be omitted.
- Currently there is support for only a single link between paired ToRs.
- In future releases, we may include dual pair links.
- Note that the pair link is only used in failure scenarios, and not in normal operation.
+- **Use of "paired" ToRs**: Each rack of compute nodes have exactly two
+ Top-of-Rack switches (ToRs), that are linked to each other via a single link
+ - such a link is referred to as a **pair link**. This pairing should NOT be
+ omitted.
-- **Dual-homed servers (compute-nodes)**: Each server is connected to both ToRs.
- The links to the paired ToRs are (Linux) bonded
+ Currently there is support for only a single link between paired ToRs. In
+ future releases, we may include dual pair links. Note that the pair link is
+ only used in failure scenarios, and not in normal operation.
-- **Dual-homed upstream routers**: The upstream routers MUST be connected to the two ToRs that are part of a leaf-pair.
- You cannot connect them to leafs that are not paired. This feature also requires two Quagga instances.
+- **Dual-homed servers (compute-nodes)**: Each server is connected to both
+ ToRs. The links to the paired ToRs are (Linux) bonded
+
+- **Dual-homed upstream routers**: The upstream routers MUST be connected to
+ the two ToRs that are part of a leaf-pair. You cannot connect them to leafs
+ that are not paired. This feature also requires two Quagga instances.
- **Dual-homed access devices**. This component will be added in the future.
Paired ToRs
-----------
-The reasoning behind two ToR (leaf) switches is simple.
-If you only have a single ToR switch, and you lose it, the entire rack goes down.
-Using two ToR switches increases your odds for continued connectivity for dual homed servers.
-The reasoning behind pairing the two ToR switches is more involved, as is explained in the Usage section below.
+The reasoning behind two ToR (leaf) switches is simple. If you only have a
+single ToR switch, and you lose it, the entire rack goes down. Using two ToR
+switches increases your odds for continued connectivity for dual homed servers.
+The reasoning behind pairing the two ToR switches is more involved, as is
+explained in the Usage section below.
Configure pair ToRs
^^^^^^^^^^^^^^^^^^^
-Configuring paired-ToRs involves device configuration. Assume switches of:205 and of:206 are paired ToRs.
+Configuring paired-ToRs involves device configuration. Assume switches of:205
+and of:206 are paired ToRs.
.. code-block:: json
@@ -71,15 +76,21 @@
There are two new pieces of device configuration.
-Each device in the ToR pair needs to specify the **deviceId of the leaf it is paired to**, in the ``pairDeviceId`` field.
-For example, in of:205 configuration the pairDeviceid is specified as of:206, and similarly in of:206 configuration the pairDeviceId is of:205
-Each device in the ToR pair needs to specify the **port on the device used for the pair link** in the ``pairLocalPort`` field.
-For example, the pair link in the config above show that port 20 on of:205 is connected to port 30 on of:206.
+Each device in the ToR pair needs to specify the **deviceId of the leaf it is
+paired to**, in the ``pairDeviceId`` field. For example, in ``of:205``
+configuration the ``pairDeviceId`` is specified as ``of:206``, and similarly in ``of:206``
+configuration the ``pairDeviceId`` is ``of:205``. Each device in the ToR pair needs to
+specify the **port on the device used for the pair link** in the
+``pairLocalPort`` field. For example, the pair link in the config above show
+that port 20 on of:205 is connected to port 30 on of:206.
-In addition, there is one crucial piece of config that needs to **match for both ToRs** – the ``routerMac`` address.
-The paired-ToRs MUST have the same routerMac - in the example above, they both have identical 00:00:02:05:00:01 routerMacs.
+In addition, there is one crucial piece of config that needs to **match for
+both ToRs** – the ``routerMac`` address. The paired-ToRs MUST have the same
+routerMac - in the example above, they both have identical 00:00:02:05:00:01
+routerMacs.
-All other fields are the same as before, as explained in :doc:`Device Configuration <device-config>` section.
+All other fields are the same as before, as explained in :doc:`Device
+Configuration <device-config>` section.
Usage of pair link
@@ -90,19 +101,26 @@
Dual-Homed Servers
------------------
+
There are a number of things to note when connecting dual-homed servers to paired-ToRs.
-- The switch ports on the two ToRs have to be configured the same way, when connecting a dual-homed server to the two ToRs.
+- The switch ports on the two ToRs have to be configured the same way, when
+ connecting a dual-homed server to the two ToRs.
+
- The server ports have to be Linux-bonded in a particular mode.
Configure Switch Ports
^^^^^^^^^^^^^^^^^^^^^^
-The way to configure ports are similar as described in :doc:`Bridging and Unicast <bridging-unicast>`.
-However, there are a couple of things to note.
-**First**, dual-homed servers should have the **identical configuration on each switch port they connect to on the ToR pairs**.
-The example below shows that the ``vlans`` and ``ips`` configured are the same on both switch ports ``of:205/12`` and ``of:206/29``.
-They are both configured to be access ports in ``VLAN 20``, the subnet ``10.0.2.0/24`` is assigned to these ports, and the gateway-IP is ``10.0.2.254/32``.
+The way to configure ports are similar as described in :doc:`Bridging and
+Unicast <bridging-unicast>`. However, there are a couple of things to note.
+
+**First**, dual-homed servers should have the **identical configuration on each
+switch port they connect to on the ToR pairs**. The example below shows that
+the ``vlans`` and ``ips`` configured are the same on both switch ports
+``of:205/12`` and ``of:206/29``. They are both configured to be access ports
+in ``VLAN 20``, the subnet ``10.0.2.0/24`` is assigned to these ports, and the
+gateway-IP is ``10.0.2.254/32``.
.. code-block:: json
@@ -125,17 +143,24 @@
}
}
-It is worth noting the meaning behind the configuration above from a routing perspective.
-Simply put, by configuring the same subnets on these switch ports, the fabric now believes that the entire subnet ``10.0.2.0/24`` is reachable by BOTH ToR switches ``of:205`` and ``of:206``.
+It is worth noting the meaning behind the configuration above from a routing
+perspective. Simply put, by configuring the same subnets on these switch
+ports, the fabric now believes that the entire subnet ``10.0.2.0/24`` is
+reachable by BOTH ToR switches ``of:205`` and ``of:206``.
.. caution::
- Configuring different VLANs, or different subnets, or mismatches like "vlan-untagged" in one switch port and "vlan-tagged" in the corresponding switch port facing the dual-homed server, will result in incorrect behavior.
+ Configuring different VLANs, or different subnets, or mismatches like
+ "vlan-untagged" in one switch port and "vlan-tagged" in the corresponding
+ switch port facing the dual-homed server, will result in incorrect
+ behavior.
-**Second**, we need to configure the **pair link ports on both ToR switches to be trunk (vlan-tagged) ports that contains all dual-homed VLANs and subnets**.
-This is an extra piece of configuration, the need for which will be removed in future releases.
-In the example above, a dual-homed server connects to the ToR pair on port 12 on of:205 and port 29 on of:206.
-Assume that the pair link between the two ToRs is connected to port 5 of both of:205 and of:206.
-The config for these switch ports is shown below:
+**Second**, we need to configure the **pair link ports on both ToR switches to
+be trunk (vlan-tagged) ports that contains all dual-homed VLANs and subnets**.
+This is an extra piece of configuration, the need for which will be removed in
+future releases. In the example above, a dual-homed server connects to the ToR
+pair on port 12 on of:205 and port 29 on of:206. Assume that the pair link
+between the two ToRs is connected to port 5 of both of:205 and of:206. The
+config for these switch ports is shown below:
.. code-block:: json
@@ -159,13 +184,18 @@
}
.. note::
- Even though the ports ``of:205/12`` and ``of:206/`` facing the dual-homed server are configured as ``vlan-untagged``,
- the same vlan MUST be configured as ``vlan-tagged`` on the pair-ports.
- If additional subnets and VLANs are configured facing other dual-homed servers, they need to be similarly added to the ``ips`` and ``vlan-tagged`` arrays in the pair port config.
+ Even though the ports ``of:205/12`` and ``of:206/`` facing the dual-homed
+ server are configured as ``vlan-untagged``, the same vlan MUST be
+ configured as ``vlan-tagged`` on the pair-ports.
+
+ If additional subnets and VLANs are configured facing other dual-homed
+ servers, they need to be similarly added to the ``ips`` and ``vlan-tagged``
+ arrays in the pair port config.
Configure Servers
^^^^^^^^^^^^^^^^^
+
Assuming the interfaces we are going to use for bonding are ``eth1`` and ``eth2``.
- Bring down interfaces
@@ -235,19 +265,24 @@
.. caution::
**Dual-homed host should not be statically configured.**
- Currently in ONOS, configured hosts are not updated when the connectPoint is lost.
- This is not a problem with single-homed hosts because there is no other way to reach them anyway if their connectPoint goes down.
- But in dual-homed scenarios, the controller should take corrective action if one of the connectPoints go down –
- the trigger for this event does not happen when the dual-homed host's connect points are configured (not discovered).
+ Currently in ONOS, configured hosts are not updated when the connectPoint
+ is lost. This is not a problem with single-homed hosts because there is no
+ other way to reach them anyway if their connectPoint goes down. But in
+ dual-homed scenarios, the controller should take corrective action if one
+ of the connectPoints go down – the trigger for this event does not happen
+ when the dual-homed host's connect points are configured (not discovered).
.. note::
- We also support static routes with dual-homed next hop.
- The way to configure it is exactly the same as regular single-homed next hop, as described in :doc:`External Connectivity <external-connectivity>`.
+ We also support static routes with dual-homed next hop. The way to
+ configure it is exactly the same as regular single-homed next hop, as
+ described in :doc:`External Connectivity <external-connectivity>`.
- ONOS will automatically recognize when the next-hop IP resolves to a dual-homed host and program both switches (the host connects to) accordingly.
+ ONOS will automatically recognize when the next-hop IP resolves to a
+ dual-homed host and program both switches (the host connects to)
+ accordingly.
- The failure recovery mechanism for dual-homed hosts also applies to static routes that point to the host as their next hop.
-
+ The failure recovery mechanism for dual-homed hosts also applies to static
+ routes that point to the host as their next hop.
Dual External Routers
---------------------
@@ -257,22 +292,25 @@
.. image:: ../images/config-dh-vr-logical.png
:width: 200px
-In addition to what we describe in :doc:`External Connectivity <external-connectivity>`,
-Trellis also supports dual external routers, which view the Trellis fabric as 2 individual routers, as shown above.
+In addition to what we describe in :doc:`External Connectivity
+<external-connectivity>`, Trellis also supports dual external routers, which
+view the Trellis fabric as 2 individual routers, as shown above.
As before the vRouter control plane is implemented as a combination of Quagga,
-which peers with the upstream routers, and ONOS which listens to Quagga (via FPM) and programs the underlying fabric.
-**In dual-router scenarios, there are two instances of Quagga required**.
+which peers with the upstream routers, and ONOS which listens to Quagga (via
+FPM) and programs the underlying fabric. **In dual-router scenarios, there are
+two instances of Quagga required**.
-As before the hardware fabric serves as the data-plane of vRouter.
-In dual-router scenarios, the **external routers MUST be connected to paired-ToRs**.
-
+As before the hardware fabric serves as the data-plane of vRouter. In
+dual-router scenarios, the **external routers MUST be connected to
+paired-ToRs**.
ToR connects to one upstream
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Lets consider the simpler case where the external routers are each connected to a single leaf in a ToR pair.
-The figure on the left below shows the logical view. The figure on the right shows the physical connectivity.
+Lets consider the simpler case where the external routers are each connected to
+a single leaf in a ToR pair. The figure on the left below shows the logical
+view. The figure on the right shows the physical connectivity.
.. image:: ../images/config-dh-vr-logical-simple.png
:width: 200px
@@ -280,19 +318,22 @@
.. image:: ../images/config-dh-vr-physical-simple.png
:width: 400px
-One of the upstream routers is connected to ``of:205`` and the other is connected to ``of:206``.
-Note that ``of:205`` and ``of:206`` are paired ToRs.
+One of the upstream routers is connected to ``of:205`` and the other is
+connected to ``of:206``. Note that ``of:205`` and ``of:206`` are paired ToRs.
-The ToRs are connected via a physical port to separate Quagga VMs or containers.
-These Quagga instances can be placed in any compute node. They do not need to be in the same server, and are only shown to be co-located for simplicity.
+The ToRs are connected via a physical port to separate Quagga VMs or
+containers. These Quagga instances can be placed in any compute node. They do
+not need to be in the same server, and are only shown to be co-located for
+simplicity.
-The two Quagga instances do NOT talk to each other.
-
+The two Quagga instances do NOT talk to each other.
Switch port configuration
"""""""""""""""""""""""""
-The ToRs follow the same rules as single router case described in :doc:`External Connectivity <external-connectivity>`.
-In the example shown above, the switch port config would look like this:
+
+The ToRs follow the same rules as single router case described in
+:doc:`External Connectivity <external-connectivity>`. In the example shown
+above, the switch port config would look like this:
.. code-block:: json
@@ -333,35 +374,51 @@
}
.. note::
- In the example shown above, switch ``of:205`` uses ``VLAN 100`` for bridging the peering session between Quagga1 and ExtRouter1,
- while switch ``of:205`` uses ``VLAN 200`` to do the same for the other peering session.
- But since these vlans and bridging domains are defined on different switches, the VLAN ids could have been the same.
+ In the example shown above, switch ``of:205`` uses ``VLAN 100`` for
+ bridging the peering session between Quagga1 and ExtRouter1, while switch
+ ``of:205`` uses ``VLAN 200`` to do the same for the other peering session.
+ But since these vlans and bridging domains are defined on different
+ switches, the VLAN ids could have been the same.
- This philosophy is consistent with the fabric use of :doc:`bridging <bridging-unicast>`.
+ This philosophy is consistent with the fabric use of :doc:`bridging
+ <bridging-unicast>`.
Quagga configuration
""""""""""""""""""""
-Configuring Quagga for dual external routers are similar to what we described in :doc:`External Connectivity <external-connectivity>`. However, it is worth noting that:
+Configuring Quagga for dual external routers are similar to what we described
+in :doc:`External Connectivity <external-connectivity>`. However, it is worth
+noting that:
-- The two Zebra instances **should point to two different ONOS instances** for their FPM connections.
- For example Zebra in Quagga1 could point to ONOS instance with ``fpm connection ip 10.6.0.1 port 2620``,
- while the other Zebra should point to a different ONOS instance with ``fpm connection ip 10.6.0.2 port 2620``.
- It does not matter which ONOS instances they point to as long as they are different.
-- The two Quagga BGP sessions should appear to come from different routers but still use the same AS number –
- i.e. the two Quaggas' belong to the same AS, the one used to represent the entire Trellis infrastructure.
-- The two upstream routers can belong to the same or different AS,
- but these AS numbers should be different from the one used to represent the Trellis AS.
+- The two Zebra instances **should point to two different ONOS instances** for
+ their FPM connections. For example Zebra in Quagga1 could point to ONOS
+ instance with ``fpm connection ip 10.6.0.1 port 2620``, while the other Zebra
+ should point to a different ONOS instance with ``fpm connection ip 10.6.0.2
+ port 2620``. It does not matter which ONOS instances they point to as long
+ as they are different.
+
+- The two Quagga BGP sessions should appear to come from different routers but
+ still use the same AS number – i.e. the two Quaggas' belong to the same AS,
+ the one used to represent the entire Trellis infrastructure.
+
+- The two upstream routers can belong to the same or different AS, but these AS
+ numbers should be different from the one used to represent the Trellis AS.
+
- Typically both Quagga instances advertise the same routes to the upstream.
- These prefixes belonging to various infrastructure nodes in the deployment should be reachable from either of the leaf switches connected to the upstream routers.
-- The upstream routers may or may not advertise the same routes.
- Trellis will ensure that traffic directed to a route reachable only one upstream router is directed to the appropriate leaf.
+ These prefixes belonging to various infrastructure nodes in the deployment
+ should be reachable from either of the leaf switches connected to the
+ upstream routers.
+- The upstream routers may or may not advertise the same routes. Trellis will
+ ensure that traffic directed to a route reachable only one upstream router is
+ directed to the appropriate leaf.
ToR connects to both upstream
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Now lets consider the **more-complicated but more fault-tolerant** case of each Quagga instance peering with BOTH external routers.
-Again the logical view is shown on the left and the physical view on the right.
+
+Now lets consider the **more-complicated but more fault-tolerant** case of each
+Quagga instance peering with BOTH external routers. Again the logical view is
+shown on the left and the physical view on the right.
.. image:: ../images/config-dh-vr-logical.png
:width: 200px
@@ -379,19 +436,24 @@
- Quagga instance 2 peers with external router R1 via port 2 on switch of:206
- Quagga instance 2 peers with external router R2 via port 1 on switch of:206
-To distinguish between the two peering sessions in the same physical switch, say of:205,
-the physical ports 1 and 2 need to be configured in **different VLANs and subnets**.
-For example, port 1 on of:205 is (untagged) in VLAN 100, while port 2 is in VLAN 101.
+To distinguish between the two peering sessions in the same physical switch,
+say of:205, the physical ports 1 and 2 need to be configured in **different
+VLANs and subnets**. For example, port 1 on of:205 is (untagged) in VLAN 100,
+while port 2 is in VLAN 101.
-Note that peering for **Quagga1 and R1** happens with IPs in the ``10.0.100.0/29`` subnet,
-and for **Quagga 1 and R2** in the **10.0.101.0/29** subnet.
+Note that peering for **Quagga1 and R1** happens with IPs in the
+``10.0.100.0/29`` subnet, and for **Quagga 1 and R2** in the **10.0.101.0/29**
+subnet.
-Furthermore, **pair link** (port 48) on of:205 carries both peering sessions to Quagga1.
-Thus port 48 should now be configured as a **trunk port (vlan-tagged) with both VLANs and both subnets**.
+Furthermore, **pair link** (port 48) on of:205 carries both peering sessions to
+Quagga1. Thus port 48 should now be configured as a **trunk port (vlan-tagged)
+with both VLANs and both subnets**.
-Finally the **Quagga interface** on the VM now needs **sub-interface configuration for each VLAN ID**.
+Finally the **Quagga interface** on the VM now needs **sub-interface
+configuration for each VLAN ID**.
-Similar configuration concepts apply to IPv6 as well. Here is a look at the switch port config in ONOS for of:205
+Similar configuration concepts apply to IPv6 as well. Here is a look at the
+switch port config in ONOS for of:205
.. code-block:: json