Added dual homing
Change-Id: I541b898fc158749a9c8ce5e00a40ec222c9b098f
diff --git a/configuration/dual-homing.rst b/configuration/dual-homing.rst
index 367152c..3e5f729 100644
--- a/configuration/dual-homing.rst
+++ b/configuration/dual-homing.rst
@@ -1,2 +1,425 @@
Dual Homing
===========
+
+Overview
+--------
+
+.. image:: ../images/config-dh.png
+
+The dual-homing feature includes several sub components
+
+- **Use of "paired" ToRs**: Each rack of compute nodes have exactly two Top-of-Rack switches (ToRs),
+ that are linked to each other via a single link - such a link is referred to as a **pair link**.
+ This pairing should NOT be omitted.
+ Currently there is support for only a single link between paired ToRs.
+ In future releases, we may include dual pair links.
+ Note that the pair link is only used in failure scenarios, and not in normal operation.
+
+- **Dual-homed servers (compute-nodes)**: Each server is connected to both ToRs.
+ The links to the paired ToRs are (Linux) bonded
+
+- **Dual-homed upstream routers**: The upstream routers MUST be connected to the two ToRs that are part of a leaf-pair.
+ You cannot connect them to leafs that are not paired. This feature also requires two Quagga instances.
+
+- **Dual-homed access devices**. This component will be added in the future.
+
+Paired ToRs
+-----------
+The reasoning behind two ToR (leaf) switches is simple.
+If you only have a single ToR switch, and you lose it, the entire rack goes down.
+Using two ToR switches increases your odds for continued connectivity for dual homed servers.
+The reasoning behind pairing the two ToR switches is more involved, as is explained in the Usage section below.
+
+Configure pair ToRs
+^^^^^^^^^^^^^^^^^^^
+Configuring paired-ToRs involves device configuration. Assume switches of:205 and of:206 are paired ToRs.
+
+.. code-block:: json
+
+ {
+ "devices" : {
+ "of:0000000000000205" : {
+ "segmentrouting" : {
+ "name" : "Leaf1-R2",
+ "ipv4NodeSid" : 205,
+ "ipv4Loopback" : "192.168.0.205",
+ "ipv6NodeSid" : 205,
+ "ipv6Loopback" : "2000::c0a8:0205",
+ "routerMac" : "00:00:02:05:00:01",
+ "pairDeviceId" : "of:0000000000000206",
+ "pairLocalPort" : 20,
+ "isEdgeRouter" : true,
+ "adjacencySids" : []
+ }
+ },
+ "of:0000000000000206" : {
+ "segmentrouting" : {
+ "name" : "Leaf2-R2",
+ "ipv4NodeSid" : 206,
+ "ipv4Loopback" : "192.168.0.206",
+ "ipv6NodeSid" : 206,
+ "ipv6Loopback" : "2000::c0a8:0206",
+ "routerMac" : "00:00:02:05:00:01",
+ "pairDeviceId" : "of:0000000000000205",
+ "pairLocalPort" : 30,
+ "isEdgeRouter" : true,
+ "adjacencySids" : []
+ }
+ }
+ }
+ }
+
+There are two new pieces of device configuration.
+
+Each device in the ToR pair needs to specify the **deviceId of the leaf it is paired to**, in the ``pairDeviceId`` field.
+For example, in of:205 configuration the pairDeviceid is specified as of:206, and similarly in of:206 configuration the pairDeviceId is of:205
+Each device in the ToR pair needs to specify the **port on the device used for the pair link** in the ``pairLocalPort`` field.
+For example, the pair link in the config above show that port 20 on of:205 is connected to port 30 on of:206.
+
+In addition, there is one crucial piece of config that needs to **match for both ToRs** – the ``routerMac`` address.
+The paired-ToRs MUST have the same routerMac - in the example above, they both have identical 00:00:02:05:00:01 routerMacs.
+
+All other fields are the same as before, as explained in :doc:`Device Configuration <device-config>` section.
+
+
+Usage of pair link
+^^^^^^^^^^^^^^^^^^
+
+.. image:: ../images/config-dh-pair-link.png
+
+
+Dual-Homed Servers
+------------------
+There are a number of things to note when connecting dual-homed servers to paired-ToRs.
+
+- The switch ports on the two ToRs have to be configured the same way, when connecting a dual-homed server to the two ToRs.
+- The server ports have to be Linux-bonded in a particular mode.
+
+Configure Switch Ports
+^^^^^^^^^^^^^^^^^^^^^^
+The way to configure ports are similar as described in :doc:`Bridging and Unicast <bridging-unicast>`.
+However, there are a couple of things to note.
+
+**First**, dual-homed servers should have the **identical configuration on each switch port they connect to on the ToR pairs**.
+The example below shows that the ``vlans`` and ``ips`` configured are the same on both switch ports ``of:205/12`` and ``of:206/29``.
+They are both configured to be access ports in ``VLAN 20``, the subnet ``10.0.2.0/24`` is assigned to these ports, and the gateway-IP is ``10.0.2.254/32``.
+
+.. code-block:: json
+
+ {
+ "ports" : {
+ "of:0000000000000205/12" : {
+ "interfaces" : [{
+ "name" : "h3-intf-1",
+ "ips" : [ "10.0.2.254/24"],
+ "vlan-untagged": 20
+ }]
+ },
+ "of:0000000000000206/29" : {
+ "interfaces" : [{
+ "name" : "h3-intf-2",
+ "ips" : [ "10.0.2.254/24"],
+ "vlan-untagged": 20
+ }]
+ }
+ }
+ }
+
+It is worth noting the meaning behind the configuration above from a routing perspective.
+Simply put, by configuring the same subnets on these switch ports, the fabric now believes that the entire subnet ``10.0.2.0/24`` is reachable by BOTH ToR switches ``of:205`` and ``of:206``.
+
+.. caution::
+ Configuring different VLANs, or different subnets, or mismatches like "vlan-untagged" in one switch port and "vlan-tagged" in the corresponding switch port facing the dual-homed server, will result in incorrect behavior.
+
+**Second**, we need to configure the **pair link ports on both ToR switches to be trunk (vlan-tagged) ports that contains all dual-homed VLANs and subnets**.
+This is an extra piece of configuration, the need for which will be removed in future releases.
+In the example above, a dual-homed server connects to the ToR pair on port 12 on of:205 and port 29 on of:206.
+Assume that the pair link between the two ToRs is connected to port 5 of both of:205 and of:206.
+The config for these switch ports is shown below:
+
+.. code-block:: json
+
+ {
+ "ports": {
+ "of:0000000000000205/5" : {
+ "interfaces" : [{
+ "name" : "205-pair-port",
+ "ips" : [ "10.0.2.254/24"],
+ "vlan-tagged": [20]
+ }]
+ },
+ "of:0000000000000206/5" : {
+ "interfaces" : [{
+ "name" : "206-pair-port",
+ "ips" : [ "10.0.2.254/24"],
+ "vlan-tagged": [20]
+ }]
+ }
+ }
+ }
+
+.. note::
+ Even though the ports ``of:205/12`` and ``of:206/`` facing the dual-homed server are configured as ``vlan-untagged``,
+ the same vlan MUST be configured as ``vlan-tagged`` on the pair-ports.
+ If additional subnets and VLANs are configured facing other dual-homed servers, they need to be similarly added to the ``ips`` and ``vlan-tagged`` arrays in the pair port config.
+
+
+Configure Servers
+^^^^^^^^^^^^^^^^^
+Assuming the interfaces we are going to use for bonding are ``eth1`` and ``eth2``.
+
+- Bring down interfaces
+
+ .. code-block:: console
+
+ $ sudo ifdown eth1
+ $ sudo ifdown eth2
+
+- Modify ``/etc/network/interfaces``
+
+ .. code-block:: text
+
+ auto bond0
+ iface bond0 inet dhcp
+ bond-mode balance-xor
+ bond-xmit_hash_policy layer2+3
+ bond-slaves none
+
+ auto eth1
+ iface eth1 inet manual
+ bond-master bond0
+
+ auto eth2
+ iface eth2 inet manual
+ bond-master bond0
+
+
+- Start interfaces
+
+ .. code-block:: console
+
+ $ sudo ifup bond0
+ $ sudo ifup eth1
+ $ sudo ifup eth2
+
+- Useful command to check bonding status
+
+ .. code-block:: console
+
+ # cat /proc/net/bonding/bond0
+ Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
+
+ Bonding Mode: load balancing (xor)
+ Transmit Hash Policy: layer2+3 (2)
+ MII Status: up
+ MII Polling Interval (ms): 0
+ Up Delay (ms): 0
+ Down Delay (ms): 0
+
+ Slave Interface: eth1
+ MII Status: up
+ Speed: 1000 Mbps
+ Duplex: full
+ Link Failure Count: 0
+ Permanent HW addr: 00:1c:42:5b:07:6a
+ Slave queue ID: 0
+
+ Slave Interface: eth2
+ MII Status: up
+ Speed: Unknown
+ Duplex: Unknown
+ Link Failure Count: 0
+ Permanent HW addr: 00:1c:42:1c:a1:7c
+ Slave queue ID: 0
+
+.. caution::
+ **Dual-homed host should not be statically configured.**
+
+ Currently in ONOS, configured hosts are not updated when the connectPoint is lost.
+ This is not a problem with single-homed hosts because there is no other way to reach them anyway if their connectPoint goes down.
+ But in dual-homed scenarios, the controller should take corrective action if one of the connectPoints go down –
+ the trigger for this event does not happen when the dual-homed host's connect points are configured (not discovered).
+
+.. note::
+ We also support static routes with dual-homed next hop.
+ The way to configure it is exactly the same as regular single-homed next hop, as described in :doc:`External Connectivity <external-connectivity>`.
+
+ ONOS will automatically recognize when the next-hop IP resolves to a dual-homed host and program both switches (the host connects to) accordingly.
+
+ The failure recovery mechanism for dual-homed hosts also applies to static routes that point to the host as their next hop.
+
+
+Dual External Routers
+---------------------
+
+.. image:: ../images/config-dh-vr.png
+
+.. image:: ../images/config-dh-vr-logical.png
+ :width: 200px
+
+In addition to what we describe in :doc:`External Connectivity <external-connectivity>`,
+Trellis also supports dual external routers, which view the Trellis fabric as 2 individual routers, as shown above.
+
+As before the vRouter control plane is implemented as a combination of Quagga,
+which peers with the upstream routers, and ONOS which listens to Quagga (via FPM) and programs the underlying fabric.
+**In dual-router scenarios, there are two instances of Quagga required**.
+
+As before the hardware fabric serves as the data-plane of vRouter.
+In dual-router scenarios, the **external routers MUST be connected to paired-ToRs**.
+
+
+ToR connects to one upstream
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Lets consider the simpler case where the external routers are each connected to a single leaf in a ToR pair.
+The figure on the left below shows the logical view. The figure on the right shows the physical connectivity.
+
+.. image:: ../images/config-dh-vr-logical-simple.png
+ :width: 200px
+
+.. image:: ../images/config-dh-vr-physical-simple.png
+ :width: 400px
+
+One of the upstream routers is connected to ``of:205`` and the other is connected to ``of:206``.
+Note that ``of:205`` and ``of:206`` are paired ToRs.
+
+The ToRs are connected via a physical port to separate Quagga VMs or containers.
+These Quagga instances can be placed in any compute node. They do not need to be in the same server, and are only shown to be co-located for simplicity.
+
+The two Quagga instances do NOT talk to each other.
+
+
+Switch port configuration
+"""""""""""""""""""""""""
+The ToRs follow the same rules as single router case described in :doc:`External Connectivity <external-connectivity>`.
+In the example shown above, the switch port config would look like this:
+
+.. code-block:: json
+
+ {
+ "ports": {
+ "of:0000000000000205/1" : {
+ "interfaces" : [{
+ "ips" : [ "10.0.100.3/29", "2000::6403/125" ],
+ "vlan-untagged": 100,
+ "name" : "internet-router-1"
+ }]
+ },
+
+ "of:0000000000000205/48" : {
+ "interfaces" : [{
+ "ips" : [ "10.0.100.3/29", "2000::6403/125" ],
+ "vlan-untagged": 100,
+ "name" : "quagga-1"
+ }]
+ },
+
+ "of:0000000000000206/1" : {
+ "interfaces" : [{
+ "ips" : [ "10.0.200.3/29", "2000::6503/125" ],
+ "vlan-untagged": 200,
+ "name" : "internet-router-2"
+ }]
+ },
+
+ "of:0000000000000206/48" : {
+ "interfaces" : [{
+ "ips" : [ "10.0.200.3/29", "2000::6503/125" ],
+ "vlan-untagged": 200,
+ "name" : "quagga2"
+ }]
+ }
+ }
+ }
+
+.. note::
+ In the example shown above, switch ``of:205`` uses ``VLAN 100`` for bridging the peering session between Quagga1 and ExtRouter1,
+ while switch ``of:205`` uses ``VLAN 200`` to do the same for the other peering session.
+ But since these vlans and bridging domains are defined on different switches, the VLAN ids could have been the same.
+
+ This philosophy is consistent with the fabric use of :doc:`bridging <bridging-unicast>`.
+
+
+Quagga configuration
+""""""""""""""""""""
+Configuring Quagga for dual external routers are similar to what we described in :doc:`External Connectivity <external-connectivity>`. However, it is worth noting that:
+
+- The two Zebra instances **should point to two different ONOS instances** for their FPM connections.
+ For example Zebra in Quagga1 could point to ONOS instance with ``fpm connection ip 10.6.0.1 port 2620``,
+ while the other Zebra should point to a different ONOS instance with ``fpm connection ip 10.6.0.2 port 2620``.
+ It does not matter which ONOS instances they point to as long as they are different.
+- The two Quagga BGP sessions should appear to come from different routers but still use the same AS number –
+ i.e. the two Quaggas' belong to the same AS, the one used to represent the entire Trellis infrastructure.
+- The two upstream routers can belong to the same or different AS,
+ but these AS numbers should be different from the one used to represent the Trellis AS.
+- Typically both Quagga instances advertise the same routes to the upstream.
+ These prefixes belonging to various infrastructure nodes in the deployment should be reachable from either of the leaf switches connected to the upstream routers.
+- The upstream routers may or may not advertise the same routes.
+ Trellis will ensure that traffic directed to a route reachable only one upstream router is directed to the appropriate leaf.
+
+
+ToR connects to both upstream
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Now lets consider the **more-complicated but more fault-tolerant** case of each Quagga instance peering with BOTH external routers.
+Again the logical view is shown on the left and the physical view on the right.
+
+.. image:: ../images/config-dh-vr-logical.png
+ :width: 200px
+
+.. image:: ../images/config-dh-vr-physical.png
+ :width: 500px
+
+First lets talk about the physical connectivity
+
+- Quagga instance 1 peers with external router R1 via port 1 on switch of:205
+- Quagga instance 1 peers with external router R2 via port 2 on switch of:205
+
+Similarly
+
+- Quagga instance 2 peers with external router R1 via port 2 on switch of:206
+- Quagga instance 2 peers with external router R2 via port 1 on switch of:206
+
+To distinguish between the two peering sessions in the same physical switch, say of:205,
+the physical ports 1 and 2 need to be configured in **different VLANs and subnets**.
+For example, port 1 on of:205 is (untagged) in VLAN 100, while port 2 is in VLAN 101.
+
+Note that peering for **Quagga1 and R1** happens with IPs in the ``10.0.100.0/29`` subnet,
+and for **Quagga 1 and R2** in the **10.0.101.0/29** subnet.
+
+Furthermore, **pair link** (port 48) on of:205 carries both peering sessions to Quagga1.
+Thus port 48 should now be configured as a **trunk port (vlan-tagged) with both VLANs and both subnets**.
+
+Finally the **Quagga interface** on the VM now needs **sub-interface configuration for each VLAN ID**.
+
+Similar configuration concepts apply to IPv6 as well. Here is a look at the switch port config in ONOS for of:205
+
+.. code-block:: json
+
+ {
+ "ports": {
+ "of:0000000000000205/1" : {
+ "interfaces" : [{
+ "ips" : [ "10.0.100.3/29", "2000::6403/125" ],
+ "vlan-untagged": 100,
+ "name" : "internet-router1"
+ }]
+ },
+
+
+ "of:0000000000000205/2" : {
+ "interfaces" : [{
+ "ips" : [ "10.0.101.3/29", "2000::7403/125" ],
+ "vlan-untagged": 101,
+ "name" : "internet-router2"
+ }]
+ },
+ "of:0000000000000205/48" : {
+ "interfaces" : [{
+ "ips" : [ "10.0.100.3/29", "2000::6403/125", "10.0.101.3/29", "2000::7403/125" ],
+ "vlan-tagged": [100, 101],
+ "name" : "quagga1"
+ }]
+
+ }
+ }
+ }