Dual Homing
===========

Overview
--------

.. image:: ../images/config-dh.png

The dual-homing feature includes several sub components

- **Use of "paired" ToRs**: Each rack of compute nodes have exactly two
  Top-of-Rack switches (ToRs), that are linked to each other via a single link
  - such a link is referred to as a **pair link**.  This pairing should NOT be
  omitted.

  Currently there is support for only a single link between paired ToRs.  In
  future releases, we may include dual pair links.  Note that the pair link is
  only used in failure scenarios, and not in normal operation.

- **Dual-homed servers (compute-nodes)**: Each server is connected to both
  ToRs.  The links to the paired ToRs are (Linux) bonded

- **Dual-homed upstream routers**: The upstream routers MUST be connected to
  the two ToRs that are part of a leaf-pair.  You cannot connect them to leafs
  that are not paired. This feature also requires two Quagga instances.

- **Dual-homed access devices**. This component will be added in the future.

Paired ToRs
-----------
The reasoning behind two ToR (leaf) switches is simple.  If you only have a
single ToR switch, and you lose it, the entire rack goes down.  Using two ToR
switches increases your odds for continued connectivity for dual homed servers.
The reasoning behind pairing the two ToR switches is more involved, as is
explained in the Usage section below.

Configure pair ToRs
^^^^^^^^^^^^^^^^^^^
Configuring paired-ToRs involves device configuration. Assume switches of:205
and of:206 are paired ToRs.

.. code-block:: json

    {
      "devices" : {
        "of:0000000000000205" : {
          "segmentrouting" : {
            "name" : "Leaf1-R2",
            "ipv4NodeSid" : 205,
            "ipv4Loopback" : "192.168.0.205",
            "ipv6NodeSid" : 205,
            "ipv6Loopback" : "2000::c0a8:0205",
            "routerMac" : "00:00:02:05:00:01",
            "pairDeviceId" : "of:0000000000000206",
            "pairLocalPort" : 20,
            "isEdgeRouter" : true,
            "adjacencySids" : []
          }
        },
        "of:0000000000000206" : {
          "segmentrouting" : {
            "name" : "Leaf2-R2",
            "ipv4NodeSid" : 206,
            "ipv4Loopback" : "192.168.0.206",
            "ipv6NodeSid" : 206,
            "ipv6Loopback" : "2000::c0a8:0206",
            "routerMac" : "00:00:02:05:00:01",
            "pairDeviceId" : "of:0000000000000205",
            "pairLocalPort" : 30,
            "isEdgeRouter" : true,
            "adjacencySids" : []
          }
        }
      }
    }

There are two new pieces of device configuration.

Each device in the ToR pair needs to specify the **deviceId of the leaf it is
paired to**, in the ``pairDeviceId`` field.  For example, in ``of:205``
configuration the ``pairDeviceId`` is specified as ``of:206``, and similarly in ``of:206``
configuration the ``pairDeviceId`` is ``of:205``. Each device in the ToR pair needs to
specify the **port on the device used for the pair link** in the
``pairLocalPort`` field.  For example, the pair link in the config above show
that port 20 on of:205 is connected to port 30 on of:206.

In addition, there is one crucial piece of config that needs to **match for
both ToRs** – the ``routerMac`` address.  The paired-ToRs MUST have the same
routerMac - in the example above, they both have identical 00:00:02:05:00:01
routerMacs.

All other fields are the same as before, as explained in :doc:`Device
Configuration <device-config>` section.


Usage of pair link
^^^^^^^^^^^^^^^^^^

.. image:: ../images/config-dh-pair-link.png


Dual-Homed Servers
------------------

There are a number of things to note when connecting dual-homed servers to paired-ToRs.

- The switch ports on the two ToRs have to be configured the same way, when
  connecting a dual-homed server to the two ToRs.

- The server ports have to be Linux-bonded in a particular mode.

Configure Switch Ports
^^^^^^^^^^^^^^^^^^^^^^

The way to configure ports are similar as described in :doc:`Bridging and
Unicast <bridging-unicast>`.  However, there are a couple of things to note.

**First**, dual-homed servers should have the **identical configuration on each
switch port they connect to on the ToR pairs**.  The example below shows that
the ``vlans`` and ``ips`` configured are the same on both switch ports
``of:205/12`` and ``of:206/29``.  They are both configured to be access ports
in ``VLAN 20``, the subnet ``10.0.2.0/24`` is assigned to these ports, and the
gateway-IP is ``10.0.2.254/32``.

.. code-block:: json

    {
      "ports" : {
        "of:0000000000000205/12" : {
          "interfaces" : [{
            "name" : "h3-intf-1",
            "ips" : [ "10.0.2.254/24"],
            "vlan-untagged": 20
          }]
        },
        "of:0000000000000206/29" : {
          "interfaces" : [{
            "name" : "h3-intf-2",
            "ips" : [ "10.0.2.254/24"],
            "vlan-untagged": 20
          }]
        }
      }
    }

It is worth noting the meaning behind the configuration above from a routing
perspective.  Simply put, by configuring the same subnets on these switch
ports, the fabric now believes that the entire subnet ``10.0.2.0/24`` is
reachable by BOTH ToR switches ``of:205`` and ``of:206``.

.. caution::
    Configuring different VLANs, or different subnets, or mismatches like
    "vlan-untagged" in one switch port and "vlan-tagged" in the corresponding
    switch port facing the dual-homed server, will result in incorrect
    behavior.

**Second**, we need to configure the **pair link ports on both ToR switches to
be trunk (vlan-tagged) ports that contains all dual-homed VLANs and subnets**.
This is an extra piece of configuration, the need for which will be removed in
future releases.  In the example above, a dual-homed server connects to the ToR
pair on port 12 on of:205 and port 29 on of:206.  Assume that the pair link
between the two ToRs is connected to port 5 of both of:205 and of:206.  The
config for these switch ports is shown below:

.. code-block:: json

    {
      "ports": {
        "of:0000000000000205/5" : {
          "interfaces" : [{
              "name" : "205-pair-port",
              "ips" : [ "10.0.2.254/24"],
              "vlan-tagged": [20]
          }]
        },
        "of:0000000000000206/5" : {
          "interfaces" : [{
            "name" : "206-pair-port",
            "ips" : [ "10.0.2.254/24"],
            "vlan-tagged": [20]
          }]
        }
      }
    }

.. note::
    Even though the ports ``of:205/12`` and ``of:206/29`` facing the dual-homed
    server are configured as ``vlan-untagged``, the same vlan MUST be
    configured as ``vlan-tagged`` on the pair-ports.

    If additional subnets and VLANs are configured facing other dual-homed
    servers, they need to be similarly added to the ``ips`` and ``vlan-tagged``
    arrays in the pair port config.


Configure Servers
^^^^^^^^^^^^^^^^^

Assuming the interfaces we are going to use for bonding are ``eth1`` and ``eth2``.

- Bring down interfaces

  .. code-block:: console

      $ sudo ifdown eth1
      $ sudo ifdown eth2

- Modify ``/etc/network/interfaces``

  .. code-block:: text

      auto bond0
      iface bond0 inet dhcp
      bond-mode balance-xor
      bond-xmit_hash_policy layer2+3
      bond-slaves none

      auto eth1
      iface eth1 inet manual
      bond-master bond0

      auto eth2
      iface eth2 inet manual
      bond-master bond0


- Start interfaces

  .. code-block:: console

      $ sudo ifup bond0
      $ sudo ifup eth1
      $ sudo ifup eth2

- Useful command to check bonding status

  .. code-block:: console

      # cat /proc/net/bonding/bond0
      Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

      Bonding Mode: load balancing (xor)
      Transmit Hash Policy: layer2+3 (2)
      MII Status: up
      MII Polling Interval (ms): 0
      Up Delay (ms): 0
      Down Delay (ms): 0

      Slave Interface: eth1
      MII Status: up
      Speed: 1000 Mbps
      Duplex: full
      Link Failure Count: 0
      Permanent HW addr: 00:1c:42:5b:07:6a
      Slave queue ID: 0

      Slave Interface: eth2
      MII Status: up
      Speed: Unknown
      Duplex: Unknown
      Link Failure Count: 0
      Permanent HW addr: 00:1c:42:1c:a1:7c
      Slave queue ID: 0

.. caution::
    **Dual-homed host should not be statically configured.**

    Currently in ONOS, configured hosts are not updated when the connectPoint
    is lost.  This is not a problem with single-homed hosts because there is no
    other way to reach them anyway if their connectPoint goes down.  But in
    dual-homed scenarios, the controller should take corrective action if one
    of the connectPoints go down – the trigger for this event does not happen
    when the dual-homed host's connect points are configured (not discovered).

.. note::
    We also support static routes with dual-homed next hop.  The way to
    configure it is exactly the same as regular single-homed next hop, as
    described in :doc:`External Connectivity <external-connectivity>`.

    ONOS will automatically recognize when the next-hop IP resolves to a
    dual-homed host and program both switches (the host connects to)
    accordingly.

    The failure recovery mechanism for dual-homed hosts also applies to static
    routes that point to the host as their next hop.

Dual External Routers
---------------------

.. image:: ../images/config-dh-vr.png

.. image:: ../images/config-dh-vr-logical.png
    :width: 200px

In addition to what we describe in :doc:`External Connectivity
<external-connectivity>`, Trellis also supports dual external routers, which
view the Trellis fabric as 2 individual routers, as shown above.

As before the vRouter control plane is implemented as a combination of Quagga,
which peers with the upstream routers, and ONOS which listens to Quagga (via
FPM) and programs the underlying fabric.  **In dual-router scenarios, there are
two instances of Quagga required**.

As before the hardware fabric serves as the data-plane of vRouter.  In
dual-router scenarios, the **external routers MUST be connected to
paired-ToRs**.

ToR connects to one upstream
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Lets consider the simpler case where the external routers are each connected to
a single leaf in a ToR pair.  The figure on the left below shows the logical
view. The figure on the right shows the physical connectivity.

.. image:: ../images/config-dh-vr-logical-simple.png
    :width: 200px

.. image:: ../images/config-dh-vr-physical-simple.png
    :width: 400px

One of the upstream routers is connected to ``of:205`` and the other is
connected to ``of:206``.  Note that ``of:205`` and ``of:206`` are paired ToRs.

The ToRs are connected via a physical port to separate Quagga VMs or
containers.  These Quagga instances can be placed in any compute node. They do
not need to be in the same server, and are only shown to be co-located for
simplicity.

The two Quagga instances do NOT talk to each other.

Switch port configuration
"""""""""""""""""""""""""

The ToRs follow the same rules as single router case described in
:doc:`External Connectivity <external-connectivity>`.  In the example shown
above, the switch port config would look like this:

.. code-block:: json

    {
      "ports": {
        "of:0000000000000205/1" : {
          "interfaces" : [{
            "ips" : [ "10.0.100.3/29", "2000::6403/125" ],
            "vlan-untagged": 100,
            "name" : "internet-router-1"
          }]
        },

        "of:0000000000000205/48" : {
          "interfaces" : [{
            "ips" : [ "10.0.100.3/29", "2000::6403/125" ],
            "vlan-untagged": 100,
            "name" : "quagga-1"
          }]
        },

        "of:0000000000000206/1" : {
          "interfaces" : [{
            "ips" : [ "10.0.200.3/29", "2000::6503/125" ],
            "vlan-untagged": 200,
            "name" : "internet-router-2"
          }]
        },

        "of:0000000000000206/48" : {
          "interfaces" : [{
            "ips" : [ "10.0.200.3/29", "2000::6503/125" ],
            "vlan-untagged": 200,
            "name" : "quagga2"
          }]
        }
      }
    }

.. note::
    In the example shown above, switch ``of:205`` uses ``VLAN 100`` for
    bridging the peering session between Quagga1 and ExtRouter1, while switch
    ``of:205`` uses ``VLAN 200`` to do the same for the other peering session.
    But since these vlans and bridging domains are defined on different
    switches, the VLAN ids could have been the same.

    This philosophy is consistent with the fabric use of :doc:`bridging
    <bridging-unicast>`.


Quagga configuration
""""""""""""""""""""
Configuring Quagga for dual external routers are similar to what we described
in :doc:`External Connectivity <external-connectivity>`. However, it is worth
noting that:

- The two Zebra instances **should point to two different ONOS instances** for
  their FPM connections.  For example Zebra in Quagga1 could point to ONOS
  instance with ``fpm connection ip 10.6.0.1 port 2620``, while the other Zebra
  should point to a different ONOS instance with ``fpm connection ip 10.6.0.2
  port 2620``.  It does not matter which ONOS instances they point to as long
  as they are different.

- The two Quagga BGP sessions should appear to come from different routers but
  still use the same AS number – i.e. the two Quaggas' belong to the same AS,
  the one used to represent the entire Trellis infrastructure.

- The two upstream routers can belong to the same or different AS, but these AS
  numbers should be different from the one used to represent the Trellis AS.

- Typically both Quagga instances advertise the same routes to the upstream.
  These prefixes belonging to various infrastructure nodes in the deployment
  should be reachable from either of the leaf switches connected to the
  upstream routers.

- The upstream routers may or may not advertise the same routes.  Trellis will
  ensure that traffic directed to a route reachable only one upstream router is
  directed to the appropriate leaf.

ToR connects to both upstream
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Now lets consider the **more-complicated but more fault-tolerant** case of each
Quagga instance peering with BOTH external routers.  Again the logical view is
shown on the left and the physical view on the right.

.. image:: ../images/config-dh-vr-logical.png
    :width: 200px

.. image:: ../images/config-dh-vr-physical.png
    :width: 500px

First lets talk about the physical connectivity

- Quagga instance 1 peers with external router R1 via port 1 on switch of:205
- Quagga instance 1 peers with external router R2 via port 2 on switch of:205

Similarly

- Quagga instance 2 peers with external router R1 via port 2 on switch of:206
- Quagga instance 2 peers with external router R2 via port 1 on switch of:206

To distinguish between the two peering sessions in the same physical switch,
say of:205, the physical ports 1 and 2 need to be configured in **different
VLANs and subnets**.  For example, port 1 on of:205 is (untagged) in VLAN 100,
while port 2 is in VLAN 101.

Note that peering for **Quagga1 and R1** happens with IPs in the
``10.0.100.0/29`` subnet, and for **Quagga 1 and R2** in the **10.0.101.0/29**
subnet.

Furthermore, **pair link** (port 48) on of:205 carries both peering sessions to
Quagga1.  Thus port 48 should now be configured as a **trunk port (vlan-tagged)
with both VLANs and both subnets**.

Finally the **Quagga interface** on the VM now needs **sub-interface
configuration for each VLAN ID**.

Similar configuration concepts apply to IPv6 as well. Here is a look at the
switch port config in ONOS for of:205

.. code-block:: json

    {
      "ports": {
        "of:0000000000000205/1" : {
            "interfaces" : [{
                "ips" : [ "10.0.100.3/29", "2000::6403/125" ],
                "vlan-untagged": 100,
                "name" : "internet-router1"
            }]
        },


        "of:0000000000000205/2" : {
            "interfaces" : [{
                "ips" : [ "10.0.101.3/29", "2000::7403/125" ],
                "vlan-untagged": 101,
                "name" : "internet-router2"
            }]
        },
        "of:0000000000000205/48" : {
            "interfaces" : [{
                "ips" : [ "10.0.100.3/29", "2000::6403/125", "10.0.101.3/29", "2000::7403/125" ],
                "vlan-tagged": [100, 101],
                "name" : "quagga1"
            }]

        }
      }
    }