| Dual Homing |
| =========== |
| |
| Overview |
| -------- |
| |
| .. image:: ../images/config-dh.png |
| |
| The dual-homing feature includes several sub components |
| |
| - **Use of "paired" ToRs**: Each rack of compute nodes have exactly two |
| Top-of-Rack switches (ToRs), that are linked to each other via a single link |
| - such a link is referred to as a **pair link**. This pairing should NOT be |
| omitted. |
| |
| Currently there is support for only a single link between paired ToRs. In |
| future releases, we may include dual pair links. Note that the pair link is |
| only used in failure scenarios, and not in normal operation. |
| |
| - **Dual-homed servers (compute-nodes)**: Each server is connected to both |
| ToRs. The links to the paired ToRs are (Linux) bonded |
| |
| - **Dual-homed upstream routers**: The upstream routers MUST be connected to |
| the two ToRs that are part of a leaf-pair. You cannot connect them to leafs |
| that are not paired. This feature also requires two Quagga instances. |
| |
| - **Dual-homed access devices**. This component will be added in the future. |
| |
| Paired ToRs |
| ----------- |
| The reasoning behind two ToR (leaf) switches is simple. If you only have a |
| single ToR switch, and you lose it, the entire rack goes down. Using two ToR |
| switches increases your odds for continued connectivity for dual homed servers. |
| The reasoning behind pairing the two ToR switches is more involved, as is |
| explained in the Usage section below. |
| |
| Configure pair ToRs |
| ^^^^^^^^^^^^^^^^^^^ |
| Configuring paired-ToRs involves device configuration. Assume switches of:205 |
| and of:206 are paired ToRs. |
| |
| .. code-block:: json |
| |
| { |
| "devices" : { |
| "of:0000000000000205" : { |
| "segmentrouting" : { |
| "name" : "Leaf1-R2", |
| "ipv4NodeSid" : 205, |
| "ipv4Loopback" : "192.168.0.205", |
| "ipv6NodeSid" : 205, |
| "ipv6Loopback" : "2000::c0a8:0205", |
| "routerMac" : "00:00:02:05:00:01", |
| "pairDeviceId" : "of:0000000000000206", |
| "pairLocalPort" : 20, |
| "isEdgeRouter" : true, |
| "adjacencySids" : [] |
| } |
| }, |
| "of:0000000000000206" : { |
| "segmentrouting" : { |
| "name" : "Leaf2-R2", |
| "ipv4NodeSid" : 206, |
| "ipv4Loopback" : "192.168.0.206", |
| "ipv6NodeSid" : 206, |
| "ipv6Loopback" : "2000::c0a8:0206", |
| "routerMac" : "00:00:02:05:00:01", |
| "pairDeviceId" : "of:0000000000000205", |
| "pairLocalPort" : 30, |
| "isEdgeRouter" : true, |
| "adjacencySids" : [] |
| } |
| } |
| } |
| } |
| |
| There are two new pieces of device configuration. |
| |
| Each device in the ToR pair needs to specify the **deviceId of the leaf it is |
| paired to**, in the ``pairDeviceId`` field. For example, in ``of:205`` |
| configuration the ``pairDeviceId`` is specified as ``of:206``, and similarly in ``of:206`` |
| configuration the ``pairDeviceId`` is ``of:205``. Each device in the ToR pair needs to |
| specify the **port on the device used for the pair link** in the |
| ``pairLocalPort`` field. For example, the pair link in the config above show |
| that port 20 on of:205 is connected to port 30 on of:206. |
| |
| In addition, there is one crucial piece of config that needs to **match for |
| both ToRs** – the ``routerMac`` address. The paired-ToRs MUST have the same |
| routerMac - in the example above, they both have identical 00:00:02:05:00:01 |
| routerMacs. |
| |
| All other fields are the same as before, as explained in :doc:`Device |
| Configuration <device-config>` section. |
| |
| |
| Usage of pair link |
| ^^^^^^^^^^^^^^^^^^ |
| |
| .. image:: ../images/config-dh-pair-link.png |
| |
| |
| Dual-Homed Servers |
| ------------------ |
| |
| There are a number of things to note when connecting dual-homed servers to paired-ToRs. |
| |
| - The switch ports on the two ToRs have to be configured the same way, when |
| connecting a dual-homed server to the two ToRs. |
| |
| - The server ports have to be Linux-bonded in a particular mode. |
| |
| Configure Switch Ports |
| ^^^^^^^^^^^^^^^^^^^^^^ |
| |
| The way to configure ports are similar as described in :doc:`Bridging and |
| Unicast <bridging-unicast>`. However, there are a couple of things to note. |
| |
| **First**, dual-homed servers should have the **identical configuration on each |
| switch port they connect to on the ToR pairs**. The example below shows that |
| the ``vlans`` and ``ips`` configured are the same on both switch ports |
| ``of:205/12`` and ``of:206/29``. They are both configured to be access ports |
| in ``VLAN 20``, the subnet ``10.0.2.0/24`` is assigned to these ports, and the |
| gateway-IP is ``10.0.2.254/32``. |
| |
| .. code-block:: json |
| |
| { |
| "ports" : { |
| "of:0000000000000205/12" : { |
| "interfaces" : [{ |
| "name" : "h3-intf-1", |
| "ips" : [ "10.0.2.254/24"], |
| "vlan-untagged": 20 |
| }] |
| }, |
| "of:0000000000000206/29" : { |
| "interfaces" : [{ |
| "name" : "h3-intf-2", |
| "ips" : [ "10.0.2.254/24"], |
| "vlan-untagged": 20 |
| }] |
| } |
| } |
| } |
| |
| It is worth noting the meaning behind the configuration above from a routing |
| perspective. Simply put, by configuring the same subnets on these switch |
| ports, the fabric now believes that the entire subnet ``10.0.2.0/24`` is |
| reachable by BOTH ToR switches ``of:205`` and ``of:206``. |
| |
| .. caution:: |
| Configuring different VLANs, or different subnets, or mismatches like |
| "vlan-untagged" in one switch port and "vlan-tagged" in the corresponding |
| switch port facing the dual-homed server, will result in incorrect |
| behavior. |
| |
| **Second**, we need to configure the **pair link ports on both ToR switches to |
| be trunk (vlan-tagged) ports that contains all dual-homed VLANs and subnets**. |
| This is an extra piece of configuration, the need for which will be removed in |
| future releases. In the example above, a dual-homed server connects to the ToR |
| pair on port 12 on of:205 and port 29 on of:206. Assume that the pair link |
| between the two ToRs is connected to port 5 of both of:205 and of:206. The |
| config for these switch ports is shown below: |
| |
| .. code-block:: json |
| |
| { |
| "ports": { |
| "of:0000000000000205/5" : { |
| "interfaces" : [{ |
| "name" : "205-pair-port", |
| "ips" : [ "10.0.2.254/24"], |
| "vlan-tagged": [20] |
| }] |
| }, |
| "of:0000000000000206/5" : { |
| "interfaces" : [{ |
| "name" : "206-pair-port", |
| "ips" : [ "10.0.2.254/24"], |
| "vlan-tagged": [20] |
| }] |
| } |
| } |
| } |
| |
| .. note:: |
| Even though the ports ``of:205/12`` and ``of:206/29`` facing the dual-homed |
| server are configured as ``vlan-untagged``, the same vlan MUST be |
| configured as ``vlan-tagged`` on the pair-ports. |
| |
| If additional subnets and VLANs are configured facing other dual-homed |
| servers, they need to be similarly added to the ``ips`` and ``vlan-tagged`` |
| arrays in the pair port config. |
| |
| |
| Configure Servers |
| ^^^^^^^^^^^^^^^^^ |
| |
| Assuming the interfaces we are going to use for bonding are ``eth1`` and ``eth2``. |
| |
| - Bring down interfaces |
| |
| .. code-block:: console |
| |
| $ sudo ifdown eth1 |
| $ sudo ifdown eth2 |
| |
| - Modify ``/etc/network/interfaces`` |
| |
| .. code-block:: text |
| |
| auto bond0 |
| iface bond0 inet dhcp |
| bond-mode balance-xor |
| bond-xmit_hash_policy layer2+3 |
| bond-slaves none |
| |
| auto eth1 |
| iface eth1 inet manual |
| bond-master bond0 |
| |
| auto eth2 |
| iface eth2 inet manual |
| bond-master bond0 |
| |
| |
| - Start interfaces |
| |
| .. code-block:: console |
| |
| $ sudo ifup bond0 |
| $ sudo ifup eth1 |
| $ sudo ifup eth2 |
| |
| - Useful command to check bonding status |
| |
| .. code-block:: console |
| |
| # cat /proc/net/bonding/bond0 |
| Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) |
| |
| Bonding Mode: load balancing (xor) |
| Transmit Hash Policy: layer2+3 (2) |
| MII Status: up |
| MII Polling Interval (ms): 0 |
| Up Delay (ms): 0 |
| Down Delay (ms): 0 |
| |
| Slave Interface: eth1 |
| MII Status: up |
| Speed: 1000 Mbps |
| Duplex: full |
| Link Failure Count: 0 |
| Permanent HW addr: 00:1c:42:5b:07:6a |
| Slave queue ID: 0 |
| |
| Slave Interface: eth2 |
| MII Status: up |
| Speed: Unknown |
| Duplex: Unknown |
| Link Failure Count: 0 |
| Permanent HW addr: 00:1c:42:1c:a1:7c |
| Slave queue ID: 0 |
| |
| .. caution:: |
| **Dual-homed host should not be statically configured.** |
| |
| Currently in ONOS, configured hosts are not updated when the connectPoint |
| is lost. This is not a problem with single-homed hosts because there is no |
| other way to reach them anyway if their connectPoint goes down. But in |
| dual-homed scenarios, the controller should take corrective action if one |
| of the connectPoints go down – the trigger for this event does not happen |
| when the dual-homed host's connect points are configured (not discovered). |
| |
| .. note:: |
| We also support static routes with dual-homed next hop. The way to |
| configure it is exactly the same as regular single-homed next hop, as |
| described in :doc:`External Connectivity <external-connectivity>`. |
| |
| ONOS will automatically recognize when the next-hop IP resolves to a |
| dual-homed host and program both switches (the host connects to) |
| accordingly. |
| |
| The failure recovery mechanism for dual-homed hosts also applies to static |
| routes that point to the host as their next hop. |
| |
| Dual External Routers |
| --------------------- |
| |
| .. image:: ../images/config-dh-vr.png |
| |
| .. image:: ../images/config-dh-vr-logical.png |
| :width: 200px |
| |
| In addition to what we describe in :doc:`External Connectivity |
| <external-connectivity>`, Trellis also supports dual external routers, which |
| view the Trellis fabric as 2 individual routers, as shown above. |
| |
| As before the vRouter control plane is implemented as a combination of Quagga, |
| which peers with the upstream routers, and ONOS which listens to Quagga (via |
| FPM) and programs the underlying fabric. **In dual-router scenarios, there are |
| two instances of Quagga required**. |
| |
| As before the hardware fabric serves as the data-plane of vRouter. In |
| dual-router scenarios, the **external routers MUST be connected to |
| paired-ToRs**. |
| |
| ToR connects to one upstream |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| |
| Lets consider the simpler case where the external routers are each connected to |
| a single leaf in a ToR pair. The figure on the left below shows the logical |
| view. The figure on the right shows the physical connectivity. |
| |
| .. image:: ../images/config-dh-vr-logical-simple.png |
| :width: 200px |
| |
| .. image:: ../images/config-dh-vr-physical-simple.png |
| :width: 400px |
| |
| One of the upstream routers is connected to ``of:205`` and the other is |
| connected to ``of:206``. Note that ``of:205`` and ``of:206`` are paired ToRs. |
| |
| The ToRs are connected via a physical port to separate Quagga VMs or |
| containers. These Quagga instances can be placed in any compute node. They do |
| not need to be in the same server, and are only shown to be co-located for |
| simplicity. |
| |
| The two Quagga instances do NOT talk to each other. |
| |
| Switch port configuration |
| """"""""""""""""""""""""" |
| |
| The ToRs follow the same rules as single router case described in |
| :doc:`External Connectivity <external-connectivity>`. In the example shown |
| above, the switch port config would look like this: |
| |
| .. code-block:: json |
| |
| { |
| "ports": { |
| "of:0000000000000205/1" : { |
| "interfaces" : [{ |
| "ips" : [ "10.0.100.3/29", "2000::6403/125" ], |
| "vlan-untagged": 100, |
| "name" : "internet-router-1" |
| }] |
| }, |
| |
| "of:0000000000000205/48" : { |
| "interfaces" : [{ |
| "ips" : [ "10.0.100.3/29", "2000::6403/125" ], |
| "vlan-untagged": 100, |
| "name" : "quagga-1" |
| }] |
| }, |
| |
| "of:0000000000000206/1" : { |
| "interfaces" : [{ |
| "ips" : [ "10.0.200.3/29", "2000::6503/125" ], |
| "vlan-untagged": 200, |
| "name" : "internet-router-2" |
| }] |
| }, |
| |
| "of:0000000000000206/48" : { |
| "interfaces" : [{ |
| "ips" : [ "10.0.200.3/29", "2000::6503/125" ], |
| "vlan-untagged": 200, |
| "name" : "quagga2" |
| }] |
| } |
| } |
| } |
| |
| .. note:: |
| In the example shown above, switch ``of:205`` uses ``VLAN 100`` for |
| bridging the peering session between Quagga1 and ExtRouter1, while switch |
| ``of:205`` uses ``VLAN 200`` to do the same for the other peering session. |
| But since these vlans and bridging domains are defined on different |
| switches, the VLAN ids could have been the same. |
| |
| This philosophy is consistent with the fabric use of :doc:`bridging |
| <bridging-unicast>`. |
| |
| |
| Quagga configuration |
| """""""""""""""""""" |
| Configuring Quagga for dual external routers are similar to what we described |
| in :doc:`External Connectivity <external-connectivity>`. However, it is worth |
| noting that: |
| |
| - The two Zebra instances **should point to two different ONOS instances** for |
| their FPM connections. For example Zebra in Quagga1 could point to ONOS |
| instance with ``fpm connection ip 10.6.0.1 port 2620``, while the other Zebra |
| should point to a different ONOS instance with ``fpm connection ip 10.6.0.2 |
| port 2620``. It does not matter which ONOS instances they point to as long |
| as they are different. |
| |
| - The two Quagga BGP sessions should appear to come from different routers but |
| still use the same AS number – i.e. the two Quaggas' belong to the same AS, |
| the one used to represent the entire Trellis infrastructure. |
| |
| - The two upstream routers can belong to the same or different AS, but these AS |
| numbers should be different from the one used to represent the Trellis AS. |
| |
| - Typically both Quagga instances advertise the same routes to the upstream. |
| These prefixes belonging to various infrastructure nodes in the deployment |
| should be reachable from either of the leaf switches connected to the |
| upstream routers. |
| |
| - The upstream routers may or may not advertise the same routes. Trellis will |
| ensure that traffic directed to a route reachable only one upstream router is |
| directed to the appropriate leaf. |
| |
| ToR connects to both upstream |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| |
| Now lets consider the **more-complicated but more fault-tolerant** case of each |
| Quagga instance peering with BOTH external routers. Again the logical view is |
| shown on the left and the physical view on the right. |
| |
| .. image:: ../images/config-dh-vr-logical.png |
| :width: 200px |
| |
| .. image:: ../images/config-dh-vr-physical.png |
| :width: 500px |
| |
| First lets talk about the physical connectivity |
| |
| - Quagga instance 1 peers with external router R1 via port 1 on switch of:205 |
| - Quagga instance 1 peers with external router R2 via port 2 on switch of:205 |
| |
| Similarly |
| |
| - Quagga instance 2 peers with external router R1 via port 2 on switch of:206 |
| - Quagga instance 2 peers with external router R2 via port 1 on switch of:206 |
| |
| To distinguish between the two peering sessions in the same physical switch, |
| say of:205, the physical ports 1 and 2 need to be configured in **different |
| VLANs and subnets**. For example, port 1 on of:205 is (untagged) in VLAN 100, |
| while port 2 is in VLAN 101. |
| |
| Note that peering for **Quagga1 and R1** happens with IPs in the |
| ``10.0.100.0/29`` subnet, and for **Quagga 1 and R2** in the **10.0.101.0/29** |
| subnet. |
| |
| Furthermore, **pair link** (port 48) on of:205 carries both peering sessions to |
| Quagga1. Thus port 48 should now be configured as a **trunk port (vlan-tagged) |
| with both VLANs and both subnets**. |
| |
| Finally the **Quagga interface** on the VM now needs **sub-interface |
| configuration for each VLAN ID**. |
| |
| Similar configuration concepts apply to IPv6 as well. Here is a look at the |
| switch port config in ONOS for of:205 |
| |
| .. code-block:: json |
| |
| { |
| "ports": { |
| "of:0000000000000205/1" : { |
| "interfaces" : [{ |
| "ips" : [ "10.0.100.3/29", "2000::6403/125" ], |
| "vlan-untagged": 100, |
| "name" : "internet-router1" |
| }] |
| }, |
| |
| |
| "of:0000000000000205/2" : { |
| "interfaces" : [{ |
| "ips" : [ "10.0.101.3/29", "2000::7403/125" ], |
| "vlan-untagged": 101, |
| "name" : "internet-router2" |
| }] |
| }, |
| "of:0000000000000205/48" : { |
| "interfaces" : [{ |
| "ips" : [ "10.0.100.3/29", "2000::6403/125", "10.0.101.3/29", "2000::7403/125" ], |
| "vlan-tagged": [100, 101], |
| "name" : "quagga1" |
| }] |
| |
| } |
| } |
| } |