Basic Trellis environment
Done:
- 2x2 leaf-spine
- IPv4 hosts (h1-h4)
- IPv6 hosts (h1v6-h4v6)
- Internal quagga
- External quagga (+1 IPv4 and 1 IPv6 host behind it)
- DHCP server
Change-Id: I05781f6507d897a80d668d465abd9e86c8dbac4d
diff --git a/trellis/README.md b/trellis/README.md
new file mode 100644
index 0000000..b9f3901
--- /dev/null
+++ b/trellis/README.md
@@ -0,0 +1,90 @@
+Trellis Leaf-Spine Fabric
+=========================
+
+# Introduction
+This folder contains Mininet scripts and corresponding config files that
+can be used to emulate Trellis leaf-spine fabric, vRouter and DHCP relay.
+
+# Download
+`git clone https://gerrit.onosproject.org/routing`
+
+# Installation
+
+## Ubuntu 16.04 LTS
+Some dependencies need to be installed for a fresh Ubuntu.
+```
+sudo apt-get update
+sudo apt-get install texinfo
+sudo apt-get install python-pip
+sudo pip install ipaddress
+```
+
+## Mininet
+`sudo apt-get install mininet`
+
+## OpenvSwitch
+Mininet should install OVS for you.
+Please run `sudo ovs-vsctl --version` and make sure the OVS version is above 2.5.0+.
+
+## DHCP server
+`sudo apt-get install isc-dhcp-server`
+
+## Quagga
+Trellis needs a special FPM patch for Quagga.
+
+```
+wget http://download.savannah.gnu.org/releases/quagga/quagga-0.99.23.tar.gz
+wget https://wiki.opencord.org/download/attachments/1278529/fpm-remote.diff
+tar -xzvf quagga-0.99.23.tar.gz
+cd quagga-0.99.23
+patch -p1 < ../fpm-remote.diff
+./configure --enable-fpm --sbindir=/usr/lib
+make
+sudo make install
+cd ..
+```
+
+## ONOS - Installation
+Learn about how to setup ONOS at: https://wiki.onosproject.org/.
+After installation, the following ONOS apps need to be activated.
+
+`export ONOS_APPS=drivers,openflow,segmentrouting,fpm,dhcprelay,netcfghostprovider`
+
+## ONOS - Network Config
+`onos-netcfg <onos-ip> routing/trellis/trellis.json`
+
+
+## Update Controller IP
+The location of ONOS controller needs to be updated in several places, including
+Mininet script and zebra config.
+
+In `routing/trellis/trellis.py`
+```
+net.addController(RemoteController('c0', ip='192.168.56.11'))
+net.addController(RemoteController('c1', ip='192.168.56.12'))
+net.addController(RemoteController('c2', ip='192.168.56.13'))
+```
+
+In `routing/trellis/zebrabgp1.conf`
+```
+fpm connection ip 192.168.56.11 port 2620
+```
+
+## Start Mininet Emulation
+```
+cd routing/trellis
+sudo ./trellis.py
+```
+
+## Verify Network Connectivity
+In Mininet, run
+- `h1 ping 10.0.99.2` to check IPv4 connectivity
+- `h1v6 ping6 2000::9902` to check IPv6 connectivity
+
+# Troubleshooting
+- Services in the emulated hosts may still be alive if Mininet is not terminated properly.
+In that case, simply run the following command to clean up.
+```
+sudo killall -9 dhclient dhcpd zebra bgpd
+sudo mn -c
+```
diff --git a/trellis/bgpdbgp1.conf b/trellis/bgpdbgp1.conf
new file mode 100644
index 0000000..383460c
--- /dev/null
+++ b/trellis/bgpdbgp1.conf
@@ -0,0 +1,53 @@
+log file /var/log/quagga/bgpdbgp1.log
+hostname bgp1
+password quagga
+!
+! Different next hop for IPv4
+!
+ip prefix-list 1 seq 10 permit 10.0.2.0/24
+ip prefix-list 1 seq 20 permit 10.0.3.0/24
+!
+route-map NEXTHOP4 permit 10
+match ip address prefix-list 1
+set ip next-hop 10.0.1.254
+!
+! Different next hop for IPv6
+!
+ipv6 prefix-list 2 seq 10 permit 2000::200/120
+ipv6 prefix-list 2 seq 20 permit 2000::300/120
+!
+route-map NEXTHOP6 permit 10
+match ipv6 address prefix-list 2
+set ipv6 next-hop global 2000::1ff
+set ipv6 next-hop local 2000::1ff
+!
+! Basic router config
+!
+router bgp 65002
+bgp router-id 10.0.1.2
+timers bgp 3 9
+!
+! IPv4
+!
+neighbor 10.0.1.1 remote-as 65001
+neighbor 10.0.1.1 ebgp-multihop
+neighbor 10.0.1.1 timers connect 5
+neighbor 10.0.1.1 advertisement-interval 5
+neighbor 10.0.1.1 route-map NEXTHOP4 out
+!
+neighbor 2000::101 remote-as 65001
+neighbor 2000::101 timers connect 5
+neighbor 2000::101 advertisement-interval 1
+no neighbor 2000::101 activate
+!
+network 10.0.2.0/24
+network 10.0.3.0/24
+!
+! IPv6
+!
+address-family ipv6
+network 2000::200/120
+network 2000::300/120
+neighbor 2000::101 activate
+neighbor 2000::101 route-map NEXTHOP6 out
+exit-address-family
diff --git a/trellis/bgpdr1.conf b/trellis/bgpdr1.conf
new file mode 100644
index 0000000..9a05ac9
--- /dev/null
+++ b/trellis/bgpdr1.conf
@@ -0,0 +1,30 @@
+log file /var/log/quagga/bgpdr1.log
+hostname r1
+password quagga
+!
+! Basic router config
+!
+router bgp 65001
+bgp router-id 10.0.1.1
+timers bgp 3 9
+!
+! IPv4
+!
+neighbor 10.0.1.2 remote-as 65002
+neighbor 10.0.1.2 ebgp-multihop
+neighbor 10.0.1.2 timers connect 5
+neighbor 10.0.1.2 advertisement-interval 5
+!
+neighbor 2000::102 remote-as 65002
+neighbor 2000::102 timers connect 5
+neighbor 2000::102 advertisement-interval 1
+no neighbor 2000::102 activate
+!
+network 10.0.99.0/24
+!
+! IPv6
+!
+address-family ipv6
+network 2000::9900/120
+neighbor 2000::102 activate
+exit-address-family
diff --git a/trellis/dhcpd.conf b/trellis/dhcpd.conf
new file mode 100644
index 0000000..9dd56dd
--- /dev/null
+++ b/trellis/dhcpd.conf
@@ -0,0 +1,34 @@
+ddns-update-style none;
+
+default-lease-time 600;
+max-lease-time 7200;
+
+subnet 10.0.2.0 netmask 255.255.255.0 {
+ range 10.0.2.100 10.0.2.240;
+ option routers 10.0.2.254;
+}
+
+subnet 10.0.3.0 netmask 255.255.255.0 {
+ range 10.0.3.100 10.0.3.240;
+ option routers 10.0.3.254;
+}
+
+host h1 {
+ hardware ethernet 00:aa:00:00:00:01;
+ fixed-address 10.0.2.1;
+}
+
+host h2 {
+ hardware ethernet 00:aa:00:00:00:02;
+ fixed-address 10.0.2.2;
+}
+
+host h3 {
+ hardware ethernet 00:aa:00:00:00:03;
+ fixed-address 10.0.3.1;
+}
+
+host h4 {
+ hardware ethernet 00:aa:00:00:00:04;
+ fixed-address 10.0.3.2;
+}
diff --git a/trellis/trellis.json b/trellis/trellis.json
new file mode 100644
index 0000000..8aef52a
--- /dev/null
+++ b/trellis/trellis.json
@@ -0,0 +1,192 @@
+{
+ "ports" : {
+ "of:0000000000000204/3" : {
+ "interfaces" : [
+ {
+ "ips" : [ "10.0.2.254/24" ],
+ "vlan-untagged": 20
+ }
+ ]
+ },
+ "of:0000000000000204/4" : {
+ "interfaces" : [
+ {
+ "ips" : [ "10.0.2.254/24" ],
+ "vlan-untagged": 20
+ }
+ ]
+ },
+ "of:0000000000000204/5" : {
+ "interfaces" : [
+ {
+ "ips" : [ "2000::2ff/120" ],
+ "vlan-untagged": 40
+ }
+ ]
+ },
+ "of:0000000000000204/6" : {
+ "interfaces" : [
+ {
+ "ips" : [ "2000::2ff/120" ],
+ "vlan-untagged": 40
+ }
+ ]
+ },
+ "of:0000000000000205/3" : {
+ "interfaces" : [
+ {
+ "ips" : [ "10.0.3.254/24" ],
+ "vlan-untagged": 30
+ }
+ ]
+ },
+ "of:0000000000000205/4" : {
+ "interfaces" : [
+ {
+ "ips" : [ "10.0.3.254/24" ],
+ "vlan-untagged": 30
+ }
+ ]
+ },
+ "of:0000000000000205/5" : {
+ "interfaces" : [
+ {
+ "ips" : [ "2000::3ff/120" ],
+ "vlan-untagged": 50
+ }
+ ]
+ },
+ "of:0000000000000205/6" : {
+ "interfaces" : [
+ {
+ "ips" : [ "2000::3ff/120" ],
+ "vlan-untagged": 50
+ }
+ ]
+ },
+ "of:0000000000000205/7" : {
+ "interfaces" : [
+ {
+ "ips" : [ "10.0.3.254/24" ],
+ "vlan-untagged": 30
+ }
+ ]
+ },
+ "of:0000000000000205/8" : {
+ "interfaces" : [
+ {
+ "ips" : [ "10.0.1.254/24", "2000::1ff/120" ],
+ "vlan-untagged": 10
+ }
+ ]
+ },
+ "of:0000000000000205/9" : {
+ "interfaces" : [
+ {
+ "ips" : [ "10.0.1.254/24", "2000::1ff/120" ],
+ "vlan-untagged": 10
+ }
+ ]
+ }
+ },
+ "devices" : {
+ "of:0000000000000204" : {
+ "segmentrouting" : {
+ "name" : "s204",
+ "ipv4NodeSid" : 204,
+ "ipv4Loopback" : "192.168.0.204",
+ "ipv6NodeSid" : 214,
+ "ipv6Loopback" : "2000::c0a8:0204",
+ "routerMac" : "00:00:00:00:02:04",
+ "isEdgeRouter" : true,
+ "adjacencySids" : []
+ },
+ "basic" : {
+ "name": "s204",
+ "driver" : "ofdpa-ovs"
+ }
+ },
+ "of:0000000000000205" : {
+ "segmentrouting" : {
+ "name" : "s205",
+ "ipv4NodeSid" : 205,
+ "ipv4Loopback" : "192.168.0.205",
+ "ipv6NodeSid" : 215,
+ "ipv6Loopback" : "2000::c0a8:0205",
+ "routerMac" : "00:00:00:00:02:05",
+ "isEdgeRouter" : true,
+ "adjacencySids" : []
+ },
+ "basic" : {
+ "name": "s205",
+ "driver" : "ofdpa-ovs"
+ }
+ },
+ "of:0000000000000226" : {
+ "segmentrouting" : {
+ "name" : "s226",
+ "ipv4NodeSid" : 226,
+ "ipv4Loopback" : "192.168.0.226",
+ "ipv6NodeSid" : 236,
+ "ipv6Loopback" : "2000::c0a8:0226",
+ "routerMac" : "00:00:00:00:02:26",
+ "isEdgeRouter" : false,
+ "adjacencySids" : []
+ },
+ "basic" : {
+ "name": "s226",
+ "driver" : "ofdpa-ovs"
+ }
+ },
+ "of:0000000000000227" : {
+ "segmentrouting" : {
+ "name" : "s227",
+ "ipv4NodeSid" : 227,
+ "ipv4Loopback" : "192.168.0.227",
+ "ipv6NodeSid" : 237,
+ "ipv6Loopback" : "2000::c0a8:0227",
+ "routerMac" : "00:00:00:00:02:27",
+ "isEdgeRouter" : false,
+ "adjacencySids" : []
+ },
+ "basic" : {
+ "name": "s227",
+ "driver" : "ofdpa-ovs"
+ }
+ }
+ },
+ "hosts": {
+ "00:bb:00:00:00:01/None": {
+ "basic": {
+ "ips": ["2000::201"],
+ "location": "of:0000000000000204/5"
+ }
+ },
+ "00:bb:00:00:00:02/None": {
+ "basic": {
+ "ips": ["2000::202"],
+ "location": "of:0000000000000204/6"
+ }
+ },
+ "00:bb:00:00:00:03/None": {
+ "basic": {
+ "ips": ["2000::301"],
+ "location": "of:0000000000000205/5"
+ }
+ },
+ "00:bb:00:00:00:04/None": {
+ "basic": {
+ "ips": ["2000::302"],
+ "location": "of:0000000000000205/6"
+ }
+ }
+ },
+ "apps" : {
+ "org.onosproject.dhcp-relay" : {
+ "dhcprelay" : {
+ "dhcpserverConnectPoint": "of:0000000000000205/7",
+ "serverip": "10.0.3.253"
+ }
+ }
+ }
+}
diff --git a/trellis/trellis.py b/trellis/trellis.py
new file mode 100755
index 0000000..78cdc7b
--- /dev/null
+++ b/trellis/trellis.py
@@ -0,0 +1,139 @@
+#!/usr/bin/python
+
+import sys
+sys.path.append('..')
+from mininet.topo import Topo
+from mininet.net import Mininet
+from mininet.cli import CLI
+from mininet.log import setLogLevel
+from mininet.node import RemoteController, OVSBridge, Host
+from mininet.nodelib import NAT
+from ipaddress import ip_network
+from routinglib import BgpRouter
+from routinglib import RoutedHost
+
+class Trellis( Topo ):
+ "Trellis basic topology"
+
+ def __init__( self, *args, **kwargs ):
+ Topo.__init__( self, *args, **kwargs )
+
+ # Spines
+ s226 = self.addSwitch('s226', dpid='226')
+ s227 = self.addSwitch('s227', dpid='227')
+
+ # Leaves
+ s204 = self.addSwitch('s204', dpid='204')
+ s205 = self.addSwitch('s205', dpid='205')
+
+ # Switch Links
+ self.addLink(s226, s204)
+ self.addLink(s226, s205)
+ self.addLink(s227, s204)
+ self.addLink(s227, s205)
+
+ # NOTE avoid using 10.0.1.0/24 which is the default subnet of quaggas
+ # NOTE avoid using 00:00:00:00:00:xx which is the default mac of host behind upstream router
+ # IPv4 Hosts
+ h1 = self.addHost('h1', cls=DhcpClient, mac='00:aa:00:00:00:01')
+ h2 = self.addHost('h2', cls=DhcpClient, mac='00:aa:00:00:00:02')
+ h3 = self.addHost('h3', cls=DhcpClient, mac='00:aa:00:00:00:03')
+ h4 = self.addHost('h4', cls=DhcpClient, mac='00:aa:00:00:00:04')
+ self.addLink(h1, s204)
+ self.addLink(h2, s204)
+ self.addLink(h3, s205)
+ self.addLink(h4, s205)
+
+ # IPv6 Hosts
+ h1v6 = self.addHost('h1v6', cls=RoutedHost, mac='00:bb:00:00:00:01', ips=['2000::201/120'], gateway='2000::2ff')
+ h2v6 = self.addHost('h2v6', cls=RoutedHost, mac='00:bb:00:00:00:02', ips=['2000::202/120'], gateway='2000::2ff')
+ h3v6 = self.addHost('h3v6', cls=RoutedHost, mac='00:bb:00:00:00:03', ips=['2000::301/120'], gateway='2000::3ff')
+ h4v6 = self.addHost('h4v6', cls=RoutedHost, mac='00:bb:00:00:00:04', ips=['2000::302/120'], gateway='2000::3ff')
+ self.addLink(h1v6, s204)
+ self.addLink(h2v6, s204)
+ self.addLink(h3v6, s205)
+ self.addLink(h4v6, s205)
+
+ # DHCP server
+ dhcp = self.addHost('dhcp', cls=DhcpServer, mac='00:99:00:00:00:01', ips=['10.0.3.253/24'], gateway='10.0.3.254')
+ self.addLink(dhcp, s205)
+
+ # Control plane switch (for quagga fpm)
+ cs0 = self.addSwitch('cs0', cls=OVSBridge)
+
+ # Control plane NAT (for quagga fpm)
+ nat = self.addHost('nat', cls=NAT,
+ ip='172.16.0.1/12',
+ subnet=str(ip_network(u'172.16.0.0/12')), inNamespace=False)
+ self.addLink(cs0, nat)
+
+ # Internal Quagga bgp1
+ intfs = {'bgp1-eth0': {'ipAddrs': ['10.0.1.2/24', '2000::102/120'], 'mac': '00:88:00:00:00:02'},
+ 'bgp1-eth1': {'ipAddrs': ['172.16.0.2/12']}}
+ bgp1 = self.addHost('bgp1', cls=BgpRouter,
+ interfaces=intfs,
+ quaggaConfFile='./bgpdbgp1.conf',
+ zebraConfFile='./zebradbgp1.conf')
+ self.addLink(bgp1, s205)
+ self.addLink(bgp1, cs0)
+
+ # External Quagga r1
+ intfs = {'r1-eth0': {'ipAddrs': ['10.0.1.1/24', '2000::101/120'], 'mac': '00:88:00:00:00:01'},
+ 'r1-eth1': {'ipAddrs': ['10.0.99.1/16']},
+ 'r1-eth2': {'ipAddrs': ['2000::9901/120']}}
+ r1 = self.addHost('r1', cls=BgpRouter,
+ interfaces=intfs,
+ quaggaConfFile='./bgpdr1.conf')
+ self.addLink(r1, s205)
+
+ # External IPv4 Host behind r1
+ rh1 = self.addHost('rh1', cls=RoutedHost, ips=['10.0.99.2/24'], gateway='10.0.99.1')
+ self.addLink(r1, rh1)
+
+ # External IPv6 Host behind r1
+ rh1v6 = self.addHost('rh1v6', cls=RoutedHost, ips=['2000::9902/120'], gateway='2000::9901')
+ self.addLink(r1, rh1v6)
+
+topos = { 'trellis' : Trellis }
+
+class DhcpClient(Host):
+ def __init__(self, name, *args, **kwargs):
+ super(DhcpClient, self).__init__(name, **kwargs)
+ self.pidFile = '/run/dhclient-%s.pid' % self.name
+
+ def config(self, **kwargs):
+ super(DhcpClient, self).config(**kwargs)
+ self.cmd('ip addr flush dev %s' % self.defaultIntf())
+ self.cmd('dhclient -q -4 -nw -pf %s %s' % (self.pidFile, self.defaultIntf()))
+
+ def terminate(self, **kwargs):
+ self.cmd('kill -9 `cat %s`' % self.pidFile)
+ self.cmd('rm -rf %s' % self.pidFile)
+ super(DhcpClient, self).terminate()
+
+class DhcpServer(RoutedHost):
+ binFile = '/usr/sbin/dhcpd'
+ pidFile = '/run/dhcp-server/dhcpd.pid'
+ configFile = './dhcpd.conf'
+
+ def config(self, **kwargs):
+ super(DhcpServer, self).config(**kwargs)
+ self.cmd('%s -q -4 -pf %s -cf %s %s' % (self.binFile, self.pidFile, self.configFile, self.defaultIntf()))
+
+ def terminate(self, **kwargs):
+ self.cmd('kill -9 `cat %s`' % self.pidFile)
+ self.cmd('rm -rf %s' % self.pidFile)
+ super(DhcpServer, self).terminate()
+
+if __name__ == "__main__":
+ setLogLevel('debug')
+ topo = Trellis()
+
+ net = Mininet(topo=topo, controller=None)
+ net.addController(RemoteController('c0', ip='192.168.56.11'))
+ net.addController(RemoteController('c1', ip='192.168.56.12'))
+ net.addController(RemoteController('c2', ip='192.168.56.13'))
+
+ net.start()
+ CLI(net)
+ net.stop()
diff --git a/trellis/zebradbgp1.conf b/trellis/zebradbgp1.conf
new file mode 100644
index 0000000..51991a4
--- /dev/null
+++ b/trellis/zebradbgp1.conf
@@ -0,0 +1,9 @@
+log file /var/log/quagga/zebradbgp1.log
+hostname zebra-bgp1
+password quagga
+!
+! Default route via virtual management switch
+!
+ip route 0.0.0.0/0 172.16.0.1
+!
+fpm connection ip 192.168.56.11 port 2620