blob: 2ebea504deefda90e92a6c7ab7de92ebbcbf813a [file] [log] [blame]
.. |onos_version| replace:: 1.12.2
Installation Guide
Building a Hardware Pod
See :doc:`Supported Hardware <supported-hardware>` for details.
Install Controller - ONOS
You could run Trellis with a `single instance of ONOS <>`_. But it is recommended to run ONOS as a cluster.
The idea is to have a "build machine", where you host and build ONOS source code. But to launch it in operation you use other "target machines" (VMs, containers or servers).
Typically we use `STC <>`_ to launch ONOS ins 3 target machines which form a ONOS cluster.
Download ONOS |onos_version|
Trellis currently is released as part of ONOS and therefore it follows ONOS version number.
This document is written based on ONOS |onos_version|.
You can find more information about how to setup build environment and fetch ONOS source code from `Development Environment Setup <>`_.
Prepare Your Target Machine
Please refer to `target machines requirements <>`_ for details (you only have to do this once).
You do not need to follow the directions for the "Mininet Target Machine" if you are using hardware switches.
Create a Cell File
We need to compose a cell file in order to give the information about target machines to the build machine. Please refer to `test cells <>`_ for details. For example:
.. code-block:: bash
export ONOS_CELL=menlo
export OC1=
export OC2=
export OC3=
export ONOS_APPS=drivers,gui,openflow,segmentrouting,fpm,dhcprelay,routeradvertisement,hostprobingprovider,t3
export ONOS_USER=admin
export ONOS_GROUP=admin
export ONOS_USE_SSH=true
export ONOS_WEB_USER=onos
export ONOS_WEB_PASS=rocks
- ``OC1, OC2, OC3`` are the IP addresses of the 3 target machines where the ONOS cluster will be deployed.
- ``ONOS_APPS`` are the apps you want to automatically deploy at launch.
- ``drivers`` includes drivers for various devices and pipelines. Always required.
- ``gui`` enables graphic user interface. Highly recommended.
- ``openflow`` is a meta app that loads ``openflow-base``, ``lldpprovider`` and ``hostprovider``. Always required.
- ``segmentrouting`` controls forwarding in the fabric. Always required.
- ``fpm`` (Forwarding Plane Manager) exchanges forwarding information with Quagga. Required if connecting to external router.
- ``dhcprelay`` relays DHCP packets between clients and servers. Required if using DHCP to configure IP addresses for hosts
- ``routeradvertisement`` periodically sends IPv6 router advertisement packets on configured interfaces. Required if using IPv6
- ``hostprobingprovider`` probes and verifies locations of dual-homed hosts. Required if using dual-homing feature with paired switches
- ``t3`` (Trellis Troubleshooting Tool) is very useful for debugging. Highly recommended.
- ``ONOS_USER`` is used to login to the target machines,
and the password is not required as you have setup the target machines for password-less access over ssh.
Once that's done, you can use the ``cell`` command to load the cell file.
.. code-block:: console
$ source onos/tools/dev/bash_profile
$ cell menlo
Check Your Target Environment
Check if your environment is OK with ``stc prerequisites`` (you only have to do this once).
.. code-block:: console
$ stc prerequisites
2017-02-07 12:10:23 Prerequisites started
2017-02-07 12:10:23 Check-Passwordless-Login-1 started -- ssh -n -o ConnectTimeout=3 -o PasswordAuthentication=no admin@ date
2017-02-07 12:10:23 Check-ONOS-Bits started -- onos-check-bits
2017-02-07 12:10:23 Check-Passwordless-Login-3 started -- ssh -n -o ConnectTimeout=3 -o PasswordAuthentication=no admin@ date
2017-02-07 12:10:23 Check-Passwordless-Login-2 started -- ssh -n -o ConnectTimeout=3 -o PasswordAuthentication=no admin@ date
2017-02-07 12:10:23 Check-Environment started -- test -n /Users/sauravdas/onos -a -n 10.128.0.* -a -n
2017-02-07 12:10:23 Check-Environment completed
2017-02-07 12:10:23 Check-Passwordless-Login-1 completed
2017-02-07 12:10:23 Check-Passwordless-Login-2 completed
2017-02-07 12:10:23 Check-Passwordless-Login-3 completed
2017-02-07 12:10:25 Check-ONOS-Bits completed
2017-02-07 12:10:25 Prerequisites completed
0:01 Passed! 6 steps succeeded
Launch STC Command
Finally launch with ``stc setup`` once you have built ONOS. Here is what it looks like when I launch
.. code-block:: console
$ stc setup
2016-12-23 12:26:44 Setup started
2016-12-23 12:26:44 Uninstall-3 started -- onos-uninstall
2016-12-23 12:26:44 Uninstall-2 started -- onos-uninstall
2016-12-23 12:26:44 Uninstall-1 started -- onos-uninstall
2016-12-23 12:26:44 Push-Bits-2 started -- onos-push-bits
2016-12-23 12:26:44 Push-Bits-3 started -- onos-push-bits
2016-12-23 12:26:44 Push-Bits-1 started -- onos-push-bits
2016-12-23 12:26:45 Push-Bits-1 completed
2016-12-23 12:26:46 Push-Bits-2 completed
2016-12-23 12:26:46 Push-Bits-3 completed
2016-12-23 12:26:46 Uninstall-3 completed
2016-12-23 12:26:46 Kill-3 started -- onos-kill
2016-12-23 12:26:46 Uninstall-2 completed
2016-12-23 12:26:46 Kill-2 started -- onos-kill
2016-12-23 12:26:46 Uninstall-1 completed
2016-12-23 12:26:46 Kill-1 started -- onos-kill
2016-12-23 12:26:46 Kill-3 completed
2016-12-23 12:26:46 Install-3 started -- onos-install
2016-12-23 12:26:46 Kill-1 completed
2016-12-23 12:26:46 Install-1 started -- onos-install
2016-12-23 12:26:46 Kill-2 completed
2016-12-23 12:26:46 Install-2 started -- onos-install
2016-12-23 12:26:52 Install-2 completed
2016-12-23 12:26:52 Secure-SSH-2 started -- onos-secure-ssh -u onos -p rocks
2016-12-23 12:26:52 Install-1 completed
2016-12-23 12:26:52 Secure-SSH-1 started -- onos-secure-ssh -u onos -p rocks
2016-12-23 12:26:52 Install-3 completed
2016-12-23 12:26:52 Secure-SSH-3 started -- onos-secure-ssh -u onos -p rocks
2016-12-23 12:27:07 Secure-SSH-1 completed
2016-12-23 12:27:07 Wait-for-Start-1 started -- onos-wait-for-start
2016-12-23 12:27:09 Secure-SSH-3 completed
2016-12-23 12:27:09 Wait-for-Start-3 started -- onos-wait-for-start
2016-12-23 12:27:13 Secure-SSH-2 completed
2016-12-23 12:27:13 Wait-for-Start-2 started -- onos-wait-for-start
2016-12-23 12:27:14 Wait-for-Start-1 completed
2016-12-23 12:27:14 Check-Components-1 started -- onos-check-components
2016-12-23 12:27:14 Check-Nodes-1 started -- onos-check-nodes
2016-12-23 12:27:14 Wait-for-Start-3 completed
2016-12-23 12:27:14 Check-Nodes-3 started -- onos-check-nodes
2016-12-23 12:27:14 Check-Components-3 started -- onos-check-components
2016-12-23 12:27:16 Wait-for-Start-2 completed
2016-12-23 12:27:16 Check-Nodes-2 started -- onos-check-nodes
2016-12-23 12:27:16 Check-Components-2 started -- onos-check-components
2016-12-23 12:27:18 Check-Nodes-1 completed
2016-12-23 12:27:18 Check-Nodes-3 completed
2016-12-23 12:27:19 Check-Nodes-2 completed
2016-12-23 12:27:20 Check-Components-1 completed
2016-12-23 12:27:20 Check-Apps-1 started -- onos-check-apps drivers,gui,openflow,segmentrouting,fpm,dhcprelay,routeradvertisement,t3,hostprobingprovider includes
2016-12-23 12:27:20 Check-Logs-1 started -- onos-check-logs
2016-12-23 12:27:20 Check-Components-3 completed
2016-12-23 12:27:20 Check-Apps-3 started -- onos-check-apps drivers,gui,openflow,segmentrouting,fpm,dhcprelay,routeradvertisement,t3,hostprobingprovider includes
2016-12-23 12:27:20 Check-Logs-3 started -- onos-check-logs
2016-12-23 12:27:20 Check-Apps-1 completed
2016-12-23 12:27:20 Check-Apps-3 completed
2016-12-23 12:27:20 Check-Logs-1 completed
2016-12-23 12:27:20 Check-Logs-3 completed
2016-12-23 12:27:21 Check-Components-2 completed
2016-12-23 12:27:21 Check-Logs-2 started -- onos-check-logs
2016-12-23 12:27:21 Check-Apps-2 started -- onos-check-apps drivers,gui,openflow,segmentrouting,fpm,dhcprelay,routeradvertisement,t3,hostprobingprovider includes
2016-12-23 12:27:21 Check-Apps-2 completed
2016-12-23 12:27:21 Check-Logs-2 completed
2016-12-23 12:27:21 Setup completed
0:36 Passed! 31 steps succeeded
Useful Utilities
- Push network configuration to one particular instance of ONOS (all instances will automatically get the same configuration)
.. code-block:: console
$ onos-netcfg $OC1 network-cfg.json
- Remotely log in to CLI of one particular ONOS instance
.. code-block:: console
$ onos $OC1
- Remotely access the log of one particular ONOS instance
.. code-block:: console
$ onos-log $OC1
Install Switch OS - ONL
The switches listed in the :doc:`Supported Hardware <supported-hardware>` are shipped with Open Networking Install Environment (ONIE) boot loader.
After booting up, we should see the ONIE prompt from console.
Here we assume that the management port on the switch already has Internet access. (via DHCP)
Enter ONIE
Has no ONL or Has ONL 1.x previously installed
The way to install ONL 2.x is the same as ONL 1.x.
However, if you have no ONL installed or have older version of ONL, you might find it tricky to (re)install a newer version.
Here's the instruction:
1. Plug in the console cable and reboot the switch
2. (If your boot loader is grub) When you see the boot menu, select **ONIE -> ONIE: Rescue**
3. (If your boot loader is uboot) When you see **Hit any key to stop autoboot** instead of the boot menu,
press any key and then enter ``run onie_rescue`` to enter ONIE rescue mode.
Has ONL 2.x previously installed
It would be much more easier to reboot into ONIE if you have a previously installed ONL 2. Simply run:
.. code-block:: console
# onl-onie-boot-mode rescue
# reboot
Available ONIE modes are: *install, rescue, uninstall, update, embed, diag, none*.
This can be helpful when you are in a normal ONL and want to upgrade your system.
Install ONL
At the ONIE prompt, we need to download and install ONL. To fetch and install the latest compatible ONL, run:
.. code-block:: console
ONIE:/ # wget
ONIE:/ # sh ONL-2.0.0_ONL-OS_2017-10-19.2200-1211610_AMD64_INSTALLED_INSTALLER
**Checksum**: *sha256:2db316ea83f5dc761b9b11cc8542f153f092f3b49d82ffc0a36a2c41290f5421*
The switch will automatically reboot into ONL after installation. Default login credential of ONL is: ``root/onl``
We might want to configure a fixed IP address for the management interface.
First edit ``/etc/network/interfaces`` and configure ``ma1``, which is the management interface.
.. code-block:: text
auto ma1
iface ma1 inet static
Install Switch Agent - OF-DPA
.. note::
**Community vs. Premium Version**
The OF-DPA image we distribute for free with Trellis is a community version.
This documentation is also written based on the community version.
The **community version** has most of the features available and therefore it is **good for small scale deployments** such as lab trials.
However, we highly recommend you to get the **premium version** from Broadcom if you are aiming for **production deployments at scale**.
Install OF-DPA
We need to use different OF-DPA images depending on the switch model you are using.
Please find the installer URL corresponding to each switch model in the `Vendor Specific Information`_ section.
Copy the image to the switch and start the installation process by running:
.. code-block:: console
$ scp ${OFDPA_DEB} ${SWITCH_IP}:/root
$ dpkg -i --force-overwrite /root/${OFDPA_DEB}
For example, assuming the OF-DPA image is ``ofdpa_3.0.5.5+accton1.7-1_amd64.deb`` and the switch management IP is ````, you should run:
.. code-block:: console
$ scp ofdpa_3.0.5.5+accton1.7-1_amd64.deb
$ dpkg -i --force-overwrite /root/ofdpa_3.0.5.5+accton1.7-1_amd64.deb
Connect Switch to Controller
.. caution::
We are going to describe two different ways to start OF-DPA and OpenFlow agent in this section.
Some vendors use ``ofagentd`` service while others use old ``launcher`` app to start the agent.
Please check the `Vendor Specific Information`_ to understand which method you should use before continuing this section.
Use ofagentd service
Launch with ofagentd
The OFDPA software and the Indigo agent are now a single process launched by Linux service. First of all, we need to configure **/etc/ofagent/ofagent.conf**.
Uncomment and edit the following line. Make sure all other lines stay commented.
.. code-block:: bash
OPT_ARGS="-t <controller_ip_1> -t <controller_ip_2> -t <controller_ip_3> -i <dpid>"
We can choose to run OF-DPA in **debug mode**
.. code-block:: bash
OPT_ARGS="-d 2 -c 2 -c 4 -t <controller_ip_1> -t <controller_ip_2> -t <controller_ip_3> -i <dpid>"
- To **start** the ofagent, run ``service ofagentd start``
- To **stop** the agent, run ``service ofagentd stop``
- More ``ofagentapp`` options can be found by running ``ofagentapp --help``
Launch in listen mode with ofagentd
An optional ``-l`` parameter can be added for switch to listen on specific **IP:port**.
This allows us to use tools like **ovs-ofctl** or **dpctl** to connect and control the switch.
.. code-block:: bash
OPT_ARGS="-t <controller_ip_1> -t <controller_ip_2> -t <controller_ip_3> -i <dpid> -l <ip>:<port>"
Please refer to `Connect to Switch in Listen Mode`_ to learn more about how to access switches in listen mode.
Use launcher app
Launch with launcher
- To **start** OF-DPA and OpenFlow agent, run:
.. code-block:: console
# ./launcher ofagentapp -t <controller_ip_1> -t <controller_ip_2> -t <controller_ip_3> -i <dpid>
- We can choose to run OF-DPA in **debug mode**
.. code-block:: console
# ./launcher ofagentapp -d 2 -c 2 -c 4 -t <controller_ip_1> -t <controller_ip_2> -t <controller_ip_3> -i <dpid>
- To **stop** OF-DPA and OpenFlow agent, run:
.. code-block:: consol
# killall ofagentapp
Launch in listen mode with launcher
An optional ``-l`` parameter can be added for switch to listen on specific **IP:port**.
This allows us to use tools like **ovs-ofctl** or **dpctl** to connect and control the switch.
.. code-block:: console
# ./launcher ofagentapp <controller_ip_1> -t <controller_ip_2> -t <controller_ip_3> -i <dpid> -l <ip>:<port>
Please refer to `Connect to Switch in Listen Mode`_ to learn more about how to access switches in listen mode.
Vendor Specific Information
- **OF-DPA Image**
- `EdgeCore 5712-54X / 5812-54X / 6712-32X / 7712-32X <>`_ - **Checksum**: *sha256:db228b6e79fb15f77497b59689235606b60abc157e72fc3356071bcc8dc4c01f*
- **Start OF-DPA and OpenFlow agent**
- Please refer to `Use ofagentd service`_
- **Configure Port Speed and Breakout**
- By default all the switch ports are running at maximum speed. The port speed can be modified in **/etc/accton/ofdpa.conf**
.. code-block:: text
port_speed_1=1000 # configure front port 1 to run at 1G
- We can also configure the same file to use break out cables on certain ports.
.. code-block:: text
port_mode_1=4x10g # configure front port 1 to break out from 40G to 4x10G
OF-DPA service needs to be restarted after modifying this configuration.
- OF-DPA Image
- `QuantaMesh T3048-LY8 <>`_ - **Checksum**: *sha256:f8201530b1452145c1a0956ea1d3c0402c3568d090553d0d7b3c91a79137da9e*
- `QuantaMesh T7032-IX1 / IX1B <>`_ **Checksum**: *sha256:278b8ffed8a8fc705a1b60d16f8e70377e78342a27a11568a1d80b1efd706a46*
- **Start OF-DPA and OpenFlow agent**
- Please refer to `Use launcher app`_
- Configure Port Speed and Breakout
The command ``client_drivshell`` can be used to configure the port speed.
Use the parameter ``ps`` to list all ports or ``ps <port number>`` to list one specific port.
.. code-block:: console
# client_drivshell ps 1
Calling ofdpaBcmCommand rpc with command = "ps 1 ".
ena/ speed/ link auto STP lrn inter max loop
port link duplex scan neg? state pause discrd ops face frame back
xe0( 1) down 10G FD SW No Forward Untag F SFI 9412
Returned from ofdpaBcmCommand rpc with rc = 0.
Then use the parameter ``port <port number> sp=<port speed>`` to change the port speed.
.. code-block:: console
# client_drivshell port xe0 sp=1000
Calling ofdpaBcmCommand rpc with command = "port xe0 sp=1000 ".
Returned from ofdpaBcmCommand rpc with rc = 0.
# client_drivshell ps 1
Calling ofdpaBcmCommand rpc with command = "ps 1 ".
ena/ speed/ link auto STP lrn inter max loop
port link duplex scan neg? state pause discrd ops face frame back
xe0( 1) down 1G FD SW No Forward Untag F GMII 9412
Returned from ofdpaBcmCommand rpc with rc = 0.
- OF-DPA Image
- `Delta AG7648 <>`_ - **Checksum**: *sha256:ddfc13cb98ca47291dce5e6938b1d65f0b99bbe77f0585e36ac0007017397f23*
- **Start OF-DPA and OpenFlow agent**
- Please refer to `Use launcher app`_
- Special instructions to install ONL
Make sure ``/etc/machine.conf`` looks like the following in ONIE before running ONL installation script:
.. code-block:: text
onie_platform=x86_64-delta_<platform name>-r0
onie_machine=delta_<platform name>
After the installation of ONL, if you don't see '/usr/bin' in your PATH variable, please run the following command:
.. code-block:: console
# export PATH=$PATH:/usr/bin
Useful Information
OF-DPA Commands
There are some useful OF-DPA commands under ``/usr/bin/``
.. code-block:: text
ONL Boot Mode
OFDPA offers two boot mode: **INSTALLED** and **SWI**.
- In **INSTALLED** mode, ONL mounts the root filesystem from /dev/sdb7 which is a flash drive that persists everything even during a power cycle.
- In **SWI** mode, ONL will load the root filesystem from a read-only image (\*.swi) and put the modified files in an copy-on-write overlay filesystem.
The change will not be persisted during a power cycle.
The pros and cons are obvious. INSTALLED mode is more convenient while SWI mode is more error-safe.
The boot mode will be determined by which image we used install ONL. We use the **INSTALLED** mode in this instruction for simplicity.
But we can still change it after the installation. Here's the instruction:
.. code-block:: console
# mount /mnt/onl/boot -o remount,rw
# sed -i 's/BOOTMODE=INSTALLED/BOOTMODE=SWI/g' /mnt/onl/boot/boot-config
# reboot
Persistence Mechanism
INSTALLED mode already persists everything for you. This section is mainly for SWI mode.
The mechanism to persist files on the switch is different from ONL 1.x to ONL 2.x.
Following are the steps required to persist a file (e.g. OFDPA package) in ONL 2.x.
- Files put in **/mnt/onl/data** will be persisted automatically.
- To install a deb package during start-up, put the deb file under **/mnt/onl/data/install-debs** folder and
create a plain text file **/mnt/onl/data/install-debs/list** with all the filename of the deb (e.g. ofdpa_3.0.5.5+accton1.7-1_amd64.deb)
- To execute some command during start-up, put the command in **/mnt/onl/data/rc.boot**
.. code-block:: bash
# set static ip and route (dhcp by default)
echo 'ip addr add dev ma1' >> /mnt/onl/data/rc.boot
echo 'ip route add default via' >> /mnt/onl/data/rc.boot
# grant executable permission
chmod a+x /mnt/onl/data/rc.boot
See ` <>`_ for more detail.
Build a Customized ONL Image
Sometimes we need to build our own ONL image from source to include some extra files. (The most common case is a modified /etc/network/interfaces).
The instruction of build process has already been well-documented in ONL repo ` <>`_.
But you might find it a little bit tricky to inject some extra files.
To do that, you need to put the files under **$ONL_ROOT/builds/any/rootfs/jessie/common/overlay/** and then run the make command again.
Recovery from a Faulty ONL Install
If, for some reason, the ONL install process fails, you may be brought to the grub rescue prompt upon reboot.
You may or may not find system files, or even basic grub rescue commands (i.e. 'help').
There are two options for returning to a (known working) ONIE prompt.
1. **Reinstaller** - You should be able to acquire an ISO image for resetting the firmware to factory default from your switch vendor.
Once you have the ISO, you can either use PXE or a live USB to boot the switch into the image.
2. **Manually boot to ONIE** - If the files are there, you can manually configure GRUB to load the necessary modules to boot back into ONIE.
.. code-block:: console
# This should be the partition on the ONIE partition on the AS6712.
# It might be hd1 or hd0. Change appropriately.
grub rescue> ls (hd0,gpt2)/
./ ../ lost+found/ onie/ grub/ grubenv
# These instructions will load the kernel and ONIE initrd.
grub rescue> set prefix=(hd0,gpt2)/grub
grub rescue> set root=(hd0,gpt2)
grub rescue> insmod normal
grub rescue> insmod linux
grub rescue> ls /onie
./ ../ vmlinuz-3.2.35-onie initrd.img-3.2.35-onie tools/ grub/ grub.d/ config/
# if the ",115200n8" is omitted, baud rate defaults to 9600.
grub rescue> linux /onie/vmlinuz-3.2.35-onie console=tty0 console=ttyS1,115200n8
grub rescue> initrd /onie/initrd.img-3.2.35-onie
# The system will not boot into ONIE and you can recover or re-install from there.
grub rescue> boot
Once back at the ONIE prompt, you can try to install ONL by following the same steps again.
Connect to Switch in Listen Mode
To connect to the switch in Listen Mode using **ovs-ofctl**, you can use commands such as
.. code-block:: console
$ ovs-ofctl -O OpenFlow13 show tcp:<ip>:<port>
$ ovs-ofctl -O OpenFlow13 dump-flows tcp:<ip>:<port>
$ ovs-ofctl -O OpenFlow13 dump-groups tcp:<ip>:<port>
$ ovs-ofctl -O OpenFlow13 add-flow tcp:<ip>:<port> table=60,priority=40000,eth_type=0x0800,ip_dst=,actions=controller
For more command, please see ``ovs-ofctl --help`` or the post `OpenvSwitch ovs-ofctl and OF-DPA <>`_.
- `ONL/OF-DPA installation guide by Edgecore Networks <>`_
- `ONL/OF-DPA cheat sheet by Phil Huang <>`_