.. |onos_version| replace:: 1.12.2

Installation
************

Topology
========
Trellis supports various topology, from a single switch to a full multi-stage leaf-spine fabric.

- **Single switch**: This is the minimum Trellis setup. In this setup, all servers are connected to a single switch.
- **Paired switches**: Compared to single switch, it provides more redundancy in terms of server NIC failure and link failure.
- **Single-stage leaf-spine**: Compared to single switch, it offers more redundancy in terms of switch failure and provides better scalability.
- **Single-stage leaf-spine with paired switches**: It supports all the redundancy and scalability features mentioned above.
- **Multi-stage leaf-spine**: Multi-stage is specifically designed for telco service providers.
  The first stage can be installed in the central office, while the second stage can be installed in a field office that is closer to the subscribers.
  Two stages are typically connected via long distance optical transport.


Trellis in a Box (TiaB)
=======================
TiaB is a software emulated environment that allows people to run Trellis without hardware switches.
TiaB comes with a few Mininet scripts that emulate network topology from a simple 2x2 leaf-spine to a full multi-stage fabric.
The configurations corresponding to each topology are also packaged together such that you can start the environment right away.

TiaB is currently hosted in a GitHub repository separated from ONOS.
The source code and user guide of TiaB can be found `here <https://github.com/opennetworkinglab/routing/tree/master/trellis>`_.

.. tip::
    TiaB is the most convenient and **recommended option for beginners who would like to quickly get some hands on experience** of Trellis.


Hardware Requirements
=====================
Following is a list of hardware that people from the ONF community have tested over time in lab trials.

Switches
--------

1G/10G models (with 40G uplinks)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    - OCP Accepted™ EdgeCore AS5712-54X
    - OCP Accepted™ EdgeCore AS5812-54X
    - QCT QuantaMesh T3048-LY8
    - Delta AG7648
    - Inventec D6254 (verified by Inventec)

25G models (with 100G uplinks)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    - QuantaMesh BMS T7032-IX1/IX1B (with 25G breakout cable)
    - Inventec D7054Q28B (verified by Inventec)

40G models
^^^^^^^^^^
    - OCP Accepted™ EdgeCore AS6712-32X

100G models
^^^^^^^^^^^
    - OCP Accepted™ EdgeCore AS7712-32X
    - QCT QuantaMesh BMS T7032-IX1/IX1B
    - OCP Accepted™ Inventec D7032Q28B (verified by Inventec)

Commercially supported models
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Models listed here are not supported by ONF community. You should contact switch vendor for commercial support.
    - OCP Accepted™ EdgeCore AS5912-54X


Servers
-------
ONOS can be run on any x86 servers, with the following notes on memory and CPU allocation:

    - Memory
        The memory requirement highly depends on the deployment scale.
        In general, we suggest that each ONOS instance should get **at least 16G RAM**.
        For production deployment at scale, we will definitely need not only more memory but also some fine tuning on JVM garbage collection mechanism.
    - CPU
        There are not much requirement on CPU.
        In practice, we avoid running other CPU intensive processes/containers/VMs on the same physical machine where ONOS runs.


Install Controller - ONOS
=========================
You could run Trellis with a `single instance of ONOS <https://wiki.onosproject.org/display/ONOS/Local+workflow>`_. But it is recommended to run ONOS as a cluster.

The idea is to have a "build machine", where you host and build ONOS source code. But to launch it in operation you use other "target machines" (VMs, containers or servers).
Typically we use `STC <https://wiki.onosproject.org/pages/viewpage.action?pageId=12421025>`_ to launch ONOS ins 3 target machines which form a ONOS cluster.


Download ONOS |onos_version|
---------------------------------
Trellis currently is released as part of ONOS and therefore it follows ONOS version number.
This document is written based on ONOS |onos_version|.

You can find more information about how to setup build environment and fetch ONOS source code from `Development Environment Setup <https://wiki.onosproject.org/display/ONOS/Development+Environment+Setup>`_.


Prepare Your Target Machine
---------------------------
Please refer to `target machines requirements <https://wiki.onosproject.org/display/ONOS/Cells+and+ONOS+test+scripts#CellsandONOStestscripts-Targetmachinesrequirements>`_ for details (you only have to do this once).

You do not need to follow the directions for the "Mininet Target Machine" if you are using hardware switches.


Create a Cell File
--------------------
We need to compose a cell file in order to give the information about target machines to the build machine. Please refer to `test cells <https://wiki.onosproject.org/display/ONOS/Cells+and+ONOS+test+scripts#CellsandONOStestscripts-TestCells>`_ for details. For example:

.. code-block:: bash

    export ONOS_CELL=menlo
    export OC1=10.128.0.216
    export OC2=10.128.0.217
    export OC3=10.128.0.218
    export ONOS_APPS=drivers,gui,openflow,segmentrouting,fpm,dhcprelay,routeradvertisement,hostprobingprovider,t3
    export ONOS_USER=admin
    export ONOS_GROUP=admin
    export ONOS_USE_SSH=true
    export ONOS_WEB_USER=onos
    export ONOS_WEB_PASS=rocks

- ``OC1, OC2, OC3`` are the IP addresses of the 3 target machines where the ONOS cluster will be deployed.
- ``ONOS_APPS`` are the apps you want to automatically deploy at launch.

    - ``drivers`` includes drivers for various devices and pipelines. Always required.
    - ``gui`` enables graphic user interface. Highly recommended.
    - ``openflow`` is a meta app that loads ``openflow-base``, ``lldpprovider`` and ``hostprovider``. Always required.
    - ``segmentrouting`` controls forwarding in the fabric. Always required.
    - ``fpm`` (Forwarding Plane Manager) exchanges forwarding information with Quagga. Required if connecting to external router.
    - ``dhcprelay`` relays DHCP packets between clients and servers. Required if using DHCP to configure IP addresses for hosts
    - ``routeradvertisement`` periodically sends IPv6 router advertisement packets on configured interfaces. Required if using IPv6
    - ``hostprobingprovider`` probes and verifies locations of dual-homed hosts. Required if using dual-homing feature with paired switches
    - ``t3`` (Trellis Troubleshooting Tool) is very useful for debugging. Highly recommended.

- ``ONOS_USER`` is used to login to the target machines,
  and the password is not required as you have setup the target machines for password-less access over ssh.

Once that's done, you can use the ``cell`` command to load the cell file.

.. code-block:: console

    $ source onos/tools/dev/bash_profile
    $ cell menlo


Check Your Target Environment
-----------------------------
Check if your environment is OK with ``stc prerequisites`` (you only have to do this once).

.. code-block:: console

    $ stc prerequisites
    2017-02-07 12:10:23  Prerequisites started
    2017-02-07 12:10:23  Check-Passwordless-Login-1 started -- ssh -n -o ConnectTimeout=3 -o PasswordAuthentication=no admin@10.128.0.216 date
    2017-02-07 12:10:23  Check-ONOS-Bits started -- onos-check-bits
    2017-02-07 12:10:23  Check-Passwordless-Login-3 started -- ssh -n -o ConnectTimeout=3 -o PasswordAuthentication=no admin@10.128.0.218 date
    2017-02-07 12:10:23  Check-Passwordless-Login-2 started -- ssh -n -o ConnectTimeout=3 -o PasswordAuthentication=no admin@10.128.0.217 date
    2017-02-07 12:10:23  Check-Environment started -- test -n /Users/sauravdas/onos -a -n 10.128.0.* -a -n 10.128.0.216
    2017-02-07 12:10:23  Check-Environment completed
    2017-02-07 12:10:23  Check-Passwordless-Login-1 completed
    2017-02-07 12:10:23  Check-Passwordless-Login-2 completed
    2017-02-07 12:10:23  Check-Passwordless-Login-3 completed
    2017-02-07 12:10:25  Check-ONOS-Bits completed
    2017-02-07 12:10:25  Prerequisites completed
    0:01 Passed! 6 steps succeeded


Launch STC Command
------------------
Finally launch with ``stc setup`` once you have built ONOS. Here is what it looks like when I launch

.. code-block:: console

    $ stc setup
    2016-12-23 12:26:44  Setup started
    2016-12-23 12:26:44  Uninstall-3 started -- onos-uninstall 10.128.0.218
    2016-12-23 12:26:44  Uninstall-2 started -- onos-uninstall 10.128.0.217
    2016-12-23 12:26:44  Uninstall-1 started -- onos-uninstall 10.128.0.216
    2016-12-23 12:26:44  Push-Bits-2 started -- onos-push-bits 10.128.0.217
    2016-12-23 12:26:44  Push-Bits-3 started -- onos-push-bits 10.128.0.218
    2016-12-23 12:26:44  Push-Bits-1 started -- onos-push-bits 10.128.0.216
    2016-12-23 12:26:45  Push-Bits-1 completed
    2016-12-23 12:26:46  Push-Bits-2 completed
    2016-12-23 12:26:46  Push-Bits-3 completed
    2016-12-23 12:26:46  Uninstall-3 completed
    2016-12-23 12:26:46  Kill-3 started -- onos-kill 10.128.0.218
    2016-12-23 12:26:46  Uninstall-2 completed
    2016-12-23 12:26:46  Kill-2 started -- onos-kill 10.128.0.217
    2016-12-23 12:26:46  Uninstall-1 completed
    2016-12-23 12:26:46  Kill-1 started -- onos-kill 10.128.0.216
    2016-12-23 12:26:46  Kill-3 completed
    2016-12-23 12:26:46  Install-3 started -- onos-install 10.128.0.218
    2016-12-23 12:26:46  Kill-1 completed
    2016-12-23 12:26:46  Install-1 started -- onos-install 10.128.0.216
    2016-12-23 12:26:46  Kill-2 completed
    2016-12-23 12:26:46  Install-2 started -- onos-install 10.128.0.217
    2016-12-23 12:26:52  Install-2 completed
    2016-12-23 12:26:52  Secure-SSH-2 started -- onos-secure-ssh -u onos -p rocks 10.128.0.217
    2016-12-23 12:26:52  Install-1 completed
    2016-12-23 12:26:52  Secure-SSH-1 started -- onos-secure-ssh -u onos -p rocks 10.128.0.216
    2016-12-23 12:26:52  Install-3 completed
    2016-12-23 12:26:52  Secure-SSH-3 started -- onos-secure-ssh -u onos -p rocks 10.128.0.218
    2016-12-23 12:27:07  Secure-SSH-1 completed
    2016-12-23 12:27:07  Wait-for-Start-1 started -- onos-wait-for-start 10.128.0.216
    2016-12-23 12:27:09  Secure-SSH-3 completed
    2016-12-23 12:27:09  Wait-for-Start-3 started -- onos-wait-for-start 10.128.0.218
    2016-12-23 12:27:13  Secure-SSH-2 completed
    2016-12-23 12:27:13  Wait-for-Start-2 started -- onos-wait-for-start 10.128.0.217
    2016-12-23 12:27:14  Wait-for-Start-1 completed
    2016-12-23 12:27:14  Check-Components-1 started -- onos-check-components 10.128.0.216
    2016-12-23 12:27:14  Check-Nodes-1 started -- onos-check-nodes 10.128.0.216
    2016-12-23 12:27:14  Wait-for-Start-3 completed
    2016-12-23 12:27:14  Check-Nodes-3 started -- onos-check-nodes 10.128.0.218
    2016-12-23 12:27:14  Check-Components-3 started -- onos-check-components 10.128.0.218
    2016-12-23 12:27:16  Wait-for-Start-2 completed
    2016-12-23 12:27:16  Check-Nodes-2 started -- onos-check-nodes 10.128.0.217
    2016-12-23 12:27:16  Check-Components-2 started -- onos-check-components 10.128.0.217
    2016-12-23 12:27:18  Check-Nodes-1 completed
    2016-12-23 12:27:18  Check-Nodes-3 completed
    2016-12-23 12:27:19  Check-Nodes-2 completed
    2016-12-23 12:27:20  Check-Components-1 completed
    2016-12-23 12:27:20  Check-Apps-1 started -- onos-check-apps 10.128.0.216 drivers,gui,openflow,segmentrouting,fpm,dhcprelay,routeradvertisement,t3,hostprobingprovider includes
    2016-12-23 12:27:20  Check-Logs-1 started -- onos-check-logs 10.128.0.216
    2016-12-23 12:27:20  Check-Components-3 completed
    2016-12-23 12:27:20  Check-Apps-3 started -- onos-check-apps 10.128.0.218 drivers,gui,openflow,segmentrouting,fpm,dhcprelay,routeradvertisement,t3,hostprobingprovider includes
    2016-12-23 12:27:20  Check-Logs-3 started -- onos-check-logs 10.128.0.218
    2016-12-23 12:27:20  Check-Apps-1 completed
    2016-12-23 12:27:20  Check-Apps-3 completed
    2016-12-23 12:27:20  Check-Logs-1 completed
    2016-12-23 12:27:20  Check-Logs-3 completed
    2016-12-23 12:27:21  Check-Components-2 completed
    2016-12-23 12:27:21  Check-Logs-2 started -- onos-check-logs 10.128.0.217
    2016-12-23 12:27:21  Check-Apps-2 started -- onos-check-apps 10.128.0.217 drivers,gui,openflow,segmentrouting,fpm,dhcprelay,routeradvertisement,t3,hostprobingprovider includes
    2016-12-23 12:27:21  Check-Apps-2 completed
    2016-12-23 12:27:21  Check-Logs-2 completed
    2016-12-23 12:27:21  Setup completed
    0:36 Passed! 31 steps succeeded


Useful Utilities
----------------
- Push network configuration to one particular instance of ONOS (all instances will automatically get the same configuration)

    .. code-block:: console

        $ onos-netcfg $OC1 network-cfg.json

- Remotely log in to CLI of one particular ONOS instance

    .. code-block:: console

        $ onos $OC1

- Remotely access the log of one particular ONOS instance

    .. code-block:: console

        $ onos-log $OC1


Install Switch OS - ONL
=======================
The switches listed in the `Hardware Requirements`_ section are shipped with Open Networking Install Environment (ONIE) boot loader.
After booting up, we should see the ONIE prompt from console.
Here we assume that the management port on the switch already has Internet access. (via DHCP)


Enter ONIE
----------


Has no ONL or Has ONL 1.x previously installed
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The way to install ONL 2.x is the same as ONL 1.x.
However, if you have no ONL installed or have older version of ONL, you might find it tricky to (re)install a newer version.
Here's the instruction:

    1. Plug in the console cable and reboot the switch
    2. (If your boot loader is grub) When you see the boot menu, select **ONIE -> ONIE: Rescue**
    3. (If your boot loader is uboot) When you see **Hit any key to stop autoboot** instead of the boot menu,
       press any key and then enter ``run onie_rescue`` to enter ONIE rescue mode.

Has ONL 2.x previously installed
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
It would be much more easier to reboot into ONIE if you have a previously installed ONL 2. Simply run:


.. code-block:: console

    # onl-onie-boot-mode rescue
    # reboot

Available ONIE modes are: *install, rescue, uninstall, update, embed, diag, none*.
This can be helpful when you are in a normal ONL and want to upgrade your system.


Install ONL
-----------
At the ONIE prompt, we need to download and install ONL. To fetch and install the latest compatible ONL, run:

.. code-block:: console

    ONIE:/ # wget https://github.com/opencord/OpenNetworkLinux/releases/download/2017-10-19.2200-1211610/ONL-2.0.0_ONL-OS_2017-10-19.2200-1211610_AMD64_INSTALLED_INSTALLER
    ONIE:/ # sh ONL-2.0.0_ONL-OS_2017-10-19.2200-1211610_AMD64_INSTALLED_INSTALLER

**Checksum**: *sha256:2db316ea83f5dc761b9b11cc8542f153f092f3b49d82ffc0a36a2c41290f5421*

The switch will automatically reboot into ONL after installation. Default login credential of ONL is: ``root/onl``

We might want to configure a fixed IP address for the management interface.
First edit ``/etc/network/interfaces`` and configure ``ma1``, which is the management interface.

.. code-block:: text

    auto ma1
    iface ma1 inet static
    address 10.128.10.128
    netmask 255.255.0.0
    gateway 10.128.0.1
    dns-nameservers 192.168.1.1 8.8.8.8


Install Switch Agent - OF-DPA
=============================


Community vs. Premium Version
-----------------------------
The OF-DPA image we distribute for free with Trellis is a community version.
This documentation is also written based on the community version.
The community version has most of the features available and therefore it is good for small scale deployments such as lab trials.
However, we highly recommend you to get the premium version from Broadcom if you are aiming for production deployments at scale.


Install OF-DPA
--------------
We need to use different OF-DPA images depending on the switch model you are using.
Please find the installer URL corresponding to each switch model in the `Vendor Specific Information`_ section.

Copy the image to the switch and start the installation process by running:

.. code-block:: console

    $ scp ${OFDPA_DEB} ${SWITCH_IP}:/root
    $ dpkg -i --force-overwrite /root/${OFDPA_DEB}

For example, assuming the OF-DPA image is ``ofdpa_3.0.5.5+accton1.7-1_amd64.deb`` and the switch management IP is ``10.128.0.201``, you should run:

.. code-block:: console

    $ scp ofdpa_3.0.5.5+accton1.7-1_amd64.deb 10.128.0.201:/root
    $ dpkg -i --force-overwrite /root/ofdpa_3.0.5.5+accton1.7-1_amd64.deb


Connect Switch to Controller
============================

.. caution::
    We are going to describe two different ways to start OF-DPA and OpenFlow agent in this section.
    Some vendors use ``ofagentd`` service while others use old ``launcher`` app to start the agent.
    Please check the `Vendor Specific Information`_ to understand which method you should use before continuing this section.

Use ofagentd service
--------------------

Launch with ofagentd
^^^^^^^^^^^^^^^^^^^^
The OFDPA software and the Indigo agent are now a single process launched by Linux service. First of all, we need to configure **/etc/ofagent/ofagent.conf**.
Uncomment and edit the following line. Make sure all other lines stay commented.

.. code-block:: bash

    OPT_ARGS="-t <controller_ip_1> -t <controller_ip_2> -t <controller_ip_3> -i <dpid>"

We can choose to run OF-DPA in **debug mode**

.. code-block:: bash

    OPT_ARGS="-d 2 -c 2 -c 4 -t <controller_ip_1> -t <controller_ip_2> -t <controller_ip_3> -i <dpid>"

- To **start** the ofagent, run ``service ofagentd start``
- To **stop** the agent, run ``service ofagentd stop``
- More ``ofagentapp`` options can be found by running ``ofagentapp --help``


Launch in listen mode with ofagentd
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
An optional ``-l`` parameter can be added for switch to listen on specific **IP:port**.
This allows us to use tools like **ovs-ofctl** or **dpctl** to connect and control the switch.

.. code-block:: bash

    OPT_ARGS="-t <controller_ip_1> -t <controller_ip_2> -t <controller_ip_3> -i <dpid> -l <ip>:<port>"

Please refer to `Connect to Switch in Listen Mode`_ to learn more about how to access switches in listen mode.

Use launcher app
----------------

Launch with launcher
^^^^^^^^^^^^^^^^^^^^
- To **start** OF-DPA and OpenFlow agent, run:

  .. code-block:: console

      # ./launcher ofagentapp -t <controller_ip_1> -t <controller_ip_2> -t <controller_ip_3> -i <dpid>

- We can choose to run OF-DPA in **debug mode**

  .. code-block:: console

     # ./launcher ofagentapp -d 2 -c 2 -c 4 -t <controller_ip_1> -t <controller_ip_2> -t <controller_ip_3> -i <dpid>

- To **stop** OF-DPA and OpenFlow agent, run:

  .. code-block:: consol

     # killall ofagentapp

Launch in listen mode with launcher
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
An optional ``-l`` parameter can be added for switch to listen on specific **IP:port**.
This allows us to use tools like **ovs-ofctl** or **dpctl** to connect and control the switch.

.. code-block:: console

    # ./launcher ofagentapp <controller_ip_1> -t <controller_ip_2> -t <controller_ip_3> -i <dpid> -l <ip>:<port>

Please refer to `Connect to Switch in Listen Mode`_ to learn more about how to access switches in listen mode.


Vendor Specific Information
===========================


Edgecore
--------

- **OF-DPA Image**
    - `EdgeCore 5712-54X / 5812-54X / 6712-32X / 7712-32X <https://raw.githubusercontent.com/onfsdn/atrium-docs/master/16A/ONOS/builds/ofdpa_3.0.5.5%2Baccton1.7-1_amd64.deb>`_ - **Checksum**: *sha256:db228b6e79fb15f77497b59689235606b60abc157e72fc3356071bcc8dc4c01f*

- **Start OF-DPA and OpenFlow agent**
    - Please refer to `Use ofagentd service`_

- **Configure Port Speed and Breakout**
    - By default all the switch ports are running at maximum speed. The port speed can be modified in **/etc/accton/ofdpa.conf**

      .. code-block:: text

          port_speed_1=1000   # configure front port 1 to run at 1G

    - We can also configure the same file to use break out cables on certain ports.

      .. code-block:: text

          port_mode_1=4x10g    # configure front port 1 to break out from 40G to 4x10G

    OF-DPA service needs to be restarted after modifying this configuration.


QCT
---

- OF-DPA Image
    - `QuantaMesh T3048-LY8 <https://raw.githubusercontent.com/onfsdn/atrium-docs/master/16A/ONOS/builds/ofdpa-ag7648_0.3.0.5.6_amd64.deb>`_ - **Checksum**: *sha256:f8201530b1452145c1a0956ea1d3c0402c3568d090553d0d7b3c91a79137da9e*
    - `QuantaMesh T7032-IX1 / IX1B <https://raw.githubusercontent.com/onfsdn/atrium-docs/master/16A/ONOS/builds/ofdpa-ix1_0.3.0.5.0-EA5-qct-01.00_amd64.deb>`_ **Checksum**: *sha256:278b8ffed8a8fc705a1b60d16f8e70377e78342a27a11568a1d80b1efd706a46*

- **Start OF-DPA and OpenFlow agent**
    - Please refer to `Use launcher app`_

- Configure Port Speed and Breakout
    The command ``client_drivshell`` can be used to configure the port speed.
    Use the parameter ``ps`` to list all ports or ``ps <port number>`` to list one specific port.

    .. code-block:: console

        # client_drivshell ps 1
        Calling ofdpaBcmCommand rpc with command = "ps 1 ".
                         ena/    speed/ link auto    STP                  lrn  inter   max  loop
                   port  link    duplex scan neg?   state   pause  discrd ops   face frame  back
               xe0(  1)  down   10G  FD   SW  No   Forward         Untag    F    SFI  9412

        Returned from ofdpaBcmCommand rpc with rc = 0.

    Then use the parameter ``port <port number> sp=<port speed>`` to change the port speed.

    .. code-block:: console

        # client_drivshell port xe0 sp=1000
        Calling ofdpaBcmCommand rpc with command = "port xe0 sp=1000 ".

        Returned from ofdpaBcmCommand rpc with rc = 0.

        # client_drivshell ps 1
        Calling ofdpaBcmCommand rpc with command = "ps 1 ".
                         ena/    speed/ link auto    STP                  lrn  inter   max  loop
                   port  link    duplex scan neg?   state   pause  discrd ops   face frame  back
               xe0(  1)  down    1G  FD   SW  No   Forward         Untag    F   GMII  9412

        Returned from ofdpaBcmCommand rpc with rc = 0.

Delta
-----

- OF-DPA Image
    - `Delta AG7648 <https://raw.githubusercontent.com/onfsdn/atrium-docs/master/16A/ONOS/builds/ofdpa-ly8_0.3.0.5.0-EA5-qct-01.01_amd64.deb>`_ - **Checksum**: *sha256:ddfc13cb98ca47291dce5e6938b1d65f0b99bbe77f0585e36ac0007017397f23*

- **Start OF-DPA and OpenFlow agent**
    - Please refer to `Use launcher app`_

- Special instructions to install ONL
    Make sure ``/etc/machine.conf`` looks like the following in ONIE before running ONL installation script:

    .. code-block:: text

        onie_platform=x86_64-delta_<platform name>-r0
        onie_machine=delta_<platform name>

    After the installation of ONL, if you don't see '/usr/bin' in your PATH variable, please run the following command:

    .. code-block:: console

        # export PATH=$PATH:/usr/bin


Useful Information
==================

OF-DPA Commands
---------------
There are some useful OF-DPA commands under ``/usr/bin/``

.. code-block:: text

    client_cfg_purge
    client_debugcomp
    client_drivshell
    client_flowtable_dump
    client_meter_dump
    client_port_table_dump
    client_tunnel_dump
    client_classcolortable_dump
    client_debuglvl
    client_event
    client_grouptable_dump
    client_oam_dump
    client_queue_config


ONL Boot Mode
-------------
OFDPA offers two boot mode: **INSTALLED** and **SWI**.

- In **INSTALLED** mode, ONL mounts the root filesystem from /dev/sdb7 which is a flash drive that persists everything even during a power cycle.
- In **SWI** mode, ONL will load the root filesystem from a read-only image (\*.swi) and put the modified files in an copy-on-write overlay filesystem.
  The change will not be persisted during a power cycle.
  The pros and cons are obvious. INSTALLED mode is more convenient while SWI mode is more error-safe.

The boot mode will be determined by which image we used install ONL. We use the **INSTALLED** mode in this instruction for simplicity.
But we can still change it after the installation. Here's the instruction:

.. code-block:: console

    # mount /mnt/onl/boot -o remount,rw
    # sed -i 's/BOOTMODE=INSTALLED/BOOTMODE=SWI/g' /mnt/onl/boot/boot-config
    # reboot


Persistence Mechanism
---------------------
INSTALLED mode already persists everything for you. This section is mainly for SWI mode.

The mechanism to persist files on the switch is different from ONL 1.x to ONL 2.x.
Following are the steps required to persist a file (e.g. OFDPA package) in ONL 2.x.

- Files put in **/mnt/onl/data** will be persisted automatically.
- To install a deb package during start-up, put the deb file under **/mnt/onl/data/install-debs** folder and
  create a plain text file **/mnt/onl/data/install-debs/list** with all the filename of the deb (e.g. ofdpa_3.0.5.5+accton1.7-1_amd64.deb)
- To execute some command during start-up, put the command in **/mnt/onl/data/rc.boot**

  .. code-block:: bash

      # set static ip and route (dhcp by default)
      echo 'ip addr add 10.128.0.203/16 dev ma1' >> /mnt/onl/data/rc.boot
      echo 'ip route add default via 10.128.0.1' >> /mnt/onl/data/rc.boot
      # grant executable permission
      chmod a+x /mnt/onl/data/rc.boot

See `PersistWorkflow.md <https://github.com/opencomputeproject/OpenNetworkLinux/blob/master/docs/PersistWorkflow.md>`_ for more detail.


Build a Customized ONL Image
----------------------------
Sometimes we need to build our own ONL image from source to include some extra files. (The most common case is a modified /etc/network/interfaces).
The instruction of build process has already been well-documented in ONL repo `Building.md <https://github.com/opencomputeproject/OpenNetworkLinux/blob/master/docs/Building.md>`_.
But you might find it a little bit tricky to inject some extra files.
To do that, you need to put the files under **$ONL_ROOT/builds/any/rootfs/jessie/common/overlay/** and then run the make command again.


Recovery from a Faulty ONL Install
----------------------------------
If, for some reason, the ONL install process fails, you may be brought to the grub rescue prompt upon reboot.
You may or may not find system files, or even basic grub rescue commands (i.e. 'help').
There are two options for returning to a (known working) ONIE prompt.

1. **Reinstaller** - You should be able to acquire an ISO image for resetting the firmware to factory default from your switch vendor.
   Once you have the ISO, you can either use PXE or a live USB to boot the switch into the image.
2. **Manually boot to ONIE** - If the files are there, you can manually configure GRUB to load the necessary modules to boot back into ONIE.

   .. code-block:: console

       # This should be the partition on the ONIE partition on the AS6712.
       # It might be hd1 or hd0. Change appropriately.
       grub rescue> ls (hd0,gpt2)/
       ./ ../ lost+found/ onie/ grub/ grubenv

       # These instructions will load the kernel and ONIE initrd.
       grub rescue> set prefix=(hd0,gpt2)/grub
       grub rescue> set root=(hd0,gpt2)
       grub rescue> insmod normal
       grub rescue> insmod linux
       grub rescue> ls /onie
       ./ ../ vmlinuz-3.2.35-onie initrd.img-3.2.35-onie tools/ grub/ grub.d/ config/

       # if the ",115200n8" is omitted, baud rate defaults to 9600.
       grub rescue> linux /onie/vmlinuz-3.2.35-onie console=tty0 console=ttyS1,115200n8
       grub rescue> initrd /onie/initrd.img-3.2.35-onie

       # The system will not boot into ONIE and you can recover or re-install from there.
       grub rescue> boot

Once back at the ONIE prompt, you can try to install ONL by following the same steps again.


Connect to Switch in Listen Mode
--------------------------------
To connect to the switch in Listen Mode using **ovs-ofctl**, you can use commands such as

.. code-block:: console

    $ ovs-ofctl -O OpenFlow13 show tcp:<ip>:<port>
    $ ovs-ofctl -O OpenFlow13 dump-flows tcp:<ip>:<port>
    $ ovs-ofctl -O OpenFlow13 dump-groups tcp:<ip>:<port>
    $ ovs-ofctl -O OpenFlow13 add-flow tcp:<ip>:<port> table=60,priority=40000,eth_type=0x0800,ip_dst=55.55.55.55,actions=controller

For more command, please see ``ovs-ofctl --help`` or the post `OpenvSwitch ovs-ofctl and OF-DPA <https://blog.pichuang.com.tw/20170106-ovs-ofctl-and-ofdpa/>`_.


Reference
---------
- `ONL/OF-DPA installation guide by Edgecore Networks <https://www.youtube.com/watch?v=mCUtkIc_hVI>`_
- `ONL/OF-DPA cheat sheet by Phil Huang <https://blog.pichuang.com.tw/20170104-ofdpa-with-onl-cheat-sheet/>`_