*** Settings ***
Documentation     ONOS Switch Scale Test
Suite Setup       ONOS Suite Setup    ${CONTROLLER_IP}    ${CONTROLLER_USER}
Suite Teardown    ONOS Suite Teardown
Library           SSHLibrary
Library           Collections
Library           OperatingSystem
Library           String
Library           RequestsLibrary
Library           HttpLibrary.HTTP

*** Variables ***
##Grab the environment variables sources from your "cell"
${CONTROLLER_IP}    %{OC1}
${MININET_IP}    %{OCN}
${CONTROLLER_USER}    %{ONOS_USER}
${MININET_USER}    %{ONOS_USER}
##USER_HOME used for public key
${USER_HOME}    %{HOME}
##ONOS_HOME is where the onos dist will be deployed on the controller vm
${ONOS_HOME}    /opt/onos
${RESTCONFPORT}    8181
${LINUX_PROMPT}    $
##SWITCHES_RESULT_FILE and JENKINS_WORKSPACE can be configurable...read overriding variables in README
##SWITCHES_RESULT_FILE is used to plot data. you can use a jenkins post job for this or do it manually
##NOTE: This file must exist otherwise the test will fail when trying to write to this file
${SWITCHES_RESULT_FILE}    ${USER_HOME}/workspace/tools/switches.csv
${JENKINS_WORKSPACE}    ${USER_HOME}/workspace/ONOS-Stable/
${prompt_timeout}    30s
${start}    10
${end}    100
${increments}    10
##Number of nodes in cluster. To add more nodes, create CONTROLLER_IP2/3/4 etc. variables above and change this cluster variable
${cluster}    1

*** Test Cases ***
Find Max Switches By Scaling
    [Documentation]    Find the max number of switches from ${start} until reaching ${end} in steps of ${increments}. The following checks are made through REST APIs:
    ...    -\ Verify device count is correct
    ...    -\ Verify device status is available
    ...    -\ Verify device roles are MASTER (default role in case of standalone controller)
    ...    -\ Verify topology recognizes corret number of devices (Through api "/topology")
    ...    -\ Observe each device individually
    ...    -\ Observe links, hosts, and flows through the controller
    ...    -\ Observe device info at lower level on mininet (Written for PoC). Shows flows, links, and ports. Checks can be easily implemented at that level as well
    ...    -\ STOP Mininet Topo
    ...    -\ Verify device count is zero
    ...    -\ Verify topology sees no devices (Through api "/topology")
    [Tags]    done
    Append To File    ${SWITCHES_RESULT_FILE}    Max Switches Linear Topo\n
    ${max}=    Find Max Switches    ${start}    ${end}    ${increments}
    Log    ${max}
    Append To File    ${SWITCHES_RESULT_FILE}    ${max}\n

*** Keywords   ***
ONOS Suite Setup
    [Arguments]    ${controller}    ${user}
    [Documentation]    Transfers the ONOS dist over to the test vm and start the controller. We will leverage the bash script, "onos-install" to do this.
    Create Controller IP List
    ${rc}=    Run and Return RC    onos-package
    Should Be Equal As Integers    ${rc}    0
    : FOR    ${ip}    IN    @{controller_list}
    \    ${rc}=    Run and Return RC    onos-install -f ${ip}
    \    Should Be Equal As Integers    ${rc}    0
    Create HTTP Sessions
    Wait Until Keyword Succeeds    60s    2s    Verify All Controller are up
    #If creating a cluster, create a keyword  and call it here

ONOS Suite Teardown
    [Documentation]    Stop ONOS on Controller VMs and grabs karaf logs on each controller (put them in /tmp)
    ${rc}=    Run and Return RC    onos-kill
    #Should Be Equal As Integers    ${rc}    0
    ${rc}=    Run and Return RC    cp ${SWITCHES_RESULT_FILE} ${JENKINS_WORKSPACE}
    Should Be Equal As Integers    ${rc}    0
    ${rc}=    Run and Return RC    rm ${SWITCHES_RESULT_FILE}
    Should Be Equal As Integers    ${rc}    0
    Clean Mininet System
    : FOR    ${ip}    IN    @{controller_list}
    \    Get Karaf Logs    ${ip}

Create Controller IP List
    [Documentation]    Creates a list of controller ips for a cluster. When creating a cluster, be sure to set each variable to %{OC} env vars in the variables section 
    @{controller_list}=    Create List    ${CONTROLLER_IP}
    Set Suite Variable    @{controller_list}

Create HTTP Sessions
    [Documentation]    Creates an http session with all controllers in the cluster. Session names are set to respective ips. 
    ${HEADERS}=    Create Dictionary    Content-Type    application/json
    : FOR    ${ip}    IN    @{controller_list}
    \    Create Session    ${ip}    http://${ip}:${RESTCONFPORT}    headers=${HEADERS}

Find Max Switches
    [Arguments]    ${start}    ${stop}    ${step}
    [Documentation]    Will find out max switches starting from ${start} till reaching ${stop} and in steps defined by ${step}
    ${max-switches}    Set Variable    ${0}
    ${start}    Convert to Integer    ${start}
    ${stop}    Convert to Integer    ${stop}
    ${step}    Convert to Integer    ${step}
    : FOR    ${switches}    IN RANGE    ${start}    ${stop+1}    ${step}
    \    Start Mininet Linear    ${switches}
    \    ${status}    ${result}    Run Keyword And Ignore Error    Verify Controllers are Not Dead
    \    Exit For Loop If    '${status}' == 'FAIL'
    \    ${status}    ${result}    Run Keyword And Ignore Error    Wait Until Keyword Succeeds    ${switches*2}    2s
    \    ...    Ensure Switch Count    ${switches}
    \    Exit For Loop If    '${status}' == 'FAIL'
    \    ${status}    ${result}    Run Keyword And Ignore Error    Wait Until Keyword Succeeds    ${switches*2}    2s
    \    ...    Ensure Switches are Available
    \    Exit For Loop If    '${status}' == 'FAIL'
    \    ${status}    ${result}    Run Keyword And Ignore Error    Wait Until Keyword Succeeds    ${switches*2}    2s
    \    ...    Ensure Switch Role    MASTER
    \    Exit For Loop If    '${status}' == 'FAIL'
    \    ${status}    ${result}    Run Keyword And Ignore Error    Wait Until Keyword Succeeds    ${switches*2}    2s
    \    ...    Ensure Topology    ${switches}    ${cluster}
    \    Exit For Loop If    '${status}' == 'FAIL'
    \    ${status}    ${result}    Run Keyword And Ignore Error    Wait Until Keyword Succeeds    ${switches*2}    2s
    \    ...    Experiment Links, Hosts, and Flows
    \    ${status}    ${result}    Run Keyword And Ignore Error    Wait Until Keyword Succeeds    ${switches*2}    2s
    \    ...    Check Each Switch Individually
    \    ${status}    ${result}    Run Keyword And Ignore Error    Wait Until Keyword Succeeds    ${switches*2}    2s
    \    ...    Check Mininet at Lower Level
    \    Stop Mininet
    \    ${status}    ${result}    Run Keyword And Ignore Error    Wait Until Keyword Succeeds    ${switches*2}    2s
    \    ...    Ensure No Switches are Available
    \    Exit For Loop If    '${status}' == 'FAIL'
    \    ${status}    ${result}    Run Keyword And Ignore Error    Wait Until Keyword Succeeds    ${switches*2}    2s
    \    ...    Ensure No Switches in Topology    ${cluster}
    \    Exit For Loop If    '${status}' == 'FAIL'
    \    ${max-switches}    Convert To String    ${switches}
    [Return]    ${max-switches}

Run Command On Remote System
    [Arguments]    ${remote_system}    ${cmd}    ${user}=${CONTROLLER_USER}    ${prompt}=${LINUX_PROMPT}    ${prompt_timeout}=30s
    [Documentation]    Reduces the common work of running a command on a remote system to a single higher level robot keyword,
    ...    taking care to log in with a public key and the command given is written and the output returned. No test conditions
    ...    are checked.
    Log    Attempting to execute ${cmd} on ${remote_system}
    ${conn_id}=    SSHLibrary.Open Connection    ${remote_system}    prompt=${prompt}    timeout=${prompt_timeout}
    Login With Public Key    ${user}    ${USER_HOME}/.ssh/id_rsa    any
    SSHLibrary.Write    ${cmd}
    ${output}=    SSHLibrary.Read Until    ${LINUX_PROMPT}
    SSHLibrary.Close Connection
    Log    ${output}
    [Return]    ${output}

Start Mininet Linear
    [Arguments]    ${switches}
    [Documentation]    Start mininet linear topology with ${switches} nodes
    Log To Console    \n
    Log To Console    Starting mininet linear ${switches}
    ${mininet_conn_id}=    Open Connection    ${MININET_IP}    prompt=${LINUX_PROMPT}    timeout=${switches*3}
    Set Suite Variable    ${mininet_conn_id}
    Login With Public Key    ${MININET_USER}    ${USER_HOME}/.ssh/id_rsa    any
    Write    sudo mn --controller=remote,ip=${CONTROLLER_IP} --topo linear,${switches} --switch ovsk,protocols=OpenFlow13
    Read Until    mininet>
    Sleep    6

Stop Mininet
    [Documentation]    Stop mininet topology
    Log To Console    Stopping Mininet
    Switch Connection    ${mininet_conn_id}
    Read
    Write    exit
    Read Until    ${LINUX_PROMPT}
    Close Connection

Check Mininet at Lower Level
    [Documentation]    PoC for executing mininet commands at the lower level
    Switch Connection    ${mininet_conn_id}
    Read
    Write    dpctl dump-flows -O OpenFlow13
    ${output}=    Read Until    mininet>
    Log    ${output}
    Write    dump
    ${output}=    Read Until    mininet>
    Log    ${output}
    Write    links
    ${output}=    Read Until    mininet>
    Log    ${output}
    Write    ports
    ${output}=    Read Until    mininet>
    Log    ${output}

Clean Mininet System
    [Arguments]     ${mininet_system}=${MININET_IP}
    [Documentation]    Cleans the mininet environment (sudo mn -c)
    Run Command On Remote System    ${mininet_system}   sudo mn -c    ${CONTROLLER_USER}    ${LINUX_PROMPT}    600s

Verify All Controller are up
    [Documentation]    Verifies each controller is up by issuing a rest call and getting a 200
    : FOR    ${ip}    IN    @{controller_list}
    \    ${resp}=    ONOS Get    ${ip}    devices
    \    Should Be Equal As Strings    ${resp.status_code}    200

Verify Controllers are Not Dead
    [Documentation]    Verifies each controller is not dead by making sure karaf log does not contain "OutOfMemoryError" and rest call still returns 200 
    : FOR    ${ip}    IN    @{controller_list}
    \    Verify Controller Is Not Dead    ${ip}

Verify Controller Is Not Dead
    [Arguments]    ${controller}
    ${response}=    Run Command On Remote System    ${controller}   grep java.lang.OutOfMemoryError /opt/onos/log/karaf.log
    Should Not Contain    ${response}    OutOfMemoryError
    ${resp}    RequestsLibrary.Get    ${controller}    /onos/v1/devices
    Log    ${resp.content}
    Should Be Equal As Strings    ${resp.status_code}    200

Experiment Links, Hosts, and Flows
    [Documentation]    Currently this only returns the information the controller has on links, hosts, and flows. Checks can easily be implemented
    : FOR    ${ip}    IN    @{controller_list}
    \    ${resp}=    ONOS Get    ${ip}    links
    \    ${jsondata}    To JSON    ${resp.content}
    \    ${length}=    Get Length    ${jsondata['links']}
    \    Log    ${resp.content}
    \    ${resp}=    ONOS Get    ${ip}    flows
    \    ${jsondata}    To JSON    ${resp.content}
    \    ${length}=    Get Length    ${jsondata['flows']}
    \    Log    ${resp.content}
    \    ${resp}=    ONOS Get    ${ip}    hosts
    \    ${jsondata}    To JSON    ${resp.content}
    \    ${length}=    Get Length    ${jsondata['hosts']}
    \    Log    ${resp.content}

Ensure Topology
    [Arguments]    ${device_count}    ${cluster_count}
    [Documentation]    Verifies the devices count through /topoplogy api. Currently, cluster count is inconsistent (Possible bug, will look into it), so the check on that is ignored, but logged
    : FOR    ${ip}    IN    @{controller_list}
    \    ${resp}=    ONOS Get    ${ip}    topology
    \    Log    ${resp.content}
    \    ${devices}=    Get Json Value    ${resp.content}    /devices
    \    ${clusters}=    Get Json Value    ${resp.content}    /clusters
    \    Should Be Equal As Strings    ${devices}    ${device_count}
    \    #Should Be Equal As Strings    ${clusters}    ${cluster_count}

Ensure No Switches in Topology
    [Arguments]    ${cluster_count}
    [Documentation]    Verifies the topology sees no devices
    : FOR    ${ip}    IN    @{controller_list}
    \    ${resp}=    ONOS Get    ${ip}    topology
    \    Log    ${resp.content}
    \    ${devices}=    Get Json Value    ${resp.content}    /devices
    \    ${clusters}=    Get Json Value    ${resp.content}    /clusters
    \    Should Be Equal As Strings    ${devices}    0
    \    #Should Be Equal As Strings    ${clusters}    ${cluster_count}

ONOS Get
    [Arguments]    ${session}    ${noun}
    [Documentation]    Common keyword to issue GET requests to the controller. Arguments are the session (controller ip) and api path
    ${resp}    RequestsLibrary.Get    ${session}    /onos/v1/${noun}
    Log    ${resp.content}
    [Return]    ${resp}

Ensure Switch Count
    [Arguments]    ${switch_count}
    [Documentation]    Verfies the device count (passed in as arg) on each controller is accurate.
    : FOR    ${ip}    IN    @{controller_list}
    \    ${resp}    ONOS Get    ${ip}    devices
    \    Log    ${resp.content}
    \    Should Be Equal As Strings    ${resp.status_code}    200
    \    Should Not Be Empty    ${resp.content}
    \    ${jsondata}    To JSON    ${resp.content}
    \    ${length}=    Get Length    ${jsondata['devices']}
    \    Should Be Equal As Integers    ${length}    ${switch_count}

Ensure Switches are Available
    [Documentation]    Verifies that the switch's availability state is true on all switches through all controllers
    : FOR    ${ip}    IN    @{controller_list}
    \    ${resp}    ONOS Get    ${ip}    devices
    \    Log    ${resp.content}
    \    Should Be Equal As Strings    ${resp.status_code}    200
    \    Should Not Be Empty    ${resp.content}
    \    ${jsondata}    To JSON    ${resp.content}
    \    #Robot tweak to do a nested for loop
    \    Check Each Switch Status    ${jsondata}    True

Check Each Switch Status
    [Arguments]    ${jdata}    ${bool}
    ${length}=    Get Length    ${jdata['devices']}
    : FOR    ${INDEX}    IN RANGE    0    ${length}
    \    ${data}=    Get From List    ${jdata['devices']}    ${INDEX}
    \    ${status}=    Get From Dictionary    ${data}    available
    \    Should Be Equal As Strings    ${status}    ${bool}

Ensure No Switches are Available
    [Documentation]    Verifies that the switch's availability state is false on all switches through all controllers
    : FOR    ${ip}    IN    @{controller_list}
    \    ${resp}    ONOS Get    ${ip}    devices
    \    Log    ${resp.content}
    \    Should Be Equal As Strings    ${resp.status_code}    200
    \    Should Not Be Empty    ${resp.content}
    \    ${jsondata}    To JSON    ${resp.content}
    \    Check Each Switch Status    ${jsondata}    False

Check Each Switch Individually
    [Documentation]    Currently just observe each information the device has. Checks can easily be implemented
    : FOR    ${ip}    IN    @{controller_list}
    \    ${resp}    ONOS Get    ${ip}    devices
    \    Should Be Equal As Strings    ${resp.status_code}    200
    \    ${jsondata}    To JSON    ${resp.content}
    \    #Robot tweak to do a nested for loop
    \    Check Each Switch    ${jsondata}

Check Each Switch
    [Arguments]    ${jdata}
    ${length}=    Get Length    ${jdata['devices']}
    @{dpid_list}=    Create List
    : FOR    ${INDEX}    IN RANGE    0    ${length}
    \    ${devicedata}=    Get From List    ${jdata['devices']}    ${INDEX}
    \    ${id}=    Get From Dictionary    ${devicedata}    id
    \    Append To List    ${dpid_list}    ${id}
    \    Log    ${dpid_list}
    ${length}=    Get Length    ${dpid_list}
    : FOR    ${i}    IN    @{dpid_list}
    \    ${resp}    ONOS Get    ${ip}    devices/${i}
    \    Log    ${resp.content}

Ensure Switch Role
    [Arguments]    ${role}
    [Documentation]    Verifies that the controller's role for each switch is MASTER (default in standalone mode)
    : FOR    ${ip}    IN    @{controller_list}
    \    ${resp}    ONOS Get    ${ip}    devices
    \    Log    ${resp.content}
    \    Should Be Equal As Strings    ${resp.status_code}    200
    \    Should Not Be Empty    ${resp.content}
    \    ${jsondata}    To JSON    ${resp.content}
    \    Ensure Role    ${jsondata}    ${role}

Ensure Role
    [Arguments]    ${jdata}    ${role}
    ${length}=    Get Length    ${jdata['devices']}
    : FOR    ${INDEX}    IN RANGE    0    ${length}
    \    ${data}=    Get From List    ${jdata['devices']}    ${INDEX}
    \    ${status}=    Get From Dictionary    ${data}    role
    \    Should Be Equal As Strings    ${status}    ${role}

Get Karaf Logs
    [Arguments]    ${controller}
    [Documentation]    Compresses all the karaf log files on each controler and puts them on your pybot execution machine (in /tmp)
    Run Command On Remote System    ${controller}    tar -zcvf /tmp/${SUITE NAME}.${controller}.tar.gz ${ONOS_HOME}/log
    SSHLibrary.Open Connection    ${controller}    prompt=${LINUX_PROMPT}    timeout=${prompt_timeout}
    Login With Public Key    ${CONTROLLER_USER}    ${USER_HOME}/.ssh/id_rsa    any
    SSHLibrary.Get File    /tmp/${SUITE NAME}.${controller}.tar.gz    /tmp/
    SSHLibrary.Close Connection