Add HApowerFailure test
This requires at least one patch to ONOS for the `onos-power` script to
support non-default cell usernames and another patch to the onos warden
to allow multiple node failures.
Also included:
- logging changes to help debug multithreadded sections of the test.
- Some input validation in functions that don't directly call the cli
- Remove some verbose logging
- Distribute some onos commands amongst the active nodes
- Refactor out clearing the ONOS cli pexpect buffer before sending a
command into it's own function
Change-Id: If1b868b399878209ab0394956f3b3918c0176909
diff --git a/TestON/tests/HA/HApowerFailure/README b/TestON/tests/HA/HApowerFailure/README
new file mode 100644
index 0000000..069e5af
--- /dev/null
+++ b/TestON/tests/HA/HApowerFailure/README
@@ -0,0 +1,26 @@
+This test is designed to verify that an ONOS cluster behaves correctly when
+ONOS nodes die due to power failures. Currently, we will kill nodes so that
+each raft partition will lose a member, but we make sure that there is always
+a majority of nodes available in each partition.
+
+As written, the test only supports an ONOS cluster of 3,5, or 7 nodes.
+This is because the test doesn't apply to a single node cluster, ONOS clusters
+should be deployed in odd numbers, and the partition generation and node
+killing scheme used doesn't give the same properties for clusters of more
+than 7 nodes. Namely, each partition won't have exactly one node killed.
+
+The gerneral structure for the test:
+- Startup
+- Assign switches
+- Verify ONOS state and functionality
+ - Device mastership
+ - Intents
+ - Leadership election
+ - Distributed Primitives
+- Kill some ONOS nodes
+- Verify ONOS state and functionality
+- Restart ONOS nodes
+- Verify ONOS state and functionality
+- Dataplane failures
+ - link down and up
+ - switch down and up