Bug fixes for ONOS-3509

- Forwarding behavior added to {Device,Link}Store by ONOS-490
  cauesed false update information sent from ONOS node, which has been detached from the cluster,
  to be accepted by rest of the cluster after the detached node has rejoined cluster.

- Fix for periodic mastership check was left out
  when MastershipService#requestRoleFor(..) return value was changed to Future.

- Fix for triggerProbe() related messages getting dropped,
  right after STANDBY -> MASTER role change.

- Local state (connectedDevices) was preventing
  vertical (Core -> switch) Mastership state synchronization.

- Various debug log, comment added during investigation.

Change-Id: I777beadf04db8a879830a07bfdc7ab0e2279f190
diff --git a/protocols/openflow/api/src/main/java/org/onosproject/openflow/controller/driver/AbstractOpenFlowSwitch.java b/protocols/openflow/api/src/main/java/org/onosproject/openflow/controller/driver/AbstractOpenFlowSwitch.java
index 098ff07..a877a66 100644
--- a/protocols/openflow/api/src/main/java/org/onosproject/openflow/controller/driver/AbstractOpenFlowSwitch.java
+++ b/protocols/openflow/api/src/main/java/org/onosproject/openflow/controller/driver/AbstractOpenFlowSwitch.java
@@ -97,6 +97,9 @@
     protected ExecutorService executorMsgs =
             Executors.newFixedThreadPool(2, groupedThreads("onos/of", "ctrl-msg-stats-%d"));
 
+    // messagesPendingMastership is used as synchronization variable for
+    // all mastership related changes. In this block, mastership (including
+    // role update) will have either occurred or not.
     private final AtomicReference<List<OFMessage>> messagesPendingMastership
             = new AtomicReference<>();
 
@@ -275,6 +278,8 @@
     public final void handleMessage(OFMessage m) {
         if (this.role == RoleState.MASTER || m instanceof OFPortStatus) {
             this.agent.processMessage(dpid, m);
+        } else {
+            log.trace("Dropping received message {}, was not MASTER", m);
         }
     }
 
@@ -309,7 +314,8 @@
         synchronized (messagesPendingMastership) {
             List<OFMessage> messages = messagesPendingMastership.get();
             if (messages != null) {
-                this.sendMsg(messages);
+                // Cannot use sendMsg here. It will only append to pending list.
+                sendMsgsOnChannel(messages);
                 log.debug("Sending {} pending messages to switch {}",
                           messages.size(), dpid);
                 messagesPendingMastership.set(null);