Switch RAMCloud to Stanford release version
- Switch RAMCloud source code to 1.0 release + patches
- WARN: RAMCloud binary is going to be built to obj.master instead of obj.blueprint-java
- Fix JNI code to follow TableEnumerator related API change
- Build JNI library inside ONOS directory
- Remove non-source code artifact related to BluePrint
- Output ZooKeeper log to onos-logs
Change-Id: Id6fbf9418d6b3ce2e560bbf24b405c4ccc80a8e6
diff --git a/blueprints-README.md b/blueprints-README.md
deleted file mode 100644
index 787bd36..0000000
--- a/blueprints-README.md
+++ /dev/null
@@ -1,160 +0,0 @@
-blueprints-ramcloud-graph
-=========================
-
-A TinkerPop Blueprints implementation for RAMCloud
-
-Setup
-=====
- - Copy `src/main/java/edu/stanford/ramcloud/JRamCloud.java` and `src/main/cpp/edu_stanford_ramcloud_JRamCloud.cc` to your `ramcloud/bindinds/java/edu/stanford/ramcloud` directory, overwriting what is already there.
-
- - Generate the C++ header files containing the function signatures for all the native methods in the Java RamCloud library:
-
-```
-javah -cp ../../../ edu.stanford.ramcloud.JRamCloud
-```
-
- - Compile the ramcloud C++ library (assuming ramcloud is in your ${HOME} directory and you have already compiled ramcloud):
-
-```
-c++ -Wall -O3 -shared -fPIC -std=c++0x -I/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/include/ -I/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/include/linux/ -I${HOME}/ramcloud/src/ -I${HOME}/ramcloud/obj.master/ -I${HOME}/ramcloud/logcabin/ -I${HOME}/ramcloud/gtest/include/ -L${HOME}/ramcloud/obj.master -o libedu_stanford_ramcloud_JRamCloud.so edu_stanford_ramcloud_JRamCloud.cc -lramcloud
-```
-
- - Update `LD_LIBRARY_PATH` (assuming ramcloud is in your ${HOME} directory) to include the library and also any other ramcloud libraries:
-
-```
-export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${HOME}/ramcloud/bindings/java/edu/stanford/ramcloud:${HOME}/ramcloud/obj.master
-```
-
- - Startup a ramcloud cluster somewhere and modify RamCloudGraph.java to point to the coordinator.
-
- - Compile this package (blueprints-ramcloud-graph) using maven and run :)
-
-Using Rexster
-=============
- - git clone the rexster repository:
-
-```
-git clone https://github.com/tinkerpop/rexster.git
-```
-
- - Compile rexster
-
-```
-cd rexster/
-mvn compile
-```
-
- - Compile packaged rexster server
-
-```
-cd rexster-server/
-mvn package
-```
-
- - Edit rexster-server config file
-
-```
-cd target/rexster-server-2.5.0-SNAPSHOT-standalone/
-vim config/rexster.xml
-```
-
- - Change web-root to be your public web-root directory
-
- - Add a ramcloud graph to the set of graphs to load up:
-
-```
-<graph>
- <graph-enabled>true</graph-enabled>
- <graph-name>ramcloudgraph</graph-name>
- <graph-type>com.tinkerpop.rexster.config.RamCloudGraphConfiguration</graph-type>
-</graph>
-```
-
- - Go back to the blueprints-ramcloud-graph directory and compile jar with depdencies:
-
-```
-mvn assembly:assembly -DdescriptorId=jar-with-dependencies
-```
-
- - Copy resulting jar file into the compiled rexster-server library directory:
-
-```
-cp blueprints-ramcloud-graph-2.0.0-jar-with-depdencies.jar ~/git/rexster/rexster-server/target/rexster-server-2.5.0-SNAPSHOT-standalone/lib
-```
-
- - Startup a ramcloud cluster (maybe using the cluster.py script):
-
-```
-./cluster.py --verbose --servers=5 --replicas=3 --backups=1 --masterArgs="--totalMasterMemory 80% --masterServiceThreads 4" --clients=1 --client=../obj.master/client --debug
-```
-
- - Startup the rexster server:
-
-```
-./bin/rexster.sh -start &
-```
-
- - Connect to rexster server at:
-
-```
-localhost:8182
-```
-
-
-Using Gremlin
-=============
- - git clone the gremlin repository:
-
-```
-git clone https://github.com/tinkerpop/gremlin.git
-```
-
- - Compile gremlin
-
-```
-cd rexster/
-mvn package
-```
-
- - Go back to the blueprints-ramcloud-graph directory and compile jar with depdencies:
-
-```
-mvn assembly:assembly -DdescriptorId=jar-with-dependencies
-```
-
- - Copy resulting jar file into the compiled gremlin-groovy library directory:
-
-```
-cp blueprints-ramcloud-graph-2.0.0-jar-with-depdencies.jar ~/git/gremlin/gremlin-groovy/target/gremlin-groovy-2.5.0-SNAPSHOT-standalone/lib
-```
-
- - Startup a ramcloud cluster (maybe using the cluster.py script):
-
-```
-./cluster.py --verbose --servers=5 --replicas=3 --backups=1 --masterArgs="--totalMasterMemory 80% --masterServiceThreads 4" --clients=1 --client=../obj.master/client --debug
-```
-
- - Startup gremlin shell:
-
-```
-./bin/gremlin.sh
-```
-
- - Import RamCloudGraph java libraries and create RamCloudGraph:
-
-```
- \,,,/
- (o o)
------oOOo-(_)-oOOo-----
-gremlin> g = new com.tinkerpop.blueprints.impls.ramcloud.RamCloudGraph()
-```
-
- - Have fun!
-
-Using Furnace
-=============
- - See `com.tinkerpop.blueprints.impls.ramcloud.FurnaceExamples` for some examples of using Furnace
-
-Using JUNG
-==========
- - See `com.tinkerpop.blueprints.impls.ramcloud.JUNGExamples` for some examples of using the JUNG ouplementation (mostly taken from https://github.com/tinkerpop/blueprints/wiki/JUNG-Ouplementation).
diff --git a/build-ramcloud-java-bindings.sh b/build-ramcloud-java-bindings.sh
index 687ad99..3e3e48d 100755
--- a/build-ramcloud-java-bindings.sh
+++ b/build-ramcloud-java-bindings.sh
@@ -7,10 +7,7 @@
ONOS_HOME=${ONOS_HOME:-~/ONOS}
RAMCLOUD_HOME=${RAMCLOUD_HOME:-~/ramcloud}
-# create ramcloud lib
-cp -pvf ${ONOS_HOME}/src/main/java/edu/stanford/ramcloud/JRamCloud.java ${RAMCLOUD_HOME}/bindings/java/edu/stanford/ramcloud/
-cp -pvf ${ONOS_HOME}/src/main/cpp/edu_stanford_ramcloud_JRamCloud.cc ${RAMCLOUD_HOME}/bindings/java/edu/stanford/ramcloud/
-cd ${RAMCLOUD_HOME}/bindings/java/edu/stanford/ramcloud/
-javac JRamCloud.java
-./build_so.sh
-jar cvf ${RAMCLOUD_HOME}/bindings/java/RamCloud.jar ${RAMCLOUD_HOME}/bindings/java/edu/stanford/ramcloud/*.class
+${ONOS_HOME}/ramcloud-build-scripts/build_jni_so.sh
+
+cd ${ONOS_HOME}/target/classes
+jar cfe ${ONOS_HOME}/lib/RamCloud.jar edu.stanford.ramcloud.JRamCloud edu/stanford/ramcloud/*.class
diff --git a/lib/.gitignore b/lib/.gitignore
new file mode 100644
index 0000000..4da1695
--- /dev/null
+++ b/lib/.gitignore
@@ -0,0 +1,2 @@
+libedu_stanford_ramcloud_JRamCloud.so
+RamCloud.jar
diff --git a/logcabin.patch b/logcabin.patch
deleted file mode 100644
index 100697c..0000000
--- a/logcabin.patch
+++ /dev/null
@@ -1,21 +0,0 @@
-diff --git a/Core/Time.h b/Core/Time.h
-index 1dd2e00..a9deddd 100644
---- a/Core/Time.h
-+++ b/Core/Time.h
-@@ -25,6 +25,7 @@ namespace std {
- /**
- * Prints std::time_point values in a way that is useful for unit tests.
- */
-+/*
- template<typename Clock, typename Duration>
- std::ostream&
- operator<<(std::ostream& os,
-@@ -45,7 +46,7 @@ operator<<(std::ostream& os,
- microsSinceUnixEpoch / 1000000,
- microsSinceUnixEpoch % 1000000);
- }
--
-+*/
- }
-
- namespace LogCabin {
diff --git a/ramcloud-build-scripts/MakefragPrivateTop b/ramcloud-build-scripts/MakefragPrivateTop
new file mode 100644
index 0000000..726dbba
--- /dev/null
+++ b/ramcloud-build-scripts/MakefragPrivateTop
@@ -0,0 +1,7 @@
+# rewrite to match your environment
+ZOOKEEPER_LIB := /usr/lib/x86_64-linux-gnu/libzookeeper_mt.a
+ZOOKEEPER_DIR := ~/zookeeper-3.4.5
+
+# FIXME remove if RAMCloud source is fixed
+EXTRACXXFLAGS := -Wno-error=unused-result -Wno-error=maybe-uninitialized
+
diff --git a/ramcloud-build-scripts/build_jni_so.sh b/ramcloud-build-scripts/build_jni_so.sh
new file mode 100755
index 0000000..0ee12c4
--- /dev/null
+++ b/ramcloud-build-scripts/build_jni_so.sh
@@ -0,0 +1,16 @@
+#!/bin/bash
+JAVA_HOME=${JAVA_HOME:-/usr/lib/jvm/java-7-oracle}
+ONOS_HOME=${ONOS_HOME:-~/ONOS}
+RAMCLOUD_HOME=${RAMCLOUD_HOME:-~/ramcloud}
+RAMCLOUD_BRANCH=${RAMCLOUD_BRANCH:-master}
+
+if [ ! -d ${ONOS_HOME}/target/classes ]; then
+ mvn -f ${ONOS_HOME}/pom.xml
+fi
+
+javah -cp ${ONOS_HOME}/target/classes -o ${ONOS_HOME}/src/main/cpp/edu_stanford_ramcloud_JRamCloud.h edu.stanford.ramcloud.JRamCloud
+
+g++ -g -Wall -O3 -shared -fPIC -std=c++0x -I${JAVA_HOME}/include/ -I${JAVA_HOME}/include/linux -I${RAMCLOUD_HOME}/src/ -I${RAMCLOUD_HOME}/obj.${RAMCLOUD_BRANCH}/ -I${RAMCLOUD_HOME}/logcabin/ -I${RAMCLOUD_HOME}/gtest/include/ -L${RAMCLOUD_HOME}/obj.${RAMCLOUD_BRANCH} -o ${ONOS_HOME}/lib/libedu_stanford_ramcloud_JRamCloud.so ${ONOS_HOME}/src/main/cpp/edu_stanford_ramcloud_JRamCloud.cc -lramcloud
+
+#ln -sf `pwd`/libedu_stanford_ramcloud_JRamCloud.so ${RAMCLOUD_HOME}/obj.${RAMCLOUD_BRANCH}
+
diff --git a/ramcloud-build-scripts/logcabin.patch b/ramcloud-build-scripts/logcabin.patch
new file mode 100644
index 0000000..d13ed32
--- /dev/null
+++ b/ramcloud-build-scripts/logcabin.patch
@@ -0,0 +1,12 @@
+diff --git a/Server/RaftLog.cc b/Server/RaftLog.cc
+index 5705cd3..e571251 100644
+--- a/Server/RaftLog.cc
++++ b/Server/RaftLog.cc
+@@ -16,6 +16,7 @@
+ #include <algorithm>
+ #include <fcntl.h>
+ #include <sys/stat.h>
++#include <unistd.h>
+
+ #include "build/Protocol/Raft.pb.h"
+ #include "Core/Debug.h"
diff --git a/ramcloud-build-scripts/ramcloud.patch b/ramcloud-build-scripts/ramcloud.patch
new file mode 100644
index 0000000..5869a01
--- /dev/null
+++ b/ramcloud-build-scripts/ramcloud.patch
@@ -0,0 +1,28 @@
+diff --git a/GNUmakefile b/GNUmakefile
+index e4c9a07..199fac1 100644
+--- a/GNUmakefile
++++ b/GNUmakefile
+@@ -23,8 +23,8 @@ OBJDIR := obj$(OBJSUFFIX)
+
+ TOP := $(shell echo $${PWD-`pwd`})
+ GTEST_DIR ?= $(TOP)/gtest
+-ZOOKEEPER_LIB := /usr/local/lib/libzookeeper_mt.a
+-ZOOKEEPER_DIR := /usr/local/zookeeper-3.4.5
++ZOOKEEPER_LIB ?= /usr/local/lib/libzookeeper_mt.a
++ZOOKEEPER_DIR ?= /usr/local/zookeeper-3.4.5
+
+ ifeq ($(DEBUG),yes)
+ BASECFLAGS := -g
+diff --git a/src/MasterService.cc b/src/MasterService.cc
+index 93cfb5d..02876c7 100644
+--- a/src/MasterService.cc
++++ b/src/MasterService.cc
+@@ -613,7 +613,7 @@ MasterService::multiWrite(const WireFormat::MultiOp::Request* reqHdr,
+ reqOffset, currentReq->valueLength);
+ reqOffset += currentReq->valueLength;
+
+- if (stringKey == NULL || value == NULL) {
++ if (stringKey == NULL) {
+ respHdr->common.status = STATUS_REQUEST_FORMAT_ERROR;
+ break;
+ }
diff --git a/setup-ramcloud.sh b/setup-ramcloud.sh
index 7ed99e1..c34d649 100755
--- a/setup-ramcloud.sh
+++ b/setup-ramcloud.sh
@@ -10,27 +10,35 @@
exit 1
fi
+# install dependencies
+sudo apt-get -y install build-essential git-core libcppunit-dev libboost-all-dev libpcre3-dev protobuf-compiler libprotobuf-dev libcrypto++-dev libevent-dev scons libssl-dev libzookeeper-mt-dev
+
# clone ramcloud
-git clone git://github.com/y-higuchi/ramcloud.git ${RAMCLOUD_HOME}
+git clone git://fiz.stanford.edu/git/ramcloud.git ${RAMCLOUD_HOME}
cd ${RAMCLOUD_HOME}
-git checkout custom
+git checkout master
+# Switch to release 1.0 + fix for newer gcc
+git reset --hard 6a3eb98ae14ec6f732f06775220287c7245f5ff0
+# bug fix for MultiWrite, make zookeeper path configurable. (TODO send this patch to upstream)
+git apply ${ONOS_HOME}/ramcloud-build-scripts/ramcloud.patch
-# install some app
-sudo apt-get -y install build-essential git-core libcppunit-dev libcppunit-doc doxygen libboost-all-dev libpcre3-dev protobuf-compiler libprotobuf-dev libcrypto++-dev libevent-dev scons libssl-dev
+mkdir ${RAMCLOUD_HOME}/private
+cp -vf ${ONOS_HOME}/ramcloud-build-scripts/MakefragPrivateTop ${RAMCLOUD_HOME}/private/MakefragPrivateTop
-# compile ramcloud
+# download submodule (logcabin, gtest)
git submodule update --init --recursive
-patch ${RAMCLOUD_HOME}/logcabin/Core/Time.h < ${ONOS_HOME}/logcabin.patch
+# cherry-pick logcabin bug fix
+cd logcabin
+ git cherry-pick --no-commit 77f0ea2da82e7abe71bb4caf084aa527de6dea50 3862499f477d0e371950aebcb829ddd8ee194962
+ git apply ${ONOS_HOME}/ramcloud-build-scripts/logcabin.patch
+cd ..
+
+# compile logcabin
make logcabin
-make DEBUG=no $*
+# compile ramcloud
+make DEBUG=no "$@"
-ln -s ${RAMCLOUD_HOME}/obj.custom obj.blueprint-java
-# create ramcloud lib
-cp -pvf ${ONOS_HOME}/src/main/java/edu/stanford/ramcloud/JRamCloud.java ${RAMCLOUD_HOME}/bindings/java/edu/stanford/ramcloud/
-cp -pvf ${ONOS_HOME}/src/main/cpp/edu_stanford_ramcloud_JRamCloud.cc ${RAMCLOUD_HOME}/bindings/java/edu/stanford/ramcloud/
-cd ${RAMCLOUD_HOME}/bindings/java/edu/stanford/ramcloud/
-javac JRamCloud.java
-./build_so.sh
-jar cvf ${RAMCLOUD_HOME}/bindings/java/RamCloud.jar ${RAMCLOUD_HOME}/bindings/java/edu/stanford/ramcloud/*.class
+# build ramcloud JNI lib
+${ONOS_HOME}/build-ramcloud-java-bindings.sh
diff --git a/src/main/cpp/edu_stanford_ramcloud_JRamCloud.cc b/src/main/cpp/edu_stanford_ramcloud_JRamCloud.cc
index 41d87a5..a0dd0ce 100644
--- a/src/main/cpp/edu_stanford_ramcloud_JRamCloud.cc
+++ b/src/main/cpp/edu_stanford_ramcloud_JRamCloud.cc
@@ -17,7 +17,6 @@
#include <TableEnumerator.h>
#include <Object.h>
#include "edu_stanford_ramcloud_JRamCloud.h"
-#include "edu_stanford_ramcloud_JRamCloud_TableEnumerator.h"
using namespace RAMCloud;
@@ -635,7 +634,7 @@
const static jfieldID fieldId = env->GetFieldID(cls, "ramCloudObjectPointer", "J");
RamCloud* ramcloud = reinterpret_cast<RamCloud*>(env->GetLongField(jTableEnumerator, fieldId));
- return reinterpret_cast<jlong>(new TableEnumerator(*ramcloud, jTableId));
+ return reinterpret_cast<jlong>(new TableEnumerator(*ramcloud, jTableId, false));
}
/*
@@ -727,7 +726,7 @@
bool done = false;
while (true) {
- jTabletNextHash = ramcloud->enumerateTable(jTableId, jTabletNextHash, state, objects);
+ jTabletNextHash = ramcloud->enumerateTable(jTableId, false, jTabletNextHash, state, objects);
if (objects.getTotalLength() > 0) {
break;
}
diff --git a/start-onos.sh b/start-onos.sh
index f369cf2..5e8af65 100755
--- a/start-onos.sh
+++ b/start-onos.sh
@@ -1,9 +1,13 @@
#!/bin/bash
+
ulimit -c unlimited
-export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${HOME}/ramcloud/bindings/java/edu/stanford/ramcloud:${HOME}/ramcloud/obj.blueprint-java
# Set paths
ONOS_HOME="${ONOS_HOME:-`dirname $0`}"
+RAMCLOUD_HOME=${RAMCLOUD_HOME:-~/ramcloud}
+RAMCLOUD_BRANCH=${RAMCLOUD_BRANCH:-master}
+
+export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${ONOS_HOME}/lib:${RAMCLOUD_HOME}/obj.${RAMCLOUD_BRANCH}
## Because the script change dir to $ONOS_HOME, we can set ONOS_LOGBACK and LOGDIR relative to $ONOS_HOME
ONOS_LOGBACK=${ONOS_LOGBACK:-${ONOS_HOME}/logback.`hostname`.xml}
diff --git a/start-ramcloud-coordinator.sh b/start-ramcloud-coordinator.sh
index 6a4232e..3bd3e39 100755
--- a/start-ramcloud-coordinator.sh
+++ b/start-ramcloud-coordinator.sh
@@ -1,6 +1,5 @@
#!/bin/bash
-export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${HOME}/ramcloud/bindings/java/edu/stanford/ramcloud:${HOME}/ramcloud/obj.blueprint-java
# Set paths
ONOS_HOME=`dirname $0`
@@ -10,6 +9,7 @@
coordinatorip=`grep coordinatorIp ${ONOS_HOME}/conf/ramcloud.conf | cut -d "=" -f 2,3`
coordinatorport=`grep coordinatorPort ${ONOS_HOME}/conf/ramcloud.conf | cut -d "=" -f 2,3`
RAMCLOUD_COORDINATOR=`echo $coordinatorip","$coordinatorport`
+RAMCLOUD_BRANCH=${RAMCLOUD_BRANCH:-master}
function lotate {
logfile=$1
@@ -35,7 +35,7 @@
# Run ramcloud
echo "Starting ramcloud coordinator"
- $RAMCLOUD_DIR/obj.blueprint-java/coordinator -L $RAMCLOUD_COORDINATOR > $RAMCLOUD_LOG 2>&1 &
+ $RAMCLOUD_DIR/obj.${RAMCLOUD_BRANCH}/coordinator -L $RAMCLOUD_COORDINATOR > $RAMCLOUD_LOG 2>&1 &
}
function stop {
@@ -72,7 +72,7 @@
# deldb
# ;;
status)
- n=`pgrep -f obj.blueprint-java/coordinator | wc -l`
+ n=`pgrep -f obj.${RAMCLOUD_BRANCH}/coordinator | wc -l`
echo "$n ramcloud coordinator is running"
;;
*)
diff --git a/start-ramcloud-server.sh b/start-ramcloud-server.sh
index f48f831..0158be2 100755
--- a/start-ramcloud-server.sh
+++ b/start-ramcloud-server.sh
@@ -1,7 +1,6 @@
#!/bin/bash
ulimit -c unlimited
-export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${HOME}/ramcloud/bindings/java/edu/stanford/ramcloud:${HOME}/ramcloud/obj.blueprint-java
# Set paths
ONOS_HOME=`dirname $0`
@@ -14,6 +13,7 @@
serverip=`grep serverIp ${ONOS_HOME}/conf/ramcloud.conf | cut -d "=" -f 2,3`
serverport=`grep serverPort ${ONOS_HOME}/conf/ramcloud.conf | cut -d "=" -f 2,3`
RAMCLOUD_SERVER=`echo $serverip","$serverport`
+RAMCLOUD_BRANCH=${RAMCLOUD_BRANCH:-master}
function lotate {
logfile=$1
@@ -39,12 +39,12 @@
# Run ramcloud
echo "Starting ramcloud"
- $RAMCLOUD_DIR/obj.blueprint-java/server -M -L $RAMCLOUD_SERVER -C $RAMCLOUD_COORDINATOR --masterServiceThreads 1 --logCleanerThreads 1 --detectFailures 0 > $RAMCLOUD_LOG 2>&1 &
+ $RAMCLOUD_DIR/obj.${RAMCLOUD_BRANCH}/server -M -L $RAMCLOUD_SERVER -C $RAMCLOUD_COORDINATOR --masterServiceThreads 1 --logCleanerThreads 1 --detectFailures 0 > $RAMCLOUD_LOG 2>&1 &
}
function stop {
# Kill the existing processes
- capid=`pgrep -f obj.blueprint-java/server | awk '{print $1}'`
+ capid=`pgrep -f obj.${RAMCLOUD_BRANCH}/server | awk '{print $1}'`
pids="$capid"
for p in ${pids}; do
if [ x$p != "x" ]; then
@@ -76,7 +76,7 @@
# deldb
# ;;
status)
- n=`pgrep -f obj.blueprint-java/server | wc -l`
+ n=`pgrep -f obj.${RAMCLOUD_BRANCH}/server | wc -l`
echo "$n ramcloud server running"
;;
*)
diff --git a/start-zk.sh b/start-zk.sh
index 0853a96..30de5fb 100755
--- a/start-zk.sh
+++ b/start-zk.sh
@@ -4,6 +4,7 @@
ONOS_HOME=`dirname $0`
ZK_DIR=${HOME}/zookeeper-3.4.5
#ZK_CONF=${ONOS_HOME}/conf/zoo.cfg
+ZOO_LOG_DIR=${ONOS_HOME}/onos-logs
function start {
# Run Zookeeper with our configuration