Using Rep to test, benchmark, and experiment with CouchDB and IBM Cloudant

Rep is the interactive tool bench component in CouchDyno and is probably the most fun one to play with.

Initially, Rep started with an assortment of python scripts to create replication scenarios. Some functions created databases, others filled them with data, some replicated documents, while others compared the contents of source and target databases.

We came up with the idea later to group the scripts into higher-level functions, refactor some common library code, and ultimately save the command line history to quickly recall the scenarios during testing. Often when debugging, you spend a lot of time in the Edit > Build > Run > Setup > Test cycle. Rep helps shorten the time wasted in the Run > Setup > Test part.

Let’s look at an example of how to use Rep.

Prerequisites

Rep assumes the user has:

Set up and configured the ability to build and run a local CouchDB 2.0 cluster. See the README for more details Python 2.7 virtualenv pip Setting up a local database cluster

The start command might look like:

$ cd $couchdb_source_dir
$ ./configure ...
$ make
$ ./dev/run --admin=adm:pass

After running the start command, a three-node cluster in development mode should be running. The first node listens on port 15984, with adm as the admin user and pass as the password.

Launching Rep

Then after cloning or downloading CouchDyno, run the ./build.sh script in its top directory. build.sh is just a wrapper around virtualenv and pip requirements install.

Now, let’s launch Rep.

$ ./venv/bin/rep
Interactive replication toolbox
rep, rep.getsrv, rep.getrdb and couchdb modules are auto-imported
Assumes the cluster runs on http://adm:pass@localhost:5984
Type rep. and press to auto-complete available functions
...
In[1]:
Notice this prompt looks remarkably like a Jupyter / IPython prompt. That’s because it uses Jupyter/IPython underneath and adds a few extra auto-imports and other nice things. Also, a lot of the module-level functions can be discovered by pressing the [Tab] key after typing “rep.” and the prompt should auto-complete with a set of choices.

Here is how to set up one source to two targets replication scenario. The source will contain 10 documents. These 10 documents should be replicated to the two targets. Rep will then verify that is the case. And will also repeat the whole thing thing times in a row:

rep.replicate_1_to_n_and_compare(n=2, cycles=3, num=10)
configuration:
cluster_branch = None
cluster_repo = None
cluster_reset_data = True
cluster_settings = None
cluster_tmpdir = None
connection_timeout = 30000
create_target = False
http_connections = 20
prefix = cdyno
replicator_url = None
server_url = http://adm:pass@localhost:15984
source_url = None
target_url = None
timeout = 0
worker_processes = 4
... Summary

Hopefully this blog post gave you a quick intro with how to use Rep, one of the CouchDyno projects to test, benchmark, and experiment with CouchDB and IBM Cloudant.