Replication architecture help when using ArangoDB

Question

In my quest to solve the 1000’s of clients, each with individual ArangoDB’s, recording individual information as something is running…We need to aggregate this information back to a collection in a singular Master Node server so that queries and reports can be made on it. Looking into the power of JavaScript and actions within ArangoDB… Is this possible and fast enough to ‘replicate’ data back to the Master Node..

1) Local run data is being logged on each client

2) An action trigger is executed (let’s take the holy crap scenario first, no bulk uploading or anything)…for each insert, take the locally inserted data and make that exact request to the Master Node for insertion into the common collection (the document data stored on each client has properties that uniquely identify it etc).

3) This would effectively ‘replicate’/aggregate each client’s unique data to the single Master Node collection.

Thoughts?

Thanks,

Ken

Answer 1

An initial thought on this:

You could wrap insert / update / delete operations to a specific collection(s) in an API of your own. Instead of using the regular HTTP POST, PUT/PATCH, DELETE routes that ArangoDB provides for documents, you can write your own routes inside ArangoDB. ArangoDB has a framework, Foxx, to do this.

For example, you may create your own insert route. Inside the route, you can execute any sort of JavaScript code. For example, here is a simple insert wrapper that justs inserts data into a collection mydata:

controller.post('/my-insert', function (req, res) {
  var document = req.body();

  try {
    var result = db.mydata.save(doc);
    // TODO: send off data to external server etc.

    res.json(result); // send back result to original client
  }
  catch (err) {
    // TODO: handle and report error
  }
});

In the above example, data is stored in a local collection and the response is sent back to the client. However, you may execute extra JavaScript code after the save operation. So you may actually send data to another server from there.

You may do the same for the update and delete routes. This solution requires that all your operations use the custom routes, and you don't modify data using the regular document APIs. Using the regular APIs would still work of course, but won't fire off the custom actions.

The interesting questions are if you can accept the thread blocking on the "send to another server call" and what you would do if sending data to another server fails for whatever reason.