Can MarkLogic REST API instances scale independently of their underlying database?

Question

The MarkLogic documentation states that each REST API instance must have its own Modules database. As I understand, the Modules database is used for storing XQueries. Assuming that I can live without this, and am developing a MarkLogic database without using XQuery to persist data, is it possible to:

• Create multiple applications that point to the same content database
• Create multiple REST API instances that point to the same content database
• Scale each REST API instance independently of the associated content database

What I'd like to achieve is an infrastructure where the content database is scaled independently of the REST API instances that persist and retrieve data to and from it. The purpose of this is to manage instances in an AWS AutoScaling Group, so that REST API instances can be invoked and dropped based on the degree of traffic.

Answer 1

Paul:

The REST API requires a modules database for internal use as well as extensions.

That doesn't, however, restrict your ability to create multiple REST servers for the same content database. Each REST server has its own modules database, but the modules database doesn't have to be larger -- a single forest will almost always be enough.

If you want to add and drop preconfigured REST servers, you might want to take a close look at Packaging, which can propagate the configuration for a REST server and its modules database as well as the content of the modules database to a new cluster.

Erik Hennum

Answer 2

I could be wrong, but to my understanding you don't need to create new REST api services to scale out. It should be sufficient to add a new host to the cluster, add it to the same group on which you already created a REST api, and you can access the REST api on that new host.

It won't have its necessary modules on its local disk, so needs to access the cluster to retrieve the modules from the original host. It is likely that most will be cached though, so that would mean an initial delay only.

The only trouble would be with running custom files from file-system. Those would need to be installed on each host individually. It would also implicate that if you upload custom extensions to the REST api on one of the hosts, it becomes available on all others as well. They literally share resources. It makes scaling easy, but might not be what you functionally need. In that case, Erik's suggestion to use Packaging could be usefull.

Only if you are talking about duplicating REST api's between clusters (or groups within a cluster), you really need to recreate the REST api. There is a section in the REST api guide about this topic:

http://docs.marklogic.com/guide/rest-dev/migration#chapter

It might also be worth reading a bit about working with MarkLogic clusters, particularly the distinction between so-called Evaluator-nodes and Data-nodes. You can find a introduction on that topic in the cluster guide:

http://docs.marklogic.com/guide/cluster/distributed#chapter

HTH!