Solr cloud sharding

Question

Currently I have a zookeeper instance controlling replication on 3 servers. It is the solr integrated zookeeper. It works well in my web based application.

I have a new requirement which will require sharding in the cloud and I am not sure how to implement it. Basically I want to separate the data which can only be updated by me, shard 1, from the data that users can update, shard 2. From time to time I will be completely replacing the data directory in shard 1 - but I don't want to disturb the user created data in shard 2.

Shard 1 does not need replication since I can copy the new data to each server when I chose to update it however shard 2 does need replication.

Currently I run the following command on the server running zookeeper -

java -Dbootstrap_confdir=solr/myApp/conf -Dcollection.configName=myConfig -DzkRun -DnumShards=1 -jar start.jar

And the following command on the other 2 non zookeeper servers

java -Djetty-port=8983 -DzkHost=129.**.30.11:9983 -jar start.jar&

This creates a single shard solr instance * 3

I think I just need to add 1 static shard to this configuration however I am not sure the sequence of commands to accomplish it.

Many thanks

Answer 1

Firstly you are using zookeeper to maintain your shards and leaders/replicas. So if you want to have one shard with two instances and another shard with only a leader then you will have to modify your command as: 1)provide -DnumShards=2 so that the zookeeper knows that you need two shards 2)specify the -DzkHost parameter for this first solr instance also.

java -Dbootstrap_confdir=solr/myApp/conf -Dcollection.configName=myConfig -DzkRun -DnumShards=2 -DzkHost=** -jar start.jar

When you do this you will see some errors on console since shard2 is not created as yet. Now start your other two servers and you should see a shard1 with two servers(leader and replica) and shard2 will have only one instance i.e leader

If you want separation of indexes and control over those indexes.You will have to create two collections instead of two shards.

Explanation

you have 3 servers right!!! so when you will start solrCloud using zookeeper. following things will happen as:

1) start first solr server along with the zookeeper and you will get 1 shard for solr cloud as shard1

2) start second solr server and point to the zookeeper... since you have declared DnumShards=2 ,Zookeeper will check that it needs to create 1 more shard, so it creates shard2 for your collection. By now you will be able to see your admin console with 2 shards for 1 collection.

3) Now start your 3rd server and point it to zookeeper and now zookeeper sees that 2 shards are there so it will now create a replica for shard1 instead of a new shard.

so it will be like

collection--->shard1--->server1,server3

      --->shard2--->server2