Symmetric DS taking too long on DataExtractorService

Question

I'm using SDS to migrate data from a SQL server to a Mysql database. My tests of moving the data of a database that was not in use worked correctly though they took like 48 hours to migrate all the existing data. I configured dead triggers to move all current data and triggers to move the new added data.

When moving to a live database that it is in use the data is being migrated too slow. On the log file I keep getting the message:

[corp-000] - DataExtractorService - Reached one sync byte threshold after 1 batches at 105391240 bytes. Data will continue to be synchronized on the next sync

I have like 180 tables and I have created 15 channels for the dead triggers and 6 channels for the triggers. For the configuration file I have:

job.routing.period.time.ms=2000

job.push.period.time.ms=5000

job.pull.period.time.ms=5000

I have none foreign key configuration so there wont be an issue with that. What I would like to know is how to make this process faster. Should I reduce the number of channels?

I do not know what could be the issue since the first test I ran went very well. Is there a reason why the threshold is not being clreared.

Any help will be apreciated. Thanks.

Answer 1

How large are your tables? How much memory does the SymmetricDS instance have?

I've used SymmetricDS for a while, and without having done any profiling on it I believe that reloading large databases went quicker once I increased available memory (I usually run it in a Tomcat container).

That being said, SymmetricDS isn't by far as quick as some other tools when it comes to the initial replication.

Have you had a look at the tmp folder? Can you see any progress in file size. That is, the files which SymmetricDS temporarily writes to locally before sending the batch off to the remote side? Have you tried turning on more fine grained logging to get more details? What about database timeouts? Could it be that the extraction queries are running too long, and that the database just cuts them off?