How can import Large data from MongoDB to SQL Server Like Schedulers

Question

We have application which collect huge data daily. So write operation is more, Hence my server slow down. So what we have planned use MongoDB to collect data, By using scheduler will import data to SQL.

So my problem is how can I import that much heavy data from MongoDB to SQL

Any suggestion Please. Like any tool etc.

Answer 1

I don't know any tools, but I'm sure they exist if you google them.

If it was me, without prior knowledge, I may export data to a flat file (.csv) and create either a stored procedure or an SSIS package to import the data into SQL.

Python may be my choice to automate the exports in chunks overnight where SQL can handle the importation and cleanup.

mongoexport --host yourhost --db yourdb --collection yourcollection --csv --out yourfile.csv --fields field1,field2,field3

Doing it this way allows you to define the structure before it hits the SSIS package.

Another way

Here is a good example of doing all collections. This was from another answer.

 out = `mongo  #{DB_HOST}/#{DB_NAME} --eval "printjson(db.getCollectionNames())"`

  collections = out.scan(/\".+\"/).map { |s| s.gsub('"', '') }

  collections.each do |collection|
    system "mongoexport --db #{DB_NAME}  --collection #{collection}  --host '#{DB_HOST}' --out #{collection}_dump"
  end

Answer 2

We created MongoSluice for this specific reason.

Our application interrogates a MongoDB collection and creates a full, deep schema. It then streams data and meta data to any RDBMS system (Oracle, MySQL, Postgres, HP Vertica...).

What you end up with is a representation of your NoSQL as SQL. A big use case for this is to get unstructured data into analytical databases. BI platforms, particularly.

Answer 3

You can user linked server to mongodb so you can query whatever you want. Of course before that you have to install to required drivers to MongoDB

EXEC sp_addlinkedserver @server='MongoDB',
@srvproduct='CData.Mongo DB.ODBC.Driver',
@provider='SQLNCLI10',
@datasrc='<Machine IP address>,1434',
@provstr='Network Library=DBMSSOCN;',
@catalog='CDataMongoDB';