Upload multiple files to SQL Server without access to the Server Box

Question

I am trying to create a process that would bulk upload multiple csv files periodically to individual tables in SQL server. The issue I am having is those files are not located on the same server.

I can use the bulk upload command to upload files from a folder on the SQL server box but not from our shared drive which is located on a different server. There are multiple people who would be running this process and so it might not be feasible because not all of them would get that kind of access to the SQL server box.

Is there any way of accomplishing this with the same level of efficiency and speed?
I have thought of sending each line in the csv files but those files are huge and would take a long time
Just looking for ideas to make this work.
Would SSIS be less or more or equally efficient?

Answer 1

If there are no transformations involved, you could write this process fairly simply in SSIS with a for loop in the control flow that iterates over a file directory, then passes each file name as a variable to a data flow. The filename for the csv would be bound to that variable, and your destination could similarly be bound to the file name. Just use bulk mode in your target component and you should get pretty good efficiency. This works great if the folder is accessible to all users and you add a step to either check and see that files in the directory are processed or move them somewhere else when they've been processed. The SSIS process could be invoked from the SSIS catalog on the SQL Server directly, or scheduled on a regular basis.

Alternatively, you could use the bcp Utility for a CLI experience. This is probably a lot simpler in that it doesn't require writing an SSIS process and management of that process, but it requires more knowledge on the end of the user invoking it. This tool doesn't require the files be on the server: you provide a path to the file as the invoking user, along with credentials to authenticate, and the tool will handle the data movement.