How to use IDs from another source (CSV file) in a query?

Question

I've been given a large list (~50,000) of User IDs in a CSV file, and I have to query our MSSql 2008R2 database to find details of all those users (email address, etc). How can I do this, given that the source of the IDs is not in a table in order to do a join?

I have tried pasting the entire list into the query editor and inserting them into a temporary table to join onto, but quickly ran into the 1000-row limit in the INSERT INTO syntax.

Is there a better way to do this? The only option I can think of is SELECT * FROM User WHERE UserId IN (..., ..., ... which seems like it would be horribly inefficient.

Answer 1

If you have access to copy the file directly to the SQL server, use bulk copy.

Borrowing some excellent explanations from http://blog.sqlauthority.com/2008/02/06/sql-server-import-csv-file-into-sql-server-using-bulk-insert-load-comma-delimited-file-into-sql-server/

BULK
INSERT CSVTest
FROM 'c:\csvtest.txt'
WITH
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)
GO
--Check the content of the table.
SELECT *
FROM CSVTest
GO
--Drop the table to clean up database.
DROP TABLE CSVTest
GO

Answer 2

This is too long for a comment.

Use the Import Wizard and insert them into a "temporary" table. I'm putting "temporary" in quotes, because it doesn't have be in tempdb.

50,000 rows is a bit long to insert them one at a time. For small sets of data, I would open the CSV file in Excel and create a separate insert statement for each one:

insert into tempids(...) select X, Y, Z;

where X and so on are values from cells. Then I'd create the table and copy these insert statements into SSMS studio and run them.

Answer 3

With that many records, you don't really have any other option than to load them into a temporary table. There are various ways to do this, SSIS being the most obvious if using SQL Server 2005 or newer.

But if you choose to use SSIS, you might as well query the database directly using a SSIS Lookup Transform. Basically what happens is: You create a data flow task in your SSIS-package that reads all the records from the CSV file. As its name suggests, the Lookup component is then used to look up the details from your database table, for each record in your data flow. The output can then be written to a new CSV file or stored in another table.

Note that the Import-Export Wizard actually creates a SSIS-package behind the scenes, with the option of saving the package file, so use that to your advantage.