Querying SQL Server repeatedly (loop) is taking up memory that doesn't release back into the system

Question

I have a VB web application that reads from a CSV file which contains about 300,000++ lines. The application will read line-by-line and for each line, it will query a table in a sql server database which contains about 100,000++ records and based on the results of the query, the application will do something else. sqlservr.exe maxes out memory consumption in my development machine of 4GB.

So I created a simple application to test the performance of sql server in terms of loop query.

Dim Connection As New SqlConnection(ConfigurationManager.ConnectionStrings("SiteSqlServer").ConnectionString)
Dim Command As New SqlCommand("", Connection)
Connection.Open()
For i As Integer = 0 To 20000
    Command.CommandText = "SELECT CustomerID FROM Customer WHERE CustomerID = " & i
    Command.ExecuteScalar()
Next
Connection.Close()

Every time this code is executed, sqlservr.exe will take up an additional 100MB++ of memory and it's not releasing it back even though after the code has finished executing.

Is this normal? Is there any way to mitigate this effect?

Answer 1

Yes, it is normal. SQL Server in general doesn't release memory once it has acquired it. It's purpose is to deliver results to you as quickly as possible, and if it has data that is cached in RAM, it can be queried quicker. SQL Server might repurpose the memory blocks it has acquired internally, but it doesn't release them while it is running. This is by design. Try reading this.

This is also why, in general, you should run SQL Server on a separate box in production, so it doesn't contend with other applications for memory (such as IIS).

Answer 2

Try this:

For i As Integer = 0 To 20000
    Command.CommandText = "SELECT CustomerID FROM Customer WHERE CustomerID = " & i
    Command.ExecuteScalar()
    Command.Dispose()
Next

The Dispose() call should clean up behind you and might mitigate the effect.

Answer 3

Sql Server does not immediately release memory back to the system, for several reasons (see: http://sqlnerd.blogspot.com/2006/07/memory-use-in-sql-server.html).

Anyway, you should use a Stored Procedure instead of a dynamic sql query: Sql Server can work more efficiently because your sql string does not have to be parsed each and every time.

It sure is, but double check - make sure that the table you're querying is properly indexed.

You might also consider loading your .csv file into sql server beforehand - if feasible. Having your .csv data in a sql server indexed table will make everything easier: no matter how well you optimize them, 100k round-trips to Sql Server are going to consume resources that you might be able to easily save by leveraging joins and server-side operations.