NPGSQL: Streaming results from Postgres query?

Question

I have a large table (2,000,000 rows) and I'd like to print each record to the screen, one at time, without loading the entire table into memory.

//pseudo code
var cmd = new NpgSQLCommand();
cmd.CommandText = "SELECT * FROM mytable;"
IReader reader = cmd.ExecuteReader(); //blocks until the entire set is returned

while(reader.Read()) //ideally each call to read loads more results from the db.
{
// print record name

}

So as noted in the code above, the ExecuteReader() doesn't continue until the entire set is loaded into memory. How do I change this behavior so the results are streamed?

Thanks

ETA: While this seems like homework, it's not. It's just an easier way to describe a problem that involves reading an entire table with a single query but processing the results a row at a time.

ETA x2:

From npgsql Warning: There is a known issue when calling ExecuteReader and large tables. Currently Version 1 of Npgsql gets all data from table before returning. If you are experiencing bad performance in such cases, you may need to use a server cursor to page through rows. For that, you can use a code like the following:

Answer 1

Okay, well it looks like this is a known issue with npgsql 1.0:

The workaround is to use a server cursor:

using System;
using System.Data;
using Npgsql;

public static class NpgsqlUserManual
{
  public static void Main(String[] args)
  {
    NpgsqlConnection conn = new NpgsqlConnection("Server=127.0.0.1;Port=5432;User Id=joe;Password=secret;Database=joedata;");
    conn.Open();

    NpgsqlCommand command = new NpgsqlCommand("select version()", conn);
    String serverversion;

    try
    {
      serverversion = (String)command.ExecuteScalar();
      Console.WriteLine("PostgreSQL server version: {0}", serverversion);
    }


    finally
    {
      conn.Close();
    }
  }
}

Answer 2

Npgsql2 now handles large resultsets much better. It doesn't load all data to memory. So, you don't need to use a server side cursor anymore.

I hope it helps.

Francisco Figueiredo Jr. Npgsql Lead Developer

Answer 3

The easiest way to do this is:

COPY ( select * from Table) TO STDOUT

Check the syntax of copy to see how to print it in .csv or any other format if you need to...

Answer 4

To print out the whole table with column name and value use following code:

using (var conn = new NpgsqlConnection("Host=IPADDRESS;Username=USER;Password=PASS;Database=DBNAME;"))
        {
            conn.Open();
            using (var cmd = new NpgsqlCommand())
            {
                cmd.Connection = conn;

                cmd.CommandText = "SELECT * FROM TABLE_NAME";

                StringBuilder str;
                int count;

                using (var reader = cmd.ExecuteReader())
                {
                    while (reader.Read())
                    {
                        str = new StringBuilder();
                        count = 0;

                        while (count < reader.FieldCount) {
                            str.Append(reader.GetName(count) + ": " + reader.GetValue(count));

                            if ((count + 1) < reader.FieldCount)
                            {
                                str.Append(", ");
                            }

                            count++;
                        }

                        Console.WriteLine(str.ToString());
                        Console.WriteLine("====================");
                    }
                }
            }