Overcome Runtime error when exporting large data table from PostgreSQL

Question

I am very new to using PostgreSQL. I have created a query that compiles a table with more than 100,000 rows and 20 columns of data. The query works and loads the table fine in the PostgreSQL preview window. However, when I click to export the table to CSV in PostgreSQL I get a Runtime application error that says PostgreSQL must automatically close (and I lose everything). I do not get this error when I query and CSV export much smaller tables.

I would appreciate any advice on how to overcome this Runtime application error for this large table.

I do not want to divide my large table into many small tables since this will take too much time and creates rooms for merging multiple tables errors.

Thank you for any help you can provide!

CODE:

SELECT
si.symbolvalue as GVKEY
,e.mostImportantDateUTC
,t.transcriptcreationdateutc
,tc.componentOrder
,replace(replace(tc.componenttext, chr(10), ''), chr(13), '') as ComponentText

,p.proId

FROM ciqTranscript t 
JOIN ciqEvent e  ON e.keyDevId = t.keyDevId
JOIN ciqEventToObjectToEventType eo ON eo.keyDevId = e.keyDevId
JOIN ciqEventType et ON et.keyDevEventTypeId = eo.keyDevEventTypeId
JOIN ciqCompany c ON c.companyId = eo.objectId
JOIN ciqsymbol si on si.objectid = c.companyid and si.symboltypeid = 69 -- gvkey
JOIN ciqTranscriptCollectionType tct  ON tct.transcriptCollectionTypeId = t.transcriptCollectionTypeId
JOIN ciqTranscriptComponent tc  ON tc.transcriptId = t.transcriptId

WHERE et.keyDevEventTypeId='48'
AND t.transcriptPresentationTypeId='5' 
AND tct.transcriptCollectionTypeId='1'
AND e.announceddateutc between '2008-01-01' and '2017-12-31'

ORDER BY t.transcriptid, c.companyid, tc.componentOrder

Answer 1

This problem is not with postgresql but with the IDE that you are using to manage it - PostgreSQL has nothing to do about that.

You can try to do this with COPY command (different from \copy - because is a psql command)

COPY (
SELECT
si.symbolvalue as GVKEY
,e.mostImportantDateUTC
,t.transcriptcreationdateutc
,tc.componentOrder
,replace(replace(tc.componenttext, chr(10), ''), chr(13), '') as ComponentText

,p.proId

FROM ciqTranscript t 
JOIN ciqEvent e  ON e.keyDevId = t.keyDevId
JOIN ciqEventToObjectToEventType eo ON eo.keyDevId = e.keyDevId
JOIN ciqEventType et ON et.keyDevEventTypeId = eo.keyDevEventTypeId
JOIN ciqCompany c ON c.companyId = eo.objectId
JOIN ciqsymbol si on si.objectid = c.companyid and si.symboltypeid = 69 -- gvkey
JOIN ciqTranscriptCollectionType tct  ON tct.transcriptCollectionTypeId = t.transcriptCollectionTypeId
JOIN ciqTranscriptComponent tc  ON tc.transcriptId = t.transcriptId

WHERE et.keyDevEventTypeId='48'
AND t.transcriptPresentationTypeId='5' 
AND tct.transcriptCollectionTypeId='1'
AND e.announceddateutc between '2008-01-01' and '2017-12-31'

ORDER BY t.transcriptid, c.companyid, tc.componentOrder ) 
TO '/tmp/yourfile.txt' delimiter ';'; 

If you are using Windows as a server, then you just need to use a Windows path instead of /tmp.

This is good if you have postgresql in your local machine or have some access in the server that postgresql is stored, then you can also use a path in the file name to put the file in some directory that is shared (via samba, Windows Share or NFS for example) with you or with your PHP/Java system. Guarantee postgresql user has access to the directory -- tmp is world writable because that i used it as a example.

Good lucky!