Records getting rejected in HP Vertica While Copying

StackOverflow https://stackoverflow.com/questions/23422390

  •  14-07-2023
  •  | 
  •  

Question

I am trying to load a segmented table in HP Vertica through COPY DIRECT from a pipe separated text file.

COPY CSI.MKT_RSRCH_AGG_ALL FROM '/opt/vertica/CSI/MKT_RSRCH_AGG_ALL.txt' DELIMITER '|' NULL '' DIRECT;

Result

 Rows Loaded 
-------------
      582006

dbadmin=> select get_num_rejected_rows();

 get_num_rejected_rows 
-----------------------
                  6046

I am unable to figure out what is causing the data to be rejected.

All of my dimensions are defined NOT NULL.

Can I check any logs/info for the rejected records.

Was it helpful?

Solution

I would start by verifying the integrity of the data. Then, I would send any rejected rows to a file. You can specify this in your COPY command:

COPY CSI.MKT_RSRCH_AGG_ALL 
FROM '/opt/vertica/CSI/MKT_RSRCH_AGG_ALL.txt' 
DELIMITER '|' 
NULL ''
REJECTED DATA '/path/to/rejected/data'
DIRECT;

If you haven't already, I would recommend that you start using stream names to identify the process:

COPY ... DIRECT STREAM NAME 'My stream name';

You can than easily monitor the stream:

SELECT * FROM v_monitor.load_streams WHERE stream_name = 'My stream name';

Documentation

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top