BCP not copying all rows

Question

I am using BCP to import rows into SQL server. All seemed to work fine when I realized that BCP is not importing all rows on a random basis. Some data is skipped (sometimes 50%). The BCP procedure is not returning any error. It says successfully copied X rows.

BCP command and output:

bcp MyTable IN MyTable_final.dat -f MyTable.Fmt -S <ServerIP> -U <User> -P <Pwd> -F2

Output:

Starting copy...
6 rows copied.
Network packet size (bytes): 4096
Clock Time (ms.) Total     : 63     Average : (95.24 rows per sec.)

SQLSERVER schema of the table:

CREATE TABLE MyTable (
Type varchar(255) default NULL,
Date varchar(255) default NULL,
ID varchar(255) default NULL,
VID varchar(255) default NULL,
VName varchar(255) default NULL,
LDate varchar(225) default NULL,
id BIGINT IDENTITY(1,1) NOT NULL,
PRIMARY KEY (id) ,
CONSTRAINT d_v UNIQUE (VID, VName)
)  ;

BCP Format file being used:

11.0
7
1       SQLCHAR             0       255     "|"      1     Type              SQL_Latin1_General_CP1_CI_AS
2       SQLCHAR             0       255     "|"      2     Date              SQL_Latin1_General_CP1_CI_AS
3       SQLCHAR             0       255     "|"      3     ID                  SQL_Latin1_General_CP1_CI_AS
4       SQLCHAR             0       255     "|"      4     VID                SQL_Latin1_General_CP1_CI_AS
5       SQLCHAR             0       255     "|"      5     VName              SQL_Latin1_General_CP1_CI_AS
6       SQLCHAR             0       225     "|"      6     LDate               SQL_Latin1_General_CP1_CI_AS
7       SQLCHAR             0       21      "\r\n"   7     id                     ""

Actual Data set: You can see there are 13 rows currently in the file. However, in my case, only 6 rows are imported randomly, without any trace of error.

Data File

I have not been able to figure out what is going on. Any pointers to trap the missing rows and prevent it?

Answer 1

It will happen when you have the ID with auto increment. So follow my idea

1. Create a view with out the ID field of the table
2. Insert the data in the view

for example

CREATE TABLE DIM_Vitals (
QueryType varchar(255) default NULL,
QueryDate varchar(255) default NULL,
APUID varchar(255) default NULL,
VitalID varchar(255) default NULL,
VitalName varchar(255) default NULL,
LoadDate varchar(225) default NULL,
id BIGINT IDENTITY(1,1) NOT NULL,
PRIMARY KEY (id) ,
CONSTRAINT dim_v UNIQUE (VitalID, VitalName)
)  ;

create a view for the above table

   create view DIM_Vitals_view 
as 
select 
QueryType,
QueryDate,
APUID,
VitalID,
VitalName,
LoadDate
from DIM_Vitals

now insert data into view [ DIM_Vitals_view - view name ]

bcp DIM_Vitals_view  IN DIM_Vitals_final.dat -f DIM_Vitals.Fmt -S <ServerIP> -U <User> -P <Pwd> -F2

sure it will solve the problem

make sure your view is not having the id field

Answer 2

Try setting max_errors to 0

-m max_errors

Specifies the maximum number of syntax errors and compilation errors that can occur before the bulk copy operation is canceled. Each row that cannot be copied by bcp is ignored and counted as one error. If this option is not included, the default is 10.