Bulk Insert using format file for varying number of columns between datafile and actual database table

Question

This is the Schema for a table

Create table dbo.Project
(
ProjectID (int,not null)
ManagerID (int,not null)
CompanyID(int, not null)
Title (nvarchar(50),not null)
StartDate(datetime,not null)
EndDate(datetime,null)
ProjDescription(nvarchar(max))
)

I created a datafile called bob.dat from this table which has around 15 rows with the following bcp command

bcp "Select ProjectID,ManagerID,CompanyID,Title,StartDate from CATS.dbo.Project" queryout "C:\Documents\bob.dat" -Sbob-pc -T -n 

Also a format/mapping file called bob.fmt was created using the following bcp command

bcp CATS.dbo.Project format nul -f C:\Documents\bob.fmt -x -Sbob-pc -T -n 

Then i created a copy of the table Project.

Create table dbo.ProjectCopy
(
ProjectID (int,not null)
ManagerID (int,not null)
CompanyID(int, not null)
Title (nvarchar(50),not null)
StartDate(datetime,not null)
EndDate(datetime,null)
ProjDescription(nvarchar(max))

)

What i want to do now is use the bob.dat and bob.format file to populate this table ProjectCopy using the following Bulk Insert statement.

BULK INSERT CATS.dbo.ProjectCopy
 FROM 'C:\Documents\bob.dat' 
WITH (FORMATFILE = 'C:\Documents\bob.fmt',
 LASTROW=5,
 KEEPNULLS,
  DATAFILETYPE='native');
  GO
 SELECT * FROM CATS.dbo.ProjectCopy
 GO

So basically the data file does not contain any data for the columns EndDate and ProjDescription. I want these two columns to be remained as null. Unfortunately i get the following error when i run the bulk insert statement.

 Msg 4863, Level 16, State 4, Line 2
 Bulk load data conversion error (truncation) for row 1, column 6 (EndDate).
 Msg 7399, Level 16, State 1, Line 2
 The OLE DB provider "BULK" for linked server "(null)" reported an error. 
 The provider did not give any information about the error.
 Msg 7330, Level 16, State 2, Line 2
 Cannot fetch a row from OLE DB provider "BULK" for linked server "(null)".

 (0 row(s) affected)

Anyone got any clue how this could be fixed? just to inform you all i have already been to this sections and the solution provided there didn't work out for me. BULK INSERT with inconsistent number of columns, Can't identify reason for BULK INSERT errors, BULK INSERT with inconsistent number of columns

Answer 1

First of all, for a creating a datafile called bob.dat you need to add two columns: EndDate and ProjDescription. In addition, for the bulk copy operation using Unicode-characters must be added the argument -W. Example:

bcp "Select ProjectID,ManagerID,CompanyID,Title,StartDate, NULL AS EndDate, NULL AS ProjDescription from CATS.dbo.Project" queryout "C:\Users\Pawan\Documents\bob.dat" -Sbob-pc -T -n -w

The original format file:

<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
 <RECORD>
  <FIELD ID="1" xsi:type="NCharTerm" TERMINATOR="\t\0" MAX_LENGTH="24"/>
  <FIELD ID="2" xsi:type="NCharTerm" TERMINATOR="\t\0" MAX_LENGTH="24"/>
  <FIELD ID="3" xsi:type="NCharTerm" TERMINATOR="\t\0" MAX_LENGTH="24"/>
  <FIELD ID="4" xsi:type="NCharTerm" TERMINATOR="\t\0" MAX_LENGTH="100" COLLATION="Cyrillic_General_CI_AS"/>
  <FIELD ID="5" xsi:type="NCharTerm" TERMINATOR="\t\0" MAX_LENGTH="48"/>
  <FIELD ID="6" xsi:type="NCharTerm" TERMINATOR="\t\0" MAX_LENGTH="48"/>
  <FIELD ID="7" xsi:type="NCharTerm" TERMINATOR="\r\0\n\0" COLLATION="Cyrillic_General_CI_AS"/>
 </RECORD>
 <ROW>
  <COLUMN SOURCE="1" NAME="ProjectID" xsi:type="SQLINT"/>
  <COLUMN SOURCE="2" NAME="ManagerID" xsi:type="SQLINT"/>
  <COLUMN SOURCE="3" NAME="CompanyID" xsi:type="SQLINT"/>
  <COLUMN SOURCE="4" NAME="Title" xsi:type="SQLNVARCHAR"/>
  <COLUMN SOURCE="5" NAME="StartDate" xsi:type="SQLDATETIME"/>
  <COLUMN SOURCE="6" NAME="EndDate" xsi:type="SQLDATETIME"/>
  <COLUMN SOURCE="7" NAME="ProjDescription" xsi:type="SQLNVARCHAR"/>
 </ROW>
</BCPFORMAT>

But you want a way to fill the data only upto StartDate. Therefore, this file needs to be changed:

<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
 <RECORD>
  <FIELD ID="1" xsi:type="NCharTerm" TERMINATOR="\t\0" MAX_LENGTH="24"/>
  <FIELD ID="2" xsi:type="NCharTerm" TERMINATOR="\t\0" MAX_LENGTH="24"/>
  <FIELD ID="3" xsi:type="NCharTerm" TERMINATOR="\t\0" MAX_LENGTH="24"/>
  <FIELD ID="4" xsi:type="NCharTerm" TERMINATOR="\t\0" MAX_LENGTH="100" COLLATION="Cyrillic_General_CI_AS"/>
  <FIELD ID="5" xsi:type="NCharTerm" TERMINATOR="\r\0\n\0" MAX_LENGTH="48"/>
 </RECORD>
 <ROW>
  <COLUMN SOURCE="1" NAME="ProjectID" xsi:type="SQLINT"/>
  <COLUMN SOURCE="2" NAME="ManagerID" xsi:type="SQLINT"/>
  <COLUMN SOURCE="3" NAME="CompanyID" xsi:type="SQLINT"/>
  <COLUMN SOURCE="4" NAME="Title" xsi:type="SQLNVARCHAR"/>
  <COLUMN SOURCE="5" NAME="StartDate" xsi:type="SQLDATETIME"/>
 </ROW>
</BCPFORMAT>