how to specify row delimiter when using bcp with a format file

Question

I am using bcp utility to import data into Sybase database on Solaris system

Since the bcp file is generated by another xml parsing script, I have no control on that to make it generate the fields in the order I want. And now the the order of fields in the bcp file is a little different with the order in the database table.

I want to use a format file for bcp tool to control the order of fields loaded into the database, so I have the sample bcp file as below and constructed a format file accordingly:

the bcp file:

603289|Aug 20 2011 12:00AM|YYY aaa OVD|KLYYP8
603284|Aug 22 2011 12:00AM|XXX bbb OVD|KLPK06

the format file:

10.0
4
1  SYBCHAR   0  12   "|" 3 ver
2  SYBCHAR   0  26   "|" 2 first_dt
3  SYBCHAR   0  60   "|" 4 name1
4  SYBCHAR   0  10   "|" 1 name2

Although I am stuck on the following error:

$bcp my_db..my_tbl in test.bcp -e error -f format.fmt -r\\n -S Sever -U user -P pw

Starting copy...
CSLIB Message:  - L0/O0/S0/N24/1/0:
cs_convert: cslib user api layer: common library error: The conversion/operation was stopped due to a syntax error in the source field.
Unexpected EOF encountered in BCP data-file.
bcp copy in partially failed

1 rows copied.

I suspect the cause for the error is the bcp utility can't recognize the row delimiter which is '\n' character (I have used od -c test.bcp to check this), although the -r\\n option in the command seems doesn't work, which should specify the row delimiter as '\n' character.

Does anyone have a clue?

Edit:

I did a change on the format file and it works fine now, I changed the field delimiter for the last field from '|' to '\n' as below:

the new format file:

10.0
4
1  SYBCHAR   0  12   "|" 3 ver
2  SYBCHAR   0  26   "|" 2 first_dt
3  SYBCHAR   0  60   "|" 4 name1
4  SYBCHAR   0  10   "\n" 1 name2

As Michael mentioned in comments, it's tricky to use format files. When using the original format file I have tried to add '|' at end of each records, although the file were not processed correctly.

Now the file can be processed correctly, although I am not sure what is used as row delimiter by bcp utility

Answer 1

Sybase format files can be tricky to use, and I have yet to get them to work to redorder columns. Without a format file, and since you can not control the input file you're given, your options are:

1. Rebuild the table with the columns in the correct order
2. Use a temp table, or work table for the bcp in, and move the data into your production table from there.
3. If for some reason there is logic in place the requires the table to keep it's existing column order (which should be unlikely), you can rebuild the table with the correct column order, then create a view of the table that has the columns in the old order.

Given your data format, the following bcp command line should work.

bcp DBNAME..TABLENAME in MY_FILE.bcp -U username -P password -S SERVERNAME -c -t\| -r\\n

Answer 2

As most of the loader utilities, BCP does not read rows, it just gets a bunch of bytes from the source file, therefore, you have to specify the end line character on the .fmt file mostly when the end of line character can be different depending on the encoding of the file.

For example, in the example you provided, you have this on the source file:

603289|Aug 20 2011 12:00AM|YYY aaa OVD|KLYYP8
603284|Aug 22 2011 12:00AM|XXX bbb OVD|KLPK06

But depending on the Encoding you will have this:

603289|Aug 20 2011 12:00AM|YYY aaa OVD|KLYYP8\n603284|Aug 22 2011 12:00AM|XXX bbb OVD|KLPK06

or this:

603289|Aug 20 2011 12:00AM|YYY aaa OVD|KLYYP8\n\r603284|Aug 22 2011 12:00AM|XXX bbb OVD|KLPK06

That's why the change of the terminator you did on the last column is so important:

10.0
4
1  SYBCHAR   0  12   "|" 3 ver
2  SYBCHAR   0  26   "|" 2 first_dt
3  SYBCHAR   0  60   "|" 4 name1
4  SYBCHAR   0  10   "\n" 1 name2   * The last terminator was not "|"

The following image shows what means each component on the format file (I got it from here):

• By the way, please notice the last terminator!