Microsoft LogParser CSV output format - suppression of double quoting

Question

I am having a problem with MS LogParser v2.2 and cannot seem to track down "the solution".

I am importing a UTF-16 TSV file with headers and am trying to export a subset of these fields this to a CSV file, generally with no processing, but with one instance of concatenating two of the fields with an intervening space (combining a first and last name into a full name).

The problem is that not only are all the fields double quoted irrespective of the oDQuotes argument (which I can happily ignore), but the concatenated field contains the result of that double quoting. I.e. given two fields Fred and Bloggs, the contents of the concatenated field is always

"Fred" "Bloggs"

rather than the less harmful

"Fred Bloggs"

or even

Fred Bloggs

no matter what the value of the oDQuotes parameter (OFF, AUTO or ON). The presence of these double quotes cannot be ignored or easily discarded.

I have tried this both in a batch file and in Windows Script:

e.g. Batch File:

set lp=%ProgramFiles(x86)%\Log Parser 2.2\LogParser.exe
set fields=[Buyer E-mail Address]
set fields=%fields%, [Order ID]
set fields=%fields%, [Shipping Addr 1]
set fields=%fields%, [Shipping Addr 2]
set fields=%fields%, [Shipping City]
set fields=%fields%, [Shipping Postal Code]
set fields=%fields%, [Buyer First Name]
::set fields=%fields%, strcat([Buyer First Name], ' ', [Buyer Last Name]) --- does not work. :-( 
set fields=%fields%, strcat([Buyer First Name], strcat(' ', [Buyer Last Name]))
set fields=%fields%, [Buyer Last Name]
set fields=%fields%, [Buyer Company]
set fields=%fields%, [Buyer Day Phone]

set sql=SELECT %fields% into chad_out.csv from %1

"%lp%" -q:ON -i:TSV -icodepage:-1 -nSep:1 -fixedSep:on -o:CSV -oDQuotes:OFF -fileMode:1 "%sql%"

or JScript:

function ProcessFile(filename) {
  DebugEcho(50, "D&D File name is <" + filename + ">");

  var lq = WScript.CreateObject("MSUtil.LogQuery");
  var lqif = WScript.CreateObject("MSUtil.LogQuery.TSVInputFormat");
  var lqof = WScript.CreateObject("MSUtil.LogQuery.CSVOutputFormat");

  // check that we actually have the objects in question
  if (lq && lqif && lqof) {
    DebugEcho(100, "Everything ok");
  } else {
    DebugEcho(0, "Something bad with LogQuery objects - exiting");
    WScript.Quit(1);
  }

  // see command line "> LogParser.exe -h -i:TSV" for details
  lqif.codepage = -1;   // this is for unicode
  lqif.fixedSep = true; // seems to need this
  lqif.nSep = 1;        // seems to need this?

  // see command line "> LogParser.exe -h -o:CSV" for details
  lqof.oDQuotes = "OFF";  // OFF | AUTO | ON - doesn't make any difference!      
  lqof.fileMode = 1;  // 0 - append, 1 - overwrite, 2 - ignore

  var fields = [
    "[Buyer E-mail Address]",
    "[Order ID]",
    "[Shipping Addr 1]",
    "[Shipping Addr 2]",
    "[Shipping City]",
    "[Shipping Postal Code]",
    "[Buyer First Name]",
    "strcat([Buyer First Name], strcat(' ', [Buyer Last Name]))", // 
    "[Buyer Last Name]",
    "[Buyer Company]",
    "[Buyer Day Phone]"
  ];

  var sql = [
    "SELECT",
    fields.join(", "),
    "INTO", "chad_out.csv",
    "FROM", filename
  ].join(" ");

  DebugEcho(20, "query string:", sql);

  lq.ExecuteBatch(sql, lqif, lqof);
}

I'm afraid I can't actually give any data as it's confidential, but I hope that the illustration I have given is sufficient.

I do have other alternatives (i.e. Python csv) but this requires at least packaging the script into an executable (I do not wish to install Python as generally available software).

Can anyone spot something that I have obviously missed to control the behaviour of quoting or is this a deficiency of an otherwise powerful tool? Googling oDQuotes does not seem to be very productive.

Answer 1

It sounds like the quotes are in the input TSV file, aren't they? If that's the case, then the quotes are imported with the field values and you'll need to strip them off with your query (using SUBSTR(MyField, 1, -1)).

TSV does not expect quoted fields and thus it does not remove them.