foreach thing `cat file.txt` loop - how to remove newline character from next line? and append to current? - prevent word splitting in tcsh

Question

I have a text file that I am reading line by line using foreach cat file.txt in tcsh. The last field sometimes has a comment. For some reason when I am reading each line in the file, cat (correctly) does not break up my comment into words (this is correct) during the initial cat file.txt(on the command line). But it does break up the comment field when I do foreach thing (cat file.txt) and echo.

When I start processing the file with awk (assigning fields to variables and soforth) the comment in the last field starts breaking up into a new line. This is what it is being processed like:

 SAMEWORD,1111,2622,33,.00000,.000000,1,0,9,0,9,0,
 SAMEWORD,1231,2222,33,.00000,.000000,1,0,9,0,9,0,
 SAMEWORD,1411,2422,34,.00000,.000000,1,0,9,0,9,0,COMMENT
 STARTS
 HERE
 BUT
 IS 
 BEING
 BROKEN 
 INTO
 NEW
 LINES
 SAMEWORD,1441,2672,53,.00000,.000000,1,0,9,0,2,0,
 SAMEWORD,1351,2372,63,.00000,.000000,1,0,1,0,9,0,
 SAMEWORD,1351,2372,63,.00000,.000000,1,0,6,0,9,0,
 SAMEWORD,1351,2372,63,.00000,.000000,1,0,6,0,9,0,HERE
 IS
 ANOTHER
 EXAMPLE
 SAMEWORD,1441,2672,53,.00000,.000000,1,0,9,0,2,0,
 SAMEWORD,1441,2672,53,.00000,.000000,1,0,9,0,2,0,

I tried using printf instead of print in my awk command but that hasn't helped the problem. If I am reading a current line the last field isn't "" then how do I append the next line to the current last field without having to lose my place in the main for loop?

[I should add that I need to process with awk because in my script I'm checking the first field for a condition (for example, if ($firstfield == "SAMEWORD") then...) and this is where the last field is, for some reason, being broken into new lines/records.]

I've tried so many variations of sed, awk, tr, cat, and I can't seem to come up with anything that works.

Also acceptable would be a way to (while on the current line) check what the first field is in the next line and if it's not SAMEWORD then append to last field of current line.

Answer 1

Sounds like the problem might be solved by preventing the tcsh foreach loop from splitting lines into words, so that the input lines are passed through intact.

Try double-quoting the command substitution:

foreach line ("`cat file.txt`")
    # process line with awk
end

On a side note: can't you just let awk process the input file directly?