Processing wrapped lines with Fold

Question

I have a list of lines that that have hard breaks at a specific line number, and look like this:

Header:<SmallJson>
Header:<VeryLongJson...
....>
Header:<Json>

And I need to process it so it removes the line breaks, so it becomes like this:

Header:<SmallJson>
Header:<VeryLongJson.......>
Header:<Json>

I've come up with a solution but I'm not particularly happy about it:

let rawLines = [ "Header:<SmallJson>"
                 "Header:<VeryLongJson..."
                 "....>"
                 "Header:<Json>" ]

let processedLines = 
    (([], ""), rawLines)
    ||> List.fold (fun (result, state) line -> 
        if line.StartsWith "Header:"
        then state::result, line
        else result, state + line)
    |> (fun (result, state) -> state::result)
    |> List.rev
    |> List.tail

Is there a way to make this in a simpler way? The extra state::result and List.tail at the end of the fold particularly annoy me. Preferably without using mutation

Answer 1

It's actually simpler if you process from the end with foldBack, and in particular you don't need to reverse the result so it should be faster:

let processedLines =
    (rawLines, ("", []))
    ||> List.foldBack (fun line (currentLine, allLines) ->
        if line.StartsWith "Header:" then
            "", line + currentLine :: allLines
        else
            line + currentLine, allLines)
    |> function
        | "", lines -> lines
        | _ -> failwith "The original string didn't start with 'Header:'"

Answer 2

This is essentially the "chunking" problem, which already has some good answers on SO. I like Brian's approach here, which, translated to your problem, would be:

[ let linesToJoin = ResizeArray()
  for line in rawLines do
    if line.StartsWith("Header:") && linesToJoin.Count > 0 then
      yield String.Join("", linesToJoin)
      linesToJoin.Clear()
    linesToJoin.Add(line) 
  if linesToJoin.Count > 0 then
    yield String.Join("", linesToJoin) ]

It's not more elegant but I think the intent is clearer.

Another option is to use Tomas' groupWhen function (see usage).

Answer 3

You could do a very similar tail recursive function and avoid those two steps:

let rec combineLines (currentLine:string) combinedLines = function
| (line:string)::tail when line.StartsWith "Header:" && currentLine <> "" -> 
    combineLines line (currentLine::combinedLines) tail
| line::tail -> combineLines (currentLine + line) combinedLines tail
| [] -> currentLine::combinedLines

lines |> combineLines "" [] |> List.rev