filehelpers - Parsing variable line length

Question

I have to parse (C#) a .CSV file, with a variable "width" and 2 lines of header information (the fist one being name and the second one being the unit).

The data looks like:

Example1.CSV:

"timestamp","NAME_1","NAME_2","NAME_3","NAME_4"
"ms","unit_1","unit_2","unit_3","unit_4"
0.01,1.23,4.56,7.89,0.12
0.02,1.23,4.66,7.89,0.11
0.03,1.23,4.76,7.89,0.11
0.04,56.23,4.86,7.89,0.12

Example2.CSV:

"timestamp","NAME_1","NAME_2","NAME_3","NAME_4","NAME_5",...,"NAME_N"
"ms","unit_1","unit_2","unit_3","unit_4","unit_5",...,"unit_N"
0.01,1.23,4.56,7.89,0.12,0.13,...,0.27
0.02,1.23,4.66,7.89,0.12,0.13,...,0.22
0.03,1.23,4.76,7.89,0.11,0.13,...,0.24
0.04,56.23,4.86,7.89,0.12,0.13,...,0.29

With N being the "width" of the table (value can be up to 128 and larger). I'm planning to use Filehelpers.

I thought of using [FieldOptional()] - but that gets very unhandy, especially when the "width" is variable...

My current attempt looks like

[IgnoreFirst(2)]
[DelimitedRecord(",")]
public sealed class LogData
{

    public Double ts;

    public Double Field1;

    [FieldNullValue(0.0)]
    [FieldOptional()]
    public Double Field2;

    [FieldNullValue(0.0)]
    [FieldOptional()]
    public Double Field3;

    // and so on
}

Any help on "how to solve the variable width"-Problem in a more elegant manner is appreciated - Thank you very much in advance!

Ben

Answer 1

If you are planning to convert the file into DataTable, there is a better option

Please use CsvEngine of FileHelpers library. See the code snippet below:

using (MemoryStream stream = new MemoryStream(_fileContent)) //file content can be file as byte array
            {
                TextReader reader = new StreamReader(stream);
string path = "C:\\Sample.csv";
                CsvEngine csvEngine = new CsvEngine("Model", ',', path);
                var dataTable = csvEngine.ReadStreamAsDT(reader);
//Do whatever with dataTable

}

Here the sample file can be csv file or text file contains the header of the csv file you want to process. The columns of the DataTable will be named according to the header of the sample file

Cheers

Answer 2

You can use optional array field. I think you need to be using FileHelpers 2.9.9.

[IgnoreFirst(2)]
[DelimitedRecord(",")]
public class LogData
{
    public Double TimeStamp;

    [FieldNullValue(0.0)]
    [FieldOptional, FieldArrayLength(0, 100)]
    public Double[] ManyFields;
}

Here's a working example.

class Program
{
    static String content = 
         @"""timestamp"",""NAME_1"",""NAME_2"",""NAME_3"",""NAME_4""
         ""ms"",""unit_1"",""unit_2"",""unit_3"",""unit_4""
         0.01,1.23,4.56,7.89,0.12
         0.02,1.23,4.66,7.89,0.11
         0.03,1.23,4.76,7.89,0.11
         0.04,56.23,4.86,7.89,0.12";

    private static void Main()
    {
        var engine = new FileHelperEngine<LogData>();
        var records = engine.ReadString(content);
        Assert.AreEqual(0.01, records[0].TimeStamp);
        Assert.AreEqual(1.23, records[0].ManyFields[0]);
        Assert.AreEqual(4.56, records[0].ManyFields[1]);
        Assert.AreEqual(7.89, records[0].ManyFields[2]);
        Assert.AreEqual(0.12, records[0].ManyFields[3]);
    }
}