Better solution for High Performance ETL Flat File Extraction? [closed]

Question

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.

---

Want to improve this question? Update the question so it's on-topic for Stack Overflow.

Closed 6 years ago.

Improve this question

I have a data set that is structured as follows:

[TIMESTAMP] FIXED POSITION       DATA STRING
[TIMESTAMP] FIXED POSITION       DATA STRING
[TIMESTAMP] tag1=9324, tag2=19, tag3=55, etc
[TIMESTAMP] FIXED POSITION       DATA STRING

These two types of records are intermixed, with the tag-based field containing summary data at a regular interval. These data files can be as large as 10 million rows for a single file. Additionally, some of the fixed position lines can be a couple kilobytes long - leading to a memory management problem when I process them.

I need to process the file from the beginning, grouping every entry when I hit a summary entry, and recording those summary entries along with aggregates of certain fields from the fixed position entries. Specifically, one specific part of the fixed position segment contains a 4-digit status code. I need to add that data to the summary record - # of occurrences of each summary code that appeared since the last summary record. Each summary entry, including fields for the aggregate data and timestamp, should be turned into a row in a CSV.

I am currently using ruby code / standard library only to do this. For smaller data sets it is fine, but for larger data sets, performance quickly deteriorates. This seems to me like a common ETL-type problem. I will be doing more complex types of operations on this data later, and seem to constantly be revisiting these log files to do more complex reporting on them.

Is there an existing ETL tool or library (pref. Ruby) that

1. Does high performance mixed (regex / fixed position) extraction well.
2. Is easy to learn.
3. Abstracts away memory management.
4. Is free / open source or low cost (under $200) commercial.

I am open to alternate recommendations on what to use to tackle this problem - I'm just looking for a better solution than raw standard-library ruby code.

Answer 1

I've done a java library call http://jrecordbind.org/ that uses xml schema to define the format of the input/output file

I suppose your case can be expressed in the form of a "choice" element

<xs:complexType name="Choice">
<xs:choice>
  <xs:element name="one" type="One"/>
  <xs:element name="two" type="Two"/>
</xs:choice>
</xs:complexType>
<xs:complexType name="One">
  <xs:sequence>
    <!-- fixed row definition -->
  </xs:sequence>
</xs:complexType>
<xs:complexType name="Two">
  <xs:sequence>
    <!-- tag row definition -->
  </xs:sequence>
</xs:complexType>

This piece of xsd is taken from one of the tests that runs with this input file.

You should then use the parsed java bean and choose what to do depending if one of the resulting methods getOne() and getTwo() return null.

JRecordBind has a "stream" approach (even though it really depends on the schema), so memory consumption is kept to a minimum.