Where is Pentaho Kettle's architecture?

Question

Where can I find Pentaho Kettle architecture? I'm looking for a short wiki, design document, blog post, anything to give a good overview on how things work. This question is not meant for specific "how to" starting guides but rather a good view at the technology and architecture.

Specific questions I have are:

1. How does data flow between steps? It would seem everything is in memory - am I right about this?
2. Is the above true about different transformations as well?
3. How are the Collect steps implemented?
4. Any specific performence guidelines to using it?
5. Is the ftp task reliable and performant?
6. Any other "Dos and Don'ts" ?

Answer 1

See this PDF.

Answer 2

1. How does data flow between steps? It would seem everything is in memory - am I right about this?

Data flow is row-based. For transformation every step produce a 'tuple' or a row with fields. Every field is pair of data and a metadata. Every step has input and output. Step takes rows from input, modify rows and send rows to outputs. For most cases every all information is in memory. But. Steps reads data in streaming fashion (like jdbc or other) - so typically in memory only a part of data from a stream.

2. Is the above true about different transformations as well?

There is a 'job' concept and 'transformation' concept. All written above is mostly true for transformation. Mostly - means transformation can contain very different steps, some of them - like collect steps - can try to collect all data from a stream. Jobs - is a way to perform some actions that do not follow 'streaming' concept - like send email on success, load some files from net, execute different transformations one by one.

3. How are the Collect steps implemented?

It only depend on particular step. Typically as said above - collect steps may try to collect all data from stream - having so - can be a reason of OutOfMemory exceptions. If data is too big - consider replace 'collect' steps with different approach to process data (for example use steps that do not collect all data).

4. Any specific performence guidelines to using it?

A lot of. Depends on steps transformation is consists, sources of data used. I would try to speak on exact scenario rather then general guidelines.

5. Is the ftp task reliable and performant?

As far as I remember ftp is backed by EdtFTP implementation, and there may be some issues with that steps like - some parameters not saved, or http-ftp proxy not working or other. I would say Kettle in general is reliable and perfomant - but for some not commonly used scenarios - it can be not so.

6. Any other "Dos and Don'ts" ?

I would say the Do - is to understand a tool before starting use it intensively. As mentioned in this discussion - there is a couple of literature on Kettle/Pentaho Data Integration you can try search for it on specific sites.

One of advantages of Pentaho Data Integration/Kettle is relatively big community you can ask for specific aspects.

http://forums.pentaho.com/

https://help.pentaho.com/Documentation