Talend job doesn't run properly when using several tBufferOuput/tBufferInput couples in in it

Question

I'm developing a big job in Talend Open Studio (about 90 components now and minimum 150 at the end of the development). But the number of components is limited in the subjobs because the Java method of a subjob can't exceed the 65536 bytes limit.

So I split my job in multiple subjobs using tBufferOuput/tBufferInput couple to pass data between each subjob. And now the problem is that I need to empty the globalBuffer before each tBufferOuput.

I've searched on the web and I found a solution with a tJava component using globalBuffer.clear(); in it but when I do that, my job finishes without handling any data.

Answer 1

If you aren't passing the data to a parent job and are instead keeping the data inside the same job (just multiple subjobs inside) you'd probably be better off with the tHash components. This allows you to cache some data (either in memory or temporarily to disk) and then retrieve that data by linking a tHashInput to a specific tHashOutput.

The tBuffer components just drop all the data into one specific pool and pick up from there so aren't really suited for multiple inputs and outputs to a job (although this can be a desired result). You're best off only using them for passing data back to a parent job by using a tBufferOutput in a child job and then linking the tRunJob in the parent job to whatever you want to pass the data to (which could be another job).