Merge flat files

Question

I am trying to create a JCL for merging flat files using IEBGENER. The number of input files is not constant. Can we do it using IEBGENER?

Answer 1

Can you override IEBGENER SYSUT1 DD when invoking the PROC? Something like:

//EXEC procedure
//procstep.SYSUT1 DD DSN=first.copy.file,DISP=SHR,
//          DSN=second.copy.file,DISP=SHR,
//          DSN=thrid.copy.file,DISP=SHR
etcetera...

Where procedure is the catalogued procedure and procstep is the IEBGENER step.

When multiple datasets are given for a single DD statement they are concatenated together. As far as IEBGENER is concerned they should look like a single input dataset.

Answer 2

That's simple to do when we have input file count is not certain by the time job runs. Get the files from source created as generations of a GDG base and specify the base as input, which takes all the generations created so far... but every time the generations created so far will be considered; to avoid it; create a temp file with data from all generations and delete all of them, so that next time you'll have fresh set of generations to be considered. Am i clear? lemme know if not!

Answer 3

A. If you cannot have GDG's then empty files for the missing ones, in particular you then need to have a limit (10, 20 ?).

B. if you are using Control-M or similar , you can pre-process in the DD statement with an INCLUDE. The job cannot be submitted until the include material is ready, and it must be a separate job. That is to say, dynamically build your JCL from alternative decks based on the number of files.

C. You may have to write a program or CLIST to test for existence and concatenate the files.

Anybody that is using GDG's and reading them all at once should always remember the lastest generations are read first.