Iterating over multiple txt files and creating a new dataset for each in SAS

Question

I am stuck with a problem in SAS. I have a bunch of monthly weather data in individual txt-files. My current goal is to read those in and create a separate data set for each. Alternatively, I could see it being possible to skip this step and go closer to end goal of merging all these data sets to another data set by the date and time. Below was my try at the problem. I thought a macro would work that iterates through the file names and creates matching data set names, but apparently it does not. Also, to make it more efficient the if/else if statements I think can be replaced by a DO loop but I could not figure it out. Help is much appreciated!

%macro loop; 
%do i = 11 %to 13; 
%do j = 01 %to 12; 
    %let year = i; 
    %let month = j;
    data _&year&month ; 
        infile "&path\hr_pit_&year..&month..txt" firstobs=27;  
        length Time $ 4 Month $ 3 Day $ 2 Year $ 4 temp 3; 
        input time $ Month $ 10-13 Day Year temp 32-34; 
        Date = Day||Month||Year;
        if time = '12AM' then time = 2400;
        else if time = '1AM ' then time = 100; 
        else if time = '2AM ' then time = 200; 
        else if time = '3AM ' then time = 300; 
        else if time = '4AM ' then time = 400; 
        else if time = '5AM ' then time = 500; 
        else if time = '6AM ' then time = 600; 
        else if time = '7AM ' then time = 700; 
        else if time = '8AM ' then time = 800; 
        else if time = '9AM ' then time = 900; 
        else if time = '10AM' then time = 1000;
        else if time = '11AM' then time = 1100; 
        else if time = '12PM' then time = 1200;
        else if time = '1PM ' then time = 1300;
        else if time = '2PM ' then time = 1400;
        else if time = '3PM ' then time = 1500;
        else if time = '4PM ' then time = 1600;
        else if time = '5PM ' then time = 1700;
        else if time = '6PM ' then time = 1800;
        else if time = '7PM ' then time = 1900;
        else if time = '8PM ' then time = 2000;
        else if time = '9PM ' then time = 2100;
        else if time = '10PM' then time = 2200;
        else if time = '11PM' then time = 2300;
        _time = input(time,4.);
        time = _time; 
        drop month day year; 
    run; 
%end; 
%end; 
%mend; 

%loop; run: 

In case anyone is wondering this is how a typical txt file looks: http://www.erh.noaa.gov/pbz/hourlywx/hr_pit_13.01

Here is a list of txt files in the same shape and form: http://www.erh.noaa.gov/pbz/hourlyclimate.htm

Answer 1

First fixes in:

%let year = &i; 
%let month = %sysfunc(putn(&j, z2.));

to use macro variables and add leading zero to month. The rest of changes is just dealing with AM/PM. Also the Date is now numeric.

Full code:

%macro loop; 
%do i = 11 %to 13; 
%do j = 1 %to 12; 
    %let year = &i; 
    %let month = %sysfunc(putn(&j, z2.));
    data _&year&month ;
        length Date 5 _Time $4 Time 8 Month $3 Day $2 Year $4 temp 3; 
          format Date DATE9.; 
        infile "&path\hr_pit_&year..&month..txt" firstobs=27;  

    input _time $ Month $ 10-13 Day Year temp 32-34; 
    _time = right(_time);
    Date = input(Day||Month||Year, date9.);
    if _time = '12AM' or (_time ne '12PM' and index(_time, 'PM') > 1 )
            then time=input(_time, 2.) + 12;
    else time=input(_time, 2.);
    time = time * 100;
    drop month day year;
run; 
     /* gather all data in one table */
    proc append base=work.all_data data=work._&year&month;
    run;
%end; 
%end; 
%mend; 


proc sql;
drop table work.all_data;
quit;
%let path=E:;
%loop; 

Answer 2

Sounds like the best answer may be to read them all into one dataset and then merge them to the final dataset from there. I think you also are served better by using a real time value, rather than 100-2400 (and an inconsistent 2400, that really should be 000 if you're doing that) - then you can just use input.

Anyway, if you just read the text files in like so:

data my_text_files;
infile "c:\mydirectory\*.txt" lrecl=whatever eov=eovmark;
*firstobs=27 is only respected for the first file - so we have to track with eovmark;
if eovmark then do;
  eovmark=0;
  linecounter=0;
end;
linecounter+1;
if linecounter ge 27 then do;
  input (input statement);
  (any other code you want to execute here);
  output;
end;
run;

Then merge by (whatever). If you need to know some information about the filename you can use the filename option to get access to that in the infile statement.