Splitting a foxpro table

Question 1

I don't know why you have all the text/endtext. You can just build your SQL-Select as one long statement... just use a semi-colon at the end of each line to indicate that the statement continues on the following line (unlike in C# that ; indicates end of statement)...

Anyhow, this simplified should do what you have

SELECT ;
      LEFT(ALLTRIM(group),ATC("|",ALLTRIM(group))-1) as Sim, ;
      STREXTRACT(ALLTRIM(group),"|","|",1) as Company, ;
      STREXTRACT(ALLTRIM(group),"|","|",2) as Fund, ;
      STREXTRACT(ALLTRIM(group),"|","|",3) as LPG_level, ;
      RIGHT(ALLTRIM(group),LEN(ALLTRIM(group)) - ATC("|",ALLTRIM(group),4)) as Output_group, ;
      time, ;
      SUM(as) as Asset_Share_Stressed, ;
      SUM(as_us) as Asset_Share_Unstressed ;
   FROM ;
      ( lcFile ) ;
   GROUP BY ;
      Sim, ;
      Company, ;
      Fund, ;
      LPG_level, ;
      Output_group, ;
      time ;
   ORDER BY ;
      sim asc,;
      output_group ASC ;
   INTO ;
      CURSOR bob

SELECT distinct LPG_Level ;
   FROM Bob ;
   INTO CURSOR C_TmpLevels

SELECT C_TmpLevels
SCAN 
   */ You might have to be careful if the LPG_Level has spaces or special characters
   */ that might cause problems in file name creation, but at your discretion.
   lcOutputFile = output_path + "LPG" + ALLTRIM( C_TmpLevels.LPG_Level ) + ".csv"
   SELECT Bob
   COPY TO ( lcOutputFile ) ;
      FOR LPG_Level = C_TmpLevels.LPG_Level ;
      TYPE csv
ENDSCAN

In this scenario, I just built your entire SQL query and ran it... From THAT result, I get distinct LPG_Level so it exactly matches the structure of the result set you have to work with. Notice in the "FROM" clause, I have the (lcFile) in parenthesis. This tells VFP to look to the variable name for the table name, not the actual table named "lcFile" as a literal. Similarly when I'm copying OUT to the CSV file... copy to (lcOutputFile).

Macros "&" can be powerful and useful, but can also bite you too especially if a file name path has a space in it... you are toast in that case... Try to get used to using parens in cases like this.

Question 2

Try something like:

FOR curlevel = 1 TO numlevels
    outfile = 'file' + ALLTRIM(STR(curlevel)) + '.csv'

    TEXT TO contents
        blah blah
    ENDTEXT

    = STRTOFILE(contents, outfile)
ENDFOR

You'll have to adjust things, but that's a technique to use.

Question 3

Thanks for all the help. I used the text/endtext thing on someone's advice from another question on here, when I was re-using blocks of code, but TBH its a bit overkill now so I might clean this up at some point.

I did get some help from someone else in my team who knows some SQL, but the solution looks remarkably similar to DRapp's.

Pasted below as a reminder for another time or to help anyone else out.

FOR i=1 TO lnCnt
lcFile = LOWER(output_path + laFiles[i,1])
lcFilename = LEFT(laFiles[i,1],ATC("~main4",laFiles[i,1])-1)

IF file(lcFile) = .F. then
    ERROR "File " + lcFile + " does not exist"
ENDIF

* Status window, tell user which file being processed
WAIT WINDOW (lcFile + ". File: " + ALLTRIM(STR(i)) + " of " + ALLTRIM(STR(lnCnt))) nowait

* identify the different LPG report levels
SELECT STREXTRACT(ALLTRIM(group),"|","|",3) as LPG_level FROM &lcFile GROUP BY LPG_level INTO CURSOR levels

* Build up the select string, starting with the sim number and company
TEXT to lcSql1 noshow textmerge pretext 15
   SELECT 
         LEFT(ALLTRIM(group),ATC("|",ALLTRIM(group))-1) as Sim, 
         STREXTRACT(ALLTRIM(group),"|","|",1) as Company, 
ENDTEXT

* Add the fund
TEXT to lcSql_fund2 noshow textmerge pretext 15
    STREXTRACT(ALLTRIM(group),"|","|",2) as Fund, 
ENDTEXT

* Add the Output group
TEXT to lcSqlgroup2 noshow textmerge pretext 15    
     RIGHT(ALLTRIM(group),LEN(ALLTRIM(group)) - ATC("|",ALLTRIM(group),4)) as Output_group, 
ENDTEXT

* Not actually required but helps clarify the output groups, the report level (e.g. tax-status, LoB, vclass)
TEXT to lcSql_level noshow textmerge pretext 15
    STREXTRACT(ALLTRIM(group),"|","|",3) as LPG_level, 
ENDTEXT

* Add the rest of the select, with the fields and their output names, the fields to group on and the sort order 
TEXT to lcSql2 noshow textmerge pretext 15
         time, 
         SUM(as) as Asset_Share_Stressed,
         SUM(as_us) as Asset_Share_Unstressed, 
         SUM(cogao) as CoGAO, 
         SUM(cog) as CoG, 
         SUM(cope) as CoPE, 
         SUM(cos) as CoS, 
         SUM(cope_wluk) as CoPE_WLUK, 
         SUM(copd_gteed) as CoPD_Gteed, 
         SUM(copd_other) as CoPD_Other,
         SUM(fprl_resid) as FPRL_resid,
         SUM(fprlcosadj) as FPRL_CoS_Adj,
         SUM(woc_res) as WOC_Reserve,
         SUM(bel) as BEL, 
         SUM(sht) as SH_Transfers, 
         SUM(pol_count) as Pol_Count 
      FROM <<lcFile>> 
      GROUP BY Sim, 
         Company, 
         Fund, 
         LPG_level,
         Output_group, 
         time 
      ORDER BY sim asc,
            output_group asc
      INTO CURSOR bob
ENDTEXT

* These are the variables which were created from the text above. This line combines them and runs them as a select statement
&lcSql1 + &lcSql_fund2 + &lcSql_level + &lcSqlgroup2 + &lcSql2

* Loop over the available report levels (previously exported to the cursor called levels
SELECT levels
SCAN
    * From bob (the cursor created above which has everything in it) for the current report level select out the relevant records
    * and save them to the original filename with the level appended in CSV format
    SELECT * FROM bob WHERE bob.LPG_Level = levels.LPG_Level INTO CURSOR temp
    COPY TO output_path + lcFilename + "_" + ALLTRIM(levels.LPG_Level) + ".csv" TYPE CSV
ENDSCAN


* Next file in list
NEXT i