Loop through 2 slectboxes submitted values to map to an update query. Coldfusion

Question

I have 2 columns of select boxes. The first (left) is populated by all columns of an uploaded CSV file. The second (right) is all of the columns of a "Clients" table that they can import to. The number of pairs is determined by the number of total columns in the uploaded file.

Users can then go through and set what columns of their data will update which columns in our Clients table. For instance, they would set the first box in the left to "Email" and the first box on the right to "Email" and their emails would be updated to the email column in our DB.

If they have a column called "Organization" and we only have "Company" then they can set it accordingly to update. Basically mapping their imported clients, so they can use a wider range of column name convention.

I already have the loops setup to populate from some help here.

Now I'm trying to update the query. Here's the selectboxes after the file is uploaded.

<form class="formContent960" id="csvmap" name="csvmap" method="post" action="custom_upload_update.cfm">
    <table class="form960" cellpadding="5">
        <tbody>
            <!--- Set Uploaded file to Array --->
            <cfset arrCSV = CSVToArray(CSVFilePath = #form.UploadedFile#,Delimiter = ",",Qualifier = """") />
            <!--- Create Key array from column names --->

            <cfloop from="1" to="#ArrayLen(arrCSV[1])#" index="t">
                <!--- Variable Headers --->
                <cfif Len(form.UploadedFile) GTE 5>
                <cfoutput>
                    <select name="upfield[#t#]" class="search" id="Header">
                </cfoutput>
                    <option selected value="">--- Headers Uploaded ---</option>
                <cfoutput>
                <cfloop from="1" to="1" index="i">
                    <cfloop from="1" to="#ArrayLen(arrCSV[i])#" index="j">
                    <option value="#arrCSV[i][j]#">#arrCSV[i][j]#</option>

                    </cfloop>
                </cfloop>
                </cfoutput>
                    </select> =
                </cfif>
                <!---Column Constants--->
                <cfoutput>
                    <select name="bofield[#t#]" class="search" id="Column">
                </cfoutput>
                    <option selected value="">--- Headers Clients ---</option>
                        <cfoutput>
                            <cfloop query="clientsCols">
                            <option value="#Column_name#">#Column_name#</option>
                            </cfloop>
                        </cfoutput>
                    </select><br /><br />
                </cfloop>
            </tbody>

        <cfoutput>
        <input type="hidden" name="filelength" id="filelength" value="#ArrayLen(arrCSV[1])#">
        </cfoutput>
        <input type="submit" name="csvmapsubmit" id="csvmapsubmit">

    </table>
</form>

So I'm thinking I need to set a variable containing the values of the Clients(Right) columns select string to set which columns to update in the query inside of a loop.

Then set the uploaded fields to update the data in those rows inside a sub loop for the values.

Like:

<cfloop>
<cfset bostring = "#bofields#"/> 
</cfloop>
<cfloop>
<cfquery name="addclientubmit" datasource="#request.dsn#">
        INSERT INTO Clients
        (
            #bostring#
        )
        VALUES 
        (
        <cfloop>
            #uploaded Values#
        </cfloop>   
        )
        </cfquery>
</cfloop>

Not working with proper syntax, just trying to include my general logic of the issue for discussion purposes.

Any help would be appreciated. Thank you in Advance,

Steve

Answer 1

Alternate Approach

Before I get to your current form, let me mention another option: using your database's import tools, like OPENROWSET or BULK INSERT. The former is a little more flexible it can be used from a SELECT statement. So you could do a direct insert from the CSV file, no looping. (I usually prefer to insert into a temp table first. Run a few validation queries, then insert/select the data into the primary table. But it depends on the application ..)

Anyway, once you have validated the column names, the insert with OPENROWSET is just a single query:

<!--- see below for how to validate list of column names --->
<cfquery name="insertRawData" datasource="yourDSN">
   INSERT INTO YourTable ( #theSelectedColumnNames# )
   SELECT  * 
   FROM    OPENROWSET( 'Microsoft.Jet.OLEDB.4.0'
            ,'text;HDR=YES;Database=c:\some\path\'
            , 'SELECT * FROM [yourFileName.csv]'  )
</cfquery>

Current Approach

Form:

Using your current method you would need to read the CSV file twice: once on the "mapping" page and again on the action page. Technically it could be as simple as giving the db column select lists the same name. So the names would be submitted as a comma delimited list:

<cfset csvHeaders = csvData[1]>
<cfloop array="#csvHeaders#" index="headerName">
    <cfoutput>
        Map file header: #headerName# 
        to column:  
        <select name="targetColumns">
            <option value="" selected>--- column name---</option>
            <cfloop query="getColumnNames">
                <option value="#column_name#">#column_name#</option>
            </cfloop>
        </select>
    </cfoutput>
    <br>
</cfloop>

Validate Columns:

Then re-validate the list of column names against your db metadata to prevent sql injection. Do not skip that step!. (You could also use a separate mapping table instead, so as not to expose the db schema. That is my preference.)

<cfquery name="qVerify" datasource="yourDSN">
    SELECT COUNT(COLUMN_NAME) AS NumberOfColumns    
    FROM   INFORMATION_SCHEMA.COLUMNS
    WHERE  TABLE_NAME = 'YourTableName'
    AND    COLUMN_NAME IN 
           (
        <cfqueryparam value="#form.targetColumns#" cfsqltype="cf_sql_varchar">
           )
</cfquery>

<cfif qVerify.recordCount eq 0 OR qVerify.NumberOfColumns neq listLen(form.targetColumns)>
    ERROR. Missing or invalid column name(s) detected
    <cfabort>
</cfif>

Insert Data:

Finally re-read the CSV file and loop to insert each row. Your actual code should contain a LOT more validation (handling of invalid column names, etcetera) but this is the basic idea:

<cfset csvData  = CSVToArray(....)>
<!--- deduct one to skip header row --->
<cfset numberOfRows = arrayLen(csvData) - 1>    
<cfset numberOfColumns = arrayLen(csvData[1])>  
<cfif numberOfColumns eq 0 OR numberOfColumns neq listLen(form.targetColumns)>
    ERROR. Missing or invalid column name(s) detected
    <cfabort>
</cfif>

<cfloop from="1" to="#numberOfRows#" index="rowIndex">
    <cfquery ...>
        INSERT INTO ClientColumnMappings ( #form.targetColumns# )
        VALUES 
        (
            <cfloop from="1" to="#numberOfColumns#" index="colIndex">
                 <cfif colIndex gt 1>,</cfif>
                 <cfqueryparam value="#csvData[rowIndex][colIndex]#" cfsqltype="cf_sql_varchar">
            </cfloop>
        )
    </cfquery>
</cfloop>

Answer 2

See if this will assist you. Please note that I have modified your initial code for demonstration purposes, but have denoted so you should be able to wire back up to test. This can be tricky... but should give you a good starting point.

Please note that there are new tools available within Coldfusion for processing CSV files - I wrote my utilities in 2008 for CF 8, but they still are in use today. Compare and contrast what works for you.

Hope this helps.

=== cfm page

<!---import csv utility component (modify for your pathing)--->
<cfset utilcsv = CreateObject("component","webroot.jquery.stackoverflow.csvColumnMap.utils_csv_processing_lib")>
<!---declare the csv file (modify for your pathing)--->
<cfset arrCSV = utilcsv.readinCSV(ExpandPath('./'),'Report-tstFile.csv') />
<!---declare the header row column values--->
<cfset headerRow = listToArray(arrCSV[1],',')>
<!---declare the column names query--->
<cfset q = QueryNew('offer,fname,lname,address,city,state,zip',
                    'CF_SQL_VARCHAR,CF_SQL_VARCHAR,CF_SQL_VARCHAR,CF_SQL_VARCHAR,CF_SQL_VARCHAR,CF_SQL_VARCHAR,CF_SQL_VARCHAR')>
<cfset colList = q.columnList>  

<!---form submission processing--->
<cfif isdefined("form.csvmapsubmit")>

    <cfset collection = ArrayNew(1)>
    <!---collect the column and column map values : this step could be eliminated by 
    just assigning the the arrays in the next step, however this allows reference for 
    dump and debug--->
    <cfloop collection="#form#" item="key">
        <cfif FIND('BOFIELD',key) && trim(StructFind(form,key)) neq "">
            <cfset fieldid = ReREPLACE(key,"\D","","all")>
            <cfset valueKey = 'UPFIELD[' & fieldid & ']'>
            <cfset t = { 'column'=StructFind(form,key),'value'=StructFind(form,valueKey) }>
            <cfset arrayappend(collection,t)>
        </cfif>
    </cfloop>

    <!---collect the column and column map values : this ensures that the table column is in the same position as the mapped column for the sql statement--->
    <cfset tblColsArr = ArrayNew(1)>
    <cfset valColsArr = ArrayNew(1)>
    <cfloop index="i" from="1" to="#ArrayLen(collection)#">
        <cfset arrayappend(tblColsArr, collection[i]['column'])>
        <cfset arrayappend(valColsArr, collection[i]['value'])>
    </cfloop>

    <!---convert the uploaded data into an array of stuctures for iteration--->
    <cfset uploadData = utilcsv.processToStructArray(arrCSV)>

    <!---loop uploaded data--->
    <cfloop index="y" from="1" to="#ArrayLen(uploadData)#">

        <!---create sql command for each record instance--->
        <cfset sqlCmd = "INSERT INTO Clients(" & arraytolist(tblColsArr) & ") Values(">
        <cfloop index="v" from="1" to="#ArrayLen(valColsArr)#">
            <!---loop over the column maps to pull the approriate value for the table column--->
            <cfif isNumeric(trim(valColsArr[v])) eq true>
                <cfset sqlCmd &= trim(uploadData[y][valColsArr[v]])>
             <cfelse>
                <cfset sqlCmd &= "'" & trim(uploadData[y][valColsArr[v]]) & "'">
             </cfif>
             <cfset sqlCmd &=  (v lt ArrayLen(valColsArr)) ? "," : ")" >
        </cfloop>

        <!---perform insert for record--->
        <!--- 
        <cfquery name="insert" datasource="">
        #REReplace(sqlCmd,"''","'","ALL")# <!---In the event that the quotation marks are not formatted properly for execution--->
        </cfquery>
        --->
    </cfloop>

</cfif>

<form class="formContent960" id="csvmap" name="csvmap" method="post">
<table class="form960" cellpadding="5">
    <tbody>
    <cfloop from="1" to="#ArrayLen(headerRow)#" index="t">
    <tr>
        <td>
        <!--- Variable Headers --->
        <cfif ArrayLen(headerRow) GTE 5>
            <cfoutput>
            <select name="upfield[#t#]" class="search" id="Header">
                <option selected value="">--- Headers Uploaded ---</option>
                <cfloop from="1" to="#ArrayLen(headerRow)#" index="j"><option value="#headerRow[j]#">#headerRow[j]#</option></cfloop>
            </select> =
            </cfoutput>                
        </cfif>
        </td>
        <td>
        <!---Column Constants--->
        <cfoutput>
        <select name="bofield[#t#]" class="search" id="Column">            
            <option selected value="">--- Headers Clients ---</option>
            <cfloop list="#colList#" index="li" delimiters=","><option value="#li#">#li#</option></cfloop>
        </select>
        </cfoutput>
        </td>
    </tr>
    </cfloop>
    <tr>
    <td>&nbsp;</td>
    <td>
    <cfoutput>
    <input type="hidden" name="filelength" id="filelength" value="#ArrayLen(headerRow)#">
    </cfoutput>
    <input type="submit" name="csvmapsubmit" id="csvmapsubmit">
    </td>
    </tr>
    </tbody>
</table>
</form>

== utils_csv_processing_lib.cfc

<!---////////////////////////////////////////////////////////////////////////////////
////                    CSV File Processing - Read In File                      /////
////                Return is array with each array item being a row            /////
////                            9.22.08 BP                                      /////
////                                                                            /////
/////////////////////////////////////////////////////////////////////////////////---> 
    <cffunction name="readinCSV" access="public" returntype="array">
        <cfargument name="fileDirectory" type="string" required="yes">
        <cfargument name="fileName" type="string" required="yes">  
        <!---/// 1. read in selected file ///--->
        <cffile action="read" file="#fileDirectory##fileName#" variable="csvfile">
    <!---/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
//  2. set csv file to array ***Note; the orginal csv file ListToArray only used the carrige return/line return as delimiters, ///
//  so each array value/member is a full record in comma delimited format (i.e.: 01, Fullname, Address1, City, etc) //////////--->    
        <cfset csvList2Array = ListToArray(csvfile, "#chr(10)##chr(13)#")> 

        <cfset ret = checkCSVRowLengths(csvList2Array)>
        <cfreturn ret>
    </cffunction>

<!---////////////////////////////////////////////////////////////////////////////////
////                    Create Structured Array of CSV FILE                     /////
//// Return is a structured array uing the colmn header as the struct element name //
////                            9.22.08 BP                                      /////
////                                                                            /////
////                    ****UPDATED 1.6.09**********                            /////
////            Added empty field file processing - takes empty value           /////
////                        and replaces with "nul"                             /////
////                                                                            /////
/////////////////////////////////////////////////////////////////////////////////---> 
    <cffunction name="processToStructArray" access="public" returntype="array">
        <cfargument name="recordFile" type="array" required="yes">


     <!---retrieve the placeholder we are setting for strings containing our default list delimiter (",")--->
        <cfinvoke component="utils_csv_processing_lib" method="SetGlobalDelimiter" returnvariable="glblDelimiter">

    <!---/// 1. get length of array (number of records) in csv file ///--->
        <cfset csvArrayLen = ArrayLen(recordFile)>

        <!---/////////////////////////////////////////
        ////        EMPTY VALUE Processing          //
        //////////////////////////////////////////--->
                <!---// a. create array to hold updated file for processing--->
                    <cfset updatedRowsFnlArr = ArrayNew(1)>

                <!---// b. loop entire csv file to process each row--->
                    <cfloop index="li2" from="1" to="#csvArrayLen#">

                    <!---// c. grab each column (delimited by ",") for internal loop. *******The value of each array index/item is a comma delimited list*******--->
                    <cfset currRecRow = #recordFile[li2]#>

                    <!---/// d. loop each row in file--->
                    <cfloop list="#currRecRow#" index="updateRowindex" delimiters="#chr(10)##chr(13)#">
                          <!---// e. find and replace empty column values in list with a set value for processing--->
                          <!---consolidated for single list output per array index: regenerates a value of val,val,val for a value of val,,val--->

                          <!---// process middle positions in list //--->
                          <cfset currRowListed = updateRowindex>
                          <cfset updatedRowListed = REreplace(currRowListed,",,",",nul,","ALL")>
                          <cfset updatedRowListed = REreplace(updatedRowListed,",,",",nul,","ALL")>
                          <!---// process 1st position in list //--->
                          <cfset frstpos = REFIND(",",updatedRowListed,1)>
                          <cfif frstpos EQ 1>
                          <cfset updatedRowListed = REReplace(updatedRowListed,",","nul,","one")>
                          </cfif>
                          <!---// process last position in list //--->
                          <cfset rowStrngLen = Len(updatedRowListed)>
                          <cfset lastpos = REFIND(",",updatedRowListed,rowStrngLen)>
                          <cfif lastpos EQ rowStrngLen>
                          <cfset updatedRowListed = updatedRowListed & "nul">
                          </cfif>

                          <!---// f. append current row with updated value of 'nul' for empty list positions to array--->
                          <cfset ArrayAppend(updatedRowsFnlArr, updatedRowListed)>
                     </cfloop>
                </cfloop>

        <!---/// 2. get number of records in updated array--->
        <cfset updatedRowsFnlLen = ArrayLen(updatedRowsFnlArr)>

        <!---/// 3. set the first item in the array to a variable (at postion 1). This will set the entire first record to the variable, delimited by commas ///--->
        <cfset getRecColumns = updatedRowsFnlArr[1]>

        <!---/// 4. get length of 1st record row, which will tell us hom many columns are in the csv file ///--->
        <cfset ColumnCount = ListLen(updatedRowsFnlArr[1],",")>

        <!---/// 5. create array to hold value for return and start loop of list *****Loop started at 2 to exclude header row***** ///--->
        <cfset recordArr = ArrayNew(1)>
        <cfloop index="i" from="2" to="#updatedRowsFnlLen#">

        <!---/// 6. grab each column (delimited by ",") internal loop. The value of each array index/item is a comma delimited list ///--->
        <cfset currRecRow = #updatedRowsFnlArr[i]#>

        <!---/// 7. We now create a structure and assign each row value to the corresponding header within the structure ///--->
        <cfset recordStruct = StructNew()>
        <cfloop index="internal" from="1" to="#ColumnCount#">
            <!---conditional to set the 'nul' value added for empty list position values in order to process back to empty values--->
            <cfif listGetAt(currRecRow,internal,",") NEQ 'nul'>

                    <!---check for global placeholder delimiter and reset to ","--->
                    <cfif FIND(glblDelimiter,listGetAt(currRecRow,internal,",")) NEQ 0>
                        <cfset resetDelimiterVal = Replace(listGetAt(currRecRow,internal,","),glblDelimiter,',','All')>
                    <cfelse>
                        <cfset resetDelimiterVal = listGetAt(currRecRow,internal,",")>
                    </cfif>

                <cfset recordStruct[listGetAt(getRecColumns,internal,",")] = resetDelimiterVal>
            <cfelse>
                <cfset recordStruct[listGetAt(getRecColumns,internal,",")] = "">
            </cfif>
        </cfloop>
        <!---/// 8. append the struct to the array ///--->
        <cfset ArrayAppend(recordArr,recordStruct)>
        </cfloop>
        <cfreturn recordArr>
    </cffunction>

<!---////////////////////////////////////////////////////////////////////////////////
////                              SetGlobalDelimiter                            /////
////    Sets a placeholder for strings containing the primary delimiter (",")   /////
////                                    02.6.11 BP                              /////
/////////////////////////////////////////////////////////////////////////////////--->     
    <cffunction name="SetGlobalDelimiter" access="public" returntype="string" hint="set a placeholder delimiter for the strings that contain the primary list comma delimiter">
        <cfset glblDelimiter = "{_$_}">
      <cfreturn glblDelimiter>
    </cffunction>  

===missing cfc function

<!---////////////////////////////////////////////////////////////////////////////////////////////////////////
////                        checkCSVRowLengths                                                          /////
////  due to some inconsistencies in excel, some csv files drop the delimiter if list is empty          /////
////                            7.20.11 BP                                                              /////
/////////////////////////////////////////////////////////////////////////////////////////////////////////---> 
<cffunction name="checkCSVRowLengths" access="public" returntype="array">
        <cfargument name="readArray" type="array" required="yes">

 <cfset column_row = readArray[1]>
       <cfset column_row_len = listlen(column_row,',')>

       <cfloop index="i" from="2" to="#ArrayLen(readArray)#">
            <cfset updateRowindex = readArray[i]>


            <cfif listlen(updateRowindex) lt column_row_len>

                         <!---// process middle positions in list //--->
                          <cfset currRowListed = updateRowindex>
                          <cfset updatedRowListed = REreplace(currRowListed,",,",",nul,","ALL")>
                          <cfset updatedRowListed = REreplace(updatedRowListed,",,",",nul,","ALL")>
                          <!---// process 1st position in list //--->
                          <cfset frstpos = REFIND(",",updatedRowListed,1)>
                          <cfif frstpos EQ 1>
                          <cfset updatedRowListed = REReplace(updatedRowListed,",","nul,")>
                          </cfif>
                          <!---// process last position in list //--->
                          <cfset rowStrngLen = Len(updatedRowListed)>
                          <cfset lastpos = REFIND(",",updatedRowListed,rowStrngLen)>
                          <cfif lastpos EQ rowStrngLen>
                          <cfset updatedRowListed = updatedRowListed & "nul">
                          </cfif>
             <cfelse>

                <cfset  updatedRowListed  = updateRowindex>    

            </cfif>

            <cfif listlen(updatedRowListed) lt column_row_len>

                <cfset lc = column_row_len - listlen(updatedRowListed)>

                <cfloop index="x" from="1" to="#lc#">
                    <cfset updatedRowListed = updatedRowListed & ',nul'>
                </cfloop>


            </cfif>


            <cfset readArray[i] = updatedRowListed>
       </cfloop>



        <cfreturn readArray>
    </cffunction>