Split csv according to field and create Subdirectory then save :

Question

Would like to split a csv file according to the 2nd "field". For instance the csv file contains:

Input.csv :: /c/Test/

aaa,K1,ppp
ddd,M3,ppp
bbb,K1,ppp
ccc,L2,ppp

This file would be split into three separate files according to second field.

First file: /c/Test/K1/K1.csv

aaa,K1,ppp
bbb,K1,ppp

Second file: /c/Test/M3/M3.csv

ddd,M3,ppp

Third file: /c/Test/L2/L2.csv

ccc,L2,ppp

Tried the below command to split file based on 2nd column and working fine, however the splitted files on the same directory Like: /c/Test/K1.csv and /c/Test/M3.csv etc ..

awk -F, '{ print > $2".csv"}' Input.csv

Have tried the below command is not working to create subdirectory and incomplete , please help ..

awk -F, 'BEGIN { system("mkdir -p $2") } { print > $10".csv"}' Input.csv

awk -F, '{ print >/system("mkdir -p $2")/ $2".txt"}' Input.csv

Answer 1

awk -F, '{ system("mkdir -p "$2); print > $2"/"$2".csv"}' Input.csv

Assuming your Input.csv contains:

aaa,K1,ppp
ddd,M3,ppp
bbb,K1,ppp
ccc,L2,ppp

And this file is in /c/Test/ and you want directories to be created in /c/Test/.

The main difference with your attempt is system("mkdir -p "$2) i.e put $2 outside of quotes. This will concatenate "mkdir -p " and the value of $2. When you put it inside quotes it becomes literal $2 and the value is not available to mkdir command.

After the directory is created, it prints output to the desired file which has the path $2"/"$2".csv"

Answer 2

You could do this in bash using read:

#!/bin/bash

while IFS="," read a b c; do
    mkdir -p "$b" && echo "$a,$b,$c" >> "$b/$b.csv"
done < Input.csv

read splits the input line on the input field separator (IFS). It makes the directory based on the name of the second column and echos the line to the relevant file.

Or if you can use bash arrays:

#!/bin/bash

(
    IFS=","
    while read -a l
    do 
        mkdir -p "${l[1]}" && echo "${l[*]}" >> "${l[1]}/${l[1]}.csv"
    done < Input.csv
)

The use of ( ) means that the original value of $IFS is preserved once the script has run.