Take a multi-delimited string in MySQL and convert to multiple records

Question

I am converting an old dataset into a newly structured database. Currently they have data in this format:

[quantity int]~[var1 string]|[var2 string optional]|[var3 string optional];(etc);
[quantity]~[var1]|[var2]|[var3]

where various patterns are delimited first by semicolons, then by tildes, and finally by pipe characters within the same field (no brackets) within the existing database. For instance:

3~S|Red|Top;1~S|Blue|Top; ... etc ... ;20~XL|Green|Left

I want to take that string in each record and do the following:

1. Create a new record in an existing table based off each semicolon split
2. Split each token again by the tilde, putting the first piece into one field and the second into a separate field (I don't care about splitting by pipe) in the same record

I don't know if it's even possible, but I'm hoping that it is. I am also hoping to only have to do this once, as the new system will take care of creating all this crap automatically. My sproc-fu is quite awful in MySQL, so any help is much appreciated.

Thanks so much!

Answer 1

This seems like a pretty gross thing to do in SQL, but here's a proof-of-concept script.

If the output looks right to you, replace the SELECT with an appropriate INSERT statement and it should get you what you want.

delimiter ;;

drop procedure if exists load_crazy_stuff;;

create procedure load_crazy_stuff(in s longtext)
begin
   declare pos       int;
   declare record    longtext;
   declare leftpart  int;
   declare rightpart longtext;
   set s = concat(s,';');
   while length(s)>0 DO
     set pos       = instr(s,';');
     set record    = left(s,pos-1);
     set s         = substr(s,pos+1);
     set pos       = instr(record,'~');
     set leftpart  = left(record,pos-1);
     set rightpart = substr(record,pos+1);
     select leftpart, rightpart;
   end while;
end;;

call load_crazy_stuff('3~S|Red|Top;1~S|Blue|Top;20~XL|Green|Left');;

Answer 2

The non-gross way to do it is this:

load data infile '/tmp/your-data-file' into table yourtable fields terminated by '~' lines terminated by ';';