Downloading file from online database using bash script

Question

I want to download some files from an online database, but it does not allow me to download all the files at once. Instead it offers to download a file for a searched keyword. Because I have more than 20000 keywords, it's not feasible for me. For example, I want to download whole information about miRNA-mRNA interaction from SarBase, but it does not offer an option to download all of them at once. I wonder, how can I download it by writing some scripts. Can anybody help me?

Answer 1

Make a file called getdb.sh.

#!/bin/bash
echo "Download keywords in kw.txt."
for kw in $(cat kw.txt)
do
    curl http://www.mirbase.org/cgi-bin/get_seq.pl?acc=$kw > $kw.txt
done

Create another file called kw.txt:

MI0000342
MI0000343
MI0000344

Then run this

$ chmod +x getdb.sh
$ ./getdb.sh

Download keywords in kw.txt.

$ ls -1 *.txt
kw.txt
MI0000342.txt
MI0000343.txt
MI0000344.txt

Answer 2

another way

cat kw.txt |xargs -i curl -o {}.txt http://www.mirbase.org/cgi-bin/get_seq.pl?acc={}