How to use awk for a compressed file

Question

How can I change the following command for a compressed file?

awk 'FNR==NR { array[$1,$2]=$8; next } ($1,$2) in array { print $0 ";" array[$1,$2] }' input1.vcf input2.vcf

The command working fine with normal file. I need to change the command for compressed files.

Answer 1

You need to read them compressed files like this:

awk '{ ... }' <(gzip -dc input1.vcf.gz) <(gzip -dc input2.vcf.gz)

Try this:

awk 'FNR==NR { sub(/AA=\.;/,""); array[$1,$2]=$8; next } ($1,$2) in array { print $0 ";" array[$1,$2] }' <(gzip -dc input1.vcf.gz) <(gzip -dc input2.vcf.gz) | gzip > output.vcf.gz

Answer 2

zcat FILE | awk '{ ...}'

I wouldn't be able to tell which of all these methods works best, zcat is at least quicker to type ;)

Answer 3

bzip2 -dc input1.vcf.bz2 input2.vcf.bz2 | awk 'FNR==NR { array[$1,$2]=$8; next } ($1,$2) in array { print $0 ";" array[$1,$2] }'

or

gzip -dc input1.vcf.gz input2.vcf.gz | awk 'FNR==NR { array[$1,$2]=$8; next } ($1,$2) in array { print $0 ";" array[$1,$2] }'

EDIT:

To write compressed output just append

| bzip2 >output.vcf.bz2

or

| gzip >output.vcf.gz

This will work with any program that prints results to standard output.

BTW: Editing such large command lines gets tedious very quickly. You should consider writing a small shell script to do the job. This has the additional benefit that you don't have to remember the entire thing and can easily repeat the command or modify it if necessary.

A good starting point for Linux shell programming is the Bash Programming Inroduction by Mike G.