automate multiple file conversions

Question

I need help fixing my script to automate multiple file conversions.
I am working on the server and I am trying to find a way to convert files automatically.

The conversion needs to be made through four formats: sam, bwa, and bcf, and vcf
The files increment at two spots like this. Each increment has two file formats: .sai and .fastq

004_001
004_002
004_003
004_004

Then 

005_001
005_002
005_003
005_004

I have tried:

Code:

for i in {1..4}
do 
    $bwa samse $hg19 $csai $cfas > *.sam
done 

But it only worked with the _001 file all four times.
I have tried setting the end increment to i so instead of _001 it's _00$((i)) to go 1,2,3,4 but its stays at 1.

Example of sam conversion:

for i in {1..4}
do 
./bwa samse  /hg19.fa  *_00$((i)).sai *_00$((i)).fastq  > *_00$((i)).sam
done

I wrote it to run input files with that name and output data to the sam file.  It works but it converts _001 four time instead of _001 to _004

Basically I need to write a script that will automate multiple file conversions

• 3 file inputs > output 1 file .sam
• input .sam file > output .bam file
• input .bam file > output .bcf file
• input .bcf file > output .vcf file

Answer 1

Well the syntax of the command can help you

bwa samse [-n max_occ] [-f out.sam] [-r RG_line] <prefix> <in.sai> <in.fq>

Now let's look at your bash:

for j in {004,005}; do
    for i in {001..004}; do
        bwa samse -f $j_$i.sam $hg19 $j_$i.sai $j_$i.fastq

        # Do the other conversions from $j_$i.sam
        samtools view -bS $j_$i.sam > $j_$i.bam
        samtools mpileup -uf genome.fa $j_$i.bam | bcftools view -bvcg - > $j_$i.bcf 
    done
done

I have no idea what $hg19 is.
It expects a "prefix" there so if you know what that means, substitute it in.

Anyway, that should generate 8× .sam files. You can either change the loop above so that after generating the .sam with bwa, you perform your other conversions, like so:

for j in {004,005}; do
    for i in {001..004}; do
        bwa samse -f $j_$i.sam $hg19 $j_$i.sai $j_$i.fastq

        # Do the other conversions from $j_$i.sam
        samtools view -bS $j_$i.sam > $j_$i.bam
        samtools mpileup -uf genome.fa $j_$i.bam | bcftools view -bvcg - > $j_$i.bcf
        # ... whatever is needed to convert bcf to vcf  ...
    done
done

Or you can do a separate loop(s): for f in *.sam; do command $f ...; done.

Answer 2

I wouldn't bother with the incrementing counter, you don't need it and it just complicates things. Try this instead:

for file in 0*.sai; do 
    bwa samse hg19.fa $file ${file%.sai}.fastq  > ${file%.sai}.sam
done

Explanation

• for file in 0*.sai; : iterate through all 0*.sai files in the current directory.
• ${file%.sai} : remove .sai from the end of $file. This means that if file=001_001.sai, ${file%.sai}.fastq will be 001_001.fastq.

This way, you will iterate through all files and run the conversion correctly without needing to fiddle with counters. I don't know what command you'd use to convert to the other formats since I've never worked with NGS data but you can add whatever is needed to the loop:

for file in 0*.sai; do 
    bwa samse hg19.fa $file ${file%.sai}.fastq  > ${file%.sai}.sam
    command2 $file ${file%.sai}.fastq  > ${file%.sai}.bwa, 
    command2 $file ${file%.sai}.fastq  > ${file%.sai}.bcf 
    command2 $file ${file%.sai}.fastq  > ${file%.sai}.vcf 
done

Answer 3

thanks for all the help guys. I have perfected my script to work on with multithreading. I appreciate all the help.