2

Lets say I have 1000 csv files that have random names in a folder test. and I have a python script script.py. I would like to write a bash script that would pass the name of each file in the folder to script.py then run it, therefore the bash script should run script.py 1000 times, each time with a different file name, example, first run:

with open('records.csv','r') as in_file

next run

with open('vertigo.csv','r') as in_file

I know I could do it in python without the hassle of writing a bash script like so

import glob
for filename in glob.glob('*.csv'):
   # script here

But I'm doing some tests therefore I'd like to do it in bash

6

Use sys.argv[1] so that the filename is taken from the arguments:

import sys

with open(sys.argv[1],'r') as in_file

Then, you can use various methods to pass the filename as an argument. For example, with find:

find test/ -type f -name '*.csv' -exec /path/to/script.py {} \;
3

Alternative to using sys.argv is using argparse. Very helpful if you need more command line parameter parsing.

#!/usr/bin/env python2

import argparse

def main(files):
    for f in files:
        print "Would do something with", f

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument('file', nargs='+', help='path to the file')
    args_namespace = parser.parse_args()
    args = vars(args_namespace)['file']
    main(args)

Running it with having Bash expanding our file argument list:

$ ./script.py *.csv
Would do something with file1.csv
Would do something with file2.csv
Would do something with file3.csv

And provides built-in help 'for free':

$ ./script.py --help
usage: script.py [-h] file [file ...]

positional arguments:
  file        path to the file

optional arguments:
  -h, --help  show this help message and exit
2

Extending muru's answer a bit:

Why not do it all inside your python script? Transform the script's actions on the files into a function, then run the script with the directory as an argument:

#!/usr/bin/env python3
import os

#--- your original script, transformed into a function
def some_function(file):
    print(file)
#---

# The directory with your .csv files
dr = sys.argv[1]

for f in os.listdir(dr):
    file = dr+"/"+f
    some_function(file)

Or a bit shorter:

for f in os.listdir(dr):
    some_function(dr+"/"+f)

Or, if the directory also might contain other files:

for f in [f for f in os.listdir(dr) if f.endswith(".csv")]:
    some_function(dr+"/"+f)

Then simply run it with the directory as argument:

python3 <script> <directory>
2

Use the sys.argv as list, providing all individual files as arguments and have your shell (e.g. Bash) expand it for you.

script.py looks like:

#!/usr/bin/env python2

import sys

def main(files):
    for f in files:
        print "Would do something with", f

if __name__ == "__main__":
    files = sys.argv[1:]: # slices off the first argument (executable itself)
    main(files)

File tree like this:

.
├── file1.csv
├── file2.csv
├── file3.csv
└── script.py

0 directories, 4 files

Then make it executable:

chmod +x script.py

Run it with the arguments as expanded by the shell:

./script.py *.csv

Outputs:

Would do something with file1.csv
Would do something with file2.csv
Would do something with file3.csv
1

You'll have to retrieve the filename within the python script for this to work:

#!/bin/bash
for f in *.csv; do
    python script.py "$f"
done

In a one-liner:

for f in *.csv; do python script.py "$f"; done

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.