How to rename many files using info from a separate .txt file?

Question

I believe this problem can be solved with a bash script, but I'm still a beginner so I need your help here.

I have many files (about 8 million!) with names such as IR.AZR..SHE.D.2016.001.000000.SAC. The file names follow this format:

IR.(stations name)..(component).D.(year).(day).000000.sac

They all need to be renamed to the following format:

(station name).IR.(component).(year).(day).(begin time).(endtime).sac

Note: (day) is in 365 format or maybe I should say it is in Julian day format.

The current file name, along with the additional information required for the new format, is stored in a plain text file:

In addition to the screenshot, here is a part from the file:

IR.AZR..SHE.D.2016.158.071819.SAC  2016 158  7 18 19 300        0.0000        8.2000
IR.AZR..SHE.D.2016.158.072153.SAC  2016 158  7 21 53 540        0.0000        8.2000
IR.AZR..SHE.D.2016.158.072251.SAC  2016 158  7 22 51  60        0.0000        8.1000
IR.AZR..SHE.D.2016.158.072315.SAC  2016 158  7 23 15 580        0.0000       16.3000
IR.AZR..SHE.D.2016.158.072340.SAC  2016 158  7 23 40 180        0.0000        8.2000
IR.AZR..SHE.D.2016.158.072421.SAC  2016 158  7 24 21 300        0.0000        8.1000
IR.AZR..SHE.D.2016.158.072445.SAC  2016 158  7 24 45 980        0.0000        8.2000
IR.AZR..SHE.D.2016.158.072518.SAC  2016 158  7 25 18  60        0.0000        7.3000
IR.AZR..SHE.D.2016.158.072541.SAC  2016 158  7 25 41 620        0.0000       32.9000
IR.AZR..SHE.D.2016.158.072647.SAC  2016 158  7 26 47 540        0.0000       16.4000
IR.AZR..SHE.D.2016.158.072712.SAC  2016 158  7 27 12 260        0.0000        8.1000
IR.AZR..SHE.D.2016.158.072736.SAC  2016 158  7 27 36 860        0.0000        8.2000
IR.AZR..SHE.D.2016.158.072753.SAC  2016 158  7 27 53 340        0.0000        8.2000
IR.AZR..SHE.D.2016.158.072809.SAC  2016 158  7 28  9 820        0.0000        8.1000
IR.AZR..SHE.D.2016.158.072904.SAC  2016 158  7 29  4 740        0.0000        8.2000
IR.AZR..SHE.D.2016.158.072921.SAC  2016 158  7 29 21 220        0.0000        8.2000
IR.AZR..SHE.D.2016.158.072954.SAC  2016 158  7 29 54  60        0.0000        8.2000
IR.AZR..SHE.D.2016.158.073035.SAC  2016 158  7 30 35 260        0.0000        8.1000
IR.AZR..SHE.D.2016.158.073059.SAC  2016 158  7 30 59 940        0.0000        8.2000

• The first column is a list of my files, with their current filenames.
• The second is column represents the (year).
• The third is (day) in Julian Day format.
• The eighth and ninth column are (begin time) and (end time), respectively.

In plain words the script should do the following:

First list all files and then find the file inside this .txt file and the corresponding row and column and rename as I said above.

Answer 1

What you ask can be easily accomplished with a Perl one-liner:

perl -lane '@a=split/\./,$F[0]; rename "old/$F[0]","new/$a[1].IR.$a[3].$F[1].$F[2].$F[7].$F[8].sac"' input.txt

However, your new format naming scheme does not seem appropriate for your input files, since it will create duplicate names. E.g.:

IR.AZR..SHE.D.2016.158.071819.SAC  2016 158  7 18 19 300        0.0000        8.2000
IR.AZR..SHE.D.2016.158.072153.SAC  2016 158  7 21 53 540        0.0000        8.2000

will both translate to: AZR.IR.SHE.2016.158.0.0000.8.2000.sac. Obviously, there can be only one file with a given filename at any one time, and one of them will be lost.

Please account for duplicates, rethinking the output filename format. One possible alternative could be:

perl -lane '@a=split/\./,$F[0]; rename "old/$F[0]","new/$a[1].IR.$a[3].$F[1].$F[2].$a[7].$F[7].$F[8].sac"' input.txt

---

Test run

Initial file structure:

old:
IR.AZR..SHE.D.2016.158.071819.SAC
IR.AZR..SHE.D.2016.158.072153.SAC
IR.AZR..SHE.D.2016.158.072251.SAC
IR.AZR..SHE.D.2016.158.072315.SAC
IR.AZR..SHE.D.2016.158.072340.SAC
IR.AZR..SHE.D.2016.158.072421.SAC
IR.AZR..SHE.D.2016.158.072445.SAC
IR.AZR..SHE.D.2016.158.072518.SAC
IR.AZR..SHE.D.2016.158.072541.SAC
IR.AZR..SHE.D.2016.158.072647.SAC
IR.AZR..SHE.D.2016.158.072712.SAC
IR.AZR..SHE.D.2016.158.072736.SAC
IR.AZR..SHE.D.2016.158.072753.SAC
IR.AZR..SHE.D.2016.158.072809.SAC
IR.AZR..SHE.D.2016.158.072904.SAC
IR.AZR..SHE.D.2016.158.072921.SAC
IR.AZR..SHE.D.2016.158.072954.SAC
IR.AZR..SHE.D.2016.158.073035.SAC
IR.AZR..SHE.D.2016.158.073059.SAC

new:

Test run with the OP's original format:

old:

new:
AZR.IR.SHE.2016.158.0.0000.16.3000.sac
AZR.IR.SHE.2016.158.0.0000.16.4000.sac
AZR.IR.SHE.2016.158.0.0000.32.9000.sac
AZR.IR.SHE.2016.158.0.0000.7.3000.sac
AZR.IR.SHE.2016.158.0.0000.8.1000.sac
AZR.IR.SHE.2016.158.0.0000.8.2000.sac

Test run with the altered output format:

old:

new:
AZR.IR.SHE.2016.158.071819.0.0000.8.2000.sac
AZR.IR.SHE.2016.158.072153.0.0000.8.2000.sac
AZR.IR.SHE.2016.158.072251.0.0000.8.1000.sac
AZR.IR.SHE.2016.158.072315.0.0000.16.3000.sac
AZR.IR.SHE.2016.158.072340.0.0000.8.2000.sac
AZR.IR.SHE.2016.158.072421.0.0000.8.1000.sac
AZR.IR.SHE.2016.158.072445.0.0000.8.2000.sac
AZR.IR.SHE.2016.158.072518.0.0000.7.3000.sac
AZR.IR.SHE.2016.158.072541.0.0000.32.9000.sac
AZR.IR.SHE.2016.158.072647.0.0000.16.4000.sac
AZR.IR.SHE.2016.158.072712.0.0000.8.1000.sac
AZR.IR.SHE.2016.158.072736.0.0000.8.2000.sac
AZR.IR.SHE.2016.158.072753.0.0000.8.2000.sac
AZR.IR.SHE.2016.158.072809.0.0000.8.1000.sac
AZR.IR.SHE.2016.158.072904.0.0000.8.2000.sac
AZR.IR.SHE.2016.158.072921.0.0000.8.2000.sac
AZR.IR.SHE.2016.158.072954.0.0000.8.2000.sac
AZR.IR.SHE.2016.158.073035.0.0000.8.1000.sac
AZR.IR.SHE.2016.158.073059.0.0000.8.2000.sac

Answer 2

Pure Bash + mv(1)

while read -r current year day d e f g begin end; do
  station="${current:3:3}" component="${current:8:3}"
  mv -T -- "$current" "${station}.IR.${component}.${year}.${day}.${begin}.${end}.sac"
done < file-name-data.txt

Although with 8 million files it will be quite slow to execute mv for each of them. Hence why I recommend the alternative below.

Python 3

One-liner (no error handling)

python3 -c 'import sys, os, re; for m in map(re.compile(sys.argv[1]).match, sys.stdin): os.rename(m.group("current"), sys.argv[2].format_map(m.groupdict()))' '(?P<current>IR\.(?P<station>\S+?)\.\.(?P<component>\S+?)\.\S*)\s+(?P<year>\S+)\s+(?P<day>\S+)\s+(?:\S+\s+){4}(?P<begin>\S+)\s+(?P<end>\S+)' '{station}.IR.{component}.{year}.{day}.{begin}.{end}.sac' < file-name-data.txt

Full program (with error handling)

#!/usr/bin/python3
import sys, os, re

src_pattern = re.compile(
  r'(?P<current>IR\.(?P<station>\S+?)\.\.(?P<component>\S+?)\.\S*)\s+'
  r'(?P<year>\S+)\s+(?P<day>\S+)\s+'
  r'(?:\S+\s+){4}'
  r'(?P<begin>\S+)\s+(?P<end>\S+)'
)
dst_format = '{station}.IR.{component}.{year}.{day}.{begin}.{end}.sac'

for i, line in enumerate(sys.stdin, 1):
  m = src_pattern.match(line)
  if m:
    try:
      os.rename(m.group('current'), dst_format.format_map(m.groupdict()))
    except OSError as ex:
      print(ex, file=sys.stderr)
  else:
    print(
      'Non-matching source line {:d}: {!r}'.format(i, line.rstrip('\r\n')),
      file=sys.stderr)

Usage:

python3 rename.py < file-name-data.txt

Answer 3

There is a command line tool, multiple move. It lets you move (which is also rename), copy, etc., based on patterns. Install it with

sudo apt install mmv

in a terminal window. Then run "man mmv" to see the manual page, or look here: https://www.systutorials.com/docs/linux/man/1-mmv/

It can be a bit daunting because the problem is not simple. Always test on a copy of your data first.

There is also a graphical tool, filebot. It is written in Java and runs on Ubuntu. Simply search for it in the Ubuntu Software store or install with

sudo apt install filebot

Its main target is bulk renaming of video and music files and it can use database information from MusicBrainz and so on, but if I remember correctly (but check it out yourself please because I didn't sleep much this weekend :) ) also works well for non-media file sets. Their website is here: https://www.filebot.net/