8

I have dat files' names in chronological order:

FileName_YYYY_MM_DD_HHMM.dat

Is there any commands for adding 30 minutes to each timestamp?

11
  • 1
    are those timestamps same as their creation date ? is it same as ls --full-time ? May 16, 2015 at 18:18
  • 1
    No, the timestamps are different as their created/modified date. The timestamps are based on the time that the data was measured. May 16, 2015 at 18:29
  • 1
    See, since those timestamps are custom, it's going to require a script, that has to calculate adding 30 minutes to the date, there's not gonna be any simple command. so it might take a little bit for people to respond May 16, 2015 at 18:35
  • 1
    are there any time stamps that are close to midnight , such that adding 30 minutes may result in need to change a day by one ? May 16, 2015 at 18:42
  • 1
    what format are hours ? 12 then 1,2,3 (12 hour format ) or 12 to 13 to 14, 15 . . . 23, 00 (24 hour format ) ? May 16, 2015 at 18:43

4 Answers 4

6

Using python :

#!/usr/bin/env python2
import glob, re, os, datetime
os.chdir('/path/to/dir')
for f in glob.glob('*.dat'):
    ini_time = datetime.datetime.strptime(re.search(r'(?<=_)(?:\d|_)+(?=.dat$)', f).group(), '%Y_%m_%d_%H%M')
    fin_time = (ini_time + datetime.timedelta(minutes=30)).strftime('%Y_%m_%d_%H%M%S')
    os.rename(f, 'Filename_' + str(fin_time) + '.dat')
  • os.chdir('/path/to/dir') will change the current directory to the directory containing the .dat files. Replace /path/to/dir with the actual path.

  • glob.glob('*.dat') will find the files ending in .dat

  • ini_time variable will at first cut out the date-time from the original file name using re module and then sort out which entry represents what in the string that is taken out so that we can add the required time to this

  • fin_time will contain the resultant time i.e. ini_time plus 30 minutes

  • os.rename will rename the file accordingly.

Also note that, with successive file names (differed by 30 minutes) the renamed file will overwrite the next one, hence it it is better to add the seconds to the renamed file name so that it remains safe. Otherwise you need to save the renamed files to a different directory and then replace them with the original ones later.

11
  • You and python, a great love. =) +1
    – A.B.
    May 16, 2015 at 20:09
  • You should not use the string Filename_ in the rename method.
    – A.B.
    May 16, 2015 at 20:28
  • @A.B. Why is that ? (+1 for you too)
    – heemayl
    May 16, 2015 at 20:50
  • Do you know why I have some files missing after running the python script? The timestamp for each file has 30-min difference. Below is the output for the first few filenames: Filename_2011_01_11_1630.dat Filename_2011_01_11_1830.dat Filename_2011_01_11_1900.dat Filename_2011_01_11_2030.dat Filename_2011_01_11_2100.dat May 16, 2015 at 21:11
  • @strawberrie I don't know..it have tested it and working well for me without any problem..have you run the script replacing the /path/to/file with the full path to the directory?
    – heemayl
    May 16, 2015 at 21:17
2

Using bash, the renamed files are in a new sub folder renamed.

Start the script in the folder where the files are located.

#!/bin/bash

mkdir -p renamed   

# loop over all dat files in the current folder
for f in *.dat; do

    # the filename without extension    
    filename="${f%%.*}"

    # your timestamp
    old_timestamp=$(echo $filename | grep -P "[0-9]{4}_[0-9]{2}_[0-9]{2}_[0-9]{4}$")

    if [ "$old_timestamp" == "" ]; then
        >&2 echo "not a valid filename: '$f', skipped."
    else
      # a valid date from the timestamp
      new_date=$(echo "$old_timestamp" | awk -F_ '{HM=NF; D=NF-1; M=NF-2; Y=NF-3; print $Y "-" $M "-" $D " " substr($HM,1,2) ":" substr($HM,3,2) ":00"}')

      # the new time stamp, 30 mins in the future
      changed_timestamp=$(date --date "$new_date 30 minutes" "+%Y_%m_%d_%H%M")

      # copy the file, ${f##*.} is the extension
      cp "$f" renamed/"${filename/$old_timestamp/$changed_timestamp.${f##*.}}"
    fi
done

example output:

% ls -og FileName*
-rw-rw-r-- 1 0 Mai 16 20:35 FileName_2015_05_16_2235.dat

% ./timestamp

% ls -og renamed/FileName*
-rw-rw-r-- 1 0 Mai 16 20:35 FileName_2015_05_16_2305.dat
5
  • @strawberrie the renamed files are now placed in a sub folder renamed
    – A.B.
    May 16, 2015 at 21:05
  • good idea to save renamed files into an extra folder. In case something goes awry, OP still has originals. Good thinking, hence +1 May 16, 2015 at 21:29
  • Thanks @A.B. I got the following error after running your script: TimeChange.sh: 21: TimeChange.sh: Bad substitution My actual filename or the fixed prefix before the timestamp is like FileName_123.Data_YYYY_MM_DD_HHMM.dat May 16, 2015 at 22:22
  • Per definition, for FileName_123.Data_YYYY_MM_DD_HHMM.datthe part .Data_YYYY_MM_DD_HHMM.dat is the extension. And therefore FileName_123 isn't a valid timestamp.
    – A.B.
    May 17, 2015 at 8:18
  • @strawberrie I have my script changed
    – A.B.
    May 17, 2015 at 8:39
1

SCRIPT

This is the edited version of my original script. OP originally didn't provide full information about the naming format. This script adapts to what OP mentioned in the comments was the correct file naming.

*Technical notes: *

In this script we separate filename into 6 separate fields using awk, with underscore as field delimiter. First two fields , $1 and $2 are considered static text string. Fields 3,4,5, and 6 are the timestamp at which OP's data was sampled, not the file's creation date on the filesystem.

Variable COPYDIR holds name of new directory where files with updated timestamp will go. We create that directory in current working directory with mkdir $COPYDIR

Variables TEXTSTRING and DATESTRING hold static text and timestamp respectivelly. In the sample output bellow I have used two different strings to prove that script will work regardless of what text the first two fields hold.

NEWEPOCHTIME is variable that holds calculated new timestamp in unix epoch format. NEWDATE is variable that holds converted timestamp from unix epoch to YYYY-MM-DD HH:MM format. NEWAPPEND is the actual timestamp that will be added to the file in OP's desired YYYY_MM_DD_HHMM format.

cp $file "$COPYDIR"/"%TEXTSTRING""$NEWAPPEND".dat copies the old file into "converted_files" directory ( instead of moving, to prevent data loss) with the updated datastamp.

Notice, the script will work as long as the naming format is really followed, i.e., all the files are really have SomeText_123.Data_YYYY_MM_DD_HHMM.dat format.

#!/usr/bin/env bash
#
# Author: Serg Kolo
# Description: this script takes timestamp within the filename
# (which may be different from file's actual creation date)
# converts that date and time to unix's epoch time
# adds 30 minutes to it and renames it

COPYDIR="converted_files"
mkdir $COPYDIR

for file in *.dat; do
        TEXTSTRING=$(stat -c %n $file | awk -F'_' '{print $1"_"$2"_"}' )
        DATESTRING=$( stat -c %n $file | awk -F'_' '{gsub(".dat","");  print $3"-"$4"-"$5" "$6}' )
        NEWEPOCHTIME=$( expr $( date --date="$DATESTRING" +%s ) + 1800 )
        NEWDATE=$(date --date=@"$NEWEPOCHTIME" +%F"_"%R)
        NEWAPPEND=$(echo $NEWDATE | awk '{gsub("-","_");gsub(":","");print}')
        cp $file "$COPYDIR"/"$TEXTSTRING""$NEWAPPEND".dat
done

SCRIPT IN ACTION

The demonstration bellow is direct copy from my terminal. Notice that I've created original files with two different strings in the first two fields. So this script should work no matter what is in the beginning of the filename, as long as there are really only two strings separated by underscore

The script was named notes-conversion because I developed the script from the notes I took while working on this question.

Notice that filenames which have HHMM part as 2345 (which is 15 minutes before midnight) get updated to 0015, and DD part is updated to next day. 24 hour format preserved.

In addition, because for loop only looks for .dat files, we avoid renaming other files or directories that may happen to be in the working directory, thus avoiding any potential data loss. In the example bellow, original directory holds 11 items, 3 of which are *.txt files for testing, so we only work with 8 .dat files. In the directory where updated files go, we see 8 files, all .dat and no other files. Data is safe, script does its job.

[68 ]SERGIY@UBUNTU_[/home/xieerqi/testdir/conversion/convert2]
***********************************************
85 $ ls
FileName_123.Dat_2015_05_31_1245.dat  Test.txt
FileName_123.Dat_2015_05_31_2345.dat  YoloSwag_123.Dat_2015_05_31_1245.dat
FileName_Foo.Bar_2015_05_31_1245.dat  YoloSwag_123.Dat_2015_05_31_2345.dat
FileName_Foo.Bar_2015_05_31_2345.dat  YoloSwag_Foo.Bar_2015_05_31_1245.dat
File.txt                              YoloSwag_Foo.Bar_2015_05_31_2345.dat
Random.txt

[68 ]SERGIY@UBUNTU_[/home/xieerqi/testdir/conversion/convert2]
***********************************************
86 $ ls | wc -l
11

[68 ]SERGIY@UBUNTU_[/home/xieerqi/testdir/conversion/convert2]
***********************************************
87 $ notes-conversion                                                                                

[68 ]SERGIY@UBUNTU_[/home/xieerqi/testdir/conversion/convert2]
***********************************************
88 $ ls converted_files/; ls converted_files/ | wc -l                                                
FileName_123.Dat_2015_05_31_1315.dat  YoloSwag_123.Dat_2015_05_31_1315.dat
FileName_123.Dat_2015_06_01_0015.dat  YoloSwag_123.Dat_2015_06_01_0015.dat
FileName_Foo.Bar_2015_05_31_1315.dat  YoloSwag_Foo.Bar_2015_05_31_1315.dat
FileName_Foo.Bar_2015_06_01_0015.dat  YoloSwag_Foo.Bar_2015_06_01_0015.dat
8

[67 ]SERGIY@UBUNTU_[/home/xieerqi/testdir/conversion/convert2]
***********************************************
89 $ 

EXPLANATION (from original post)

*) Today I've learned that Unix-Linux systems count time in Epoch time, or simply put seconds.

*) the script takes each file name, extracts date, converts it to epoch , adds 1800 seconds (which is exactly 30 minutes), and saves the file with than new timestamp.

*) This script addresses what OP wanted - change timestamp in filename, not update creation time of the file itself

Tools used:

  • ubuntu 15.04

  • GNU bash 4.3.30

  • GNU awk 4.1.1

  • date (GNU coreutils) 8.23

10
  • by the way, ls | wc -l before gives 36 files, and after script also 36 files, none missing May 16, 2015 at 21:12
  • Also, reason why i use stat -c %n file, is because parsing output of ls is not a good idea, generally. Some folks use find command, which is also good. May 16, 2015 at 21:22
  • Thanks @Serg. I don't know why I only left 2 out of 727 files left in the folder after running your script... May 16, 2015 at 22:25
  • Was there any difference between the actual file names and the example you posted ? May 16, 2015 at 22:33
  • also, did you run the first line in the "Script in action" ? the rm * was for cleaning up my own directory and then creating bunch of test files with different dates in file names. You were not supposed to run that May 16, 2015 at 22:35
-1

You may use this code to do what you need assuming

  1. you should take backup and test the code first to see if it's suitable for your case
  2. you are using 24H format
  3. no files will be named after 23:29 (if you have files after that time the code should be modified to change the date also )

the code is :

cd /path/to/the/files

for i in `ls`; do MM=${i:(-6): -4}; HH=${i: -8 : -6 }; NAME=${i: 0 : -8 } ; if [ "$MM" -lt 30 ] ; then  NEWMM=$((10#$MM+30)); mv -f $i $NAME$HH$NEWMM.dat ; else NEWHH=$((10#$HH+1));NEWMM=$((10#$MM-30)) ; mv -f $i $NAME$NEWHH$NEWMM.dat ; fi ; done ;

How it works: The code will check the minutes part in the file name MM then if it's less than 30 it will add 30 to the MM if it equals 30 or more it will add 1 hour to the HH part in the name and deduct 30 minutes from the MM part of the name

1
  • for down voter please notify the reason ?
    – Fat Mind
    May 17, 2015 at 7:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.