4

I'm a newbie so please help:

I keep a journal on my iPhone using Scratch that outputs all notes I make to a separate .txt file stored in Dropbox.

I have this synced with my Ubuntu 14.04 system, so in my Files I have a folder with all the text files stored in it here:

/home/stuart/Dropbox/Scratch

I want to run a command that concatenates all these files into one file, with the following provisos:

  1. Ordered by date created (earliest file first)
  2. Print the date of the file on a separate line before the contents of the file
  3. Include a blank line followed by some kind of separator line after each file

So the output file has entries that look something like this:

12-01-2014 11:01 AM:
A coffee shop in Israel. The sign outside reads:
"Coffee" - 9 shekel
"Coffee please" - 8 shekel
"Good morning, could I have a coffee please?" - 7 shekel

--
25-01-2014 11:01 AM:
You cannot outperform your ego - ole Gunnar solskjaer

--

And so on. I used to use a different app that did this kind of auto append but I have no idea how to replicate it.

I've looked over a lot of the help files on here but I've not found any that can help with the output I have in mind.

Any help greatly appreciated!


MORE INFO

I tried creating the script suggested below and followed the steps. However I get this response:

stuart@StudioClough:/home$ chmod +x $HOME/my_concat

stuart@StudioClough:/home$ ./my_concat /home/stuart/Dropbox/Scratch > new_concatenated_file

bash: new_concatenated_file: Permission denied

Do I have to somehow run it as sudo?

  • 1
    You can't get file creation times easily, unless the filename itself contains that date. – muru Sep 29 '14 at 2:46
  • @muru -- more than that, *nix systems typically don't store creation time at all, so it's actually impossible to retrieve such information. Stuart: have these files been modified at all since they were created? If not, you could sort them by mtime. – evilsoup Sep 29 '14 at 7:50
  • @evilsoup it's not impossible, depending on the filesystem. OP: Given that it's being synced by Dropbox, you are at the mercy of Dropbox to sync a bunch of files in the right order. – muru Sep 29 '14 at 7:53
  • It turns out that that @muru is right, and modification date nor creation date is saved when files are downloaded / synchronized via Dropbox. Both dates will depend on when the files are synced on your computer (I tried). After taking notes while you are not connected, modification dates / order presumably will be incorrect. I removed my answer. – Jacob Vlijm Sep 29 '14 at 15:46
  • You usually don't have write permissions on the /home directory, which is different from $HOME. Do your work in $HOME: cd $HOME; chmod +x $HOME/my_concat; ./my_concat /home/stuart/Dropbox/Scratch > new_concatenated_file. – muru Sep 29 '14 at 18:13
3

It can be done with a python script, with one sidenote: I took the modification date instead of the creation date, since the creation date will almost certainly not match the real creation date: it is the date the file was copied to the computer, while the modification date seems unchanged during copying (see discussion at @cOrps answer). You will have to see if it works in your situation.

If that is acceptable for you, you can use the script below to create a combined file with your notes. It reads the notes, sorts them and appends them to a text file (creates it if it doesn't exist).

The good news is that you can append your new notes to the same file without overwriting the old ones.

Example output:

Mon Sep 29 08:48:31 2014
This is my first note.
As you can read, I am not really awake yet.

----------
Mon Sep 29 09:04:06 2014
It is really time I am going to eat something.
I am a bit hungry.
Making it a bit longer.

----------

How to use:

  • Copy the script below into an empty file, save it as add_notes.py
  • change the directories for files_dir (where your notes are) and the file in which you want to save the notes: combined_file (the script creates the file if it doesn't exist)
  • run the script in a terminal window by typing the command:

    python3 /path/to/add_notes.py
    

The script:

#!/usr/bin/env python3

import os
import time
import subprocess

# --------------------------------------------------------
files_dir = "/path/to/your/textfiles"
combined_file = "/path/to/your/combined/file.txt"
# ---------------------------------------------------------
notes = []

if not os.path.exists(combined_file):
    subprocess.Popen(["touch", combined_file])

def read_file(file):
    with open(file) as note:
        return note.read()

def append_file(combined_file, text):
    with open(combined_file, "a") as notes:
        notes.write(text)

for root, dirs, files in os.walk(files_dir):
    for name in files:
        subject = root+"/"+name
        cr_date_text = time.ctime(os.path.getmtime(subject))
        cr_date_n = os.stat(subject).st_mtime
        notes.append((cr_date_n, cr_date_text, subject))

notes.sort(key=lambda x: x[0])

for note in notes:
    text = note[1]+"\n"+read_file(note[2])+"\n"+"-"*10+"\n"
    append_file(combined_file, text)
  • Hi Jacob - that's sorted it completely. Exactly what I was after. So every month from now on I can just run that Python file and it will add only the un-appended notes (basically those since the last time the script was run) to the output file? – Stuart Clough Sep 29 '14 at 19:34
  • @StuartClough exactly! (but don't forget to remove the old ones from the directory or they will be appended once more :)) – Jacob Vlijm Sep 29 '14 at 19:36
2

Here is a bash solution. It should work if you are using the ext4 file system. It is using file creation date that ext4 store in crtime field.

Create this script anywhere. Let's say my_concat in your $HOME directory (in your case it is /home/stuart):

#!/bin/bash

get_crtime() {
    for target in "${@}"; do
        inode=$(ls -di "${target}" | cut -d ' ' -f 1)
        fs=$(df  --output=source "${target}"  | tail -1)
        crtime=$(sudo debugfs -R 'stat <'"${inode}"'>' "${fs}" 2>/dev/null | 
        grep -oP 'crtime.*--\s*\K.*')
        printf "%s\n" "${crtime}"
    done
}

get_epoch_crtime(){
    date --date "$(get_crtime $1)" +%s
}

get_epoch_mtime() {
    stat -c %Y $1
}

# takes two date as input, returns earlier date
get_earlier_time(){
    if [[ "$1" -lt "$2" ]]; then
        echo $(date -d @$1 +%m/%d/%Y:%H:%M:%S)
    else
        echo $(date -d @$2 +%m/%d/%Y:%H:%M:%S)
    fi
}

if [ $# != 1 ]; then
    echo "Required only one argument - full path to folder"
    echo "Usage example:"
    echo "$0 /var/log/syslog/"
    exit 1
fi

if [ -d "$1" ]; then
    cd $1
    for file in *
    do 
        echo $(get_earlier_time $(get_epoch_crtime $file) $(get_epoch_mtime $file))
        cat $file
        echo -e "\n-------"
    done
else
    echo "The folder specified is not exists ($1). Please enter full path"
fi

Make it executable:

chmod +x $HOME/my_concat

Now go to your $HOME folder and run script. Script will ask you your password because script uses sudo:

./my_concat /home/stuart/Dropbox/Scratch > new_concatenated_file

Now read new_concatenated_file using some editor:

gedit new_concatenated_file

This script uses both creation date and modify date, after comparison it takes the earliest one.

Sources

  1. About creation date
  2. Creation date in other files systems
  3. Script to find creation date
  • Can you confirm that it works if you copy the files into a new directory? According to this unix.stackexchange.com/a/91200, it doesn't, even if the file system supports it. – Jacob Vlijm Sep 29 '14 at 9:16
  • @JacobVlijm here is simple test. it works, diff shows that creation time is different after copying files – c0rp Sep 29 '14 at 9:31
  • @cOrp But that's exactly the problem, it shouldn't. The files are copied from the Dropbox server to his computer, so the creation date is incorrect. Better take the mod. date then wich is stored in the file, then the creation date which is stored in the file system and thus not copied. In most cases, when taking notes, the mod. date will be close enough to the (real) creation date to use it in a script like OP is looking for. – Jacob Vlijm Sep 29 '14 at 10:48
  • @JacobVlijm you right, in this case modify date is more accurate – c0rp Sep 29 '14 at 12:19
  • I must admit, I didn't know until this question :) – Jacob Vlijm Sep 29 '14 at 12:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.