2

Basically I am looking for a script to automate stuff (See picture below) in Ubuntu. I'm thinking of using a bash script but other solutions (eg. python?) would also be excellent.

1) Suppose I have a number of real directories "Folder 1" and "Folder 2" with subfolders and files. Assuming that files in the corresponding folders 1 and 2 have unique names. How can I create a new merged folder where each of the files are symlinks to the original folders?

2) The script should also offer an option to prune broken symbolic links in the merged folder.


The reason I want to do this is because I'd like to improve how my stuff are organised. eg. "Folder 1|2" might be data obtained in different chronological time points. Then I'd create Merged_Folder1, Merged_Folder2 etc for different projects without duplicating large files.


Edit: This question differs from this post as I would like to merge corresponding nested subfolders with the same names. The question in the previous post simply links the top directories under the sources to the target and cannot merge nested subfolders. Notice that in my case none of the folders would be symlinks, only the files are symlinks.

Edit2: I should have clarified that I wish the code to merge arbitrary levels of nested subfolders, not just two levels. Hence I've added "File J" and "File I" in the example illustration.

Example of how I want the folder merge to work

  • 4
    Possible duplicate of Make Folder a symbolic link to two folders? – Jacob Vlijm Nov 30 '18 at 19:45
  • If I understand the question correctly, it is a dupe of the link. Please mention. – Jacob Vlijm Nov 30 '18 at 19:46
  • rsync is a tool that could be configured to do this. rsync - fast, versatile, remote (and local) file-copying tool – waltinator Nov 30 '18 at 19:46
  • @waltinator it is not about copying, but about creating what was/is? called a library on Mac or Windows (don't remember) in the past (or still?), which is no more than a reference to the original file(s). – Jacob Vlijm Nov 30 '18 at 19:48
  • 1
    @JacobVlijm Thanks for showing me the link. The question might look similar, but in my case I would like to merge corresponding subfolders of the same names. The python solution in the above link cannot handle subfolders. Notice that in my case none of the folders would be symlinks, only the files are symlinks. However, given the link you sent, I would be capable of extending the script myself. Unless someone beat me to it, I'll post the solution here later. In addition, I would also add 1) Pruning of target subfolders that contain no symlinks 2) options to handle conflicting filenames. – matohak Nov 30 '18 at 21:04
1

I think the following shellscript will do what you want

  • The original folders are in main
  • There can be subfolders in several levels
  • The merged folder is links
  • run the main shellscript script in the directory containing main and links

script

#!/bin/bash

mkdir -p links

find main -type d -exec bash -c \
'for pathname do
  #echo "------------------------------${pathname} ${pathname#*/*/}"
  if [ "${pathname/*\/*\/}" != "${pathname}" ]
  then
   mkdir -p "links/${pathname#*/*/}"
  fi
 done' bash {} +

find main -type f -exec bash -c \
'curdir=$(pwd)
for pathname do
  tpat=${pathname/main\/}
  ln -s "${curdir}/${pathname}" "links/${tpat#*/}" 2> /dev/null;
 done' bash {} +

find links -type l -exec bash -c \
'for pathname do
  LANG=C
  file "$pathname"|grep -o  "$pathname: broken symbolic link" > /dev/null; \
  if [ $? -eq 0 ];then rm "$pathname";fi
 done' \
 bash {} +

Demo

$ \rm -r links

$ find main
main
main/f 4
main/f 4/s 4
main/f 4/s 4/k 4
main/asdf
main/f2
main/f2/s3
main/f2/s3/h
main/f2/s3/g
main/f2/s3/ss
main/f2/s3/ss/i
main/f2/s1
main/f2/s1/c
main/f2/s1/d
main/j
main/f1
main/f1/s2
main/f1/s2/x y
main/f1/s2/f
main/f1/s2/e
main/f1/s1
main/f1/s1/a
main/f1/s1/b

$ ./script  # doing it

$ find links/ -type l -exec file {} \;
links/s2/x y: symbolic link to /media/multimed-2/test/test0/matohak/main/f1/s2/x y
links/s2/f: symbolic link to /media/multimed-2/test/test0/matohak/main/f1/s2/f
links/s2/e: symbolic link to /media/multimed-2/test/test0/matohak/main/f1/s2/e
links/s3/h: symbolic link to /media/multimed-2/test/test0/matohak/main/f2/s3/h
links/s3/g: symbolic link to /media/multimed-2/test/test0/matohak/main/f2/s3/g
links/s3/ss/i: symbolic link to /media/multimed-2/test/test0/matohak/main/f2/s3/ss/i
links/s 4/k 4: symbolic link to /media/multimed-2/test/test0/matohak/main/f 4/s 4/k 4
links/asdf: symbolic link to /media/multimed-2/test/test0/matohak/main/asdf
links/s1/a: symbolic link to /media/multimed-2/test/test0/matohak/main/f1/s1/a
links/s1/c: symbolic link to /media/multimed-2/test/test0/matohak/main/f2/s1/c
links/s1/b: symbolic link to /media/multimed-2/test/test0/matohak/main/f1/s1/b
links/s1/d: symbolic link to /media/multimed-2/test/test0/matohak/main/f2/s1/d
links/j: symbolic link to /media/multimed-2/test/test0/matohak/main/j

$ ln -s main/asdf links/asdf-b  # create a broken link

$ find links/ -type l -name "asdf*" -exec file {} \;
links/asdf-b: broken symbolic link to main/asdf
links/asdf: symbolic link to /media/multimed-2/test/test0/matohak/main/asdf

$ ./script  # this time only to remove the broken link

$ find links/ -type l -name "asdf*" -exec file {} \;
links/asdf: symbolic link to /media/multimed-2/test/test0/matohak/main/asdf
$

A case that would allow you to specify which folder under main/* to merge

#!/bin/bash
# First argument is target, following arbitrary number of target folders
# eg. ./script.sh links main/f1 main/f2 main/f3

argc=$#
argv=($@)

mkdir -p ${argv[0]}

for (( j=1; j<argc; j++ )); do

    find ${argv[j]} -type d -exec bash -c \
    'for pathname do
      #echo "------------------------------${pathname} ${pathname#*/*/}"
      if [ "${pathname/*\/*\/}" != "${pathname}" ]
      then
       mkdir -p "'${argv[0]}'/${pathname#*/*/}"
      fi
     done' bash {} +

    find ${argv[j]} -type f -exec bash -c \
    'curdir=$(pwd)
    for pathname do
      tpat=${pathname/${argv[j]}\/}
      ln -s "${curdir}/${pathname}" "'${argv[0]}'/${tpat#*/}" 2> /dev/null;
     done' bash {} +

    find ${argv[0]} -type l -exec bash -c \
    'for pathname do
      LANG=C
      file "$pathname"|grep -o  "$pathname: broken symbolic link" > /dev/null; \
      if [ $? -eq 0 ];then rm "$pathname";fi
     done' \
     bash {} +

    find ${argv[0]} -type d -empty -delete  # Removes empty dir in target

done
  • Thanks, this seems to work for files nested one level under "subfolder". I forgot to clarify that the drawing was just an example - I wish the script would work with files at arbitrary nested depths (ie. just added File J and File I in the example) – matohak Dec 1 '18 at 1:43
  • @matohak, Too bad you did say so at once. I'm afraid, it means big modifications of the scripts. I don't know if and when I can have a solution for your more general directory structure. – sudodus Dec 1 '18 at 1:49
  • @matohak, are the folder names arbitrary too? – sudodus Dec 1 '18 at 1:59
  • Yes the filenames/foldernames should be arbitrary. The numbers of base folder (ie. folder1, folder2, folder3...) to be merged should also be arbitrary. – matohak Dec 1 '18 at 2:04
  • @matohak, I think my new bash script works according to your new specs. I noticed that you made a python script. Maybe you can compare your script and mine with some corner cases. – sudodus Dec 1 '18 at 4:23
1

Got it... this python code should be able to walk through arbitrary number of nested directories and create symlinks for files all merged into the target directory. Arguments are the target and the source directories respectively. Source directories should be relative to the target dir. eg.

python script.py ./merged_folder ../folder1 ../folder2 ../folder3
import os
import sys
import time
'''
Loops through merge_symlink.
See https://askubuntu.com/questions/1097502/script-for-merging-files-in-multiple-nested-directories-by-symbolic-link/
KH Tam Nov 2018 (matohak)

Use: python merge_symlink.py ./target ../folder1 ../folder2 ../folder3
Note that if overwrite==True and there are duplicated filenames, links will be overwritten by the last argument's
'''

def merge_symlink(sources, overwrite=True, remove_empty_dir=True, verbose=False):
    '''
    See https://askubuntu.com/questions/1097502/script-for-merging-files-in-multiple-nested-directories-by-symbolic-link/
    Function to be run in the target directory.

    :param sources: a list of directories where the files in the subdirectories are to be merged symbolically. Path relative to the target. eg. ["../folder1", "../folder2"]
    :param overwrite: Bool, whether to overwrite existing symbolic links
    :param remove_empty_dir: Bool, whether to remove empty directories in target.
    :param verbose: Prints stuff.
    :return: None
    '''

    # Creating symlinks and folders
    for source in sources:
        for dirName, subdirList, fileList in os.walk(source):
            # print(dirName, fileList)  # print all source dir and files
            if source[-1] == "/": source=source[:-1]

            target_dir = dirName.replace(source, '.', 1)
            depth = dirName.count("/") - source.count("/")

            try:
                os.mkdir(os.path.join(target_dir))
            except FileExistsError:
                pass

            for file in fileList:
                targetlink = os.path.join(target_dir, file)
                try:
                    os.symlink(os.path.join("../"*depth + dirName, file), targetlink)
                except FileExistsError:
                    if overwrite and not (isvalidlink(targetlink)==2):  # Never replace a real file with a symlink!
                        os.remove(targetlink)
                        os.symlink(os.path.join("../" * depth + dirName, file), targetlink)
                        if verbose: print('overwriting {}'.format(targetlink))

    # Pruning broken links and then deleting empty folders.
    for dirName, subdirList, fileList in os.walk("./"):
        for file in fileList:
            link = os.path.join(dirName,file)
            if isvalidlink(link)==0:
                os.remove(link)
                if verbose: print("Removing broken symlink: {}".format(link))

    if remove_empty_dir:
        for dirName, subdirList, fileList in os.walk("./"):
            if fileList==[] and subdirList==[] and dirName!="./":
                os.rmdir(dirName)

# Checks if file is a broken link. 0: broken link; 1: valid link; 2: not a link
def isvalidlink(path):
    if not os.path.islink(path):
        return 2
    try:
        os.stat(path)
    except os.error:
        return 0
    return 1

if __name__ == "__main__":

    target = sys.argv[1]
    sources = sys.argv[2:]      # Inputs should be relative to the target dir.
    overwrite = False
    loop = False
    looptime = 10

    os.chdir(target)
    if not loop:
        merge_symlink(sources, overwrite=overwrite)
    else:
        while loop:
            merge_symlink(sources, overwrite=overwrite)
            time.sleep(looptime)

Thank @JacobVlijm for the link and @sudodus for helping!

  • +1; Thanks for sharing your solution. – sudodus Dec 1 '18 at 4:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.