Script for merging files in multiple nested directories by symbolic link

Basically I am looking for a script to automate stuff (See picture below) in Ubuntu. I'm thinking of using a bash script but other solutions (eg. python?) would also be excellent.

1) Suppose I have a number of real directories "Folder 1" and "Folder 2" with subfolders and files. Assuming that files in the corresponding folders 1 and 2 have unique names. How can I create a new merged folder where each of the files are symlinks to the original folders?

2) The script should also offer an option to prune broken symbolic links in the merged folder.

The reason I want to do this is because I'd like to improve how my stuff are organised. eg. "Folder 1|2" might be data obtained in different chronological time points. Then I'd create Merged_Folder1, Merged_Folder2 etc for different projects without duplicating large files.

Edit: This question differs from this post as I would like to merge corresponding nested subfolders with the same names. The question in the previous post simply links the top directories under the sources to the target and cannot merge nested subfolders. Notice that in my case none of the folders would be symlinks, only the files are symlinks.

Edit2: I should have clarified that I wish the code to merge arbitrary levels of nested subfolders, not just two levels. Hence I've added "File J" and "File I" in the example illustration.

• Possible duplicate of Make Folder a symbolic link to two folders? – Jacob Vlijm Nov 30 '18 at 19:45
• If I understand the question correctly, it is a dupe of the link. Please mention. – Jacob Vlijm Nov 30 '18 at 19:46
• rsync is a tool that could be configured to do this. rsync - fast, versatile, remote (and local) file-copying tool – waltinator Nov 30 '18 at 19:46
• @waltinator it is not about copying, but about creating what was/is? called a library on Mac or Windows (don't remember) in the past (or still?), which is no more than a reference to the original file(s). – Jacob Vlijm Nov 30 '18 at 19:48
• @JacobVlijm Thanks for showing me the link. The question might look similar, but in my case I would like to merge corresponding subfolders of the same names. The python solution in the above link cannot handle subfolders. Notice that in my case none of the folders would be symlinks, only the files are symlinks. However, given the link you sent, I would be capable of extending the script myself. Unless someone beat me to it, I'll post the solution here later. In addition, I would also add 1) Pruning of target subfolders that contain no symlinks 2) options to handle conflicting filenames. – matohak Nov 30 '18 at 21:04

I think the following shellscript will do what you want

• The original folders are in main
• There can be subfolders in several levels
• The merged folder is links
• run the main shellscript script in the directory containing main and links

script

#!/bin/bash

find main -type d -exec bash -c \
'for pathname do
#echo "------------------------------${pathname}${pathname#*/*/}"
if [ "${pathname/*\/*\/}" != "${pathname}" ]
then
mkdir -p "links/${pathname#*/*/}" fi done' bash {} + find main -type f -exec bash -c \ 'curdir=$(pwd)
for pathname do
tpat=${pathname/main\/} ln -s "${curdir}/${pathname}" "links/${tpat#*/}" 2> /dev/null;
done' bash {} +

find links -type l -exec bash -c \
'for pathname do
LANG=C
file "$pathname"|grep -o "$pathname: broken symbolic link" > /dev/null; \
if [ $? -eq 0 ];then rm "$pathname";fi
done' \
bash {} +


Demo

$\rm -r links$ find main
main
main/f 4
main/f 4/s 4
main/f 4/s 4/k 4
main/asdf
main/f2
main/f2/s3
main/f2/s3/h
main/f2/s3/g
main/f2/s3/ss
main/f2/s3/ss/i
main/f2/s1
main/f2/s1/c
main/f2/s1/d
main/j
main/f1
main/f1/s2
main/f1/s2/x y
main/f1/s2/f
main/f1/s2/e
main/f1/s1
main/f1/s1/a
main/f1/s1/b

$./script # doing it$ find links/ -type l -exec file {} \;
links/s2/x y: symbolic link to /media/multimed-2/test/test0/matohak/main/f1/s2/x y
links/s 4/k 4: symbolic link to /media/multimed-2/test/test0/matohak/main/f 4/s 4/k 4

$ln -s main/asdf links/asdf-b # create a broken link$ find links/ -type l -name "asdf*" -exec file {} \;

$./script # this time only to remove the broken link$ find links/ -type l -name "asdf*" -exec file {} \;
$ A case that would allow you to specify which folder under main/* to merge #!/bin/bash # First argument is target, following arbitrary number of target folders # eg. ./script.sh links main/f1 main/f2 main/f3 argc=$#
argv=($@) mkdir -p${argv[0]}

for (( j=1; j<argc; j++ )); do

find ${argv[j]} -type d -exec bash -c \ 'for pathname do #echo "------------------------------${pathname} ${pathname#*/*/}" if [ "${pathname/*\/*\/}" != "${pathname}" ] then mkdir -p "'${argv[0]}'/${pathname#*/*/}" fi done' bash {} + find${argv[j]} -type f -exec bash -c \
'curdir=$(pwd) for pathname do tpat=${pathname/${argv[j]}\/} ln -s "${curdir}/${pathname}" "'${argv[0]}'/${tpat#*/}" 2> /dev/null; done' bash {} + find${argv[0]} -type l -exec bash -c \
'for pathname do
LANG=C
file "$pathname"|grep -o "$pathname: broken symbolic link" > /dev/null; \
if [ $? -eq 0 ];then rm "$pathname";fi
done' \
bash {} +

find \${argv[0]} -type d -empty -delete  # Removes empty dir in target

done

• Thanks, this seems to work for files nested one level under "subfolder". I forgot to clarify that the drawing was just an example - I wish the script would work with files at arbitrary nested depths (ie. just added File J and File I in the example) – matohak Dec 1 '18 at 1:43
• @matohak, Too bad you did say so at once. I'm afraid, it means big modifications of the scripts. I don't know if and when I can have a solution for your more general directory structure. – sudodus Dec 1 '18 at 1:49
• @matohak, are the folder names arbitrary too? – sudodus Dec 1 '18 at 1:59
• Yes the filenames/foldernames should be arbitrary. The numbers of base folder (ie. folder1, folder2, folder3...) to be merged should also be arbitrary. – matohak Dec 1 '18 at 2:04
• @matohak, I think my new bash script works according to your new specs. I noticed that you made a python script. Maybe you can compare your script and mine with some corner cases. – sudodus Dec 1 '18 at 4:23

Got it... this python code should be able to walk through arbitrary number of nested directories and create symlinks for files all merged into the target directory. Arguments are the target and the source directories respectively. Source directories should be relative to the target dir. eg.

python script.py ./merged_folder ../folder1 ../folder2 ../folder3

import os
import sys
import time
'''
KH Tam Nov 2018 (matohak)

Use: python merge_symlink.py ./target ../folder1 ../folder2 ../folder3
Note that if overwrite==True and there are duplicated filenames, links will be overwritten by the last argument's
'''

def merge_symlink(sources, overwrite=True, remove_empty_dir=True, verbose=False):
'''
Function to be run in the target directory.

:param sources: a list of directories where the files in the subdirectories are to be merged symbolically. Path relative to the target. eg. ["../folder1", "../folder2"]
:param overwrite: Bool, whether to overwrite existing symbolic links
:param remove_empty_dir: Bool, whether to remove empty directories in target.
:param verbose: Prints stuff.
:return: None
'''

# Creating symlinks and folders
for source in sources:
for dirName, subdirList, fileList in os.walk(source):
# print(dirName, fileList)  # print all source dir and files
if source[-1] == "/": source=source[:-1]

target_dir = dirName.replace(source, '.', 1)
depth = dirName.count("/") - source.count("/")

try:
os.mkdir(os.path.join(target_dir))
except FileExistsError:
pass

for file in fileList:
targetlink = os.path.join(target_dir, file)
try:
except FileExistsError:
if overwrite and not (isvalidlink(targetlink)==2):  # Never replace a real file with a symlink!
os.symlink(os.path.join("../" * depth + dirName, file), targetlink)
if verbose: print('overwriting {}'.format(targetlink))

# Pruning broken links and then deleting empty folders.
for dirName, subdirList, fileList in os.walk("./"):
for file in fileList:

if remove_empty_dir:
for dirName, subdirList, fileList in os.walk("./"):
if fileList==[] and subdirList==[] and dirName!="./":
os.rmdir(dirName)

# Checks if file is a broken link. 0: broken link; 1: valid link; 2: not a link
return 2
try:
os.stat(path)
except os.error:
return 0
return 1

if __name__ == "__main__":

target = sys.argv[1]
sources = sys.argv[2:]      # Inputs should be relative to the target dir.
overwrite = False
loop = False
looptime = 10

os.chdir(target)
if not loop: