script to concatenate multiple text files found in zip archives nested in multiple folders

Question

I have 30 gig of text files and html files from various sources laying in a nested folder hierarchy. All these folders are in my home directory in a folder named zipFiles. The hierarchy is 20 levels deep-some branches deeper than others. I want to concatenate the text of ALL these files into one huge text file to use as an RDD in a Spark project. Obviously I would need to convert the html files into text by stripping out the html formatting codes as well.

My approach would be: iterate through the nested folder hierarchy (dont know how to do this in bash) unzip the file if it's html convert it to text add it to the HUGE result

The problems I see with this is running out of ram- if I do it all in memory by just adding the text to a variable in the code.

If i open the HUGE file from disk, write to it and close the file, then I am thrashing the disk and it also will go slowly.

Is bash scripting my only solution because it scares me. I guess I could do it in python but then I need to know how to unzip in python and do html to text conversion as well.

Answer 1

find is a good tool when you need to do something recursively in directories. zcat can

My approach would be something like this :

find /home/username/zipFiles -type f -name "*.zip" -exec zcat {} \; > outputfile.txt

Example:

$ df > testfile.txt                                            
$ zip testArchive testfile.txt
  adding: testfile.txt (deflated 54%)
$ find . -maxdepth 1 -name "*.zip" -exec zcat {} \;            
Filesystem     1K-blocks     Used Available Use% Mounted on
udev             1954212        4   1954208   1% /dev
tmpfs             393160     3392    389768   1% /run
/dev/sda1      115247656 82424300  26946008  76% /
none                   4        0         4   0% /sys/fs/cgroup
none                5120        0      5120   0% /run/lock
none             1965792       80   1965712   1% /run/shm
none              102400       32    102368   1% /run/user
cgmfs                100        0       100   0% /run/cgmanager/fs
/dev/sdb1       30343056       48  30343008   1% /media/xieerqi/SDCZ43-032G