2

I have 30 gig of text files and html files from various sources laying in a nested folder hierarchy. All these folders are in my home directory in a folder named zipFiles. The hierarchy is 20 levels deep-some branches deeper than others. I want to concatenate the text of ALL these files into one huge text file to use as an RDD in a Spark project. Obviously I would need to convert the html files into text by stripping out the html formatting codes as well.

My approach would be: iterate through the nested folder hierarchy (dont know how to do this in bash) unzip the file if it's html convert it to text add it to the HUGE result

The problems I see with this is running out of ram- if I do it all in memory by just adding the text to a variable in the code.

If i open the HUGE file from disk, write to it and close the file, then I am thrashing the disk and it also will go slowly.

Is bash scripting my only solution because it scares me. I guess I could do it in python but then I need to know how to unzip in python and do html to text conversion as well.

  • 2
    So is there one huge archive with many files in it, or are there multiple archives with few files in it? Or are there many archives with one file in each? What archive type do you use (tar.gz, zip, ...)? Are there only text/HTML files or also other types? If so, can we distinguish them by the file names (only use .txt, .htm, .html)? It would be nice if you uploaded some example data somewhere. – Byte Commander Feb 14 '16 at 18:21
  • Please clarify - What input files do you have? ZIP archives containing HTML and text files? What output do you need? Do you have to add it to your "RDD in a Spark project" all in one BIG lump, or can you add it in multiple, smaller lumps? – waltinator Feb 14 '16 at 18:26
  • sorry for being unclear. There is a folder hierarchy like so: – aquagremlin Feb 14 '16 at 19:11
  • folder1 has folders a,b,c,d,e,f. Folder a has folders alpha, beta, gamma in it. Folder alpha has file1.zip, file2.zip. Folder b has folders theta and zeta. Thete has files1.zip.......file1000.zip. Folder C has folder1......folder500 in it - and each of these has 10 to 20 zip files. Some zip files contain multiple text files. Some zip files have html and jpg files. I would like to add the data as a single HUGE text file. – aquagremlin Feb 14 '16 at 19:17
2

find is a good tool when you need to do something recursively in directories. zcat can

My approach would be something like this :

find /home/username/zipFiles -type f -name "*.zip" -exec zcat {} \; > outputfile.txt

Example:

$ df > testfile.txt                                            
$ zip testArchive testfile.txt
  adding: testfile.txt (deflated 54%)
$ find . -maxdepth 1 -name "*.zip" -exec zcat {} \;            
Filesystem     1K-blocks     Used Available Use% Mounted on
udev             1954212        4   1954208   1% /dev
tmpfs             393160     3392    389768   1% /run
/dev/sda1      115247656 82424300  26946008  76% /
none                   4        0         4   0% /sys/fs/cgroup
none                5120        0      5120   0% /run/lock
none             1965792       80   1965712   1% /run/shm
none              102400       32    102368   1% /run/user
cgmfs                100        0       100   0% /run/cgmanager/fs
/dev/sdb1       30343056       48  30343008   1% /media/xieerqi/SDCZ43-032G
  • as a test I tried this in a folder with only one zip file and the output was empty...........find . -type f -name ".zip" -exec zcat {}/.txt \; > outputfile.txt – aquagremlin Feb 17 '16 at 0:15
  • @aquagremlin you were searching for ".zip" file, named exactly .zip. Look at my answer - you need to include "*.zip" – Sergiy Kolodyazhnyy Feb 17 '16 at 0:34
  • Ah, my bad - I've included the {}/*.txt part. zcat doesn't need the /*.txt part. Fixing that – Sergiy Kolodyazhnyy Feb 17 '16 at 0:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.