I have 30 gig of text files and html files from various sources laying in a nested folder hierarchy. All these folders are in my home directory in a folder named zipFiles. The hierarchy is 20 levels deep-some branches deeper than others. I want to concatenate the text of ALL these files into one huge text file to use as an RDD in a Spark project. Obviously I would need to convert the html files into text by stripping out the html formatting codes as well.
My approach would be: iterate through the nested folder hierarchy (dont know how to do this in bash) unzip the file if it's html convert it to text add it to the HUGE result
The problems I see with this is running out of ram- if I do it all in memory by just adding the text to a variable in the code.
If i open the HUGE file from disk, write to it and close the file, then I am thrashing the disk and it also will go slowly.
Is bash scripting my only solution because it scares me. I guess I could do it in python but then I need to know how to unzip in python and do html to text conversion as well.