Getting specific 'Out of memory!' message when running very involved Perl script [closed]

Question

I'm running a Perl script that parses information from around 7,000 HTML files. Probably not really relevant, but I'm just trying to describe that this is, well, doing a lot of stuff. I originally had them just print out the information I was looking for right into the terminal and then I pasted that into my spreadsheet. However, about half way through, I was told that I was out of memory and it stopped.

Out of memory! - nothing else

I Googled, but I couldn't find anything that talked about the terminal specifically printing this. I could find posts related to Linux about just generally running out of memory, but I wanted to be sure. I've tried, with no success, to have it write to a .txt file instead of printing out into the terminal.

How can I fix this and let the Perl script run all the way through?

Answer 1

Instead of fighting with perl optimization, use a wrapper like this:

wrapper.sh

#!/bin/bash

for i in `ls $1`
do
    echo $i
    <perl script> $i
done

Use it as follows:

./wrapper.sh *.html

So instead of the perl script looping through 7k of files in one run, it is a fresh start for each file.

I also suggest you redirect the output into a file instead of the screen. It is possible that the terminal (xterm, gnome-term, etc) used up all your memory if set to unlimited buffer/lines.