I have an 11gb wordlist file which is already sorted as each word is on its own line.

I need to remove duplicates and lines starting from 077.

I guess I need to run sed and sort -u together but I also want a live output (display whats happening in terminal) and if possible display the time left. All of this in one command and it must be able to run optimally at full performance in ubuntu 12.10.

Time is not very important but if there is a way for me to calculate the ETA, I might be able to borrow my dad's i7 based CPU which should process it faster obviously otherwise I'll have to use an older core 2 CPU.

So please give me the complete command, be it sed/sort/awk to do this (whichever is most optimal).


As you have already understood, you need to use sort -u to remove all duplicated lines. sort however does not support showing progress.

You can however write a little script that reads the contents from the input file and outputs it to the standard output, and while it does so it prints progress. Here is an example:


set -e

byte_count=$(wc -c "$1" | cut -d" " -f1)

while read -N $chunk_size chunk
    echo -ne "\rRead $bytes_read of $byte_count bytes [$[ 100 * bytes_read / byte_count ]%]" >& 2
    echo -n "$chunk"
    bytes_read=$[ bytes_read + chunk_size ]
done < "$1"

echo >& 2

You can use this script as follows:

./script-name input-file | sort -u > output-file

The progress will not take in account the time that sort will use to actually write the output, however this is fairly less than the time it takes to read the input file. This should be the most efficient shell-based solution.

| improve this answer | |

tee and tail -f are both good commands for following the action as files are written, but neither will help you with the ETA of your sort command (nor will they let you see what's going on under the hood of sort -u; just the final output after most of the work is done)

Either pipe your output through tee (which will both write to 'output_file' and stdout):

sort -u input_file | tee output_file

or use tail -f:

sort -u input_file -o output_file &
tail -f output_file

Additionally: if your input is pre-sorted (as your question suggests) and all you want is to remove adjacent duplicate lines, then uniq is much faster than sort -u (and our tee/tail will actually be a useful way to monitor progress)

uniq input_file | tee output_file
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.