Tell me more ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.
up vote 0 down vote favorite

How do I extract all words from all files in a given directory in order to make a dictionary? The words must be different from each other(letters in caps are not taken into consideration. For example, car=Car), they will be written with small letters in the dictionary(even if in originally they had caps), they will be sorted alphabetically and each line in the dictionary will contain only one word. The dictionary will be in /home/dictionary.txt.

share|improve this question
It depends greatly on the format of the files that will be read to create the dictionary. Are they one word a line, words separated by commas or ; or <>.... Please give an example. – Luis Alvarado♦ Dec 7 '11 at 17:40
the words are extracted from a random text. For example, from "mom, dad and my brother went home." we should extract only "mom dad and my brother went home". Basicly, we only have to keep a continuous string of characters from a-z or A-Z. Any other type of character should be discarded. – johnny Dec 7 '11 at 17:49
So if "Superman2K came to my home and fought Spiderman3D" Then it should extract superman came to my home and fought spiderman" or without supermand and spiderman beause of the numbers. – Luis Alvarado♦ Dec 7 '11 at 17:56
2  
Doesn't this really belong on programmers.stackexchange.com or similar? – Caesium Dec 7 '11 at 17:58
That is the exact text it should extract(including spiderman and superman), except the words have to be sorted alphabetically. – johnny Dec 7 '11 at 18:29
show 1 more comment

3 Answers

up vote 1 down vote

johnny, I think what you want to do could easy be done in a bash script. But you are going to want to study up on redirection, input files... ; should get you going hopefully. | For getting words or patterns...

share|improve this answer
up vote 0 down vote

Depends heavily on the contents of the directory, but... this should be at least a very good start if we are talking about plain text files in the directory in question.

cd
find $directory_name -type f -exec grep -o -E '\w+' {} \; | sort -u -f > out
tr '[:upper:]' '[:lower:]' < out > dictionary.txt
rm out

About the switches, man $command can expound.

Remove numbers: sed 's/[0-9]*//g' dictionary.txt > a_different_file

Remove blank lines: sed '/^$/d' dictionary.txt > some_other_file

share|improve this answer
There is a problem with this solution. Apart from returning words from the given files, it also returns numbers. What do you think I should modify? – johnny Dec 7 '11 at 20:02
Yeah... a good start depending on what you have. See updated answer. rdh said it plainly on the learning curve... – user8290 Dec 7 '11 at 21:54
up vote 0 down vote

You can do this with awk 

 awk 'BEGIN {RS=" "}; /^[A-Za-z]*$/ {print tolower ($0)}' ./* | sed 's_[.].*$__g' | sort -u

"print tolower" simply converts to all lower case (makes sort -u work).

The sed removes trailing periods, you may or may not need to manage other symbols depending on your input files.

share|improve this answer
I still don't get the desired output. I need to extract all continuous strings of characters that contain only letters from a-z or A-Z(which have to be transformed into small letters) and then sort them alphabetically and put them in a different file. – johnny Dec 8 '11 at 16:25
awk uses regular expressions awk 'BEGIN {RS=" "}; /^[A-Za-z]*$/ {print tolower ($0)}' ./* | sed 's_[.].*$__g' | sort -u – bodhi.zazen Dec 8 '11 at 16:51
Now I have some words missing. – johnny Dec 8 '11 at 17:12
Well, we can only get you so far. You will need to read up on regular expressions, look at the output, and revise. Perhaps run through the files in steps. But it is impossible for us to write an exact script without examining the data and knowing exactly what you want , what is missing, etc. – bodhi.zazen Dec 8 '11 at 18:36
Thank you very much. I will try to adapt the script to exactly fit my needs. – johnny Dec 8 '11 at 18:43

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.