2

I want to find the number of unique words in my file named cdj.tsv. I can use head -n 1 cdj.tsv to get the first line. Now I want number of unique words in this line. How can I get that? Result of the command head -n 1 cdj.tsv looks like:

Country China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   China   Denmark Denmark Denmark Denmark Denmark DenmarkDenmark  Denmark Denmark Denmark Denmark Denmark Denmark Denmark Denmark Denmark Denmark Denmark Denmark Denmark DenmarkDenmark  Denmark Denmark Denmark Denmark Denmark Denmark Denmark Denmark Denmark Denmark Denmark Denmark Denmark DenmarkDenmark  Denmark Denmark Denmark Denmark Denmark Denmark Denmark Denmark Denmark Denmark Denmark Denmark Denmark DenmarkDenmark  Denmark Denmark Denmark Denmark Denmark Denmark Denmark Denmark Denmark Denmark Denmark Denmark Denmark DenmarkDenmark  Denmark

So, I want the output to be 3 (for Country, China and Denmark).

Thanks

3 Answers 3

3

One simple way:

  • Get the first line from a file with head -n 1 cdj.tsv (You already know that) or from multiple files by name like this head -q -n 1 cdj.tsv file2.tsv file3.tsv the -q will suppress printing extra headers / file names so that only the lines from the files are printed. You can use globbing * with input file names as well like this head -q -n 1 *.tsv to process all files in the current directory with .tsv extension as input.

  • Then, pipe | that to tr -s ' ' '\n' to process the words each on a new line i.e. one at a time ... (notice: many alternate tools can be used to do the same thing in this step even the much less efficient xargs -n 1 and the answer by @Peter Cordes in this regard is worth reading.)

  • Then, pipe that to sort -u to sort and print only unique words.

  • Then, to get both the words themselves and their count, pipe that to tee with wc -l in a process substitution syntax >(wc -l) and put both in a subshell syntax (...) to group the output like so:

    head -q -n 1 *.tsv | tr -s ' ' '\n' | sort -u | (tee >(wc -l))
    
  • The result from your example will look like this:

    China
    Country
    Denmark
    DenmarkDenmark
    4
    

Another faster way with awk or gawk:

  • Start a word (field) counter, set its initial value to 1 like this i=1, limit its maximum value to the available number of fields like this i<=NF and increment it by 1 with each new field like this i++ and put all that in an awk for control statement syntax like this for ( i=1; i<=NF; i++ ).

  • Then, for each field, check if the line being processed is the first line in the file like this NR==1 and if that is true check if the current field (word) hasn't occurred before and if true add its reference to an array like this !seen[$i]++ and print it with print $i and put all that in an awk if control statement syntax in an action group {...} like this { if ( NR==1 && !seen[$i]++ ) print $i }.

  • Then, print the total number of unique words (fields) with print length(seen) and put it in a separate action group after the awk conditional pattern element END like this END { print length(seen) }.

  • Then, Use it on a single input file like so:

    awk '{ for ( i=1; i<=NF; i++ ) { if ( NR==1 && !seen[$i]++ ) print $i }} END { print length(seen) }' cdj.tsv
    
  • Or use it on multiple input files with FNR==1 instead of NR==1 like so:

    awk '{ for ( i=1; i<=NF; i++ ) { if ( FNR==1 && !seen[$i]++ ) print $i }} END { print length(seen) }' *.tsv
    
  • Or use it on multiple input files with <(head -q -n 1 *.tsv) and without NR==1 or FNR==1 like so:

    awk '{ for ( i=1; i<=NF; i++ ) { if ( !seen[$i]++ ) print $i }} END { print length(seen) }' <(head -q -n 1 *.tsv)
    
16
  • xargs -n 1 seems an awkward and inefficient way to tokenize a string. It forks+execs echo for every input word, so it will lose words like -n that are options to /bin/echo, as well as being slow for large inputs. (And yes, I checked with strace -f xargs -n 1 that it does actually fork/exec, not handling this special case internally). Your Awk version is much better, just histogramming. You could also just use tr, sed, or awk to turn whitespace into newlines to feed to sort -u. May 21 at 16:04
  • @PeterCordes “Awkward” maybe but correct… “it will lose words” can you prove this? “/bin/echo” that’s part of how xargs work … tr,sed, or awk or even pure bash are possible alternatives but xargs comes ready out of the box with not only ability to deal with white space but newlines as well among others and this becomes handy in the case of reading from multiple files above … I agree it is slower this way but then the faster awk method is provided. Diversity is a good thing… letting folks know possible different ways of doing something is good too :)
    – Raffa
    May 21 at 16:46
  • Sure, easy to prove: echo 'foo -n' | xargs -n 1 prints foo instead of foo / -n on my Arch GNU/Linux system (GNU coreutils 9.0, xargs from GNU findutils 4.8.0). The fact that /bin/echo is part of how xargs works if you don't specify a command is exactly why you shouldn't use it this way, at least on GNU systems where echo supports some options. May 21 at 16:48
  • @PeterCordes This behavior is both expected and wanted. This is what I meant by “among others” in my previous comment:) The OP wants unique words …. Need l say more? … Nevertheless, your obsetvation is true.
    – Raffa
    May 21 at 16:56
  • 1
    Ah yes, leading whitespace does lead to an empty line in the output, which is part of what gets sorted and counted. Good point, thanks. But at no point is it an "argument" to anything. It's only data being piped around. May 22 at 4:40
3
perl -ne 'undef %c; grep($c{$_}++,split); print join(" ",scalar(%c),keys %c),"\n"'

4 Country China DenmarkDenmark Denmark
2
  • 1
    Thanks, user10489. I have multiple files. How can I specify them as input for this command?
    – DEEP
    May 21 at 6:01
  • That wasn't in your question, so this doesn't handle that.
    – user10489
    May 21 at 12:10
1
 head -q -n 1 *.tsv | tr -s '[:space:]' '\n' | sort -u | tee /dev/tty | wc -l

tr -s characters \n is a good way to split on whitespace, into separate lines. Any number of whitespace characters are "squeezed" (tr -s) down to one newline.

If you also want to remove leading whitespace (before the first word), you can use sed with two separate -e operations, one for that and one to squeeze later whitespace. -E is extended regex

... | sed -E -e 's/^[[::space:]]+// -e 's/[[:space:]]+/\n/g' | ...

Alternatively to using tee to the terminal (which interferes with redirecting the whole thing to a file), @Raffa's answer suggests | (tee >(wc -l)) process substitution. (Not sure why tee itself has to be in a subshell, though.)


A more efficient version would just histogram in awk or perl instead of piping through multiple processes to sort-unique and then count. Two other answers show ways to do that.


Another part of Raffa's answer is not a good suggestion for general use, though. xargs -n 1 (using its default command of echo) works as an alternative to tr for some cases, but is much slower (fork/exec of a whole process for every word).

Worse, GNU echo eats -n and -e so it's not robust if your text could contain those "words". For example,

echo -e 'foo \t  -n \n\n bar\n\n\nbaz' |  tr -s '[:space:]' '\n' | sort -u | tee /dev/tty | wc -l
bar
baz
foo
-n
4
# this version loses the -n
echo -e 'foo \t  -n \n\n bar\n\n\nbaz' |  xargs -n 1 | sort -u | tee /dev/tty | wc -l
bar
baz
foo
3

The xargs -n 1 hack also exposes your data to other users as command line args, which they could see via ps auxw if they ran it at just the right time. (e.g. in a loop, or via something watching /proc to grab the command line args of new PIDs.)

If you don't care about efficiency for small data sets and find it easier to type for interactive use, you can use it if you want, but it's a weird abuse of xargs's debug feature of defaulting to echo instead of a meaningful command. It's definitely not something you'd want to put in a script; if you're going to take the time to write a script, use something robust and efficient if it only takes an extra couple seconds to type or remind yourself of what a man page says.

It does have one upside of ignoring leading whitespace.

1
  • 1
    The sub-shell at the end is to group both words and their count in case of redirecting to a file … +1 :).
    – Raffa
    May 28 at 10:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.