14

Is there any short way to save the pipe output to the same file that it's processed. For example, this is what I am actually doing

$ cat filename | sort | uniq > result
$ rm -f filename
$ mv result filename

I was wondering if there was a way to do it in just one line (not appending those commands using &&)

This is not the way, but to get an idea

$ cat filename | sort | uniq > filename
18

You can use sponge from moreutils package:

LC_ALL=C sort -u filename | sponge filename

You also don't need pipe to uniq, since when sort has -u option to unique lines when sorting.

Note that on GNU system with UTF-8 locales, sort -u or sort | uniq didn't give you unique lines, but the first from sequence of lines which sort the same in current locale.

$ printf '%b\n' '\U2460' '\U2461' | LC_ALL=en_US.utf8 sort | LC_ALL=en_US.utf8 uniq
①

gave you only . Changing locale to C force the sorting order based on the byte values:

$ export LC_ALL=C
$ printf '%b\n' '\U2460' '\U2461' | LC_ALL=C sort | LC_ALL=C uniq
①
②
12

You don't need any extra command like cat and uniq and also without using rm command and mv command to removing and renaming the filename. just use simple command.

sort -u filename -o filename


 -u, --unique
        with -c, check for strict ordering; without -c, output only  the
        first of an equal run

 -o, --output=FILE
        write result to FILE instead of standard output

How it work?

sort command sorts your filename and with -u option, removes duplicate lines from it. then with -o option writes the output to the same file with in place method.

  • 3
    If system crash when sort is running, you will loose your original file. – cuonglm Sep 26 '14 at 10:20
  • @Gnouc So, this is end of unlucky!! :'( – αғsнιη Sep 26 '14 at 11:02
  • 1
    thanks! in this example, with ´sort´ particularly, I should do that. However I was thinking about a general case. @Gnouc, haha there's no way to think that if it didn't happen to you, right? – whitenoisedb Sep 27 '14 at 4:55
3

Your suggested example (below) doesn't work because you'd actually be reading from and writing to the same file simultaneously.

$ cat filename | sort | uniq > filename

The idea with a pipe or redirect is that the command on the left and right hand side of each pipe or redirect run simultaneously, in parallel. The command on the right processes information as it is handed over to it from the command on the left, while the command on the left is still running.

In order for your scenario to work, the command that reads from the file would need to finish before the command that writes to the file begins. In order for this to work you would need to redirect the output into a temporary location first, then once that's finished, send it from the temporary location back into the file.

A better way to do this is basically like in your former example, where you redirect to a temporary file then rename that file back to the original (except that you don't need to delete the file first, because moving deletes any existing target).

$ cat filename | sort | uniq > result
$ mv -f result filename

You could also save it into a string variable, except that only works when data is small enough to all fit in memory at once.

  • As someone put out in a suggested edit, you can change cat filename | sort to just sort filename - cat is unnecessary here. – thomasrutter Sep 26 '14 at 5:58
  • My example below wasn't the way to do it. Thanks for clarification. cat could be unnecessary in this case but I was focusing on the redirection part. – whitenoisedb Sep 26 '14 at 6:04
  • 1
    I was explaining why your below example wouldn't work. I know that you knew it didn't work. – thomasrutter Sep 26 '14 at 6:05
  • Thanks for clarifying! In fact I didn't know what was really happening. – whitenoisedb Sep 26 '14 at 6:08
2

You can use the tee command:

sort -u filename | tee filename > /dev/null

The tee command reads from standard input and writes to standard output and files.

  • 2
    This does not work for me. – pjvandehaar Oct 18 '15 at 5:40
  • 3
    This does not work askubuntu.com/a/752451 – Steven Penny Apr 10 '16 at 22:52
  • This does work for me. e.g., to move a line to the bottom of a file: (cat ~/file | grep -v 3662 ; printentry 3662) | tee ~/file > /dev/null works. Like the original post, this doesn't work if you just > ~/file without the tee. Tee seems similar here to sort -o file, which writes to the named file without continuing the same pipe. – Joshua Goldberg Apr 3 '17 at 4:56
  • Wait, sorry! I've seen empirically this will unpredictably lose data as explained in the link from @Steven. Make a file with the numbers 1..9 on 9 lines. The following will work several times, and then occasionally remove all of the data from the file: (cat x | grep -v 7 ; echo 7) | tee x > /dev/null; cat x I recommend a temp file and mv or perhaps the solution from @Steven's link. – Joshua Goldberg Apr 20 '17 at 16:13
  • @JoshuaGoldberg have you seen my answer on this page? – Steven Penny Apr 20 '17 at 16:20
0

You can use Vim in Ex mode:

ex -sc 'sort u|x' filename
  1. sort u sort unique

  2. x write if changes have been made (they have) and quit

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.