13

I was trying sed to replace some keywords in a large file (100 MB). I was unaware of the -i (inplace) option, so my first attempt was to redirect like this:

sed 's/original/edited/g' file.log >> file.log

what happened after that was that my PC went to a halt, almost no keyboard input. I tried a different console Ctrl + Alt + F1 but after slowly entering user name, it halted too. Without keyboard, my only option was to hardware-reset the machine. After logging in, I saw that file.log was about 8 GB.

I really would like to understand why the execution of that command was able to make the system so unresponsive, and if mechanisms exist at the system level to trigger alerts and kill the offending process?

  • 7
    Is this a single core machine? It seems very strange that this should have brought a modern computer to its knees. Filled up your disk, yes. Used up 100% of one of your cores, yes. But a full crash? – terdon Jul 6 '16 at 22:27
  • Is there anything peculiar about that file ? if this is not a problem, could you post its contents to pastebin ? – Sergiy Kolodyazhnyy Jul 6 '16 at 22:53
  • Also, what is the amount of your memory ? Could you provide us with output of free -h ? – Sergiy Kolodyazhnyy Jul 6 '16 at 23:39
  • Why use a stream editor in the first place when you want to change a file? ex -sc '%s/original/edited/ge|x' file.log should do what you want in a UNIX idiomatic way without the sed -i side effects. – David Ongaro Jul 7 '16 at 21:34
  • Note that even if you're doing it correctly (by any of the methods people are providing), it can be dicey doing this sort of thing to a log file belonging to an active process. – Random832 Jul 7 '16 at 23:35
10

As has already been said, >> appends to the file, so your sed command will sit there reading the lines it has just output, and then outputting them some more. If you wanted to replace your file in-place, > still wouldn't work, but you're aware of sed's -i option, which is definitely the one you want.

If, however, you're absolutely sure that you want to append to a file you're reading as a stream, and only want to do one pass of this, consider using sponge from the moreutils package;

sed 's/original/edited/g' file.log | sponge >> file.log

sponge reads from stdin into memory until EOF, then dumps all its contents to stdout, so sed will hit the end of the file, stop reading it, close it, and then sponge will start appending to it.

  • 2
    sponge is a nice utility to know about, but sed already has a -i option: -i[SUFFIX], --in-place[=SUFFIX], edit files in place (makes backup if SUFFIX supplied). – Joshua Taylor Jul 7 '16 at 13:36
  • @JoshuaTaylor, OP was using >>, which appends, rather than >, which replaces. Granted, OP had specifically mentioned -i in the post and it seems like a far more common use case than this one, but I thought it was worth pointing out that the specific operation OP had posted was possible without too much faff, if you're really sure it's what you want to do. – ymbirtt Jul 7 '16 at 13:50
  • 1
    I mentioned it here because it was the key in the accepted answer. That said, I am genuinely happy to learn about sponge; it's a new tool for my toolbox, and worthy of an upvote just for that. – Joshua Taylor Jul 7 '16 at 14:33
  • 1
    Ah! I see. I'll tweak my answer to make that a bit clearer. Also, if you enjoyed sponge, take a look at vipe. moreutils is just a magical package filled with things you never knew you needed – ymbirtt Jul 7 '16 at 14:50
18

Your sed command was trying to read the file it was appending to. It will never reach End-Of-File, but will eat a lot of CPU time trying. That's why ^C (interrupt current process) was invented.

  • I don't think ^C was an option there... it went to a HALT, i.e. no blinking cursor, stuck! – EKons Jul 7 '16 at 9:41
18

Appending back to the file you read from is in no case a good idea, as you will end up with an ever growing file. If you really want to write back into the file you should use the -i flag:

sed -i 's/original/edited/g' file.log

or if you want it to create a backup before doing changes you can add a file suffix to the -i flag:

sed -i.bak 's/original/edited/g' file.log

This would create a file called file.log.bak and then doing changes, what you did there by trying to append to the file you're reading from we call in programmer slang a data race, where different processes race for the same data source be it input or output. This is also why your machine came to a halt.

  • 1
    I'm surprised this is the accepted answer, because it doesn't even address OP's question "I really would like to understand why the execution of that command was able to make the system so unresponsive, and if mechanisms exist at the system level to trigger alerts and kill the offending process?" – Steve Jul 7 '16 at 20:59
  • @Steve As of why it came to an halt I addressed, but for the second part you're right. I did not address that because I not know an answer to this. We tested the command after a chat discussion extensively and came to totally different results on different machines and operating systems. Example: On a machine with arch it only lets the file grow forever, but does not render the machine unresponsive. On my Ubuntu machine I was getting the same result as the questioner without a chance to kill the process. A second machine testing the same in an Ubuntu VM came to the same halt. – Videonauth Jul 7 '16 at 22:44
  • An straceof the whole process on the other side didtn reproduce the result and this on my machine and on the machine of an other user. Sure there are mechanism with which you can kill unresponsive applications, but if your machine is rendered unresponsive, your only left with one option, resetting it. I'm still testing on this and before I not understand fully whats causing the described behavior, I'm unable to address this part of the question. – Videonauth Jul 7 '16 at 22:44
  • It's probably a difference in the kernel configurations, like a different scheduler that prioritizes IO, or differences in the disk/filesystem driver between the systems. It's good to see the investigation you guys did, that's good information. – Steve Jul 8 '16 at 2:09
  • If you're interested in another data point; I tried this on a CentOS machine with a fairly small file, and it did exactly the same thing as my sponge solution below. I imagine that for a small file sed will buffer the entire thing into memory and then close it, rather than keeping hold of the handle. With a ~100MB file, as in OP, it grew indefinitely but didn't brick the machine. – ymbirtt Jul 8 '16 at 7:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.