7

I have a remote machine I need to keep an eye on. It's running Ubuntu Studio 22.04 (KDE Plasma). A few weeks ago it crashed and journalctl showed a "Bug" that occurred a couple minutes before the crash. So I wrote a simple script that follows journalctl and if the word "Bug" appears it sends a warning email. I set that script running about 10 days ago. Yesterday, I connected to the machine remotely and checked htop and found that the script was using over 90% of CPU. I killed it and CPU usage dropped back to normal. Here's the script:

#!/bin/bash

#####################
#   THIS SCRIPT LAUNCHED AT STARTUP, CHECKS journalctl for string "Bug"
######################

while true; do

nohup journalctl --follow | grep -i -q "bug" && mutt -s "ALERT - AirchainPC may be in TROUBLE" -- [email protected] < bug_issued_by_journalctl.txt &>/dev/null &

done

Anything there that might explain the high CPU usage? BTW, I don't think I need that "nohup".

6
  • 5
    Just curious: what did you think this script will do? There's a file being redirected to mutt which otherwise isn't mentioned in the script at all, so are you just sending yourself the same file as an email infinitely many times?
    – muru
    Commented Oct 24, 2023 at 13:50
  • 7
    Sending the command to the back ground with & in a while true; ... loop will free the loop immediately resulting in rerunning the command sending hundreds/thousands to the background until your system resources are consumed ... Also detaching it with nohup will do the same ... So, remove both and your script should run as you expect it without consuming all your system's resources.
    – Raffa
    Commented Oct 24, 2023 at 14:20
  • 1
    @ArturMeinild (technically that's not the reason, the problem is the & at the end. OP's backgrounding everything in an infinite loop, so it doesn't matter what's going on with journalctl, grep, or mutt, really. I'd like to know how much OP understands shell scripting before attempting an explanation)
    – muru
    Commented Oct 24, 2023 at 14:21
  • 1
    OP (me) is not very skilled at scripting, as can clearly be seen. I'm a community radio station studio engineer trying to keep things running. My intent is to keep an eye on this remote machine's journalctl and get a notification email if the search string occurs. So, this is a crude attempt at that. I think I ran out of inotify watch descriptors due to my infinite looping. Thank you. Commented Oct 24, 2023 at 14:35
  • 3
    Note that journalctl has a --grep option, this will avoid some overhead as well.
    – G. Sliepen
    Commented Oct 25, 2023 at 9:45

1 Answer 1

15

Please, change your script to something like:

#!/bin/bash
journalctl --follow | grep --line-buffered -i bug | while read -r ; do
  mutt -s "ALERT - AirchainPC may be in TROUBLE" -- '[email protected]' < bug_issued_by_journalctl.txt
done

Here, you are following the journal trying to catch lines containing the "bug" string. If the string is matched, it will be returned by grep immediately (thanks to the --line-buffered option) into the while loop. The read command will consume each such line and return, so that the body of the while statement will be executed at that time.

I don't know what the contents of the bug_issued_by_journalctl.txt file are, but I would prefer something dynamically created and containing (part of) the string matched from journalctl's output. In such a case, replace read -r with read -r msg and use the contents of the $msg variable in the mail text. Also, if the script is to be run by another user and/or from another directory, the full path for the above-mentioned file should be used.

Because of the --follow option in journalctl command, this script will never return. If you want, you can send the script to background like this:

#!/bin/bash

check_bug () {
  journalctl --follow | grep --line-buffered -i bug | while read -r ; do
    mutt -s "ALERT - AirchainPC may be in TROUBLE" -- '[email protected]'
  done
}

check_bug &>/dev/null &

A simple advice: Always test interactively first before putting something into background! While testing, you will also want to replace the &>/dev/null in the last line, with something like &>/tmp/check_bug.out.


Update: As @G.Sliepen suggested, you may replace the command sequence of

journalctl --follow | grep --line-buffered -i bug

with

journalctl --follow --grep=bug

You should read man journalctl page though, for case-sensitivity and other details.

5
  • 1
    Well, thank you very much for the suggested and far more sensible script, and the detailed explanation! I'm just a volunteer at the station yet the pressure is on to keep things running. Your help is appreciated! Commented Oct 24, 2023 at 15:36
  • 2
    It may be worth mentioning that the line-buffered option isn’t POSIX (I’m not sure why it’s needed, anyway: something to do with buffering output?). Commented Oct 25, 2023 at 2:02
  • 1
    @D.BenKnoble on this site, there is no need since only Ubuntu is on topic, so every command is assumed to be for the GNU toolset and, more specifically, for whatever version of any tool is present in Ubuntu.
    – terdon
    Commented Oct 26, 2023 at 11:29
  • @thatjackelliott note that the reason you saw the CPU spike is because you had while true. This spams the CPU, and adding a sleep 1 for 1 second delay, or even a sleep 0.1 for a 0.1 second delay, to your loop would make everything better. FedKad's solution is much better, of course, but next time you need an infinite loop for a tool that lacks the --follow option, add a sleep command to the end of it.
    – terdon
    Commented Oct 26, 2023 at 11:31
  • It might be worth noting that journalctl's builtin option --grep='pattern' will only search the MESSAGE= field and will match case insensitive if the pattern is all lowercase or case sensitive otherwise ... So, there might be cases where piping to /bin/grep is preferred.
    – Raffa
    Commented Oct 26, 2023 at 14:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .