Shell script & "nvidia-smi" - needs right command/flag!

Question

I've got a problem regarding a shell-script and the "nvidia-smi" command!

I've made a script that as protection against CPU overheating on my Ubuntu Server 14.04.2. The scripts works nicely but I need to make it work on my 4 GPU's as well. I'm pretty green when it comes to bash scripts so I've been looking for commands which would make it easy for me to edit the script. I found and tested a lot of them, but none seems to give me the output I need! I'll show you the commands and the output below. And the scripts as well.

What I need is a command which lists the GPU's the same way the "sensors" command from "lm-sensors" does. So that I can use "grep" to select a GPU and set the variable "newstring" (the temp. two digits). I've been trying for a couple of days, but have had no luck. Mostly because the command "nvidia-smi -lso" and/or "nvidia-smi -lsa" doesn't exist anymore. Think it was an experimental command.

Here's the commands I found and tested & the output:

This command shows GPU socket number which I could put into the string "str" but the problem is that the temp. is on the next line. I've been fiddling with the flag "A 1" but haven't been able to put it into the script:

# nvidia-smi -q -d temperature | grep GPU
Attached GPUs                       : 4
GPU 0000:01:00.0
        GPU Current Temp            : 57 C
        GPU Shutdown Temp           : N/A
        GPU Slowdown Temp           : N/A
GPU 0000:02:00.0
        GPU Current Temp            : 47 C
        GPU Shutdown Temp           : N/A
        GPU Slowdown Temp           : N/A
GPU 0000:03:00.0
        GPU Current Temp            : 47 C
        GPU Shutdown Temp           : N/A
        GPU Slowdown Temp           : N/A
GPU 0000:04:00.0
        GPU Current Temp            : 48 C
        GPU Shutdown Temp           : N/A
        GPU Slowdown Temp           : N/A

This command shows the temp in the first line, but there's no GPU number!?

# nvidia-smi -q -d temperature | grep "GPU Current Temp"
        GPU Current Temp            : 58 C
        GPU Current Temp            : 47 C
        GPU Current Temp            : 47 C
        GPU Current Temp            : 48 C

This command shows the GPU number you select, but there's still no output showing the GPU numer/socket/ID!?

# nvidia-smi -q --gpu=0 | grep "GPU Current Temp"
GPU Current Temp            : 59 C

And this commands shows the GPU number and the results in the same row!! But, no temperature!!

# nvidia-smi -L
GPU 0: GeForce GTX 750 Ti (UUID: GPU-9785c7c7-732f-1f51-..........)
GPU 1: GeForce GTX 750 (UUID: GPU-b2b1a4a-4dca-0c7f-..........)
GPU 2: GeForce GTX 750 (UUID: GPU-5e6b8efd-7531-777c-..........)
GPU 3: GeForce GTX 750 Ti (UUID: GPU-5b2b1a2f-3635-2a1c-..........)

And a command which shows all 4 GPU's temp. without anything else. But still I need the GPU number/socket/ID!?

# nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader
58
47
47
48

What I'm wishing for! If I could get a command which made a output like this I would be the happiest guy around:

GPU 0: GeForce GTX 750 Ti   GPU Current Temp            : 58 C
GPU 1: GeForce GTX 750   GPU Current Temp            : 47 C
GPU 2: GeForce GTX 750   GPU Current Temp            : 47 C
GPU 3: GeForce GTX 750 Ti   GPU Current Temp            : 48 C

Here's the output that "sensors" from "lm-sensors". As you can see the unit info and the temp is in the same line:

# -----------------------------------------------------------
# coretemp-isa-0000
# Adapter: ISA adapter
# Physical id 0:  +56.0°C  (high = +80.0°C, crit = +100.0°C)
# Core 0:         +56.0°C  (high = +80.0°C, crit = +100.0°C)
# Core 1:         +54.0°C  (high = +80.0°C, crit = +100.0°C)
# Core 2:         +54.0°C  (high = +80.0°C, crit = +100.0°C)
# Core 3:         +52.0°C  (high = +80.0°C, crit = +100.0°C)
# -----------------------------------------------------------

Here's the part of the script that needs changing. As mentioned in the top, this works using the command "sensors" from the application "lm-sensors". "lm-sensors" doesn't show GPU temp. when running CUDA and the driver attached, so we need another command to get the GPU's listed and the temp. shown. You may know another way to fix my problem, if please don't hesitate to show me.:

[...]
echo "JOB RUN AT $(date)"
echo "======================================="

echo ''
echo 'CPU Warning Limit set to => '$1
echo 'CPU Shutdown Limit set to => '$2
echo ''
echo ''

sensors

echo ''
echo ''

for i in 0 1 2 3
do

  str=$(sensors | grep "Core $i:")
  newstr=${str:17:2}

  if [ ${newstr} -ge $1 ]
  then
    echo '===================================================================='         >>/home/......../logs/watchdogcputemp.log
    echo $(date)                                                                        >>/home/......../logs/watchdogcputemp.log
    echo ''                                                                             >>/home/......../logs/watchdogcputemp.log
    echo ' STATUS WARNING - NOTIFYING : TEMPERATURE CORE' $i 'EXCEEDED' $1 '=>' $newstr >>/home/......../logs/watchdogcputemp.log
    echo ' ACTION : EMAIL SENT'                                                         >>/home/......../logs/watchdogcputemp.log
    echo ''                                                                             >>/home/......../logs/watchdogcputemp.log
    echo '===================================================================='         >>/home/......../logs/watchdogcputemp.log

# Status Warning Email Sending Code
# WatchdogCpuTemp Alert! Status Warning - Notifying!"

/usr/bin/msmtp -d --read-recipients </home/......../shellscripts/messages/watchdogcputempwarning.txt

    echo 'Email Sent.....'
  fi
[...]

I hope there's a bash-script guru out there, ready to solve this issue Have a nice weekend!

Kind Regards, Dan Hansen Denmark

.

Answer 1

awk is a great all-purpose tool perfect for this. For each input line it executes all commands which match. Here I pipe the output from two of your commands into awk. When it matches the lines beginning GPU 0: it splits the line into 2 parts at the "(" character and saves the first part (x[1]) in an array indexed by the gpu number, got from field 2 ($2: fields are separated by whitespace).

When it matches lines GPU 0000:01:00.0 it splits field 2 into 3 parts at the ":" character and saves the 2nd part minus 1 as the gpu number.

When it matches lines with GPU Current Temp it save the 5th and 6th fields (concatenated with a space) into another array, indexed by the "global" variable gpu set by a previous line.

At the end of input (END) we print the arrays, now they have all the info.

mynvidia(){

 ( nvidia-smi -L
   nvidia-smi -q -d temperature | grep GPU
 ) | 
 awk '
 /^GPU [0-9]:/     { gpu=0+$2; split($0,x,"("); gputype[gpu]=x[1]; }
 /^GPU 00/         { split($2,x,":"); gpu=x[2]-1; }
 /GPU Current Temp/{ temperature[gpu] = $5 " " $6; }
 END               { for(gpu=0;gpu<99;gpu++)
                     if(gputype[gpu]!="")
                      printf "%-30s GPU Current Temp: %s\n",gputype[gpu],temperature[gpu]
                   }'
}

str=$(mynvidia | grep "GPU $i:")
newstr=${str:49:2}
# ... echo "$str"

Here's the output of function mynvidia in a script:

GPU 0: GeForce GTX 750 Ti      GPU Current Temp: 57 C
GPU 1: GeForce GTX 750         GPU Current Temp: 47 C
GPU 2: GeForce GTX 750         GPU Current Temp: 47 C
GPU 3: GeForce GTX 750 Ti      GPU Current Temp: 48 C

Answer 2

All you really need to do is to remove the newline from all lines starting with GPU. You could use this perl one liner which removes newlines from lines whose first three characters are GPU:

$ nvidia-smi -q -d temperature | grep GPU | perl -pe '/^GPU/ && s/\n//' | grep ^GPU
GPU 0000:01:00.0        GPU Current Temp            : 57 C
GPU 0000:02:00.0        GPU Current Temp            : 47 C
GPU 0000:03:00.0        GPU Current Temp            : 47 C
GPU 0000:04:00.0        GPU Current Temp            : 48 C

Alternatively, you can do the whole thing in awk:

$ nvidia-smi -q -d temperature | awk '{if(/C$/){print last,$0};last=$0};' 
GPU 0000:01:00.0         GPU Current Temp            : 57 C
GPU 0000:02:00.0         GPU Current Temp            : 47 C
GPU 0000:03:00.0         GPU Current Temp            : 47 C
GPU 0000:04:00.0         GPU Current Temp            : 48 C

That simply checks if the current line ends with a C and, if it does, it prints it ($0) along with the previous line. last=$0 saves the current line as last to make it available when the next line is processed.

Here's the same logic implemented in Perl:

$ nvidia-smi -q -d temperature | perl -lne 'print "$last $_" if /C$/; $last=$_' file 
GPU 0000:01:00.0         GPU Current Temp            : 57 C
GPU 0000:02:00.0         GPU Current Temp            : 47 C
GPU 0000:03:00.0         GPU Current Temp            : 47 C
GPU 0000:04:00.0         GPU Current Temp            : 48 C

Finally, since you're doing this in a shell script anyway, you could also get the desired output directly from bash:

$ nvidia-smi -q -d temperature | while read line; do 
    [[ $line =~ C$ ]] && printf "%s : %s\n" "$last" "$line";
    last="$line"; done
GPU 0000:01:00.0 : GPU Current Temp            : 57 C
GPU 0000:02:00.0 : GPU Current Temp            : 47 C
GPU 0000:03:00.0 : GPU Current Temp            : 47 C
GPU 0000:04:00.0 : GPU Current Temp            : 48 C

Answer 3

Thanks for your reply!! This seems to solve my problem!! The second suggestion using awk didn't look the same way here on my system, but your first suggestion was just perfect! I repeat PERFECT!! Thanks a lot for that my friend. This really helps me. I have to say that you solve the issue very nicely and I'll keep the other suggestions for further use!! Again, thanks a lot for your help my friend!

For others to see I'll and learn of this here's the results on my Ubuntu Server 14.04

This one looks like this on my system:

# nvidia-smi -q -d temperature | awk '{if(/C$/){print last,$0};last=$0};'
    Temperature         GPU Current Temp            : 53 C
    Temperature         GPU Current Temp            : 45 C
    Temperature         GPU Current Temp            : 52 C
    Temperature         GPU Current Temp            : 51 C

And this one, which is just PERFECT looks like this on my system:

# nvidia-smi -q -d temperature | grep GPU | perl -pe '/^GPU/ && s/\n//' | grep ^GPU
GPU 0000:01:00.0        GPU Current Temp            : 53 C
GPU 0000:02:00.0        GPU Current Temp            : 45 C
GPU 0000:03:00.0        GPU Current Temp            : 52 C
GPU 0000:04:00.0        GPU Current Temp            : 51 C

Here I've got the GPU text to "grep" in my script. I've got the GPU socket ID and last but not least I've got the temperature in the same line! Exactly what I asked for. I humbly bow ;)

I hope others with the same issue may use this to solve their problem.

Kind Regards, Dan

Answer 4

I tried the new "edition" of your suggestion and it work nicely! Looks great. But I'm trying to make it work in the script and it seems to be difficult...

Here's the nice output when run as a script:

# ./getgputemp.sh
GPU 0: GeForce GTX 750 Ti      GPU Current Temp: 49 C
GPU 1: GeForce GTX 750         GPU Current Temp: 39 C
GPU 2: GeForce GTX 750         GPU Current Temp: 42 C
GPU 3: GeForce GTX 750 Ti      GPU Current Temp: 51 C

Please show me how to use it in my script. I've tried multiple ways and cant make it work. I need to variables to make the script work. I need "str" which is the GPUnumber and I need "newstr" which is the temperature". Any ideas??

The script as it is right now:

    #!/bin/bash

    # --- WatchdogGpuTemp.sh v.0.1.6 ---
    # Author: DanHansen[at]Denmark
    # Application: nvidia-smi
    # Filename: watchdoggputemp.sh
    # Logfile: watchdoggputemp.log
    # Message file for status warning: watchdoggputempwarning.txt
    # Message file for status critical: watchdoggputempcritical.txt
    # Work directory: /home/username/shellscripts/
    # Log directory: /home/username/logs/
    # Message directory: /home/username/shellscripts/messages/
    #
    # --- WatchdogGpuTemp.sh v.0.1.6 ---

    echo "JOB RUN AT $(date)"
    echo "======================================="

    echo ''
    echo 'CPU Warning Limit set to => '$1
    echo 'CPU Shutdown Limit set to => '$2
    echo ''
    echo ''


( nvidia-smi -L
  nvidia-smi -q -d temperature | grep GPU
) | 
awk '
/^GPU [0-9]:/     { gpu=0+$2; split($0,x,"("); gputype[gpu]=x[1]; }
/^GPU 00/         { split($2,x,":"); gpu=x[2]-1; }
/GPU Current Temp/{ temperature[gpu] = $5 " " $6; }
END               { for(gpu=0;gpu<4;gpu++)
                      printf "%-30s GPU Current Temp: %s\n",gputype[gpu],temperature[gpu]
                  }'


    echo ''
    echo ''

    for i in 1 2 3 4
    do


      str=$( YOUR STUFF AND GREP "GPU $i:)  <------- SET "STR" TO GPU NUMBER
      newstr=${str:49:2}  <------------------------- SET "NEWSTR" TO TEMPERATURE



      if [ ${newstr} -ge $1 ]
      then
        echo '===================================================================='        >>/home/username/logs/watchdoggputemp.log
        echo $(date)                                                                       >>/home/username/logs/watchdoggputemp.log
        echo ''                                                                            >>/home/username/logs/watchdoggputemp.log
        echo ' STATUS WARNING - NOTIFYING : TEMPERATURE GPU' $i 'EXCEEDED' $1 '=>' $newstr >>/home/username/logs/watchdoggputemp.log
        echo ' ACTION : EMAIL SENT'                                                        >>/home/username/logs/watchdoggputemp.log
        echo ''                                                                            >>/home/username/logs/watchdoggputemp.log
        echo '===================================================================='        >>/home/username/logs/watchdoggputemp.log

    # Status Warning Email Sending Code 
    # WatchdogGpuTemp Alert! Status Warning - Notifying!"

    /usr/bin/msmtp -d --read-recipients </home/username/shellscripts/messages/watchdoggputempwarning.txt

        echo 'Email Sent.....'
      fi

      if [ ${newstr} -ge $2 ]
      then
        echo '===================================================================='        >>/home/username/logs/watchdoggputemp.log
        echo $(date)                                                                       >>/home/username/logs/watchdoggputemp.log
        echo ''                                                                            >>/home/username/logs/watchdoggputemp.log
        echo ' STATUS CRITICAL - SHUTDOWN : TEMPERATURE GPU' $i 'EXCEEDED' $2 '=>' $newstr >>/home/username/logs/watchdoggputemp.log
        echo ' ACTION : EMAIL SENT & SYSTEM SHUTDOWN'                                      >>/home/username/logs/watchdoggputemp.log
        echo ''                                                                            >>/home/username/logs/watchdoggputemp.log
        echo '===================================================================='        >>/home/username/logs/watchdoggputemp.log

    # Status Critical Email Sending Code:
    # WatchdogGpuTemp Alert! Status Critical - Shutdown!"

    /usr/bin/msmtp -d --read-recipients </home/username/shellscripts/messages/watchdoggputempcritical.txt

        echo 'Email Sent.....'
        echo 'System will now shutdown.....'
        /sbin/shutdown -h now
        exit

      else
        echo ' Temperature GPU '$i' OK at =>' $newstr
        echo ''
      fi
    done

    echo 'Status - All GPUs are within critical temperature limits'
    echo ''

Answer 5

Maybe you have the knowhow to fix this. I've tested my refitted script but I'm struggling with 2 variables. I need to set GPU number in "str" and the temperature in "newstr". It seems to go just fine with the first string and "grep" is working, but when it comes to the second it halts. I've been counting the spaces 101 times. Do you know if this is the problem. Does spaces need to be "counted" in another way?

Here's my script:

#!/bin/bash

# --- WatchdogGpuTemp.sh v.0.1.2 ---
# Author: DanHansen[at]Denmark
# Thanks to "Terdon" Ubuntu Forums
# Application: nvidia-smi
# Filename: watchdoggputemp.sh
# Logfile: watchdoggputemp.log
# Message file for status warning: watchdoggputempwarning.txt
# Message file for status critical: watchdoggputempcritical.txt
# Work directory: /home/username/shellscripts/
# Log directory: /home/username/logs/
# Message directory: /home/username/shellscripts/messages/
#
# --- WatchdogGpuTemp.sh v.0.1.2 ---

echo "JOB RUN AT $(date)"
echo "======================================="

echo ''
echo 'CPU Warning Limit set to => '$1
echo 'CPU Shutdown Limit set to => '$2
echo ''
echo ''

nvidia-smi -q -d temperature | grep GPU | perl -pe '/^GPU/ && s/\n//' | grep ^GPU

echo ''
echo ''

for i in 1 2 3 4
do

  str=$(nvidia-smi -q -d temperature | grep GPU | perl -pe '/^GPU/ && s/\n//' | grep ^GPU "GPU 0000:0$i:00.0")
  newstr=${str:54:2}

  if [ ${newstr} -ge $1 ]
  then
    echo '===================================================================='        >>/home/username/logs/watchdoggputemp.log
    echo $(date)                                                                       >>/home/username/logs/watchdoggputemp.log
    echo ''                                                                            >>/home/username/logs/watchdoggputemp.log
    echo ' STATUS WARNING - NOTIFYING : TEMPERATURE GPU' $i 'EXCEEDED' $1 '=>' $newstr >>/home/username/logs/watchdoggputemp.log
    echo ' ACTION : EMAIL SENT'                                                        >>/home/username/logs/watchdoggputemp.log
    echo ''                                                                            >>/home/username/logs/watchdoggputemp.log
    echo '===================================================================='        >>/home/username/logs/watchdoggputemp.log

# Status Warning Email Sending Code 
# WatchdogGpuTemp Alert! Status Warning - Notifying!"

/usr/bin/msmtp -d --read-recipients </home/username/shellscripts/messages/watchdoggputempwarning.txt

    echo 'Email Sent.....'
  fi

  if [ ${newstr} -ge $2 ]
  then
    echo '===================================================================='        >>/home/username/logs/watchdoggputemp.log
    echo $(date)                                                                       >>/home/username/logs/watchdoggputemp.log
    echo ''                                                                            >>/home/username/logs/watchdoggputemp.log
    echo ' STATUS CRITICAL - SHUTDOWN : TEMPERATURE GPU' $i 'EXCEEDED' $2 '=>' $newstr >>/home/username/logs/watchdoggputemp.log
    echo ' ACTION : EMAIL SENT & SYSTEM SHUTDOWN'                                      >>/home/username/logs/watchdoggputemp.log
    echo ''                                                                            >>/home/username/logs/watchdoggputemp.log
    echo '===================================================================='        >>/home/username/logs/watchdoggputemp.log

# Status Critical Email Sending Code:
# WatchdogGpuTemp Alert! Status Critical - Shutdown!"

/usr/bin/msmtp -d --read-recipients </home/username/shellscripts/messages/watchdoggputempcritical.txt

    echo 'Email Sent.....'
    echo 'System will now shutdown.....'
    /sbin/shutdown -h now
    exit

  else
    echo ' Temperature GPU '$i' OK at =>' $newstr
    echo ''
  fi
done

echo 'Status - All GPUs are within critical temperature limits'
echo ''

Here's the output when running the script:

# ./watchdoggputemp.sh 55 60
JOB RUN AT Sun Jun 28 10:13:57 CEST 2015
=======================================

CPU Warning Limit set to => 55
CPU Shutdown Limit set to => 60


GPU 0000:01:00.0        GPU Current Temp            : 49 C
GPU 0000:02:00.0        GPU Current Temp            : 46 C
GPU 0000:03:00.0        GPU Current Temp            : 52 C
GPU 0000:04:00.0        GPU Current Temp            : 51 C


grep: GPU 0000:01:00.0: No such file or directory
./watchdoggputemp.sh: line 68: [: -ge: unary operator expected
./watchdoggputemp.sh: line 86: [: -ge: unary operator expected
 Temperature GPU 1 OK at =>

grep: GPU 0000:02:00.0: No such file or directory
./watchdoggputemp.sh: line 68: [: -ge: unary operator expected
./watchdoggputemp.sh: line 86: [: -ge: unary operator expected
 Temperature GPU 2 OK at =>

grep: GPU 0000:03:00.0: No such file or directory
./watchdoggputemp.sh: line 68: [: -ge: unary operator expected
./watchdoggputemp.sh: line 86: [: -ge: unary operator expected
 Temperature GPU 3 OK at =>

grep: GPU 0000:04:00.0: No such file or directory
./watchdoggputemp.sh: line 68: [: -ge: unary operator expected
./watchdoggputemp.sh: line 86: [: -ge: unary operator expected
 Temperature GPU 4 OK at =>

Status - All GPUs are within critical temperature limits

I tried the 4'th suggestion, the script command but when testing it the output looks like this again. No GPU number. Therefore I think your first suggestion is the best for me, but I still struggle with the temperature variable "newstr".:

# nvidia-smi -q -d temperature | while read line; do [[ $line =~ C$ ]] && printf "%s : %s\n" "$last" "$line"; last="$line"; done
Temperature : GPU Current Temp            : 51 C
Temperature : GPU Current Temp            : 46 C
Temperature : GPU Current Temp            : 53 C
Temperature : GPU Current Temp            : 50 C

So, Terdon, if there's any way you can help me across the finish line here, I'll be very thankful. How to put the temperature into "newstr". It seems to be the best solution in my little script.

Looking so much forward to hear from you ;)