0

I'm working on assembling my personal computer.

The motherboard I'm using is a TB85.

I, at first, tested a 5 GPU coolbits configuration,

then I tested a 6 GPU coolbits configuration,

Both worked fine.

When I run "nvidia-sensors", in a terminal, it shows thermal controls and all the sliders.

also the following runs fine:

sudo nvidia-xconfig -a --cool-bits=4
let NumGPU=6
for (( n=0; n < NumGPU; n++))
do
        /usr/bin/nvidia-settings -a "[gpu:$n]/GPUFanControlState=1"
        /usr/bin/nvidia-settings -a "[fan:$n]/GPUTargetFanSpeed=100"
done

I used a 1-to-4 GPU splitter to add a 7th GPU.

I then had :

  • GTX 1070
  • GTX 1070
  • GTX 1060
  • GTX 1060
  • GTX 1060
  • GTX 1060
  • GTX 1060

(2 GTX 1070 + 5 GTX 1060)

in this configuration "SETI@home" continues to function, however two of the gpus no longer have fan control.

jstateson@tb85-nvidia:~/Desktop$ nvidia-smi
Fri Jul 26 08:21:45 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.116                Driver Version: 390.116                   |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1070    Off  | 00000000:01:00.0  On |                  N/A |
|100%   41C    P2   101W / 151W |   1499MiB /  8117MiB |     91%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 106...  Off  | 00000000:02:00.0 Off |                  N/A |
|100%   62C    P2   121W / 120W |   1300MiB /  6078MiB |     95%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX 106...  Off  | 00000000:03:00.0 Off |                  N/A |
|100%   59C    P2   115W / 120W |   1292MiB /  3019MiB |     92%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce GTX 106...  Off  | 00000000:04:00.0 Off |                  N/A |
|100%   64C    P2    83W / 120W |   1292MiB /  3019MiB |     93%      Default |
+-------------------------------+----------------------+----------------------+
|   4  GeForce GTX 1070    Off  | 00000000:05:00.0 Off |                  N/A |
|  0%   45C    P2    85W / 151W |   1315MiB /  8119MiB |     87%      Default |
+-------------------------------+----------------------+----------------------+
|   5  GeForce GTX 106...  Off  | 00000000:09:00.0 Off |                  N/A |
|100%   57C    P2    84W / 120W |   1292MiB /  3019MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
|   6  GeForce GTX 106...  Off  | 00000000:0A:00.0 Off |                  N/A |
| 48%   65C    P2    60W / 120W |   1045MiB /  3019MiB |     52%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1222      G   /usr/lib/xorg/Xorg                            86MiB |
|    0      1685      G   /usr/bin/gnome-shell                          97MiB |
|    0     12951      C   ...x41p_V0.98b1_x86_64-pc-linux-gnu_cuda90  1303MiB |
|    1      1222      G   /usr/lib/xorg/Xorg                             6MiB |
|    1     12922      C   ...x41p_V0.98b1_x86_64-pc-linux-gnu_cuda90  1281MiB |
|    2      1222      G   /usr/lib/xorg/Xorg                             6MiB |
|    2     12917      C   ...x41p_V0.98b1_x86_64-pc-linux-gnu_cuda90  1273MiB |
|    3      1222      G   /usr/lib/xorg/Xorg                             6MiB |
|    3     12944      C   ...x41p_V0.98b1_x86_64-pc-linux-gnu_cuda90  1273MiB |
|    4     12958      C   ...x41p_V0.98b1_x86_64-pc-linux-gnu_cuda90  1303MiB |
|    5      1222      G   /usr/lib/xorg/Xorg                             6MiB |
|    5     12932      C   ...x41p_V0.98b1_x86_64-pc-linux-gnu_cuda90  1273MiB |
|    6     12970      C   ...x41p_V0.98b1_x86_64-pc-linux-gnu_cuda90  1033MiB |
+-----------------------------------------------------------------------------+

The xorg.conf looks fine and the monitor works fine but I cannot access the #4 and #6 fans as shown above.

I also noticed that the buss IDs vary 1..4 then jump to 9 and A.

I changed my loop NumGPU to 7 and also tried 15 but same problem.

I also noticed that at least one of the GPUs on the splitter has working fan control.

I cant tell if the 2nd one has or not as I cant make sense out of the buss ID numbers.

Before putting the splitter in, the IDs were "0:01:0 .. 0:06:0" which makes sense.

Maybe 18.04 and the NVIDIA 390 driver cannot cope with splitters?

When the ID was 1..6 fans were 0..5 but how do I handle the 9 and the "A"?

  • I have no idea about anything related to multiple GPUs, that is not my domain, however A seems to indicate a hexadecimal value, in which case iff the IDs count from 1 (this is unusual) and the fans count from 0 (this makes more sense), then ID A would correspond to fan 9 and ID 9 would correspond to fan 8. I hope I'm being relevant and helpful – tatsu Jul 26 at 15:21
  • Yea, I tried using 15 to get the loop to go 0..14. I assume the argument to nvidia-settings in decimal. Even if it was in hex the '9" decimal should have worked as that is 9 in hex. I am going to try A, B , etc see if that works. – Joseph Stateson Jul 26 at 15:42
  • Thanks to whoever re-wrote my original post, double spacing and explaining more. I have had a post re-written by a moderator before but received a tongue lashing as I had not yet figured out how the formatting codes worked. – Joseph Stateson Jul 27 at 1:39
0

This seems to be a timing problem. Probably cause by different GPUs and behavior of the 4-in=1 PCIe multipler and the order in which the GPUs were arranged and possibly the vendor (pny, evga, msi).

I swapped the gtx1070 boards as the board that had a single fan was not spinning its fan fast enough and it sat in a rack where it did not not get a lot of air. The other gtx1070 had a pair of fans and was more exposed to the large cooling fans so if it ran slow it would still run cool. After swapping USB3 cables (all are on risers) I discovered that ALL had coolbits enabled. Every GPU had a temp slider, but only GPU #5 was not running at %100. I brought up nvidia-settings and it had a slider (as all now did) and I set it to %100. I did not have to run that script a second time.

So my conclusion is that the 4-in-1 worked OK but subtle hardware differences caused the coolbits mechanism to to not get full control. ie: could be more robust.

[EDIT] Found info on bus ID and how to use coolbits here wiki archlinux

hope this helps someone else

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.