0

We have a server with 24 custom NVMe devices connected ( the company i work is developing a custom ASIC for hardware video transcoding utilizing NVMe as a trasport) We are interacting with our NVMe devices using vendor specific commands in NVMe protocol utilizing linux IOCTL interface with standard inbox NVMe linux driver. We are testing this on both ubuntu 16.04 and 18.04 with various kernel but end up with the same issue no matter what we do. We have 24 devices connected over NVMe in a Gigabyte made storage server. When we run out tests, we are able to work at the desires transcoding FPS range with 9 of our devices at a time. We use FFMpeg with a custom AVCODEC plugin to run transcoding sessions. As soon as we try to utilize the 10th NVMe device, we see the performance drop on the other devices, which tells us that there is some kind of bottleneck in doing NVMe transfers. Our IO is very different from the normal SSD drive since we try to do as big of transfers as possible for both READ and WRITE operations (not NVMe read/write, but our read/write over vendor specific command interface)

we have tried to tune just about every kernel parameter there exists including using polling/hybrid polling/interrupts for nvme, changing max transfer sizes etc, nothing helps.

The server is more that capable of utilizing 10 or even 24 nvme drives , it was built for it and CPU usage is definately not a bottleneck. All our data transfers from user space to kernel space are Zero copy and 512 bytes aligned and this is confirmed by running perf on the instances.

If anyone had any experience tuning a big NVME storage system with more than 10 drives in Ubuntu (ubuntu is the primary linux distro for us atm) please share what else is it possible to do to get more than 10 devices running at max transfer rates.

0

It may be worth checking to see where the bottle neck is with perf.

Install with:

sudo apt-get install linux-tools

Record events, say for 20 seconds at 1000 samples per second:

sudo perf record -a -F 1000 sleep 20

..and see where the bottle neck is occurring:

sudo perf report

Refer to https://perf.wiki.kernel.org/index.php/Tutorial for more details on perf. A really useful article on monitoring memory bandwidth utilization with perf is https://yunmingzhang.wordpress.com/2015/07/22/measure-memory-bandwidth-using-uncore-counters/

Alternatively, this may be memory/cache bandwidth limited, in which case the pqos tool can be used on some modern processors to monitor the cache utilization:

sudo apt-get install intel-cmt-cat
sudo modprobe msr  
sudo pqos -r
  • Thanks for the suggestions, We have user perf and it doesn't show anything abnormal, most of the time is used by mapping user to kernel memory for transfers, will give the msr utility a try as well, but as i mentioned CPU usage is less than 30% per core so im thinking stil a lot of headroom – spectrality Aug 17 at 14:51
  • Just ran t he tools, can' figure out how to post here the output , get error saying im 4000 characters over the limit so here is the pastebin link pastebin.com/raw/BpSGPRkt – spectrality Aug 17 at 15:23
  • Just had a thought, it may be worth examining the interrupts that the devices are using, maybe the 10th NVMe device is sharing interrupts with another device and having contention. lspci -v will show the IRQs each device is using, and cat /proc/interrupts shows the interrupt rates – Colin Ian King Aug 17 at 21:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.