We have a server with 24 custom NVMe devices connected ( the company i work is developing a custom ASIC for hardware video transcoding utilizing NVMe as a trasport) We are interacting with our NVMe devices using vendor specific commands in NVMe protocol utilizing linux IOCTL interface with standard inbox NVMe linux driver. We are testing this on both ubuntu 16.04 and 18.04 with various kernel but end up with the same issue no matter what we do. We have 24 devices connected over NVMe in a Gigabyte made storage server. When we run out tests, we are able to work at the desires transcoding FPS range with 9 of our devices at a time. We use FFMpeg with a custom AVCODEC plugin to run transcoding sessions. As soon as we try to utilize the 10th NVMe device, we see the performance drop on the other devices, which tells us that there is some kind of bottleneck in doing NVMe transfers. Our IO is very different from the normal SSD drive since we try to do as big of transfers as possible for both READ and WRITE operations (not NVMe read/write, but our read/write over vendor specific command interface)
we have tried to tune just about every kernel parameter there exists including using polling/hybrid polling/interrupts for nvme, changing max transfer sizes etc, nothing helps.
The server is more that capable of utilizing 10 or even 24 nvme drives , it was built for it and CPU usage is definately not a bottleneck. All our data transfers from user space to kernel space are Zero copy and 512 bytes aligned and this is confirmed by running perf on the instances.
If anyone had any experience tuning a big NVME storage system with more than 10 drives in Ubuntu (ubuntu is the primary linux distro for us atm) please share what else is it possible to do to get more than 10 devices running at max transfer rates.