Phoronix Test Suite
This is an open source test suite licensed under the GPLv3:
Phoronix automates and standardizes the benchmarking of several real world use cases like compression, encryption and databases. Most tests benchmark open source software projects, but some also benchmark closed source software. They also host test results at: https://openbenchmarking.org which anyone can upload to, so you can compare test results with other different systems.
Since 18.04 there isn't a package on the official repos for it, so you just have to download it from https://www.phoronix-test-suite.com/?k=downloads Tested on Ubuntu 23.10:
sudo apt-get install php-cli php-xml
wget https://phoronix-test-suite.com/releases/phoronix-test-suite-10.8.4.tar.gz
tar xfz phoronix-test-suite-10.8.4.tar.gz
export PATH="$PATH:$(pwd)/phoronix-test-suite"
PHP is needed because it is actually the main scripting language of the suite. These were different times altogether!
Until 18.04 there was a package for it, tested on Ubuntu 16.10:
sudo apt-get install phoronix-test-suite
Tested on version 10.8.4, we can list all tests with:
phoronix-test-suite list-all-tests
The first few lines are:
pts/3dmark 3DMark Wild Life Extreme Graphics
pts/ai-benchmark AI Benchmark Alpha System
pts/aircrack-ng Aircrack-ng Processor
The last column categorizes the main thing being benchmarked, some possible values are:
- Processor: CPU
- Graphics: GPU
- Memory: RAM
- Network
- Disk
- System: likely mean a mixture of multiple above categories
Let's try a self explanatory processor bound one which basically benchmarks GCC:
phoronix-test-suite install pts/build-linux-kernel
phoronix-test-suite run pts/build-linux-kernel
First it asks some annoying configuration questions, first which kernel configuration we want to build, I select 1 for defconfig
:
Timed Linux Kernel Compilation 6.1:
pts/build-linux-kernel-1.15.0
Processor Test Configuration
1: defconfig [Default Kernel Build]
2: allmodconfig [This option is *much* more time consuming...]
3: Test All Options
** Multiple items can be selected, delimit by a comma. **
Build: 1
then:
Would you like to save these test results (Y/n):
Enter a name for the result file:
Enter a unique name to describe this test run / configuration:
New Description:
and then it proceeds to build the Linux kernel a bunch of times.
How to automate answering those questions was asked at: https://stackoverflow.com/questions/28581447/how-to-skip-phoronix-test-suite-inital-questions It is quite annoying really.
Running the test so many times is a bit excessive for a casual run, so let's limit it to a single run with FORCE_TIMES_TO_RUN=1
:
FORCE_TIMES_TO_RUN=1 phoronix-test-suite run pts/build-linux-kernel
After dumping my system's configuration, at the end we see:
Timed Linux Kernel Compilation 6.1:
pts/build-linux-kernel-1.15.0 [Build: defconfig]
Test 1 of 1
Estimated Trial Run Count: 1
Estimated Time To Completion: 9 Minutes [04:37 UTC]
Running Pre-Test Script @ 04:29:14
Started Run 1 @ 04:29:27
Running Post-Test Script @ 04:32:23
Build: defconfig:
173.449
Average: 173.449 Seconds
Samples: 1
so the test result was how long it took to build the kernel, which was 173.449 seconds on my system.
We can see how that compares with other systems at: https://openbenchmarking.org/test/pts/build-linux-kernel I'm running an AMD Ryzen 7 PRO 7840U on a Lenovo ThinkPad P14s, and for that CPU the public results were 129 +/- 9. So sadface, there seems to be something wrong with my system, as I'm considerably slower than those tests, maybe a performance mode issue? Setting to High Performance But at least this illustrates the awesome value of having public results available!
By tweaking "Analyze Test Configuration" we can also see:
Let's try another benchmark, how about stress-ng which was mentioned in another answer with configuration 1: CPU Stress
printf 1 | FORCE_TIMES_TO_RUN=1 phoronix-test-suite batch-run pts/stress-ng
and this time we see:
Stress-NG 0.16.04:
pts/stress-ng-1.11.0 [Test: CPU Stress]
Test 1 of 1
Estimated Trial Run Count: 1
Estimated Time To Completion: 1 Minute [12:15 UTC]
Started Run 1 @ 12:14:56
Test: CPU Stress:
22191.17
Average: 22191.17 Bogo Ops/s
Samples: 1
where the test results is 22191.17 Bogo Ops/s
, which is how stress-ng reports its output, and means just how many operations it did in a given amount of time. So we see that this is a different type of test, which rather than benchmarking the time to completion of a task rather set a timer and ran as many times as possible.
While it runs, I try to observe the final commands it is running with:
ciro 2117375 0.0 0.0 2780 1920 pts/11 S+ 12:20 0:00 /bin/sh /home/ciro/down/phoronix-test-suite/phoronix-test-suite batch-run pts/stress-ng
ciro 2117388 0.0 0.0 2780 1792 pts/11 S+ 12:20 0:00 sh -c php /home/ciro/down/phoronix-test-suite/pts-core/phoronix-test-suite.php batch-run pts/stress-ng
ciro 2117673 0.0 0.0 2784 1792 pts/11 S+ 12:20 0:00 /bin/sh ./stress-ng -t 30 --metrics-brief --cpu -1 --cpu-method all --no-rand-seed
ciro 2117674 0.0 0.0 39132 6528 pts/11 SL+ 12:20 0:00 ./stress-ng -t 30 --metrics-brief --cpu -1 --cpu-method all --no-rand-seed
...
A bit more digging shows that the shell script is something from Phoronix: ~/.phoronix-test-suite/installed-tests/pts/stress-ng-1.11.0
which simply does:
#!/bin/sh
cd stress-ng-0.16.04
./stress-ng $@ > $LOG_FILE 2>&1
echo $? > ~/test-exit-status
where stress-ng
is the final executable.
Test suites
Besides individual tests, Phoronix Test Suite also has the concept of test suites, which are just groups of tests. TODO how to list all test suites? I tried:
phoronix-test-suite list-available-suites
but it does not contain all suites. Notably, it is missing pts/cpu
, which contains several CPU benchmarks. Note that this will be slow and download a bunch of test executables
Running a suite is similar to running a single test:
phoronix-test-suite install pts/cpu
phoronix-test-suite run pts/cpu
We can see which tests are contained in a suite with:
phoronix-test-suite info pts/cpu
which contains a list such as:
Contained Tests:
Rodinia Test: OpenMP CFD Solver
Rodinia Test: OpenMP LavaMD
NAMD
Stockfish
x264 Video Input: Bosphorus 1080p
x264 Video Input: Bosphorus 4K
so we quickly understand that is it picking certain configs of certain tests. TODO: is there a way to list the actual test identifiers, e.g. pts/rodinia
and so on? One way is to look at: https://openbenchmarking.org/suite/pts/cpu
Another one of interest for CPU benchmarking which I found by randomly doing find
is:
phoronix-test-suite info pts/cpu-massive
which contains a whopping 373 tests.
Upload a test result
First we create an account: https://openbenchmarking.org/register
TODO haven't managed to login yet. Confirmation email didn't arrive? I'll try again later some day.
File structure
Finally, let's have a look at the file that Phoronix keeps around:
ncdu ~/.phoronix-test-suite/
shows:
167.1 MiB [###########################] /installed-tests
44.1 MiB [####### ] /test-profiles
2.9 MiB [ ] /test-results
1.1 MiB [ ] /openbenchmarking.org
636.0 KiB [ ] /test-suites
12.0 KiB [ ] core.pt2so
e 4.0 KiB [ ] /modules-data
e 4.0 KiB [ ] /modules
e 4.0 KiB [ ] /download-cache
4.0 KiB [ ] user-config.xml
4.0 KiB [ ] graph-config.json
4.0 KiB [ ] phoronix-test-suite-dependencies.log
0.0 B [ ] phoronix-test-suite-benchmark.log
installed-tests
contains source code and prebuilts, e.g. the Linuxk kernel build only has the source code txz:
--- /home/ciro/.phoronix-test-suite/installed-tests/pts/build-linux-kernel-1.15.0 -------------
/..
128.5 MiB [###########################] linux-6.1.tar.xz
The build dir was either deleted, or lies outside of of ~/.phoronix-test-suite
.
stress-ng
has the packed source, the unpacked source, and the built executable:
--- ~/.phoronix-test-suite/installed-tests/pts/stress-ng-1.11.0 -
/..
34.8 MiB [###########################] /stress-ng-0.16.04
3.7 MiB [## ] stress-ng-0.16.04.tar.gz
Another interesting directory is ~/.phoronix-test-suite/test-results
which stores run results under:
Enter a name for the result file:
if you entered that interactively
- a datetime such as
2023-12-23-0414
when running with batch-run