I am trying to setup a Ceph cluster on a bunch of Raspberry Pi 4 machines (8Gb memory).

Unfortunately, during the installation of the cluster, I get illegal instructions:

/usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid c339214
4-7d0d-4380-b714-ba92ef198298 --setuser ceph --setgroup ceph

*** Caught signal (Illegal instruction) **
in thread ffffae8b8930 thread_name:bstore_kv_sync
ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable)
1: __kernel_rt_sigreturn()
2: (crc32c_arm64(unsigned int, unsigned char const*, unsigned int)+0x44c) [0xaaaad4a11c1c]
3: (rocksdb::log::Writer::EmitPhysicalRecord(rocksdb::log::RecordType, char const*, unsigned long)+0x60) [0xaaaad48f47a0]
4: (rocksdb::log::Writer::AddRecord(rocksdb::Slice const&)+0x98) [0xaaaad48f4938]
5: (rocksdb::DBImpl::WriteToWAL(rocksdb::WriteBatch const&, rocksdb::log::Writer*, unsigned long*, unsigned long*)+0x104) [0xaaaad488aee8]
6: (rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned long)+0xe0) [0xaaaad488c104]
7: (rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb::PreReleaseCallback*)+0x1860) [0xaaaad4891f34]
8: (rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x44) [0xaaaad4892284]
9: (RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x8c) [0xaaaad482c9fc]
10: (RocksDBStore::submit_transaction(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x78) [0xaaaad482d4e8]
11: (BlueStore::_txc_apply_kv(BlueStore::TransContext*, bool)+0x3e0) [0xaaaad4332ee0]
12: (BlueStore::_kv_sync_thread()+0xb94) [0xaaaad4354604]
13: (BlueStore::KVSyncThread::entry()+0x10) [0xaaaad437bc30]
14: /lib/aarch64-linux-gnu/libpthread.so.0(+0x7e84) [0xffffb8606e84]
15: /lib/aarch64-linux-gnu/libc.so.6(+0xd6d1c) [0xffffb8558d1c]

What could be causing this? I would expect that the Ubuntu version I downloaded from https://ubuntu.com/download/raspberry-pi would be compiled for use on the Raspberry Pi 4 (seeing that it has a "Works on:" banner below it showing the Raspberry Pi 4. I did not expect binaries with instructions that are not available on the Pi4.

  • The official ubuntu image should work with Raspberry Pi 4 (I use the 20.04 version on my own pi4). Given the specific failing call looks architecture specific perhaps there is an issue in the porting layer of the package istelf? is Ceph known to be stable on arm64 (eg debian - or raspiOS)? Alternatively is the illegal instruction signal "precise" - ie is that the code that caused the failure, or just the code that recieved the signal (eg due to signal masking). The simplest way to get an illegal instruction is a programming error causing one to jump through a bad pointer to function.
    – Greg
    May 25 at 11:42

It appears to be a known (and solved bug):


So I'll just have to wait until 16.2.5 is released which should solve my problem.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.