3

I was running mysqld on 14.04 with a non-standard --datadir (/usr/local/mysqld-test) & port (33306). After upgrading to 16.04, I was no longer able to start up my test mysqld, so, following the instructions here and in the bug it links to here, I wound up going as far as:

sudo apt purge mariadb-client-10.0 mariadb-common mysql-client mysql-client-5.7 mysql-client-core-5.7 mysql-common dbconfig-mysql
sudo apt-get install mysql-client mysql-server

(In the process, it uninstalled amarok!? snarl)

That still wasn't working, so I wound up nuking my entire /usr/local/mysqld-test (intentionally; in theory I have scripts to rebuild it). The mysql_install_db command I had used before is deprecated now, and doesn't seem to understand the -P 33306 argument, so instead I'm trying to use mysqld --initialize-insecure.

Here's what I'm doing:

mysqld --initialize-insecure --basedir=/usr/local/mysqld-test --datadir=/usr/local/mysqld-test/data

It fails:

mysqld: Can't create directory '/usr/local/mysqld-test/data/' (Errcode: 13 - Permission denied)
2016-11-02T10:21:37.547443Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-11-02T10:21:37.549183Z 0 [ERROR] Aborting

Huh. I thought I made that world-writable:

$ ls -al /usr/local/mysqld-test
total 8
drwxrwxrwx  2 kuhrusty kuhrusty 4096 Nov  2 03:13 .
drwxr-xr-x 20 root     root     4096 Nov  2 03:13 ..

WTF? Let's try that with strace:

execve("/usr/sbin/mysqld", ["mysqld", "--initialize-insecure", "--basedir=/usr/local/mysqld-test", "--datadir=/usr/local/mysqld-test"...], [/* 69 vars */]) = 0
...

It mmaps a bunch of shared libraries, sets some signal handlers, and then looks for config files (which I don't think I've messed with--I start it up manually, passing --datadir, -P, etc.):

stat("/etc/my.cnf", 0x7ffe13e99af0)     = -1 ENOENT (No such file or directory)
stat("/etc/mysql/my.cnf", {st_mode=S_IFREG|0644, st_size=682, ...}) = 0
open("/etc/mysql/my.cnf", O_RDONLY)     = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=682, ...}) = 0
read(3, "#\n# The MySQL database server co"..., 4096) = 682
open("/etc/mysql/conf.d/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 4
fstat(4, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
brk(0x271d000)                          = 0x271d000
getdents(4, /* 4 entries */, 32768)     = 120
getdents(4, /* 0 entries */, 32768)     = 0
close(4)                                = 0
stat("/etc/mysql/conf.d/mysql.cnf", {st_mode=S_IFREG|0644, st_size=8, ...}) = 0
open("/etc/mysql/conf.d/mysql.cnf", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=8, ...}) = 0
read(4, "[mysql]\n", 4096)              = 8
read(4, "", 4096)                       = 0
close(4)                                = 0
stat("/etc/mysql/conf.d/mysqldump.cnf", {st_mode=S_IFREG|0644, st_size=55, ...}) = 0
open("/etc/mysql/conf.d/mysqldump.cnf", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=55, ...}) = 0
read(4, "[mysqldump]\nquick\nquote-names\nma"..., 4096) = 55
read(4, "", 4096)                       = 0
close(4)                                = 0
brk(0x2713000)                          = 0x2713000
brk(0x270a000)                          = 0x270a000
open("/etc/mysql/mysql.conf.d/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 4
fstat(4, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
getdents(4, /* 4 entries */, 32768)     = 128
getdents(4, /* 0 entries */, 32768)     = 0
close(4)                                = 0
stat("/etc/mysql/mysql.conf.d/mysqld.cnf", {st_mode=S_IFREG|0644, st_size=3028, ...}) = 0
open("/etc/mysql/mysql.conf.d/mysqld.cnf", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=3028, ...}) = 0
read(4, "#\n# The MySQL database server co"..., 4096) = 3028
read(4, "", 4096)                       = 0
close(4)                                = 0
stat("/etc/mysql/mysql.conf.d/mysqld_safe_syslog.cnf", {st_mode=S_IFREG|0644, st_size=21, ...}) = 0
open("/etc/mysql/mysql.conf.d/mysqld_safe_syslog.cnf", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=21, ...}) = 0
read(4, "[mysqld_safe]\nsyslog\n", 4096) = 21
read(4, "", 4096)                       = 0
close(4)                                = 0
read(3, "", 4096)                       = 0
close(3)                                = 0
stat("/home/kuhrusty/.my.cnf", 0x7ffe13e99af0) = -1 ENOENT (No such file or directory)
stat("/home/kuhrusty/.mylogin.cnf", 0x7ffe13e99af0) = -1 ENOENT (No such file or directory)
getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=64*1024}) = 0
setrlimit(RLIMIT_NOFILE, {rlim_cur=5000, rlim_max=5000}) = 0
getrlimit(RLIMIT_NOFILE, {rlim_cur=5000, rlim_max=5000}) = 0
umask(026)                              = 02
open("/etc/localtime", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=2845, ...}) = 0
fstat(3, {st_mode=S_IFREG|0644, st_size=2845, ...}) = 0
read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\5\0\0\0\5\0\0\0\0"..., 4096) = 2845
lseek(3, -1811, SEEK_CUR)               = 1034
read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\5\0\0\0\5\0\0\0\0"..., 4096) = 1811
close(3)                                = 0
uname({sysname="Linux", nodename="tentacle", ...}) = 0
brk(0x2732000)                          = 0x2732000
brk(0x2724000)                          = 0x2724000
lstat("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/usr/local", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/usr/local/mysqld-test", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
lstat("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/usr/local", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/usr/local/mysqld-test", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
lstat("/usr/local/mysqld-test/data", 0x7ffe13e9d1d0) = -1 ENOENT (No such file or directory)
open("/usr/share/mysql/english/errmsg.sys", O_RDONLY) = 3
read(3, "\376\376\3\1\1\0\335\26\1\0008\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10\0", 32) = 32
read(3, "\0\0\0\0\10\0\0\0\20\0\0\0\23\0\0\0\27\0\0\0D\0\0\0m\0\0\0\231\0\0\0"..., 4320) = 4320
read(3, "hashchk\0isamchk\0NO\0YES\0Can't cre"..., 71389) = 71389
close(3)                                = 0
stat("/usr/local/mysqld-test/share/mysql/charsets/Index.xml", 0x7ffe13e9e2e0) = -1 ENOENT (No such file or directory)
futex(0x1ddcf48, FUTEX_WAKE_PRIVATE, 2147483647) = 0
brk(0x2745000)                          = 0x2745000
open("/usr/local/mysqld-test/data/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
mkdir("/usr/local/mysqld-test/data/", 0750) = -1 EACCES (Permission denied)

OK, now, I think that's where things go wrong (note: make sure you scroll to the bottom)... but why? Anyone can create a file there!

write(2, "mysqld: ", 8)                 = 8
write(2, "Can't create directory '/usr/loc"..., 87) = 87
write(2, "\n", 1)                       = 1
mysqld: Can't create directory '/usr/local/mysqld-test/data/' (Errcode: 13 - Permission denied)

WTF?? With this problem, the 14.04 -> 16.04 upgrade now ranks as about the third biggest mistake of my life... I feel like I've suffered one of those traumatic brain injuries where you forget how to read. Those permissions are right, right?? Running mysqld --initialize under sudo doesn't make a difference. Is there some incredibly stupid thing I'm doing wrong?

3

ARGHH AppArmor is what hosed me. The second answer here is what gave me the hint I needed to look in the log:

[137519.814104] audit: type=1400 audit(1478218238.219:177): apparmor="DENIED" operation="mkdir" profile="/usr/sbin/mysqld" name="/usr/local/mysqld-test/data/" pid=7126 comm="mysqld" requested_mask="c" denied_mask="c" fsuid=1000 ouid=1000

Following these instructions, I disabled AppArmor for MySQL:

sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
cd /etc/apparmor.d/disable
sudo ln -s /etc/apparmor.d/usr.sbin.mysqld .

And then confirmed that mysqld was no longer in enforce mode:

sudo aa-status

And then mysqld --initialize-insecure started & ran it the way I expected it to work two days ago.

(How/why did AppArmor get installed/enabled during the 14.04 -> 16.04 upgrade? I don't think I'd heard of AppArmor before this; "your stuff is no longer going to work the way it used to" seems like the kind of information I should've been given during the upgrade!)

  • I see your problem. You're expecting Oracle MySQL to be a well-managed and sensible project. FYI, percona-server-server doesn't pull in AppArmor, and it doesn't have 5.7 perf regressions. – Paul Draper Sep 11 '17 at 20:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.