3

Related to another post where I was finally able to delete files within /tmp thanks to another poster.

However, in the end the sysadmin had to restore a snapshot of a few days ago to get the mysql service back up.

I deduced the mysql server was not running, and it wasn't. One error was that it couldn't write to the tmp dir to create the mysqld.sock/.pid files so they didn't exist.

So the server was restarted/rebooted and finally restored from a previous snapshot so all is now working but that snapshot was from a few days ago so the tell-tell signs are in this output:

drwxrwxrwt  2 root  root  34471936 Oct  8 20:47 tmp

That big number is still being 'remembered' somehow and I'm trying to find out what is writing to that folder? In Drupal, the path for preview files is set to that directory but I just did a series of tests (the site has no login accounts/just for data) downloading, previewing etc and the file number never changed on the /tmp folder so what is writing to that folder?

This is a web server which does not get shut down/rebooted unless there's some problem. Even so, the /tmp directory does not get emptied upon restart. Also, in the rsC file, the TMPTIME=0 is still at the default.

So, I'm not sure where else to look or how to investigate what is keeping that 3441936 size when the actual directory is empty? Or, how to search for what processes are using that directory.

This is just for a Drupal install. Ubuntu 12.

4
  • How are you determining that there are no files? are you checking for hidden (dot) files/folders as well? Have you tried the du command e.g. sudo du -bc /tmp Oct 16, 2013 at 22:04
  • just cd into it and then ll is shows 0. Also, after I was able to delete the files after using sudo su and rm filesA* for example, I'd recall ll and they would be gone. So no files show inside there. Trying your code above, I get du: cannot access /tmp/#sql_314_1.MYI': No such file or directory` and then 741247078 /tmp 741247078 total Oct 16, 2013 at 22:37
  • to show all hidden files, try: ls -a Oct 16, 2013 at 22:55
  • ahhh... crap.. they're all still there! I deleted about 100 of them but when I ll again, though they are not in the list, the 3441936 hasn't changed? Oct 16, 2013 at 22:57

2 Answers 2

6

Try this.

$ sudo lsof | egrep '^COMMAND|deleted'

Example output I see on my system...

COMMAND     PID        USER   FD      TYPE             DEVICE   SIZE/OFF       NODE NAME
mysqld     2842      mysql    5u      REG                9,1        4633    6209543 /tmp/ibLRNV1i (deleted)
mysqld     2842      mysql    6u      REG                9,1         424    6209545 /tmp/ibnnYl1y (deleted)
mysqld     2842      mysql    7u      REG                9,1           0    6209546 /tmp/ib5ViM0O (deleted)
mysqld     2842      mysql    8u      REG                9,1           0    6209547 /tmp/ibh9U55F (deleted)
mysqld     2842      mysql   13u      REG                9,1           0    6209548 /tmp/ibRzqtwm (deleted)
mysqld     2842      mysql  319u      REG                9,1    25894907    6209553 /tmp/MLFxqPDX (deleted)
mysqld     2842      mysql  350u      REG                9,1   267677827    6209550 /tmp/MLFBkHbP (deleted)

A common characteristic of many *nix filesystems is the fact that there's nothing preventing a process from continuing to read/write to/from a deleted file, and there's nothing preventing the deletion of a file that a process still has open.

One way some programs ensure that temporary files don't linger around (and possibly also as a security measure, though I'm unsure how "secure" this is) is to open them, and immediately "unlink" (delete) them. If the process terminates unexpectedly, the disk space is freed. MySQL does this:

MySQL arranges that temporary files are removed if mysqld is terminated. On platforms that support it (such as Unix), this is done by unlinking the file after opening it. The disadvantage of this is that the name does not appear in directory listings and you do not see a big temporary file that fills up the file system in which the temporary file directory is located. (In such cases, lsof +L1 may be helpful in identifying large files associated with mysqld.)

http://dev.mysql.com/doc/refman/5.6/en/temporary-files.html

Important Point: When a file that is still open is deleted, the space is not actually freed on the disk until the last process holding a handle on the file closes it. Until the file is closed, nobody, including root, can free that space other than by killing the process. Conversely, killing (or gracefully stopping, if possible) the process should always free the space.

So I would recommend you take a look at what your system may have in the way of open and deleted files and this may give you something to go on.

In the case of my system, MySQL does have a number of temporary files open, some of them fairly large.

If you want to peek inside those files, it's easily done, though the nature of what you'll be able to discern varies widely. Take the PID and the digits under FD that do this... example, PID 2842 FD 350:

$ sudo strings /proc/2842/fd/350 | less

If MySQL does have deleted temporary files open, stopping and restarting MySQL will free up that space, and then you'll need to investigate what might be keeping those open of they are lingering longer than appropriate and whether you have queries that are generating inordinately large temporary tables and exhausting your /tmp space. If your /tmp directory is its own partition/filesystem then df -h will give you a better answer regarding free and used space. You probably don't need to worry too much about the actual size of the directory entry itself.

And of course it might not be MySQL. :)

1
  • 1
    Precisely what I needed to know (well, with a lot more detail than I required, but I'd rather have too much than too little). Thanks! :)
    – rinogo
    Feb 3, 2017 at 1:35
0

ext[234] does not shrink the size of directories when you delete files; it just marks the entries as deleted so they can be reused later. If you really want to reclaim those 34 MB, you will need to delete and recreate the directory, or boot into rescue mode and run e2fsck -D on the volume.

1
  • I tried to delete, not permitted. I renamed it, created a new tmp and tried to delete but not permitted. I went sudo su and still couldn't delete. Oct 16, 2013 at 22:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .