24

I just realized I am able to move a running active program to a different directory. In my experience that was not possible in MacOs or Windows. How does it work in Ubuntu?

Edit: I thought it was not possible on Mac but apparently it's possible as comments verify. It is only not possible on Windows maybe.Thanks for all the answers.

5
  • 2
    Pretty much a cross-site dupe: stackoverflow.com/a/196910/1394393. – jpmc26 Aug 19 '16 at 23:56
  • 1
    You can't rename(2) a running executable on OS X? What happens, do you get EBUSY or something? Why doesn't it work? The rename(2) man page doesn't document ETXTBUSY for that system call, and only talks about EBUSY being possible for directory renames, so I didn't know a POSIX system even could disallow renaming executables. – Peter Cordes Aug 20 '16 at 6:57
  • 3
    macOS apps can also be moved whilst they run, just not trashed. I presume some apps may err after that, e.g., if they store file URLs to their binary or bundled resources somewhere as a variable instead of generating such URL via NSBundle et al. I suspect it's macOS' POSIX compliance. – Constantino Tsarouhas Aug 20 '16 at 18:31
  • 1
    It actually works as Linux intends, you should know what you are doing. :P – userDepth Aug 21 '16 at 9:35
  • 2
    I guess another way to think about it is, why wouldn't it be possible? Just because Windows doesn't let you, doesn't necessarily mean it's fundamentally not possible because of how processes work or something. – Thomas Aug 21 '16 at 11:33
32

Let me break it down.

When you run an executable, a sequence of system calls are executed, most notably fork() and execve():

  • fork() creates a child process of the calling process, which is (mostly) an exact copy of the parent, both still running the same executable (using copy-on-write memory pages, so it's efficient). It returns twice: In the parent, it returns the child PID. In the child, it returns 0. Normally, the child process calls execve right away:

  • execve() takes a full path to the executable as an argument and replaces the calling process with the executable. At this point the newly created process gets its own virtual address space i.e. virtual memory, and execution begins at its entry point (in a state specified by the platform ABI's rules for fresh processes).

At this point, the kernel's ELF loader has mapped the text and data segments of the executable into memory, as if it had used the mmap() system call (with shared read-only and private read-write mappings respectively). The BSS is also mapped as if with MAP_ANONYMOUS. (BTW, I'm ignoring dynamic linking here for simplicity: The dynamic linker open()s and mmap()s all the dynamic libraries before jumping to the main executable's entry point.)

Only a few pages are actually loaded into memory from disk before a newly-exec()ed starts running its own code. Further pages are demand paged in as needed, if/when the process touches those parts of its virtual address space. (Pre-loading any pages of code or data before starting to execute user-space code is just a performance optimization.)


The executable file is identified by the inode on the lower level. After the file has started to be executed, the kernel keeps the file content intact by the inode reference, not by file name, like for open file descriptors or file-backed memory mappings. So you can easily move the executable to another location of the filesystem or even on a different filesystem. As a side note, to check process's various stat you can peek into the /proc/PID (PID is the Process ID of the given process) directory. You can even open the executable file as /proc/PID/exe, even it's been unlinked from disk.


Now let's dig down the moving:

When you move a file within a same filesystem, the system call that is executed is rename(), which just renames the file to another name, the file's inode remain the same.

Whereas between two different filesystems, two things happen:

  • The content of the file in copied first to the new location, by read() and write()

  • After that, the file is unlinked from the source directory using unlink() and obviously the file will get a new inode on the new filesystem.

rm is actually just unlink()-ing the given file from the directory tree, so having the write permission on the directory will get you sufficient right to remove any file from that directory.

Now for fun, imagine what happens when you are moving files between two filesytems and you do not have permission to unlink() the file from source?

Well, the file will be copied to the destination at first (read(), write()) and then unlink() will fail due to insufficient permission. So, the file will remain in both filesystems!!

11
  • 5
    Your are somewhat confusing virtual and physical memory. Your description of the way the program is loaded to physical memory is inaccurate. The exec system call doesn't at all copy the various sections of an executable to physical memory but only load the one it needs to start the process. Afterwhile, required pages are loaded on demand, possibly a long time later. The executable file bytes are part of the process virtual memory and might be read, and possibly read again during the whole life of the process. – jlliagre Aug 19 '16 at 23:26
  • @jlliagre Edited, I hope it's clarified now. Thanks. – heemayl Aug 20 '16 at 1:50
  • 6
    "The process is not using the file system anymore" statement is still questionable. – jlliagre Aug 20 '16 at 8:52
  • 2
    The basic understanding that a given file in the file system is not directly identified by the file name should be much clearer. – Thorbjørn Ravn Andersen Aug 20 '16 at 11:38
  • 2
    The are still inaccuracies in your update. The mmap and unmap system calls are not used to load and unload the pages on demand, pages are loaded by the kernel when accessing them generate a page fault, pages are unloaded from memory when the OS feels the RAM would be better used for something else. No system call is involved in these load/unload operations. – jlliagre Aug 20 '16 at 21:32
14

Well, that is pretty straighforward. Let's take an executable named /usr/local/bin/whoopdeedoo. That is only a reference to so called inode (basic structure of files on Unix Filesystems). It's the inode that gets marked "in use".

Now when you delete or move the file /usr/local/whoopdeedoo, the only thing that is moved (or wiped) is the reference to the inode. The inode itself remains unchanged. That's basically it.

I should verify it, but I believe you can do this on Mac OS X filesystems too.

Windows takes a different approach. Why? Who knows...? I am not familiar with the internals of NTFS. Theoretically, all filesystems that use references to intenal structures for filesnames should be able to do this.

I admit, I overly simplified, but go read the section "Implications" on Wikipedia, which does a much better job than me.

3
  • 1
    Well if you use a shortcut in Windows to start the executable, you can wipe the shortcut too, if you want to compare it like that, maybe? =3 – Ray Aug 20 '16 at 13:01
  • 2
    No, that would be like wiping a symbolic link. Somewhere in other comments, it is stated that the behaviour is due to legacy support with FAT file systems. That sounds like a probable reason. – jawtheshark Aug 20 '16 at 13:54
  • 1
    This doesn't have anything to do specifically with inodes. NTFS uses MFT records to track file state, and FAT uses directory entries for this, but Linux still works the same way with these filesystems — from user's point of view. – Ruslan Aug 20 '16 at 18:24
13

One thing that seems missing from all other answers is that: once a file is opened and a program holds an open file descriptor the file will not be removed from the system until that file descriptor is closed.

Attempts to delete the referenced inode will be delayed until the file is closed: renaming in the same or different file system cannot affect the open file, independently of the behaviour of the rename, nor explicitly deleting or overwriting the file with a new one. The only way in which you can mess a file up is by explicitly opening its inode and mess with the contents, not by operations on the directory such as renaming/deleting the file.

Moreover when the kernel executes a file it keeps a reference to the executable file and this will again prevent any modification of it during execution.

So in the end even if it looks like that you are able to delete/move the files that make up a running program, actually the contents of those files are kept in memory until the program ends.

7
  • 1
    This is not right. execve() does not return any FD, it simply executes the program. So for example, if you run tail -f /foo.log then their is a FD (/proc/PID/fd/<fd_num>) associated with tail for the foo.log but not for the executable itself, tail, not on its parent as well. This is true for the single executables too. – heemayl Aug 19 '16 at 20:17
  • @heemayl I didn't mention execve so I don't see how this is relevant. Once the kernel starts executing a file, trying to replace the file will not modify the program the kernel is going to load rendering the point moot. If you want to "update" the executable while it is running you could call execve at some point to make the kernel re-read the file, but I don't see how this matters. The point is: deleting a "running executable" doesn't really trigger any data deletion until the executable stops. – Bakuriu Aug 19 '16 at 20:26
  • I am talking about this part if the program consists of a single executable file once you start execution the program will run fine independently of any change in the directory: renaming in the same or different file system cannot affect the open handler , you are necessarily talking about execve() and a FD when there is not FD involved in this case. – heemayl Aug 19 '16 at 20:33
  • 2
    You don't need a file handle in order to have a reference to the file -- having pages mapped is also sufficient. – Simon Richter Aug 19 '16 at 23:15
  • 1
    Unix doesn't have "file handles". open() returns a file descriptor, which is heemayl's talking about here with execve(). Yes, a running process has a reference to its executable, but that isn't a file descriptor. Probably even if it munmap()ed all its mappings of its executable, it would still have a reference (reflected in /proc/self/exe) that stopped the inode from being freed. (This would be possible without crashing if it did this from a library function that never returned.) BTW, truncating or modifying an in-use executable might give you ETXTBUSY, but might work. – Peter Cordes Aug 21 '16 at 1:54
7

In a Linux filesystem, when you move a file, so long as it doesn't cross filesystem boundaries (read: stays on the same disk/partition) all you are changing is the inode of .. (parent directory) to that of the new location. The actual data hasn't moved at all on the disk, just the pointer so that the filesystem knows where to find it.

This is why move operations are so quick and likely why there's no issue moving a running program as you aren't actually moving the program itself.

1
  • Your answer seems to imply moving a binary executable to another file system would impact running processes launched from that binary. – jlliagre Aug 20 '16 at 21:36
6

It is possible because moving a program doesn't affect running processes started by launching it.

Once a program is launched, its on-disk bits are protected against being overwritten but there is no need to protect the file to be renamed, moved to a different location on the same file system, which is equivalent to rename the file, or moved to a different file system, which is equivalent to copy the file elsewhere then remove it.

Removing a file that is in use, either because a process has a file descriptor open on it, or because a process is executing it, doesn't remove the file data, which stays referenced by the file inode but only removes the directory entry, i.e. a path from which the inode can be reached.

Note that launching a program doesn't load everything at once in (physical) memory. On the opposite, only the strict minimum required for the process to start is loaded. Then, required pages are loaded on demand during the whole life of the process. this is called demand paging. If there is RAM shortage, the OS is free to release the RAM holding these pages so it is well possible for a process to load multiple times the very same page from the executable inode.

The reason why it was not possible with Windows is originally likely due to the fact the underlying file system (FAT) wasn't supporting the split concept of directory entries vs inodes. This limitation was no more present with NTFS but the OS design has been kept for a long time, leading to the obnoxious constraint to have to reboot when installing a new version of a binary, which is no more the case with recent versions of Windows.

6
  • 1
    I believe newer versions of Windows can replace binaries in use without rebooting. – Thorbjørn Ravn Andersen Aug 20 '16 at 11:39
  • @ThorbjørnRavnAndersen I wonder why all updates still require restart :( – Braiam Aug 20 '16 at 19:06
  • 1
    @Braiam They don't. Have a closer look. Even though binaries can be updated the kernel cannot (to my knowledge) and requires a reboot to be replaced with a newer version. This is valid for most operating system kernels. Smarter people than I have written kpatch for Linux which can patch a Linux kernel while running - see en.wikipedia.org/wiki/Kpatch – Thorbjørn Ravn Andersen Aug 20 '16 at 22:19
  • @ThorbjørnRavnAndersen I meant "all Windows updates" – Braiam Aug 20 '16 at 22:23
  • @Braiam yes - so did I. Please have a closer look. – Thorbjørn Ravn Andersen Aug 20 '16 at 22:29
4

Basically, in Unix and its ilk, a file name (including the directory path leading to it) is used for associating/finding a file when opening it (executing a file is one way of opening it in a manner). After that moment, the identity of the file (via its "inode") is established and no longer questioned. You can remove the file, rename it, change its permissions. As long as any process or a file path has a handle on that file/inode, it will stick around, just like a pipe between processes does (actually, in historic UNIX a pipe was a nameless inode with a size that just fitted in the "direct blocks" disk storage reference in the inode, something like 10 blocks).

If you have a PDF viewer open on a PDF file, you can delete that file and open a new one with the same name, and as long as the old viewer is open it will still be fine accessing the old file (unless it actively watches the file system in order to notice when the file disappears under its original name).

Programs needing temporary files can just open such a file under some name and then immediately remove it (or rather its directory entry) while it is still open. Afterwards the file is no longer accessible by name, but any processes having an open file descriptor to the file can still access it, and if there is an unexpected exit of the program afterwards, the file will get removed and the storage reclaimed automatically.

So the path to a file is not a property of the file itself (in fact, hard links can provide several different such paths) and is needed only for opening it, not for continued access by processes already having it open.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.