Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

After adding the preload package, my applications seem to speed up but if I copy a large file, the file cache grows by more than double the size of the file.

By transferring a single 3-4 GB virtualbox image or video file to an external drive, this huge cache seems to remove all the preloaded applications from memory, leading to increased load times and general performance drops.

Is there a way to copy large, multi-gigabyte files without caching them (i.e. bypassing the file cache)? Or a way to whitelist or blacklist specific folders from being cached?

share|improve this question
add comment

2 Answers

up vote 7 down vote accepted

There is the nocache utility, which can prepended to a command like ionice and nice. It works by preloading a library which adds posix_fadvise with the POSIX_FADV_DONTNEED flag to any open calls.

In simple terms, it advises the kernel that caching is not needed for that particular file; the kernel will then normally not cache the file. See here for the technical details.

It does wonders for any huge copy jobs, e. g. if you want to backup a multi terabyte disk in the background with the least possible impact on you running system, you can do something along nice -n19 ionice -c3 nocache cp -a /vol /vol2.

A package will be available in Ubuntu 13.10 and up. If you are on a previous release you can either install the 13.10 package or opt for this 12.04 backport by François Marier.

share|improve this answer
    
Are you the author? because nocache says something different: The nocache tool tries to minimize the effect an application has on the Linux file system cache. This is done by intercepting the open and close system calls and calling posix_fadvise with the POSIX_FADV_DONTNEED parameter. –  izx Jul 3 '12 at 22:53
    
No, I am not the author. I wrote this answer off my head, but I read the source a year ago and since then must have mixed up POSIX_FADV_DONTNEED and O_DIRECT . –  hirschhornsalz Jul 4 '12 at 8:48
    
Sorry about that, I was browsing with the "mobile view" and didn't notice the edit :). –  izx Jul 4 '12 at 17:13
    
I was hoping for something that could be done via the GUI, as well as a way to simply blacklist the 'no cache' folders, but this will have to do for now. –  Veazer Jul 8 '12 at 9:56
add comment

For single large files, use dd with direct I/O to bypass the file cache:

If you want to transfer one (or a few) large multi-gigabyte files, it's easy to do with dd:

dd if=/path/to/source of=/path/to/destination bs=4M iflag=direct oflag=direct
  • The direct flags tell dd to use the kernel's direct I/O option (O_DIRECT) while reading and writing, thus completely bypassing the file cache.
  • The bs blocksize option must be set to a reasonably large value since to minimize the number of physical disk operations dd must perform, since reads/writes are no longer cached and too many small direct operations can result in a serious slowdown.
    • Feel free to experiment with values from 1 to 32 MB; the setting above is 4 MB (4M).

For multiple/recursive directory copies, unfortunately, there are no easily available tools; the usual cp,etc do not support direct I/O.

/e iflags & oflags changed to the correct iflag & oflag

share|improve this answer
1  
Recursive could be done with zsh's ** operator. zsh needs to be installed manually from the repos. –  hexafraction Jul 4 '12 at 17:23
1  
Actually, no. dd's weird syntax fouls up the ** oprtator. You could still use a shell script that got arguments normally(dd.sh in.file out.file with ** in the filenames) and gave the filenames to dd using $1, $2, etc, which should not be fouled by dd's weird syntax. –  hexafraction Jul 5 '12 at 17:04
1  
Direct makes is very slow, since it AFAIK also disables the readahead caches, which is probably not what you want, and is not realistic in a benchmarking scenario either. Use "iflag=nocache oflag=nocache" instead, which exactly says to the OS that you don't need the in-file or the out-file cached. –  stolsvik Nov 18 '12 at 15:44
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.