Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

What would be the best way to list 100 first files in a directory sorted by the created timestamp (oldest first). The directory is quite large (around 100000 files).

ls piped to head takes an awful long time to complete.

Edit:

  • Filesystem is ext3.
  • limiting the number of files in the folder is not worth the effort, since this is going to be a rare "clean-up" operation and files are generated by a 3rd party software.
  • Using file modified time, instead of the creation time provides and acceptable solution.

share|improve this question
1  
If it's a clean-up operation, maybe what you want is just find -mtime +<number of days> -delete to clean up all files older than a certain age. That will mean no sort is necessary. –  Mikel Jan 28 '11 at 22:35
add comment

2 Answers

up vote 9 down vote accepted

You say that "ls piped to head takes an awful long time to complete".

The cause of this is not ls, but the number of files in your directory. If you have 100,000 files in a single directory, any way of solving this problem would have to get information about all 100,000 files before it could even think about sorting them or printing any output.

If it's taking too long, the real solution is to split the files over multiple directories.

If you can't spread the files over several directories, is there any way you can narrow down the number of files to consider? e.g. if the file names include a date, maybe you can include a wildcard so the system doesn't have to sort 100,000 files. Or maybe they're sequentially numbered? (This may or may not help, but is worth trying.)

How many times are you trying to do this? Maybe it's worth saving/caching the output for re-use.


Now, a question.

Are you sure you mean "creation time" and not "change time"? Most tools can only display "change time", not "creation time".

Getting "creation time" is a very new thing, which requires an ext4 filesystem, and some tools that aren't easy to install.


If you want change time

Change time (ctime for short) means the time the attributes of the file last changed.

ls -c sorts by ctime.

You want the output in ascending order, not descending, so you need to reverse the output with the -r option too.

So you could do it like this:

ls -cr | head -n 100

A longer solution to the same problem using stat:

find . -mindepth 1 -maxdepth 1 -exec stat -c $'%Z\t%n' '{}' \; |
    sort -k 1n |
    cut -f 2 -d $'\t' |
    head -n 10 |
    sed -e 's/^\.\///'

but that runs slower than ls -cr on my system.


If you want modification time

Modification time (mtime for short) means the time the contents of the file last changed.

ls -t sorts by mtime.

Change ls -cr to ls -tr (best option) or change stat -c $'%Z\t%n' to stat -c $'%Y\t%n'.


If you need creation time

(crtime for short)

This is harder.

First, ensure the directory is on a filesystem that is formatted using ext4. You can use tune2fs -l <device name> to check this.

Then, there is a new stat format called %W, which can help you here. To get it, you'll have to download a version of GNU Coreutils released in October 2010 or after, extract it, compile it, and install it.

Then, depending on your kernel, this might work (haven't tried it).

find . -mindepth 1 -maxdepth 1 -exec stat -c $'%W\t%n' '{}' \; |
    sort -k 1n |
    cut -f 2 -d $'\t' |
    head -n 10 |
    sed -e 's/^\.\///'

See also:


If you get errors about "'$\t'

The '$\t' notation requires bash or zsh: it will not work in dash or sh on Ubuntu. If you really need to use those shells, you will need to change any \t to Ctrl+V, Tab and remove the leading $ from just before the opening quote.

share|improve this answer
    
It's possible he may not be running ext4. I run Ubuntu 10.04 on all my machines, but run JFS on several drives. AFAIK JFS does support creation time stamps. –  jwernerny Jan 28 '11 at 13:23
    
Indeed. We know that it's not supported on ext3 and it is supported on ext4. A quick search suggests it might work with zfs, or the FreeBSD ufs, but neither of those are common on Ubuntu! Not sure about jfs or xfs or anything else. Would be happy to hear more if you can find info/links. –  Mikel Jan 28 '11 at 14:43
    
Thank you for that very comprehensive answer and for the subtle reminder to be write more specific questions ;) "Find" turned out to be a winner performance-wise, fs-type turned out to be ext3. –  user9565 Jan 28 '11 at 15:30
add comment

Another way if found of doing things today might be relevant to your performance issues:

I=0; ls -cr /dir/ | while read file; do I=`expr $I + 1`; echo "$file"; if [ $I == 100 ]; then break; fi; done

That should in theory start outputting a lot faster but I guess it depends where the lag is coming from. It could just be taking ls a long time to sort the files.

share|improve this answer
    
I doubt it. head actually exits as soon as it has read enough input. Try running both with time at the front. The head version is much faster on my system anyway. –  Mikel Jan 28 '11 at 22:36
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.