17

I've just added a predictive search (see example below) feature to my site that runs on a Ubuntu Server. This runs direct from a database. I want to cache the result for each search and use that if it exists, else create it.

Would there be any problem with me saving the potential cira 10 million results in separate files in one directory? Or is it advisable to split them down into folders?

Example:

  • 5
    It would be better to split. Any command which tries to list the contents of that directory will likely decide to shoot itself. – muru Feb 11 '15 at 17:12
  • 8
    So if you already have a database, why not use that? I'm sure the DBMS will be better able to handle millions of records vs. the filesystem. If you're dead-set on using the filesystem you need to come up with a splitting scheme using some sort of hash, at this point IMHO it sounds like using the DB will be less work. – roadmr Feb 11 '15 at 18:54
  • 3
    Another option for caching that would fit your model better might be memcached or redis. They are key value stores (so they act like a single directory and you access items just by name). Redis is persistent (won't lose data when it is restarted) where as memcached is for more temporary items. – Stephen Ostermiller Feb 11 '15 at 20:45
  • 2
    There's a chicken-and-egg problem here. Tool developers don't handle directories with large numbers of files because people don't do that. And people don't make directories with large numbers of files because tools don't support it well. e.g. I understand at one time (and I believe this to still be true), a feature request to make a generator version of os.listdir in python was flatly denied for this reason. – user378002 Feb 12 '15 at 7:39
  • From my own experience I've seen breakage when going over 32k files in a single directory on Linux 2.6. It's possible to tune beyond this point of course, but I wouldn't recommend it. Just split into a few layers of subdirectories and it will be much better. Personally I'd limit it to around 10,000 per directory which would give you 2 layers. – Wolph Feb 12 '15 at 11:17
25

Would there be any problem with me saving the potential circa 10 million results in separate files in one directory?

Yes. There probably are more reasons but these I can post off the top of my head:

  • tune2fs has an option called dir_index that tends to be turned on by default (on Ubuntu it is) that lets you store roughly 100k files in a directory before you see a performance hit. That is not even close to the 10m files you are thinking about.

  • ext filesystems have a fixed maximum number of inodes. Every file and directory uses 1 inode. Use df -i for a view of your partitions and inodes free. When you run out of inodes you can not make new files or folders.

  • commands like rm and ls when using wildcards expand the command and will end up with a "argument list too long". You will have to use find to delete or list files. And find tends to be slow.

Or is it advisable to split them down into folders?

Yes. Most definitely. Basically you can not even store 10m files in 1 directory.

I would use the database. If you want to cache it for a website have a look at "solr" ("providing distributed indexing, replication and load-balanced querying").

8

Ended up with same issue. Run my own benchmarks to find out if you can place everything in the same folder vs. having multiple folders. It appears you can and it's faster!

Benchmark

Ref: https://medium.com/@hartator/benchmark-deep-directory-structure-vs-flat-directory-structure-to-store-millions-of-files-on-ext4-cac1000ca28

  • 1
    Thanks, this is very useful. I am using this on a site and it's been a real problem to re-program this part to have /abc/efg.html directory structures. So I will move back to a flat directory in future. – Kohjah Breese Dec 22 '18 at 4:01
0

A binary search can easily handle millions of records so searching the single directory would not be a problem. It will do so very fast.

Basically if you are using a 32 bit system, binary search upto 2Gb records is easy and good.

Berekely DB, an open source software, would readily allow you to store the full result under one entry and would have the search built in.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.