2

I want all my files in an ext4 filesystem not fragmented because of reasons. Sadly e4defrag (advised here: How to defrag an ext4 filesystem) fails to defragment several files. What are my alternatives?

The filesystem has all files in it already (they are not to be changed in any way) and it is almost full. There are several free blocks (according to df -h: 434M available of 85G and 80G used) which can be used as buffer. I do not need the filesystem mounted while defragmenting. Moreover, I have other filesystem available with enough space to use as a buffer.

One idea I have is to move the files to other filesystem and then copy them back somehow telling the filesystem to store them contiguously.

[EDIT]

I have just found that I cannot rely on e4defrag output. It counts files with more than one extent as fragmented, while it knows that the extents are contiguous:

$ sudo filefrag file.file
file.file: 1 extent found
$ sudo e4defrag -vc file.file
e4defrag 1.45.5 (07-Jan-2020)
<File>
[ext 1]: start 22388736: logical 0: len 32768
[ext 2]: start 22421504: logical 32768: len 32768
[ext 3]: start 22454272: logical 65536: len 32768
[ext 4]: start 22487040: logical 98304: len 27962

 Total/best extents 4/1
 Average size per extent 126266 KB
 Fragmentation score 0
 [0-30 no problem: 31-55 a little bit fragmented: 56- needs defrag]
 This file (file.file) does not need defragmentation.
 Done.
19
  • 5
    There is no need in defragmentation in ext4.
    – Mahler
    Nov 13, 2022 at 12:08
  • Is the purpose of this to shrink an ext4 virtual disk image? 🤔
    – matigo
    Nov 13, 2022 at 12:22
  • 1
    @Mahler then what is your way to have all files contiguous?
    – abukaj
    Nov 13, 2022 at 12:38
  • 8
    This sounds very much like an XY problem. If you can explain why you think you need exactly zero fragmentation, people can probably give you a better answer. Based on your comments about the filesystem reasonably being read-only though, ext4 is probably not the best choice here, and you should be looking at either SquashFS or EROFS instead (or possibly CramFS, but that’s inferior to either SquashFS or EROFS in most ways). Nov 14, 2022 at 2:35
  • 2
    @Mahler Each individual IO request still has a cost in the OS. It’s not ‘free’ to ask the drive for data, the OS has to set up the region of memory which the data will be transferred to, actually send the request to the storage device (which can be very time consuming depending on how it’s connected, wait for completion (yes, there is a wait, even for an SSD), and then once it has the data clean everything up. That overhead is per-request, so by issuing larger requests you get lower overhead and faster bulk data transfers. Nov 15, 2022 at 12:47

3 Answers 3

12

I want all my files in an ext4 filesystem not fragmented because of reasons.

While there are legitimate reasons to defrag, none require every single file to be defragged and contiguous. The main reasons anyone might want every file to be defragged are OCPD related, which is a complete waste of time because the file system will become "fragmented" again shortly after being mounted rw.

The filesystem... is almost full...

In that scenario, you probably will not be able to defrag every file because Linux defrag programs tend to work at the file level and you do not necessarily have enough contiguous free space to defrag every file.

One idea I have is to move the files to other filesystem and then copy them back...

That is your most viable option. However, specific file allocation is determined by the file system driver.


To reorder blocks it is enough to have one free block as a buffer.

Linux file system devs have not given defrag the same priority that Windows devs have. So the problem is not so much that it is technically impossible, but that no one has bothered to write any programs to do so.

The fs may be set to ro after defragmentation.

Then use a filesystem designed for ro use, like squashfs. All files will be defragged, contiguous, and even compressed.

2
  • "the file system will become "fragmented" again shortly after being mounted rw" As I mentioned, no file is going to be changed. The fs may be set to ro after defragmentation. "will not be able to defrag every file" May you elaborate why is it so? To reorder blocks it is enough to have one free block as a buffer. "That is your most viable option." How do I tell the ext4 not to leave empty blocks between files?
    – abukaj
    Nov 13, 2022 at 13:58
  • 1
    Edited to answer some of your questions.
    – xiota
    Nov 13, 2022 at 14:09
4

If some of your files are big, it might be technically impossible to defragment them all without reformating the filesystem.

Any ext4 filesystem is composed of a sequence of block groups. By default, each block group is 128 MiB long.

Each block group starts with a bunch of filesystem metadata (superblock, group descriptor, allocation bitmaps, and inode tables) followed by actual data blocks used by files belonging to that block group. This means that filesystem metadata are scattered mostly uniformly across the entire device.

However, thanks to the optional flex_bg feature, several block groups can be aggregated together into a single bigger one. mke2fs has been creating filesystems by default with 16 block groups packed together since 2008-ish. Assuming you haven't changed this when making the filesystem using the -G option to mkfs, your filesystem is thus likely split into 2-GiB flex groups.

Unless all your files are significantly smaller than 2 GiB, you would thus inevitably run into a situation where the next file to store would have to be fragmented across two or more (flex) block groups. Of course this is guaranteed to happen if any of your files is bigger than the usable data blocks in a (flex) block group.

To achieve your goal, you will thus likely have to reformat the filesystem with a much higher setting of the -G option than the default 16 to make the filesystem use really big flex block groups.

4
  • Now I wonder why e4defrag reports "success" for files 900M+. ;)
    – abukaj
    Nov 13, 2022 at 21:23
  • 1
    @abukaj Not sure what exactly you mean by "reporting success", e4defrag can print several different messages meaning different things. But in general, e4defrag is smart enough to calculate the "best possible number of fragments a given file could have with a given block group size" and uses that to find if a file is worth trying to defragment further. (See the source of get_best_count().)
    – TooTea
    Nov 13, 2022 at 22:55
  • @abukaj After further checking of the code, I must conclude that my original version of this answer was wrong. It's actually perfectly possible to make a filesystem with huge flex block groups using the standard tools (without any hacking). You might want to un-accept this answer if it doesn't apply anymore.
    – TooTea
    Nov 14, 2022 at 9:25
  • I meant output like: $ e4defrag 986M.file <NL> e4defrag 1.45.5 (07-Jan-2020) <NL> ext4 defragmentation for 986M.file <NL> [1/1]986M.file: 100% [ OK ] <NL> Success: [1/1] Meanwhile I ran sudo e4defrag -v . and had 986M.file listed as a fragmented file (now/best = 1/1)...
    – abukaj
    Nov 15, 2022 at 9:07
0

Back before DOS 6, the usual advice for defragmenting a FAT partition was:

  1. Copy the files from the partition to something else;
  2. Wipe the partition;
  3. Re-create the directory structure on the empty partition; and
  4. Copy the files back.

I never tried this because MS-DOS 6 came out (with its included defrag utility) before defragmenting came to be an issue for me.

4
  • 2
    The major problem with that approach is that "Linux does not need defragmentation" or rather that its filesystems are designed to avoid fragmentation. That means that while FAT fs lines files as close to each other as possible (which leads to file fragmentation if you append to it), extX scatters files around the disk leaving free blocks in case they grow. That means as you approach the filesystem capacity (which should be avoided in most circumstances), your copied files start to be fragmented.
    – abukaj
    Nov 13, 2022 at 16:23
  • @abukaj I'm sure you're correct. And I haven't looked into it deeply enough to offer more than the paraphrased "Four Yorkshiremen" skit. ("You had a File Allocation Table? Luxury!") Seems to me that if conventional tools aren't giving you what you want, though, and you insist on ext4, the old-school approach is something you can try. Now that I think of it, because you don't intend to change these files, what if you created a UDF-formatted disk image, moved the files there, and mounted it when you needed to? Would that be too slow?
    – MDeBusk
    Nov 13, 2022 at 20:51
  • I think I will either drop ext4 or the "no fragmentation" requirement, whichever is easier (see the accepted answer). BTW: I have not known the original skit.
    – abukaj
    Nov 13, 2022 at 21:24
  • 2
    @abukaj The Four Yorkshiremen is one of Monty Python's more famous sketches. Four well-off men trying to one-up one another over their difficult childhhoods.
    – MDeBusk
    Nov 13, 2022 at 23:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.