59

Yesterday I was copying a single, 8 GB file to a USB with a slow, 7 MB/s write speed, while my RAM is 3 GB. While copying the system froze, to the point where I couldn't even move the cursor.

I managed to log into the text console, and ran iotop, it showed that a process named kswapd0 was taking 99.99% of IO.

Are there workarounds so copying a large file doesn't make my system unusable?

  • 14
    This bug is so ridiculous... – king_julien Jul 2 '14 at 19:08
  • Yes, It not only happens with Ubuntu but also in other Debian flavors. I've also seen the same issue in Kali Linux and Parrot OS. Kali has the worst-case scenario, while parrot makes it smooth but even though hangs for very big sizes. I think it's the problem in the Linux Kernel and how it is written. This is and will remain the worst-nightmare of the Linux of all time. – Mohith7548 Nov 4 '19 at 15:32
  • in 19.10 swap management caused the problem . Try disabling the swap usage if you have no intention to use applications like LibeOffice and more than one Browser tabs which requires more ram . it is recommanded not to disable the swap partition completly , try reducing swappiness to 10 or less . – Afsal Dec 11 '19 at 14:15
36

According to this bug report I solved it adding following lines

vm.dirty_background_ratio = 5
vm.dirty_ratio = 10

into /etc/sysctl.conf

and running

sudo sysctl -p
| improve this answer | |
  • 14
    Care to explain what the above lines do? – nsane May 2 '15 at 6:39
  • 3
    @nisargshah95 sorry, but dont have a clue, search for yourself ;-) – Philippe Gachoud Feb 15 '16 at 9:45
  • 4
    @nisargshah95 The details of the problem are explained in the two LWN articles linked in unix.stackexchange.com/a/107722/52205 – Rmano May 15 '16 at 15:37
  • 1
    Thank you, I just found that my Ubuntu 16.04 can not write two 1.4 GB files to USB without these two lines, I was getting frozen for hours, this solved problem, who cares what it does, sometimes you just want to copy files and move on. – Mike Aug 10 '16 at 19:29
  • 1
    I had values of 5 and 60. These control the percent of memory used for operations, while dirty_background_bytes and dirty_bytes use absolute bytes values. I've fixed this issue with second answer, but to make it persistent add it to sysctl.conf, see this answer. So when using percent values tweak them when upgrading memory. – PeterM Sep 14 '17 at 13:15
22
+50

I ran into the similar issue. Mine is 64 bit Ubuntu 14.04. So After a long struggle I found a answer which solves my issue. For easy use I added the commands below used in that above mentioned answer. Check the answer for detailed explanation.

echo $((16*1024*1024)) > /proc/sys/vm/dirty_background_bytes
echo $((48*1024*1024)) > /proc/sys/vm/dirty_bytes

After using the above command system started to work normally on copying files.

Thanks goes to @Rmano.

| improve this answer | |
  • 2
    The ratio settings didn't help on my 12.04 system with a slow NAS share. But after setting bytes directly as suggested here my system is usable again while copying to the NAS. – mivk Dec 24 '15 at 12:04
  • 6
    This question is 3 years old and this is still required to avoid getting an unusable system while copying to pendrive. Some info: if the pendrive is formatted using a Linux fs like ext4 this doesn't happen. When I said "unusable system" I really mean it, mouse pointer becomes unresponsive and you have to insist to move it around the screen, you look at the system monitor and there isn't any abnormal resource usage. Are the kernel people all using 6th gen Intel CPUs and SSD drives? How come they don't notice this while testing. – Hatoru Hansou Feb 22 '17 at 13:03
  • 3
    @HatoruHansou I feel the same, I just installed fresh Debian Stretch and this error is present here also. I know it does not depends on distribution, but on kernel sources, but men, how come this is still not fixed? – Marecky May 19 '17 at 15:20
  • 1
    @Marecky After some reading it seems that the dirty_bytes aren't a usb thing alone. They affect all I/O so after doing the echo thing you are changing them system global, not for pendrives alone. For the current session only, I think. It seems that kernel's current values are tweaked thinking in newer storage devices. Slow pendrives suffer as a side effect. Sorry, no link, but this must be easy to find by googling for it. – Hatoru Hansou May 20 '17 at 1:47
  • 3
    See this answer to make it persistent – PeterM Sep 14 '17 at 13:16
6

I am experiencing a similar issue with the system freezing when copying to a flash drive. I submitted a bug report about it: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1267648

As workaround I found that disabling swap completely eliminates the problem.

| improve this answer | |
  • 1
    Unfortunately this didn't work for me on Ubuntu 16.04. – Programster Jun 4 '17 at 17:23
  • Didn't work for me either on Ubuntu 16.04.3 LTS - this on an Alienware 17 R2 laptop. – AnthonyK Nov 17 '17 at 7:36
5

I have just had exactly the same problem (in 2019), on ubuntu 19.10, while copying large number of files from USB disk to the SATA disk. Both filesystems are ext4. When I turned off the swap, the problem disappeared. It looks like some bug in memory allocation for disk buffers - apparently, the kernel tries to allocate as much memory for disk buffers, as possible in such situation, which doesn't make sense (making disk buffers in swap...), or it just wrongly calculates the memory size than can be used for caching...

As someone pointed out, setting swappiness to 1 doesn't solve the problem, which is logical, if the files you copy have total size more, than the RAM size...

By the way - can anybody explain, WHY turning off the swap is not recommended? If I have 32GB of RAM, what difference does it make to add another 4GB of swap? I can only think of some obscure applications that actually expect the swap to be there. But I would just stop using such application, since properly written one shouldn't care about swap. The swap should only be managed on the OS level...

| improve this answer | |
  • 1
    Did any of the solutions here work for you? I started suffering from this issue too after upgrading to 19.10 & in the process updating kernel too. In my case, this happens when I copy a large file from partition A to B of the same SSD. Disabling SWAP "fixes" the problem. – pileofrocks Nov 10 '19 at 10:02
  • 1
    I just keep my swap turned off. I have 32GB of RAM, and I don't see a point in having swap, actually. So far, everything works. – chaker Nov 28 '19 at 11:31
  • Try not to disable swap completely in 19.10 . you can adjust the swappiness to very small value such as 1 . swappiness determines when to use swap instead of ram . setting swappiness 100 will cause system to allocate mostly to swap and setting a value 1 will cause to utilize swap when there is no ram left . – Afsal Dec 11 '19 at 14:21
  • @pileofrocks This is the only working solution for me . Problem exists only in 19.10 . – Afsal Dec 18 '19 at 16:55
  • 1
    @Afsal Previously I've had it value 20, now tried 1 - but doesn't make any difference to this issue. kswapd0 still uses some cpu, mouse lags and entire system crawls for as long as the file writing is processing. – pileofrocks Dec 18 '19 at 20:21
4

Yes, there are kernel settings you can tweak specifying how much data has to be marked as written before it actually gets written to disk. Look here for a pretty comprehensive description of them. In particular, you'll want to find a value of dirty_ratio that works well for you (it's generally too high for desktop/laptop by default, but there's no one magic number that works for everyone).

| improve this answer | |
  • 2
    Hey, could you please suggest me what numbers I need to set based on my laptop specifications ? refer askubuntu.com/questions/713723/… – user441517 Dec 28 '15 at 19:21
2

in Ubuntu 19.10 this problem happens due to problem with swap management . Tinkering with dirty_background broke my nautilus file explorer . Along with freezing when copying large files , i have encountered several performance issues due to bad swap management in 19.10 . Since completely disabling swap is not recommended and we can control the degree of writing into swap partition by setting swappiness .

swappiness=0 tells the kernel to avoid swapping processes out of physical memory for as long as possible . swappiness=100 tells the kernel to aggressively swap processes out of physical memory and move them to swap cache

so , we are going to set swappiness =1 .

To change the swappiness value A temporary change (lost on reboot) with a swappiness value of 1 can be made with

sudo sysctl vm.swappiness=1

To make a change permanent, edit the configuration file with your favorite editor:

sudo gedit /etc/sysctl.conf

Search for vm.swappiness and change its value as desired. If vm.swappiness does not exist, add it to the end of the file like so:

vm.swappiness=1

Save the file and reboot.

This helped me to improve overall performance issues in Ubuntu 19.10 while disabling swap completely eliminated all performance issues that i had with 19.10 but still i won't recommend doing that in normal situations .

| improve this answer | |
2

As an experiment, I've set sysctl.conf to:

vm.dirty_background_ratio=1
vm.dirty_ratio=2

and this solved the problem. Previous values 5/10 didn't help. Looks like everybody should play with those params to find the best one. Copying of 12 GB file from SSD to pen flash drive took about 15 min without system hangups (USB 2.0). Probably I'll check values 2/4, etc after several days of testing.

System: Ubuntu 19.10, i7, 8GB Ram, SSD.

| improve this answer | |
1

I had similar problems when copying files to an exfat drive. I had less trouble using an ext4 filesystem on my USB hard drive .

| improve this answer | |
  • 1
    Had this issue on ext4 too – PeterM Sep 14 '17 at 13:01
  • Fedora 27 (kernel 4.17.5-100) copying from USB-attached spinning rust to USB-attached flash stick. This seems to go as far as freezing the screenlocker in mid-fade. :-( ~~~ – David Tonhofer Jul 18 '18 at 9:33
0

Turning off swap while copying worked for me.

sudo swapoff -a

After the copying is done, I turn on the swap

sudo swapon

Edit: Do note, as Robby1212 mentions in the comment, that swapping off is dangerous and should be done after you have tried out the other answers. I am fortunate that nothing has happened to me when I swap off. I only swap off during copying large files.

| improve this answer | |
  • Turning off the swap is a very dangerous proposition and should not be done without exhausting all other troubleshooting steps. – Robby1212 Apr 6 at 1:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.