72

Yesterday I was copying a single, 8 GB file to a USB with a slow, 7 MB/s write speed, while my RAM is 3 GB. While copying the system froze, to the point where I couldn't even move the cursor.

I managed to log into the text console, and ran iotop, it showed that a process named kswapd0 was taking 99.99% of IO.

Are there workarounds so copying a large file doesn't make my system unusable?

Improve this question
3
  • 20
    This bug is so ridiculous...
    – king_julien
    Commented Jul 2, 2014 at 19:08
  • Yes, It not only happens with Ubuntu but also in other Debian flavors. I've also seen the same issue in Kali Linux and Parrot OS. Kali has the worst-case scenario, while parrot makes it smooth but even though hangs for very big sizes. I think it's the problem in the Linux Kernel and how it is written. This is and will remain the worst-nightmare of the Linux of all time.
    – Mohith7548
    Commented Nov 4, 2019 at 15:32
  • in 19.10 swap management caused the problem . Try disabling the swap usage if you have no intention to use applications like LibeOffice and more than one Browser tabs which requires more ram . it is recommanded not to disable the swap partition completly , try reducing swappiness to 10 or less .
    – Afsal
    Commented Dec 11, 2019 at 14:15

9 Answers 9

Reset to default
42

According to this bug report I solved it adding following lines

vm.dirty_background_ratio = 5
vm.dirty_ratio = 10

into /etc/sysctl.conf

and running 

sudo sysctl -p
Improve this answer
8
  • 15
    Care to explain what the above lines do?
    – nsane
    Commented May 2, 2015 at 6:39
  • 3
    @nisargshah95 sorry, but dont have a clue, search for yourself ;-)
    – Philippe Gachoud
    Commented Feb 15, 2016 at 9:45
  • 4
    @nisargshah95 The details of the problem are explained in the two LWN articles linked in unix.stackexchange.com/a/107722/52205
    – Rmano
    Commented May 15, 2016 at 15:37
  • 1
    Thank you, I just found that my Ubuntu 16.04 can not write two 1.4 GB files to USB without these two lines, I was getting frozen for hours, this solved problem, who cares what it does, sometimes you just want to copy files and move on.
    – Mike
    Commented Aug 10, 2016 at 19:29
  • 2
    I had values of 5 and 60. These control the percent of memory used for operations, while dirty_background_bytes and dirty_bytes use absolute bytes values. I've fixed this issue with second answer, but to make it persistent add it to sysctl.conf, see this answer. So when using percent values tweak them when upgrading memory.
    – PeterM
    Commented Sep 14, 2017 at 13:15
29
+50

I ran into the similar issue. Mine is 64 bit Ubuntu 14.04. So After a long struggle I found a answer which solves my issue. For easy use I added the commands below used in that above mentioned answer. Check the answer for detailed explanation.

echo $((16*1024*1024)) > /proc/sys/vm/dirty_background_bytes
echo $((48*1024*1024)) > /proc/sys/vm/dirty_bytes

After using the above command system started to work normally on copying files.

Thanks goes to @Rmano.

Improve this answer
7
  • 2
    The ratio settings didn't help on my 12.04 system with a slow NAS share. But after setting bytes directly as suggested here my system is usable again while copying to the NAS.
    – mivk
    Commented Dec 24, 2015 at 12:04
  • 9
    This question is 3 years old and this is still required to avoid getting an unusable system while copying to pendrive. Some info: if the pendrive is formatted using a Linux fs like ext4 this doesn't happen. When I said "unusable system" I really mean it, mouse pointer becomes unresponsive and you have to insist to move it around the screen, you look at the system monitor and there isn't any abnormal resource usage. Are the kernel people all using 6th gen Intel CPUs and SSD drives? How come they don't notice this while testing.
    – Hatoru Hansou
    Commented Feb 22, 2017 at 13:03
  • 4
    @HatoruHansou I feel the same, I just installed fresh Debian Stretch and this error is present here also. I know it does not depends on distribution, but on kernel sources, but men, how come this is still not fixed?
    – Marecky
    Commented May 19, 2017 at 15:20
  • 2
    @Marecky After some reading it seems that the dirty_bytes aren't a usb thing alone. They affect all I/O so after doing the echo thing you are changing them system global, not for pendrives alone. For the current session only, I think. It seems that kernel's current values are tweaked thinking in newer storage devices. Slow pendrives suffer as a side effect. Sorry, no link, but this must be easy to find by googling for it.
    – Hatoru Hansou
    Commented May 20, 2017 at 1:47
  • 4
    See this answer to make it persistent
    – PeterM
    Commented Sep 14, 2017 at 13:16
6

I am experiencing a similar issue with the system freezing when copying to a flash drive. I submitted a bug report about it: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1267648

As workaround I found that disabling swap completely eliminates the problem.

Improve this answer
2
  • 1
    Unfortunately this didn't work for me on Ubuntu 16.04.
    – Programster
    Commented Jun 4, 2017 at 17:23
  • Didn't work for me either on Ubuntu 16.04.3 LTS - this on an Alienware 17 R2 laptop.
    – AnthonyK
    Commented Nov 17, 2017 at 7:36
5

Yes, there are kernel settings you can tweak specifying how much data has to be marked as written before it actually gets written to disk. Look here for a pretty comprehensive description of them. In particular, you'll want to find a value of dirty_ratio that works well for you (it's generally too high for desktop/laptop by default, but there's no one magic number that works for everyone).

Improve this answer
1
  • 2
    Hey, could you please suggest me what numbers I need to set based on my laptop specifications ? refer askubuntu.com/questions/713723/…
    – user441517
    Commented Dec 28, 2015 at 19:21
5

I have just had exactly the same problem (in 2019), on ubuntu 19.10, while copying large number of files from USB disk to the SATA disk. Both filesystems are ext4. When I turned off the swap, the problem disappeared. It looks like some bug in memory allocation for disk buffers - apparently, the kernel tries to allocate as much memory for disk buffers, as possible in such situation, which doesn't make sense (making disk buffers in swap...), or it just wrongly calculates the memory size than can be used for caching...

As someone pointed out, setting swappiness to 1 doesn't solve the problem, which is logical, if the files you copy have total size more, than the RAM size...

By the way - can anybody explain, WHY turning off the swap is not recommended? If I have 32GB of RAM, what difference does it make to add another 4GB of swap? I can only think of some obscure applications that actually expect the swap to be there. But I would just stop using such application, since properly written one shouldn't care about swap. The swap should only be managed on the OS level...

Improve this answer
5
  • 1
    Did any of the solutions here work for you? I started suffering from this issue too after upgrading to 19.10 & in the process updating kernel too. In my case, this happens when I copy a large file from partition A to B of the same SSD. Disabling SWAP "fixes" the problem.
    – pileofrocks
    Commented Nov 10, 2019 at 10:02
  • 1
    I just keep my swap turned off. I have 32GB of RAM, and I don't see a point in having swap, actually. So far, everything works.
    – chaker
    Commented Nov 28, 2019 at 11:31
  • 1
    Try not to disable swap completely in 19.10 . you can adjust the swappiness to very small value such as 1 . swappiness determines when to use swap instead of ram . setting swappiness 100 will cause system to allocate mostly to swap and setting a value 1 will cause to utilize swap when there is no ram left .
    – Afsal
    Commented Dec 11, 2019 at 14:21
  • @pileofrocks This is the only working solution for me . Problem exists only in 19.10 .
    – Afsal
    Commented Dec 18, 2019 at 16:55
  • 1
    @Afsal Previously I've had it value 20, now tried 1 - but doesn't make any difference to this issue. kswapd0 still uses some cpu, mouse lags and entire system crawls for as long as the file writing is processing.
    – pileofrocks
    Commented Dec 18, 2019 at 20:21
3

in Ubuntu 19.10 this problem happens due to problem with swap management . Tinkering with dirty_background broke my nautilus file explorer . Along with freezing when copying large files , i have encountered several performance issues due to bad swap management in 19.10 . Since completely disabling swap is not recommended and we can control the degree of writing into swap partition by setting swappiness .

swappiness=0 tells the kernel to avoid swapping processes out of physical memory for as long as possible . swappiness=100 tells the kernel to aggressively swap processes out of physical memory and move them to swap cache

so , we are going to set swappiness =1 .

To change the swappiness value A temporary change (lost on reboot) with a swappiness value of 1 can be made with

sudo sysctl vm.swappiness=1

To make a change permanent, edit the configuration file with your favorite editor:

sudo gedit /etc/sysctl.conf

Search for vm.swappiness and change its value as desired. If vm.swappiness does not exist, add it to the end of the file like so:

vm.swappiness=1

Save the file and reboot.

This helped me to improve overall performance issues in Ubuntu 19.10 while disabling swap completely eliminated all performance issues that i had with 19.10 but still i won't recommend doing that in normal situations .

Improve this answer
3

As an experiment, I've set sysctl.conf to: 

vm.dirty_background_ratio=1
vm.dirty_ratio=2

and this solved the problem. Previous values 5/10 didn't help. Looks like everybody should play with those params to find the best one. Copying of 12 GB file from SSD to pen flash drive took about 15 min without system hangups (USB 2.0). Probably I'll check values 2/4, etc after several days of testing.

System: Ubuntu 19.10, i7, 8GB Ram, SSD.

Improve this answer
3

Turning off swap while copying worked for me.

sudo swapoff -a

After the copying is done, I turn on the swap

sudo swapon

Edit: Do note, as Robby1212 mentions in the comment, that swapping off is dangerous and should be done after you have tried out the other answers. I am fortunate that nothing has happened to me when I swap off. I only swap off during copying large files.

Improve this answer
3
  • 1
    Turning off the swap is a very dangerous proposition and should not be done without exhausting all other troubleshooting steps.
    – Robby1212
    Commented Apr 6, 2020 at 1:35
  • Tried almost everything. Problems are not only when using USB but unpacking large files on disk. Only disabling swap helped although there is some fancy logic behind this :(
    – DevilaN
    Commented Mar 31, 2021 at 12:36
  • Why is it so dangerous? What's the worst that can happen if my laptop has 16GB RAM? I thought I'd turn it off and not bother turning it back on if that's causing such problems.
    – Daniel Katz
    Commented Aug 12, 2022 at 11:11
1

I had similar problems when copying files to an exfat drive. I had less trouble using an ext4 filesystem on my USB hard drive .

Improve this answer
2
  • 1
    Had this issue on ext4 too
    – PeterM
    Commented Sep 14, 2017 at 13:01
  • Fedora 27 (kernel 4.17.5-100) copying from USB-attached spinning rust to USB-attached flash stick. This seems to go as far as freezing the screenlocker in mid-fade. :-( ~~~
    – David Tonhofer
    Commented Jul 18, 2018 at 9:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .