17

Situation: fix Out of Memory errors in Ubuntu's Matlab
Goal: allocate some virtual memory and/or Swap on external HDD/SSD; the read/write decrease from 20 GBps to 0.1 GBps, which is ok!
Terminology: swap and virtual memory here

No, swap and virtual memory are completely different. For example, memory mapping a 1GB file uses an additional 1GB of virtual memory but has no change in the use of swap. Swap is a form of backing store. Many uses of virtual memory don't have anything to do with backing store. (And there have been systems with virtual memory and no swap as well as systems with swap but no virtual memory.)

Consumer hardware is limited so I need to use more virtual memory and/or swap with my external HDDs. Matlab says about the swap memory (TODO excluding virtual memory?)

Linux Systems — Change your swap space by using the mkswap and swapon commands.

Characteristics of the system

  • You can see how much you have it by swapon -s

    Filename                Type        Size    Used    Priority
    /dev/sda3               partition   8326140 0       -1
    
  • Matlab's configurations

    % https://stackoverflow.com/a/35971040/54964
    com.mathworks.services.Prefs.setIntegerPref('JavaMemHeapMax', 2048); % MB
    
    % TODO cannot find ways how to put Matlab use /dev/sda3
    
  • So you see that my Matlab is not using it. I receive Out of Memory errors with big matrices in Matlab. I have been unsuccessful in reshape'ing matrices to vectors and writing parallel code. So I want to use virtual memory because I need to get work done; the rate does not matter.

Pseudocode

  1. Shell script that creates a swap, start MATLAB, and delete the swap when MATLAB exits. (MichaelHooreman)
  2. Enable swap on external HDD. How to use sudo swapon -a here?
  3. Start Matlab.
  4. Put Matlab use the swap.
  5. Delete swap when Matlab exits.

Temporary Swap, Running Client and Closing/Removing Swap in Micheal's script

Situation: cannot control errors in setting up the environment (1), running Matlab (2) and closing the environment (3)
Script

#!/usr/bin/env bash

# https://stackoverflow.com/a/69808/54964
set -e 
# TODO How to do swapoff if any error?

SWAP_FILE="/media/masi/SamiSwapVirtual/.swap_file_20.7.2016"
SIZE_MB=16000
TO_RUN="matlab"

dd if="/dev/zero" of=${SWAP_FILE} bs="1M" count=${SIZE_MB} status="progress"
mkswap ${SWAP_FILE}
chmod 0600 ${SWAP_FILE}
sudo chown 0.0 ${SWAP_FILE} # https://unix.stackexchange.com/a/297153/16920
sudo swapon -v ${SWAP_FILE}
echo "Swap enabled. Press enter to continue"; read
${TO_RUN}
echo "I will remove the swap. Press enter to continue"; read
sudo swapoff -v ${SWAP_FILE}
rm -vf ${SWAP_FILE}

Iteration 1 with Transcend 25M3 1 TB with little use berofe where filesystem ext4

  1. Logs after starting the script

    sh start_matlab_with_swap.sh 
    16000+0 records in
    16000+0 records out
    16777216000 bytes (17 GB, 16 GiB) copied, 134.489 s, 125 MB/s
    Setting up swapspace version 1, size = 15.6 GiB (16777211904 bytes)
    no label, UUID=48c2835b-4499-4534-aa49-0648e15bd5d9
    [sudo] password for masi: 
    swapon /media/masi/SamiWeek/tmp/swap_file_18.7.2016
    swapon: /media/masi/SamiWeek/tmp/swap_file_18.7.2016: insecure file owner 1000, 0 (root) suggested.
    swapon: /media/masi/SamiWeek/tmp/swap_file_18.7.2016: found swap signature: version 1d, page-size 4, same byte order
    swapon: /media/masi/SamiWeek/tmp/swap_file_18.7.2016: pagesize=4096, swapsize=16777216000, devsize=16777216000
    Swap enabled. Press enter to continue
    start_matlab_with_swap.sh: 11: read: arg count
    
  2. Running the client

    • Command history gone in the first start of Matlab in the swap memory (ticket #02075943), with the error There was a problem reading your command history - -. Just restart your Matlab and and the problem is solved if you have default settings. Command prefdir gives /home/masi/.matlab/R2016a which is the default location (/home/{username}/.matlab/R2016a. File /home/masi/.matlab/R2016a/matlab.prf exists after the restart, here.

    • ... [other errors] ...

  3. Closing Matlab and typing password again in Terminal

    [sudo] password for masi: 
    swapoff /media/masi/SamiWeek/tmp/swap_file_18.7.2016
    [ bugs here! ]
    

Open: How to Apply the Better Error-handling of Error trapping here? See my script for the example in the source. Thread How to Do Error-trapping and Swapoff if Error/Warning?

Permanent Swap = Separate Swap Setup from Running Client

Setting up Swap

# https://unix.stackexchange.com/q/297767/16920
masi@masi:~$ sudo fallocate -l 20G /mnt/.swapfile

masi@masi:~$ sudo mkswap /mnt/.swapfile 
Setting up swapspace version 1, size = 20 GiB (21474832384 bytes)
no label, UUID=45df9e48-1760-47e8-84d7-7a14f56bbd72

masi@masi:~$ sudo swapon /mnt/.swapfile
swapon: /mnt/.swapfile: insecure permissions 0644, 0600 suggested.

masi@masi:~$ sudo chmod 600 /mnt/.swapfile

masi@masi:~$ free -m
              total        used        free      shared  buff/cache   available
Mem:           7925        1494         175         196        6255        5892
Swap:         28610           0       28610

Put the following in the end of /etc/fstab for the permanent change

# https://unix.stackexchange.com/a/298212/16920
# https://unix.stackexchange.com/a/298543/16920

# If swap is on SSD, trim blocks each time at startup.
#/mnt/.swapfile  none    swap    defaults,discard      0        0

# If swap on External HDD, just use sw.
/media/masi/SamiWeek/.swapfile  none    swap    sw      0        0

System: Linux Ubuntu 16.04 64 bit
Linux kernel: 4.6
Linux kernel options: wl
Matlab: 2016a
Official Matlab docs: Resolve "Out of Memory" Errors
External HDD: Transcend 1 TB StoreJet 25M3 review, Transcend 2 TB StoreJet 25M3
External HDD filesystem: ext4
External HDD buffer: 8 MB
Related threads: How to increase MATLAB memory limitation in ubuntu? (how to use mkswap, swapon for MATLAB?), How to Reduce Physical memory increase in Matlab?, How to Resolve out of memory error in Matlab?, How to Fix Out of Memory error in Matlab for 10800x10800 matrix?, How do I increase memory limit (contiguous as well as overall) in Matlab r2012b?, How to Increase Array Block and Resolve Out of Memory Error in Matlab 2009b?, How to Resolve this Out of Memory Issue for a Small Variable in Matlab?, 'Out of memory' in Matlab. A slow but a permanent solution?

6
  • 4
    Sorry this isn't answering the question. But are you sure you want this? Doing computations from swap takes unbelievably long. Ram reads at at least 20 GB per second, your external HDD will be reading at less than 0.1 GBps
    – Anake
    Jul 17 '16 at 19:04
  • 1
    @Anake Yes, I know. It is not a problem. I have huge matrices which can be computed with 32/64 GB but not with my current 8 GB ultrabook. I still need to do tho computation on holidays. Jul 17 '16 at 19:05
  • 1
    Another non-useful comment sorry. Could you leave a computer on at home so that you could SSH over and run them at home/uni?
    – Anake
    Jul 17 '16 at 19:07
  • @Anake Not possible, sorry, at the moment. Also, Matlab here so need local computation. Also, do not have sufficient keys to do it at the moment. Jul 17 '16 at 19:09
  • What is special about MATLAB in this context? Isn't this just a duplicate of How to increase swap space? Jul 17 '16 at 20:43
2

OK, quite a list you have there. Let me respond inline

  1. How to Apply the Better Error-handling of Error trapping here? See my script for the example in the source. Thread How to Do Error-trapping and Swapoff if Error/Warning?.

I don't like the concept of this script at all. That you have an external harddrive that you're trying to use as swap is just a bad idea. If you really intend to do this on a regular basis then resize your partitions to put a proper swap partition in, add a swap file, or just buy a bigger internal disk.

  1. How to put warnings if the matrix size exceeds the swap size?

Just do do the math. If you know the size of the matrix before the program begins then compute the size in MiB and compare it to the available swap.

  1. How to have a progress bar in computing your huge matrix in Matlab?

matlab has an API right? I don't think this is the right forum for that question. Even if you had an API, you'll be blocking on IO via swap so it'll just be a jerky progress bar that doesn't actually reflect reality.

  1. How to kill busy progress and/or swapon -s/swapoff in iteration (2)?

You don't. just because you're done with the computation doesn't mean the operating system is done with the resources you allocated. When it's done writing out to swap, it'll free up. You've consumed so much memory that lots of applications can't get the memory they need so they're using swap too. Just leave it on and let the operating system do it's thing. Before you perform your next run clear the caches.

echo 3 > /proc/sys/vm/drop_caches 

There's probably more to it than that, I'm not a Linux VM expert. It would be worthwhile to investigate how the SLAB/SLUB allocator works and how to tune it for your large memory requirements. You may be able to MLOCK matlab into memory. That forces the OS to reserve memory for you, or it just doesn't start, you also have to unlock it when you're done. I can do this with the C API just fine but I not sure how you would do that outside of a process I can't recompile, that would require some research.

Finally, this is the sort of stuff EC2 was made for. It looks like 16G is what you need, a m4.4xlarge has 64G ram @ $0.958 per Hour. That's less than a cup coffee. Script your install of matlab using a juju charm or similar and turn the whole thing into a computation as a service.

Is 16G 16 GB?

  • Yes, normally when we leave off the suffix we mean base2 numbers in units of bytes. If you want to concise you would write 16GiB.

"I need matrices that are > 100 GB. I do not know if you can do it with EC2."

Should you clear your caches also by echo 3 > /proc/sys/vm/drop_caches?

  • Yes, it doesn't hurt to always do that. See Documentation/sysctl/vm.txt in the linux kernel.

How can you MLOCK Matlab into memory?

  • man mlock. Though I goofed when I quoted that. This call assures that you can allocate all the memory you want and keep in from being swapped out, it'll never use virtual memory. That's not what you want.

I think you can bind C API to Matlab. - - Do you have any idea of turning off swap if there is any failure in the processes?

  • I'm going to be frank here, the concept of micromanaging swap files in the manner you propose is ridiculous. The operating system's job is to manage resources and hand them out in a fair & consistent manner. Once you give it more resources, it's going to use them as it sees fit. You don't get to tell it when you're done and yank resources out from underneath it, the OS tells you when it's done.

When I ask the OS for an memory address space, sometimes it doesn't always succeed, that doesn't mean I can't try again. That matlab can't figure out to call malloc twice is matlab's problem.

So, to affect the change you want, if that 100G of space is really at a premium then you need to figure out how to tell the operating system to trim it's memory footprint (by clearing caches for starters) so the memory manager doesn't feel the need to use the additional swap space that it was provided. Then and only then can you ask the memory manager to release the swap file.

It's easy to grow things like memory and disks, it's a lot harder to shrink them. Shrinking forces a re-balancing of every user who has resources allocated in that space. If I instead said "I have a 100TB storage array but now I only need 60TB, why is it when I remove 40TB of disk that the array stops working?" Well, the answer would be obvious right?

So here are your options as I see it.

  1. investigate the matlab C API to see if you can get better control over how memory is allocated for these massive working sets.

  2. refactor your computation to compute what you have now using sub-matrices or some other sparse data representation.

  3. write your own program in C/C++ using the plethora of linear algebra libraries out there to perform the computation and use malloc or mmap anonymous to allocate the address space you need.

3
  • Yes, but it might not be enough and you may have to wait and retry before it succeeds, if it ever does.
    – ppetraki
    Jul 21 '16 at 14:48
  • 1
    heh, the rest of your comment didn't show up in my smartphone. you can't sudo echo like that, instead do. echo 3 | sudo tee /proc/sys/vm/drop_caches
    – ppetraki
    Jul 21 '16 at 14:57
  • I accept this answer because it has the right sense in its answer. To run Client in the same script as setting up the environment is insanely error-prone. More discussion about it here unix.stackexchange.com/a/298543/16920 - - It would be great if you could clean up your body little. Jul 27 '16 at 19:18
12
+200

You cannot dedicate swap for a software. What you can do is create a shell script that creates a swap, start MATLAB, and delete the swap when MATLAB exits.

Here's an example script which creates a swap of 10Mb in the /tmp directory, mounts it, starts R (I don't have matlab), wait R exits, umount the swap file and delete it.

Please note that: - you will have a warning because the swap file is not owned by root. That's because the system will use if for any software, maybe not ran by you, and you can read on this file... I let you fix it. - if you [ctrl]-[c] the script, or logoff, or etc., the swap will remain mounted. I let you fix it as well.

#!/usr/bin/env bash

SWAP_FILE=/tmp/my_swap_file
SIZE_MB=10
TO_RUN="R"

dd if=/dev/zero of=${SWAP_FILE} bs=1M count=${SIZE_MB}
mkswap ${SWAP_FILE}
chmod 0600 ${SWAP_FILE}
sudo swapon -v ${SWAP_FILE}
echo "Swap enabled. Press enter to continue"; read
${TO_RUN}
echo "I will remove the swap. Press enter to continue"; read
sudo swapoff -v ${SWAP_FILE}
rm -vf ${SWAP_FILE}
4
  • 1
    Well, it is not really dedicated to MATLAB, but for any situation needing temporarily more RAM. Please note that swapon/swapoff needs root right (so: sudo) Jul 17 '16 at 20:07
  • 1
    Yes, you have. That's how to become root. Jul 18 '16 at 4:51
  • Good answer. I personally would approach it this was, and keep my own add-swap.sh script for thise lovely moments when memory gets eaten up Jul 20 '16 at 16:52
  • I awarded the bounty here because it made me thing the correct way. I cannot however accept the answer because the error-management is very difficult and incomplete. It is best to setup the swap separately from running Matlab, please see the answer here unix.stackexchange.com/a/298543/16920 Jul 27 '16 at 19:17
2

Use zswap if you have swap portions on HDD/SSD. Tho module zram is without swap portions on HDD/SSD, so Hakala's answer is not applicable. See the thread zram vs zswap vs zcache Ultimate guide: when to use which one for explanations. Setup zswap as described in the thread How to Activate Zswap Successfully for Matlab Computation in Ubuntu 16.04?

  • Replace the corresponding line with the following line in /etc/default/grub

    # https://wiki.archlinux.org/index.php/Zswap
    GRUB_CMDLINE_LINUX_DEFAULT="quiet splash zswap.enabled=1 zswap.max_pool_percent=25 zswap.compressor=lzo"
    
  • Run sudo update-grub.

1

This is how you can expand your SWAP memory using an external hard drive:

  • First, write down your actual SWAP memory by running:

    free -m
    
  • Second, have at hand the folder of your HDD. It should be something like /media/myhdd.

  • Decide the size of extra SWAP you want to add. Let's say, X GB.
  • Calculate the amount of bytes that quantity is. Using GB, this is: Y=X*1024^3, where Y is the result of your calculation.
  • Choose the block size of the file (it is in bytes). Let's use the default here: 4096 (See more about this here).
  • Calculate the number of blocks the file will have: Z=Y/4096
  • Create a file of size X GB using the following command in a terminal:

    dd if=/dev/zero of=/media/myhdd/swapfile bs=4096 count=Z
    
  • Create the SWAP in the file using:

    sudo mkswap /media/myhdd/swapfile -f
    
  • Finally, enable the SWAP:

    sudo swapon -p 1000 /media/myhdd/swapfile
    

Now your SWAP has increased. Check again with free -m


We can set up this in a non-interactive script (sudo powers needed):

#!/bin/bash

### Inputs ###

swap_GB=$(expr 1) # Enter here size of the swap memory to create, in GB.
swap_bs=$(expr 4096) # Enter here block size, in bytes (must be a multiple of 8).
HDD_folder="/media/myhdd/" # Enter absolute path of HDD inside the brackets.

### Swap creation ###

swap_size=$(expr $swap_GB \* 1024 \* 1024 \* 1024 / $swap_bs)
dd if=/dev/zero of=${HDD_folder}"/swapfile" bs=$swap_bs count=$swap_size
mkswap ${HDD_folder}"/swapfile" -f
swapon -p 1000 ${HDD_folder}"/swapfile"

### EOF ###

PS: please optimize/correct if possible. As said, it's my first script ever :)

3
  • Well, swap is already slow, but I can't imagine having swap on USB drive. It will become slow like hell IMHO Jul 20 '16 at 21:06
  • A script as a summary would be great here. Jul 21 '16 at 5:48
  • 1
    I never have created a script, but will give it a go.
    – user308164
    Jul 21 '16 at 7:17
1

I would at least test how well compressed RAM (zram kernel module, available since kernel version 3.14) performs.

Following archlinux wiki instructions

modprobe zram
echo lz4 > /sys/block/zram0/comp_algorithm
echo 4G > /sys/block/zram0/disksize
mkswap --label zram0 /dev/zram0
swapon --priority 100 /dev/zram0

My guess is that compressed RAM should be faster than disk I/O.

To keep the change also after the restart, place boot time commands in /etc/rc.local and run sudo systemctl enable rc-local.service.

4
  • I extended the answer to the thread unix.stackexchange.com/q/297752/16920 for the effect of zram on the memory size and paging rate. I cannot find any effect on the memory size, probably mostly on the paging. However, systemd/... warning have increased with runit. It would be great to know about the error rate of zram. Jul 23 '16 at 7:53
  • In case of zram0, there are some statistics available in /sys/block/zram0 in files orig_data_size and compr_data_size. I don't have much data swapped there, but for what is there the ratio is about 0.35. With that ratio 11 GB of data might fit in to 4 GB of RAM. That could mean 7 GB of data not swapped to disk. Jul 23 '16 at 8:00
  • 1
    You can place boot time commands in /etc/rc.local and then sudo systemctl enable rc-local.service Jul 23 '16 at 8:53
  • I found out that zram is not applicable for swap on HDD/SSD so zram is not applicable here, please see the thread askubuntu.com/a/472227/25388 Jul 25 '16 at 16:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.