Situation: fix Out of Memory
errors in Ubuntu's Matlab
Goal: allocate some virtual memory and/or Swap on external HDD/SSD; the read/write decrease from 20 GBps to 0.1 GBps, which is ok!
Terminology: swap and virtual memory here
No, swap and virtual memory are completely different. For example, memory mapping a 1GB file uses an additional 1GB of virtual memory but has no change in the use of swap. Swap is a form of backing store. Many uses of virtual memory don't have anything to do with backing store. (And there have been systems with virtual memory and no swap as well as systems with swap but no virtual memory.)
Consumer hardware is limited so I need to use more virtual memory and/or swap with my external HDDs. Matlab says about the swap memory (TODO excluding virtual memory?)
Linux Systems — Change your swap space by using the
mkswap
andswapon
commands.
Characteristics of the system
You can see how much you have it by
swapon -s
Filename Type Size Used Priority /dev/sda3 partition 8326140 0 -1
Matlab's configurations
% https://stackoverflow.com/a/35971040/54964 com.mathworks.services.Prefs.setIntegerPref('JavaMemHeapMax', 2048); % MB % TODO cannot find ways how to put Matlab use /dev/sda3
So you see that my Matlab is not using it. I receive
Out of Memory
errors with big matrices in Matlab. I have been unsuccessful inreshape
'ing matrices to vectors and writing parallel code. So I want to use virtual memory because I need to get work done; the rate does not matter.
Pseudocode
- Shell script that creates a swap, start MATLAB, and delete the swap when MATLAB exits. (MichaelHooreman)
- Enable swap on external HDD. How to use
sudo swapon -a
here? - Start Matlab.
- Put Matlab use the swap.
- Delete swap when Matlab exits.
Temporary Swap, Running Client and Closing/Removing Swap in Micheal's script
Situation: cannot control errors in setting up the environment (1), running Matlab (2) and closing the environment (3)
Script
#!/usr/bin/env bash
# https://stackoverflow.com/a/69808/54964
set -e
# TODO How to do swapoff if any error?
SWAP_FILE="/media/masi/SamiSwapVirtual/.swap_file_20.7.2016"
SIZE_MB=16000
TO_RUN="matlab"
dd if="/dev/zero" of=${SWAP_FILE} bs="1M" count=${SIZE_MB} status="progress"
mkswap ${SWAP_FILE}
chmod 0600 ${SWAP_FILE}
sudo chown 0.0 ${SWAP_FILE} # https://unix.stackexchange.com/a/297153/16920
sudo swapon -v ${SWAP_FILE}
echo "Swap enabled. Press enter to continue"; read
${TO_RUN}
echo "I will remove the swap. Press enter to continue"; read
sudo swapoff -v ${SWAP_FILE}
rm -vf ${SWAP_FILE}
Iteration 1 with Transcend 25M3 1 TB with little use berofe where filesystem ext4
Logs after starting the script
sh start_matlab_with_swap.sh 16000+0 records in 16000+0 records out 16777216000 bytes (17 GB, 16 GiB) copied, 134.489 s, 125 MB/s Setting up swapspace version 1, size = 15.6 GiB (16777211904 bytes) no label, UUID=48c2835b-4499-4534-aa49-0648e15bd5d9 [sudo] password for masi: swapon /media/masi/SamiWeek/tmp/swap_file_18.7.2016 swapon: /media/masi/SamiWeek/tmp/swap_file_18.7.2016: insecure file owner 1000, 0 (root) suggested. swapon: /media/masi/SamiWeek/tmp/swap_file_18.7.2016: found swap signature: version 1d, page-size 4, same byte order swapon: /media/masi/SamiWeek/tmp/swap_file_18.7.2016: pagesize=4096, swapsize=16777216000, devsize=16777216000 Swap enabled. Press enter to continue start_matlab_with_swap.sh: 11: read: arg count
Running the client
Command history gone in the first start of Matlab in the swap memory (ticket #02075943), with the error There was a problem reading your command history - -. Just restart your Matlab and and the problem is solved if you have default settings. Command
prefdir
gives/home/masi/.matlab/R2016a
which is the default location (/home/{username}/.matlab/R2016a
. File/home/masi/.matlab/R2016a/matlab.prf
exists after the restart, here.... [other errors] ...
Closing Matlab and typing password again in Terminal
[sudo] password for masi: swapoff /media/masi/SamiWeek/tmp/swap_file_18.7.2016 [ bugs here! ]
Open: How to Apply the Better Error-handling of Error trapping here? See my script for the example in the source. Thread How to Do Error-trapping and Swapoff if Error/Warning?
Permanent Swap = Separate Swap Setup from Running Client
Setting up Swap
# https://unix.stackexchange.com/q/297767/16920
masi@masi:~$ sudo fallocate -l 20G /mnt/.swapfile
masi@masi:~$ sudo mkswap /mnt/.swapfile
Setting up swapspace version 1, size = 20 GiB (21474832384 bytes)
no label, UUID=45df9e48-1760-47e8-84d7-7a14f56bbd72
masi@masi:~$ sudo swapon /mnt/.swapfile
swapon: /mnt/.swapfile: insecure permissions 0644, 0600 suggested.
masi@masi:~$ sudo chmod 600 /mnt/.swapfile
masi@masi:~$ free -m
total used free shared buff/cache available
Mem: 7925 1494 175 196 6255 5892
Swap: 28610 0 28610
Put the following in the end of /etc/fstab
for the permanent change
# https://unix.stackexchange.com/a/298212/16920
# https://unix.stackexchange.com/a/298543/16920
# If swap is on SSD, trim blocks each time at startup.
#/mnt/.swapfile none swap defaults,discard 0 0
# If swap on External HDD, just use sw.
/media/masi/SamiWeek/.swapfile none swap sw 0 0
System: Linux Ubuntu 16.04 64 bit
Linux kernel: 4.6
Linux kernel options: wl
Matlab: 2016a
Official Matlab docs: Resolve "Out of Memory" Errors
External HDD: Transcend 1 TB StoreJet 25M3 review, Transcend 2 TB StoreJet 25M3
External HDD filesystem: ext4
External HDD buffer: 8 MB
Related threads: How to increase MATLAB memory limitation in ubuntu? (how to use mkswap, swapon for MATLAB?), How to Reduce Physical memory increase in Matlab?, How to Resolve out of memory error in Matlab?, How to Fix Out of Memory error in Matlab for 10800x10800 matrix?, How do I increase memory limit (contiguous as well as overall) in Matlab r2012b?, How to Increase Array Block and Resolve Out of Memory Error in Matlab 2009b?, How to Resolve this Out of Memory Issue for a Small Variable in Matlab?, 'Out of memory' in Matlab. A slow but a permanent solution?