Configuring ISCSI timeout settings for AWS Storage Gateway volume gateway

Question

I have an EC2 instance which is configured to attach to an volume gateway volume configured using the AWS Storage Gateway service. The instance connects to the iSCSI interface provided by the volume gateway via the iSCSI Initiator (provided by the open-iscsi APT package).

Recently Amazon contacted us to alert us of an update they were rolling out to our Storage Gateway host which might result in some downtime. To minimise the risk of disruption Amazon recommends increasing the iSCSI request queue and disk timeouts used for volume gateway. However, the instructions they provide on how to do this are specific to Windows and RHEL.

Does anyone know the best way to configure iSCSI disk and request timeouts, as used by the iSCSI initiator in Ubuntu 16.04?

Answer 1

After a bit of experimentation and reading the man pages for the iscsiadm utility in Ubuntu 16.04, I have come up with the following solution:

For the iSCSI request queue timeouts I modified the following entries in /etc/iscsi/iscsid.conf, to the exact values suggested by AWS in their documentation for RHEL:

node.session.timeo.replacement_timeout = 600 
node.conn[0].timeo.noop_out_interval = 60
node.conn[0].timeo.noop_out_timeout = 600

To load in these settings I then ran the following to clear and repopulate the iSCSI descovery database:

sudo iscsiadm -m discoverydb -t sendtargets -p [GATEWAY_IP]:3260 -o delete
sudo /sbin/iscsiadm --mode discovery --type sendtargets --portal 10.79.1.4:3260

Replacing [GATEWAY_IP] with the IP address used to connect to my Storage Gateway host.

I then verified the updated timeout values had been pulled in by running:

sudo iscsiadm --mode node --op show

For configuring the disk timeouts I created a new udev rules file, /etc/udev/rules.d/60-storage-gateway.rules, containing the same rule suggested by AWS for RHEL 6/7:

ACTION=="add", SUBSYSTEMS=="scsi" , ATTRS{model}=="Storage Gateway", 
RUN+="/bin/sh -c 'echo 600 > /sys$$DEVPATH/timeout'"

I then forced udev to reconfigure the volume gateway iSCSI disk (in my case /dev/sda) by running:

udevadm test /sys/block/sda

I was then verified the new timeout value had been properly configured by running:

cat /sys/block/sda/device/timeout