1

I'm trying to transfer files over 10 gigabit ethernet. However the maximum speed I can achieve between 2 RAM-Disks (mounted on /mnt on each node) is around 200MB/s:

root@odin$ scp /mnt/largeFile marc@10.0.0.12:/mnt
marc@10.0.0.12's password:
largeFile                                                            86% 6904MB 192.4MB/s   00:05 ETA

When running Iperf I can achieve 9.89 Gb/s:

root@odin$ iperf -c 10.0.0.12
------------------------------------------------------------
Client connecting to 10.0.0.12, TCP port 5001
TCP window size:  325 KByte (default)
------------------------------------------------------------
[  3] local 10.0.0.11 port 59028 connected with 10.0.0.12 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  11.5 GBytes  9.89 Gbits/sec

The RAM-Disks achieve write speeds of 1.2 GB/s.

The network config on the 2 machines:

Server:

enp1s0    Link encap:Ethernet  HWaddr 00:02:c9:4d:a3:12
          inet addr:10.0.0.12  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::202:c9ff:fe4d:a312/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1

Client:

ens1      Link encap:Ethernet  HWaddr 00:02:c9:4f:98:14
          inet addr:10.0.0.11  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::202:c9ff:fe4f:9814/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1

I'm wondering how I can achieve speeds similar to the iperf ones for file Transfers?

0

It's due to the transfer limits caused by various aspects of scp/OpenSSH.

If you look at your destination machines resources while doing lperf and scp tests you should notice that even at 5x the speed the CPU will be under a lower load with lperf than with scp , this solely due to encryption.

There is a great set of resources and patches available from the Pittsburgh Supercomputing Center, https://www.psc.edu/index.php/hpn-ssh

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.