7

I'm transferring huge files, over a network to a destination server, totaling ~ 67 GB with individual zip files of size ~ 7GB. I tried copying to the destination using both sftp and rsync. rsync performs (2x) faster copying than sftp. sftp was achieving around 700 kbps while rsync transfers the data at a rate north of 1.4 Mbps. I know that SFTP and SCP uses the same SSH connection for transferring files. So, does this imply rsync is much (2x) faster than SCP which uses SSH under the hood?

Or above all, is there an even faster way to transfer (such huge) files over a network?

P.S.: I'm transferring completely new data to the destination server with the following rsync command:

rsync --info=progress2 -az source_folder username@destination_server:/location/
  • 1
    Is rsync connecting using SSH? Are you just updating data or is this transferring entirely new data? What options did you use for rsync? – muru Sep 30 '17 at 9:17
  • @muru please see my updated question! – kmario23 Sep 30 '17 at 9:24
  • So you're using compression -z. Try without it. – muru Sep 30 '17 at 9:31
  • @muru On average, the transfer speed is same whether I use compression -z or not :( – kmario23 Sep 30 '17 at 9:41
  • Do you mean sftp or scp? scp is known to be slower because it requires more overhead. – barrycarter Sep 30 '17 at 14:18
1

Or above all, is there an even faster way to transfer (such huge) files over a network?

You can speed up scp with

scp -p -C -o 'CompressionLevel 9' -o 'IPQoS throughput'  -c arcfour machine:file .

I saw up to 5 times better speed using scp but of course external factors play a big role here.

  • I'm getting command-line line 0: Unsupported option "compressionlevel" and Unknown cipher type 'arcfour' errors while trying your suggestion – kmario23 Mar 21 at 1:23
  • Also, I found exactly same answer here: superuser.com/a/787479 . Not whether this answer was copied from this link – kmario23 Mar 21 at 1:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.