1

I have to copy a lot of files (60000+) between two servers, which is approx 5TB of data.

I've tried mounting the backup server as a folder, and copying the files that way, but I couldn't get the permissions to write files right.

So I thought of bonding the remaining 3 nic ports, connect crosscables and using cp/scp to copy everything. Now I have no experience with bonding nic's en transfering data that way.

Would it be faster? Can anyone relate or give me some advice on better solutions? Would be much appreciated.

5
  • Are scp and rsync not usable in your case? Or, connecting the two servers over a single server-to-server network NIC connection (static IPs on both NICs with the same IP range) on a network range not managed by your network? (A similar situation showed up at work hence why I ask) – Thomas Ward Jun 18 '15 at 13:17
  • Both options are usable, i was just wondering if bounding the nic's would speed up the proces, as one cat6 crosscable with scp command will take between 12 and 14 hours, if not more. – Roffel Jun 18 '15 at 13:35
  • Check this askubuntu.com/a/580645/216503 – heemayl Jun 18 '15 at 13:37
  • @Roffel ideally going forward you would be doing transactional updates, rather than copying between servers. You may have a ton of data to transfer initially but you should be keeping backups, and doing transactional updates (i.e. "This data is new, send it to the other server" rather than all-at-once) – Thomas Ward Jun 18 '15 at 14:09
  • @ThomasW. normaly that is what happes, we have 3 backup servers to store the files. only now we get an OS upgrade, with new kernel and averything. downside, the new kernel and OS cant deal with the current tree structure, thus will erase all harddrive on install. So this is more like a one time thing. To prevent our system going down for to long, i want to copy it fast, as it has to go back and forth, for all 3 servers. – Roffel Jun 19 '15 at 7:24
4

When copying large amounts of files I usually use these commands:

Target:

nc -q 1 -l 1234 | pv -pterb -s <filesize>G | tar xv

Source:

tar cv <DIR>  | nc -q 1 <targetip> 1234

This will directly stream all the data, without a lot of protocol overhead from source to target over port 1234. This proved to be the fastest way for me to copy the data in a local network. As an addition I added the pv command in the target, so I can get a rough overview how far along the files are.

For a more advanced, yet maybe slower transfer, I'd recommend using rsync.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.