3

I have to copy a lot of files (60000+) between two servers, which is approx 5TB of data.

I've tried mounting the backup server as a folder, and copying the files that way, but I couldn't get the permissions to write files right.

So I thought of bonding the remaining 3 nic ports, connect crosscables and using cp/scp to copy everything. Now I have no experience with bonding nic's en transfering data that way.

Would it be faster? Can anyone relate or give me some advice on better solutions? Would be much appreciated.

5
  • Are scp and rsync not usable in your case? Or, connecting the two servers over a single server-to-server network NIC connection (static IPs on both NICs with the same IP range) on a network range not managed by your network? (A similar situation showed up at work hence why I ask)
    – Thomas Ward
    Jun 18, 2015 at 13:17
  • Both options are usable, i was just wondering if bounding the nic's would speed up the proces, as one cat6 crosscable with scp command will take between 12 and 14 hours, if not more.
    – Roffel
    Jun 18, 2015 at 13:35
  • Check this askubuntu.com/a/580645/216503
    – heemayl
    Jun 18, 2015 at 13:37
  • @Roffel ideally going forward you would be doing transactional updates, rather than copying between servers. You may have a ton of data to transfer initially but you should be keeping backups, and doing transactional updates (i.e. "This data is new, send it to the other server" rather than all-at-once)
    – Thomas Ward
    Jun 18, 2015 at 14:09
  • @ThomasW. normaly that is what happes, we have 3 backup servers to store the files. only now we get an OS upgrade, with new kernel and averything. downside, the new kernel and OS cant deal with the current tree structure, thus will erase all harddrive on install. So this is more like a one time thing. To prevent our system going down for to long, i want to copy it fast, as it has to go back and forth, for all 3 servers.
    – Roffel
    Jun 19, 2015 at 7:24

1 Answer 1

6

When copying large amounts of files I usually use these commands:

Target:

nc -q 1 -l 1234 | pv -pterb -s <filesize>G | tar xv

Source:

tar cv <DIR>  | nc -q 1 <targetip> 1234

This will directly stream all the data, without a lot of protocol overhead from source to target over port 1234. This proved to be the fastest way for me to copy the data in a local network. As an addition I added the pv command in the target, so I can get a rough overview how far along the files are.

For a more advanced, yet maybe slower transfer, I'd recommend using rsync.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .