0

I have a file on my backup drive that is foo.txt (30 GiB).

I have a file on my regular drive that is foo.txt (60 GiB, first 30 GiB is guaranteed to be exactly the same).

How can I only append the missing part without recopying the whole file?

Maybe something with dd would work?

2

To sync files there’s rsync, it has an --append option to “append data onto shorter files”:

rsync --append /path/to/foo.txt /path/to/foo.txt
#              ^- original      ^- copy

Example run

Test scenario flagrantly copied from steeldriver’s answer – I added the -P and -v options for verbose output.

$ dd if=/dev/urandom bs=1M iflag=fullblock count=60 of=origfile
60+0 records in
60+0 records out
62914560 bytes (63 MB, 60 MiB) copied, 0.328983 s, 191 MB/s
$ dd if=origfile bs=1M iflag=fullblock count=30 of=newfile
30+0 records in
30+0 records out
31457280 bytes (31 MB, 30 MiB) copied, 0.0292976 s, 1.1 GB/s
$ cmp origfile newfile
cmp: EOF on newfile
$ rsync -Pv --append origfile newfile
origfile
     62,914,560 100%  365.47MB/s    0:00:00 (xfr#1, to-chk=0/1)

sent 31,465,039 bytes  received 35 bytes  20,976,716.00 bytes/sec
total size is 62,914,560  speedup is 2.00
$ cmp origfile newfile
$ 
| improve this answer | |
  • Nice - I thought there should be a way to do it with rsync – steeldriver Jun 4 '18 at 12:02
2

Yes you can use dd - the trick is to choose both bs x skip and obs x seek equal to the exact required offset

Ex.

First lets generate a test file - I've chosen 60MiB rather than 60GiB for the sake of illustration:

$ dd if=/dev/urandom bs=1M iflag=fullblock count=60 of=origfile
60+0 records in
60+0 records out
62914560 bytes (63 MB, 60 MiB) copied, 0.376846 s, 167 MB/s

Now lets copy exactly the first half of it - again using dd (though that's not required)

$ dd if=origfile bs=1M iflag=fullblock count=30 of=newfile
30+0 records in
30+0 records out
31457280 bytes (31 MB, 30 MiB) copied, 0.063891 s, 492 MB/s

Verify that they're different:

$ cmp origfile newfile
cmp: EOF on newfile after byte 31457280, in line 122106

Now let's copy from origfile to newfile, skipping the first 30 x 1M blocks of both files:

$ dd if=origfile bs=1M iflag=fullblock skip=30 count=30 of=newfile seek=30
30+0 records in
30+0 records out
31457280 bytes (31 MB, 30 MiB) copied, 0.0632964 s, 497 MB/s

Finally, verify that the files are now identical:

$ cmp origfile newfile
$ 
| improve this answer | |
0

That is a huge file, but if it handles the size, you could use the split command to split the 60GB file into file1 and file2. Then use cat to put the parts you want back together.

Example:

split -n2 60Gfile
cat xab >> 30Gfile

split -n2 splits the file in half and makes 2 files called xaa and xab

If this doesn't do what you want, read the split manual as the command has other options.

| improve this answer | |
  • Now I know about another command. To future readers: split -n 2/2 foo.txt outputs second half of file to stdout; this is useful to pair with other commands. --byte option splits every n bytes. – tfstwbbnb Jun 4 '18 at 2:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.