1

bsdiff uses max(17*n,9*n+m) bytes of memory (where n is the size of the old file and m is the size of the new file) according to http://www.daemonology.net/bsdiff/

Is there a lighter (in terms of memory) alternative for me given the following circumstances?

  • The changes will only take place when I am aware
  • I have sudo power
  • The file is between 1 GiB and 10 GiB

My use:

  • Copy file from external drive to tmpfs
  • Many, many writes will take place on the file in the tmpfs (I do not know where in the large binary the writes will take place)
  • Make the file in the external drive the same as the one in the tmpfs

Right now, I have two solutions:

  • Run bsdiff to get a patch (and apply the patch to the file in an external drive) - this does not work for me; the computer I am using does not have enough memory for running bsdiff on this large file.
  • Copy the whole file over to the external drive. This works, but uses up the flash erase (remember, I am working with large files) and takes a long time (redundant copying).

Note: I am able to have both the "original copy" (from the external drive) and the "working copy" (same as the one from the external drive, but will be making many changes to this file) in the tmpfs.

Would monitoring changes as they are made to the file and building a patch from that be possible (and also more memory efficient)? If so, how?

|improve this question
  • Related: superuser.com/questions/125376/…; stackoverflow.com/questions/12118403/… – Melebius Aug 29 '18 at 7:56
  • if all you need is to know if they are different compare md5 hashes? I would assume that is going to be quicker than a diff (but I will let you test that ;-) ) – Rinzwind Oct 21 '18 at 4:44
  • @Rinzwind Thanks for your suggestion, but I need to apply the changes back to the "master copy" after use – tfstwbbnb Oct 21 '18 at 4:45
  • Ok. The next method I would choose would be to redirect those patches to a backup file and then execute those patches on the original. But that might will require code to be inserted into your software. That we can not help with since we can not look into your software. It could be as simple as a cat {patch} >> /dir/to/patchfile.txt ;-) – Rinzwind Oct 21 '18 at 5:09
  • @Rinzwind I might be misunderstanding your suggestion (or maybe you're misinterpreting my question?). I'm trying to get the difference (generate a patch) between the master copy and the used version. I am able to apply the patch to the master copy once I get the patch. My question is how I can generate such a patch on large binary files without using excessive memory. – tfstwbbnb Oct 21 '18 at 16:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.