I am trying deduplication with sdfs.(ubuntu 13.10 on Vm).

(http://opendedup.org/quickstart) followed the same steps as this

mounted the file system on /media/pool0/

When copied a directory containing some similar pdf's its not removing the duplicates and the size of the folder remains the same as the original.

Original directory size 1.6G /Worksheet/ 1.6G total

Sdfs directory size 1.7G /media/pool0/Worksheet/ 1.7G tota

Is there any specific commands for enabling deduplicaiton feature in the SDFS volume. Or will dedpulication get applied to the files that we copy into the volume.?

Can some1 please also explain me how sdfs file system works?

Thanks in advance

  • Please let me know if u require anymore details – Sharath Jan 27 '14 at 4:43

In sdfs, there are two ways of deduplication used:

  • Fixed block deduplication

    • Requires aligned blocks of 4096 bytes to be the same
    • That will just not happen often with your data.
  • Variable block deduplication

    • Also requires blocks of 4096 bytes to be the same, but they do not need to be aligned.
    • I assume that your test files ("Similar PDF files") have only very few equal ranges of 4096 bytes or more.

So for your kind of data,
there is just no deduplication available!
Still, I expect it to work really well with the data they had in mind to support:

From Administration Guide for Version 2.0:

Fixed and Variable Block Deduplication

SDFS Can perform both fixed and variable block deduplication. Fixed block deduplication takes fixed blocks of data and hashes those blocks. Variable block deduplication attempts to find natural breaks within stream of data an creates variable blocks at those break points.

Fixed block deduplication is performed at volume defined fixed byte buffers within SDFS. These fixed blocks are defined when the volume is created and is set at 4k by default but can be set to a maximum value of 128k. Fixed block deduplication is very useful for active structured data such as running VMDKs or Databases. Fixed block deduplication is simple to perform and can therefore be very fast for most applications.

Variable block deduplication is performed using Rabin Window Borders (http://en.wikipedia.org/wiki/Rabin_fingerprint). SDFS uses fixed buffers of 128k and then runs a rolling hash across that buffer to find natural breaks. The minimum size of a variable block is 4k and the maximum size is 128k. Variable block deduplication is very good at finding deduplicate blocks in unstructured data such as uncompressed tar files and documents. Variable Block deduplication typically will create blocks of 10k-16k. This makes Variable block deduplication more scalable than fixed block deduplication when it is performed at 4k block sizes. The downside of Variable block deduplication is that it can be computationally intensive and sometimes slower for write processing.

Variable block deduplication can only enabled when the volume is created using the --hash-type=VARIABLE_MURMUR3.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.