If it is used, the re-assembled file system will likely not be identical as the original one, because unused space is rarely ever filled with zeros.

Index empty space: Read the NTFS bitmap ( $Bitmap), find unused sections and mark them as sparse.For each chunk checksum, compare if it’s in the global chunk index and only store new chunks. 32M sized chunks and then hash each chunk using blake2b. For each entry, read all data runs (extents, fragments) sequentially (i.e. Index files within NTFS: Read all entries in the master file table ( $MFT) that are larger than X (e.g.Instead of fixed offset chunking with 4K chunks, I decided to parse the NTFS filesystem using a combination of content aware chunking (for the file system itself) and fixed offset chunking (for the files within the file system).

Since this was a proof-of-concept, and I wanted to learn about NTFS and write some fun Go code, I chose to go a different route: That said, given that we’re talking about hundreds of thousands (maybe even millions? I should really check …) of NTFS file systems, it might not matter if the objects are 4K or 128K in size. On top of that, many object stores or file systems are not really good with small files like that, so a higher chunk size is desirable. In addition to that, the more chunks you produce the higher the chance of hash collisions (unless you increase the hash size, which in turn increases the metadata overhead …). While that will lead to the best dedup results, it also produces a lot of metadata and the metadata-to-data ratio (metadata overhead) is pretty bad. The obvious approach to dedup NTFS would be to simply dedup blocks (= clusters), so typically that would mean deduping with a fixed offset of 4096 bytes (4K cluster size). If you’re impatient, here’s a link to the fsdup tool I created as part of this proof-of-concept.

File deduplicator