Someone posted on Elektronauts this month with a memory problem. He was running a Mac companion app against a sampler card holding 64,417 files, and Activity Monitor showed it climbing about 100MB every refresh. It looked like a leak.
It wasn't. Nothing was growing without limit. One file was being read into memory in full, and that file happened to be large. The number looked like a leak because the reads never stopped coming.
The stage it happened in was the duplicate check. That part is worth sitting with, because duplicate detection is one of those problems that looks trivial, and then quietly becomes the most expensive thing your app does.
Hashing everything costs you the whole disk
Hash every file, group the matching digests, and you have found every duplicate. It is four lines of code and it is correct. It is also O(every byte you own). On a 200GB sample library that is 200GB of reads before you can tell the user anything, and you paid it to discover that almost nothing was a duplicate.
The fix is not a faster hash. The fix is not hashing most of the files at all.
Size is free
Two files with different byte counts cannot be identical. The filesystem already knows every file's size, so grouping by size costs no reads at all. You get it for free.
On a normal folder this kills most of the work immediately. Sizes are effectively random, collisions are rare, and the vast majority of files land alone in their bucket and are proven unique without ever being opened.
Then hash a little
For whatever survives, do not jump to the full hash. Read the first 100k or so and hash that. Files that share a size but differ in their first hundred kilobytes are done, and they cost you one small read instead of a whole file.
Only when the prefixes match do you pay for a full hash. In practice almost nothing gets that far, and the things that do are usually genuine duplicates, which means the expensive read was the one you actually wanted.
Three tiers: free, cheap, expensive. Most files stop at free.
Sample libraries break the usual assumption
Here is where it gets interesting, and where general-purpose advice starts lying to you.
The size-grouping tier assumes sizes are spread out. In a folder of documents, they are. In a sample library, they are not.
One shots and stems get bounced at fixed bar lengths. A pack of 16-bar loops at the same sample rate and bit depth produces files with identical byte counts, over and over, by construction. Kits are worse. Export a hundred one shots trimmed to the same length and you have a hundred files that are byte-for-byte the same size and completely different sounds.
So instead of a clean spread of tiny buckets, you get a handful of very fat ones. The developer in that thread checked his own test card and found 8,000 files of identical size sitting in a single bucket. Without a prefix check, every one of those gets read end to end to prove it is not a copy of the other 7,999.
That is the whole bug, and it only shows up on the kind of data this software exists to handle. You will not find it on your Documents folder.
The allocation is the other half
Even after you have cut the reads down, how you read matters.
Pulling a file into memory to hash it means allocating a buffer the size of the file. Do that for a 300MB stereo stem and you have a 300MB allocation that exists for the length of one hash. Do it in a loop and the allocator spends the whole scan handing back memory and taking it again, which is exactly the sawtooth that looks like a leak in a profiler.
Stream instead. One buffer, allocated once, reused for every file. The hash does not care whether it received the bytes all at once, and a fixed 1MB buffer will hash a 300MB file just as correctly as a 300MB one will.
And then the bookkeeping
At 64,000 entries, the accounting starts to cost real memory on its own, separate from any file content.
A digest is 32 raw bytes. Written as a hex string it is 64 characters, and as a string object it carries overhead on top of that. Put those in a map keyed by path and you are storing two allocated objects per file for something that could have been 32 bytes hanging off a row you already have.
Multiply by 64,000. It is not the headline number, but it is free to avoid and it stops being free to fix later.
Why it hides
None of this is new. Size bucketing, prefix hashing and buffer reuse have all been around for decades, and anyone who reads them will recognize them.
What makes the duplicate pass worth a second look is that it stays invisible until the data gets strange. On ordinary files the size tier does its job and nothing downstream ever costs much. It takes a folder full of material bounced to the same length before the buckets go lopsided and the reads start adding up, and that is not a shape you go looking for. It arrives with the library.
Discover hidden gems in your sample folder
Sample Miner turns the sample folders you already own into playable drum kits and instruments — entirely on your machine.