Mac 911: Why is my Mac full of a million tiny files?

Last time, I talked about the “dark matter” of your iOS devices and OS X systems: stuff that seems to occupy space without offering up information about why. In this Mac 911, I’ll answer more questions about storage and backup.

As many as grains of sand on the beach

Doug Eldred writes in with a concern about a form of file bloat—but not about bloated sizes. Rather, the sheer number of items that seem to appear on his drive.

Whatever or wherever they are, they must not be terribly large, but my Mac used to have less than a million files (according to various tools, including SuperDuper!), and now it’s [up] to 1.3 million. Trust me, I haven’t knowingly created 300K new file recently!

Mac OS X (and, invisibly, iOS) has always had an inordinate quantity of files because of its Unix underpinnings. There is something about Unix that loves a multiplicity of tiny files rather than monolithic larger ones, hence these huge counts.

In the way back, in the long ago, it used to matter, because each file consumed a minimum amount of hard disk space regardless of the actual amount of data in it, and no more than 65,536 files on the drive. With HFS+, however, a drive can have nearly 4.3 billion files without each consuming unreasonable amounts of space.

Doug continues:

Those zillions of stray files don’t seem to consume much space, but tools like SuperDuper! and Disk Utility “verify disk” need to process each and every one of them in one way or another.

To my recollection and experience, the number of files shouldn’t contribute to any system slowdowns, because they’re inert unless needed. But during the cases that Doug mentions, could it cause delays? And why are there all these files being created, anyway?

Shirt Pocket

I turned to an expert: Dave Nanian, the founder of Shirt Pocket, makers of the SuperDuper! app that Doug uses. SuperDuper! is a disk backup utility that makes an exact, bootable clone of a drive, and I have it scheduled to run every night on my main computer in addition to two other forms of incremental document backup.

Dave writes:

Ah, 1.3 million files. I remember those early days, back when laptops were made of plastic, and chips by IBM. But today, you’re not alone. The laptop I’m typing on right now has—according to Disk Utility—6,709,682 files. Make that, two seconds later, 6,709,687.

In general, especially if you’re not seeing excessive disk usage, this just isn’t something to worry about. There are a lot of hidden folders on your drive, sometimes entire hierarchies that are emulating things that can’t be directly expressed in HFS+, but are still counted as “files,” even though you wouldn’t ever access them that way.

Apple implemented Time Machine in a very peculiar way by most developers’ reckoning to create an exact copy without duplicating every file. Using so-called hard links, which allows a single copy of data to appear multiple times in a folder hierarchy, Time Machine creates hidden folders that fully replicate a hard disk’s file structure. A Time Machine backup comprises mostly hard links pointing to existing files or folders unchanged since the previous backup. Each hard link is counted by OS X as a separate file.

Dave notes that Time Machine also stores a sort of local backup on the startup drive to improve its performance in copying to a local drive or a remote one over the network. Most other backup software makes an initial copy of a file and then a “delta,” or a kind of summary of differences, for every version stored thereafter.

And Spotlight contributes to the file count: for better indexing, Mac apps have reference files for each quantum of data, such as an email message, to match a result up with an item. On one machine, I have hundreds of thousands of metadata files associated with email messages. Combine Spotlight and Time Machine, and you can see where the file count comes from. I don’t use Time Machine and have nearly 1.8 million files on my MacBook Air.

Dave has consoling words, though:

I wouldn’t worry about it too much — let the system handle its files, and don’t be too concerned about the count. As long as your drive isn’t mysteriously filling up, you’re good.

Selective Time Machine

On the related suject of Time Machine, Larry Landen asks:

When I reinstall Mac OS X (to solve performance problems, or clear space) I have a restore option to migrate data from my Time Capsule, but it currently only gives me options to migrate entire user profiles, applications, and settings.

How may I pull only selected data (such as images, and iTunes media like music and movies, and documents) rather than an entire profile? I fear that restoring an entire profile may also restore any problems that made me want to reinstall OS X.

It’s a sensible and reasonable question: if some kind of corrupted or inaccurate setting is causing system problems, aren’t you just asking for trouble by bringing all your settings over? Likely no. With the except of specific applications having corrupted configuration files, a clean installation and a migration of settings generally seems to avoid causing identical problems in OS X. (Solving corrupt app settings varies, but often involves tossing a file or several from ~Library/Preferences/ with the advice of the software’s maker.)

That’s in part because the corruption or other setting issues you’re having can result from missing or overwritten system files or configurations that aren’t copied back as part of a Time Machine migration.

With Migration Assistant, however, which you can launch on a computer after you’ve reinstalled and updated OS X, you can select certain sets of data to reimport, and can omit all your settings.

Another option, is to use the aforementioned SuperDuper! or Carbon Copy Cloner to back up a volume, or software/service combinations like Backblaze or Crashplan to specific files or folders. Then you can more easily pick and choose what to restore.

Ask Mac 911

You can email things that perplex you or need solving to mac911@macworld.com, tweet them at me (if brief) @glennf, or call 206–337–5833 and leave a voicemail. (We’ll be experimenting with some audio in the future, and may put your question “on the air.”)