The perfect storm of data loss: a cautionary tale

(The following item originally appeared on our MacUser blog.)

As the old saying goes, you learn something new everyday. But sometimes you learn something and forget something. Yesterday was one of those days. While writing up VoodooPad 4, I thought I’d open up some of my VoodooPad files to check out the new features.

I mostly use VP files to keep track of notes and ideas for fiction writing that I do in my spare time; it’s great for building connections between different concepts and characters. And as I used to maintain multiple computers, I’d gotten into the habit of using my iDisk for file storage so I could access said files no matter where I was. As an added benefit, I figured that would give me multiple synchronized copies of files in different locations to help safeguard against data loss.

I’d also assumed that Apple itself backed up the iDisk files, as most large server installations do. This wasn’t blind assumption either; back in 2005, Apple recovered files that had disappeared from my iDisk:

We isolated an issue with your iDisk, which is now fixed and we recovered the missing items for your iDisk Documents folder. To access the recovered data, please connect to your iDisk and navigate to the Recovered Items folder that is now present in your iDisk Documents folder. Please note there may be duplicate copies of some of the files.

Fast-forward to yesterday. I open up the VoodooPad files on my iDisk only to find they won’t open. And when I check them out in the Finder, I discover that’s because they register as 0KB in size.

Read more...

I should mention that I usually work directly off those files on my iDisk: it appears in the Finder like any other mountable volume, so why shouldn’t it be treated as such? Now, I had encountered sync errors from time to time while working on those files, but if you’ve used an iDisk for any reasonable amount of time, you know that sync errors come up a fair amount. At one point, when presented with the option to choose between two copies of one of my VP files, one from .Mac (as it was at the time) and one from my Mac, I made sure to keep both, just in case.

That’s why it was all the more surprising to end up with suddenly empty files yesterday. So I headed over to MobileMe Support, figuring I would fire off an email and see if they could help me out. Now, despite the many problems MobileMe users have run into, I’ve only had to contact support a couple times in the many years I’ve been a subscriber. So my next surprise of the day was that MobileMe support email is a thing of the past; your only option now is to chat online with a support rep.

With no other option, I jumped into a chat with MobileMe rep Cody H. After explaining my problem, Cody offered this diagnosis:

Cody H.: Okay, I do not recommend running a database, such as Voodoopad, on your MobileMe iDisk. The iDisk is designed to hold data that is static, which means photos, movies, and other files that users can view and download.

Cody H.: The iDisk is not designed to hold files that users actively manipulate, such as a database. You will find that using a database on a local hard disk is much faster than using the database on the iDisk.

Dan Moren: What’s the difference between that and, say, a text file that you edit on the disk?

Cody H.: Manipulating files while they reside on the iDisk, especially when using iDisk Sync could lead to unusable files if computer tries to sync while you are making changes to that file.

That, as I said to Cody, was news to me. I’d been editing files on the iDisk for years, and while I’m not a filesystem engineer, you could say that I’m reasonably computer savvy. Maybe I should have known better, but the past is the past.

That being surprise number three, we move on to surprise number four—the biggie:

Cody H.: I do apologize that you have lost data on your iDisk due to this issue. Please remember that to protect your privacy, Apple does not maintain any data that MobileMe members have manually deleted from the MobileMe servers. This includes files deleted, whether normally or due to corruption, from the iDisk and messages deleted from MobileMe Mail. Preserving data is the responsibility of each MobileMe member.

Cody H.: Please see the “Limitation of Liability” section of the MobileMe Terms of Service.

Ouch. Not good. Well. I’m not a complete idiot, so I do use Time Machine to backup my laptop, which is my main computer. Cody had earlier intimated that maybe that could help.

Er. Surprise number five.

Dan Moren: In terms of trying to recover iDisk files from a Time Machine backup, how would that be accomplished?

Cody H.: Unfortunately Time Machine may not backup the local iDisk, but any files saved to your computer should be backed up. You can contact AppleCare for further information on how to restore information using Time Machine.

Now, this dislodged another piece of information in my brain from last year when Leopard rolled out. In previous versions of OS X, your iDisk was stored as a disk image (DMG) in your Library. This lead to the problem of the miraculously inflating iDisk. Apple sort of compensated for that issue in Leopard by switching to a “sparsebundle” format that can dynamically shift its size.

The other problem with the iDisk image is backing up with Time Machine. Normally, TM checks to see if a file has been altered, and if it has, backs it up. But if you change one file on your iDisk, it can change the entire disk image, which could lead you to backing up a multi-gigabyte file every time you change a single file.

But the sparsebundle format should address that too: Time Machine can look at the file as 8MB-sized “bands” and back up only those bands that have changed. The problem there is that it renders Time Machine useless if you want to go back and recover an individual file. All you can recover is the most recent entire image of your iDisk. If you want to recover an earlier version of any single file, you’re out of luck unless you’ve engaged in some elaborate trickery.

So despite feeling secure in the knowledge that I had multiple copies of these valuable files, the truth ended up being that I had multiple copies of one corrupt version of the same file. Which, as it turns out, is actually worse than useless. It’s more like “maniacal rage-inducing.”

That, then, is my tragic tale of woe and…tragedy. If you take anything away from it, I hope that it is this: don’t rely on your iDisk for backups of critical data. Make sure that you have a copy on a local disk at all times, lest you end up crushed and bereft, like me. I imagine the chances that you’ll be in this exact same scenario are probably slim: I seem to have an encountered a perfect storm of data loss, certainly at least partially of my own making.

I don’t know if there’re things Apple could do to improve the situation so that other people don’t fall through the cracks in the system as I did. From a technical standpoint, I’d sure like to see them have a more robust system for handling working off files on the iDisk, as well as being a little more upfront about what the limitations of the system are.

From my perspective, this situation has encouraged me to take ownership of my data and do an extensive overhaul of the many, many documents I have scattered around myriad computers and hard drives. My goal is to never run into this problem ever again. Digital data has become an increasingly integral part of our lives, and losing it can be as frustrating as losing physical possessions. Take it from me.

To comment on this article and other Macworld content, visit our Facebook page or our Twitter feed.
Shop Tech Products at Amazon