Cloud Computing Is Like the Food Industry and Needs to Behave the Same Way
It was a big day today for cloud computing. At least it certainly seems that way after two of the most egregious issues witnessed in cloud computing service provisioning, both emerging in the past 24 hours. I feel compelled to make some comments in response to the events of Distribute.IT’s catastrophic and irrecoverable data loss and DropBox’s temporary lapse of control over their security system, allowing any user’s password to access any user’s account.
Let’s first of all recap what happened.
- DropBox had an incident they described as an “Authentication Bug” resulting from a code update released that allowed any user to log into any account using any password. The security hole was open for almost four hours.
- Distribute.IT, who provide web hosting services had what they described as a “deliberate, premeditated and targeted attack ” on their network. This impacted pretty much their entire business infrastructure and four major servers were irrecoverably lost including all “production data, key backups, snapshots and other information that would allow us to reconstruct these Servers from the remaining data.” The upshot: more than 4800 customer websites, data and email reservoirs eradicated.
- DropBox, whose business is based on storing people’s private files, cannot afford to let something like that happen – it is tantamount to a physical storage facility providing master keys to all their clients that can open anyone’s storage area. Or the post office providing master keys to all the post office boxes. Storage privacy preservation is the special thing they do – it is not a sideline – it is part of their core competence. So any breach of that functionality is a breach of confidence, a breach of trust. The fact that the fix took five minutes means that it should never have been overlooked. Change management around core competency is critical and no doubt they are having a good hard look at their processes. I dare say a repeat performance might just kill them.
- Distribute.IT: I really am lost for words. OK they are victims, it would appear, of a pretty malicious attack, but where are the offsite backups? How is it conceivable that the company cannot recover almost 5000 customer websites? Many of these businesses will have lost everything as a result of this, naively trusting their data to this provider. Easy for a professional like me to say they had it coming, but most of the clients will undoubtedly have been small business with no idea of Information Technology and would have entrusted this data to the so-called professionals to look after it. There is simply no excuse for a professional service provider to not have this most fundamental control in place – what about fire? The only thing I can think of is that they have the backups but do not have a viable means of recovery that is cost effective en-masse. I simply cannot accept that they don’t have offsite backups. (As an aside, naming the servers that failed Drought, Hurricane, Blizzard and Cyclone is a secondary PR disaster).
In my post dealing with the Amazon outage, I wrote that CIOs need to take personal responsibility for their sites being down – the fact that their underlying infrastructure provider is down is irrelevant. And in the spirit of that, these customers should have personal backups, and certainly some of them do have according to the comments on the website. But the Amazon outage was about pushing the envelope, pushing cloud boundaries. This is different – this is a failure of fundamentals that have been known and understood for 20+ years. I mean, who doesn’t understand the need for offsite backups?
After a day like this it is much harder to be as sanguine about cloud computing in a broadbrush way: I feel today the Cloud lost its innocence. Something very ugly happened and nothing will be quite the same again. Like asking a builder for their insurance certificate of currency, like asking a potential supplier to provide references, like checking a new employee’s resume claims. Perhaps today is the day the Cloud comes of age, the day when we separate out the cowboys from the pros.
Cloud Computing is like the food industry – we go to a restaurant and we trust that the food has not been handled by people who have been to the toilet, that the produce is fresh, that the cooks haven’t entertained themselves at risk to our health. We have systems in Australia and I am sure elsewhere that enable health inspections and public reporting of those who fail them. We have common and statutory law that ensures food is handled appropriately from the farm to the table and that people are punished when things go wrong. We even have public registers that shame bad examples of food processing facilities and restaurants when things go wrong or inspections are failed badly.
I like the fact that DropBox publicised their problem – transparency is good, but you have to have the fundamentals right. And when food producers let rats grow, they should be banned. The same thing applies to businesses making money out of protecting or managing people’s data.
Cloud81 
Great article and what a business opportunity – being the industry oversight and compliance watchdog – either industry or government backed!
Thanks Neville,
An interesting idea – perhaps one could call oneself a technometeorologist.
But seriously, this is something that is needed and a watchdog sponsored by the client side rather than a certification coming from the vendor side might be an interesting idea here. I will give this some thought.
Hi Alan, I would say the industry needs to be more like the aerospace industry. In reality lives may well depend on the infrastructure and there needs to be the same learning from mistakes. It’s partly why ultimately I see a handful of big infrastructure / platform providers and the likes of Distribute IT will no longer do things on such a small scale – it won’t be viable to meet the level of protection needed.
I’m surprised overall there hasn’t been much more of this – there’s a gold rush going on and mistakes happen. People get things out to market quickly and they forget the basics (like paying the bills!). We partnered with Salesforce because all of those sorts of issues as close to being non-issues as there possibly could be right now. Until others catch up on the level of security and good management they have a clear advantage over the rest IMHO.
incidents like this are all part of the natural evolution of internet based computing. Without the messes to learn from there wouldn’t eventually be the rock-solid infrastructure that will come out of it. The lesson for any business is know the risks and mitigate against them. If you don’t think you can fully understand the risks with your provider, it’s time to look elsewhere.
Ian Ryder
CEO, appiChar Australia
A great comment Ian, thank you for it.
great article.Thanks