Cloud vs On-Premise – A False Dichotomy
I often hear people talking about Cloud Computing as an opposite to On-Premise. This is based on an incorrect assumption that one of the key characteristics of Cloud is that it is delivered from off-premise.
The above diagram shows the perception people have – that there is a choice to be made – choosing Cloud means sacrificing on premise, and choosing on-premise means sacrificing Cloud.
The reality is more like the above – we can choose for any workload whether to adopt Cloud or use more traditional approaches, and we can, separately, choose whether to run these work loads on -premise or elsewhere. This is not an all-or nothing proposition.
Speaking of all-or-nothing propositions, I really like what Gartner’s Lydia Leong had to say about bimodal IT. Her message is that we should not try to compromise our IT practices – use best-practice traditional approaches for traditional workloads, and best-practice “Cloudy” approaches for new workloads based on the New Style of IT. One of the biggest causes of difficulty in adopting new approaches to IT delivery is creating an anemic set of practices that can’t do anything well.
Go All-in when you can, and recognize that despite what the public-cloud-only vendors would have you believe, the location of the infrastructure is a completely independent decision from whether you choose to Go Cloud.
DevOps: the Solution to Disintermediation
Around 1986/87, I approached a motorcycle courier and asked him, “what impact, if any, has the fax machine had on your business”. His reply surprised me: “It has been the best thing that ever happened – people now expect instant delivery, but the fax is not a valid original”.
At first glance, I thought this was counter-intuitive – I expected the fax machine to kill the couriers, or to have no impact – I certainly didn’t expect it to have the positive impact he described. Suddenly people knew they could get their hands on a facsimile of a document instantaneously, and so they came to expect instant gratification, instant delivery. And so, they would call on couriers rather than use the postal service to ensure they got the documents quickly. The fax machine changed the paradigm.
Similarly, the proliferation of SaaS (Software as a Service) offerings like Salesforce, Yammer, Marketo, GMail, Clarizen, Zoho, Zuora etc have changed end users’ expectations of what is an acceptable timeframe for delivery of new systems or modifications. Marketing, Finance, HR and other teams now expect new functionality to be available in days/weeks/months rather than quarters and years. They are making new demands of their IT teams that are in the eyes of traditional CIOs unconscionable, impossible, reckless. This leads to frustration and tension, and a strong desire to bypass the IT department – a process that has become known as disintermediation: cutting out the middle man.
This process is so well established that Gartner stated in early 2012 that by 2017 marketing departments will be spending more on IT systems than CIOs. The trouble is that marketers are good at marketing, not IT, and so they will end up creating siloed “crapplications” rather than extensions of a single source of truth. There will be a plethora of disjoint standalone copies of customer databases, product catalogs, charts of accounts etc distributed through the enterprise.
CIOs can well look at this and feel frustrated and concerned – how do they convey the risk being taken by bypassing the architectural Design in order to get instant gratification? How do they get back in control of the systems agenda? How do they learn to respond to the accelerated expectations of business departments that cannot afford to stand by and watch their business erode by far more responsive newcomers?
The answer, I believe, largely lies in moving to a DevOps model where the dichotomy between software development and operations infrastructure provisioning disappears, and hardware and any required software is provisioned on demand by scripting and automation. The DevOps approach leads to continuous deployment – in some cases up to 75 times per day. This model aligns with agile programming techniques like Scrum, and supports the idea of disposable “cattle” hardware provisioning so synonymous with Cloud and Big Data.
Critics may argue that such approaches are risky, even reckless, that new versions must be thoroughly tested before seeing the light of day. New systems and changes to hardware environments must go through rigorous testing before being published. This has consequences for change management complexity and thus risk: keeping development, staging and production environments in sync is difficult when major changes are bundled together and released in one large batch.
It is perhaps paradoxical that the systems developed using DevOps principles will be delivered faster at lower risk. This is due to the combination of:
- Being able to continuously deploy updates using automated deployment systems like Jenkins;
- Procuring scripted environment using the likes of Chef, Puppet, Saltstack or Ansible that guarantee all the environments (development, staging and production) are materially the same;
- Autonomous failover and recovery systems that know how to self-repair or self-replace parts of the environment in real time;
- Agile software practices that are designed to iteratively deal with intrinsic and extrinsic problems, scaling needs, and are capable of responding to changes in direction;
- Systems being designed with testing as a first foundation rather than an afterthought; and
- Systems being designed from the ground up to cope with fallible, unreliable infrastructure.
CIOs often want to wait “until Cloud matures” before they move workloads or change practices. Such wait-and-see approach can doom them to fail, or at least consign them to a much more difficult road into the new world. Marketers and other functional heads are not going to wait, and once the systems do become disparate and disconnected as a result of them going it alone, it will become increasingly difficult for the CIOs to introduce DevOps processes and re-establish themselves as the credible go-to resource for any form of IT related system.
Unfortunately too many CIOs have been convinced by Cloud vendors that claim that the only true Cloud is a public multi-tenanted one. I myself felt that way in the early days. The reality is that the location of the system is a secondary matter. Cloud and DevOps principles apply regardless of whether the systems are on premise or off premise. Too many CIOs see their biggest challenge of deciding whether to run IT on premise or in the Cloud, but this is a false dichotomy, a false choice – it is not a question of on-premise OR Cloud – there are two choices: one a choice of on-premise or off-premise and the other a choice of Cloud or not Cloud, DevOps or not DevOps.
Chief Marketing Officers, Financial Officers, HR Officers are not going to wait for their visions to be implemented by IT teams encumbered by old practices – nor should they. They need to be able to rely on their CIOs to deliver what they need when they need it. DevOps and Cloud technologies will empower the IT departments to be enablers leading the organisation forward into the new world instead of roadblocks preventing the business from moving ahead.
The Internet of Things: Interconnectedness is the key
I was at an Internet of Things event a couple of weeks ago and listening to the examples it was clear there is too much focus on connecting devices, and not enough focus on interconnecting devices.
Connecting devices implies building devices that are designed specifically to work within a closed ecosystem, to report back to some central hub that manages the relationship with the purpose-built device. Interconnected devices are designed in such a way that they can learn to collaborate with devices they were never designed to work with and react to events of interest to them.
So what will this look like? For one possible scenario, let’s start with the ubiquitous “smart fridge” example and expand this to look at the way we buy our food. There has been talk for years about how fridges will be telling us about the contents, how old they are, whether anything in them has been reserved for a special meal, what is on the shopping list etc. Even to the idea of placing automatic orders with the food suppliers, but what if we want to still be involved in the physical purchasing process, how will the Internet of Things, with interconnected devices work in that scenario? Here’s a chain of steps involved:
- Assuming our fridge is the central point for our shopping list, and we want to physically do the shopping ourselves, we can tap the fridge with our phones and the shopping list will be transferred to the phone.
- The fridge or our phone can tell us how busy the nearby supermarkets currently are, and based on regular shopping patterns, how many people will likely be there at certain times in the immediate future. Sensors in the checkout will let us know what the average time is for people to be cleared. Any specials that we regularly buy will be listed for us to help make the decision about which store to visit.
- We go to the supermarket and the first thing that happens is the supermarket re-orders our shopping list in accordance with the layout of the store.
- The phone notifies our family members that we are at the supermarket and lets them know we are there so they can modify our shopping list.
- We get a shopping trolley, which immediately introduces itself to our phone. It checks with our preferences in the phone as to whether we want its assistance, whether it is allowed to record our shopping experience for our use, or to assist the store with store planning
- As we walk around the store, the phone or the trolley alerts us to the fact that we are near one of the items on our shopping list.
- If we have allowed it, the trolley can make recommendations based on our shopping list of related products, compatible recipes, with current costs, and offer to place the additional products into the shopping list on the phone and even into our shopping list template stored in the fridge if we want.
- As we make our way to the checkout, the trolley checks its contents against what is on our shopping list and alerts us to anything missing. Clever incentives might also be offered at this time based on the current purchase.
- As soon as the trolley is told by the cash register that the goods have been paid for, it will clear its memory, first uploading any pertinent information you have allowed.
- Independent of the shopping experience and the identifiability of the shopper and their habits, the store will be able to store the movements of the trolley through the store, and identify how fast, any stopping points to identify interest and analyse for product placement.
- Once we get home, we stock the cupboard and the fridge, both of which update our shopping list.
- As soon as we put the empty wrapper in the trash, the trash can will read the wrapper and add the item to a provisional entry in the shopping list, unless we have explicitly pre-authorised that product for future purchase.
Another example would be linking an airline live schedule to your alarm clock and taxi booking, to give you extra sleep in the morning if the flight is delayed. Or having your car notify the home that it looks like it is heading home and to have the air conditioner check whether it should turn on.
While we focus only on pre-ordaining the way devices should work during their design, we limit their ability to improve our lives. By building devices that are capable of being interconnected with other devices in ways that can be exploited at run time we open up a world of possibilities we haven’t begun to imagine.
Preparing for the Big Data Revolution
It is no accident that we have recently seen a surge in the amount of interest in big data. Businesses are faced with unprecedented opportunities to understand their customers, achieve efficiencies and predict future trends thanks to the convergence of a number of technologies.
Businesses need to take every opportunity to store everything they can. Lost data represents lost opportunities to understand customer behaviour and interests, drivers for efficiency and industry trends.
A perfect storm
Data storage costs have fallen dramatically. For instance, in 1956 IBM released the first hard disk drive, the RAMAC 305. It allowed the user to store five megabytes of data at a cost of $50,000 – that’s around $435,000 in today’s dollars. In comparison, a four-terabyte drive today can fit in your hand and costs around $180. If you were to build the four-terabyte drive using 1956 technology, it would cost $350 billion and would take up a floor area of 1600 km2 – 2.5 times the area of Singapore. Also, 10-megabyte personal hard drives were advertised circa 1981 for $3398 – that’s $11,000 today, or $4.4 billion for four terabytes.
Gordon Moore’s prediction in 1965 that processing capacity doubles approximately every two years has proved astoundingly accurate. Yet the amount of data we can generate has far outstripped even this exponential growth rate. Data capture has evolved from requiring specialised engineers, then specialised clerical staff, to the point where the interactive web allowed people to capture their own data. While this was a revolutionary step forward in the amount of data we had at our disposal, it pales before the most recent step: the ‘Internet of Things’, which has opened the door for machines to automatically capture huge amounts of data, resulting in a veritable explosion of data, way outstripping Moore’s Law. The result: the data load became too much for our computers, so we simply threw a lot away or stopped looking for new data to store.
With the price of storage decreasing sharply, the economies of storage have meant we can afford to capture more data: it has become increasingly important to find new ways to process all the data being stored at the petabyte scale. A number of technologies have emerged to do this.
Pets versus cattle
Traditionally computer servers were all-important – they were treated like pets. Each server was named and maintained with great attention to ensure that everything was performing as expected. After all, when a server failed, bad things would happen. Under the new model, servers are more like cattle: they are expendable, easily replaced. Parallel processing technologies have superseded monolithic approaches and allow us to take advantage of using many low-cost machines rather than increasingly more powerful central servers.
Hadoop is one project that has emerged to handle very large data sets using the cattle approach. Hadoop uses a ‘divide and conquer’ approach, which enables extremely large workloads to be distributed across multiple computers, with the results brought back together for aggregation once each intermediate step has been performed. To illustrate Hadoop: imagine having a deck of cards and someone asks you to locate the Jack of Diamonds. Under a traditional approach you have to search through the cards until you locate the card. With Hadoop, you can effectively give one card each to 52 people, or four cards each to 13 people, and ask who has the Jack of Diamonds. Much faster and much simpler when complex processes can be broken into manageable steps.
NoSQL, which was intended to mean “not only SQL”, is a collection of database technologies designed to handle large volumes of data – typically with less structure required than in a typical relational database like SQL Server or MySQL. Databases like this are designed to scale out to multiple machines, whereas traditional relational databases are more suited to scaling up on single bigger servers. NoSQL databases can handle semi-structured data; for example, if you need to capture multiple values of one type or obscure values for one person. In a traditional database, the structure of the database is typically more rigid. NoSQL databases are great for handling large workloads but they are typically not designed to handle atomic transactions: relational SQL databases are better designed for workloads where you have to guarantee that all changes are made to the database at the same time, or no changes are made.
Network science
Network science studies the way relationships between nodes develop and behave in complex networks. Network concepts apply in many scenarios; examples include computer networks, telecommunications networks, airports or social networks. Given a randomly growing network, some nodes emerge as the most significant and, like gravity, continue to attract additional connections from new nodes. For example, some airports develop into significant hubs while others are left behind. As an airport grows, with more connections and flights, there are increasingly compelling reasons why new airlines will decide to fly to that airport. Likewise, in social networks, some people are far more influential either due to the number of associations they develop or because of the effectiveness of their communication skills or powers of persuasion.
Big data can help us to identify the important nodes in any contextual network. Games console companies have identified the most popular children in the playground and given them a free console on the basis that they will have a lot of influence over their friends. Epidemiologists can identify significant factors in the spread of diseases by looking at the significant nodes and then take steps to prevent further contamination or plan for contagion. Similarly, marketers can use the same approaches to figure out what is more likely to ‘go viral’.
Benefits
Big data assists businesses to gain a better understanding of customers, treating each customer as an individual – the so-called marketing segment of one. Understanding what moves customers can build strong brand loyalty and evoke an emotional response that can be very powerful. Imagine an airline that recognises that a particular passenger travels from A to B every Monday to Thursday. However, if that passenger plans to stay in B for two weeks, imagine how much loyalty could be generated by offering them a free flight over the weekend to C, a discounted flight for their spouse from A to C, and a discounted hire car and room for the weekend away together.
Digital body language and buying habits can lead online retailers to be able to make astute decisions about what product to offer customers. Target was able to identify pregnant customers very early by their shopping patterns: customers buying certain combinations of cosmetics, magazines, clothes would go on to buy certain maternity products months later.
Big data can be used to drive efficiencies in a business. The freight company UPS, for example, was able to save almost 32 million litres of fuel and shave 147 million km off the distance its trucks travelled in 2011 by placing sensors throughout the trucks. As a side benefit, they learned that the short battery life of their trucks was due to the drivers leaving the headlights on.
By analysing customer relationships, T-Mobile was able to mitigate the risk of a domino effect when one customer decided to leave its service. It did this by identifying the customers who were most closely related digitally to the person churning and making a very attractive offer to those people, preventing the churn from spreading. Further, by analysing people’s billing, call dropout rates and public comments, they were able to act in advance to reduce churn by 50% in a quarter.
CERN conducts physics experiments at the Large Hadron Collider involving sending 3.5 trillion electron volts in each direction around an underground ring, resulting in particle collisions that provide an understanding of the basic building blocks of matter. The Higgs-Boson was proven by analysing the data that was generated in smashing the particles together. 15,000 servers are used to analyse the one petabyte of data that is generated per second and 20 gigabytes is actually stored. This is orchestrated using cloud techniques built on OpenStack and designed and supported by Rackspace.
Conclusion
We have reached a point where it is now better to start storing everything today so that we have a business case for analytical tools tomorrow. Once we start getting used to the idea that everything is available to us, we will find new ways to think about how we leverage our information. The businesses that succeed in the future will be those that constantly look for ways to mine the information they have gleaned.
[This article has been slightly modified from an article I wrote that was previously published in Technology Decisions magazine.]
CIOs: Focusing on Obstacles will Limit your Success with Cloud Computing
No matter where you stand on the new Cloud technologies, there is no escaping the fact that Cloud Computing has everyone’s attention. For some business executives it is seen as an opportunity to financially restructure their IT expenditure. Some focus only on the risks they perceive of placing their data and systems in the hands of some external third party. Still others see it as providing the means to focus on their core business and new business ideas without having to worry about whether the computer infrastructure will be able to cope.
While IT teams must ensure that systems are safe and data is secure, it is ironic that by focusing too much on security and availability, many CIOs are exposing themselves, and their employers, to a far greater risk – the risk of missing the opportunities presented by new technologies emerging from Cloud Computing.
The CIOs who will provide the greatest value to their employer will be those who approach Cloud by asking themselves “what can we now achieve that was previously inconceivable?”
While there are many distracting arguments about what constitutes Cloud, the key characteristic that differentiates it from more traditional approaches is that Cloud provides the freedom to be remarkable – the freedom for a business to focus on what it does best without constraints imposed by infrastructure.
Traditional approaches to IT see the acquisition of dedicated equipment on a project basis, with each new system requiring new equipment and administration. This leads to ever-increasing IT complexity, with the IT department working to prevent things from getting out of hand. In many cases this has led to a perception that the IT department is the problem, and many IT budgets are shifting to marketing as a result. Under a Cloud model, IT should evolve to becoming a reservoir from which new equipment is instantly sourced, and a platform underpinning whatever the business or the marketplace throws at it, scaling up and down to meet changing demands. While traditional approaches add complexity, Cloud provides the freedom to focus on the business imperatives.
IT leaders who embrace Cloud computing as an enabler will not be seen as roadblocks by the marketing or sales departments. Especially when they adopt Open Cloud approaches such as OpenStack that overcome vendor lock-in and allow for hosting data on-premise, off-premise, or a mix of the two.
Cloud computing enables businesses to take advantage of the relationships their customers have with each other. For the cost of a coffee, businesses are able to experiment with new technologies by renting computers for the few hours it might take to trial an idea. They can continuously update their web presence in response to constantly changing patterns of behaviour. They can forge ahead with an initiative knowing that if it exceeds beyond their expectations, the platform can grow to accommodate it, and then shrink when the job is done. They can scale while maintaining a specialized relationship with each individual client. They can identify trends and make predictions based on analysing unprecedented amounts of data. Their employees can collaborate, find information and respond to events and customer demands with far greater agility than ever before. Those who truly adopt this approach fundamentally understand that the Cloud is independent of issues such as on-premise or off-premise – provisioning can include a mixture of both – even bare metal machines can be incorporated into a Cloud-oriented approach to provisioning.
CEOs need to understand that the opportunities to stand out have never been greater. They can help their businesses succeed in capitalizing by making it clear to their CIOs that it is no longer enough just to ensure that systems are operating and data is safe. Cloud computing opens up opportunities for a level playing field like never before and CEOs need to put their CIOs on notice that they need to be first to come up with the next wave of innovation or there will be more at risk than their jobs.
A Review of Bruce McCabe’s Skinjob
It’s not every day you get to read a book written by one of your professional associates; it’s rare when the book happens to be a gripping yarn that has you wanting to tell everyone about it. Skinjob by Dr. Bruce McCabe is such a book.
I first met Bruce in 2006 when he came to interview me about my journey in taking Altium to the Cloud. At the time the things we were doing at Altium with what later became known as ‘Cloud Computing’ were pretty revolutionary, Some senior people at Salesforce told me that some of my emails about how I was using their system sent shock waves through the entire organisation and were instrumental in the development of Force.com. One person has even described me, rather embarrassingly I think, as the “Father of Force.com”. Anyway I digress. Bruce was researching some of the new technologies emerging for his consulting firm and had visited San Francisco to see several companies including Salesforce. He was interested in meeting people who were pushing the envelope and he was referred to me. Here he was in America looking for innovators globally, and was referred to someone that lived almost on his doorstep.
Since those days, Bruce and I have maintained a good professional relationship, and he mentioned to me over a coffee that he was writing a work of fiction – a thriller set in the immediate future. I didn’t think much at the time – I have seen lots of people talk about writing books, but there was a glint in his eye and he seemed serious enough. Fast forward almost a year, and I have just read his first novel – Skinjob. I was seriously impressed.
Once I got past the initial thoughts of “I know this author”, and settled into the book, I was absorbed. I had intended to finish on a plane next week, but I simply couldn’t put it down.
Without giving anything away, the book grabs you on multiple levels. Firstly there is the whodunnit mystery laced with enough threat to the main characters to consider it a thriller. At this level the book is entertaining with a complex plot that is managed and presented well, each dimension to the story doled out at a steady pace that balances beautifully the need to know what’s happening, with the desire for more. The astute reader is given enough hints to solve the mystery. Few readers will see the giveaways though, and I am certain no-one will predict the complete plot line.
Delve one level deeper and we are presented with a number of different predictions about how technology will impact on our world in the next five to ten years. McCabe’s has thoughtfully woven a number of technical advances into our daily lives, focusing on how they relate to things that are very close to all of us, examining intimate entertainment, policing and religion. I like the way in which he illustrates how data will be collected and analysed, and how computing power will be made available for the public ‘good’. In particular the iterative analysis of unstructured data is well thought through. I don’t want to say too much about this because it will give part of the story away. Suffice it to say the book makes a great case study on how Big Data and Cloud Computing will impact on our lives.
At a third level we are challenged with questions of morality and ethics about our behaviour and our rights to privacy. When people see new opportunities to make money or achieve other self interested goals, they will often overlook their moral compass and push on towards what can ultimately be a very slippery slope. McCabe does a great job of raising issues about how technology will impact on the moral decisions we make at the social and individual level, without being at all judgemental.
I think the author is destined for big things. And I am not the only one Bruce McCabe appears to have piqued the interest of the literary agent who made Harry Potter’s J.K. Rowling famous.
I thoroughly enjoyed reading Skinjob. I think others will too.
Links:
- Amazon – http://www.amazon.com/Skinjob-ebook/dp/B00CLCZEOM
- Skinjob The Book – http://skinjob-the-book.com
Techs and Non-Techs: Society’s Left Brain and Right Brain
Our progress as an ever advancing civilization is being held back by the way we approach the education of information technology. We have created a false dichotomy: we have those who come out of the education system understanding technology but not the way the real world works, and we have those who learn some aspect of the business world, but have no idea how technology is applied to their domain. It seems the more powerful the software developer, the less grounded they are in the real world, and the same is probably true for those who are strong in some vertical business functional area.
Over time, this one-sidedness is mitigated by experience and exposure, but it is not the same as having the fundamental understanding of what goes on over the fence. It is like having two separate brain hemispheres – one that is focused on how stuff can be built, and another that is focused on what needs to happen. The left brain (the software developers) know how to mix ingredients and build something, but it takes the right-brain to see how things need to be used.
The trouble is, without some means of conveying their expertise, a lot is lost in translation. Nontechs are unaware of what is possible, or have no idea whether something is technically risky or feasible. Technologists know lots of cool tech stuff but have no idea how some gem can be applied to the real world.
Technology is so pervasive, so fundamental to the way we now live that we need to rethink our education strategy or miss out on generations of possibilities. If you think we are doing just fine, why has it taken us 30+ years to apply social media principles to our computing, with publisher-subscriber models only beginning to permeate into our IT systems in natural human-facing ways? These new modes of operation are natural, what we have been doing previously is not natural, hence our historic fear of IT, our frustration with information overload, our expensive overruns and ridiculously high rates of project failures.
I was talking to some software department heads at one of Australia’s leading universities recently and I asked them when they thought we should begin teaching HTML and CSS to our students. Their response: grade two – that’s seven year olds. With this kind of fundamental understanding of the building blocks in web pages, these students will be much better prepared to build an understanding of what is possible.
On the same topic, why are we not teaching secondary students the fundamentals of object-oriented programming? I was rather shocked to learn that in the State of Victoria, Australia, there are only 14 secondary teachers who are qualified computer scientists.
Society will benefit greatly when the two hemispheres are able to communicate more effectively. Current workarounds like product managers and business analysts are a necessary glue, but how much more effective will be if there is a more fundamental understanding of what is going on in the other half of the brain. Imagine constructing buildings where the builder and architect have only a vague understanding of what the building’s purpose might be, or a prospective customer has no sense of the cost of adding a room after the walls have gone up.
I believe we need to start teaching the fundamentals of IT as part of our primary and secondary education, and carry that through to the all the university vertical domains so that computer technology is an intrinsic part of the education of every discipline. Likewise, we need to be introducing Applied Computer Science courses into the CompSci and InfoSys courses on offer so that graduates learn things like the application of Big Data, Publisher Subscriber models, marketing automation, the cost of downtime, basic risk etc, and are able to apply them to real world problems.
We need to be able to cultivate a society where both sides can make meaningful contributions to the other’s discipline by seeing through the other’s perspectives. Only then will be begin to recognise our full potential.
Transitioning to the Cloud
Today I am presenting for you the same talk I gave at the CeBIT Cloud 2012 conference in Sydney. Entitled, “Transitioning to the Cloud”, the presentation covers three areas:
- If you want to transition your business to the cloud you need to Think Cloud – Cloud, as I see it, is as much a state of mind and you need to embrace this thinking to really make full use of its potential;
- Some examples from my personal experience of using some of the large Cloud providers’ offerings, and why they are more than what they superficially appear to be; and
- Some tips on adopting Cloud (previously covered in an earlier post)
Social Media Facilitates Broader Change
Thanks to Cloud concepts, we are discovering that using computers in interconnected ways is much more powerful because it is much more natural to the way people have always worked.
Social media facilitates interactions between people with common interests who would otherwise not be able to find each other, amplifying each others’ messages and providing opportunities to synergize, collaborate, share, compare etc.
When we started talking about the semantic web quite a few years ago now, I don’t think we quite envisaged that the ontology would evolve naturally as a result of the interactions between people based on them voting with their feet. Yes, the academic community continues to move inexorably towards intelligently classified and centrally categorized data, but the real momentum is coming from dynamic ontological discoveries, crowdsourced and crowdwitnessed. People are gravitating to concepts in Twitter through hash tags, lists. They are coming together in groups in LinkedIn, Facebook and Google+, and sharing video channels in YouTube and applications in all sorts of platforms.
What has been accomplished with these tools is phenomenal, yet I have a sense that we are only just beginning to get a glimpse of the possibilities inherent in this phenomenon, possibilities largely based on publish-subscribe metaphors. Read more 
My Top 7 Tips for Going to the Cloud
A lot of people ask me for my advice on what are the most important things to consider when moving the business into the Cloud. So here are some of the things that I think business people need to consider when thinking about going to the Cloud:
1. Make sure you know how to get your data out again
Often people think about how they are going to put their data into the Cloud – if they are using Software as a Service, like Salesforce or Netsuite or Intacct or Clarizen, or Google Apps for that matter, they will be thinking about how to get their data into a shape that can go into the system. The documentation for these systems make clear reference to how to prepare and then import the customer’s data, and there are usually consultants who can assist with this process. Typically this process is well planned, but often little thought is given to how exactly you go about extracting the data out again in a way that is of value to you going forward. Often, lip service is paid to the issue by asking questions like “can I get a back up of my data?”, and a reassuring yes is provided to the now comforted prospective customer. It is one thing to be told it can be done, but you need to check that the data is actually in a format that is useful to you. And if the system is mission critical, it needs to not just be useful, it needs to be readily convertible for immediate use.
Some of the things I have done to ensure that my data is safe include writing programs that automatically read the data for updates every fifteen minutes and write them into a Relational Database hosted separately, and even replicated both in house and in the Cloud. All customisations are programmatically managed so that the relational database copy always reflects the structure in the live system. For example, I did this from Salesforce, where there were more than 300 custom objects created. Another example is to write a program that knows how to extract all the data from a system, such as an accounting system, using the API provided. Not until you have actually proven tangibly that you can get your data into a format you can actually use, it is meaningless to have access to a copy of it.
Even without programming, many systems provide some access to your data in a way you can extract it. For example Salesforce provide a once-per-week csv file you can download. If you don’t have an alternative means it is worth setting up a routine with someone responsible to take this data and copy it.
On line databases such as Amazon RDS or Simple DB can be accesed easily enough through OLEDB connections or similar, or copies of the backups can be stored locally in a format that can be opened by alternative data stores.
No matter how you do it, the principle is important here: you should have a fully tested means of accessing your data off line. The more mission critical the data, the more real-time the recoverability needs to be.
2. Think Differently
Steve Jobs’ passing reminded everyone of the Apple Think Different campaign, but seriously, you need to think Differently when it comes to the Cloud in order to leverage it successfully. It truly is different to anything we have seen, and if you are only seeing it as a cost mitigator or a means of outsourcing infrastructure, you are missing a lot of (pardon the pun) blue sky behind the Cloud. Social networking, crowdsourcing, ubiquity of device and location, Metcalfe’s law in general, scalability, the ability to fail fast and loosely coupled web services are all factors of the Cloud that lend itself to being different.
One example is the way that Salesforce enables you to leverage the power of Twitter and Facebook by recording people’s Twitter and Facebook details against their record and if they tweet or post something with a given hashtag, the system is watching and can automatically create a case for them, assign it to a support officer who can find a solution, link the solution and automatically have the system tweet them with a response and a link.
Another example is the way captchas are being used to get the masses to perform optical character recognition on historical documents that are too poor for a machine to read. The system uses a known control word to determine whether you are human or not and poses a second one that is not known. The results are compared against the results entered by others who have received the same word – a high correlation between results from different users indicates what the text is likely to be.
A third example comes from my own testing of the Amazon EC2 platform to test some ideas concerning a new database design that enabled end users to change the structure of the database without programming, kind of like the way Salesforce allows end users to do custom objects. The test was in two parts – the first, which was easy to test, was could it handle more than a billion records. The second, a little more difficult, was, can it handle one thousand simultaneous users on cheap virtual hardware. For this test I needed a simulation that ran across eleven machines. Traditionally I would need to acquire these eleven machines and set them up – an expensive and time consuming exercise. Using Amazon EC2, I was able to set up the machines from scratch in thirty minutes, run my tests in three hours, and then analyse the results. Total cost? Less than five dollars.
There are plenty of ways the Cloud can transform how you do business if you allow it. Get your sales team to focus on harder sells while the Cloud is engineered around a Marketing Automation experience that drives their behaviour for all the low hanging fruit. The Cloud itself, if you configure it correctly, will tell you where the low hanging fruit are.
3. Make sure your systems interactions are atomic
One of the issues with having Cloud-based systems is that you can build compelling processes out of tools from a number of vendors’ systems working together. Linking your CRM to your financials, or your website to marketing automation and analytics for example. While these may seem obvious examples, the point being made here is that we need to ensure when multiple systems are involved that we are thinking about how to prevent a situation where only part of a system succeeds. This is a much more common problem when different types of systems are talking together. So make sure you are not telling the customer that his request for information has been placed in a queue unless you know for sure that the request has been placed in a queue.
4. Start with Upside, not Downside
When I first started looking at Cloud concepts about six years ago I was looking with the eyes of a sceptic and I was asking the question “What can’t I do if I adopt this approach?” By taking this kind of view I found there were plenty of things I didn’t think I could do, and this thinking led me to see restrictions and obstacles. Once I started to ask myself rather contrary question “What can I do if I adopt this approach?”, I started to see all sorts of opportunities emerge. I understand from Salesforce I was possibly the first person in the world to see their CRM product as a business platform rather than a CRM product. This led to building all sorts of systems within Salesforce including purchase requisitioning, customer software licensing, electronic production management systems with automated QA built in and tested on the finished manufactured products (with the results of the tests stored against each product and displayed to the end user when he or she finally purchased the product and plugged it into a computer). Other systems included Human Resources systems with annual leave management systems, individual development plans and hierarchical cost management for each line manager, who could also see things like who had the most leave accrued in the team.
Thinking of what is possible also leads to being able to try things experimentally with a “fail-fast” attitude. The example provided above about the eleven computers is an example of this. But being able to put ideas into practice quickly makes all sorts of innovative approaches viable that may be otherwise ignored or side stepped as pipe dreams.
In traditional approaches, a startup may need to think of architecting a business for the first generation of clients. As the numbers grow, a different architecture may be required, or investment may be required in infrastructure just in case growth may occur. One of the risks of any business that grows too quickly is one of running out of liquid cash. All this can be very limiting in an entrepreneurs thinking, with a real chance that the fear of succeeding too quickly may cause them to underperform. Often the Cloud allows an architecture to scale far further than using traditional approaches, with the ability to consume infrastructure and related services as required, scaling rapidly up, and then if necessary, scaling rapidly back down again. Traditional models require risky investments, Cloud models are far more flexible. And this allows for more optimistic thinking.
5. Check what API options are available
Most mainstream cloud vendors, whether they be offering Software as a Service, Infrastructure as a Service or a Platform as a Service, will have some sort of API that enables you to read and write data, change metadata, set permissions etc. This is important if you want to truly leverage the power that is available to you. For example, you can use Amazon’s Simple Notification Service and Simple Queueing Service to provide asynchronous connections between systems and plan to notify managers when a VIP customer representative has mentioned your company in a tweet. Having a rich API in your bag of tricks enables you to innovate with freedom, seeing the Cloud as one Cloud rather than a disparate products offered by a host of different people.
6. Seek to understand the inner workings of the vendors various risk mitigation strategies
This is something I was guilty of in the early days. I used to say “these guys know better so you can trust them to make sure your data is safe”. Recent events have made me a little more open eyed about the inner workings. If you are not sure how your data is being backed up, ask. Imagine you are having to satisfy your auditor about the safety of your data. Imagine you are having to satisfy your customer that their data is safe, secure and reliably stored. If you don’t know yourself what steps are being taken to guarantee the preservation of the data, you won’t be able to tell them, and you will come across poorly.
I have written an earlier post about an Australian ISP that collapsed after an attack that took out the server with all of their clients’ websites. They had no offsite backup. Recently, Salesforce, one of the most respected companies had two outages on Sandboxes that caused the loss of the customer data on those sandboxes and the data was down for several days. Amazon had a well publicised outage earlier in the year that brought into question the way their system handled mass failure. Separate zones, designed to remain up when others failed, went down simply due to the overload caused by the failure of one. These failures, or at least the Salesforce and Amazon ones cited, have resulted in those companies making some changes, but an astute customer robustly challenging the methods may well have picked them up before a major problem occurred.
7. Remember, it’s your data, and the buck still stops with you
I wrote a post at the time of the major Amazon outage that was picked up by the CIO Magazine. Several companies hosting their data on Amazon Web Services were posting during the outage as if they were innocent bystanders observing the fallout. The reality is that if your services are down it is your responsibility no matter how you host them. Imagine an airline losing an aircraft saying “oops, luckily we outsourced the maintenance on that plane or else it would have looked really bad for us LOL!”. I don’t think so.
Remember, it is your data and you are entitled to it, and your are responsible for its availability and its security.
Cloud81 