SAN FRANCISCO (06/07/2012) - I'm standing in the lobby of the downtown San Francisco Hyatt Regency, where the 2012 OpenStack Conference has just commenced. As with most open source events, I feel like I'm surrounded by ComicCon refugees.
It's a big change of scene for me. My last full-time job was in Windows Server product marketing, which prevented me from writing for InfoWorld or anyone else except my Redmond bosses for four years. Now I'm back in the game, laptop battery fully charged, ready to chronicle the next big thing in open source.
OpenStack is an evolving mountain of Apache 2-licensed code billed as a "cloud operating system" for the data center. At the same time, OpenStack is a movement, confirmed by the high-energy jabber in the air around me. As with the early days of Linux, the buzz around OpenStack has risen to a roar, with thousands of community members flocking to conferences from Paris to Seoul. The level of interest and growth is phenomenal.
People with money are excited about OpenStack, too. Investors like True Ventures and big-name corporations like AT&T, Dell, Cisco, and HP, and IBM are jumping in the game. The conference is filled with true believers hawking OpenStack startups, developing for OpenStack startups, or just talking about OpenStack startups. It's a late-'90s gold rush in miniature.
The allure of OpenStack is clear: Like Linux, OpenStack aims to provide a kernel around which all kinds of software vendors can build businesses. But with OpenStack, we're talking multiple projects to provide agile cloud management of compute, storage, and networking resources across the data center -- plus authentication, self-service, resource monitoring, and a slew of other projects. It's hugely ambitious, perhaps the most far-reaching open source project ever, although still at a very early stage.
OpenStack is staking out a huge swath of territory in a hotly contested area. VMware is already shipping software that covers much the same ground, building on its stellar technology development in virtualization management. My alma mater, Microsoft, is moving in a similar direction with Windows Server and System Center. There are many other smaller competitors -- led by Eucalyptus, which offers private cloud software compatible with Amazon Web Services APIs.
As the crowd settles in for the keynote, I'm reminded of a big player that isn't here: Citrix, an early OpenStack supporter that exited the consortium in a flurry of destructive trash talk -- to launch its own competing cloud operating system, CloudStack. Clearly, the sky-high aspirations of OpenStack both fuel its outrageous momentum and incur the risk of overreach and collapse, as it incites all manner of competition. The promise is big, but the success of OpenStack is by no means assured.
Present at the creation
First up on stage is Chris Kemp, which is appropriate enough. Smart, self-possessed, and just 32 years old, Kemp is CEO of an OpenStack startup called Nebula, which counts as it backers Silicon Valley legends John Doerr and Andy Bechtolsheim. More to the point, Kemp is OpenStack's most compelling evangelist and a key figure in its genesis.
As the story goes, when Kemp was at NASA's Ames Research Center, he realized the agency's habit of procuring a supercomputer when it needed horsepower for a big, new project was not sustainable. Why couldn't NASA have an infrastructure more like Google's, where you could allocate compute power as needed from a massive pool of machines? Kemp and some forward-looking developers set about writing software for a private NASA cloud that supported this commodity computing goal.
Around the same time, public cloud provider Rackspace, occupying a distant second behind Amazon Web Services in the IaaS (infrastructure as a service) market, decided to cook up an open source cloud management system that would help raise the company's profile.
In early 2010, not long after Kemp had been promoted to NASA's CTO, NASA's endeavors caught the notice of Rackspace. Rackspace and NASA were headed in the same direction; they were even using the same language, Python, to accomplish their goals. As it turned out, the NASA crew was further along on the "compute" service, code-named Nova, to enable admins to provision and manage server instances on the fly. Meanwhile, Rackspace had written the code for a storage service, dubbed Swift.
Rackspace sought to establish a collaborative relationship with NASA -- and OpenStack was the result. Along with Nova and Swift, Glance (for managing VM images) was added to the mix for the first OpenStack code release in October 2010. Meanwhile, Kemp was lured into leaving his position as NASA CTO to found Nebula, whose principal product will be a private cloud appliance built in part on OpenStack code.
On stage at the OpenStack event, Kemp revs up the crowd as he introduces Essex, the fifth revision of OpenStack, loaded with 150 new features, including a cloud management dashboard (Horizon), an identity management service (Keystone), and more. He also covers projects not yet ready for release: Quantum, the cloud networking service under development by Cisco, and OpenvSwitch, a Linux-based virtual switch.
The crowd cheers, but I hear a loud snort next to me when Kemp wraps up his speech with a line about the participants in the OpenStack consortium being happy to work together. The source of the snort, a guy in an XXL black T-shirt that says "Chaotic evil means never having to say you're sorry," turns to me and gleefully confides: "Maybe they're working together now, but most of 'em hate each other."
Backstabbing in the cloud
That seems harsh; as far as I can tell, the vendor participants in OpenStack seem to be getting along pretty well, considering the vast breadth of project. There's one glaring exception: the ugly departure of Citrix, one of the first big vendors to lend legitimacy to OpenStack, just before the gala event in San Francisco.
Citrix cited two main reasons for abandoning OpenStack in favor of CloudStack: insurmountable technological incompatibilities between the two platforms and OpenStack's insufficient responsiveness to customer needs. Citrix also went out of its way to criticize the Nova compute platform as immature and implied that Rackspace, which has run the OpenStack project from the beginning, didn't know what it was doing because it had no experience managing an open source community.
According to Citrix, CloudStack, also an open source project under Apache 2 license, has a bunch of customers using it in production already. In comparison, OpenStack still has a long way to go -- and Citrix customers want a solution now.
A couple of days before the show, I asked Kemp for his reaction to Citrix's move, and he didn't pull any punches: "I think Citrix lied about aligning with OpenStack and then completely changed their position and viciously attacked OpenStack and threw them under the bus. And I think they're irrelevant."
It's the kind of public spat that seems inevitable with an initiative of such enormous scale. According to both Citrix and its critics, the company pushed to replace Nova with the compute kernel from Cloud.com, which it purchased in July 2011 for more than $200 million. But the Cloud.com software was written in Java, not Python, and OpenStack wasn't having any. So -- pick your hypothetical motive -- Citrix pulled out, burning bridges along the way.
OpenStack itself is a product of this kind of technology feuding. It was Eucalyptus that pioneered the idea of an open source cloud computing stack; the company's offering is now mature enough to have working production implementations to brag about. Before the Rackspace partnership, an attempt was made to integrate Nova with Eucalyptus, but incompatibilities in technology and culture led to failure and finger-pointing.
Now OpenStack faces another crossroads. For the past two years, Rackspace has provided most of the code and leadership. Sometime this year, an independent entity called the OpenStack Foundation will spin off from Rackspace and take the reins, presumably with a more open leadership structure -- which critics have been demanding for some time. But will that help or hinder OpenStack's quest to deliver a production-ready cloud operating system that can be widely adopted?
Momentum vs. adoption
Back on stage, Ubuntu's Mark Shuttleworth does a sweet demo during his portion of the keynote. When he starts his spiel, he kicks off an OpenStack installation on a remote server rack using Ubuntu's Juju deployment project. When his speech ends, 30 minutes later, there's a working private cloud deployed in the rack. Impressive.
Kemp's Nebula and another NASA alum-led startup, Piston Cloud Computing, have a similar mission: Make OpenStack deployments as easy as possible. They go about it in different ways; Nebula uses a dedicated hardware management appliance, and Piston opts for a USB-based CloudKey that can turn a rack into a cloud in 10 minutes or less. Both Kemp and Piston's CTO, Christopher McGowan, explained that their technologies are possible only by limiting the options in in a typical deployment.
Later, when I got McGowan on the phone, he told me that CloudKey works only on a specific kind of server sold by Piston. "There's no other way at this stage," said McGowan. "We can only manage this kind of simplification by reducing the number of variables, and part of that is limiting hardware options." He explained the company will expand those options in the future.
Real live OpenStack customers are hard to find, but I was able to interview one early evaluator who gave OpenStack a try and was left nonplussed. "We don't know what happened," Father Ballecer grumped on a phone call. He's the National Director of Vocation Promotion for the Jesuit Conference and the "Digital Jesuit" founder of TheTechStop; he's heavily involved in IT decision-making for the Catholic Church.
"We're testing these guys for things like moderate scalability, price-performance, maintenance capabilities, and long-term relevance," said Ballecer. "But when we ran workloads across our OpenStack test bed, we crashed the whole stack. That's not even supposed to be possible in a cloud model. We still don't know whether it was something in the code because it's beta or whether we set something up wrong."
Ballecer's misadventure is not terribly surprising -- and not just because OpenStack is early-stage technology. OpenStack is a basic set of services. If you want to make OpenStack work in the real world, you're best advised to go for a precooked solution like that provided by Piston Cloud or Nebula. Otherwise, be prepared to write your own code around those services.
Who exactly is making it work? Rackspace, for one. In mid-April, the company announced a limited-availability public cloud service powered by OpenStack, which will eventually replace Rackspace's current IaaS platform.
But the really big deal is the HP Cloud, OpenStack's most important implementation to date. Launched on May 1 in beta form, the HP Cloud is an IaaS offering built on OpenStack Compute, Object Storage, and Identity. Like Rackspace, HP doesn't have a choice -- it has to make OpenStack work for its customers. The company is confident enough of the service's stability to charge customers for it, even though HP's Cloud Services are in beta.
In a May 2012 interview, Michael Crandell, CEO of RightScale, told InfoWorld he believes HP's support of OpenStack is significant. Crandell has a great perspective on the cloud market since his company provides a SaaS offering that helps customers run production workloads on public and private clouds ranging from Amazon Web Services to Citrix's CloudStack. He thinks HP has "probably worked some of the kinks out of operating OpenStack at scale. They've contributed a lot to the code."
A common denominator for the cloud?
But OpenStack isn't all HP Cloud will offer. For example, a cloud version of MySQL and a content distribution network are already available as beta services. Late last year, HP showed off VMware's open source cloud development platform, Cloud Foundry, running on HP Cloud.
"You've got consider that no one is a pure OpenStack play," Kemp told me. "No one is supposed to be. We're not. OpenStack is just one of 50 technologies we use at Nebula."
At the OpenStack event, I met Boris Renski, co-founder of an OpenStack consultancy called Mirantis, which has worked on 25 OpenStack deployment projects, all but four of them for service providers. His take on OpenStack reveals where the action is now: "It's not a cloud computing stack. It's a framework of standards and APIs so that people can build their own cloud solutions."
The industry term is "API-driven infrastructure," and the OpenStackers are exploiting this new concept whole hog. Do it right, and you've removed many of the compatibility, management, and performance issues associated with today's data center. Don't think of private clouds as being administered by standard-issue IT pros running around with patch cables; think of them being managed from a command line by IT pros who are just as well versed with programming as they are with server management.
A big takeaway from my OpenStack experience is that world will no longer settle for the old, slow, labor-intensive way of maintaining infrastructure. Although the enthusiasm surrounding OpenStack right now is contagious, it's too early to tell whether it will be the lodestone around which this new hyper-automated world of private and public clouds coalesces. But that world is coming, and like it or not, many fewer admins will be required to keep it running.
Today, OpenStack is at the epicenter of this seminal shift. If it's successful, as with Linux, a vast ecosystem will grow up around it. It took more than a decade for Linux to evolve from a mere toy to solid server operating system and years longer to yield profitable businesses. Less than two years after its inception, OpenStack is way ahead of the curve.