Architect Openstack

Architecting OpenStack for enterprise reality By Paul Miller April 7, 2014

This report was underwritten by Canonical.

TABLE OF CONTENTS Executive summary ................................................................................................................................... 3 Adding cloud to the enterprise IT mix ........................................................................................................ 4 From virtualization to the cloud .............................................................................................................. 5 Public, Private, Hybrid ........................................................................................................................... 5 The role of VMware ............................................................................................................................... 6 The road to the cloud ............................................................................................................................ 6 OpenStack ................................................................................................................................................ 7 Key components ................................................................................................................................... 8 Adoption to date .................................................................................................................................. 10 Building a bridge ..................................................................................................................................... 11 SDN – hype or value? ............................................................................................................................. 13 Managing change.................................................................................................................................... 14 Key takeaways ........................................................................................................................................ 16 About Paul Miller ..................................................................................................................................... 17 About Gigaom Research ......................................................................................................................... 17

Architecting OpenStack for Enterprise Reality

2

Executive summary Enterprise IT managers are watching the open-source cloud infrastructure project OpenStack with interest, hoping it might offer an easy way to begin exploiting the cloud alongside their existing IT estates. In this report, we briefly introduce each of OpenStack’s core components before exploring the ways OpenStack might realistically add value alongside existing investment in widely deployed on-premise solutions such as those dependent on VMware’s product family. Today’s enterprise data center is typically already heavily virtualized. Pools of servers are available for use across the organization, in a manner that appears increasingly cloud-like. With VMware still dominating this market for on-premise virtualization, we could argue that customers who have embraced VMware’s model of virtualization have no real need to take the additional steps required to deploy either public or private cloud solutions. In this report, we explore some of the ways in which VMware virtualization and OpenStack-powered clouds complement each other, and we discuss the efforts of OpenStack Foundation member VMware and other project participants to simplify the process by which existing enterprise IT investments might be enriched with the addition of OpenStack.

Architecting OpenStack for Enterprise Reality

3

Adding cloud to the enterprise IT mix The enterprise IT landscape has changed dramatically in recent years, and many of those changes continue to affect IT planning and procurement decisions today. The full implications of cloud computing’s rise are not yet fully understood, but an earlier wave of transformation — the widespread adoption of virtualization — is increasingly seen as a logical step on the way toward enterprise cloud adoption. With well over 50 percent of the world’s x86-based servers likely now devoted to hosting virtualized workloads and virtualization often exceeding 75 percent of the server estate in larger enterprises, virtualization is clearly an established technique in the IT toolkit. These virtualized pools of computing capacity change the way in which IT is provisioned and managed, and they set adopters on a path that typically leads them toward the even greater flexibility offered by a cloud solution. Virtualization offers a number of benefits over hardware-based provisioning of IT, including: 

Centralized management of IT capacity, offering economies of scale in purchasing, more-efficient resource utilization, and so on



Cost, power, cooling, and space savings, as a smaller number of servers can be operated at higher levels of utilization (virtualized servers typically operate at 80 percent to 90 percent of capacity, compared with 50 percent to 60 percent or less for non-virtualized servers)



Reduction of vendor lock-in, as the virtualization process creates a layer of abstraction between the applications and the physical hardware on which they happen to be running today



Faster provisioning, as new virtual machines can be created from a pool of available capacity far faster (minutes) than a new physical server can be specified, approved, procured, delivered, installed, and made available (weeks or even months)



Improved reliability, as virtual machines and their applications can often be moved from one physical server to another without significant impact on users

Architecting OpenStack for Enterprise Reality

4

Virtualization is not the answer to every IT challenge: 

The hypervisor that controls the virtualization process introduces a slight performance overhead, perhaps making it more efficient to leave servers devoted to a single application un-virtualized.



Some applications require dedicated access to specific hardware (such as a GPU for intensive processing), and these will usually perform better without virtualization.



In certain circumstances, ensuring that a mission-critical application is able to draw on all of a server’s resources may be more cost-effective, even if those resources may be underutilized much of the time when that application is idle.



A number of applications still ship with licenses that do not permit virtualization.



Some older applications may not perform reliably in a virtualized environment.

From virtualization to the cloud Once an organization recognizes and embraces virtualization’s core proposition of a device-independent pool of computing capacity, seeing the additional value offered by cloud computing is relatively simple. Self-service provisioning of virtual machines, elastic scaling up and down of compute capacity, the ability to access additional computing power outside the data center when required, fine-grained metering, and billing on the basis of consumption all offer clear and achievable benefits. For an organization that is already virtualizing a lot of its workloads, the additional step to a cloud solution is often — in theory, at least — not a large one.

Public, private, hybrid Early cloud solutions, such as those offered by Amazon Web Services (AWS), tended to be in the public cloud. They were compelling to startups without existing IT infrastructure, and even for larger organizations they made a lot of sense for short bursts of activity such as the New York Times’ batch conversion of unwieldy image formats for use online back in 2008. But for organizations with existing IT infrastructure, established workflows, and compliance frameworks, moving mainstream workloads to the public cloud was more complicated. Whether justified or not, the perception that the cloud might be less secure, less reliable, or simply too different to existing systems created hurdles that slowed adoption. Typically, those hurdles were only tackled when a pressing business requirement made change less painful than preserving the status quo.

Architecting OpenStack for Enterprise Reality

5

Private cloud solutions and, more recently, hybrid cloud solutions have emerged to tackle these perceived shortcomings in the public cloud, lowering the barriers to adoption and simplifying the process of realizing at least some of cloud computing’s benefits. Activities such as the Eucalyptus project from the University of California, Santa Barbara (UCSB) quickly offered software that allowed customers to run Amazon-compatible private clouds in their own data centers. More recently other open-source initiatives like the CloudStack and OpenStack projects gained traction and grew to become widely supported by a significant proportion of vendors operating in the market. OpenStack, for example, powers public cloud offerings from Rackspace, Hewlett-Packard, and others, and it can be downloaded to create private clouds that run inside customer data centers. In principle, at least, public and private OpenStack clouds can be combined to create a hybrid cloud, and the OpenStack code distributions from the likes of Rackspace and Canonical are explicitly marketed on this promise.

The role of VMware VMware dominates the server-virtualization market today, with IDC cited as suggesting a market share of 50 to 55 percent. That dominance is being squeezed — mostly by competitors Microsoft and Citrix — but VMware remains a significant player in the space. The company’s vSphere hypervisor is likely to be deployed at many organizations considering a move to the cloud, and some of the company’s other products may also be used to manage some or all the existing on-premise infrastructure. VMware is increasingly pushing cloud-like solutions such as the private vCloud Suite and its more recently launched hybrid equivalent. Both of these are most likely to appeal to customers with an ongoing and near-exclusive commitment to VMware’s family of products. More-cautious customers may be wary of the growing risk of lock-in and will therefore look elsewhere.

The road to the cloud The perception that VMware’s cloud products are the only way to move from a VMware virtualized data center to a cloud-based model is not necessarily true. The perception that companies with heavily virtualized IT infrastructure (using VMware or one of its competitors) need to throw a lot of that investment away and begin again as they adopt a different cloud solution such as OpenStack’s is also not the case, as we shall see below.

Architecting OpenStack for Enterprise Reality

6

OpenStack Launched in 2010 by Rackspace and NASA and supported by a broad and growing set of technology companies, the OpenStack project today dominates the discussion of private and hybrid clouds. Significant backers such as HP and Rackspace also offer public clouds to compete with AWS, powered by OpenStack. Other open-source cloud projects such as CloudStack have loyal followings of their own, and they are frequently described as easier to deploy than OpenStack. But OpenStack’s broad industry backing, plus the speed with which projects form to tackle perceived weaknesses in the code, make it the open-source AWS alternative to beat.

Google Trends data, tracking interest in competing open-source cloud projects

Source: Google

OpenStack continues to evolve rapidly, with new versions of the code released roughly every six months. The current version, OpenStack Havana, was released in October 2013. Core capabilities around compute and storage are relatively mature, but other aspects of the project are not so complete. Across the project, more emphasis tends to be paid to core functionality than to ease of use, sometimes leading newcomers to consider OpenStack modules complex or difficult to deploy. A wide range of companies, including Canonical, Mirantis, and Rackspace, offer professional-services engagements designed to mask some of

Architecting OpenStack for Enterprise Reality

7

this complexity behind delivery of an installation tailored to meet their clients’ requirements. These companies and others also offer their own distributions of the OpenStack code, often adding richer installation tools or tighter integration with other open-source projects (such as Ubuntu, in Canonical’s case) or their own products.

Key components Core components of the OpenStack cloud

Source: OpenStack

OpenStack originally launched with a focus on two core modules, an object-storage module (Swift), contributed by founding partner Rackspace, and a compute module (Nova), contributed by founding partner NASA. Development on each of these has continued, with a growing number of contributions from others too. The OpenStack project now offers nine core modules, composed of: 1. Nova (compute). One of the original OpenStack modules and still the most widely deployed, Nova is broadly equivalent to Amazon’s Elastic Compute Cloud (EC2). Nova is central to any OpenStack deployment, providing the APIs that developers use to start, manage, and stop virtual machines within an OpenStack cloud. Nova is designed to be horizontally scalable and to operate effectively on commodity hardware. Nova does not include a hypervisor of its own, but it is

Architecting OpenStack for Enterprise Reality

8

designed to manage the deployment of most major hypervisors, including KVM, Xen, and VMware’s ESX (via an API call to vCenter). As well as the x86 architectures typically found in today’s data centers, Nova can also run on alternative infrastructures such as those using lowpower Atom chips designed by ARM. 2. Swift (object storage). The second of OpenStack’s original modules, Swift is loosely similar to Amazon’s Simple Storage Service (S3). Swift provides OpenStack users with a scalable and redundant object-storage solution, and it should not be confused with the block-storage module Cinder. Contributors such as SwiftStack have also commercialized Swift for use in OpenStack and non-OpenStack environments. 3. Cinder (block storage). Cinder is OpenStack’s block-storage module, designed to manage a wide range of commercial storage arrays in delivering persistent block-level storage to highperformance applications such as databases. A further project, Ceph, has been growing in popularity as a replacement for (or adjunct to) both Swift and Cinder. Ceph is offered as a supported option within the OpenStack distributions of companies such as Canonical. 4. Neutron (networking). Neutron (previously known as Quantum) is OpenStack’s networking module, designed to manage communication among OpenStack instances across a wide range of physical and virtual network architectures. Neutron supports OpenFlow, one of the principal specifications for the emerging area of software-defined networking (SDN). 5. Horizon (dashboard). Horizon is OpenStack’s web-based dashboard, augmenting the APIs offered by each OpenStack module with a single graphical management console. 6. Keystone (identity service). Keystone is OpenStack’s central directory service, which manages registration, authorization, and authentication of users. Keystone can integrate with existing authentication services such as LDAP to reuse user credentials created elsewhere. 7. Glance (image service). Glance is OpenStack’s repository of disk and server images, which can be used to store and quickly deploy predefined virtual machines (for example, an Ubuntu web server or database server or a CentOS development machine). Images may be stored locally within a single OpenStack cloud or shared across a number of clouds with querying via a standard REST interface.

Architecting OpenStack for Enterprise Reality

9

8. Ceilometer (telemetry). Ceilometer offers a single repository for storing usage data from across an OpenStack cloud. This usage data is intended to support billing systems and audit processes, and it also aids in the general monitoring of a cloud’s performance under load. 9. Heat (orchestration). Heat is OpenStack’s orchestration service, designed to support human and machine-driven management of a cloud, its infrastructure, and its applications. Heat’s primary focus is the management of infrastructure, but it is designed to work with widely used software-configuration tools such as Puppet (see disclosure) and Chef in order to offer an integrated view across the whole. (Disclosure: Puppet Labs is backed by True, a venture capital firm that is an investor in the parent company of Gigaom.)

Adoption to date According to October 2013 results from the OpenStack Foundation’s ongoing survey of its users, OpenStack adoption broadly mirrors trends observed in other cloud activities. The majority of reported deployments are small, with 45 percent constituting less than 100 virtual machine instances and only 6 percent with more than 10,000 instances. Similarly, 67 percent of deployments are across fewer than 50 physical servers, and only 8 percent require more than 1,000. OpenStack use is still dominated by proofs of concept, with 32 percent of survey respondents reporting running some form of production workload. Open-source technologies dominate the environments in which OpenStack was deployed at the time of the survey, with Linux distributions such as Ubuntu (55 percent overall) and CentOS (24 percent overall) clearly the default choice for host operating systems at all scales of deployment. The KVM hypervisor used by many Linux distributions is also dominant in 62 percent of responses, but Microsoft’s HyperV and VMware’s ESX also make the list of chosen hypervisors (3 percent and 8 percent, respectively). The appearance of enterprise-grade networking from Cisco (10 percent) and VMware’s Nicira (6 percent) as well as storage solutions from the likes of NetApp (8 percent) and EMC (3 percent) combine to suggest that some, at least, are trying to integrate OpenStack with solutions less frequently associated with adopters of open-source projects. Effective deployments that include these companies’ mainstream solutions will, of course, be key to more-widespread adoption of OpenStack in the future.

Architecting OpenStack for Enterprise Reality

10

Building a bridge VMware as a company is keen to remain relevant as its biggest customers move from a largely virtualized IT infrastructure (that VMware dominates) toward a model in which public and/or private clouds play an increasingly significant role. Equally, those advocating the greater adoption of cloud infrastructure benefit if prospective customers see that their new cloud projects will be able to leverage existing investment in the virtualization of their data centers. For the moment, at least, it is in the interests of both VMware and the cloud’s champions to be seen to be working together, even as each works to extend the reach and capability of its own emergent alternative solutions (VMware’s private cloud and hybrid cloud vCloud offerings, for example).

Integration between OpenStack and VMware

Source: VMware

Architecting OpenStack for Enterprise Reality

11

Officially sanctioned and supported free drivers already exist to support interoperation between OpenStack’s Nova nodes and vSphere’s compute cluster capabilities and to direct OpenStack Cinder requests to vSphere’s storage services. There are also drivers in Canonical’s OpenStack distribution to exploit the software-defined networking (SDN) capabilities of Nicira NVP (acquired by VMware and now marketed as VMware NSX) within OpenStack’s Neutron.

Architecting OpenStack for Enterprise Reality

12

SDN: hype or value? The virtualization of compute and storage is relatively well-understood and accepted within the broader IT industry. As discussed above, the majority of enterprise-compute workloads are now virtualized, and the creation of virtual pools of storage is also well-advanced. The virtualization of networking — or software-defined networking (SDN) — is at an earlier stage of adoption. Many organizations have significant investments in perfectly serviceable physical network devices from established incumbents such as OpenStack Foundation member Cisco and are at an early stage in evaluating the additional benefits of virtualizing their network. VMware’s 2012 acquisition of SDN startup Nicira was one validation of the trend, and even stalwarts of the physical-networking paradigm today offer SDN products. OpenStack’s Neutron module is designed to integrate with existing SDN projects such as OpenFlow, as well as connecting relatively easily to commercial SDN products from VMware and others. Software-defined networking is at an early point in adoption, but most indicators suggest that the SDN market is heading toward significant growth. The foundations laid in Neutron should enable those deploying OpenStack clouds to benefit from a wide range of SDN solutions as these emerge in the market.

Architecting OpenStack for Enterprise Reality

13

Managing change Organizations with an existing investment in server virtualization from VMware or one of its competitors would not likely consider throwing that investment away in order to move wholesale to a completely new cloud. However, even organizations with fully virtualized IT infrastructure will benefit from the elastic and self-service nature of a well-architected cloud solution. Adding the ability to draw on additional compute capacity from outside the data center when required simply makes the proposition more compelling. VMware’s own cloud products offer one means of achieving these ends, but it is also increasingly feasible to implement more-open cloud environments (such as OpenStack) without giving up any of the benefits seen in the already virtualized data center. Use of the same hypervisor (e.g., KVM) and operating system (e.g., Ubuntu) both on- and off-premise certainly simplifies that process of extending a cloud, but cooperation among the technology companies in this space means it is often possible to move workloads across architectures. PayPal, for example, integrates its existing VMware investment with an OpenStack cloud. That cloud combines virtual machines using both OpenStack’s dominant KVM hypervisor and VMware’s ESX under a single management layer. As OpenStack matures, the code distributions from various partners are becoming increasingly robust and more tailored to deployment in the sort of mixed environments likely to be found in many production settings. Both Canonical and Mirantis, for example, offer their own OpenStack distributions, and both have signed agreements and undertaken development work with VMware to simplify real-world deployments like PayPal’s. Production environments are rarely as neat and single-source as the clusters used for pilot deployments or devtest activities. There are no convincing indications that IT buyers are likely to restrict their options by buying more from a smaller set of vendors, which would suggest that the IT landscape will continue to be diverse and complex. Indeed, as the number of choices on the market continues to expand, the complexly diverse nature of most IT deployments will only grow. As such, efforts to improve interoperability among different pieces of the whole should be welcomed, and activity to improve interoperability among VMware solutions and open-source clouds powered by OpenStack is one recent example of this.

Architecting OpenStack for Enterprise Reality

14

We have come a long way since early hype-filled discussions in which OpenStack was often unrealistically pitched as a direct replacement for much of an enterprise's existing IT estate. There is now far less interest in simply replacing existing systems and processes and far more in discovering the most costeffective and advantageous ways to blend the best of old and new. OpenStack has clearly reached a level of maturity at which it is feasible to deploy for key workloads inside the enterprise data center. The project's rich partner ecosystem includes both the technical underpinnings to integrate established infrastructure and systems (such as VMware-based virtualization) and the consultancy and services expertise to support these deployments in production environments. For those who are ready to embrace a hybridized solution and who wish to reduce the perceived risk of becoming too dependent on a single technology partner, it's time to seriously explore the opportunity offered by the OpenStack ecosystem.

Architecting OpenStack for Enterprise Reality

15

Key takeaways 

The virtualization of servers is increasingly common, especially in larger enterprise data centers, and VMware continues to dominate this market today.



OpenStack attracts much of the attention in the open-source cloud space. Adoption still lags far behind industry leader Amazon, but a growing number of organizations publicly support OpenStack. These include public and private cloud operators such as Rackspace, Hewlett-Packard, IBM, and others, as well as smaller companies like Canonical and Mirantis, which can help with local OpenStack deployments.



Virtualization is a step on the path toward cloud deployment, and it introduces many of the concepts and procedures needed for an effective cloud.



Organizations do not need to adopt a VMware cloud solution to benefit from existing investment in VMware virtualization.



Equally, there is no need to throw away existing investment in virtualization in order to build an OpenStack cloud.



VMware is an active member of the OpenStack Foundation, and there are supported drivers that simplify the process of managing VMware virtual machines within an OpenStack cloud.



OpenStack continues to evolve, with new code released every six months. There may be value in working with a partner if you are deploying an OpenStack cloud for production workloads.

Architecting OpenStack for Enterprise Reality

16

About Paul Miller Paul Miller is an analyst and consultant, based in the East Yorkshire (U.K.) market town of Beverley and working with clients worldwide. He helps clients understand the opportunities and pitfalls around cloud computing, big data, and open data, as well as presents, podcasts, and writes for a number of industry channels. His background includes public policy and standards roles, several years in senior management at a U.K. software company, and a Ph.D. in Archaeology. Paul was the curator for GigaOM Research’s infrastructure and cloud computing channel during 2011, routinely acts as a moderator for Gigaom Research webinars, and has authored a number of underwritten research papers such as this one.

About Gigaom Research Gigaom Research gives you insider access to expert industry insights on emerging markets. Focused on delivering highly relevant and timely research to the people who need it most, our analysis, reports, and original research come from the most respected voices in the industry. Whether you’re beginning to learn about a new market or are an industry insider, Gigaom Research addresses the need for relevant, illuminating insights into the industry’s most dynamic markets.

Visit us at: research.gigaom.com.

© 2014 Giga Omni Media, Inc. All Rights Reserved. This publication may be used only as expressly permitted by license from Gigaom and may not be accessed, used, copied, distributed, published, sold, publicly displayed, or otherwise exploited without the express prior written permission of Gigaom. For licensing information, please contact us.

Architecting OpenStack for Enterprise Reality

17