DC/OS Enables Data Center “App Stores”

Containers are getting widely adopted as they simplify the application development and deployment process, the transition into continuous integration with Microservices and Cloud-Native architectures will have significant impact on the businesses agility and competitiveness- They bring the notion of Smartphone Apps to the datacenter.

Unfortunately, today there are still gaps, and Cloud-Native implementations require quite a bit of integration and DevOps. A few commercial and proprietary offerings try to tackle the integration challenge, but without a real community behind them they may not gain enough momentum. This is why Mesosphere latest move to open source DC/OS can be a significant milestone in enabling the Enterprise “App Store” experience.

DC/OS Enabling Data Center “App Stores”

The Enterprise “App Store” Micro-Services Stack

When building a Micro-Services architecture, we need several key ingredients such as:

• Trusted and searchable image repository (Application Marketplace)
• A way to monitor and manage the physical or virtual cluster (The Devices)
• Scheduler & Orchestrator to automate deployment and resource management of apps
• Cloud-Native Storage to host shared data and state

For a real enterprise deployment, you would need a bunch of additional components like Service discovery, Network isolation, Identity management, and the list goes on.

Integrating the above discrete components manually is resource consuming and useless given everyone would need similar components, which is why we see the emergence of commercial “Cloud-Native Stacks” and formation of standard organizations like CNCF (Cloud-Native Foundation). Commercial or “best of bread” stacks have limited impact since they do not attract collaboration from multiple vendors, this is where DC/OS fits in, now there is a holistic Cloud-Native stack to which people can contribute and vendors can integrate with.

As a background DC/OS (Datacenter Operating System) is based on Mesos, a popular cluster resource manager and orchestrator which is used by some large cloud operators, Mesos have a unique two level approach allowing it to manage a cluster of clusters and support a wide array of workloads such as Docker containers, VMs, Hadoop, Spark, etc. over physical and virtual infrastructure spanning on-prem and cloud. Mesos can work with different scheduling paradigm for batch processing, continuous services, and Chronos (Scheduled) tasks over the same resource pool, making it very attractive for a broad set of applications.

DC/OS extends Mesos by providing a fully integrated stack with:

–        Different scheduling tools for Spark, Kafka, Cassandra, Docker etc.

–        Integrated resource monitoring across nodes, CPUs, memory

–        Application packages and configuration repository, which already pre-package some of the common development tools and apps

–        Service discovery (DNS), Authentication, proxy & load-balancing service, ..

Some integrations are in the works or can be extended by partners for example using project Calico for network virtualization and isolations, and persistent volumes for external storage, but the most important part is that as an open framework new components can be added or modified. In the long run it would be most valuable if various micro-services platforms will converge on APIs, hopefully this can be accomplished by CNCF.

Storing “Apps” Data

In mobile “Apps” we store the application image in Google Play Store or Apple App Store, but the data and its configuration are stored somewhere in the cloud, so when our device is stolen or breaks we load the App on the new phone, enter our credentials, and we are back in business.

Micro-Service “Apps” should be no different, Data should not be stored “locally” on stateless and elastic containers, rather in shared and highly available repositories exposing stateless (atomic) APIs for accessing files, objects, records, and streams/message-queues. Multiple micro-service instances may access the same data simultaneously without imposing locks or inconsistency, access should be location independent, and management of the data should be as simple as with mobile “Apps”, where users and developers work in a self-service environment.

This is the case today with cloud native Apps deployed in Amazon, they use managed data services like S3, DynamoDB, Kinesis, RedShift, etc. We still don’t have the on-prem equivalents of “Cloud-Native Storage Services”, but stay tuned, those will arrive if we want to deliver the “App Store” experience for on-prem Applications.