Is there an intermediate computing model between docker containers and xen hypervisors?

Question

Docker is amazing. You can use it to replicate production and all its quirks on your local machine. Then you can deploy that instance straight through all the staging workflows super-quick. Ie your software stack becomes an immutable repeatable widget in a CD pipeline.

The challenge is that the understanding the docker ecosystem is a mental leap for some people.

The benefit of Xen Hypervisors is that to the ordinary user, it walks and talks exactly like a physical machine. If you know how to ssh into a machine, then there is no learning curve. This is part of what makes AWS's EC2 so popular.

The disadvantage of Xen Hypervisors is that you don't get all the deployment and versioning benefits that allow great software deployment workflows as you would with Docker.

Is there some middle ground with the benefits of both? (I know that you can do terraform CD pipelines with amazon EC2, but it doesn't run as smoothly as Docker does.)

My question is: Is there an intermediate computing model between docker containers and xen hypervisors?

EDIT - Update:

I've had some helpful feedback about what specifically you would want to do in a docker container that is easier in a Xen Hypervisor. What I'm writing is based on my experience, trying to sell the docker concept to some very smart co-workers.

Things that can't be done in a docker container:

• Changing the open ports after the container has started.
• Changing the mapping between internal and external ports after the container has started.

(For the following - I acknowledge that these are not the docker pattern, but people learning docker don't care about that. They compare it to Xen hypervisors and go 'not as good'.)

Things that come for free in a Xen Hypervisor that take more work in a docker container

• Yes it is possible to set up ssh in a docker container. (I acknowledge that it's not the natural docker pattern - but people learning docker don't care about that.) But anyone who has gone through this process knows how badly documented it is, how much time it takes to set this up from scratch.
• Yes it is possible to set up a service manager (systemd style) after the container has started. But you don't get systemd, you get something else(supervisord) that people have to learn. Also modifying services in the container after you have started the container is painful.
• Downloading a new application service and installing it is possible, with supervisord - but is quite a lot of work.

Answer 1

Different ways to provision your servers have different tradeoffs. We have control, performance, isolation, and a focus on complete systems on one end of the spectrum, with convenience, higher-level abstractions, and a focus on individual applications on the other. Xen and Docker are almost on the different ends of this spectrum, so it doesn't make too much sense to compare them. Importantly it's not either–or, but likely both.

Let's look at different ways to manage/install/provision your server systems. We'll reach across bare-metal, hardware-level virtualization, OS-level virtualization (containers), and application-level configuration management.

• You could install and configure the server software directly on the server hardware. This requires physical access to the servers, but may actually be preferable and more secure for normal deployments. You have full control of the server. We have one OS per hardware server.

• Most server-grade systems come with an integrated management interface in their firmware. This can be used to install a server over a network, without physical access. The server can also be configured to boot from an image stored on the network which makes it feasible to uniformly administer many servers. We still have on OS per hardware server.

• Instead of installing a normal operating system we could install a hypervisor like Xen. So when the server is booted, it will actually load the Xen kernel. Xen can run other OSes in virtual machines. This allows us to have multiple isolated OSes per hardware server. Benefits of Xen and other virtualization technologies:

  • Because the virtualized OSes are isolated from each other and from the host OS, it is possible to securely share the same hardware server between multiple organizations.
  • If each OS will only consume few resources, running multiple small systems on the same hardware server leads to better utilization of the available resources.
  • Xen also makes it possible to transfer running virtual machines between physical hosts.

• Virtualization can also be used on top of an ordinary host OS, though a full host OS will consume more resources. This may still be sensible since it may be easier to administer for single-server deployments.

• Containers do not virtualize the whole operating system, but only the user space. All containers on a system still share the same kernel. While this should still provide sufficiently secure isolation, the shared kernel means this isolation is significantly weaker than with a full virtual machine. As a notable drawback, a single container can't use a kernel that is tuned for its task. Instead, the host OS must use a kernel configuration that balances the requirements of all containers.

  So unless you're fiddling with kernel-level stuff, a container may be indistinguishable from a normal virtual machine.

  Because containers are managed by the OS they also have many advantages over hardware level virtualization:

  • Container images can often inherit from another image, which means that individual images are very small, down to a few MB. So once the base images are installed, containers can be copied around very quickly.
  • When each container is specialized for a particular application, non-essential parts of the user-space can be removed. That reduces the footprint compared to a conventional virtual machine.
  • Containers can be started and suspended quickly. It can often be avoided to do a full boot of the container's OS.
  • The host OS can implement fine-grained quotas, e.g. regarding CPU usage or I/O priorities.
  • Versioning, cloning, and copy-on-write: new containers can be created so cheaply that they can be treated as disposable.
  • Many container implementations offer convenient, high-level abstractions that are easily scriptable.

• Docker is an abstraction layer on top of containers. It makes it easier to create, manage, and run container images. Docker is geared towards packaging a single stateless application per container, in contrast to virtualizing a complete OS. The main advantage of Docker is the large ecosystem of related tools.

  At this point we also have the answer to your main question: the convenience of Docker and the restrictions of Docker are two sides of the same coin, without them we have ordinary OS-level virtualization/containers.

• When packaging applications, the main focus lies on ensuring that the application runs in an appropriate environment, with the necessary configuration, and all required dependencies. Instead of creating a container where all this is provided, we could also bundle all resources into a self-contained archive and deploy it, e.g. as used by .snap packages or Java EE .war files. Those packages are effectively mounted as a read-only file system, then executed as a normal process. As a similar example, many languages can statically link their dependencies so that no dependencies are necessary at run time.

• Instead of bundling all dependencies, the dependencies can be made explicit and get resolved during installation. There's a long list of language- and OS-specific package managers, for example APT on Debian and Pip for Python. Additionally, there are configuration management tools that bring a server system into a particular state. Dependencies are now less fixed. As a notable advantage, this makes it easier to (automatically) install security updates for dependencies.

That was a pretty long list, because this is a large topic with many very good solutions. The point is, these solutions address a spectrum of problems. There is some problem-space overlap for Xen and Docker: running multiple instanced of an isolated, virtualized operating system on the same hardware.

But the proposed solutions are so different that you may reasonably combine them, i.e. run a server which runs Xen which runs an OS which runs Docker which runs your application. You might eliminate Docker from that stack if you don't want Docker-style image deployment, or might eliminate Xen if the server only runs a single OS.

Note that if the problems you are addressing are less about system configuration and more about reliability and hardware utilization, then the abstractions provided by containers are a bad fit. E.g. a production database is a kind of system that benefits from dedicated hardware, and not as much from Docker-style deployment and container isolation.

When you view Docker as a way to package applications, then the comparison to hypervisors like Xen ceases to make any sense. Instead, we now have to compare Docker against solutions like package managers where we have a similar “not either–or but probably both” relationship.

Answer 2

Other things to consider: (keeping these as notes for later)

• Kata Containers - The speed of containers, the security of VMs
• Intel Clear Containers
• Hyper RunV (Hyper Containers)