Building Custom Kubernetes Operators Part 6: Building Operators using Metacontroller
Kubernetes operators were introduced as an implementation of the Infrastructure as software concept. Using them you can abstract the deployment of applications and services in a Kubernetes cluster. This is the sixth and last of a series of articles explaining how operators work and how they can be implemented in different languages.
Introduction
In the previous articles in this series we saw how operators can be implemented in different programming languages. We also saw that, regardless of the chosen language, all operator controllers work the same way, executing the following actions:
- Watch for events about relevant resources.
- Run the reconcile loop to transform the current state into the desired one.
2.1. Query current state and desired state information stored in objects, using Kubernetes API.
2.2. Calculate what the state should be.
2.3. Execute actions, using Kubernetes API, to correct the state.
Metacontroller is an add-on for Kubernetes that handles the tasks common to every operator controller, (events watching, objects’ states querying, and actions execution). The decoupling of these tasks from the functionality specific to each controller allows the developer to focus on the latter: calculation of the correct state.
In the next sections of this article we are going to see how Metacontroller works and how to implement a simple operator using it.
This article assumes you have Python (version at least 3.6) installed in your computer. You will also need access to a Kubernetes cluster to try the operator. You can use minikube to create a development cluster.
The complete source code for the operator described in this article can be found here
https://github.com/flugel-it/k8s-metacontroller-operator
How Metacontroller works
A controller implementation using the Metacontroller has two components: the Metacontroller and one or more lambda controllers.
The Metacontroller is a server that extends Kubernetes API and encapsulates the common parts of writing custom controllers: events watching, states querying and actions execution.
Each lambda controller contains the business logic specific to a custom controller, that is, the functionality determining what the state should be.
The Metacontroller watches multiple resources in order to detect changes in the actual or desired state. Detection of a change invokes the relevant lambda controllers. Using the information from the lambda controller about the desired state, the Metacontroller applies the necessary changes, executing actions calling the Kubernetes API.
The Metacontroller communicates with lambda controllers using webhooks. This greatly simplifies the design and implementation of the lamda controllers, eliminating the need for direct use of Kubernetes API and allowing implementation of controllers in any language that can understand HTTP and JSON.
With Metacontroller, developers have only to create and register their lamda controllers. Metacontroller provides :
- Label selectors (for defining flexible collections of objects)
- Orphan/adopt semantics (controller reference)
- Garbage collection (owner references for automatic cleanup)
- Watches (for low latency)
- Caching (shared informers/reflectors/listers)
- Work queues (deduplicated parallelism)
- Optimistic concurrency (resource version)
- Retries with exponential backoff
- Periodic relist/resync
Currently, Metacontroller supports the implementation of two kind of controllers:
- CompositeController: We use this when the controller manages objects composed of other objects. This arrangement implies a parent-child relationship between objects, commonly various objects (children) are populated from user-provided information in a parent object that belongs to a custom resource.
- DecoratorController: Deals with attaching new behavior to existing objects, such as pods, for example.
We are focusing on CompositeControllers in this article. When registering a composite controller, the developer specifies the parent and child resources and the URL of the lambda controller webhook. Metacontroller watches the specified resources and calls the webhook when a change occurs.
When invoked, the webhook receives the state of the parent and its children. It must return the desired state of the parent and a list of all the children that should exist. The Metacontroller then takes care of the actions necessary to move from the current state to the desired one.
In the next sections we are going to use Metacontroller to implement the example operator, immortal containers.
The immortal containers operator
As we said in previous articles, the purpose of the immortal containers operator is to enable users to define containers that should run forever — that is, whenever such containers terminate for any reason, they will be restarted.
Keep in mind that the operator demonstrated in this article is just a toy example which serves only to illustrate the steps involved in the implementation of an operator. The functionality it provides can be achieved with already existing Kubernetes features, such as deployments.
This operator defines a new object kind named ImmortalContainer. Users create objects of this kind to specify containers that must run forever. In each object the user specifies the image he wants to run.
For each ImmortalContainer object the operator’s controller creates a pod to run the container and then recreates the pod whenever it terminates or is deleted. In the same object the operator exposes the name of the created pod. — In previous implementations we added a field to count the number of times the pod has been created; we will omit that step in the current implementation to make it simpler.
Each ImmortalContainer object has the following structure:
Let’s say the operator has been installed and the user wants to create an immortal container to run the image nginx:latest. To do so, he can use kubectl to create an ImmortalContainer object.
The controller will detect the new immortal container and respond by creating a pod to run the image nginx:latest. The user can then view the running pod using the following command:
If someone deletes the pod, it will be recreated.
Finally, the user can edit the ImmortalContainer object he has created to see the CurrentPod field.
$ kubectl edit immortalcontainer example-immortalcontainer
Implementing an operator using Metacontroller
Custom Resource
The custom resources are used to expose the desired and actual states. They define endpoints that give access to collections of objects.
The operator we implemented uses a custom resource to expose a collection of objects belonging to the ImmortalContainer object kind. Users create objects of this kind to specify containers that need to run forever.
As we said previously, each ImmortalContainer object has the following structure:
We used a Custom Resource Definition to create the operator’s custom resource. Again, as in the previous article, we had to write the CRD yaml file. This file defines the new object kind, ImmortalContainer, with its fields and validations.
Note that we’ve indicated that our API group is immortalcontainers.flugel.it, the API version is v1alpha1, and the name of the new object kindis ImmortalContainer.
Lambda controller
Our operator’s lambda controller provides a webhook. This webhook receives a JSON object containing the current state of an ImmortalContainer object and its Pod child (if any), and returns information specifying the desired state.
The following block shows the complete source code of this lambda controller:
You might have noticed that the code is much shorter and simpler than the ones from previous implementations in Go and in Python without Metacontroller. This is one of the main advantages of working with Metacontroller.
The code above uses the following rules to compute desired state for an ImmortalContainer object.
- If there is currently one child, then the CurrentPod field of the ImmortalContainer object must be equal to the name of the pod. Else, the CurrentPod field must be blank.
- There must be one and only one child and it must be a Pod with one container running the specified image.
The lambda controller must be registered with the Metacontroller. Registration involves creation of a CompositeController object specifying parent and child resources and the webhook URL. We will see how to do this later.
Operator deployment
In order to use the operator we must deploy it to the cluster using the following steps:
- Install Metacontroller in the cluster.
- Deploy the lambda controller code to the cluster.
- Create a service to publish the webhook URL.
- Register the lambda controller.
Installing Metacontroller
Metacontroller can be installed in any Kubernetes cluster following its installation guide. Here we are reproducing the instructions:
Execute these commands to install Metacontroller to the cluster.
Nota that, in order for Metacontroller to function properly, the Kubernetes DNS Service must be enabled in the cluster. This is the default on most clusters, but on some development clusters it’s disabled.
After installing Metacontroller you are ready to deploy any existing lambda controller or create your own.
Deploying the lambda controller
Since the code of our controller is just one file with no extra dependencies, we run it using the default python image instead of building a custom image. To do this we created a ConfigMap to store the code, using kubectl.
sync.py is the file containing the webhook code. Note that we created the ConfigMap inside the immortalcontainers namespace (using the -n option).
Then, we created a deployment that runs sync.py using the official Python image. It mounts the code from the ConfigMap using a volume.
Publishing the webhook using a service
Since the Metacontroller must be able to resolve and connect to the webhook URL, we created a service to make the webhook callable from outside its pod.
Lambda controller registration
The lambda controller must be registered with the Metacontroller. Registration requires creation of a CompositeController object containing the names of parent and child resources and the webhook URL.
Full reference about CompositeController objects can be found here.
this information tells the Metacontroller what resources to watch and what webhook to invoke when a change occurs.
Trying the operator
Now that the operator is installed in the cluster, we can try it. We are going to create an ImmortalContainer object to run the nginx:latest image. To do this we need to edit the file config/example-use.yaml to make it look like this:
We then use kubectl to create the ImmortalContainer object in the cluster.
The controller will detect the new immortal container and create a pod to run its image. Let’s verify this.
Next, let’s see that the pod is recreated if we delete it.
Finally, we can edit the ImmortalContainer object to see its status.
As you can see, the operator works as expected.
Clean up
Finally, if you’ve been following and reproducing these steps in your cluster, you can use these commands to remove all the changes that have been made.
In Conclusion
In this article we have seen how, by using Metacontroller, developers can avoid much of the complexity related to operators development. Metacontroller decouples and abstracts the tasks of events watching, state querying, and API calls, allowing developers to focus on the operators’ logic.
Another advantage of Metacontroller, as we have already mentioned, is that its architecture enables developers to use almost any programming language. Also, since lambda controllers don’t depend on any service, they are easy to test.
Finally, we think it’s worth noting that we have used only a subset of the Metacontroller features. You can learn more about the Metacontroller in the official documentation, which also provides examples in various programming languages.
