Compendium of Kubernetes Application Deployment Tools

Deploying applications to Kubernetes can be as simple as writing a few resource definitions in yaml or json and applying them with kubectl, but it can also be a whole lot more automated (and complicated).

A popular meme in application deployment is the combination of Continuous Deployment and GitOps: the automatic deployment of resources after each change to the source code. In order to for you to use GitOps to deploy applications to Kubernetes, you need several things:

• Container Image Building to build your source code and local dependencies into container images.
• Resource Templating to customize deployment resources for your environment(s).
• Package Management to bundle multiple resources into versioned releases and manage package dependencies.
• Continuous Deployment to roll out new changes to your environment(s), often using a pipeline of steps and stages.
• Imperative Deployment to manage complex service lifecycles programmatically and reduce manual or fragile scripted steps.
• Autoscaling to manage the replication and resource allocation of your application over time, based on usage and consumption.

In this article, I have listed many tools (both popular and obscure) for each of these stages in application lifecycle management. Since it’s hard to judge popularity or success objectively, I’ve tried to annotate these tools in a way that makes it easy to see which big corporate backers have invested in these projects. Keep in mind, a large cloud backer may have multiple competing investments, so just because it has a known investor doesn’t mean it will survive and thrive in the long term.

Hopefully this list will give you a place to start when searching for solutions to your application deployment problems.

Container Image Building

• Moby / buildkit (Docker) — A toolkit for converting source code to build artifacts.
• kaniko (Google) — A tool to build container images from a Dockerfile, inside a container or Kubernetes cluster.
• img (Jess Frazelle) — A standalone, daemon-less, unprivileged Dockerfile and OCI compatible container image builder.
• buildah (IBM/Red Hat) — A tool that facilitates building Open Container Initiative (OCI) container images.
• Source-To-Image (IBM/Red Hat) — A tool for building artifacts from source and injecting into container images.
• Tanzu Build Service / kpack / pack (VMware/Pivotal) — A CLI and service for building apps using Cloud Native Buildpacks.
• Carvel / kbld (VMware/Pivotal) — A service for building and pushing images into development and deployment workflows.
• Google Cloud Buildpacks (Google) — Builders and buildpacks designed to run on Google Cloud’s container platforms.
• Makisu (Uber) — A fast and flexible Docker image building tool, that works in unprivileged containerized environments like Mesos and Kubernetes.

Resource Templating

• Helm (Microsoft, Google) — A Kubernetes Package Manager
• Kustomize (Google, Apple) — A CLI to customize raw, template-free YAML files, leaving the original YAML untouched and usable as-is.
• Carvel / ytt (VMware/Pivotal) — A YAML templating tool that works on YAML structure instead of text
• jsonnet / go-jsonnet (Google) — A JSON templating language.
• gomplate (Dave Henderson) — A CLI for golang template rendering, supporting local and remote datasources.
• Mustache (Github) — A framework-agnostic JSON templating engine.

Package Management

• Helm (Microsoft, Google) — A Kubernetes Package Manager
• Cloud Native Application Bundles (CNAB) / Porter / Duffle (Microsoft/Deis, Docker) — A package format specification, bundler, and installer for managing cloud agnostic distributed applications.

Continuous Deployment

• Spinnaker (Netflix, Google) — A multi-cloud continuous delivery platform for releasing software changes with high velocity and confidence.
• Terraform Kubernetes Provider (Hashicorp) — A Terraform plugin that enables full lifecycle management of Kubernetes resources.
• Concourse (VMware/Pivotal) — A container-based continuous thing-doer written in Go and Elm.
• JenkinsX (CloudBees) — An automated CI/CD for Kubernetes with preview environments on pull requests using Tekton, Knative, Lighthouse, Skaffold and Helm
• Argo CD (Intuit) — Adeclarative, GitOps continuous delivery tool for Kubernetes.
• Tekton / Tekton Pipelines (Google) — A Kubernetes controller providing CI/CD-style pipeline resources.
• Cloud Build (Google) — A service that executes builds on Google Cloud Platform infrastructure.
• Skaffold (Google) — A CLI that facilitates continuous development for Kubernetes applications.
• Azure DevOps / Azure Pipelines (Microsoft) — A cloud service that automatically builds and tests your project code and makes it available to other users.
• Brigade (Microsoft) — Event-based Scripting for Kubernetes.
• Habitat / habitat-operator (Chef) — A Kubernetes controller that runs and manages Habitat Services on Kubernetes.
• gitkube (Hasura) — A tool for building and deploying Docker images on Kubernetes using git push.

Imperative Deployment

• Kubebuilder (CNCF, Google, Apple, IBM/Red Hat) — An SDK for building Kubernetes APIs (and controllers and operators) using CRDs.
• Operator Framework / Operator SDK (IBM/Red Hat/CoreOS) — An SDK for building Kubernetes application operators.
• KUDO (D2IQ) — A framework for building production-grade Kubernetes Operators using a declarative approach.
• Pulumi (Pulumi) — An Infrastructure as Code SDK for creating and deploying cloud software that use containers, serverless functions, hosted services, and infrastructure, on any cloud.
• Carvel / kapp / kapp-controller (VMware/Pivotal) — A CLI and Kubernetes controller for installing configuration (helm charts, ytt templates, plain yaml) as described by App CRD.
• Isopod (Cruise) — An expressive DSL and framework for Kubernetes resource configuration without YAML.

Autoscaling

• Horizontal Pod Autoscaler (built-in) — A Kubernetes controller that automatically scales the number of pods in a replication controller, deployment, replica set or stateful set based on a configured metric.
• Vertical Pod Autoscaler (Google) — A set of Kubernetes components that automatically adjusts the amount of CPU and memory requested by pods running in the Kubernetes Cluster.
• Addon Resizer (Google) — A simplified version of vertical pod autoscaler that modifies resource requests of a deployment based on the number of nodes in the Kubernetes Cluster.
• KEDA (Microsoft) — A Kubernetes-based Event Driven Autoscaling component.
• Watermark Pod Autoscaler Controller (DataDog) — A custom controller that extends the Horizontal Pod Autoscaler (HPA).
• Pangolin (Damian Peckett) — An enhanced Horizontal Pod Autoscaler for Kubernetes that scales deployments based on their Prometheus metrics, using a variety of highly configurable control strategies.
• Predictive Horizontal Pod Autoscaler (IBM) — A custom pod autoscaler, similar to Horizontal Pod Autoscaler, however with added predictive elements.
• Horizontal Pod Autoscaler Operator (Banzai Cloud) — A Kubernetes controller that watches Deployments or StatefulSets and automatically creates HorizontalPodAutoscaler resources, based on autoscale annotations.

In The End…

As any DevOps advocate will tell you, it’s not about the tools but about the mindset. No one tool will give you an end to end application lifecycle management experience that delights you, because everyone uses their own permutation of tools, glued together with scripts and integration code.

You can look for tools that do one thing well, being easily replaceable and extensible, or tools that provide the most value, being less to manage, cheaper to integrate, and the best end to end user experience. There’s not really a wrong answer.

Because of those tradeoffs, it pays to look at who is behind each project, how many companies are investing, and how popular the tool is. Popular tools with large, diverse investors are more likely to keep growing as you use them, rather than stagnate and become abandoned, requiring you to replace the tool or replace the investment with your own time and energy.

Hopefully, this taxonomy will be useful and provide you with a starting place as you consider your options. Good luck!

Did I forget your favorite tool? Leave a comment or let me know on Twitter!