151 lines
6.4 KiB
Markdown
151 lines
6.4 KiB
Markdown
---
|
|
title: How do Kubernetes Deployments work?
|
|
h1: "How do Kubernetes Deployments work? An adversarial perspective"
|
|
date: "2018-10-03"
|
|
meta_desc: "In this blog, we take a closer look at what happens during a rollout, what happens if we kill a pod, and what happens if we add or remove labels from a pod."
|
|
meta_image: "deployment-rollout.png"
|
|
authors: ["alex-clemmer"]
|
|
tags: ["Kubernetes"]
|
|
---
|
|
|
|
*This post is part 3 in a series on the Kubernetes API. Earlier,
|
|
[Part 1](/blog/kubespy-and-the-lifecycle-of-a-kubernetes-pod-in-four-images/)
|
|
focused on the lifecycle of a `Pod` and
|
|
[Part 2](/blog/kubespy-trace-a-real-time-view-into-the-heart-of-a-kubernetes-service/)
|
|
focused on the lifecycle of a `Service`.*
|
|
|
|
What is happening when a `Deployment` rolls out a change to your app?
|
|
What does it actually do when a `Pod` crashes or is killed? What happens
|
|
when a `Pod` is re-labled so that it's not targeted by the
|
|
`Deployment`?
|
|
|
|
`Deployment` is probably the most complex resource type in Kubernetes
|
|
core. `Deployment` specifies how changes should be rolled out over
|
|
`ReplicaSet`s, which themselves specify how `Pod`s should be replicated
|
|
in a cluster.
|
|
|
|
In this post we continue our exploration of the Kubernetes API, cracking
|
|
`Deployment` open using `kubespy`, a small tool we developed to observe
|
|
Kubernetes resources in real-time.
|
|
<!--more-->
|
|
|
|
Using `kubespy trace`, for example, we can observe at a high level what
|
|
happens when `Deployment` rolls out a new version of an application:
|
|
|
|
|
|
|
|
But this post also comes with a twist, because in addition to being
|
|
complex, `Deployment` is also the most *dynamic* resource in Kubernetes
|
|
core. A `Pod` can crash or be killed. A node can disappear. A user can
|
|
trigger a rollout. In each of these cases, some combination of the
|
|
`Deployment` controller, the `ReplicaSet` controller, and the Kubelet
|
|
have to cope.
|
|
|
|
In other words, understanding `Deployment` necessarily involves
|
|
understanding how it *does* this coping. We therefore will also use
|
|
`kubespy trace` to observe what happens when we kill and otherwise
|
|
bother our `Pod`s.
|
|
|
|
## Following along
|
|
|
|
The `kubespy` repository comes with a simple [example Kubernetes app](https://github.com/pulumi/kubespy/tree/master/examples/trivial-service-trace-example),
|
|
which is used in each of these examples. If you want to try it out for
|
|
yourself the README contains detailed instructions.
|
|
|
|
You can use `kubectl` or `pulumi` --- `kubespy` CLI itself is powered by
|
|
the same code that underlies the core (OSS) [Pulumi engine](/kubernetes/). If you like this, and would
|
|
like to see information like it in CI/CD, we hope you'll give it a
|
|
shot! To get a flavor of what this looks like in practice, you might
|
|
also check out [my tweetstorm](https://twitter.com/hausdorff_space/status/1039940379301179392)
|
|
on the subject.
|
|
|
|
## What happens during a rollout?
|
|
|
|
The gif in the introduction shows what happens when:
|
|
|
|
- We deploy the example application.
|
|
- Then, later, change the image tag to `nginx:1.12-alpine` and deploy
|
|
again.
|
|
|
|
When `kubespy trace deploy nginx` is run, it will watch for changes to
|
|
(1) the `Deployment` called `nginx`, (1) the `ReplicaSet`s it controls,
|
|
and (3) the `Pod`s they control over time. The overall status is
|
|
aggregated and printed to the console as they change.
|
|
|
|
From this gif, we can see 3 distinct phases:
|
|
|
|
**First, the initial state: the `Deployment` is in a steady state.** It
|
|
controls a single `ReplicaSet`, which in turn controls three replicas of
|
|
the application `Pod`. Each `Pod` is marked as available. We can see
|
|
from `Deployment`'s status that the application is on **revision 2**,
|
|
which means the application has been deployed twice --- one initial
|
|
deployment, and one update.
|
|
|
|
|
|
|
|
**Second: the user submits a change to the `Deployment`, which triggers
|
|
a rollout.** The `Deployment` creates a new revision, **revision 3.** It
|
|
creates a new `ReplicaSet` to represent this revision, and begins to
|
|
start `Pod`s with the new app version on it.
|
|
|
|
This gif is slowed down to show each of these individual changes --- in
|
|
the gif above, you can see these happen in very quick succession.
|
|
|
|
|
|
|
|
**Third: the new version of the app comes online; the old `ReplicaSet`
|
|
is killed.** As the new `Pod`s in **revision 3** come online and are
|
|
marked available, the `Deployment` controller begins killing off
|
|
replicas from **revision 2.** Eventually it succeeds and the rollout is
|
|
complete.
|
|
|
|
|
|
|
|
## What happens if we kill a Pod?
|
|
|
|
Now that we understand the semantics of `Deployment`'s rollout, we can
|
|
see what happens when we use `kubectl delete pod <name>` on one of the
|
|
`Pod`s that is controlled by our `Deployment`.
|
|
|
|
|
|
|
|
As expected, we can see that the `ReplicaSet` controller notices the
|
|
`Pod` goes missing and spins up a new one. Note that it does *not*
|
|
trigger a rollout, and does *not* increment the revision.
|
|
|
|
Notice also that the `Pod` that was destroyed hangs around for a few
|
|
seconds, even though the new one has been booted up and the `ReplicaSet`
|
|
has been marked available.
|
|
|
|
## What happens if we add or remove labels from a Pod?
|
|
|
|
Try using `kubectl edit` to delete the labels on one of your `Pods`.
|
|
`kubespy trace` will show you something the following:
|
|
|
|
|
|
|
|
Unlike the "killing a `Pod`" example above, the old `Pod` seems to
|
|
*disappear*, replaced by a new `Pod` that, when booted up, causes the
|
|
`Deployment` to be marked as available again.
|
|
|
|
What's happening here? It turns out that if you remove the app labels
|
|
for a `Pod`, the `ReplicaSet` controller notices, removes itself from
|
|
`.metadata.ownerRef`, and then treats the `Pod` as if it's been
|
|
deleted, spinning up a new one immediately.
|
|
|
|
This is useful: if you notice one of your `Pod`s is behaving strangely,
|
|
you can simply take it out of rotation by removing those labels, so that
|
|
you can pull it aside and test it.
|
|
|
|
## Conclusion
|
|
|
|
The Kubernetes API is rich, packed with useful information about the
|
|
state of your cluster. It's remarkable how little it is directly used
|
|
to make tools. Combine a little knowledge of `Deployment`'s `.status`
|
|
field with the Kubernetes `Watch` API and a bit of terminal UI
|
|
programming, you're most of your way to `kubespy trace deployment`.
|
|
|
|
If you enjoyed this post, or are curious to see how this lifecycle is
|
|
baked into the Pulumi CLI, please [give us a shot](/kubernetes/)!
|
|
We'd love to hear your feedback.
|