Kubernetes Pod Policies — imagePullPolicy

When a pod is launched in a Kubernetes cluster, it starts with several policies. You can list these policies using the below command.

kubectl get pod <pod-name> -o yaml | grep Policy | sort | uniq

Replace the <pod-name> with a running pod in your cluster. You should see a similar output.

imagePullPolicy: Always
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
preemptionPolicy: PreemptLowerPriority
restartPolicy: Always

In this series, we will learn and understand these policies, how they work, when to change their default values, etc.

Let’s start with understanding imagePullPolicy.

imagePullPolicy
Default image pull policy
Key takeaways

imagePullPolicy

You might be familiar with this one. This policy tells kubelet when (and when not) to pull the container images once a pod is launched.

There are three possible values for this policy. Let’s see how each work and differ from each other.

Here’s a quick summary for you.

IfNotPresent

This tells kubelet to fetch the pod’s container image(s) only when they are not present on the host where the pod is placed.

Let’s see an example of this.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.25.3
        imagePullPolicy: IfNotPresent

Save the above definition imagePullPolicy.yml and apply.

Note the last line. imagePullPolicy: IfNotPresent.

> kubectl apply -f imagePullPolicy.yml

You should see the image being pulled from the docker registry in the pod events.

> POD=$(kubectl get pod -l app=nginx -o jsonpath="{.items[0].metadata.name}")

> kubectl describe pod $POD

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  36s   default-scheduler  Successfully assigned default/nginx-deployment-ccbb8fcb5-vhr77 to kind-control-plane
  Normal  Pulling    36s   kubelet            Pulling image "nginx:1.25.3"
  Normal  Pulled     31s   kubelet            Successfully pulled image "nginx:1.25.3" in 4.481862769s (4.481866978s including waiting)
  Normal  Created    31s   kubelet            Created container nginx
  Normal  Started    31s   kubelet            Started container nginx

It took kubelet ~5 seconds to pull the image.

Now, let’s delete the deployment and re-apply.

> kubectl delete -f imagePullPolicy.yml
> kubectl apply -f imagePullPolicy.yml

Retake a look at pod events.

> POD=$(kubectl get pod -l app=nginx -o jsonpath="{.items[0].metadata.name}")
> kubectl describe pod $POD


Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  6s    default-scheduler  Successfully assigned default/nginx-deployment-ccbb8fcb5-4mgrl to kind-control-plane
  Normal  Pulling    6s    kubelet            Pulling image "nginx:1.25.3"
  Normal  Pulled     4s    kubelet            Successfully pulled image "nginx:1.25.3" in 2.145746866s (2.145753366s including waiting)
  Normal  Created    4s    kubelet            Created container nginx
  Normal  Started    4s    kubelet            Started container nginx

This time, it says it’s pulling the image; however, the image with the same tag was already on the host, so it only took ~2 seconds to start the pod.

That’s how imagePullPolicy: IfNotPresent works.

Image Digest

Before the next section, there is an essential concept of image digest. Image digest is different from the image tag and uniquely identifies a particular version of an image.

Image tag can change for an image version, but image digest will not.

You might have read this when it comes to the K8S best practices,

Pin your images with image digest.

Why? Because if the image code were to change for the same tag, you would have two different code bases for the same application.

To avoid this, container registries have a Tag immutability feature that restricts the image push if the same image tag exists. However, it’s a toggle feature; most of the time, it’s turned off by default.

Let’s see how image digest works in the context of imagePullPolicy.

First, let’s delete the deployment and remove the nginx docker image from local.

> kubectl delete -f imagePullPolicy.yml

# Since I am using Kind to run the k8s cluster locally,
# I will remove the docker image using crictl
> docker exec kind-control-plane crictl rmi nginx:1.25.3

# You can use the below command to remove the nginx docker image from local
> docker rmi nginx:1.25.3

Here’s the plan to test how image digest works.

Apply the deployment with the nginx:1.25.3 image tag.
It will pull the correct image with the 1.25.3 tag and start the pod.
Tag a different nginx image (I will use 1.24-perl here) as 1.25.3 locally while the pod runs. This simulates the registry image update for the same tag. Since I am using Kind to run the k8s cluster locally, I need to push this modified image to Kind using kind load
Scale the deployment to 2 replicas. Note we are not changing the image tag.
Check the Nginx version in both pods.

Action time.

> kubectl apply -f imagePullPolicy.yml

> POD=$(kubectl get pod -l app=nginx -o jsonpath="{.items[0].metadata.name}")

> kubectl describe pod $POD

> docker tag nginx:1.24-perl nginx:1.25.3

# check docker images
> docker images

# you should have two nginx images with same image id but different tags.
# REPOSITORY               TAG            IMAGE ID       CREATED        SIZE
# nginx                    1.24-perl      1a97e991c9a2   9 months ago   181MB
# nginx                    1.25.3         1a97e991c9a2   9 months ago   181MB

# before loading to the Kind, let's see how the correct image is in Kind
# > docker exec kind-control-plane crictl images |  grep nginx
# docker.io/library/nginx                    1.25.3               6c7be49d2a11c       196MB

# now I will load this image to Kind
# > kind load docker-image docker.io/library/nginx:1.25.3
# Image: "docker.io/library/nginx:1.25.3" with ID "sha256:1a97e991c9a269689f1f8dbfbe88281ffd5930e050527d1dac1f6028884c6bac" not yet present on node "kind-control-plane", loading...

# we can see the image digest has updated to 1a97e991c**
# > docker exec kind-control-plane crictl images |  grep nginx
# docker.io/library/nginx                    1.25.3               1a97e991c9a26       187MB

# now scale the nginx deployment to 2 pods.
> kubectl scale deployment --replicas 2 nginx-deployment

# once the second pod is up, exec in each pod and check nginx version
> kubectl exec -i -t $(kubectl get pod -l "app=nginx" -o jsonpath='{.items[0].metadata.name}') -- nginx -v

> kubectl exec -i -t $(kubectl get pod -l "app=nginx" -o jsonpath='{.items[1].metadata.name}') -- nginx -v

We got two different Nginx versions even though we haven’t changed the image tag on our deployment definition. That can’t be good for the Production environment if there is a drastic change in those two versions.

Always

This tells kubelet to query the image registry to resolve the image name and tag to the image digest. If the same image with the exact digest exists locally, kubelet starts the pod with a local image. If the image doesn’t exist locally, kubelet pulls the image with the resolved digest and creates the pod.

kubelet does this every time the pod is launched.

As before, let’s verify this practically.

Based on the last deployment, here’s where we are now.

Deployment definition with imagePullPolicy: IfNotPresent
The local image tag is nginx:1.25.3 with a different image digest that does not match the registry.
Two pods running with the same image tag but with different image digest
One digest correctly matches with the registry based on the tag
The second one does not.

So, to test how imagePullPolicy: Always works, here are two changes we need to apply.

Change the imagePullPolicy value from IfNotPresent to Always.
Update replica count to 2
Below should happen.

kubelet should pull the correct image based on the resolved image tag.
Pods should be started with a new image.
If we check the nginx version in both pods, both should match.

Let’s apply these changes and observe.

# before applying, I will list the image digest present on my Kind cluster
> docker exec kind-control-plane crictl images |  grep nginx
docker.io/library/nginx                    1.25.3               1a97e991c9a26       187MB

# now let's apply
> kubectl apply -f imagePullPolicy.yml
deployment.apps/nginx-deployment configured

> kubectl get pods
# both pods just started.

# list the image digest present on my Kind cluster. 
> docker exec kind-control-plane crictl images |  grep nginx
docker.io/library/nginx                    1.25.3               6c7be49d2a11c       196MB

# note the image digest/ ID is different and correctly matching with the tag from the registry

# last, check the nginx version in both pods.
> kubectl exec -i -t $(kubectl get pod -l "app=nginx" -o jsonpath='{.items[0].metadata.name}') -- nginx -v
nginx version: nginx/1.25.3

> kubectl exec -i -t $(kubectl get pod -l "app=nginx" -o jsonpath='{.items[1].metadata.name}') -- nginx -v
nginx version: nginx/1.25.3

That is how imagePullPolicy: Always works.

Never

This one is easy to guess. imagePullPolicy: Never tells kubelet not to pull any image from the registry. It will start the pods only if the specified image is present locally; otherwise, it will give an ErrImageNeverPull error.

In what use cases does this make sense?

If you want the pod started with minimal delay, you can preload its image before starting it.
You work with private image registries and don’t want to authenticate with registries every time the pod is started.

Here’s the plan to test this.

Delete the previous deployment.
Remove the local nginx image.
Update the imagePullPolicy to Never.
Apply the deployment.
Describe the pod.

# Delete the previous deployment.
> kubectl delete -f imagePullPolicy.yml

# i will remove nginx image from my kind cluster.
# > docker exec kind-control-plane crictl rmi nginx:1.25.3
# you can use 
> docker rmi nginx:1.25.3

# Update the imagePullPolicy to Never.

# Apply the deployment.
> kubectl apply -f imagePullPolicy.yml

# Describe the pod.
> POD=$(kubectl get pod -l app=nginx -o jsonpath="{.items[0].metadata.name}")

> kubectl describe pod $POD

Events:
  Type     Reason             Age                     From               Message
  ----     ------             ----                    ----               -------
  Normal   Scheduled          5m27s                   default-scheduler  Successfully assigned default/nginx-deployment-6dbdc4957f-dk7zc to kind-control-plane
  Warning  Failed             3m21s (x12 over 5m27s)  kubelet            Error: ErrImageNeverPull
  Warning  ErrImageNeverPull  19s (x26 over 5m27s)    kubelet            Container image "nginx:1.25.3" is not present with pull policy of Never

That was imagePullPolicy: Never in action.

Default image pull policy

You might have read/ know this: by default, imagePullPolicy is set to IfNotPresent.

Well, that’s only sometimes the case.

So far, in our deployment file, we have explicitly set the imagePullPolicy field; this time, let’s clean up the pods and images and do a quick test.

Here’s the plan.

Clean up pods.
Remove the docker image if present.
Remove/comment out the imagePullPolicy field from the deployment file.
Update container image field to just nginx, no tag.
Apply the deployment.
Check the value for imagePullPolicy by describing the pod.

# Delete the previous deployment.
> kubectl delete -f imagePullPolicy.yml

# we did remove the docker image in above section.

# comment out the imagePullPolicy field from deployment file.
        # imagePullPolicy: Never

# Update container image field to just nginx , no tag.
image: nginx

# Apply the deployment.
> kubectl apply -f imagePullPolicy.yml

# check pod imagePullPolicy
> POD=$(kubectl get pod -l app=nginx -o jsonpath="{.items[0].metadata.name}")

> kubectl get pod $POD -o yaml | grep imagePullPolicy

Well, that’s not IfNotPresent.

Why? It is because specific conditions exist for setting the imagePullPolicy if not given explicitly.

You can try out these scenarios and understand how the default values work.

Key takeaways

Pinning your images with Digest whenever possible is always better. However, there’s a catch. As we learned earlier, image digests are per image version. These versions will differ based on the OS architecture they were built for. So, if you are running arm64 CPU hosts, the image digest will differ from amd64.
Enable the Tag immutability feature on your image registry.
Explicitly mention the imagePullPolicy field in your yaml definitions.

🙏 I am grateful for your time and attention all the way through!

If this guide sparked a new idea,
a question, or desire to collaborate,
I’d love to hear from you:
🔗 E-Mail
🔗 LinkedIn
🔗 Upwork

Till we meet again, keep making waves.🌊 🚀