CNCF — Kyverno: Sigstore Cosign Updates

5 min readAug 28, 2023

I’ve recently been presented with an amazing opportunity to be mentored in Kyverno as part of the Linux Foundation mentorship program. I’ve been involved in tasks related to Sigstore Cosign updates, Kyverno CLI enhancements, and the optimization of the verifyImage operation through the implementation of a TTL cache. The purpose of this blog is to spotlight my journey as a mentee and to showcase the exciting progress achieved within Kyverno.

About Kyverno :

Kyverno, is a Kubernetes policy engine and admission controller that can be used to describe policies and validate resource requests against those policies. It allows us to create Policies for our Kubernetes cluster on different levels. It enables us to validate, change, and create resources based on the policies we define.

Kyverno runs as a dynamic admission controller in a Kubernetes cluster. Kyverno receives validating and mutating admission webhook HTTP callbacks from the Kubernetes API server and applies matching policies to return results that enforce admission policies or reject requests.

Kyverno policies can match resources using the resource kind, name, label selectors, and much more.

Significance of my project :

Kyverno supports verifying OCI images before pulling them into a cluster. OCI images may be signed using Notary or Cosign to ensure supply chain security is maintained. Those signatures can be verified against the provided key. It also supports the keyless image verification.

During my mentorship I have worked on Sigstore cosign updates, kyverno CLI and optimization of verifyImage.

Currently, when applying a policy containing an image verification rule to a resource, the verification process takes place. However, if we deploy this same resource thousands of times, the verification would also occur thousands of times, resulting in significant costs. To address this issue, I have devised a plan to implement a “verifyImage” cache.

Considering our requirements, we are seeking an optimal cache solution. Specifically, once an image has been verified, we don’t intend for it to remain verified indefinitely. Additionally, given limitations on cache size, we need to manage entries that have expired or are less relevant, necessitating eviction. After conducting benchmarking across various caching options, we have determined that Ristretto is the most suitable choice for our needs. Ristretto is a fast, concurrent cache library built with a focus on performance and correctness. Ristretto admission policy is based on TinyLFU: A Highly Efficient Cache Admission Policy.

By default cache size is 1000 and it’s TTL value is 60 minutes. But Users can set these values according to their requirements.

Upstream PR : https://github.com/kyverno/kyverno/pull/7969

Verify Sigstore Cosign signatures and attestations using TTL cache:

Consider the following ClusterPolicy which contains a verifyImage rule.
This policy checks the images in the repoghcr.io/kyverno/test-verify-image:* and ensures that it has been signed by verifying its signature against the provided cosign.pub key :

apiVersion: v1
kind: Namespace
metadata:
  name: test-verify-images
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: secret-in-keys
spec:
  validationFailureAction: Enforce
  background: false
  webhookTimeoutSeconds: 30
  failurePolicy: Fail
  rules:
  - name: check-secret-in-keys
    match:
      any:
      - resources:
          kinds:
          - Pod
    verifyImages:
    - imageReferences:
      - "ghcr.io/kyverno/test-verify-image:*"
      attestors:
      - entries:
        - keys:
            secret:
              name: testsecret
              namespace: test-verify-images
            rekor:
              url: https://rekor.sigstore.dev
              ignoreTlog: true

We are currently utilizing the cosign.pub key sourced from the mentioned secret. However, it’s also possible to directly provide the cosign.pub key within the cluster policy :

apiVersion: v1
kind: Secret
metadata:
  name: testsecret
  namespace: test-verify-images
data:
  cosign.pub: LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tLS0KTUZrd0V3WUhLb1pJemowQ0FRWUlLb1pJemowREFRY0RRZ0FFOG5YUmg5NTBJWmJSajhSYS9OOXNicU9QWnJmTQo1L0tBUU4wL0tqSGNvcm0vSjV5Y3RWZDdpRWNuZXNzUlFqVTkxN2htS082SldWR0hwRGd1SXlha1pBPT0KLS0tLS1FTkQgUFVCTElDIEtFWS0tLS0t
type: Opaque

After creating this policy now create the following pod resource :

apiVersion: v1
kind: Pod
metadata:
  name: test-secret-pod
  namespace: test-verify-images
spec:
  containers:
  - image: ghcr.io/kyverno/test-verify-image:signed
    name: test-secret

Inside this pod, there’s a container named “test-secret” that holds the image “ghcr.io/kyverno/test-verify-image:signed”. The said image has been signed using the cosign key pair as the one employed in our policy.

After applying this pod now generate the Kyverno log by using :

kubectl logs deployment/kyverno-admission-controller -n kyverno

Now You can check in logs that image verify operation has taken around 15 seconds.

Time taken by the image verify operation : %!(EXTRA time.Duration=14.772056s)

Now create another Pod resource by using the same image.

apiVersion: v1
kind: Pod
metadata:
  name: test-secret-pod-2
  namespace: test-verify-images
spec:
  containers:
  - image: ghcr.io/kyverno/test-verify-image:signed
    name: test-secret-2

Once you have applied this new Pod, regenerate kyverno logs using the identical command. Then, examine the logs to verify that the image verification operation was completed within a mere 39 microseconds.

Time taken by the image verify operation : %!(EXTRA time.Duration=39.892µs)

Imagine making thousands of similar calls; in such a scenario, substantial time savings could be achieved.

Verifying Notary Signatures and attestations using TTL cache:

apiVersion: v1
kind: Namespace
metadata:
  name: notary-verify-images
---
apiVersion: kyverno.io/v2beta1
kind: ClusterPolicy
metadata:
  name: check-image-notary
spec:
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 30
  failurePolicy: Fail  
  rules:
    - name: verify-signature-notary
      context:
      - name: keys
        configMap:
          name: keys
          namespace: notary-verify-images
      match:
        any:
        - resources:
            kinds:
              - Pod
      verifyImages:
      - type: Notary
        imageReferences:
        - "ghcr.io/kyverno/test-verify-image*"
        attestors:
        - count: 1
          entries:
          - certificates:
              cert: "{{ keys.data.certificate }}"

While we’ve currently supplied certificates through a ConfigMap, there’s also the option to include them directly within a ClusterPolicy.

apiVersion: v1
kind: ConfigMap
metadata:
  name: keys
  namespace: notary-verify-images
data:
  certificate: |-
    -----BEGIN CERTIFICATE-----
    MIIDTTCCAjWgAwIBAgIJAPI+zAzn4s0xMA0GCSqGSIb3DQEBCwUAMEwxCzAJBgNV
    BAYTAlVTMQswCQYDVQQIDAJXQTEQMA4GA1UEBwwHU2VhdHRsZTEPMA0GA1UECgwG
    Tm90YXJ5MQ0wCwYDVQQDDAR0ZXN0MB4XDTIzMDUyMjIxMTUxOFoXDTMzMDUxOTIx
    MTUxOFowTDELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAldBMRAwDgYDVQQHDAdTZWF0
    dGxlMQ8wDQYDVQQKDAZOb3RhcnkxDTALBgNVBAMMBHRlc3QwggEiMA0GCSqGSIb3
    DQEBAQUAA4IBDwAwggEKAoIBAQDNhTwv+QMk7jEHufFfIFlBjn2NiJaYPgL4eBS+
    b+o37ve5Zn9nzRppV6kGsa161r9s2KkLXmJrojNy6vo9a6g6RtZ3F6xKiWLUmbAL
    hVTCfYw/2n7xNlVMjyyUpE+7e193PF8HfQrfDFxe2JnX5LHtGe+X9vdvo2l41R6m
    Iia04DvpMdG4+da2tKPzXIuLUz/FDb6IODO3+qsqQLwEKmmUee+KX+3yw8I6G1y0
    Vp0mnHfsfutlHeG8gazCDlzEsuD4QJ9BKeRf2Vrb0ywqNLkGCbcCWF2H5Q80Iq/f
    ETVO9z88R7WheVdEjUB8UrY7ZMLdADM14IPhY2Y+tLaSzEVZAgMBAAGjMjAwMAkG
    A1UdEwQCMAAwDgYDVR0PAQH/BAQDAgeAMBMGA1UdJQQMMAoGCCsGAQUFBwMDMA0G
    CSqGSIb3DQEBCwUAA4IBAQBX7x4Ucre8AIUmXZ5PUK/zUBVOrZZzR1YE8w86J4X9
    kYeTtlijf9i2LTZMfGuG0dEVFN4ae3CCpBst+ilhIndnoxTyzP+sNy4RCRQ2Y/k8
    Zq235KIh7uucq96PL0qsF9s2RpTKXxyOGdtp9+HO0Ty5txJE2txtLDUIVPK5WNDF
    ByCEQNhtHgN6V20b8KU2oLBZ9vyB8V010dQz0NRTDLhkcvJig00535/LUylECYAJ
    5/jn6XKt6UYCQJbVNzBg/YPGc1RF4xdsGVDBben/JXpeGEmkdmXPILTKd9tZ5TC0
    uOKpF5rWAruB5PCIrquamOejpXV9aQA/K2JQDuc0mcKz
    -----END CERTIFICATE----

This policy verifies a container image against a Notary certificate. And After creating the policy now create the following pod resource :

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: test
  name: test
  namespace: notary-verify-images
spec:
  containers:
  - image: ghcr.io/kyverno/test-verify-image:signed
    name: test
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

Inside this pod, there’s a container named “test” that holds the image “ghcr.io/kyverno/test-verify-image:signed”. The said image has been signed using the notary.

After applying this pod now generate the Kyverno log by using :

kubectl logs deployment/kyverno-admission-controller -n kyverno

Now You can check in logs that image verify operation has taken around 5 seconds.

imageverifier.go:275: Time taken by the image verify operation : %!(EXTRA time.Duration=4.875285835s)

Now create another Pod resource by using the same image.

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: test
  name: test-2
  namespace: notary-verify-images
spec:
  containers:
  - image: ghcr.io/kyverno/test-verify-image:signed
    name: test-2
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

 Time taken by the image verify operation : %!(EXTRA time.Duration=20.183µs)

Imagine making thousands of similar calls; in such a scenario, substantial time savings could be achieved.

Conclusion :

I would Like to thank my Mentors Shuting Zhao, Jim Bugwadia, Vishal Choudhary, Chip Zoller and Charles-Edouard Brétéché for helping me out throught this mentorship. Without their constant support it’s not going to happen.

LFX Mentorship is a great place for a student and a working professional to work on a project that motivates and exponentially increases the ability of a developer to work in a large team. I would strongly recommend it to anyone who has a passion for the open-source culture to apply for LFX Mentorship.