HTTPS Client Certificate Authentication with Sidecar

posted Jul 4, 2020, 12:24 PM by Chris G [ updated Jul 4, 2020, 12:25 PM ]

From: https://medium.com/@zhimin.wen/https-client-certificate-authentication-with-sidecar-9b07d82a6389

Image from epicurious.com

This paper is a continuous exploration of enabling HTTPS for the app without https implemented. (The first paper can be reached here.) Here we will enable client certificate authentication for a non-https app using the sidecar pattern.

When client certificate authentication is turned on, the client HTTPS connection must submit with a valid cert that signed by the CA. Otherwise, the connection will be rejected.

In the last part of the paper, we examine the Prometheus in IBM Cloud Private which is using the same https sidecar pattern.

Steps to setup client certificate authentication

First, we enable the client certificate authentication by adding the following lines in the nginx.conf file as what we have in the first paper.

...
server {
      listen       443 ssl;
      server_name  localhost;      ssl_certificate      /app/cert/hello-server.pem;
      ssl_certificate_key  /app/cert/hello-server-key.pem;
      
      ssl_client_certificate /app/cert/hello-server-ca.pem;
      ssl_verify_client on;      ssl_protocols TLSv1.2;
...

When ssl_verify_client is set on, the ssl_client_certificate need to be set as the CA cert that is used to sign the server and client cert. If a client doesn’t use a cert signed by this CA, the https connection will be rejected.

Secondly, we create the K8s secret with all the certs required,

kubectl create secret generic hello-sidecar-nginx-certs --from-file=hello-server-cert=./hello-server.pem --from-file=hello-server-key=./hello-server-key.pem --from-file=hello-server-ca-cert=./myca.pem

Then update the K8s deployment file, to mount the CA to the Nginx container.

...
spec:
  containers:
  - name: hello
    image: zhiminwen/hello:v1
    imagePullPolicy: IfNotPresent
    env:
      - name: LISTENING_PORT
        value: "8080"
  - name: tls-sidecar
    image: nginx
    imagePullPolicy: IfNotPresent
    volumeMounts:
      - name: secret-volume
        mountPath: /app/cert
      - name: config-volume
        mountPath: /etc/nginx/nginx.conf
        subPath: nginx.conf
  volumes:
  - name: secret-volume
    secret:
      secretName: hello-sidecar-nginx-certs
      items:
        - key: hello-server-cert
          path: hello-server.pem
        - key: hello-server-key
          path: hello-server-key.pem
        - key: hello-server-ca-cert
          path: hello-server-ca.pem
  - name: config-volume
    configMap:
      name: hello-sidecar-nginx-conf

Apply the updated yaml file. Now the application is HTTPS client certificate authentication enabled.

Positive and Negative Test

Test without the valid cert.

curl -k https://192.168.64.244:31463/date
<html>
<head><title>400 No required SSL certificate was sent</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>No required SSL certificate was sent</center>
<hr><center>nginx/1.15.7</center>
</body>
</html>

To test with the valid cert, let’s generate the cert signed by the CA first.

Create a json file as below, save it as “clientRequest.json”

{
  "CN": "client-for-hello-server",
  "hosts": [
    ""
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  }
}

Generate the client cert with the client profile (as defined in the first paper).

cd certscfssl gencert -ca=myca.pem -ca-key=myca-key.pem -config=ca-config.json -profile=client -hostname="127.0.0.1" clientRequest.json | cfssljson -bare hello-client

You will have

hello-client.pem, the public key
hello-client-key.pem, the private key

Test with these keys,

curl -k --cert certs\hello-client.pem --key certs\hello-client-key.pem https://192.168.64.244:31463/date
time now: 03:36:36

At this point, we implemented the HTTPS with client cert authentication for the non-https application using the K8s sidecar pattern.

ICP Prometheus

By default, Prometheus doesn’t provide any HTTP/TLS capability. IBM Cloud Private, ICP, uses the sidecar technique to enable the HTTPS/TLS and client certificate authentication.

Attached is part of the result of kubectl -n kube-system describe pods monitoring-prometheus-74c6d846d7-plb2n

The sidecar container, router, in the pods

router:
    Container ID:  docker://04b9191d822ddb7f14e063e264afaeb2299d6d846777d230b17ec7404f92bade
    Image:         devcluster.icp:8500/ibmcom/icp-management-ingress:2.2.2
    Image ID:      docker-pullable://devcluster.icp:8500/ibmcom/icp-management-ingress@sha256:c6e8be6e465e69b0d9e045a78a42ec22ef9e79d1886dfce691b9e7f1e9738d6a
    Port:          8080/TCP
    Host Port:     0/TCP
    Command:
      /opt/ibm/router/entry/entrypoint.sh
    State:          Running
      Started:      Wed, 28 Nov 2018 15:16:22 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /opt/ibm/router/caCerts from monitoring-ca-certs (rw)
      /opt/ibm/router/certs from monitoring-certs (rw)
      /opt/ibm/router/conf from router-config (rw)
      /opt/ibm/router/entry from router-entry (rw)
      /opt/ibm/router/lua-scripts from lua-scripts-config-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-tvb2h (ro)

The nginx.conf can be found by running kubectl -n kube-system describe cm monitoring-prometheus-router-nginx-config

The server block excerpt is listed as below,

server {
        listen 8443 ssl default_server;
        ssl_certificate server.crt;
        ssl_certificate_key server.key;
        ssl_client_certificate /opt/ibm/router/caCerts/tls.crt;
        ssl_verify_client on;
        ssl_protocols TLSv1.2;
        # Modulo ChaCha20 cipher.
        ssl_ciphers EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:!EECDH+3DES:!RSA+3DES:!MD5;
        ssl_prefer_server_ciphers on;...

Because of the ssl_verify_client setting, the client which needs to contact to Prometheus must, therefore, use the certificate signed by the CA cert.

The location block, which redirects the traffic to the Prometheus, is shown below,

location /federate {
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header Host $http_host;  proxy_pass http://127.0.0.1:9090/prometheus/federate;
}location /api/v1/series {
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header Host $http_host;  if ($arg_match[] = "helm_release_info") {
     content_by_lua 'rewrite.write_release_response()';
  }
  rewrite_by_lua 'rewrite.rewrite_query()';
  proxy_pass http://127.0.0.1:9090/prometheus/api/v1/series;}location / {
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header Host $http_host;rewrite_by_lua 'rewrite.rewrite_query()';  proxy_pass http://127.0.0.1:9090/prometheus/;
}

The API call to Prometheus will be redirected to the Prometheus container in the same pod.

Conclusion

These two papers explore the sidecar pattern for HTTPS and client certificate authentication. With the knowledge, it is useful to understand the ICP Prometheus and to further extend Prometheus functionality.

Comments