System Tools‎ > ‎

HTTPS Client Certificate Authentication with Sidecar

posted Jul 4, 2020, 12:24 PM by Chris G   [ updated Jul 4, 2020, 12:25 PM ]

Image from

This paper is a continuous exploration of enabling HTTPS for the app without https implemented. (The first paper can be reached here.) Here we will enable client certificate authentication for a non-https app using the sidecar pattern.

When client certificate authentication is turned on, the client HTTPS connection must submit with a valid cert that signed by the CA. Otherwise, the connection will be rejected.

In the last part of the paper, we examine the Prometheus in IBM Cloud Private which is using the same https sidecar pattern.

Steps to setup client certificate authentication

First, we enable the client certificate authentication by adding the following lines in the nginx.conf file as what we have in the first paper.

server {
listen 443 ssl;
server_name localhost;
ssl_certificate /app/cert/hello-server.pem;
ssl_certificate_key /app/cert/hello-server-key.pem;

ssl_client_certificate /app/cert/hello-server-ca.pem;
ssl_verify_client on;
ssl_protocols TLSv1.2;

When ssl_verify_client is set on, the ssl_client_certificate need to be set as the CA cert that is used to sign the server and client cert. If a client doesn’t use a cert signed by this CA, the https connection will be rejected.

Secondly, we create the K8s secret with all the certs required,

kubectl create secret generic hello-sidecar-nginx-certs --from-file=hello-server-cert=./hello-server.pem --from-file=hello-server-key=./hello-server-key.pem --from-file=hello-server-ca-cert=./myca.pem

Then update the K8s deployment file, to mount the CA to the Nginx container.

- name: hello
image: zhiminwen/hello:v1
imagePullPolicy: IfNotPresent
value: "8080"
- name: tls-sidecar
image: nginx
imagePullPolicy: IfNotPresent
- name: secret-volume
mountPath: /app/cert
- name: config-volume
mountPath: /etc/nginx/nginx.conf
subPath: nginx.conf
- name: secret-volume
secretName: hello-sidecar-nginx-certs
- key: hello-server-cert
path: hello-server.pem
- key: hello-server-key
path: hello-server-key.pem
- key: hello-server-ca-cert
path: hello-server-ca.pem
- name: config-volume
name: hello-sidecar-nginx-conf

Apply the updated yaml file. Now the application is HTTPS client certificate authentication enabled.

Positive and Negative Test

Test without the valid cert.

curl -k
<head><title>400 No required SSL certificate was sent</title></head>
<center><h1>400 Bad Request</h1></center>
<center>No required SSL certificate was sent</center>

To test with the valid cert, let’s generate the cert signed by the CA first.

Create a json file as below, save it as “clientRequest.json”

"CN": "client-for-hello-server",
"hosts": [
"key": {
"algo": "rsa",
"size": 2048

Generate the client cert with the client profile (as defined in the first paper).

cd certscfssl gencert -ca=myca.pem -ca-key=myca-key.pem -config=ca-config.json -profile=client -hostname="" clientRequest.json | cfssljson -bare hello-client

You will have

  • hello-client.pem, the public key
  • hello-client-key.pem, the private key

Test with these keys,

curl -k --cert certs\hello-client.pem --key certs\hello-client-key.pem
time now: 03:36:36

At this point, we implemented the HTTPS with client cert authentication for the non-https application using the K8s sidecar pattern.

ICP Prometheus

By default, Prometheus doesn’t provide any HTTP/TLS capability. IBM Cloud Private, ICP, uses the sidecar technique to enable the HTTPS/TLS and client certificate authentication.

Attached is part of the result of kubectl -n kube-system describe pods monitoring-prometheus-74c6d846d7-plb2n

The sidecar container, router, in the pods

Container ID: docker://04b9191d822ddb7f14e063e264afaeb2299d6d846777d230b17ec7404f92bade
Image: devcluster.icp:8500/ibmcom/icp-management-ingress:2.2.2
Image ID: docker-pullable://devcluster.icp:8500/ibmcom/icp-management-ingress@sha256:c6e8be6e465e69b0d9e045a78a42ec22ef9e79d1886dfce691b9e7f1e9738d6a
Port: 8080/TCP
Host Port: 0/TCP
State: Running
Started: Wed, 28 Nov 2018 15:16:22 +0800
Ready: True
Restart Count: 0
Environment: <none>
/opt/ibm/router/caCerts from monitoring-ca-certs (rw)
/opt/ibm/router/certs from monitoring-certs (rw)
/opt/ibm/router/conf from router-config (rw)
/opt/ibm/router/entry from router-entry (rw)
/opt/ibm/router/lua-scripts from lua-scripts-config-config (rw)
/var/run/secrets/ from default-token-tvb2h (ro)

The nginx.conf can be found by running kubectl -n kube-system describe cm monitoring-prometheus-router-nginx-config

The server block excerpt is listed as below,

server {
listen 8443 ssl default_server;
ssl_certificate server.crt;
ssl_certificate_key server.key;
ssl_client_certificate /opt/ibm/router/caCerts/tls.crt;
ssl_verify_client on;
ssl_protocols TLSv1.2;
# Modulo ChaCha20 cipher.
ssl_prefer_server_ciphers on;

Because of the ssl_verify_client setting, the client which needs to contact to Prometheus must, therefore, use the certificate signed by the CA cert.

The location block, which redirects the traffic to the Prometheus, is shown below,

location /federate {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
location /api/v1/series {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
if ($arg_match[] = "helm_release_info") {
content_by_lua 'rewrite.write_release_response()';
rewrite_by_lua 'rewrite.rewrite_query()';
}location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;rewrite_by_lua 'rewrite.rewrite_query()';

The API call to Prometheus will be redirected to the Prometheus container in the same pod.


These two papers explore the sidecar pattern for HTTPS and client certificate authentication. With the knowledge, it is useful to understand the ICP Prometheus and to further extend Prometheus functionality.