Making Kubernetes services externally reachable through public URLs, with certificates and hostname DNS records, all controlled by Kubernetes configuration, can be a big challenge. In this post we’ll look at some of theses challenges and explain how Otomi Stack solves all of them by providing a unified ingress experience on all Clouds. Now you can expose services, together with certificates, SSO and DNS with only 4 values in a yaml file.
Ingress controller limitations
There are multiple ways to give Kubernetes Services externally-reachable URLs. Usually something like a Nginx ingress controller is used that spins up an external (cloud native) load balancer. There are ingress controllers available for most Cloud (L7) load balancers. But ingress controllers do have some implications and limitations. When using the controller with a service of type NodePort, decryption will take place on the cluster worker nodes. Using the controller with a Service.Type=LoadBalancer spins up a separate external load balancer for each service. That’s gonna be costly! Other ingress controllers like the AKS Application Gateway Ingress Controller or the AWS ALB Ingress Controller allow utilizing managed Cloud services for L7 load balancing to provide high availability, path-based and host-based routing, SSL offloading and WAF integration. But these ingress controllers also have some limitations. You still need to deploy multiple ingress controllers (at least one per namespace) and configuring them together with automated certificate management and DNS integration can be challenging.
Otomi Stack and ingress
In Otomi Stack we created two flavours of routing that most companies use nowadays:
We preconfigured the ingress controllers for Azure Application Gateways, AWS ALBs and Google Cloud LBs to terminate incoming traffic. All traffic from there is handled in the cluster, in a cloud agnostic way: Nginx ingress controller handles authentication for private SSO protected apps and passes through all traffic to Istio.
Of course everybody expects domain registration + validation to work out of the box, so we wired up External DNS and automated certificate management. We use cert-manager to create Let’s Encrypt certificates, except for when AWS is configured to use ALBs. This requires certificate ARNs, so with Otomi Stack it’s possible to create and register certificates in AWS certificate manager automatically.
All traffic is protected with mutual TLS and handled by Istio to provide consistent security best practices for microservices.
In summary. With Otomi Stack you can:
How it works
When Otomi Cloud’s cluster.hasCloudLB flag is set, a cloud native L7 load balancer will be instantiated (AWS and GCP) or expected to exist for integration. The load balancer will then handle TLS termination and host-based routing for all cluster services (when they are configured for ingress). The Azure Application Gateway can not (yet) be automatically instantiated by Kubernetes and needs to be installed separately. The installation of the Azure Application Gateway is included in the Otomi Stack Kubernetes install scripts.
All ingress traffic is passed to an auto scaling Nginx ingress, which can use an oauth2-proxy to redirect unauthenticated traffic to an OIDC provider of choice when a service needs SSO authentication. Behind the nginx-ingress is an Istio IngressGateway, responsible for routing and policy management. Each team namespace is provisioned with Istio VirtualServices to connect services deployed in the namespace with the outside world.
Creating an ingress
To create an ingress for a Service in Otomi Stack, you only need to add a service to the services section in the team config. A service needs to have a name. This name will be used as a short name for URLs. When a team is called ‘team1’, the default URL for the service will be servicename.team-team1.yourdomain.com.
When running on Kubernetes version 1.15 or higher, you can choose to deploy using Knative. In this case you don’t add a backing svc for the service, but the specs of the image to deploy. Otomi Stack will then automatically deploy the image and configure ingress.
If you don’t like to configure oauth2 authentication for the ingress, add isPublic: true.
The default URL is not really user friendly. So we have added the option to use a custom URL. To configure a custom URL, add domain: app-name.yourdomain.com to the service config.
By default, Otomi Stack will create a certificate for the hostname and add it to the 443 listener on the external Gateway. If you would like to use a custom certificate, you can add it as a secret.
Below you’ll see a simplified example of a service configuration for a service (hello) to be deployed on an Otomi Stack controlled AWS EKS cluster:
As said before, only when you run Kubernetes version 1.15 or higher, you can take advantage of Knative. Unfortunately, EKS on AWS is always a little behind when it comes to supported Kubernetes versions. The latest supported version now is 1.14.9.
But Azure AKS does support version 1.15. So let’s look at an example service configuration for an Otomi Stack controlled AKS cluster:
After committing the teams.yaml values file, Otomi Stack will automatically do the following:
Otomi Stack will also add the service to the team dashboard. Now team members don’t need to remember all the externally-reachable URLs for their apps deployed in multiple stages.
A real cloud agnostic experience
Although Otomi Stack uses Cloud provider services for L7 Load Balancing and DNS, deploying an ingress with Otomi Stack is completely cloud agnostic. The only thing you need to do is deploy a (Knative or ClusterIP) service and add the service to the team configuration as shown before. This makes it possible to deploy your service to all supported clouds in the same fashion, without having to know on which Cloud your application is running. When using the default service configuration, you can now deploy your application to multiple Kubernetes clusters on different clouds.