03-02-2020

Ingress on all clouds made easy with Otomi Container Platform

Making Kubernetes services externally reachable through public URLs, with certificates and hostname DNS records, all controlled by Kubernetes configuration, can be a big challenge. In this post we’ll look at some of theses challenges and explain how Otomi Container Platform solves all of them by providing a unified ingress experience on all Clouds. Now you can expose services, together with certificates, SSO and DNS with only 4 values in a yaml file.

Ingress controller limitations

There are multiple ways to give Kubernetes Services externally-reachable URLs. Usually something like a Nginx ingress controller is used that spins up an external (cloud native) load balancer. There are ingress controllers available for most Cloud (L7) load balancers. But ingress controllers do have some implications and limitations. When using the controller with a service of type NodePort, decryption will take place on the cluster worker nodes. Using the controller with a Service.Type=LoadBalancer spins up a separate external load balancer for each service. That’s gonna be costly! Other ingress controllers like the AKS Application Gateway Ingress Controller or the AWS ALB Ingress Controller allow utilizing managed Cloud services for L7 load balancing to provide high availability, path-based and host-based routing, SSL offloading and WAF integration. But these ingress controllers also have some limitations. You still need to deploy multiple ingress controllers (at least one per namespace) and configuring them together with automated certificate management and DNS integration can be challenging.

Otomi Container Platform and ingress

In Otomi Container Platform we created two flavours of routing that most companies use nowadays:

  • With Cloud LB: cloud native LB > nginx (auth only) GW > istio GW
  • Without Cloud LB: nginx (auth + termination, extras) GW > istio GW

We preconfigured the ingress controllers for Azure Application Gateways, AWS ALBs and Google Cloud LBs to terminate incoming traffic. All traffic from there is handled in the cluster, in a cloud agnostic way: Nginx ingress controller handles authentication for private SSO protected apps and passes through all traffic to Istio.

Of course everybody expects domain registration + validation to work out of the box, so we wired up External DNS and automated certificate management. We use cert-manager to create Let’s Encrypt certificates, except for when AWS is configured to use ALBs. This requires certificate ARNs, so with Otomi Container Platform it’s possible to create and register certificates in AWS certificate manager automatically.

All traffic is protected with mutual TLS and handled by Istio to provide consistent security best practices for microservices.

In summary. With Otomi Container Platform you can:

  • Take advantage of preconfigured ingress controllers (Azure, AWS and GCP)
  • Automatically create and attach certificates to external load balancers
  • Automatically create host names for services in Cloud DNS services
  • Configure SSO on services
  • Use one external Cloud L7 load balancer per Kubernetes cluster

How it works

When Otomi Cloud’s cluster.hasCloudLB flag is set, a cloud native L7 load balancer will be instantiated (AWS and GCP) or expected to exist for integration. The load balancer will then handle TLS termination and host-based routing for all cluster services (when they are configured for ingress). The Azure Application Gateway can not (yet) be automatically instantiated by Kubernetes and needs to be installed separately. The installation of the Azure Application Gateway is included in the Otomi Container Platform Kubernetes install scripts.

All ingress traffic is passed to an auto scaling Nginx ingress, which can use an oauth2-proxy to redirect unauthenticated traffic to an OIDC provider of choice when a service needs SSO authentication. Behind the nginx-ingress is an Istio IngressGateway, responsible for routing and policy management. Each team namespace is provisioned with Istio VirtualServices to connect services deployed in the namespace with the outside world.

Creating an ingress

To create an ingress for a Service in Otomi Container Platform, you only need to add a service to the services section in the team config. A service needs to have a name. This name will be used as a short name for URLs. When a team is called ‘team1’, the default URL for the service will be servicename.team-team1.yourdomain.com.

When running on Kubernetes version 1.15 or higher, you can choose to deploy using Knative. In this case you don’t add a backing svc for the service, but the specs of the image to deploy. Otomi Container Platform will then automatically deploy the image and configure ingress.

If you don’t like to configure oauth2 authentication for the ingress, add isPublic: true.

The default URL is not really user friendly. So we have added the option to use a custom URL. To configure a custom URL, add domain: app-name.yourdomain.com to the service config.

By default, Otomi Container Platform will create a certificate for the hostname and add it to the 443 listener on the external Gateway. If you would like to use a custom certificate, you can add it as a secret.

Below you’ll see a simplified example of a service configuration for a service (hello) to be deployed on an Otomi Container Platform controlled AWS EKS cluster:

As said before, only when you run Kubernetes version 1.15 or higher, you can take advantage of Knative. Unfortunately, EKS on AWS is always a little behind when it comes to supported Kubernetes versions. The latest supported version now is 1.14.9. 

But Azure AKS does support version 1.15. So let’s look at an example service configuration for an Otomi Container Platform controlled AKS cluster:

After committing the teams.yaml values file, Otomi Container Platform will automatically do the following:

  • Create a listener on the external load balancer and configure the listener with the certificate created for the service
  • Add a new record in the DNS hosted zone configured for the cluster and point it to the public IP of the external load balancer
  • Configure the Nginx ingress controller
  • Configure the oauth2 proxy (if the service is not public and oidc is configured for the team)
  • Configure the internal Istio ingress gateway

Otomi Container Platform will also add the service to the team dashboard. Now team members don’t need to remember all the externally-reachable URLs for their apps deployed in multiple stages.

A real cloud agnostic experience

Although Otomi Container Platform uses Cloud provider services for L7 Load Balancing and DNS, deploying an ingress with Otomi Stack is completely cloud agnostic. The only thing you need to do is deploy a (Knative or ClusterIP) service and add the service to the team configuration as shown before. This makes it possible to deploy your service to all supported clouds in the same fashion, without having to know on which Cloud your application is running. When using the default service configuration, you can now deploy your application to multiple Kubernetes clusters on different clouds.

Would you like to know more about Otomi Container Platform? Request a free demo.

Let's get in touch!

Contact us