OpenZiti Ingress to Azure K8s Service
Prerequisites
- Azure Subscription, Resource Group and Azure Cli
- OpenZiti Nginx Module Repo
- Terraform
- Openziti Network
- Desktop Tunneler
Architecture
Before
After
Create OpenZiti Network
A couple of ways to do that:
- Follow the guide @Host OpenZiti
- Follow the guide @Terraform LKE Setup with OpenZiti
Tested with OpenZiti Network deployed to the LKE Cluster using Terraform.
Create Identities
Follow the guide @Create Identities for the ngx-ziti-module and windows client.
client name = client-nginx with Attribute: #clients, server module name = server-nginx with Attribute: #servers
Download jwt files and enroll identities.
- Windows ZDE Identity can be enrolled by following this enrolling process
- Nginx Module Identity can be enrolled by following Enrolling an Identity Guide
Build Docker Image
Currently, configmaps have a binary file limit of 1MB and the size of the ngx-ziti-module is around 2~3MBs. Therefore, it can not be uploaded to the existing nginx image. One needs to build a custom docker image and add it during the build process.
- Follow steps to build @ngx-ziti-module
- Follow steps to create @nginx ingress controller image
One way to update the build is to add to the Dockerfile (build/Dockerfile) this snippet of code under the common section, i.e. FROM ${BUILD_OS} as common
# copy ziti module
COPY ./ngx_ziti_module.so /usr/lib/nginx/modules
Also, need to add the following package libc6 to the debian build in the same Dockerfile, i.e. FROM nginx:1.23.3 AS debian. Did not try the alpine build but the assumption is that would be the same.
&& apt-get install --no-install-recommends --no-install-suggests -y libcap2-bin libc6 \
Lastly, if you don't want the image name to have a postfix ofSNAPSHOT... , comment it out in Makefile.
VERSION = $(GIT_TAG)##-SNAPSHOT-$(GIT_COMMIT_SHORT)
Once the image is built, upload it to your container registry. You will need it during customization of the nginx ingress controller deployment to the AKS Cluster.
if you don't have time to build your own, can use our test image based on debian 11 and nginx v1.23.3
set {
name = "controller.image.repository"
value = "docker.io/elblag91/ziti-nginx-ingress"
}
Deploy AKS Cluster
Deploy the AKS Infrustructure in Azure using terraform. The terraform code is located @OpenZiti Nginx Ingress Terraform Repo. The following resources will be created, when the plan is applied.
- Virtual Network
- AKS Private Cluster with Kubenet CNI and Nginx Ingress Controller
Set a few environmental variables for Azure Authentication and Authorization.
export ARM_SUBSCRIPTION_ID = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
export ARM_CLIENT_ID = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
export ARM_CLIENT_SECRET = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
export ARM_TENANT_ID = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
Run terraform plan.
git clone https://github.com/dariuszSki/openziti-nginx-ingress-terraform.git
cd openziti-nginx-ingress-terraform/tf-provider
terraform init
terraform plan -var include_aks_nginx=true -out aks
terraform apply "aks"
Once completed, grab cluster_public_fqdn under outputs as shown in the example below.
cluster_name = "akssandeastus"
cluster_private_fqdn = ""
cluster_public_fqdn = "akssand-2ift1yqr.hcp.eastus.azmk8s.io"
Configuration Code Snippets
If you are using your own deployment method, here are some configuration details from helm chart that need to be passed to the nginx ingress controller deployment.
- Ziti Nginx Module Identity
nginx_ziti_identity = "${file("./server-nginx.json")}"
resource "kubernetes_secret" "ziti-identity" {
metadata {
name = "nginx-ziti-identity"
}
data = {
"nginx-ziti-identity" = var.nginx_ziti_identity
}
type = "Opaque"
}
- Image Reference
set {
name = "controller.image.repository"
value = "docker.io/elblag91/ziti-nginx-ingress"
}
set {
name = "controller.image.tag"
value = "3.0.2"
}
- Configuration File - Main section blockinfo
Services are commented out until they are created. Then, terraform plan can be re-run to enable them.
:::
controller:
service:
create: false
config:
entries:
main-snippets: |
load_module modules/ngx_ziti_module.so;
thread_pool ngx_ziti_tp threads=32 max_queue=65536;
#ziti identity1 {
# identity_file /var/run/secrets/openziti.io/${kubernetes_secret.ziti-identity.metadata[0].name};
# bind k8s-api {
# upstream kubernetes.default:443;
# }
#}
- Volume section and path to secrets. Added openziti.io folder.
volumes:
- name: "ziti-nginx-files"
projected:
defaultMode: 420
sources:
- secret:
name: ${kubernetes_secret.ziti-identity.metadata[0].name}
items:
- key: ${kubernetes_secret.ziti-identity.metadata[0].name}
path: ${kubernetes_secret.ziti-identity.metadata[0].name}
volumeMounts:
- mountPath: /var/run/secrets/openziti.io
name: ziti-nginx-files
readOnly: true
OpenZiti Service Configurations
- Create configs
ziti edge create config k8s-api-intercept.v1 intercept.v1 "{\"protocols\": [\"tcp\"], \"addresses\": [\"akssand-2ift1yqr.hcp.eastus.azmk8s.io\"],\"portRanges\": [{\"low\": 443,\"high\": 443}]}"
- Create Service
ziti edge create service k8s-api --configs "k8s-api-intercept.v1" --role-attributes "service-nginx"
- Create Service Bind Policy
ziti edge create service-policy k8s-api-bind Bind --semantic "AnyOf" --identity-roles "#servers" --service-roles "#service-nginx"
- Create Service Dial Policy
ziti edge create service-policy k8s-api-dial Dial --semantic "AnyOf" --identity-roles "#clients" --service-roles "#service-nginx"
Enable Ziti Service in Ingress Controller
Uncomment ziti identity1 block in resource.helm_release.nginx-ingress
config:
entries:
main-snippets: |
load_module modules/ngx_ziti_module.so;
thread_pool ngx_ziti_tp threads=32 max_queue=65536;
ziti identity1 {
identity_file /var/run/secrets/openziti.io/${kubernetes_secret.ziti-identity.metadata[0].name};
bind k8s-api {
upstream kubernetes.default:443;
}
}
Need to disable ZDE for this network before the next step, so the nginx updates will not get intercepted while the service is not ready yet.
- Re-run terraform
terraform plan -var include_aks_nginx=true -out aks
terraform apply "aks"
Let's test our service
If the terraform was run, the kube-config file was created in the tf root directory. One can also use az cli to get the kube config downloaded.
# Configure your local kube configuration file using azure cli
az login # if not already logged in
# Windows WSL
export RG_NAME = 'resource group name'
export ARM_SUBSCRIPTION_ID = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
az aks get-credentials --resource-group $RG_NAME --name {cluster_name} --subscription $ARM_SUBSCRIPTION_ID
- Check context in the kubectl config file
kubectl config get-contexts
Expected Output
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* akssandeastus akssandeastus clusterUser_nginx_module_rg_eastus_akssandeastus
- Let's check the status of nodes in the cluster.
kubectl get nodes
Expected Output
NAME STATUS ROLES AGE VERSION
aks-agentpool-20887740-vmss000000 Ready agent 151m v1.24.9
aks-agentpool-20887740-vmss000001 Ready agent 151m v1.24.9
- List cluster info
kubectl cluster-info
Expected Output
Kubernetes control plane is running at https://akssand-2ift1yqr.hcp.eastus.azmk8s.io:443
CoreDNS is running at https://akssand-2ift1yqr.hcp.eastus.azmk8s.io:443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://akssand-2ift1yqr.hcp.eastus.azmk8s.io:443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
- List pods
kubectl get pods --all-namespaces
Expected Output
NAMESPACE NAME READY STATUS RESTARTS AGE
default nginx-ingress-nginx-ingress-7ffd564557-ngt69 1/1 Running 0 134m
kube-system azure-ip-masq-agent-9scx6 1/1 Running 0 166m
kube-system azure-ip-masq-agent-xps5g 1/1 Running 0 166m
kube-system cloud-node-manager-8q7fk 1/1 Running 0 166m
kube-system cloud-node-manager-czg95 1/1 Running 0 166m
kube-system coredns-59b6bf8b4f-8t5kj 1/1 Running 0 165m
kube-system coredns-59b6bf8b4f-rc6b8 1/1 Running 0 167m
kube-system coredns-autoscaler-5655d66f64-pbjhr 1/1 Running 0 167m
kube-system csi-azuredisk-node-2mp98 3/3 Running 0 166m
kube-system csi-azuredisk-node-tq6x6 3/3 Running 0 166m
kube-system csi-azurefile-node-9q4f5 3/3 Running 0 166m
kube-system csi-azurefile-node-bzr4z 3/3 Running 0 166m
kube-system konnectivity-agent-86cdc66d6b-tlhkw 1/1 Running 0 128m
kube-system konnectivity-agent-86cdc66d6b-ttmq6 1/1 Running 0 128m
kube-system kube-proxy-q48qv 1/1 Running 0 166m
kube-system kube-proxy-t2dhr 1/1 Running 0 166m
kube-system metrics-server-8655f897d8-mr2bq 2/2 Running 0 165m
kube-system metrics-server-8655f897d8-tnpzt 2/2 Running 0 165m
- List services
kubectl get services --all-namespaces
Expected Output
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 169m
kube-system kube-dns ClusterIP 10.0.0.10 <none> 53/UDP,53/TCP 169m
kube-system metrics-server ClusterIP 10.0.218.224 <none> 443/TCP 169m
At this point the public access is still available even though the API Kubectl queries are routed through Ziti Network. You can disable ZDE client and test that.
Block IPs to API Server
Pass the following variable to only allow 192.168.1.1/32 source IP to essentially disable Public Access.
terraform plan -var include_aks_nginx=true -var authorized_source_ip_list=[\"192.168.1.1/32\"] -out aks
terraform apply "aks"
Retest with ZDE enabled and disabled for this network.
Clean up
Run terraform to open up the AKS API to public before deleting AKS resources, so you dont get locked out.
terraform plan -var include_aks_nginx=true -var authorized_source_ip_list=[\"0.0.0.0/0\"] -out aks
terraform apply "aks"
Delete all resources
terraform plan --destroy -var include_aks_nginx=true -out aks