In the fast-paced world of software development, time-sensitive projects can often lead to both breakthroughs and challenges. Recently, I embarked on a project to deploy a Helm chart to a Kubernetes cluster using Go. This was because the helm Terraform provider is currently not working as I would expect it to and does not allow you to build an Azure Kubernetes Cluster and Deploy a Helm chart in the same run.
Despite my best efforts, a critical bug in a dependency and several deployment hurdles made the journey incredibly frustrating. Here's an account of my experience and the code I used, highlighting the pain points and lessons learned.
The Goal
The objective was simple: automate the deployment of a Helm chart to a Kubernetes cluster. This included setting up the Kubernetes Cert-Manager objects and leveraging the github.com/mittwald/go-helm-client package to manage Helm operations. Theoretically, it was straightforward. In practice, it proved to be anything but.
The Code
Here is the code I used for this project. Note that sensitive information has been omitted for security reasons.
package main
import (
"bytes"
"context"
"fmt"
"log"
"os"
"path/filepath"
"time"
"github.com/Azure/azure-sdk-for-go/sdk/azidentity"
"github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v5"
certmanagerv1 "github.com/cert-manager/cert-manager"
helmclient "github.com/mittwald/go-helm-client"
"github.com/mittwald/go-helm-client/values"
"helm.sh/helm/v3/pkg/repo"
core "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
_ "k8s.io/client-go/plugin/pkg/client/auth"
clientcmd "k8s.io/client-go/tools/clientcmd"
"github.com/Azure/azure-sdk-for-go/sdk/azidentity"
"github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v5"
)
func main() {
var outputBuffer bytes.Buffer
ctx := context.Background()
mycontext, _ := context.WithTimeout(ctx, 80000*time.Second)
certmanageremail := "your-email@example.com"
subid := "your-subscription-id"
rg := "your-resource-group"
cluster := "your-cluster-name"
fmt.Println("Connecting to Cluster Via Azure Call\n")
myaksconnect := connectToAks(ctx, subid, rg, cluster)
mypublicip := "0.0.0.0"
fmt.Printf("Moved %v to KubeConfig File", &myaksconnect.Name)
fmt.Println("Connecting to Cluster using KubeConfig File\n")
kubeClient := connectToK8s()
fmt.Println("Building Helm Client\n")
opt := &helmclient.Options{
Namespace: "default",
RepositoryCache: "/tmp/.helmcache",
RepositoryConfig: "/tmp/.helmrepo",
Debug: false,
Linting: true,
DebugLog: func(format string, v ...interface{}) {},
Output: &outputBuffer,
}
myHelmClient, err := helmclient.New(opt)
if err != nil {
panic(err)
}
fmt.Println("Deploying nginx-ingress\n")
nginxchartRepo := repo.Entry{
Name: "nginx-ingress",
URL: "https://kubernetes.github.io/ingress-nginx",
}
if err := myHelmClient.AddOrUpdateChartRepo(nginxchartRepo); err != nil {
log.Fatal(err)
} else {
fmt.Printf("Added Chart Repo %s\n", nginxchartRepo.Name)
}
if err := myHelmClient.UpdateChartRepos(); err != nil {
log.Fatal(err)
} else {
fmt.Printf("Updating Chart Repo\n")
}
nginxchartSpec := helmclient.ChartSpec{
ReleaseName: "nginx-ingress/nginx-ingress",
ChartName: "nginx-ingress",
Namespace: "nginx-ingress",
CreateNamespace: true,
SkipCRDs: false,
Wait: true,
ValuesOptions: values.Options{
StringValues: []string{
"rbac.create=false",
"controller.service.externalTrafficPolicy=Local",
fmt.Sprintf("controller.service.loadBalancerIP=%v", mypublicip),
"controller.replicaCount=2",
"controller.nodeSelector.kubernetes\\.io/os=linux",
"defaultBackend.nodeSelector.kubernetes\\.io/os=linux",
"controller.admissionWebhooks.patch.nodeSelector.kubernetes\\.io/os=linux",
"controller.publishService.enabled=true",
"controller.service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path=/healthz",
},
},
}
nginxInstalledHelmChart, err := myHelmClient.InstallChart(mycontext, &nginxchartSpec, nil)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Status of Chart Install %v,\n", *nginxInstalledHelmChart.Info)
fmt.Println("Deploying cert-manager\n")
certmanagerchartRepo := repo.Entry{
Name: "cert-manager",
URL: "https://charts.jetstack.io",
}
if err := myHelmClient.AddOrUpdateChartRepo(certmanagerchartRepo); err != nil {
log.Fatal(err)
} else {
fmt.Printf("Added Chart Repo %s\n", certmanagerchartRepo.Name)
}
if err := myHelmClient.UpdateChartRepos(); err != nil {
log.Fatal(err)
} else {
fmt.Printf("Updating Chart Repo\n")
}
certmanagerchartSpec := helmclient.ChartSpec{
ReleaseName: "cert-manager/cert-manager",
ChartName: "cert-manager",
Version: "v1.14.5",
Namespace: "cert-manager",
CreateNamespace: true,
ValuesOptions: values.Options{
StringValues: []string{
"extraArgs = {--dns01-recursive-nameservers=1.1.1.1:53}",
"controller.nodeSelector.kubernetes\\.io/os=linux",
},
},
SkipCRDs: false,
Wait: true,
}
certmanagermyInstalledHelmChart, err := myHelmClient.InstallChart(mycontext, &certmanagerchartSpec, nil)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Status of Chart Install %v,\n", *certmanagermyInstalledHelmChart.Info)
fmt.Println("Finished Deployment\n")
}
func connectToK8s() *kubernetes.Clientset {
home, exists := os.LookupEnv("HOME")
if !exists {
home = "C:\\Users\\your-user"
}
configPath := filepath.Join(home, ".kube", "config")
config, err := clientcmd.BuildConfigFromFlags("", configPath)
if err != nil {
log.Panicln("Failed to create K8s config")
}
clientset, err := kubernetes.NewForConfig(config)
if err != nil {
log.Panicln("Failed to create K8s clientset")
}
return clientset
}
func connectToAks(ctx context.Context, subid string, rg string, aksname string) *armcontainerservice.ManagedClustersClientGetAccessProfileResponse {
cred, err := azidentity.NewDefaultAzureCredential(nil)
if err != nil {
log.Fatalf("Failed to obtain a credential: %v", err)
}
clientFactory, err := armcontainerservice.NewClientFactory(subid, cred, nil)
if err != nil {
log.Fatalf("Failed to create client: %v", err)
}
res, err := clientFactory.NewManagedClustersClient().GetAccessProfile(ctx, rg, aksname, "clusterUser", nil)
if err != nil {
log.Fatalf("Failed to finish the request: %v", err)
}
return &res
}
func Ptr[T any](v T) *T {
return &v
}
The Challenges
Dependency Bugs: The biggest hurdle was a bug in the github.com/mittwald/go-helm-client package. Despite following the documentation and best practices, the Helm charts would not deploy correctly. I reported the issue here: mittwald/go-helm-client#209. This bug made deploying a working Helm chart impossible, derailing the project timeline.
Time Sensitivity: This project was highly time-sensitive. Every minute spent troubleshooting the bug was a minute closer to the deadline, adding to the stress and frustration.
Cert-Manager Objects: Deploying Kubernetes Cert-Manager objects using the native Kubernetes Go package was another pain point. Despite their theoretical simplicity, the objects did not deploy as expected, which was a significant setback, as Cert-Manager is critical for managing SSL/TLS certificates.
Lessons Learned
Dependency Management: Always vet your dependencies thoroughly. Consider contingency plans for potential issues with third-party libraries in critical projects.
Community and Support: Don't hesitate to reach out to the community or report issues.
Time Management: Factor in potential delays when working with new or less-tested tools. Buffer time can help you better manage unexpected challenges.
Conclusion
While this project was fraught with challenges, it was a valuable learning experience. I learned a ton about the internals of Kubernetes and Helm, which has improved my understanding of how to build Apps for Kubernetes. I also hope it gives you an insight into how it could be done.
Note: Please don't use this code, as it currently does not work. Once I have fixed the bug in the helm package, I will write another blog post about it.
