The Painful Journey of Deploying a Helm Chart to Kubernetes with Go

In the fast-paced world of software development, time-sensitive projects can often lead to both breakthroughs and challenges. Recently, I embarked on a project to deploy a Helm chart to a Kubernetes cluster using Go. This was because the helm Terraform provider is currently not working as I would expect it to and does not allow you to build an Azure Kubernetes Cluster and Deploy a Helm chart in the same run.

Despite my best efforts, a critical bug in a dependency and several deployment hurdles made the journey incredibly frustrating. Here's an account of my experience and the code I used, highlighting the pain points and lessons learned.

The Goal

The objective was simple: automate the deployment of a Helm chart to a Kubernetes cluster. This included setting up the Kubernetes Cert-Manager objects and leveraging the github.com/mittwald/go-helm-client package to manage Helm operations. Theoretically, it was straightforward. In practice, it proved to be anything but.

The Code

Here is the code I used for this project. Note that sensitive information has been omitted for security reasons.

package main

import (
    "bytes"
    "context"
    "fmt"
    "log"
    "os"
    "path/filepath"
    "time"

    "github.com/Azure/azure-sdk-for-go/sdk/azidentity"
    "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v5"
    certmanagerv1 "github.com/cert-manager/cert-manager"
    helmclient "github.com/mittwald/go-helm-client"
    "github.com/mittwald/go-helm-client/values"
    "helm.sh/helm/v3/pkg/repo"
    core "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
    _ "k8s.io/client-go/plugin/pkg/client/auth"
    clientcmd "k8s.io/client-go/tools/clientcmd"

    "github.com/Azure/azure-sdk-for-go/sdk/azidentity"
    "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v5"
)

func main() {
    var outputBuffer bytes.Buffer
    ctx := context.Background()
    mycontext, _ := context.WithTimeout(ctx, 80000*time.Second)

    certmanageremail := "your-email@example.com"
    subid := "your-subscription-id"
    rg := "your-resource-group"
    cluster := "your-cluster-name"

    fmt.Println("Connecting to Cluster Via Azure Call\n")
    myaksconnect := connectToAks(ctx, subid, rg, cluster)
    mypublicip := "0.0.0.0"
    fmt.Printf("Moved %v to KubeConfig File", &myaksconnect.Name)
    fmt.Println("Connecting to Cluster using KubeConfig File\n")
    kubeClient := connectToK8s()

    fmt.Println("Building Helm Client\n")
    opt := &helmclient.Options{
        Namespace:        "default",
        RepositoryCache:  "/tmp/.helmcache",
        RepositoryConfig: "/tmp/.helmrepo",
        Debug:            false,
        Linting:          true,
        DebugLog:         func(format string, v ...interface{}) {},
        Output:           &outputBuffer,
    }

    myHelmClient, err := helmclient.New(opt)
    if err != nil {
        panic(err)
    }

    fmt.Println("Deploying nginx-ingress\n")
    nginxchartRepo := repo.Entry{
        Name: "nginx-ingress",
        URL:  "https://kubernetes.github.io/ingress-nginx",
    }

    if err := myHelmClient.AddOrUpdateChartRepo(nginxchartRepo); err != nil {
        log.Fatal(err)
    } else {
        fmt.Printf("Added Chart Repo %s\n", nginxchartRepo.Name)
    }

    if err := myHelmClient.UpdateChartRepos(); err != nil {
        log.Fatal(err)
    } else {
        fmt.Printf("Updating Chart Repo\n")
    }

    nginxchartSpec := helmclient.ChartSpec{
        ReleaseName:     "nginx-ingress/nginx-ingress",
        ChartName:       "nginx-ingress",
        Namespace:       "nginx-ingress",
        CreateNamespace: true,
        SkipCRDs:        false,
        Wait:            true,
        ValuesOptions: values.Options{
            StringValues: []string{
                "rbac.create=false",
                "controller.service.externalTrafficPolicy=Local",
                fmt.Sprintf("controller.service.loadBalancerIP=%v", mypublicip),
                "controller.replicaCount=2",
                "controller.nodeSelector.kubernetes\\.io/os=linux",
                "defaultBackend.nodeSelector.kubernetes\\.io/os=linux",
                "controller.admissionWebhooks.patch.nodeSelector.kubernetes\\.io/os=linux",
                "controller.publishService.enabled=true",
                "controller.service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path=/healthz",
            },
        },
    }
    nginxInstalledHelmChart, err := myHelmClient.InstallChart(mycontext, &nginxchartSpec, nil)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("Status of Chart Install %v,\n", *nginxInstalledHelmChart.Info)

    fmt.Println("Deploying cert-manager\n")
    certmanagerchartRepo := repo.Entry{
        Name: "cert-manager",
        URL:  "https://charts.jetstack.io",
    }

    if err := myHelmClient.AddOrUpdateChartRepo(certmanagerchartRepo); err != nil {
        log.Fatal(err)
    } else {
        fmt.Printf("Added Chart Repo %s\n", certmanagerchartRepo.Name)
    }

    if err := myHelmClient.UpdateChartRepos(); err != nil {
        log.Fatal(err)
    } else {
        fmt.Printf("Updating Chart Repo\n")
    }

    certmanagerchartSpec := helmclient.ChartSpec{
        ReleaseName:     "cert-manager/cert-manager",
        ChartName:       "cert-manager",
        Version:         "v1.14.5",
        Namespace:       "cert-manager",
        CreateNamespace: true,
        ValuesOptions: values.Options{
            StringValues: []string{
                "extraArgs = {--dns01-recursive-nameservers=1.1.1.1:53}",
                "controller.nodeSelector.kubernetes\\.io/os=linux",
            },
        },
        SkipCRDs: false,
        Wait:     true,
    }
    certmanagermyInstalledHelmChart, err := myHelmClient.InstallChart(mycontext, &certmanagerchartSpec, nil)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("Status of Chart Install %v,\n", *certmanagermyInstalledHelmChart.Info)

    // Further deployments omitted for brevity...

    fmt.Println("Finished Deployment\n")
}

func connectToK8s() *kubernetes.Clientset {
    home, exists := os.LookupEnv("HOME")
    if !exists {
        home = "C:\\Users\\your-user"
    }

    configPath := filepath.Join(home, ".kube", "config")

    config, err := clientcmd.BuildConfigFromFlags("", configPath)
    if err != nil {
        log.Panicln("Failed to create K8s config")
    }

    clientset, err := kubernetes.NewForConfig(config)
    if err != nil {
        log.Panicln("Failed to create K8s clientset")
    }

    return clientset
}

func connectToAks(ctx context.Context, subid string, rg string, aksname string) *armcontainerservice.ManagedClustersClientGetAccessProfileResponse {
    cred, err := azidentity.NewDefaultAzureCredential(nil)
    if err != nil {
        log.Fatalf("Failed to obtain a credential: %v", err)
    }

    clientFactory, err := armcontainerservice.NewClientFactory(subid, cred, nil)
    if err != nil {
        log.Fatalf("Failed to create client: %v", err)
    }
    res, err := clientFactory.NewManagedClustersClient().GetAccessProfile(ctx, rg, aksname, "clusterUser", nil)
    if err != nil {
        log.Fatalf("Failed to finish the request: %v", err)
    }
    return &res
}

func Ptr[T any](v T) *T {
    return &v
}

The Challenges

1. Dependency Bugs: The biggest hurdle was a bug in the github.com/mittwald/go-helm-client package. Despite following the documentation and best practices, the Helm charts would not deploy correctly. I reported the issue here: mittwald/go-helm-client#209. This bug made deploying a working Helm chart impossible, derailing the project timeline.

2. Time Sensitivity: This project was highly time-sensitive. Every minute spent troubleshooting the bug was a minute closer to the deadline, adding to the stress and frustration.

3. Cert-Manager Objects: Deploying Kubernetes Cert-Manager objects using the native Kubernetes Go package was another pain point. Despite their theoretical simplicity, the objects did not deploy as expected, which was a significant setback, as Cert-Manager is critical for managing SSL/TLS certificates.

Lessons Learned

1. Dependency Management: Always vet your dependencies thoroughly. Consider contingency plans for potential issues with third-party libraries in critical projects.

2. Community and Support: Don't hesitate to reach out to the community or report issues.

3. Time Management: Factor in potential delays when working with new or less-tested tools. Buffer time can help you better manage unexpected challenges.

Conclusion

While this project was fraught with challenges, it was a valuable learning experience. I learned a ton about the internals of Kubernetes and Helm, which has improved my understanding of how to build Apps for Kubernetes. I also hope it gives you an insight into how it could be done.

Note: Please don't use this code, as it currently does not work. Once I have fixed the bug in the helm package, I will write another blog post about it.