ec2_instances = [
  {
    name        = "jc-ec2-app001"
    role        = "web"
    size        = "t3.micro"
    application = "app"
    ami         = "ami-008687c5b5546727c"
    family      = "windows"
    volumes = [
      {
        device_name           = "/dev/sda1"
        size                  = 30
        volume_type           = "gp3"
        iops                  = 3000
        throughput            = 125
        encrypted             = true
        delete_on_termination = true
      },
      {
        device_name           = "/dev/xvdf"
        size                  = 20
        volume_type           = "gp3"
        iops                  = 3000
        throughput            = 125
        encrypted             = false
        delete_on_termination = true
      }
    ]
  },
]

The first thing I noticed when using Pulumi with Python is that you don’t have the option of setting up a resource with default values and then passing through new values from the yaml file to replace those default values and essentially getting Pulumi to work in a Terraform way.

This is where I build on the dictionary work I wrote about an article ago. If you want to take a look, check out the following link: Playing Around with Dictionaries in Python

For this example, I will make an AWS backup plan with default values and show how we can replace those with new ones from a yaml file.

I first created my arguments class, where I will pass through the new values I wish to replace. I have already put default values into that argument class under a variable called. backup_configs Of type dictionary. This Class will be the Argument class that will be passed on to the BackupPlanCreator
Class, which will do the heavy lifting of making the AWS Backup Vault and the AWS Backup Plan.

A snippet of the code for the argument class is here:

class BackupPlanCreatorArgs:
    def __init__(
        self,
        name: str,
        vault_name: str,
        base_tags: dict,
    ):
        self.name = name
        self.vault_name = vault_name
        self.base_tags = base_tags

        self.backup_configs = {
            "hourly": {
                "name": "bkp-hourly-ret7-jc",
                "rule_name": "hourly-backup-rule",
                "schedule": "cron(0 * * * ? *)",
                "delete_after_days": 7,
                "start_window": 60,
                "completion_window": 120,
                "recovery_point_tags": {
                    "aws_profile": "jc-profile",
                    "aws_region": "us-east-2",
                    "type": "hourly-backup",
                },
                "enable_windows_vss": True,
            },
            "daily": {
                "name": "bkp-daily-ret14-jc",
                "rule_name": "daily-backup-rule",
                "schedule": "cron(0 5 ? * * *)",
                "delete_after_days": 30,
                "start_window": 60,
                "completion_window": 125,
                "recovery_point_tags": {
                    "aws_profile": "jc-profile",
                    "aws_region": "us-east-2",
                    "type": "daily-backup",
                },
                "enable_windows_vss": True,
            },
            "weekly": {
                "name": "bkp-weekly-ret365-jc",
                "rule_name": "weekly-backup-rule",
                "schedule": "cron(0 6 ? * 2 *)",
                "delete_after_days": 365,
                "start_window": 60,
                "completion_window": 120,
                "recovery_point_tags": {
                    "aws_profile": "jc-profile",
                    "aws_region": "us-east-2",
                    "type": "weekly-backup",
                },
                "enable_windows_vss": True,
            },
            "monthly": {
                "name": "bkp-monthly-ret365-jc",
                "rule_name": "monthly-backup-rule",
                "schedule": "cron(0 6 1 * ? *)",
                "delete_after_days": 365,
                "start_window": 60,
                "completion_window": 120,
                "recovery_point_tags": {
                    "aws_profile": "jc-profile",
                    "aws_region": "us-east-2",
                    "type": "monthly-backup",
                },
                "enable_windows_vss": True,
            },
        }

I wrote the following code into the class, which will replace values in what is essentially a nested dictionary with multiple values. I will go through this code and explain what it does.

        (1) self.final_backup_config = self.backup_configs.copy()

           (2) for key, value in new_backup_config_values.items():
                if (
                   (3) isinstance(value, dict) and key in self.final_backup_config
                ):  # isinstance() checks to see if it has been passed a dictionary.
                    print("Found BackUp Config Dictionary Merging Values")
                   (4) for nested_key, nested_value in value.items():
                        self.final_backup_config[key][nested_key] = nested_value
                else:
               (5)     print("Do not have any new values keeping BackUp Config the same.")

1. Here, I take a copy of the backup_configs variable values. This goes back to my GoLang days; when passing a variable around, Go will make a copy of that variable so you still have the original values if you want to call them later. We will do more on that later.

2. The items() method I found to be quite clever in Python. Essentially, you can pass a whole dictionary to items() It will return a view object that displays a list of a dictionary’s key-value tuple pairs. Which is precisely what I needed for this project. To take that one step further and start manipulating that data, you can put the items() method into a for loop. I thought this was quite cool and did not require me to start looking at programming algorithms and tricks to manipulate this data.

3. The isinstance() the function I added because I am double-checking my code here, again, a nifty function that basically checks before continuing that we are working with a dictionary. Its because of this function that you could essentially put all of this into a module or a function, it allows your code to start becoming quite dynamic and perhaps called elsewhere in your project. I will need to do this further down the line.

4. for nested_key, nested_value in value.items(): As we are working with a nested dictionary, I need to access the nested value of the dictionary, as this is the actual value that the user will change.

5. print("Do not have any new values keeping BackUp Config the same."). Now, say that we have passed no changed values to this whole for loop. We then need a way to skip through and keep our values. When I first wrote this if/else statement, I thought it would work with changing defaults and that I would not have to write anything more. I thought I had finished the piece of code. Yet I later found that this code is step one in about five more steps to make it all optional.

Right now, we have something that we can pass from the yaml to manipulate the dictionary. If you were to do this every time and not keep your default values, then this would be all you need to do. Yet that’s not very Terraformy. Terraform allows you to either keep the values or change them. Therefore, to achieve that functionality, we need to do more work. Onwards!!

        (1) if new_backup_config_values is not None:
                for key, value in new_backup_config_values.items():
                    if (
                        isinstance(value, dict) and key in self.final_backup_config
                    ):  # isinstance() checks to see if it has been passed a dictionary.
                        print("Found BackUp Config Dictionary Merging Values")
                        for nested_key, nested_value in value.items():
                            self.final_backup_config[key][nested_key] = nested_value
        (2) else:
                print("Do not have any new values keeping BackUp Config the same.")

1. if new_backup_config_values is not None: The first part we need to put into all of this is an option to handle type None. This is because in our main.py the Pulumi function pulumi.Config().get("backupConfigs") reads from the Yaml file will sometimes have values, or it will return None for no values. Therefore, we need to handle that here.

2. else: If the function returns no values, then basically, we are keeping our defaults and handing them over to the BackupPlanCreator Class.

Yet, there is still one issue with our code. Our BackupPlanCreatorArgs Class does have a parameter that expects to receive the changed default values. That parameter is called: new_backup_config_values, If you remember rightly from earlier, this is the foundation for our whole for loop. Yet the idea is sometimes we will pass on new values, and sometimes we will not.

How to solve this one…

(1) backup_configs_yaml = pulumi.Config().get("backupConfigs")
  (2) if backup_configs_yaml == None:
            backup_creator_args = backup.BackupPlanCreatorArgs(
                 name="backupPlanCreator",
                 vault_name=vault_name,
             (3) new_backup_config_values=None,
                 base_tags=base_tags,
            )
(4) else:
      (5)  new_backup_config_values = yaml.safe_load(
            backup_configs_yaml
        )  # This safe loads yaml without Embedding Json and having future issues.
        backup_creator_args = backup.BackupPlanCreatorArgs(
            name="backupPlanCreator",
            vault_name=vault_name,
        (6) new_backup_config_values=new_backup_config_values,
            base_tags=base_tags,
        )

1. backup_configs_yaml = pulumi.Config().get("backupConfigs"): As described earlier, this function is the key to it all. It will either pass the values that we are changing or will pass the type None

2. if backup_configs_yaml == None: Once I figured that out, I could start building logic around that function and of passing type None

3. new_backup_config_values=None, If none is passed from the yaml file, then within the if statement logic, I am simply going to call the args Class and pass None. Taking a copy of backup_configs now allows us to pass the original variable through the for loop logic and to the BackupPlanCreator class with no changes. If we had not taken a copy earlier, our lives would have been made a bit harder, and new code would have had to be written to handle this outcome.

4. else: Well, this becomes self-explanatory, really, and perhaps I am teaching to suck eggs, but if the function pulumi.Config().get("backupConfigs") does pass values, then the code below will be executed.

5. new_backup_config_values = yaml.safe_load (backup_configs_yaml): This will safely load the yaml into the class and allow our For loop with the items() method to go through and read the new values. I liked this yaml.safeload() function because it means I don’t have to try and put json formatted code into YAML anyone who has been doing this for some time would know that’s an absolute headache. It also keeps my Yaml file very yaml code-standard and easy to read.

6. new_backup_config_values=new_backup_config_values This passes our new variable values into the class.

Here is a snippet of Yaml code for reference.

  backupConfigs:
    hourly:
      enable_windows_vss: false

That's how you can use Python to build Terraform-like behaviour into Pulumi and make your code more usable for future engineers.

I hope this has been helpful. Until next time,

Happy Coding,

The Cloud Dude.

When approaching this task, the first milestone was to get Python to read through the dictionary. I did this by hardcoding the dictionaries into my Python class and choosing a toggle to select that dictionary to see if Python was reading my dictionaries correctly.

The code below shows the passing of the toggle backup_type and the hard coding of the dictionary.

class BackupPlanCreator:
    def __init__(self, vault_name: str, base_tags: dict, backup_type: str):
        self.vault_name = vault_name
        self.base_tags = base_tags
        self.backup_type = backup_type

        # Create the backup vault
        self.backup_vault = aws.backup.Vault(
            resource_name="backupVault",  # Unique resource name
            name=self.vault_name,  # Actual vault name
            tags=self.base_tags,
        )

        # Define configurations for each backup type
        self.backup_configs = {
            "hourly": {
                "name": "bkp-hourly-ret7-jc",
                "rule_name": "hourly-backup-rule",
                "target_vault_name": self.backup_vault.name,
                "schedule": "cron(0 * * * ? *)",  # Every hour
                "delete_after_days": 7,
                "start_window": 60,
                "completion_window": 120,
                "enable_windows_vss": True,
            },
            "daily": {
                "name": "bkp-daily-ret14-jc",
                "rule_name": "daily-backup-rule",
                "target_vault_name": self.backup_vault.name,
                "schedule": "cron(0 5 ? * * *)",  # Daily at 5 AM
                "delete_after_days": 30,
                "start_window": 60,
                "completion_window": 120,
                "enable_windows_vss": True,
            },
            "weekly": {
                "name": "bkp-weekly-ret365-jc",
                "rule_name": "weekly-backup-rule",
                "target_vault_name": self.backup_vault.name,
                "schedule": "cron(0 6 1 * ? *)",  # Every Monday at 5 AM
                "delete_after_days": 365,
                "start_window": 60,
                "completion_window": 120,
                "enable_windows_vss": True,
            },
            "monthly": {
                "name": "bkp-monthly-ret365-jc",
                "rule_name": "monthly-backup-rule",
                "target_vault_name": self.backup_vault.name,
                "schedule": "cron(0 6 1 * ? *)",  # 1st of every month at 5 AM
                "delete_after_days": 365,
                "start_window": 60,
                "completion_window": 120,
                "enable_windows_vss": True,
            },
        }

Below is the same class reading through the dictionary and compiling the backup_rules for the ultimate plan.

# Get the configuration for the specified backup type
        config = self.backup_configs.get(self.backup_type)
        if not config:
            raise ValueError(
                "Invalid backup type specified. Choose from: 'hourly', 'daily', 'weekly', 'monthly'."
            )

        # Define the backup rules using the configuration from the dictionary
        backup_rules = [
            {
                "rule_name": config["rule_name"],
                "target_vault_name": self.backup_vault.name,
                "completion_window": config["completion_window"],
                "copy_actions": [
                    {
                        "destination_vault_arn": self.backup_vault.arn,
                        "lifecycle": {
                            "cold_storage_after": 0,
                            "delete_after": config["delete_after_days"],
                            "opt_in_to_archive_for_supported_resources": False,
                        },
                    }
                ],
                "enable_continuous_backup": False,
                "lifecycle": {
                    "cold_storage_after": 0,
                    "delete_after": config["delete_after_days"],
                    "opt_in_to_archive_for_supported_resources": False,
                },
                "schedule": config["schedule"],
                "schedule_expression_timezone": "UTC",
                "start_window": config["start_window"],
            }
        ]

        # Define the advanced backup settings
        advanced_backup_settings = [
            {
                "backup_options": {
                    "WindowsVSS": (
                        "enabled" if config["enable_windows_vss"] else "disabled"
                    ),
                },
                "resource_type": "EC2",
            }
        ]

As you can see, a for loop is not currently needed, as I am only seeing if Python can read through and select the correct dictionary. I then pass backup_type to the Python Backup Plan Function as a string. This one is daily to choose the daily backup plan out of the dictionary.

# Create the backup plan
        backup_creator = backup.BackupPlanCreator(vault_name, base_tags, "daily")

This is great and shows me that I can process a dictionary. However, I want to take this a step further and pass the dictionary around. To achieve this, I moved the dictionary out of the class and into a variable to see if I could pass it through and get the same result.

import pulumi
import pulumi_aws as aws
import jc_aws_backup as backup

# Define configurations for each backup type
   backup_configs = {
            "hourly": {
                "name": "bkp-hourly-ret7-jc",
                "rule_name": "hourly-backup-rule",
                "target_vault_name": self.backup_vault.name,
                "schedule": "cron(0 * * * ? *)",  # Every hour
                "delete_after_days": 7,
                "start_window": 60,
                "completion_window": 120,
                "enable_windows_vss": True,
            },
            "daily": {
                "name": "bkp-daily-ret14-jc",
                "rule_name": "daily-backup-rule",
                "target_vault_name": self.backup_vault.name,
                "schedule": "cron(0 5 ? * * *)",  # Daily at 5 AM
                "delete_after_days": 30,
                "start_window": 60,
                "completion_window": 125,
                "enable_windows_vss": True,
            },
            "weekly": {
                "name": "bkp-weekly-ret365-jc",
                "rule_name": "weekly-backup-rule",
                "target_vault_name": self.backup_vault.name,
                "schedule": "cron(0 6 1 * ? *)",  # Every Monday at 5 AM
                "delete_after_days": 365,
                "start_window": 60,
                "completion_window": 120,
                "enable_windows_vss": True,
            },
            "monthly": {
                "name": "bkp-monthly-ret365-jc",
                "rule_name": "monthly-backup-rule",
                "target_vault_name": self.backup_vault.name,
                "schedule": "cron(0 6 1 * ? *)",  # 1st of every month at 5 AM
                "delete_after_days": 365,
                "start_window": 60,
                "completion_window": 120,
                "enable_windows_vss": True,
            },
        }

# Create the backup plan
        backup_creator = backup.BackupPlanCreator(vault_name, base_tags, backup_configs, "daily")

import pulumi
import pulumi_aws as aws


class CorityBackupPlanCreator:
    def __init__(
        self, vault_name: str, base_tags: dict, backup_configs: dict, backup_type: str
    ):
        self.base_tags = base_tags
        self.backup_type = backup_type
        self.backup_configs = backup_configs

        # Create the backup vault
        self.backup_vault = aws.backup.Vault(
            vault_name,
            tags=self.base_tags,
        )

        config = self.backup_configs.get(self.backup_type)
        if not config:
            raise ValueError(
                "Invalid backup type specified. Choose from: 'hourly', 'daily', 'weekly', 'monthly'."
            )
        # Define the backup rules using the configuration from the dictionary
        backup_rules = []
        for _, backup_details in self.backup_configs.items():
            backup_rules.append(
                aws.backup.PlanRuleArgs(
                    rule_name=backup_details["rule_name"],
                    target_vault_name=self.backup_vault.name,
                    schedule=backup_details["schedule"],
                    lifecycle=aws.backup.PlanRuleLifecycleArgs(
                        delete_after=backup_details["delete_after_days"]
                    ),
                    start_window=backup_details["start_window"],
                    completion_window=backup_details["completion_window"],
                    recovery_point_tags=self.base_tags,
                )
            )

        # Define the advanced backup settings
        advanced_backup_settings = []
        for _, backup_details in self.backup_configs.items():
            advanced_backup_settings.append(
                aws.backup.PlanAdvancedBackupSettingArgs(
                    backup_options={
                        "WindowsVSS": (
                            "enabled"
                            if backup_details["enable_windows_vss"]
                            else "disabled"
                        )
                    },
                    resource_type="EC2",
                )
            )

        # Create the backup plan
        self.plan_resource = aws.backup.Plan(
            resource_name=backup_details["rule_name"],
            rules=backup_rules,
            advanced_backup_settings=advanced_backup_settings,
            tags=self.base_tags,
        )

        # Export the backup plan ID
        pulumi.export("backupPlanId", self.plan_resource.id)

What I also did here was add a for loop because my logic will change later, and I will want all four backup plans made at once from the dictionary. Yet this showed me that I could pass around a dictionary, and If I could pass around a dictionary, that meant I could also make it come from the central pulumi.yaml file and load it as an json object into the class. This is because when working with Pulumi, the idea you want to get to is to pass everything through one central yaml file that is linked to each environment stack. That is always the end goal. I then changed my main.py code around a bit like this to achieve this.

import pulumi
import pulumi_aws as aws
import json 
import jc_aws_backup as backup

backup_configs_json = pulumi.Config().require("backupConfigs")
vault_name = f"bkp-vault-{project_profile}-{project_short_region}"[:42]

# Load the Backup configs from the Yaml and pass as a Json String.
# Do this with Error Handling so you can see where in JSON it is failing.
try:
    backup_configs = json.loads(backup_configs_json)
except json.JSONDecodeError as e:
    raise ValueError(f"Invalid JSON in backupConfigs: {e}")

# Create the backup plan
backup_creator = backup.BackupPlanCreator(
    vault_name=vault_name,
    base_tags=base_tags,
    backup_type="daily"
    backup_configs=backup_configs,
)

I also added some Error Handling. This is because when I first passed through the json from the yaml file, I got a panic error shown below, and it was a pretty unhelpful error in figuring out where in the json it was failing.

Diagnostics:
  pulumi:pulumi:Stack (jcontent-dev-v2-jcontent_dev_v2):
    error: Program failed with an unhandled exception:
    Traceback (most recent call last):
      File "C:\DevOps\infrastructure.pulumi\iac-development\jcontent-dev-v2\__main__.py", line 70, in <module>
        backup_configs = json.loads(backup_configs_json)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "C:\Python312\Lib\json\__init__.py", line 346, in loads
        return _default_decoder.decode(s)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "C:\Python312\Lib\json\decoder.py", line 337, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "C:\Python312\Lib\json\decoder.py", line 355, in raw_decode
        raise JSONDecodeError("Expecting value", s, err.value) from None
    json.decoder.JSONDecodeError: Expecting value: line 23 column 27 (char 602)

By creating my Error handling, I get it to show me where exactly in the json object it is failing.
I have also supplied the yaml file below to show you how to pass in the object. This stumped me for a little while, and having an example would have helped.

encryptionsalt: VALUE
config: 
aws:region: us-east-2
environment: jc_testing_pulumi
backupConfigs: |
    {
      "hourly": {
        "name": "bkp-hourly-ret7-jc",
        "rule_name": "hourly-backup-rule",
        "schedule": "cron(0 * * * ? *)",
        "delete_after_days": 7,
        "start_window": 60,
        "completion_window": 120,
        "enable_windows_vss": true
      },
      "daily": {
        "name": "bkp-daily-ret14-jc",
        "rule_name": "daily-backup-rule",
        "schedule": "cron(0 5 ? * * *)",
        "delete_after_days": 30,
        "start_window": 60,
        "completion_window": 125,
        "enable_windows_vss": true
      },
      "weekly": {
        "name": "bkp-weekly-ret365-jc",
        "rule_name": "weekly-backup-rule",
        "schedule": "cron(0 6 ? * 2 *)",
        "delete_after_days": 365,
        "start_window": 60,
        "completion_window": 120,
        "enable_windows_vss": true
      },
      "monthly": {
        "name": "bkp-monthly-ret365-jc",
        "rule_name": "monthly-backup-rule",
        "schedule": "cron(0 6 1 * ? *)",
        "delete_after_days": 365,
        "start_window": 60,
        "completion_window": 120,
        "enable_windows_vss": true
      }
    }

This now prepares me for my final hurdle and task; what I actually want to happen is for the backup plan class to make all four backup plans in one run and name them accordingly. How I achieved this was first to remove backup_type from my Class and to change my for loop around to include the aws.backup.plan() function. Yet, I still needed it to name each plan for me. The code below shows how I achieved this.

import json
import pulumi
import pulumi_aws as aws


class BackupPlanCreator:
    def __init__(self, vault_name: str, base_tags: dict, backup_configs: dict):
        self.base_tags = base_tags
        self.backup_configs = backup_configs

        # Create the backup vault
        self.backup_vault = aws.backup.Vault(
            vault_name,
            tags=self.base_tags,
        )

        # Loop through each backup configuration and create a separate backup plan
        for backup_name, backup_details in self.backup_configs.items():
            # Define the backup rules
            backup_plan = aws.backup.Plan(
                resource_name=f"{backup_name}-backup-plan",
                rules=[
                    {
                        "rule_name": backup_details["rule_name"],
                        "target_vault_name": self.backup_vault.name,
                        "schedule": backup_details["schedule"],
                        "lifecycle": {
                            "delete_after": backup_details["delete_after_days"],
                        },
                        "start_window": backup_details["start_window"],
                        "completion_window": backup_details["completion_window"],
                        "recovery_point_tags": self.base_tags,
                    }
                ],
                # Define the advanced backup settings
                advanced_backup_settings=[
                    {
                        "backup_options": {
                            "WindowsVSS": (
                                "enabled"
                                if backup_details["enable_windows_vss"]
                                else "disabled"
                            ),
                        },
                        "resource_type": "EC2",
                    }
                ],
                tags=self.base_tags,
            )

The call to this function is similar to before, without the backup_type part being passed.

import pulumi
import pulumi_aws as aws
import json 
import jc_aws_backup as backup

backup_configs_json = pulumi.Config().require("backupConfigs")
vault_name = f"bkp-vault-{project_profile}-{project_short_region}"[:42]

# Load the Backup configs from the Yaml and pass as a Json String.
# Do this with Error Handling so you can see where in JSON it is failing.
try:
    backup_configs = json.loads(backup_configs_json)
except json.JSONDecodeError as e:
    raise ValueError(f"Invalid JSON in backupConfigs: {e}")

# Create the backup plan
backup_creator = backup.BackupPlanCreator(
    vault_name=vault_name,
    base_tags=base_tags,
    backup_configs=backup_configs,
)

And that’s it. That’s how I have managed to pass around a dictionary and evolve that into an actual, useful json object to then make multiple plans for one backup plan run.

I will build on this dictionary now and put in some functionality that you see in Terraform. If you want to use default values, you can, or if you're going to override those values, you can. The thing with my dictionary right now is everything is being passed through, but essentially, the engineer will not want to do that. They will probably only want to configure certain parts of the dictionary from the yaml file.

Yet this is a cliffhanger moment for another blog post for another time.

Till next time, happy coding.

Managing AWS resources efficiently is a critical challenge for cloud architects and developers, especially using a dynamic language like Python. While Python offers flexibility and rapid development capabilities, its lack of built-in type safety can lead to runtime errors that are difficult to debug. Inspired by Programming Design concepts from Will Kennedy's Go Lesson at GopherCon this year, this article explores how implementing robust input validation can mitigate these issues, offering a clearer, more maintainable approach to developing AWS infrastructure with Pulumi and Python.

Why Input Validation Matters

Error Prevention and Pipeline Efficiency

Input validation helps catch potential errors early in the development process, well before they reach production environments. By ensuring that inputs to your system—such as configuration variables for AWS resources—are of the correct type and format, you mitigate the risk of runtime errors that can disrupt services and degrade user experience. This preemptive approach reduces the need for rollbacks and hotfixes, enhancing the reliability of your deployment processes.

Type Safety in Python

Python's dynamic typing offers flexibility but lacks the inherent type safety of a language like Go. Implementing input validation through decorators can bring some of the benefits of static typing to Python, ensuring that functions receive arguments of the expected types.

Implementing Input Validation in Python

The Decorator Pattern

The decorator pattern provides a simple yet powerful way to enforce input validation in Python. By wrapping class methods with a decorator, you can systematically check the types of arguments and raise descriptive errors when mismatches occur.

class CloudDudeFSxArgs:
    @staticmethod
    def type_check_decorator(func):
        def wrapper(self, *args, **kwargs):
            # Map positional arguments to their names
            arg_names = [
                "create_active_directory",
                "service_name",
                "domain_name_suffix",
                "administrator_password",
                "directory_service_id",
                "deployment_type",
                "storage_type",
                "storage_capacity",
                "throughput_capacity",
                "automatic_backup_retention_days",
                "copy_tags_to_backups",
                "weekly_maintenance_start_time",
                "environment",
                "base_tags",
                "cor_selected_vpc",
                "cor_subnets_2",
                "cor_subnets_1",
                "cority_ems_sg",
            ]

            # Combine args and kwargs into a single dictionary
            all_args = {**dict(zip(arg_names, args)), **kwargs}

            # Check types of specific arguments
            bool_vars = ["create_active_directory", "copy_tags_to_backups"]
            for boolvar in bool_vars:
                if not isinstance(all_args.get(boolvar), bool):
                    raise TypeError(f"{boolvar} must be a boolean")

            int_vars = [
                "storage_capacity",
                "throughput_capacity",
                "automatic_backup_retention_days",
            ]
            for var in int_vars:
                if not isinstance(all_args.get(var), int):
                    raise TypeError(f"{var} must be an integer")

            str_vars = [
                "service_name",
                "domain_name_suffix",
                "administrator_password",
                "directory_service_id",
                "deployment_type",
                "storage_type",
                "weekly_maintenance_start_time",
                "environment",
                "cor_selected_vpc",
                "cor_subnets_2",
                "cor_subnets_1",
                "cority_ems_sg",
            ]
            for strvar in str_vars:
                if not isinstance(all_args.get(strvar), str):
                    raise TypeError(f"{strvar} must be a string")

            # Call the original function
            return func(self, *args, **kwargs)

        return wrapper

    @type_check_decorator
    def __init__(self, *args, **kwargs):
        # Initialize attributes
        self.create_active_directory = kwargs.get("create_active_directory")
        self.service_name = kwargs.get("service_name")
        self.domain_name_suffix = kwargs.get("domain_name_suffix")
        self.administrator_password = kwargs.get("administrator_password")
        self.directory_service_id = kwargs.get("directory_service_id")
        self.deployment_type = kwargs.get("deployment_type")
        self.storage_type = kwargs.get("storage_type")
        self.storage_capacity = kwargs.get("storage_capacity")
        self.throughput_capacity = kwargs.get("throughput_capacity")
        self.automatic_backup_retention_days = kwargs.get(
            "automatic_backup_retention_days"
        )
        self.copy_tags_to_backups = kwargs.get("copy_tags_to_backups")
        self.weekly_maintenance_start_time = kwargs.get("weekly_maintenance_start_time")
        self.environment = kwargs.get("environment")
        self.base_tags = kwargs.get("base_tags")
        self.cor_selected_vpc = kwargs.get("cor_selected_vpc")
        self.cor_subnets_2 = kwargs.get("cor_subnets_2")
        self.cor_subnets_1 = kwargs.get("cor_subnets_1")
        self.cority_ems_sg = kwargs.get("cority_ems_sg")

Benefits of Input Validation

Readable Error Messages

Without input validation, errors can manifest in verbose stack traces that are often hard to decipher. Consider this typical Python error:

Traceback (most recent call last):
  File "C:\Program Files (x86)\Pulumi\pulumi-language-python-exec", line 191, in <module>
    loop.run_until_complete(coro)
  File "C:\Python312\Lib\asyncio\base_events.py", line 687, in run_until_complete
    return future.result()
  File "C:\DevOps\awsinfrastructure.pulumi\iac-development\jcontent-dev\venv\Lib\site-packages\pulumi\runtime\stack.py", line 142, in run_in_stack
    await run_pulumi_func(run)
  File "C:\DevOps\awsinfrastructure.pulumi\iac-development\jcontent-dev\venv\Lib\site-packages\pulumi\runtime\stack.py", line 56, in run_pulumi_func
    await wait_for_rpcs()
  File "C:\DevOps\awsinfrastructure.pulumi\iac-development\jcontent-dev\venv\Lib\site-packages\pulumi\runtime\stack.py", line 89, in wait_for_rpcs
    raise exn from cause
  File "C:\DevOps\awsinfrastructure.pulumi\iac-development\jcontent-dev\venv\Lib\site-packages\pulumi\runtime\stack.py", line 81, in wait_for_rpcs
    await rpc_manager.rpcs.pop()
AssertionError: Unexpected type. Expected 'list' got '<class 'str'>'

In contrast, input validation provides clearer, more concise error messages:

Diagnostics:
  pulumi:pulumi:Stack (jcontent-dev-v2-jcontent_dev_v2):
    error: Program failed with an unhandled exception:
    Traceback (most recent call last):
      File "C:\DevOps\infrastructure.pulumi\iac-development\jcontent-dev-v2\__main__.py", line 60, in <module>
        fsxArgs = fsx.CorityFSxArgs(
                  ^^^^^^^^^^^^^^^^^^
      File "C:\DevOps\infrastructure.pulumi\iac-development\jcontent-dev-v2\../../pkg\cority_aws_fsx.py", line 144, in wrapper
        raise TypeError(f"{boolvar} must be a boolean")
    TypeError: copy_tags_to_backups must be a boolean

The bottom line of the error message is more concise and tells you where the problem is.

TypeError: copy_tags_to_backups must be a boolean

These specific messages reduce debugging time and improve developer productivity by directly pointing to the source of the error.

Improved Debugging

With input validation, errors are caught and reported closer to their source, making tracing and fixing issues easier. This clarity accelerates the development process and ensures smoother project execution.

Cross-Package Consistency

Input validation ensures that data flows consistently between different modules and packages. This consistency is crucial when using functions like pulumi.export to pass outputs between resources, allowing for controlled and predictable data handling. For instance, when handling Pulumi export values from one class to another, input validation ensures that the data types remain consistent, preventing integration issues and facilitating smoother transitions between components.

Conclusion

Input validation is a foundational practice that enhances the robustness and reliability of AWS resource management in Python. By adopting these techniques, developers can leverage Python's flexibility while mitigating the risks associated with dynamic typing. As cloud infrastructures become increasingly complex, the importance of such practices cannot be overstated. Implementing robust input validation improves code quality and fosters a culture of precision and accountability in software development, ultimately leading to more efficient and error-free deployment pipelines.

Happy Coding,

The Cloud Dude

When designing a scalable and flexible architecture on AWS, it's common to use an Application Load Balancer (ALB) to distribute traffic across multiple targets, such as EC2 instances and ECS tasks. However, a challenge arises when you must support both EC2 instances and ECS Fargate tasks with the same ALB, especially when the target types differ. This article explains why and how multiple target groups for an ALB can be used to support both EC2 instances and ECS Fargate tasks.

Problem Statement

In a recent work project, I encountered an issue where I needed to attach both EC2 instances and ECS Fargate tasks to an ALB. The ALB target group for ECS Fargate tasks requires target_type="ip", while the target group for EC2 instances requires target_type="instance". Attempting to use a single target group for both resulted in compatibility issues. When coding with Pulumi, AWS interprets the instance ID as an IPv6 address using target_type="ip", resulting in a failed creation attempt.

Solution: Multiple Target Groups

To resolve this issue, we decided to create two separate target groups:

1. Target Group for EC2 Instances: Uses target_type="instance".

2. Target Group for ECS Fargate Tasks: Uses target_type="ip".

This approach allows us to leverage the strengths of both EC2 instances and ECS Fargate tasks while ensuring compatibility with the ALB.

Implementation

Here’s a step-by-step guide on implementing this solution using Pulumi in Python.

Step 1: Define the ALB with Two Target Groups

First, we must create an ALB with two target groups: one for EC2 instances and one for ECS Fargate tasks.

import pulumi
import pulumi_aws as aws
from pulumi import ResourceOptions, Output
from typing import Mapping, List

class ALBArgs:
    """The arguments necessary to construct an `ALB` resource"""

    def __init__(
        self,
        base_tags: Mapping[str, str],
        internal: bool,
        subnets: List[str],
        security_groups: List[str],
        enable_deletion_protection: bool,
        vpc_id: str,
        webserver_id: str,
        alb_listener: str,
    ):
        self.internal = internal
        self.subnets = subnets
        self.security_groups = security_groups
        self.enable_deletion_protection = enable_deletion_protection
        self.vpc_id = vpc_id
        self.webserver_id = webserver_id
        self.alb_listener = alb_listener
        self.base_tags = base_tags

class JCProjectALB(pulumi.ComponentResource):
    def __init__(self, resource_name: str, args: ALBArgs, opts: ResourceOptions = None):
        super().__init__("cority:aws:alb", resource_name, {}, opts)

        self.alb_name = resource_name
        self.internal = args.internal
        self.subnets = args.subnets
        self.security_groups = args.security_groups
        self.enable_deletion_protection = args.enable_deletion_protection
        self.vpc_id = args.vpc_id
        self.webserver_id = args.webserver_id
        self.alb_listener = args.alb_listener
        self.base_tags = args.base_tags

        self.alb = aws.lb.LoadBalancer(
            self.alb_name,
            internal=self.internal,
            load_balancer_type="application",
            security_groups=self.security_groups,
            subnets=self.subnets,
            enable_deletion_protection=self.enable_deletion_protection,
            tags=self.base_tags,
            opts=ResourceOptions(parent=self),
        )

        # Target group for EC2 instances
        self.alb_target_group_instance = aws.lb.TargetGroup(
            f"{self.alb_name[:20]}-tg-instance",
            port=80,
            protocol="HTTP",
            target_type="instance",
            vpc_id=self.vpc_id,
            opts=ResourceOptions(parent=self.alb),
        )

        # Target group for Fargate tasks
        self.alb_target_group_ip = aws.lb.TargetGroup(
            f"{self.alb_name[:20]}-tg-ip",
            port=80,
            protocol="HTTP",
            target_type="ip",
            vpc_id=self.vpc_id,
            opts=ResourceOptions(parent=self.alb),
        )

        self.alb_target_group_attachment = aws.lb.TargetGroupAttachment(
            f"{self.alb_name[:10]}-tg-attachment",
            target_id=self.webserver_id,
            target_group_arn=self.alb_target_group_instance.arn,
            port=80,
            opts=ResourceOptions(parent=self.alb_target_group_instance),
        )

        self.listener = aws.lb.Listener(
            f"{self.alb_name}-listener",
            load_balancer_arn=self.alb.arn,
            port=80,
            default_actions=[
                aws.lb.ListenerDefaultActionArgs(
                    type="forward",
                    target_group_arn=self.alb_target_group_ip.arn,
                )
            ],
            opts=ResourceOptions(parent=self.alb),
        )

        pulumi.export("alb_name", self.alb_name)
        pulumi.export("listener_arn", self.listener.arn)
        pulumi.export("targetgroup_instance_arn", self.alb_target_group_instance.arn)
        pulumi.export("targetgroup_ip_arn", self.alb_target_group_ip.arn)

        self.register_outputs(
            {
                "alb_name": self.alb_name,
                "listener_arn": self.listener.arn,
                "targetgroup_instance_arn": self.alb_target_group_instance.arn,
                "targetgroup_ip_arn": self.alb_target_group_ip.arn,
            }
        )

Step 2: Define the ECS Service to Use the IP Target Group

Next, we must configure the ECS service to use the target group with target_type="ip".

import pulumi
import pulumi_aws as aws
from pulumi import ResourceOptions, Output
from typing import Mapping, List
import applicationloadbalancer
import compute


class ECSArgs:
    """The arguments necessary to construct an `ECS` resource"""

    def __init__(
        self,
        base_tags: Mapping[str, str],
        internal: bool,
        subnets: List[str],
        security_groups: List[str],
        enable_deletion_protection: bool,
        vpc_id: str,
        container_name: str,
        alb_listener: aws.lb.Listener,
        alb_target_group_arn: str,
    ):
        self.internal = internal
        self.subnets = subnets
        self.security_groups = security_groups
        self.enable_deletion_protection = enable_deletion_protection
        self.vpc_id = vpc_id
        self.base_tags = base_tags
        self.alb_target_group_arn = alb_target_group_arn
        self.alb_listener = alb_listener
        self.container_name = container_name


class JCProjectECS(pulumi.ComponentResource):
    def __init__(self, resource_name: str, args: ECSArgs, opts: ResourceOptions = None):
        super().__init__("cority:aws:ecs", resource_name, {}, opts)

        cluster_name = resource_name
        self.internal = args.internal
        self.subnets = args.subnets
        self.security_groups = args.security_groups
        self.enable_deletion_protection = args.enable_deletion_protection
        self.vpc_id = args.vpc_id
        self.base_tags = args.base_tags
        self.alb_target_group_arn = args.alb_target_group_arn
        self.alb_listener = args.alb_listener
        self.container_name = args.container_name

        # Create an ECS cluster
        self.cluster = aws.ecs.Cluster(cluster_name)

        # Create an IAM role for task execution
        task_execution_role = aws.iam.Role(
            "taskExecutionRole",
            assume_role_policy="""{
                "Version": "2012-10-17",
                "Statement": [
                    {
                        "Effect": "Allow",
                        "Principal": {
                            "Service": "ecs-tasks.amazonaws.com"
                        },
                        "Action": "sts:AssumeRole"
                    }
                ]
            }""",
        )

        # Attach the required policies to the task execution role
        task_execution_role_policy = aws.iam.RolePolicyAttachment(
            "taskExecutionRolePolicy",
            role=task_execution_role.name,
            policy_arn="arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy",
        )

        # Create a Fargate Task Definition
        self.task_definition = aws.ecs.TaskDefinition(
            "task_definition",
            family="my-fargate-task",
            cpu="256",
            memory="512",
            network_mode="awsvpc",
            requires_compatibilities=["FARGATE"],
            execution_role_arn=task_execution_role.arn,
            container_definitions=Output.all().apply(
                lambda _: """[
                    {
                        "name": "investorportalcont",
                        "image": "nginx",
                        "cpu": 256,
                        "memory": 512,
                        "essential": true,
                        "portMappings": [
                            {
                                "containerPort": 80,
                                "hostPort": 80,
                                "protocol": "tcp"
                            }
                        ]
                    }
                ]"""
            ),
        )
     # ECS Service connected to the existing ALB
        self.service = aws.ecs.Service(
            "ecs_service",
            cluster=self.cluster.arn,
            task_definition=self.task_definition.arn,
            desired_count=1,
            launch_type="FARGATE",
            network_configuration=aws.ecs.ServiceNetworkConfigurationArgs(
                subnets=args.subnets,
                security_groups=args.security_groups,
            ),
            load_balancers=[
                aws.ecs.ServiceLoadBalancerArgs(
                    target_group_arn=args.alb_target_group_arn,
                    container_name=args.container_name,
                    container_port=80,
                ),
            ],
            opts=ResourceOptions(depends_on=[args.alb_listener]),
        )

        pulumi.export("cluster_name", self.cluster.name)
        pulumi.export("service_name", self.service.name)

        self.register_outputs(
            {
                "cluster_name": self.cluster.name,
                "service_name": self.service.name,
            }
        )

Step 3: Putting everything together main.py

We must now code main.py to pass through both alb target groups, one for the ECS and one for the EC2 instance; the actual connection for the EC2 instance happens within the ALB code as we point it to the `

 target_group_arn=self.alb_target_group_ip.arn` line.

import pulumi
import pulumi_aws as aws
import os
import sys

#### This Section of Code will go and look for the PKG folder to then be able to import the modules
#### It also sets the PYTHONPATH which is needed to succesfully see the modules.
module_path = "../pkg"
if module_path not in sys.path:
    sys.path.append(module_path)

# Try to import your custom module
try:
    import compute  ## This is just a module that we are testing for.

    pulumi.export("module_import_status", "success")
except ImportError as e:
    pulumi.export("module_import_status", "failed")
    pulumi.export("import_error_message", str(e))
#### END OF Import checking

# Import custom modules
import compute
import applicationloadbalancer
import networking
import ecs

# Define the VPC ID (Replace with your actual VPC ID)
vpc_id = "vpc-0f0587443694534b4"

# Networking infrastructure
cnetworking = networking.ExistingVPC(vpc_id)
vpcsubnet_id = cnetworking.subnet_id
vpcsecgroup_id = cnetworking.security_group_id

# Convert Security Groups into Input of Strings
security_group_strings = [sg for sg in vpcsecgroup_id]

# Compute infrastructure
ccompute = compute.CreateEC2Instances(
    "my-ec2-instance",
    compute.EC2Args(
        base_tags={"Project": "Investor Portal"},
        internal=False,
        ami_instance_size="t2.micro",
        subnet_id=vpcsubnet_id[0],
        security_groups=security_group_strings,
    ),
)

# Define ALB arguments and create ALB
alb_args = applicationloadbalancer.ALBArgs(
    subnets=vpcsubnet_id,
    security_groups=vpcsecgroup_id,
    base_tags={"Project": "Investor Portal"},
    internal=False,
    enable_deletion_protection=False,
    vpc_id=vpc_id,
    webserver_id=ccompute.server_instance.id,
    alb_listener="corityalblistener",
)

build_alb = applicationloadbalancer.JCProjectALB(
    "investorpalb",
    alb_args,
    opts=pulumi.ResourceOptions(depends_on=[ccompute]),
)

# Export ALB status
pulumi.export("alb_status", build_alb)

# Define ECS arguments
ecs_args = ecs.ECSArgs(
    base_tags={"Project": "Investor Portal"},
    internal=False,
    subnets=vpcsubnet_id,
    security_groups=security_group_strings,
    enable_deletion_protection=False,
    vpc_id=vpc_id,
    container_name="investorportalcont",
    alb_target_group_arn=build_alb.alb_target_group_ip.arn,
    alb_listener=build_alb.listener,
)

# Create ECS
ecs_cluster = ecs.JCProjectECS(
    "my-ecs", ecs_args, opts=pulumi.ResourceOptions(depends_on=[build_alb.listener])
)

# Export ECS status
pulumi.export("ecs_cluster_name", ecs_cluster.cluster.name)
pulumi.export("ecs_service_name", ecs_cluster.service.name)

Conclusion

When working with cloud infrastructure and Pulumi, you sometimes have to think outside the box as different errors come flying at you. In this instance, I hope I have shown you how to get around the simple issue of putting two completely different resources onto the same ALB.

Happy coding,
Cloud Dude

In the world of software development, encountering roadblocks is part of the journey. Recently, I faced a significant challenge while trying to deploy a Helm chart to a Kubernetes cluster using Go. Despite my best efforts and countless hours of troubleshooting, a critical bug in the github.com/mittwald/go-helm-client package thwarted my attempts. I wrote about that here: Painful Experience with Go and Kubernetes. However, Bash came to the rescue, allowing me to complete the deployment in a time-sensitive project at work. Here’s the story of how a simple Bash CLI Program saved the day.

The Challenge

The task was to automate the deployment of several Helm charts to a Kubernetes cluster. Initially, I opted to use Go for this purpose, leveraging the github.com/mittwald/go-helm-client package. Unfortunately, a bug in the package prevented the successful deployment of the Helm charts, as detailed in my bug report here. With the project deadline looming, I needed a quick and reliable alternative.

The Solution: Bash CLI Program

Bash scripts are often underestimated, but their simplicity and power can be a lifesaver in critical situations. I wrote a Bash CLI program to handle the Helm deployments and integrated it into my Terraform script using the null function. Here’s the Bash script that made it all possible:

#!/bin/bash
### Pass Flags
#Define the help function
function help(){
    echo "Options:";
    echo "-r    Resource Group Name"
    echo "-l    Location of Resource Group"
    echo "-c    Cluster Name"
    echo "-e    Email for LetsEncrypt"
    echo "-d    Domain name for Nginx"
    exit 1;
}

# Initialize the default values for the variables.
r="rg";
l="location";
c="cluster";
e="email";
d="domainname";

# Define the getopts variables
options=":r:l:c:e:d:h";

# Start the getopts code
while getopts ${options} opt; do
    case $opt in
            r) # Get the resource group name
                    rg=${OPTARG}
            ;;
            l) # Get the location
                  location=${OPTARG}
            ;;
            c) # Get the cluster name
                    cluster=${OPTARG}
            ;;
            e) # Get the email
                    email=${OPTARG}
            ;;
            d) # Get the domain name
                    domainname=${OPTARG}
            ;;
            h) # Execute the help function
                    help;
            ;;
            \?) # Unrecognized option - show help
                    echo "Invalid option."
                    help;
            ;;
    esac
done

# This tells getopts to move on to the next argument.
shift $((OPTIND-1))
# End getopts code

## Connect to Cluster
echo "Connecting to AKS Cluster"
az aks get-credentials --resource-group "$rg" --name "$cluster" --overwrite-existing

## Wait for the Cluster to Come online.
sleep 2m
echo "Waiting For the Cluster to Fully Wake Up. This is so that Helm can install properly and not error. It's a 2-minute wait, go make a tea or coffee."

## Add the Repos
helm repo add --force-update ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm repo add --force-update cert-manager https://charts.jetstack.io
helm repo update
helm repo add --force-update argo https://argoproj.github.io/argo-helm
helm repo update

## Install nginx-ingress
helm install nginx-ingress ingress-nginx/ingress-nginx --force \
--namespace ingress-nginx --create-namespace \
--set controller.replicaCount=2 \
--set controller.admissionWebhooks.patch.nodeSelector."kubernetes\.io/os"=linux \
--set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz \
--set controller.service.externalTrafficPolicy=Local \
--set controller.nodeSelector."kubernetes\.io/os"=linux \
--set defaultBackend.nodeSelector."kubernetes\.io/os"=linux 

## Install Cert Manager
helm install cert-manager cert-manager/cert-manager --force \
--namespace cert-manager --create-namespace \
--version v1.14.5 \
--set nodeSelector."kubernetes\.io/os"=linux \
--set installCRDs=true \
--set 'extraArgs={--dns01-recursive-nameservers=1.1.1.1:53}'

## Install ArgoCD 
helm install argocd argo/argo-cd --force \
--namespace argocd --create-namespace \
--version "6.9.2" \
--set installCRDs=true \
-f argocd-values.yaml \
-f deployenvcongifmap.yaml

cat << EOF | kubectl - 
port-forward svc/argocd-server 8080:443 
EOF

git clone https://github.com/cloudflare/origin-ca-issuer.git

cd origin-ca-issuer

kubectl apply -f deploy/crds

kubectl apply -f deploy/rbac

kubectl apply -f deploy/manifests

kubectl create secret generic jasons-api-key \
    -n cert-manager \
    --from-literal api-token='API-TOKEN'

sleep 60

cat  << EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: lets-encrypt-jasons-cert
  namespace: cert-manager
spec:
  acme:
    email: "$email"
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: lets-encrypt-jasons-cert
    solvers:
    - dns01:
        cloudflare:
          email: "$email"
          apiTokenSecretRef:
            name: jasons-api-key
            key: api-token
      selector:
        dnsZones:
        - 'companydomain.com'
EOF

cat << EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: letsencrypt-jasons-cert
spec:
  secretName: letsencrypt-jasons-cert
  issuerRef:
    name: lets-encrypt-jasons-cert
    kind: ClusterIssuer
  commonName: 'testing-jc2.comapnydomain.com'
  dnsNames:
  - 'testing-jc2.comapnydomain.com'
EOF

cat << EOF | kubectl replace -f -
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: server
    app.kubernetes.io/name: argocd-server
    app.kubernetes.io/part-of: argocd
  name: argocd-server
  namespace: argocd
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 8080
    - name: https
      port: 443
      protocol: TCP
      targetPort: 8080
  selector:
    app.kubernetes.io/name: argocd-server
EOF

cat << EOF | kubectl apply -f - 
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server-ingress
  namespace: argocd
  annotations:
    cert-manager.io/cluster-issuer: lets-encrypt-jasons-cert
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: 'true'
    nginx.ingress.kubernetes.io/ssl-passthrough: 'true'
    nginx.ingress.kubernetes.io/backend-protocol: 'HTTPS'
spec:
  rules:
    - host: "$domainname"
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: argocd-server
                port:
                  name: https
  tls:
    - hosts:
        - "$domainname"
      secretName: argocdingress-cert
EOF

echo "Deployment Complete"

Integrating with Terraform

To automate the execution of this Bash CLI program within my Terraform script, I used the null_resource function. This allowed me to seamlessly call the Bash script and handle the Helm deployments as part of the Terraform workflow. Here’s a snippet of how I integrated the script with Terraform; I actually pass the variable values using a locals file:

resource "null_resource" "aks_configure" {
  provisioner "local-exec" {
    command = "./setupcluster.sh -r ${local.rg_name} -l ${local.location_for_bash_script} -c ${local.kubernetesclustername} -e ${local.letsencryptemail} -d ${local.fulldnsname}"
    interpreter = ["bash", "-c" ]
  }
  depends_on = [azurerm_kubernetes_cluster.new_prod_kubernetes, azurerm_resource_group.new_prod_rg]
}

The Benefits of Bash

1. Simplicity: Bash scripts are easy to write and understand. They don’t require complex setup or dependencies, making them ideal for quick automation tasks.

2. Flexibility: With Bash, I could easily integrate various Helm commands and Kubernetes configurations and even clone Git repositories, it also uses the latest version of the CLI's.

3. Reliability: The script provided a reliable way to deploy the Helm charts, overcoming the limitations I faced with Go.

Conclusion

While encountering challenges is inevitable in software development, finding alternative solutions is crucial. In this case, Bash proved to be a reliable and efficient tool, allowing me to complete the Helm deployments on time. This experience reinforced the importance of having multiple tools in your arsenal and being adaptable in the face of obstacles.

If you ever find yourself in a similar situation, don’t underestimate yourself and the knowledge you have.

As the day unfolded, it became clear that the focus was on the future—namely, IoT and cloud devices. AWS had set up a series of quests that required us to harness the full potential of their cloud services. But what truly caught me off guard was the whimsical theme woven into the technical challenges: unicorns.

We were divided into teams and tasked with completing various quests, all centred around these mythical creatures. It was a clever way to blend creativity with technology, making the learning process not just informative but thoroughly enjoyable. The competitive spirit was high, spurred on by an intimidating leaderboard that kept us all on our toes.

Throughout the day, I had the pleasure of meeting many talented individuals, but one person who stood out was Bret. Our team quickly bonded over our shared goals and mutual enthusiasm for the tasks at hand. We navigated AWS Cloud's vast landscape, solving problems and unlocking new levels of the competition.

By the end of the event, our team proudly secured third place. While we didn't take home the grand prize, the experience was rewarding in itself. Plus, my laptop now sports two cool new stickers, a tangible reminder of an unforgettable day.

This GameDay event was more than just a competition; it was a testament to the power of collaboration and the endless possibilities within the AWS ecosystem. Whether you're an IoT enthusiast, a cloud computing aficionado, or just someone who loves a good challenge, events like these are a fantastic way to expand your horizons and make new connections.

As I look at my newly adorned laptop, I'm reminded of the camaraderie, the learning, and, yes, the unicorns that made the day truly magical. Here's to more cloud conquests and the friendships forged along the way.

Till Next Time
Happy Coding
The Cloud Dude

Introduction

In this series, we're diving deep into the world of LLMs, like ChatGPT, to understand how they work and how to build one from scratch. This post covers Chapter 2 of the book, which focuses on building a dataset for training our LLM.

Chapter 2: Building a Dataset

Understanding the Dataset

The core idea behind an LLM is to take user input, process it, and generate meaningful output. For instance, if you ask the model to write a book about different breeds of cats, it needs to understand the request, fetch relevant information, and then generate the content. This process involves training the model on a dataset that acts as its knowledge base.

Below is an image on how this might look from a high level.

Training the Dataset

Training a dataset involves encoding a large amount of text into tokens that the model can understand and use. Essentially, you have to give it a vocabulary to then be able to go out into the world and understand and process the information. I mentioned in the YouTube video that I believe you should give it the English Dictionary because then it would have all the words and letters and would be able to process the information.

In the book, Sebastian Raschka uses "The Verdict" as an example dataset. The goal is to encode this book into tokens, which the model can then use to process other tasks.

Practical Implementation

Using Jupyter Notebooks

The book provides code snippets in Jupyter Notebooks, which are great for running pieces of code interactively. However, as a developer, you might prefer to build a complete program. This involves creating classes and importing them into a main file to run the entire process as a cohesive unit.

Your folder structure would look like so:

┣ 📜main.py
┣ 📜simple_tokenizer.py

You would have the class in simple_tokenizer.py and that would look like this:

import re
class SimpleTokenizerV1:
    def __init__(self, vocab):
        self.str_to_int = vocab
        self.int_to_str = {i:s for s,i in vocab.items()}

    def encode(self, text):
        preprocessed = re.split(r'([,.?_!"()\']|--|\s)', text)
        preprocessed = [
            item.strip() for item in preprocessed if item.strip()
        ]
        #ids = [self.str_to_int[s] for s in preprocessed]
        ids = []
        for s in preprocessed:
            if s in self.str_to_int:
                ids.append(self.str_to_int[s])
            else:
                print(f"Warning: '{s}' not found in vocabulary. Using '<UNK>' token.")
                ids.append(self.str_to_int[self.unknown_token])
        return ids

    def decode(self, ids):
        text = " ".join([self.int_to_str[i] for i in ids]) 

        text = re.sub(r'\s+([,.?!"()\'])', r'\1', text)
        return text

Then, as we are working in Python, you would import that class into your m̀ain.py file. Like so

import simple_tokenizer

To now use that class and process the vocabulary so that it can be put into tokens, you would write something similar to the following:

import urllib.request
import re
import simple_tokenizer

## Download the Text File
url = ("https://raw.githubusercontent.com/rasbt/"
       "LLMs-from-scratch/main/ch02/01_main-chapter-code/"
       "the-verdict.txt")
file_path = "the-verdict.txt"
urllib.request.urlretrieve(url, file_path)

## Read the Text File
with open("the-verdict.txt", "r", encoding="utf-8") as f:
    raw_text = f.read()

# Define The Vocab
# For the Vocab to work on a book you need to pass it a lot of vocabulary. Essentially to get this working I had to pass it the book
# to process the book. Is this not counter interuitive?
# Once it had all the vocab then it can run and process the book into tokens. 
#vocab = {
#    'The': 1, 'verdict': 2, 'is': 3, 'in': 4, '.': 5,
#    'It': 6, 'was': 7, 'a': 8, 'great': 9, 'success': 10, 's':11
#}

## Now Process the Book and convert it to Tokens.
preprocessed = re.split(r'([,.?_!"()\']|--|\s)', raw_text)
preprocessed = [item.strip() for item in preprocessed if item.strip()]
print(len(preprocessed))

## Sort words Alphabetically This I believe removes white space and vocab punctuation as well.
all_words = sorted(set(preprocessed))
vocab_size = len(all_words)
print(vocab_size)
all_tokens = sorted(list(set(preprocessed)))
all_tokens.extend(["<|endoftext|>", "<|unk|>"])

## Now print the first 51 TokenIDs. 
vocab = {token:integer for integer,token in enumerate(all_words)}
vocab = {token:integer for integer,token in enumerate(all_tokens)}
for i, item in enumerate(vocab.items()):
    print(item)
    if i >= 50:
        break

## Initialise the tokenizer
tokenizetext = simple_tokenizer.SimpleTokenizerV1(vocab)

## Encode the Text
converttoids = tokenizetext.encode(raw_text)
print("Encoded Text:", converttoids)

## Decode the Text
decodeids = tokenizetext.decode(converttoids)
print("Decoded Text", decodeids)

This would be simple enough to build a very basic vocabulary to process the book.

Handling Errors

One common issue when working with datasets is handling unknown tokens. For example, if the model encounters a word that isn't in its vocabulary, it should handle this gracefully rather than throwing an error. In Python, this can be managed by checking for the presence of each token in the vocabulary and using a placeholder for unknown tokens.

You could do this like so:

class SimpleTokenizerBetterErrorV1:
    def __init__(self, vocab):
        self.str_to_int = vocab
        self.int_to_str = {i: s for s, i in vocab.items()}
        self.unknown_token = '<UNK>'
        self.str_to_int[self.unknown_token] = len(self.str_to_int) + 1
        self.int_to_str[len(self.int_to_str) + 1] = self.unknown_token

    def encode(self, text):
        preprocessed = re.split(r'([,.?_!"()\']|--|\s)', text)
        preprocessed = [item.strip() for item in preprocessed if item.strip()]
        ids = []
        for s in preprocessed:
            if s in self.str_to_int:
                ids.append(self.str_to_int[s])
            else:
                print(f"Warning: '{s}' not found in vocabulary. Using '<UNK>' token.")
                ids.append(self.str_to_int[self.unknown_token])
        return ids

    def decode(self, ids):
        text = " ".join([self.int_to_str[i] for i in ids])
        text = re.sub(r'\s+([,.?!"()\'])', r'\1', text)
        return text

Instead of Python just spitting out to me lines of code in my error that really do not make much sense:

Traceback (most recent call last):
  File "/home/jason/Dev_Ops/LLMS/MyLLMVersion/Ch2/main.py", line 52, in <module>
    converttoids = tokenizetext.encode(raw_text)
  File "/home/jason/Dev_Ops/LLMS/MyLLMVersion/Ch2/simple_tokenizer.py", line 12, in encode
    ids = [self.str_to_int[s] for s in preprocessed]
  File "/home/jason/Dev_Ops/LLMS/MyLLMVersion/Ch2/simple_tokenizer.py", line 12, in <listcomp>
    ids = [self.str_to_int[s] for s in preprocessed]
KeyError: 'I'

With the above code, I am able to make the error make better sense.

Warning: 'I' not found in vocabulary. Using '<UNK>' token.
Traceback (most recent call last):
  File "/home/jason/Dev_Ops/LLMS/MyLLMVersion/Ch2/main.py", line 52, in <module>
    converttoids = tokenizetext.encode(raw_text)
  File "/home/jason/Dev_Ops/LLMS/MyLLMVersion/Ch2/simple_tokenizer.py", line 19, in encode
    ids.append(self.str_to_int[self.unknown_token])
AttributeError: 'SimpleTokenizerV1' object has no attribute 'unknown_token'

As I touched on in the video, this really is a letdown of the Python language, and in Go, you can't do this. Go error handles well, and it also gets you, as a developer, to think about what type of error is going to come back to the user. Making you essentially a better developer.

Conclusion

I would highly recommend buying this book; you can get it from here: https://www.manning.com/books/build-a-large-language-model-from-scratch

To begin building an LLM, you need to first build a tokenizer that can take in human language and convert it into numbers or, essentially, a database. Then, you can ask it to process data and return human-readable results.

I hope that was insightful, and I look forward to writing the next post about this project.

Happy coding,
Cloud Dude

Despite my best efforts, a critical bug in a dependency and several deployment hurdles made the journey incredibly frustrating. Here's an account of my experience and the code I used, highlighting the pain points and lessons learned.

The Goal

The objective was simple: automate the deployment of a Helm chart to a Kubernetes cluster. This included setting up the Kubernetes Cert-Manager objects and leveraging the github.com/mittwald/go-helm-client package to manage Helm operations. Theoretically, it was straightforward. In practice, it proved to be anything but.

The Code

Here is the code I used for this project. Note that sensitive information has been omitted for security reasons.

package main

import (
    "bytes"
    "context"
    "fmt"
    "log"
    "os"
    "path/filepath"
    "time"

    "github.com/Azure/azure-sdk-for-go/sdk/azidentity"
    "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v5"
    certmanagerv1 "github.com/cert-manager/cert-manager"
    helmclient "github.com/mittwald/go-helm-client"
    "github.com/mittwald/go-helm-client/values"
    "helm.sh/helm/v3/pkg/repo"
    core "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
    _ "k8s.io/client-go/plugin/pkg/client/auth"
    clientcmd "k8s.io/client-go/tools/clientcmd"

    "github.com/Azure/azure-sdk-for-go/sdk/azidentity"
    "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v5"
)

func main() {
    var outputBuffer bytes.Buffer
    ctx := context.Background()
    mycontext, _ := context.WithTimeout(ctx, 80000*time.Second)

    certmanageremail := "your-email@example.com"
    subid := "your-subscription-id"
    rg := "your-resource-group"
    cluster := "your-cluster-name"

    fmt.Println("Connecting to Cluster Via Azure Call\n")
    myaksconnect := connectToAks(ctx, subid, rg, cluster)
    mypublicip := "0.0.0.0"
    fmt.Printf("Moved %v to KubeConfig File", &myaksconnect.Name)
    fmt.Println("Connecting to Cluster using KubeConfig File\n")
    kubeClient := connectToK8s()

    fmt.Println("Building Helm Client\n")
    opt := &helmclient.Options{
        Namespace:        "default",
        RepositoryCache:  "/tmp/.helmcache",
        RepositoryConfig: "/tmp/.helmrepo",
        Debug:            false,
        Linting:          true,
        DebugLog:         func(format string, v ...interface{}) {},
        Output:           &outputBuffer,
    }

    myHelmClient, err := helmclient.New(opt)
    if err != nil {
        panic(err)
    }

    fmt.Println("Deploying nginx-ingress\n")
    nginxchartRepo := repo.Entry{
        Name: "nginx-ingress",
        URL:  "https://kubernetes.github.io/ingress-nginx",
    }

    if err := myHelmClient.AddOrUpdateChartRepo(nginxchartRepo); err != nil {
        log.Fatal(err)
    } else {
        fmt.Printf("Added Chart Repo %s\n", nginxchartRepo.Name)
    }

    if err := myHelmClient.UpdateChartRepos(); err != nil {
        log.Fatal(err)
    } else {
        fmt.Printf("Updating Chart Repo\n")
    }

    nginxchartSpec := helmclient.ChartSpec{
        ReleaseName:     "nginx-ingress/nginx-ingress",
        ChartName:       "nginx-ingress",
        Namespace:       "nginx-ingress",
        CreateNamespace: true,
        SkipCRDs:        false,
        Wait:            true,
        ValuesOptions: values.Options{
            StringValues: []string{
                "rbac.create=false",
                "controller.service.externalTrafficPolicy=Local",
                fmt.Sprintf("controller.service.loadBalancerIP=%v", mypublicip),
                "controller.replicaCount=2",
                "controller.nodeSelector.kubernetes\\.io/os=linux",
                "defaultBackend.nodeSelector.kubernetes\\.io/os=linux",
                "controller.admissionWebhooks.patch.nodeSelector.kubernetes\\.io/os=linux",
                "controller.publishService.enabled=true",
                "controller.service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path=/healthz",
            },
        },
    }
    nginxInstalledHelmChart, err := myHelmClient.InstallChart(mycontext, &nginxchartSpec, nil)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("Status of Chart Install %v,\n", *nginxInstalledHelmChart.Info)

    fmt.Println("Deploying cert-manager\n")
    certmanagerchartRepo := repo.Entry{
        Name: "cert-manager",
        URL:  "https://charts.jetstack.io",
    }

    if err := myHelmClient.AddOrUpdateChartRepo(certmanagerchartRepo); err != nil {
        log.Fatal(err)
    } else {
        fmt.Printf("Added Chart Repo %s\n", certmanagerchartRepo.Name)
    }

    if err := myHelmClient.UpdateChartRepos(); err != nil {
        log.Fatal(err)
    } else {
        fmt.Printf("Updating Chart Repo\n")
    }

    certmanagerchartSpec := helmclient.ChartSpec{
        ReleaseName:     "cert-manager/cert-manager",
        ChartName:       "cert-manager",
        Version:         "v1.14.5",
        Namespace:       "cert-manager",
        CreateNamespace: true,
        ValuesOptions: values.Options{
            StringValues: []string{
                "extraArgs = {--dns01-recursive-nameservers=1.1.1.1:53}",
                "controller.nodeSelector.kubernetes\\.io/os=linux",
            },
        },
        SkipCRDs: false,
        Wait:     true,
    }
    certmanagermyInstalledHelmChart, err := myHelmClient.InstallChart(mycontext, &certmanagerchartSpec, nil)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("Status of Chart Install %v,\n", *certmanagermyInstalledHelmChart.Info)

    // Further deployments omitted for brevity...

    fmt.Println("Finished Deployment\n")
}

func connectToK8s() *kubernetes.Clientset {
    home, exists := os.LookupEnv("HOME")
    if !exists {
        home = "C:\\Users\\your-user"
    }

    configPath := filepath.Join(home, ".kube", "config")

    config, err := clientcmd.BuildConfigFromFlags("", configPath)
    if err != nil {
        log.Panicln("Failed to create K8s config")
    }

    clientset, err := kubernetes.NewForConfig(config)
    if err != nil {
        log.Panicln("Failed to create K8s clientset")
    }

    return clientset
}

func connectToAks(ctx context.Context, subid string, rg string, aksname string) *armcontainerservice.ManagedClustersClientGetAccessProfileResponse {
    cred, err := azidentity.NewDefaultAzureCredential(nil)
    if err != nil {
        log.Fatalf("Failed to obtain a credential: %v", err)
    }

    clientFactory, err := armcontainerservice.NewClientFactory(subid, cred, nil)
    if err != nil {
        log.Fatalf("Failed to create client: %v", err)
    }
    res, err := clientFactory.NewManagedClustersClient().GetAccessProfile(ctx, rg, aksname, "clusterUser", nil)
    if err != nil {
        log.Fatalf("Failed to finish the request: %v", err)
    }
    return &res
}

func Ptr[T any](v T) *T {
    return &v
}

The Challenges

1. Dependency Bugs: The biggest hurdle was a bug in the github.com/mittwald/go-helm-client package. Despite following the documentation and best practices, the Helm charts would not deploy correctly. I reported the issue here: mittwald/go-helm-client#209. This bug made deploying a working Helm chart impossible, derailing the project timeline.

2. Time Sensitivity: This project was highly time-sensitive. Every minute spent troubleshooting the bug was a minute closer to the deadline, adding to the stress and frustration.

3. Cert-Manager Objects: Deploying Kubernetes Cert-Manager objects using the native Kubernetes Go package was another pain point. Despite their theoretical simplicity, the objects did not deploy as expected, which was a significant setback, as Cert-Manager is critical for managing SSL/TLS certificates.

Lessons Learned

1. Dependency Management: Always vet your dependencies thoroughly. Consider contingency plans for potential issues with third-party libraries in critical projects.

2. Community and Support: Don't hesitate to reach out to the community or report issues.

3. Time Management: Factor in potential delays when working with new or less-tested tools. Buffer time can help you better manage unexpected challenges.

Conclusion

While this project was fraught with challenges, it was a valuable learning experience. I learned a ton about the internals of Kubernetes and Helm, which has improved my understanding of how to build Apps for Kubernetes. I also hope it gives you an insight into how it could be done.

Note: Please don't use this code, as it currently does not work. Once I have fixed the bug in the helm package, I will write another blog post about it.

Introduction

Large Language Models (LLMs), like ChatGPT, are powerful text generation and problem-solving tools. Building your own LLM ensures data privacy and control, allowing you to tailor the model to your specific needs and datasets. In this post, I will guide you through setting up the development environment required to follow along with the book.

Setup

1. Clone the Repository

First, clone the repository from Rashbits GitHub to follow along with the book. This repository contains all the code and resources you will need.

2. Install Miniforge

Miniforge is essential for installing the dependencies needed for our LLM. Use the following commands to install Miniforge:

curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
bash Miniforge3-$(uname)-$(uname -m).sh

This command downloads and installs Miniforge, setting up the underpinnings for your LLM.

3. Create a Virtual Environment

Using Conda, create a virtual environment for Python. This is crucial for ensuring that Python runs correctly and that dependencies do not conflict with other projects on your machine.

conda create -n llm_env python=3.8
conda activate llm_env

4. Install Required Libraries

To get the code from the book to work and to use the right libraries, you need to install JupyterLab and watermark:

conda install jupyterlab watermark

5. Run Environment Check Script

To ensure your environment matches the one used in the book, run Sebastian Raschka's setup script:

python /LLMs-from-scratch/setup/02_installing-python-libraries/python_environment_check.py

This script will tell you what is missing in your environment.

6. Install Additional Dependencies

To fix any missing dependencies and align your environment with the book, run:

pip install -r /LLMs-from-scratch/requirements.txt --upgrade

This command installs all the required Python libraries listed in the requirements.txt file, ensuring that your environment is fully set up.

A Small Overview Of The Book So Far

• Chapter 1: Introduces the theory behind LLMs and explains how they work.

• Chapter 2: Delves into the practical implementation of creating your own LLM. I am going to write a future Blog Post about this.

The ultimate goal of the book and my blog posts is for you to have your own LLM running on your machine, trained on datasets of your choice. For example, you could train it on code to create a programming assistant.

Conclusion

Stay tuned for upcoming videos in this series, which aim to guide you through building and using your own LLM. Follow along with the book for a comprehensive learning experience. Happy coding!

I would highly recommend buying this book; you can get it from here: https://www.manning.com/books/build-a-large-language-model-from-scratch

And I look forward to you following along with me as we explore how to do this together over the next few weeks or months...

As I said on my YouTube Channel (https://www.youtube.com/@TheCloudDude-24), I will eventually convert this LLM into a GoLang version just because I believe it will run better. I will also cover that as a new series after this one.

Happy coding,
Cloud Dude

One common way to secure these communications is through the use of authentication mechanisms. Today, I'll walk you through creating a custom Authentication API in Go, providing a flexible and reusable way to handle different authentication methods.

Overview

Our goal is to create a middleware-based HTTP client that can handle various types of authentication, such as username/password and API key authentication. We'll leverage Go's powerful type system and interfaces to achieve this.

The Code

Here's the code we'll be working with:

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
)

// HTTPClient is an interface that can be used to perform HTTP requests
type HTTPClient interface {
    Do(req *http.Request) (*http.Response, error)
}

// HTTPClientFunc is a function that implements HTTPClient
type HTTPClientFunc func(req *http.Request) (*http.Response, error)

// Do implements HTTPClient
func (fn HTTPClientFunc) Do(req *http.Request) (*http.Response, error) {
    return fn(req)
}

// Middleware is a function that can mutate the request before it is sent
type Middleware func(HTTPClient) HTTPClient

// UserPassAuthentication is middleware that adds username/password authentication
// to the request
func UserPassAuthentication(username, password string) Middleware {
    return func(client HTTPClient) HTTPClient {
        return HTTPClientFunc(func(req *http.Request) (*http.Response, error) {
            req.Header.Set("Authorization", "Bearer "+username+password)
            return client.Do(req)
        })
    }
}

// APIKeyAuthentication is middleware that adds APIKey-based authentication to requests
func APIKeyAuthentication(apikey string) Middleware {
    return func(client HTTPClient) HTTPClient {
        return HTTPClientFunc(func(req *http.Request) (*http.Response, error) {
            req.Header.Set("Authorization", "Bearer "+apikey)
            return client.Do(req)
        })
    }
}

type Client struct {
    client HTTPClient
}

func GCNewClient(httpClient *http.Client, middleware ...Middleware) Client {
    client := Client{client: httpClient}

    // Apply the middleware
    for _, m := range middleware {
        client.client = m(client.client)
    }

    return client
}

func (c *Client) Get(ctx context.Context, GCUrl string) ([]byte, error) {
    req, err := http.NewRequestWithContext(ctx, "GET", GCUrl, nil)
    if err != nil {
        return nil, err
    }

    resp, err := c.client.Do(req)
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()

    // Read the response body
    body, err := io.ReadAll(resp.Body)
    if err != nil {
        return nil, fmt.Errorf("failed to read http response body: %w", err)
    }

    fmt.Println(string(body))
    return body, err
}

func main() {
    // Replace with your API key
    apiKey := "your-api-key-here"

    // Replace with the API endpoint
    url := "https://your-api-endpoint.com"

    client := GCNewClient(http.DefaultClient, APIKeyAuthentication(apiKey))
    client.Get(context.TODO(), url)
}

Explanation

Interfaces and Middleware

We start by defining an HTTPClient interface and a function type HTTPClientFunc that implements this interface. This allows us to create flexible middleware that can modify requests before they are sent.

Authentication Middleware

We then define two middleware functions:

1. UserPassAuthentication: Adds a username and password to the request header.

2. APIKeyAuthentication: Adds an API key to the request header.

These middleware functions return a new HTTPClient that modifies the request headers as needed.

Client Struct and Constructor

The Client Struct holds an HTTPClient, and the GCNewClient Function applies the provided middleware to this client.

GET Request Method

The Get Method constructs an HTTP GET request and uses the client to execute it. It reads the response body and returns it.

Putting It All Together

In the main Function, we create a new client with API key authentication and use it to make a GET request to a specified endpoint. Remember to replace the placeholders with your actual API key and endpoint.

Conclusion

This approach provides a clean and modular way to handle different authentication mechanisms in Go. By using middleware, we can easily add or modify authentication methods without changing the core logic of our HTTP client.

Feel free to use and adapt this code for your projects. Happy coding!

I would like to thank Adam Talbot from the Kubernetes Go Community for helping with this project. Without his help, I would still be stuck looking at the NET/HTTP Package. You can find him here: https://uk.linkedin.com/in/mrtalbot

Please ensure you replace sensitive information, such as actual API keys, with placeholders before sharing your code publicly.

At first, Pulumi did not make much sense to me. My initial thought was they were trying to copy Hashicorp. Yet the more I used it, the more it has grown on me.

I like it because if you are a Dev and not a DevOps person, I believe you will like Pulumi perhaps more than Terraform. The main reason is that you can bring your language of choice to the platform and start making cloud Infrastructure without going off and learning a whole new language.

After two weeks of messing with Pulumi and Go, let me show you what I did.

The most impressive part I have seen from Pulumi and Go is the Auto Package. Auto PKG Link The Auto package brings a particular element of A. I to your program. It essentially lets you automate the whole process of running a Pulumi program.

If you want to run a Pulumi program, in the cmd line, you run Pulumi Up. In the picture below, you will see when Pulumi runs, it will ask if you want to create a Stack. A Stack is like a state file if you come from a Terraform background. You say yes and give your Stack a name.

Now, that's all good if you're running this locally and don't mind interacting with Pulumi.

But what about if you want to run this in a Pipeline? You can't have the pipeline prompt you to create a new stack. This is where the Auto Pkg starts to solve this for you. There may be other options for this. It would be worth checking out pulumi --config argument. Pulumi Up Docs

First, you must set your context, as many Pulumi Functions use the Context package. (I am not sure why this is; their documentation says it's necessary for Go to work, yet I've seen Go work fine without context. Yet that's what is required.)

So, you start with context.Background() then, you need to declare the following variables. Now, with the pipeline, You could put these in as ENV variables, but for the sake of this demo, we will hard-code these. (Later on, I will write more blog posts on this, and it will probably show a pipeline run in GitLab, so keep tuned in if you want to see that.) The Variables are: stackName , and the projectName. For obvious reasons, this stackName is what we will call the Stack. The projectName It is my understanding, and this could be wrong, but I believe the project is the top-level folder of what you will create; for example, this project will build an EC2 instance in AWS, so I called my Project ec2_deploy. If you run Pulumi manually, it will just go and grab the top-level folder name and call that the project.

Now, we are going to pass. context.Background(), stackName, and the ProjectName with a separate deploy func variable function called deployFunc (More on this later). To the function UpsertStackInlineSource What this function does is it takes all our variables and returns a created stack and an error to us. If the Stack is already created, the function won't error. (This may enable us to handle errors better than just running the pulumi up command and capturing a CLI error.) It will select the Stack and return the created Stack to us. This is handy if you run your pipeline over an already-created environment. So, the code will look a bit like this. (Side note: ensure you have "github.com/pulumi/pulumi/sdk/v3/go/auto" on your imports; otherwise, this won't work.)

package main

import (
    "context"
    "fmt"
    "os"

    "github.com/pulumi/pulumi/sdk/v3/go/auto"
)

func main() {

    ctx := context.Background()

    projectName := "ec2_deploy"
    // we use a simple stack name here, but recommend using auto.FullyQualifiedStackName for maximum specificity.
    stackName := "dev"
    // stackName := auto.FullyQualifiedStackName("myOrgOrUser", projectName, stackName)

    // create or select a stack matching the specified name and project.
    // this will set up a workspace with everything necessary to run our inline program (deployFunc)
    createdStack, err := auto.UpsertStackInlineSource(ctx, stackName, projectName, deployFunc)
    if err != nil {
        fmt.Printf("Failed to set up a workspace: %v\n", err)
        os.Exit(1)
    }

}

As we are working with AWS, we need to create a workspace; my only explanation for this workspace idea is the Python equivalent of a Virtual Environment. I still don't fully understand this concept, but according to various examples, we need it. We also need to install the AWS tools to deploy our EC2 instance. The code will look like this.

 workspace := createdStack.Workspace()

    fmt.Println("Installing the AWS plugin")

    // for inline source programs, we must manage plugins ourselves
    err = workspace.InstallPlugin(ctx, "aws", "v6.22.0")
    if err != nil {
        fmt.Printf("Failed to install program plugins: %v\n", err)
        os.Exit(1)
    }

    fmt.Println("Successfully installed AWS plugin")

Depending on how you are signing into AWS from Pulumi, where I am, we are using SSO profiles. Again, this can be a manual process when running through the Pulumi command line, but when operating the Auto Pkg, you can set the config to use the AWS profile to sign in and deploy the EC2 instance.

How we are going to do that is by using a function called, SetAllConfig() There is another function called. SetConfig() But that function will only let you set one item to the configuration. In our use case, we need to send over the AWS Region; for the project I'm working on, I need to send over a couple of custom values. It was also cool to send many values to see how this would work.

These values, though, are weirdly not sent to the workspace but to the Stack we made at the beginning. When I was initially coding this, I presumed, with the linear process the program is running in, that we had moved from the Stack to the workspace and that all settings I was configuring were now in the workspace, not the Stack. Yet that's incorrect, all settings and configs are set to the workspace.

To send our config to the Stack, we are going to use the map called ConfigMap in the Auto PKG, which contains a field called ConfigValue you need to set the name fields of this map in a YAML format; for example, aws:region don't do aws:region: as this won't work. (Hours of troubleshooting and pain brought me to this conclusion). An example of what I am talking about is below. Side Note: You must also pass the context variable we set earlier to this function.

cfg := auto.ConfigMap{
        "aws:region":             auto.ConfigValue{Value: "us-east-1"},
        "company:ticket":          auto.ConfigValue{Value: "CLOUD"},
        "company:hubname": auto.ConfigValue{Value: "MYHUB"},
    }

    createdStack.SetAllConfig(ctx, cfg)

Then, as we are sending it to the Stack, we also need to refresh our Stack to receive the new config. For that, we are going to use the Refresh() function again, you need to send your context variable to the function for this to work.

I can hear the cogs turning. Why don't you just set the configuration to the Stack at the beginning with the initial function? That is a great question; simply, it doesn't work; the only way you can set the config to the Stack that I could see with the documentation was this way. Again, I could be wrong, as I have just started this process with Pulumi.

    _, err = createdStack.Refresh(ctx)
    if err != nil {
        fmt.Printf("Failed to refresh stack: %v\n", err)
        os.Exit(1)
    }

The last thing we need to code for working with Stacks is the Pulumi-up part to make it all work. For that, we are going to use the Up() function. This works the same way as typing in pulumi up -y into your terminal.

For this function to work, it uses the package optup function ProgressStreams() , which uses io.writer to write the output of what Pulumi is sending to a console. This should still work in a pipeline scenario, which I will see at a later date when I build one. Pkgs Documentation for more details: Opt Upt PKG Link

Your code for this part should look like this

    // wire up our update to stream progress to stdout
    stdoutStreamer := optup.ProgressStreams(os.Stdout)

    // run the update to deploy our EC2 Cluster
    _, err = createdStack.Up(ctx, stdoutStreamer)
    if err != nil {
        fmt.Printf("Failed to update stack: %v\n\n", err)
        os.Exit(1)
    }

Your program code at this point should look like this:

package main

import (
    "context"
    "fmt"
    "os"

    "github.com/pulumi/pulumi/sdk/v3/go/auto"
    "github.com/pulumi/pulumi/sdk/v3/go/auto/optup"
)

func main() {

    ctx := context.Background()

    projectName := "ec2_deploy"
    // we use a simple stack name here, but recommend using auto.FullyQualifiedStackName for maximum specificity.
    stackName := "dev"
    // stackName := auto.FullyQualifiedStackName("myOrgOrUser", projectName, stackName)

    // create or select a stack matching the specified name and project.
    // this will set up a workspace with everything necessary to run our inline program (deployFunc)
    createdStack, err := auto.UpsertStackInlineSource(ctx, stackName, projectName, deployFunc)
    if err != nil {
        fmt.Printf("Failed to set up a workspace: %v\n", err)
        os.Exit(1)
    }

    workspace := createdStack.Workspace()

    fmt.Println("Installing the AWS plugin")

    // for inline source programs, we must manage plugins ourselves
    err = workspace.InstallPlugin(ctx, "aws", "v6.22.0")
    if err != nil {
        fmt.Printf("Failed to install program plugins: %v\n", err)
        os.Exit(1)
    }

    fmt.Println("Successfully installed AWS plugin")

    cfg := auto.ConfigMap{
        "aws:region":             auto.ConfigValue{Value: "us-east-1"},
        "company:ticket":          auto.ConfigValue{Value: "CLOUD"},
        "company:hubname": auto.ConfigValue{Value: "MYHUB"},
    }

    createdStack.SetAllConfig(ctx, cfg)

    _, err = createdStack.Refresh(ctx)
    if err != nil {
        fmt.Printf("Failed to refresh stack: %v\n", err)
        os.Exit(1)
    }

    // wire up our update to stream progress to stdout
    stdoutStreamer := optup.ProgressStreams(os.Stdout)

    // run the update to deploy our EC2 Cluster
    _, err = createdStack.Up(ctx, stdoutStreamer)
    if err != nil {
        fmt.Printf("Failed to update stack: %v\n\n", err)
        os.Exit(1)
    }

}

Yet, if you were to run this program like in my picture below, it would fail...

WHY?

It's failing because our first function UpsertStackInlineSource() to piece all this together is missing the deployFunc variable we have not yet set to deploy our EC2 instance. Now, I must say I don't like this, the reason being that the way you code a function essentially is with a variable, not the usual GO way with func(){} . The UpsertStackInlineSource() it's not very GO Idiomatic. I couldn't figure out how to pass a function to this function. I will at a later point in time, but for the purpose of this blog post, we will keep it a variable with the note to change it later so that we can keep to Go's Idiomaticness. (If that's a word.) Again, as always, I could be wrong and missing something here, but I could not get this function to work with another function during my tired hours.

// Deploy the EC2 Instance
    deployFunc := func(ctx *pulumi.Context) error {
        ubuntu, err := ec2.LookupAmi(ctx, &ec2.LookupAmiArgs{
            MostRecent: pulumi.BoolRef(true),
            Filters: []ec2.GetAmiFilter{
                {
                    Name: "name",
                    Values: []string{
                        "ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*",
                    },
                },
                {
                    Name: "virtualization-type",
                    Values: []string{
                        "hvm",
                    },
                },
            },
            Owners: []string{
                "099720109477",
            },
        }, nil)
        if err != nil {
            return err
        }
        amiinstanceid := pulumi.Sprintf(ubuntu.Id)
        _, err = ec2.NewInstance(ctx, "web", &ec2.InstanceArgs{
            Ami:          amiinstanceid,
            InstanceType: pulumi.String("t3.micro"),
            Tags: pulumi.StringMap{
                "Name": pulumi.String("HelloWorld"),
            },
        })
        if err != nil {
            return err
        }
        return nil
    }

The above code originated from the Pulumi website on how to deploy an EC2 instance. Yet there is a big error: It's on the line before _, err = ec2.NewInstance(ctx, "web", &ec2.InstanceArgs{ where you have to get the Instance ID of the AMI. There seems to be an issue with Pulumi and passing IDs around. The way I got around this was to use pulumi.Sprintf , which converted the ID into a string that was acceptable to the AMI field.

Also, please make sure you have the following library in your import statement:
"github.com/pulumi/pulumi/sdk/v3/go/pulumi" and "github.com/pulumi/pulumi-aws/sdk/v6/go/aws/ec2" otherwise, this won't work.

I won't go into the EC2 code too much now, as this Blog post is long enough. I will be, however, writing another post about deploying the EC2 instance with Go and Pulumi at a later date :).

The code for this variable has to go up the top now in main(){}, before our UpsertStackInlineSource() function call because Go runs in a sequential order, so if you put the deployFunc after the UperStackInlineSource() call, it will error and say deployFunc does not exist. Your whole program code should now look like this:

package main

import (
    "context"
    "fmt"
    "os"

    "github.com/pulumi/pulumi-aws/sdk/v6/go/aws/ec2"
    "github.com/pulumi/pulumi/sdk/v3/go/auto"
    "github.com/pulumi/pulumi/sdk/v3/go/auto/optup"
    "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {

    // Deploy the EC2 Instance
    deployFunc := func(ctx *pulumi.Context) error {
        ubuntu, err := ec2.LookupAmi(ctx, &ec2.LookupAmiArgs{
            MostRecent: pulumi.BoolRef(true),
            Filters: []ec2.GetAmiFilter{
                {
                    Name: "name",
                    Values: []string{
                        "ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*",
                    },
                },
                {
                    Name: "virtualization-type",
                    Values: []string{
                        "hvm",
                    },
                },
            },
            Owners: []string{
                "099720109477",
            },
        }, nil)
        if err != nil {
            return err
        }
        amiinstanceid := pulumi.Sprintf(ubuntu.Id)
        _, err = ec2.NewInstance(ctx, "web", &ec2.InstanceArgs{
            Ami:          amiinstanceid,
            InstanceType: pulumi.String("t3.micro"),
            Tags: pulumi.StringMap{
                "Name": pulumi.String("HelloWorld"),
            },
        })
        if err != nil {
            return err
        }
        return nil
    }

    ctx := context.Background()

    projectName := "ec2_deploy"
    // we use a simple stack name here, but recommend using auto.FullyQualifiedStackName for maximum specificity.
    stackName := "dev"

    // create or select a stack matching the specified name and project.
    // this will set up a workspace with everything necessary to run our inline program (deployFunc)
    createdStack, err := auto.UpsertStackInlineSource(ctx, stackName, projectName, deployFunc)
    if err != nil {
        fmt.Printf("Failed to set up a workspace: %v\n", err)
        os.Exit(1)
    }

    workspace := createdStack.Workspace()

    fmt.Println("Installing the AWS plugin")

    // for inline source programs, we must manage plugins ourselves
    err = workspace.InstallPlugin(ctx, "aws", "v6.22.0")
    if err != nil {
        fmt.Printf("Failed to install program plugins: %v\n", err)
        os.Exit(1)
    }

    fmt.Println("Successfully installed AWS plugin")

    cfg := auto.ConfigMap{
        "aws:region":             auto.ConfigValue{Value: "us-east-1"},
        "company:ticket":          auto.ConfigValue{Value: "CLOUD"},
        "company:hubname": auto.ConfigValue{Value: "MYHUB"},
    }

    createdStack.SetAllConfig(ctx, cfg)

    _, err = createdStack.Refresh(ctx)
    if err != nil {
        fmt.Printf("Failed to refresh stack: %v\n", err)
        os.Exit(1)
    }

    // wire up our update to stream progress to stdout
    stdoutStreamer := optup.ProgressStreams(os.Stdout)

    // run the update to deploy our EC2 Cluster
    _, err = createdStack.Up(ctx, stdoutStreamer)
    if err != nil {
        fmt.Printf("Failed to update stack: %v\n\n", err)
        os.Exit(1)
    }

}

I hope this Blog post is helpful to you and shows you how Auto PKG works with Go in Pulumi.

Till next time :)

Saying Hello to the Brave API

I first tried the curl command to see how the API worked. (Just a side note for the sake of this blog: the command is on two lines to make the curl command work: move the -H "Accept-Encoding: gzip" -H "X-Subscription-Token: API_KEY_VALUE" up a level to make the command all one line.)

curl -s --compressed "https://api.search.brave.com/res/v1/web/search?q=brave+search" -H "Accept: application/json"
    -H "Accept-Encoding: gzip" -H "X-Subscription-Token: API_KEY_VALUE"

As you can see, the Headers are not the normal restful headers when using an API. For example, most restful headers accept the following Bearer format when working with a token. -H "Authorization: Bearer <ACCESS_TOKEN>"

Using this Curl command will only get you authenticated to the API and will return the following: (This is just a snippet of what is returned; the whole JSON is very long and would make this a very long article.)

{"query":{"original":"brave search","show_strict_warning":false,"is_navigational":true,"local_decision":"drop","local_locations_idx":0,"is_news_breaking":false,"spellcheck_off":true,"country":"us","bad_results":false,"should_fallback":false,"postal_code":"","city":"","header_country":"","more_results_available":true,"state":""},"mixed":{"type":"mixed","main":[{"type":"web","index":0,"all":false},{"type":"web","index":1,"all":false},{"type":"videos","all":true},{"type":"web","index":2,"all":false},{"type":"web","index":3,"all":false},{"type":"web","index":4,"all":false},{"type":"web","index":5,"all":false},{"type":"web","index":6,"all":false},{"type":"web","index":7,"all":false},{"type":"web","index":8,"all":false},{"type":"web","index":9,"all":false},{"type":"web","index":10,"all":false},{"type":"web","index":11,"all":false},{"type":"web","index":12,"all":false},{"type":"web","index":13,"all":false},{"type":"web","index":14,"all":false},{"type":"web","index":15,"all":false},{"type":"web","index":16,"all":false},{"type":"web","index":17,"all":false},{"type":"web","index":18,"all":false},{"type":"web","index":19,"all":false}],"top":[],"side":[]},"type":"search","videos":{"type":"videos","results":[{"type":"video_result","url":"https://www.youtube.com/watch?v=gwho1EuwkRQ","title":"How To Change The Default Search Engine In The Brave ...","description":"How To Change The Default Search Engine In The Brave Web Browser | PC | *2022*This is a video tutorial on how to change the default search engine in the Brav...","age":"September 20, 2022","video":{},"meta_url":{"scheme":"https","netloc":"youtube.com","hostname":"www.youtube.com","favicon":"https://imgs.search.brave.com/Ux4Hee4evZhvjuTKwtapBycOGjGDci2Gvn2pbSzvbC0/rs:fit:32:32:1/g:ce/aHR0cDovL2Zhdmlj/b25zLnNlYXJjaC5i/cmF2ZS5jb20vaWNv/bnMvOTkyZTZiMWU3/YzU3Nzc5YjExYzUy/N2VhZTIxOWNlYjM5/ZGVjN2MyZDY4Nzdh/ZDYzMTYxNmI5N2Rk/Y2Q3N2FkNy93d3cu/eW91dHViZS5jb20v","path":"› watch"},"thumbnail":{"src":"https://imgs.search.brave.com/6g23ttPyzgvCVreUZ2pcnuTnGAhNHt8yQLyVNL3mlsc/rs:fit:200:200:1/g:ce/aHR0cHM6Ly9pLnl0/aW1nLmNvbS92aS9n/d2hvMUV1d2tSUS9t/YXhyZXNkZWZhdWx0/LmpwZz9zcXA9LW9h/eW13RW1DSUFLRU5B/RjhxdUtxUU1hOEFF/Qi1BSC1DWUFDMEFX/S0Fnd0lBQkFCR0Vn/Z0V5aF9NQTg9JmFt/cDtycz1BT240Q0xB/dzdIUkZsVTd1OUZT/YUR2WERZSE1TNnNj/U0p3"}},{"type":"video_result","url":"https://www.youtube.com/watch?v=ux4VacF3hxc","title":"Brave Browser And Search Engine Keep Getting Better - YouTube",

Lets Get Searching With Go

As you can see, we have yet to get searching. For that, we turn to Go.

In Go, you can use specific packages to interact with APIs. They are: "encoding/json", "net/http", and "net/url." There are a few others we are going to use, but they are just there to make this all work. I won't go into detail about them here; maybe in a future blog post.

Essentially, what we are dealing with here is JSON being passed to us, and we have to do something with that JSON. Now you have structured JSON and Unstructured JSON. To learn more about the difference and more about JSON, there is a great article on the Go Blog here: https://go.dev/blog/json

As we know the fields we are getting from the CURL command, we know to use a structured JSON approach. To do that, we set structs to match the fields of the Returned JSON. We do that like this:

type Searchrequest struct {
    base       *url.URL
    token      string
    searchterm string
}

type Main struct {
    Type  string `json:"type"`
    Index int    `json:"index"`
    All   bool   `json:"all"`
}

type Query struct {
    Original             string `json:"original"`
    ShowStrictWarning    bool   `json:"show_strict_warning"`
    IsNavigational       bool   `json:"is_navigational"`
    IsNewsBreaking       bool   `json:"is_news_breaking"`
    SpellcheckOff        bool   `json:"spellcheck_off"`
    Country              string `json:"country"`
    BadResults           bool   `json:"bad_results"`
    ShouldFallback       bool   `json:"should_fallback"`
    PostalCode           string `json:"postal_code"`
    City                 string `json:"city"`
    HeaderCountry        string `json:"header_country"`
    MoreResultsAvailable bool   `json:"more_results_available"`
    State                string `json:"state"`
}

type Mixed struct {
    Type string        `json:"type"`
    Main []Main        `json:"main"`
    Top  []interface{} `json:"top"`
    Side []interface{} `json:"side"`
}

type MetaURL struct {
    Scheme   string `json:"scheme"`
    Netloc   string `json:"netloc"`
    Hostname string `json:"hostname"`
    Favicon  string `json:"favicon"`
    Path     string `json:"path"`
}

type Results struct {
    Title          string  `json:"title"`
    URL            string  `json:"url"`
    IsSourceLocal  bool    `json:"is_source_local"`
    IsSourceBoth   bool    `json:"is_source_both"`
    Description    string  `json:"description"`
    Language       string  `json:"language"`
    FamilyFriendly bool    `json:"family_friendly"`
    Type           string  `json:"type"`
    Subtype        string  `json:"subtype"`
    MetaURL        MetaURL `json:"meta_url"`
    Age            string  `json:"age,omitempty"`
}

type Web struct {
    Type           string    `json:"type"`
    Results        []Results `json:"results"`
    FamilyFriendly bool      `json:"family_friendly"`
}

type Response struct {
    Query Query  `json:"query"`
    Mixed Mixed  `json:"mixed"`
    Type  string `json:"type"`
    Web   Web    `json:"web"`
}

This gets us ready to receive and send data and do something with it. We will touch on that in a minute first to tell Go to go (no pun intended) and interact with the API. We have to do what I like to call building the request.

Let's Build our Request

To build the request, we use the "net/http" package. We essentially tell the package this is going to be a GET request with a token, and in the "net/http" package, there is a function called http.NewRequest this basically controls our request and allows us to pass certain parameters to the function. We use http.MethodGet to be able to pass in a token and our search query.

The following code builds our request:

req, err := http.NewRequest(http.MethodGet, encodedurl, nil)
    if err != nil {
        log.Fatal(err)
    }
    req.Header = http.Header{
        "Accept":               {"application/json"},
        "X-Subscription-Token": {searchrequest.token},
    }

As you can see, our req variable receives the encoded URL (we will touch on this after and nil for error handling). We then pass the variable into a struct and add the following two headers "Accept:" which is how we wish to receive our data in this instance json.

The other header "X-Subscription-Token" is the token we are passing in. We are getting this token from a flag when we run the program as it is not good practice to put API keys into your code; we pass the value from this flag into a struct field called searchrequest.token

Passing in Flags

This piece of code is at the beginning of our program and looks like this:

    // Ask For Token and Search Term
    searchrequest := Searchrequest{
        token:      "",
        searchterm: "",
    }

    flag.StringVar(&searchrequest.token, "token", "", "token")
    flag.StringVar(&searchrequest.searchterm, "searchterm", "", "searchterm")
    flag.Parse()

This asks our user for the token and the search term. Input is as follows:

go run getwebpage.go -token "VALUE" -searchterm "VALUE"

Working with Search Engines Search Terms

    encodedsearchterm := url.PathEscape(searchrequest.searchterm)

    encodedurl := fmt.Sprintf("https://api.search.brave.com/res/v1/web/search")
    queryparam := url.Values{}
    queryparam.Set("q", encodedsearchterm)
    if len(searchrequest.searchterm) > 0 {
        encodedurl += "?" + queryparam.Encode()
    }

The reason why you see in the code the following variable encodedurl is not because of security but because of a simple problem when working with search engines. When you enter a search term into the search engine, you put your term like this ** Show Me Some Cats**. As you can see, you put your text in with spaces.

Now, the search engine automatically encodes your search term to something like this: show%20me%20some%20cats.This tells the rest of the engine this is a whole string with spaces.

Using the "net/url" package, you can encode the term first by passing it to the url struct called url.Values{} (this is in the net/url package). Then you get the search query and pass it to url.Values{} but not before you have encoded it with the Encode function. Now, for this to work, you also have to attach it to the "q" header, as this is what the Brave API only accepts to be able to use your term to search. The code that attaches our encoded query to the header of "q" is queryparam.Set("q", encodedsearchterm)

Finishing our Request

Now we have the API token and the query (our search term), we build a client to handle our request and then pass the req variable to that client.

// Send Request
    client := &http.Client{}
    repsonse, err := client.Do(req)
    if err != nil {
        log.Fatal(err)
    }
    defer repsonse.Body.Close()

Receiving The Request

Essentially, by this point, all I wanted to do was receive my data and write to a file in the format of JSON.

To do that, I used the "json/unmarshall" package, and in this piece of code, I received the JSON and unmarshal it: json.Unmarshal([]byte(body), &res)

Then, this code will indent my JSON so that It is semi-readable, and it will write it to a file in the same directory as my program and call it reply.json.

content := fmt.Sprintf("Query: %+v\n, Results: %+v\n", res.Query, res.Web.Results)
    bytes, _ := json.MarshalIndent(content, "", "")
    contentjsonerr := os.WriteFile("reply.json", bytes, 0644)
    if contentjsonerr != nil {
        log.Fatal(contentjsonerr)
    }

You can do a ton with that data, but this Blog post is already long enough, so I will stop here. I think I might write another post soon about what we can do with our cat's data.

If you want to write this program or run it yourself, you can find the code here: https://github.com/jasric89/headfirstgo/tree/master/Chapter13/HTTPProg

I wish to thank the Slack Gopher community (Join here: gophers.slack.com), particularly a fellow Gopher called Peter Hellberg, who helped me understand how APIs work. Without him, this blog post would not exist.

Happy Coding and The Cloud Dude will see you on the next Post.

I have recently just finished reading Head First Go by Jay McGavren. When I opened the book, I thought this was a very different way of learning to program from a book.

It's a bit like a school workbook; you have exercises where you need to use a pen or pencil for, and then there are also times when you need to build what they are writing about. My Github Repo for this book is here: https://github.com/jasric89/headfirstgo if you wish to take a look.

I have read quite a few books on Go, and honestly, I get halfway through the book and give up. The reason for this was I had spent the last two years trying to grasp the language, and no book for me personally could do that until I got to this book.

The way Jay McGavren writes is like someone guiding you through the language. Each chapter creates an interesting problem that naturally, your brain kicks in and goes ohh, I wonder how this problem is solved. It's sort of like a Sherlock Holmes episode of who committed the crime. Or an NCIS series, depending on your taste. (If you haven't already figured it out, I like crime series.)

Then there are the exercises where you have to get a pen out and fill in the blanks, which, in an interesting way, really starts cementing what you are learning. For me, when this first started happening. I had to take a moment with a smile and realised oh wait, this is actually going in.

As I said, there are tons of books I have read on Go where you read the chapter, then the Author leaves you an exercise to complete, which really most of that exercise has no real bearing on what you were reading, then you spend most of your time on Stack Overflow trying to figure out how to complete the exercise. Then you get disheartened about why you can't do it; the book gets tossed to the side and back to the drawing board. Not with this book.

The only major drawback with this book is that it was written before modules came out, so essentially, it uses GoPath for a lot of the book. I did head to their website, https://headfirstgo.com/, when I first noticed this problem, but I could not see any notice about the issue or any other versions of the chapters released with the new Go Pkg framework.

At first, this was frustrating because I could not get my head around how Packages work in Go. Yet I was already so impressed with the book that I spent two to three hours more learning how to write packages in Go. I have also written a blog post on it here: https://theclouddude.co.uk/mastering-writing-packages-with-go-when-working-with-the-azure-sdk

This may save you the hours of figuring out how packages work if you give that post a read and then apply it to the parts of the book where GoPath is being spoken about. It could be enough, though, to put the person off buying this book because, essentially, that part is so out of date.

There is also another area where things are out of date, and that is with the ioutil package that's been depreciated in Go. It's now the io or os package. Information about this is given here: https://pkg.go.dev/io/ioutil

Other than those things, this book, in my opinion, is very good. It teaches you the Language in a way you remember; you are not lost and disheartened when something does not work, which I think when starting out with a Language is one of the most important aspects.

I would say this book is very good for Beginners to Go; if you have come from another Language, you may start skimming through the book, which in turn would actually hinder you from finishing it properly and getting the programs working. I guess I am saying that this book is for you if you have basic or no programming knowledge. If you are a seasoned Developer in another language, perhaps choose another book. I think this one is very good for seasoned developers. https://www.gopl.io/

Overall, I enjoyed reading this book; it was easy reading, did not send me to sleep, made me confident in the Go Language, and learned a lot. I highly recommend this book if you can see past its outdated issues.

One thing you need to know well when working with the Go Azure SDK is how packages work with the removal of GOPATH.

This has taken me some time to get my head around.

Since Go 1.11 there was a fundamental change with the Go Compiler. GOPATH was gone; no more building your Go Projects in your machine's src/bin folder. Now, you can build your Go projects anywhere. Yay, I hear many of you say. Well, at first, when learning Go, I thought that made sense until I started messing about with making and importing packages.

The way making packages work is you must run go mod init module_name first. It's not such a big deal and easy enough to master.

Until you get to the point of wanting to import that package.

Take this example, for instance:

┣ 📂pkgwork
┃ ┗ 📂mypkg
┃   ┗ 📜mypkg.go
┣ 📜go.mod
┗ 📜packagework.go

Here, I have made a package called packagework, which just sits outside the folder pkgwork.

I tried to import the package like this in my code:

package main

import (
    "fmt"
    mypkg "pkgwork/mypkg"
)

func main() {
    var value mypkg.MyInterface
    value = mypkg.MyType(5)
    value.MethodWithoutParameters()
    value.MethodWithParamaters(127.3)
    fmt.Println(value.MethodWithReturnValue())

}

In my head, that makes sense because I'm thinking Well, the root of my package is pkgwork/mypkg

Yet, when I run go run packagework.go I get the following error:

package pkgwork/mypkg is not in GOROOT (/usr/local/go/src/pkgwork/mypkg)

This means the GO compiler went to look for my package within the module and expected my module name, pkgwork, to be in the pkgwork folder. Yet my module_name in this instance, is not called pkgwork it's called packagework.

Once Go realised that the module was not in this destination, what it did was It went outside the pkgwork folder. Therefore, its next course of action is to go and check the main Go libraries in the GOROOT folder. When all places are searched and not found, it comes back with the error we just saw.

Essentially, my understanding of importing packages was wrong, and I have seen many others think the same way. To import your package, it's not package_path/package_name it's module_name/package_name

To fix this and to make this easier to understand in my own head. I put my main go file called packagework.go in a folder called pkgwork, so my folder structure looks like this:

┗ 📂pkgwork
  ┣ 📂mypkg
  ┃ ┗ 📜mypkg.go
  ┣ 📜go.mod
  ┗ 📜packagework.go

I re-modulated my project with the module_name of packagework and then imported that into my packagework.go file like this:

mypkg "packagework/mypkg"

So essentially, my main Go file looks like this:

package main

import (
    "fmt"
    mypkg "packagework/mypkg"
)

func main() {
    var value mypkg.MyInterface
    value = mypkg.MyType(5)
    value.MethodWithoutParameters()
    value.MethodWithParamaters(127.3)
    fmt.Println(value.MethodWithReturnValue())

}

This then allowed Go to find the package I was referring to.

One other thing I have picked up whilst learning Go is labelling my packages when importing them; if you noticed, I import like this: mypkg "packagework/mypkg" which is label module_name/pkgname

I find it is then much easier to call to my package when coding in main; it's also easier to read. Just a little hint.

Now, you have come down this far, and I have not mentioned the Azure SDK once, so rightly you are thinking, why is this important to the Azure SDK?

Well, what I have found when working with the Azure SDK is everything is split up into packages, which are essentially modules with pkg names. Let's look at the main one you will find yourself working with a lot:

The azidentity package https://tinyurl.com/azidentity. This package is used to authenticate into Azure. To work with it, you have to import it into your project like this:

"github.com/Azure/azure-sdk-for-go/sdk/azidentity"

And you reference it like this: without a label, you take the end part of the full path, so in this instance, azindentity Here is a small example:

azCLI, err := azidentity.NewAzureCLICredential(nil)
    if err != nil {
        fmt.Println(err)
    } else {
        fmt.Println("Authenticated!!")
        armcompute.NewVirtualMachinesClient(subscriptionID, azCLI, nil)
    }

On another note, the Github.com URLS are no different to just calling your module_name something like packagework. Essentially, Go does not see them as URLS when you are first making your packages, so If I call my module github.com/jasric89/packagework, then Go will see that as a name. Only when it goes into GitHub does it become something to download as a package for another developer using the go get -u URL to package_name.

I hope this helps someone. It took me a while to get my head around this subject.

There is a project I was recently involved in where they wanted a VM in Azure running IIS. If you don't wish to do this with Go Lang, you can follow this Blog post by Facundo Gauna. He explains how to do this with PowerShell: https://gaunacode.com/install-iis-on-azure-vm-using-terraform

This was also a learning exercise to see how Go interacts with the Azure SDK; I've been learning and using Go for around a year, and the saying. "Go is easy to learn but hard to master." It could not be more true. So please, if you don't wish to go on this learning exercise and are thinking well, this can just be done with Powershell, then be my guest in following that link ;). Now that we have housekeeping out of the way let me explain what I did.

I made the VM using Terraform as this blog post is not about Terraform, but Go I will leave this link here to make a VM using Terraform.

https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/windows_virtual_machine.html

Our Journey begins at the NewVirtualMachineExtensionsClient function from the package armcompute (links at the bottom of the post for more information): NewVirtualMachineExtensionsClient(AzureSubscriptionID, cred, nil) This function tells the Azure API that I am about to create a new Virtual Machine Extension function and here are my Subscription ID and Credentials. This function does return an error, the way I handled the error is with the following piece of code:

if err != nil {
        log.Fatalf("failed to create client: %v", err)
    }

This is simple error handling. The Program will terminate and tell me the whole error right from the SDK. (One Tip: When working with the Azure SDK, don't write custom error messages; the SDK will forward detailed error messages for you, making researching these errors much easier.)

Here's the whole function in its entirety:

client, err := armcompute.NewVirtualMachineExtensionsClient(AzureSubscriptionID, cred, nil)
    if err != nil {
        log.Fatalf("failed to create client: %v", err)
    }

The first major hurdle I found in this journey is the following function, BeginCreateOrUpdate, actually updating the VM extension. The docs give an example of how this function is used, but when you fill the fields in with pointers, you will run into all sorts of errors with the SDK talking to Microsoft Azure.

What I found was that actually, it was not the best example of working with an extensions script; in researching for a few hours, I came across this Microsoft Article here: https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/template-tutorial-deploy-vm-extensions#:~:text=Save%20As.-,Edit%20the%20template,-Add%20a%20virtual

I then edited my calls to the struct.VirtualMachineExtensionsClient, which I believe all these fields were in the example JSON file given in the MS document. Here is an example of that JSON:

{
  "type": "Microsoft.Compute/virtualMachines/extensions",
  "apiVersion": "2021-04-01",
  "name": "[format('{0}/{1}', variables('vmName'), 'InstallWebServer')]",
  "location": "[parameters('location')]",
  "dependsOn": [
    "[format('Microsoft.Compute/virtualMachines/{0}',variables('vmName'))]"
  ],
  "properties": {
    "publisher": "Microsoft.Compute",
    "type": "CustomScriptExtension",
    "typeHandlerVersion": "1.7",
    "autoUpgradeMinorVersion": true,
    "settings": {
      "fileUris": [
        "https://raw.githubusercontent.com/Azure/azure-docs-json-samples/master/tutorial-vm-extension/installWebServer.ps1"
      ],
      "commandToExecute": "powershell.exe -ExecutionPolicy Unrestricted -File installWebServer.ps1"
    }
  }
}

As you can see from my code, I am now writing the following to the struct VirtualMachineExtensionProperties through a pointer using the armcompute package:

Properties: &armcompute.VirtualMachineExtensionProperties{
            Type:                    to.Ptr("CustomScriptExtension"),
            AutoUpgradeMinorVersion: to.Ptr(true),
            ForceUpdateTag:          to.Ptr("False"),
            InstanceView: &armcompute.VirtualMachineExtensionInstanceView{
                Name: to.Ptr("InstallWebServer"),
                Type: to.Ptr("CustomScriptExtension"),
            },
            Publisher: to.Ptr("Microsoft.Compute"),
            Settings: map[string]interface{}{
                "commandToExecute": "powershell -ExecutionPolicy Unrestricted Install-WindowsFeature -Name Web-Server -IncludeAllSubFeature -IncludeManagementTools",
            },
            SuppressFailures:   to.Ptr(true),
            TypeHandlerVersion: to.Ptr("1.7"),
        },

What I have found to be quite hard when working with Azure SDK is that there are many nested structs within structs. The struct fields themselves are based on JSON files. I do believe these are essentially just ARM templates being called. It's actually very hard to work out in the first instance how to change a property of a struct in the Azure SDK.

For example, the current function we are working on requires us first to pass in: armcompute.VirtualMachineExtension, which in itself is a struct from the package armcompute. Then because the struct we actually want to edit is VirtualMachineExtensionProperties, you have to do some crazy pointer work to edit those fields like this:

Type:                    to.Ptr("CustomScriptExtension"),
            AutoUpgradeMinorVersion: to.Ptr(true),
            ForceUpdateTag:          to.Ptr("False"),

Miki Tebeka (Who has written the book Go Brain Teasers") showed me another way to do this, which I may try another time. For now, this works.

Essentially to make a VM Extension using Go you are left with the following Code. I hope I have explained various parts of this code well enough for you to understand what is happening.

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/Azure/azure-sdk-for-go/sdk/azcore/to"
    "github.com/Azure/azure-sdk-for-go/sdk/azidentity"
    "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v4"
)

func main() {
    AzureSubscriptionID := "VALUE"
    myVM := "vm-onpremvm"
    myVMLocation := "North Europe"
    resourceGroup := "RG-JC-Sandbox"
    myVMExtension := "IIS"
    // Authenticate To Azure

    azCLI, err := azidentity.NewAzureCLICredential(nil)
    if err != nil {
        fmt.Println(err)
    } else {
        fmt.Println("Authenticated!!")
    }
    cred := azCLI
    ctx := context.Background()
    // Edit the VM Extension
    client, err := armcompute.NewVirtualMachineExtensionsClient(AzureSubscriptionID, cred, nil)
    if err != nil {
        log.Fatalf("failed to create client: %v", err)
    }
    poller, err := client.BeginCreateOrUpdate(ctx, resourceGroup, myVM, myVMExtension, armcompute.VirtualMachineExtension{
        Location: to.Ptr(myVMLocation),
        Tags: map[string]*string{
            "IISExtension": to.Ptr("IISExtension"),
        },
        Properties: &armcompute.VirtualMachineExtensionProperties{
            Type:                    to.Ptr("CustomScriptExtension"),
            AutoUpgradeMinorVersion: to.Ptr(true),
            ForceUpdateTag:          to.Ptr("False"),
            InstanceView: &armcompute.VirtualMachineExtensionInstanceView{
                Name: to.Ptr("InstallWebServer"),
                Type: to.Ptr("CustomScriptExtension"),
            },
            Publisher: to.Ptr("Microsoft.Compute"),
            Settings: map[string]interface{}{
                "commandToExecute": "powershell -ExecutionPolicy Unrestricted Install-WindowsFeature -Name Web-Server -IncludeAllSubFeature -IncludeManagementTools",
            },
            SuppressFailures:   to.Ptr(true),
            TypeHandlerVersion: to.Ptr("1.7"),
        },
    },
        nil)
    if err != nil {
        log.Fatalf("failed to finish the request: %v", err)
    }
    res, err := poller.PollUntilDone(ctx, nil)
    if err != nil {
        log.Fatalf("failed to pull the result: %v", err)
    }
    fmt.Printf("%v", res)
}

I hope this has helped someone; this was my first ever blog post about Go; I hope you have enjoyed it. I am no Go expert. If you wish to get in touch with me I can be reached here at jason@theclouddude.co.uk Id like to say thank you for reading this far. :)

I would also like to thank the Gophers at the Gopher Slack found here: gophers.slack.com for helping me hugely with this project and pointing me in the right direction.

The Resources I used were:

https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/template-tutorial-deploy-vm-extensions

https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v4#VirtualMachineExtensionsClient

https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v4#ClientFactory.NewVirtualMachineExtensionsClient:~:text=README,ClientFactory)%20NewVirtualMachineExtensionsClient%20%C2%B6