Dr Saurabh Sawhney, Prateek Bhatia
The story begins one fine weekend, when our production Kubernetes cluster on Amazon Web Services (AWS) decided to call it quits, possibly influenced by the gorgeous weather outside. I mean, who wants to manage pods and daemonsets when spring is in the air, right?
Of course, the humans involved did not want to manage those pods either, but it had to be done. The episode lead to some soul-searching and a long overdue decision to tackle some of our software design debt, which clung to the organization in ever-expanding streamers like seaweed on a surfer’s board. The upshot of it all was, we decided we needed more robust systems for deploying and rolling back our services on K8s.
Till then, the images we were pushing to the GitHub Container Registry (ghcr) were tagged simply as staging or production. Now, finally, we implemented semver tags. Things would be simple — just deploy the image with the tag you want, and if it crashes and burns, quickly deploy the one that was doing so well before it was unceremoniously booted out.
The new plan is sound, of course, but how does one tell which containers have which tags in which cluster, and which tagged images are available to us? The answers are not far — one connects to the Kubernetes cluster and issues a somewhat convoluted command (but aren’t they all) to get hold of the current deployments. The command goes like this.
kubectl get deployment -o=jsonpath=”{range .items[*]}{‘\n’}{.metadata.name}{‘:\t’}{range .spec.template.spec.containers[*]}{.image}{‘, ‘}{end}{end}”Just make sure you are authenticated and kubernetes knows who you are, before running this.
After this, cuppa Joe in hand, one can saunter over to the individual packages and take a look at the tags, which Github provides in a chronological order.
Now, as you can see, this is not the most convenient way of doing things, even if there’s coffee in the middle. So we decided that we will use a combination of Github actions and scripts to get the work done, under the everlasting philosophy of using code wherever possible, and effort:reward ratios can take that backseat they frequently occupy anyway.
So we launched Project Tag, and here’s how it went.
There are two things we’re primarily interested in.
- For the staging and production Kubernetes clusters, what are the tags on the containers running there?
- Across all packages, what are the recently tagged images stored in the GitHub Container Registry?
The first part is tackled easily enough, but there is a bit of trouble with the second one. GitHub doesn’t seem to provide, at the time of writing this, any easy command line thingy to get all package tags from all the packages that one owns. But it does have a library called Octokit which can be roped in. Here’s what the Octokit logo looks like.
If you are still around, then this ferocious kitten has obviously failed in its task. But that means we can go ahead and take a look at the code we need.
We plan to host the code for this project in a GitHub repository. You can create a new one of your own, or use an existing one that you have access to.
Inside the repository root directory, we will create a folder called node-packs, to hold all our scripts, both javascript and python.
JAVASCRIPTS BITS
The first script we discuss attempts to tame the Octokat. We call this script packs.js
Please note that all packs references are for ghcr packages and not six-packs. Excuse the pun, please.
Right, back to the serious stuff and the javascript code we promised.
In the code that follows, username, starts_with, whitelist and blacklist are all variables that relate to the way packages are named in our organization — you may not need any of these, in fact.
// nvm use 14
const { writeFile } = require("fs/promises");
const { Octokit } = require("@octokit/rest");
const octokit = new Octokit({
auth: process.env.GH_READ_PACKAGE_TOKEN,
});
const username = "n*****";
const starts_with = "hea****nt/";
const whitelist = [
"kli***/***marr",
"hsa-****/****ber",
];
const blacklist = [
"hea***/***res",
"hea***/***end",
"hea***/***inx",
"hea***/***tel",
];
octokit.rest.packages
.listPackagesForOrganization({ package_type: "container", org: username })
.then(async ({ data }) =>
(
await Promise.all(
data
.filter(
(container) =>
(container.name.startsWith(starts_with) ||
whitelist.includes(container.name)) &&
!blacklist.includes(container.name)
)
.map(
async (container) =>
await octokit.rest.packages
.getAllPackageVersionsForPackageOwnedByOrg({
package_type: container.package_type,
org: username,
package_name: container.name,
})
.then(({ data }) => {
const tagsString = data
.filter((image) => image?.metadata?.container?.tags?.length)
.map((image) => {
const tags = image?.metadata?.container?.tags ?? [];
return tags.join(" : ");
});
return {
"Container Name": container.name,
"Available Tags": tagsString.slice(0, 5).join(", "),
};
})
.catch((err) => console.log(err))
)
)
).filter((v) => v !== null && v !== undefined)
)
.then((result) => writeFile("result.json", JSON.stringify(result, null, 2)))
.catch((err) => console.log(err));So what this does is, it gets hold of information about package tags and puts this treasure into a file called result.json.
But before we can execute this file, we need authentication, and that comes in the form of a PAT. We need to create a classic Personal Access Token (PAT), which can be done on GitHub under your developer settings. In the interest of security, we will not hardcode the PAT into the script. Instead, will save the PAT as a secret in the Github repo where we plan to run this script as part of a Github Workflow (see later in this blog). In the context of the code above, our PAT is called GH_READ_PACKAGE_TOKEN.
A note: Make sure the PAT has enough permissions and the user who is creating the PAT has access to the package in the first place.
With this PAT, you can directly run the file locally as well, but as our aim is to put this into a Github workflow, we move on.
PYTHON BITS
Next up, we need information about which cluster has which image running on it. This is achieved by running the kubectl command that we shared earlier. Now to run a kubectl command, there are options. We could run it in the GitHub runner compute space, or set it up inside a script. We chose a python script for the purpose, for no good reason at all. Here’s what it looks like. We call it cluster-packs.py
import subprocess
import json
import sys
cluster = sys.argv[1]
def run_kubectl_command():
command = ["kubectl", "get", "deployment", "-o", "json"]
try:
completed_process = subprocess.run(
command, check=True, capture_output=True, text=True)
return completed_process.stdout.strip()
except subprocess.CalledProcessError as e:
print(f"Error executing '{command}': {e.stderr}")
return None
output = run_kubectl_command()
if output:
try:
json_data = json.loads(output)
with open(f"cluster-packs-{cluster}.json", "w") as file:
json.dump(json_data, file)
except json.JSONDecodeError as e:
print(f"Error decoding JSON: {e}")
except IOError as e:
print(f"Error writing to file: {e}")The purpose of this script is to run a kubectl command and get hold of the tags which the current containers are sporting. Then it puts this intel into a file called cluster-packs-… json. The name of the file produced depends on the argument we provide at runtime. For example, if the script is called thus
python3 cluster-packs.py stagingThen the output file will be called cluster-packs-staging.json
The actual identification of the cluster and authentication to access it is a task delegated to the GitHub workflow, as we shall soon see.
At this point, if these scripts work well, we are in a position to retrieve all elements of information we need and have them available as json files. The next step is to compile them and present them nicely. We need a red bow to tie it all up.
THE RED BOW
That red bow is provided by another python script that we call output.py
Here’s a look at that code
import json
from tabulate import tabulate
def get_deployed_images_dict(cluster):
'''
Obtain a dictionary of image-name and deployed version numbers for cluster passed as argument
'''
json_file_path = f"cluster-packs-{cluster}.json"
with open(json_file_path, "r") as json_file:
cluster_pack_data = json.load(json_file)
deployed_images_dict = {}
for i in range((len(cluster_pack_data['items']))):
image = (cluster_pack_data['items'][i]['spec']
['template']['spec']['containers'][0]['image'])
if 'your-organization' in image:
key, value = image.split('your-organization/')[1].split(':')
deployed_images_dict[key] = value
return deployed_images_dict
### END OF FUNCTION ###
# Part 1 - read in info about available tags
json_file_path = "result.json"
with open(json_file_path, "r") as json_file:
available_tags_data = json.load(json_file)
# Part 2 - read in info about images deployed to staging and production
staging_deployed_images_dict = get_deployed_images_dict('staging')
production_deployed_images_dict = get_deployed_images_dict('production')
# Part 3 - Combine the info from two sources
for structure in available_tags_data:
foreign_key = structure['Container Name']
structure[f'Version Deployed\n to Staging'] = staging_deployed_images_dict.get(
foreign_key)
structure[f'Version Deployed\n to Production'] = production_deployed_images_dict.get(
foreign_key)
temp = structure['Available Tags']
del structure['Available Tags']
structure['Available Versions'] = temp
del temp
# Part 4 - Display the report
rowIDs = range(1, len(available_tags_data)+1)
table = tabulate(available_tags_data,
headers="keys",
tablefmt="fancy_grid", # heavy_grid, double_grid, fancy_grid
showindex=rowIDs,
maxcolwidths=[None, 55, 20, 20, 50])
print(table)Let’s unpack this a little.
The script will require the existence of the json files produced earlier. It will then read those files and compile the data in a format that can be used by the tabulate library to print out the final dashboard on the Github runner terminal.
A word of caution: The creation of the deployed_images_dict in the code uses logic specific to the way packages in our organization are named. If you find errors or unexpected outputs, then the function get_deployed_images_dict(cluster) should be your first port of debugging. Start with how the key:value pair is being generated.
You can read more about the tabulate library here.
Now we move on to the final step — the GitHub workflow. If you are unfamiliar with workflows, do read up a brief intro here.
THE WORKFLOW
The workflow file is a yml file that we call List GHCR Packages.yml
Like all github workflows, it needs to be placed inside the <my-github-repo>\.github\workflows directory of the Github repo where all the code is. Let’s look at the yml file itself.
name: List GHCR Packages
on:
workflow_dispatch:
jobs:
information-about-packages:
runs-on: ubuntu-latest
env:
AWS_REGION: <name-of-the-region-where-your-eks-cluster-is>
GH_READ_PACKAGE_TOKEN: ${{ secrets.GH_READ_PACKAGE_TOKEN }}
steps:
- name: Checkout Repository
uses: actions/checkout@v2
- name: Set up Node.js
uses: actions/setup-node@v2
with:
node-version: 14
- name: Install Dependencies
run: npm install @octokit/rest
- name: Configure AWS
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ env.AWS_REGION }}
- name: Install kubectl
uses: azure/setup-kubectl@v3.0
with:
version: "v1.23.6"
id: install
- name: Install tabulate
run: pip install tabulate
- name: Packages Report
run: |
echo "Generating report..."
cd node-packs
node packs.js
aws eks --region ${{ env.AWS_REGION }} update-kubeconfig --name ****mycluster****-staging
python3 cluster-packs.py staging
aws eks --region ${{ env.AWS_REGION }} update-kubeconfig --name ****mycluster****-production
python3 cluster-packs.py production
echo -e "\033[1;32mPackages report for My Company\033[0m"
echo ""
python3 output.pyThe GH_READ_PACKAGE_TOKEN is the name of the variable stored in the github repository’s secrets and variables section. We keep it as a secret and not as a variable for security reasons, of course. This is the same PAT we discussed a while ago.
Other secrets that must be setup include the AWS_ACCESS_KEY_ID and the AWS_SECRET_ACCESS_KEY, which are available through your AWS account. The actual naming of these secrets is up to you, of course, but they do need to be saved as secrets in Github as well. Make sure that the credentials supplied ensure access to the Kubernetes cluster.
For a small organization or a solo flyer, it is possible to have the person who created the cluster to provide their own programmatic-access credentials to the workflow. This is not ideal, of course.
Another way is create a new programmatic user in AWS Identity and Access Management (IAM) with no console access and then to modify the role-bindings at the level of the kubernetes cluster itself, along with changes to the access list. This decouples the creator from the accessor.
If you have trouble with configuring access to Kubernetes, then these resource may be helpful.
The rest of the yml does not require fiddling, except the last step called Packages Report. Here, you’ll need to replace the ****mycluster****-staging and ****mycluster****-production with the names of your own clusters.
That’s the code. Once you have the files modified as per your organization, you should be able to visit the Actions tab on the Github repo and trigger a run of the workflow. The sequence of steps takes very little time.
This is the kind of output that can be expected. This will be available on the GitHub Actions terminal output itself.
Again, our apologies for the redactions, but this is super-secret stuff that must never see the light of the day.
As you can see, we’re still transitioning and a few of the non-semver tags still float around, fighting a losing battle against obsolescence.
But what this lets our team do is this. Any team member with access to the Github repo can quickly determine the state of staging and production deployment with respect to tags, see what tags are available and whether staging and production are using the same image for a particular container or not. The dashboard shows 5 of the most recent images, but that number can be altered if required (in packs.js just before it sets out to catch all those errors, if you must know).
That’s it for today then. This is a first version of the dashboard and there are tons of optimizations that can be done, but they all hide very effectively behind that eternal backbencher, the effort:reward ratio. Of course, your comments and feedback are always welcome, dear reader, and we are never averse to picking any low-hanging fruit.
Take care, save your weekends,
Saurabh and Prateek
