- Kubernetes Admission Webhook
- Packaging With Python Using Self-Signed Certs
- Python Versions
- My First Powershell
- Conclusion
I thought mayyyyyybee I'd have an article planned out today but hacker dev things(tm) have kept me at bay. Over the last week I've been doing open-source type work, or at least preparation for it. Don't worry it's not like I've run away form article writing. It's just that in order for me to write articles I kind of need to do the things mentioned in the articles so I know it works. In this particular article I show you just how much of a rabbit hole that can be!
Kubernetes Admission Webhook
I've had recent posts about Kubernetes which is quite an interesting piece of technology for someone who likes seeing how things work (except BGP... ugh). One day I found out about Admission Webhooks. I came across this due to trying to figure out how to enforce certain Kubernetes resource constraints. It sounded like a nifty thing to talk about since I had done a piece on WSGI application deployments to a container with pex. While doing this I learned about the Python Kuberenetes library and had some fun there:
from simple_webhook_deploy.certificate import WebhookCertificate
from kubernetes.client import CoreV1Api
from kubernetes.client import AppsV1Api
from kubernetes.client import (
V1ObjectMeta, V1Deployment,
V1DeploymentSpec, V1PodTemplateSpec,
V1PodSpec, V1Container, V1LabelSelector,
V1VolumeMount, V1SecretVolumeSource, V1Volume
)
class WebhookDeployment(object):
def __init__(self, service_name: str, namespace: str,
core_client: CoreV1Api, app_client: AppsV1Api,
certificate: WebhookCertificate):
self.service_name = service_name
self.namespace = namespace
self.core_client = core_client
self.app_client = app_client
self.certificate = certificate
def has_namespace(self) -> bool:
namespaces = self.core_client.list_namespace()
for namespace in namespaces.items:
if namespace.metadata.name == self.namespace:
return True
return False
def create(self, image_name: str, replicas: int = 2) -> None:
deployment = V1Deployment(
metadata=V1ObjectMeta(
name=f"{self.service_name}-deployment",
labels={
'app': self.service_name
}
),
spec=V1DeploymentSpec(
replicas=replicas,
template=V1PodTemplateSpec(
metadata=V1ObjectMeta(
name=f"{self.service_name}-pod",
namespace=self.namespace
),
spec=V1PodSpec(
)
),
selector=V1LabelSelector(
match_labels={
'app': self.service_name
}
)
)
)
self.app_client.create_namespaced_deployment(
namespace=self.namespace,
body=deployment
)
Yikes, so.much.nesting... I'm not even done with the pod spec yet! Honestly while this is pretty close to the YAML file in deployment, I think if you have a lot of re-usable parts that can be broken out into their own classes there's some really good usability with this. When going through all this I found that making self-signed certs to go back and forth between microservices was a bit tedious. So I began to think of a programmatic way to handle this...
Packaging With Python Using Self-Signed Certs
To start the journey I put together code in Python that would programmatically handle generating a self signed cert. The result was... tedious to say the least:
import datetime
from cryptography import x509
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.x509.oid import NameOID
from typing import Tuple
class WebhookCertificate(object):
def __init__(self, service_name: str, namespace: str = 'default') -> None:
self.service_name = service_name
self.namespace = namespace
self._key_object = None
self._cert_object = None
self.key, self.cert = self._generate_self_cert()
def _generate_self_cert(self) -> Tuple[bytes, bytes]:
self._key_object = ec.generate_private_key(ec.SECP256R1(), backend=default_backend())
pem_key = self._key_object.private_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PrivateFormat.PKCS8,
encryption_algorithm=serialization.NoEncryption()
)
subject = issuer = x509.Name([
x509.NameAttribute(NameOID.COUNTRY_NAME, "US"),
x509.NameAttribute(NameOID.STATE_OR_PROVINCE_NAME, "Virginia"),
x509.NameAttribute(NameOID.LOCALITY_NAME, "Richmond"),
x509.NameAttribute(NameOID.ORGANIZATION_NAME, "Kubernetes"),
x509.NameAttribute(NameOID.COMMON_NAME, f"${self.service_name}.${self.namespace}.svc.local"),
])
self._cert_object = x509.CertificateBuilder().subject_name(
subject
).issuer_name(
issuer
).public_key(
self._key_object.public_key()
).serial_number(
x509.random_serial_number()
).not_valid_before(
datetime.datetime.utcnow()
).not_valid_after(
datetime.datetime.utcnow() + datetime.timedelta(days=365)
).add_extension(
x509.SubjectAlternativeName([
x509.DNSName(f"${self.service_name}.${self.namespace}.svc.local")
]),
critical=False,
).sign(self._key_object, hashes.SHA256())
return (pem_key, self._cert_object.public_bytes(encoding=serialization.Encoding.PEM))
This essentially creates a proper self-signed cert using the cryptography pypi package along with sweat and tears. Most of this was just trying to figure out how to get the private key in PEM format while dealing with who knows what serialization fun. It also needs a few more x509 extensions.
Then I thought "Hey this would make a nice reusable pypi package" which meant I also thought "Hey this would make a nice tutorial on how to make a pypi package". I started to work on that thinking things would be somewhat straightforward. Unfortunately, it took a weird turn when I tried to deal with the Windows side of things. Part of it was I wanted to show how to test multiple Python versions using tox. Trying to get things consistent with multiple Python versions was weird.
Python Versions
What made this weird was the inconsistency between *NIX and Windows systems. On Windows the Python Launcher is used. This allows you to switch between different Python versions which are registered in the Windows Registry via PEP514 logic. Unfortunately, one of the popular implementations, PyPy, didn't have great support for it. On *NIX systems pyenv made this easy, while on Windows pyenv-win exists but it's currently not able to pull the PyPy mirrors. I wanted a more simplistic way to integrate PyPy into Windows for easy Python Launcher integration. So I started to do something really crazy: write Powershell.
My First Powershell
While I had a small bit of Powershell experience I didn't really take it to the full extent. It was mostly done on an as-needed basis. This time though I decided to really take a stab at things. One thing I find interesting is that Powershell feels like more of the UNIX philosophy of gluing commands together than most of what I've done on Linux systems (shove things into a bash script). Even more weird is Powershell being open source. With that said I began my journey with this empty repository. That's because I'm trying to get that sweet code coverage:
Tests completed in 1.93s
Tests Passed: 15, Failed: 0, Skipped: 0 NotRun: 0
Processing code coverage result.
Covered 83.56% / 90%. 73 analyzed Commands in 5 Files.
It was actually at 97% at one point but I added some new code that I'm writing tests for. Which by the way:
BeforeAll {
$PackageRoot = "$PSScriptRoot\..\src\PyPyInstaller\"
Import-Module $PackageRoot
. "$PackageRoot\Functions\Private\Utility.ps1"
}
InModuleScope PyPyInstaller {
Describe "Get-PyPyDownload" {
BeforeAll {
$TestRootPath = "$env:temp"
Mock -CommandName Read-PyPyInstallerConfig -MockWith { return @{ RootPath = $TestRootPath } }
Mock -CommandName Get-Content -ParameterFilter { $Path -eq "$TestRootPath\versions.json" } -MockWith { return Get-Content "$PSScriptRoot\fixtures\test_versions.json" }
}
Context "Python Version Provided" {
BeforeAll {
Mock -CommandName New-Item -ParameterFilter { $ItemType -eq "Directory" -and $Path-eq "$TestRootPath\Installs\7.3.12-3.10.12" -and $Force -eq $true } -MockWith { return New-Object -TypeName System.Object -Property @{} }
Mock -CommandName Invoke-WebRequest -ParameterFilter { $Uri -eq "https://downloads.python.org/pypy/pypy3.10-v7.3.12-win64.zip" } -MockWith { return @{ Content = "Test" } }
Mock -CommandName Expand-Archive -ParameterFilter { $Path -eq "$TestRootPath\Downloads\pypy3.10-v7.3.12-win64.zip" -and $DestinationPath -eq "$TestRootPath\Installs\7.3.12-3.10.12"} -MockWith { return $null }
Mock -CommandName Out-File -ParameterFilter { $FilePath -eq "$TestRootPath\Downloads\pypy3.10-v7.3.12-win64.zip" } -MockWith { return $null }
}
It "Downloads and Extracts To Folder" {
Get-PyPyDownload "3.10.12"
Assert-MockCalled Get-Content -Exactly -Times 1
Assert-MockCalled New-Item -Exactly -Times 1
Assert-MockCalled Invoke-WebRequest -Exactly -Times 1
Assert-MockCalled Expand-Archive -Exactly -Times 1
}
}
}
}
This is the fun that is Powershell mocking, or at least the form that someone completely new to Powershell would write. One thing I must say is that Pester's mocking capabilities are quite easy to work with. It reminded me of pytest mock patching, only well, simpler. Most of the time spent fighting tests was due to the nuances of pipeline flow. For example, you can see that your mocked method was called X many times through something like this:
Assert-MockCalled Get-Content -Exactly -Times 1
This Get-Content loads up a JSON file that's obtained from the PyPy mirror version list in many areas of code. Since I should only need to load the file once, I ensure that's exactly what happens. It's pretty intuitive. However this lead to an interesting issue here:
Invoke-WebRequest -Uri "https://buildbot.pypy.org/mirror/versions.json" | Select-Object -ExpandProperty Content | Out-File -FilePath "$($PyPyInstallerConfig.RootPath)\versions.json"
Seeing this I had to mock Out-File so it didn't actually to write anything. My first thought was "Make sure it gets called once". What I found though was that it was called a lot more than that, because the downloaded content was getting written in a buffered manner. In essence it was being called at least 250 times. This sort of makes sense when you think about it, but it certainly threw me off.
For now I'm mostly working on getting the final details sorted out on the Powershell. Also re-working my version comparison logic due to well... whatever this is (time to add another test!):
"3.9", "3.10" | Find-PyPyLatest
Name Value
---- -----
7.3.11 @{pypy_version=7.3.11; python_version=3.9.16; stable=True; latest_pypy=True; date=2022-12-29; files=System.Object[]}
7.3.12 @{pypy_version=7.3.12; python_version=3.9.17; stable=True; latest_pypy=True; date=2023-06-16; files=System.Object[]}
7.3.9 @{pypy_version=7.3.9; python_version=3.9.12; stable=True; latest_pypy=True; date=2022-03-30; files=System.Object[]}
7.3.8 @{pypy_version=7.3.8; python_version=3.9.10; stable=True; latest_pypy=True; date=2022-02-19; files=System.Object[]}
7.3.10 @{pypy_version=7.3.10; python_version=3.9.15; stable=True; latest_pypy=True; date=2022-12-06; files=System.Object[]}
7.3.12 @{pypy_version=7.3.12; python_version=3.10.12; stable=True; latest_pypy=True; date=2023-06-16; files=System.Object[]}
At least the download and unpack code works! Also need to get the setup parts that add to path and the Windows Registry ala PEP514 logic. This may take one or two days to fully flush out but I must say it's been quite the fun experience.
Conclusion
See, what a fine rabbit hole I've plunged into! Once I get code up and running and some form of installation up on GitHub I'll probably have another article up to announce it. Then I can finally go backwards in the article queue train (my word that's a lot of drafts).
Top comments (0)