</th>	RAW</th>	Nuxt</th>	Zola</th></tr></thead>
JS</td>	140KB</td>	4.2MB</td>	156KB</td></tr>
HTML</td>	544KB</td>	13MB</td>	1.4MB</td></tr>
Build Time</td>	0s</td>	14s</td>	2.2s</td></tr> </tbody></table> 🧾📖 NOTE: You might want to check out v1</a>, v2</a> and v3</a> of my site.</p> Benchmarking what actually drive our containers 2024-07-01T00:00:00+00:00 The Engines that run our Kubernetes Workloads</h1> 🧾📖 PDF version</a></p> Target audience:</em> This article is a deep dive for people living the Kubernetes lifestyle, for people who know or want to know how the low level stuff works and performs. I do not explain every container or Kubernetes component as it is expected to be known.</p> Kubernetes has become THE standard for container orchestration. It is not just a software tool, it is a framework with extensive extensibility features. Entire businesses are built on top of Kubernetes and offer essential, nice to have and abstruse services. Tools like ArgoCD and Crossplane are build for Kubernetes and changed the way how we deploy software. There is even a new category of operating systems like CoreOS, Bottlerocket and Talos that are build purposely to run containers. All that happened in just 10 years. Happy late birthday K8s!</strong></p> </p> Kubernetes is dominating almost every Ted-talk and system architecture presentation, yet the actual engine that drive our workloads sometimes gets forgotten. Kubernetes is just an orchestrator, and its primary (but not only) task is to manage containers. It does not run any containers itself. That gets delegated to the container runtime interface.</p> A Little History</h2> Initially, Kubernetes was build to manage Docker Containers. Docker is the technology that made containers accessible to the mainstream. Containers were not an entirely new technology. LXC containers existed before and Linux namespaces are a thing since 2002, but Docker made it so easy to use containers. By handling software packaging, distribution and all low level network and device configuration, Docker allowed every developer to start any application in an isolated environment. Promising to finally overcoming the famous: It works on my machine</em> meme. Even the Kubernetes core team admits that Kubernetes would’ve been such a success without Docker. In an alternative universe, we are all using Mesos</a> 😉</p> At some point, a lot of different people and companies tried to solve similar problems regarding containers. Unlike some dinosaur companies with mediocre stubborn management, they agreed to build some common interfaces for their solutions so that parts of their implementations could be swapped while relying on certain standards. This led to the creation of the Open Container Initiative (OCI), Container Network Interface (CNI), Container Storage Interface (CSI) and Container Runtime Interface (CRI). Today, I want to talk about the Container Runtime Interface (CRI). With version 1.24, Kubernetes has overcome its dependence on Docker and now only relies on any CRI compliant runtime to start containers. And there are quite a few.</p> CRIs</h2> The most common CRIs are containerd, CRI-O and Mirantis Container Runtime. Of these three, containerd is the most wildly used option. The containerd project was originally developed by Docker Inc and still is the core of Docker, but its ownership was since donated to the CNCF. When a new Pod is scheduled to a node, the kubelet communicates with containerd over the container runtime interface to actually pull the image and start the container with the configuration specified in the pod definition. Containerd is now in charge to set up the Linux namespace, mount volumes or devices and apply seccomp profiles and much more for every new container. That’s a lot of tasks, and not quite according to the UNIX philosophy to do one thing and do it well</em>.</p> Digging deeper</h3> So containerd doesn’t do that all on its own. It uses something I call the container engine (others also call it OCI runtime, but I believe this would cause some confusion). By default, containerd calls `runc</code>. The runc binary actually spawns the container process with all its applied properties. It does not pull images, manages the containers file system, sets up networking or monitors the container process. Overly simplified it is just a spawn</code> syscall with a bunch of extra properties for some isolation, but it still runs on the same kernel since containers are NOT VMs.</p>` Most of the tooling for Kubernetes and containerd is written in go. Go is a great language, but some people argue that it is not the best choice for low level system functions. Discourse and different opinions are great and drive innovation. And since a bunch of smart people choose to standardize container interfaces, everyone can create their own OCI compliant implementation and use it with the existing CRIs and tools. And that is exactly what people did, and I will take a look at some of them and compare them.</p> Benchmarking Container Engines</h2> Containers are created all the time. We don’t even think about them anymore when we schedule a deployment with 42 replicas. But since it happens so frequently, we might take an occasional look at what actually happens at a low level and how it impacts our cluster’s efficiency. Do container engines impact start-up time and memory consumption? Since I didn’t find any real up-to-date comparisons, I took a look for myself and ended up comparing these implementations:</p>runc - The default OCI engine written in go that comes with containerd</li> crun - Alternative OCI implantation written in C, claiming to be -49.4% faster than runc for running /usr/bin/true</code></li> gvisor/runsc - Googles safe OCI implantation (also written in go) that aims to improve security by running many syscalls in userspace</li> youki - Quiet young alternative OCI implantation written in rust</li> </ul> Testsetup</h3> Disclaimer:</em> This is NOT a scientific benchmark. I’m quite a noop in profiling such applications, and I cannot claim full correctness for my findings. If you know how to do it better, please show me.</p> I created a VM on the Hetzner-Cloud using their dedicated offerings to avoid the effect of noisy neighbors. I redid the test on another instance to ensure I didn’t get a low-performing one. System Data:</p> 1</span>r</span>egion</span>:</span> n</span>bg1-dc3</span></span> 2</span>o</span>s</span>:</span> L</span>inux 6.1.0-21-amd64 Debian 12 (2024-05-03) x86_64 GNU/Linux</span></span> 3</span>s</span>ku</span>:</span></span> 4</span> t</span>ype</span>:</span> C</span>CX33</span></span> 5</span> c</span>pu</span>:</span> 8</span></span> 6</span> r</span>am</span>:</span> 32</span></span></code></pre> The test environment was configured with an ansible playbook and my personal Kubernetes role</a> I use to play around. My benchmark script uses the time</code> tool to measure execution time and calculates min, max and mean for 1000 container creations for the busybox image running /usr/bin/true</code>. I use nerdctl</code> to create containers rather than calling the OCI implementation directly, since it mimics a more realistic use of an CRI. Before every test, I run a small warm-up to allow the CPU to boost clock speed. I found it hard to measure CPU and memory consumption for a short-lived process, so I installed node-exporter and have it scraped by Prometheus (running on another system) every 5 seconds.</p> 1</span>containerd version 1.7.18</span></span> 2</span>crun version 1.15</span></span> 3</span>runc version 1.1.13</span></span> 4</span>runsc version release-20240617.0</span></span> 5</span>youki version 0.3.3</span></span></code></pre>Results</h3> The node’s CPU and RAM were not even remotely utilized as seen in the Grafana dashboard below. In contrast to the production environment where the kubelet can create several containers simultaneously, my benchmark runs in serial. Nevertheless, some conclusions can be drawn. The test showed indeed that crun outperforms containerd’s default OCI engine, yet the difference is not as big as claimed on the crun website. However, you have to bear in mind that it was not the same test as on the crun page. The performed benchmark includes the overhead of containerd, to represent a more realistic deployment compared to calling the OCI engine directly.</p> </p> Given the claims from crun I was expecting a bigger difference between crun and runc. A 21% difference is still impressive. Surprisingly, the quiet new youki implementation is right up there head to head with runc when it comes to pure performance. Unfortunately, youki didn’t perform as consistent and errored out quite some times. I reported the error to the authors and hope it will be fixed soon, despite its good performance an error rate of 3.6% is not acceptable in a production environment. As expected, runsc performs worst by far. However, performance is not its primary goal, it aims to improve security by implementing syscalls in userspace. The chosen test may also not favorable for runsc since running a short-lived container with true</code> does not have many syscalls, the lower context switche are still noticeable as shown in the last dashboard below. The main findings are summarized in the table below.</p> OCI</th> Mean</th> Min</th> Max</th> Errors</th> Total Time</th> Min Mem</th></tr></thead> crun</td> 0.28s</td> 0.13s</td> 0.41s</td> 0</td> 306s</td> 160kb</td></tr> runc</td> 0.34s</td> 0.16s</td> 0.44s</td> 1</td> 362s</td> 8mb</td></tr> runsc</td> 0.62s</td> 0.44s</td> 1.64s</td> 0</td> 642s</td> 32mb</td></tr> youki</td> 0.32s</td> 0.16s</td> 0.46s</td> 36</td> 354s</td> 2mb</td></tr> </tbody></table> Memory was hard to test, and I found no significant differences here. From the data node-exporter provided it seems like crun and runc use a little less memory compared to the others, but I can not prove that no other process caused the increase, even when I did my best to keep variations low.</p> </p> A claim that I was able to verify is that cruns authors say it requires far less memory to start a container than runc. I tested how low I could set the memory limit via the CRI for each engine via nerdctl run --memory 4mb ...</code>. The authors claim it requires less than 4mb, which is the lowest limit podman allows. I was able to start busybox with a memory limit of just 160kb. This is indeed impressive, even though it is not directly relevant in a production environment, since most applications within a container will need and have much higher limits.</p> Honorable findings</h3> I wanted to compare the runtime and memory performance of these OCI implementations, but when monitoring, I also noticed some spikes in IO. I did not expect that starting a small container like busybox would cause some significant IO writes, and I don’t really know why write activity is “so high” for just starting a container. Both crun and runc caused around ≈700-800 IOPS and ≈4-4.5 MB/s in writes. Slightly less was needed for youki and runsc was using just half of IOPS and MB/s in comparison with crun and runc. According to its promises, runsc also caused fewer context switches for syscalls. I’m quite interested in learning more about the “high” writes, though. Especially where the differences come from when the same snaphotter is used across all OCI engines.</p> </p> The complete raw output can be found on my GitHub Gist</a>.</p> Conclusion</h2> The performance differences between the tested OCI implementations are not as big as I expected, and even though crun performs better, I doubt that switching to it justifies the operational overhead for most companies - at least for managed Kubernetes offerings. As far as I know most managed Kubernetes offerings use containerd with runc. Being able to just use AWS managed AMIs (or Azures) is a huge time and cost saver. Unless the last percent of performance is highly important, I would just stick with runc. It would still be nice to see hyperscaler’s include crun in their AMIs (which costs 2MB of space) and give users the choice via runtimeclasses. For self-hosted and embedded Kubernetes offerings, crun might be worth a look. Even more if companies are running cutting edge technology like WASM within their clusters. Both crun and youki can be compiled with WASM support. Meaning users can run WASM applications in their clusters with the same OCI as all their other containers. Which is SICK! I’ve been using this for quite some time and it works great. The only downside is that WASM support is not enabled by default yet and you have to compile it yourself. (I’ve set up an GH Action automation just for that.) The first look of youki is promising, but without having these errors solved, I wouldn’t use it in production. When it comes to gvisor/runsc I’m a little unsure. It offers additional security, but most people run their own or trusted workloads in their clusters. If one would run random untrusted code, like someone’s leetcode or lambda code I might want to use things like kata containers or firecracker VMs which offer a higher level of isolation. If someone wants to include these in the tests, I’m very interested - I couldn’t run these since they require nested virtualization, which is not supported on Hetzners CCX VMs. This leaves gvisor in some weird middle ground, especially with Kubernetes upcoming ability to run containers in user namespaces further increasing security.</p> Innovation in the container world has settled down a bit since a lot of the technology as reached mainstream adoption and has standardized. The existing solutions have become more accessible, and the defaults cover the needs of the majority of use cases. Yet, it is still interesting how new solutions hit the market, resulting in a drop in replacements for solving specific problems.</p> Using GitLab to manage Kubernetes access 2024-06-12T00:00:00+00:00 Using GitLab to manage Kubernetes access</h1> Recently I have been tasked to share access to a Kubernetes test cluster with other teams. Easy: Add them to a group in your OIDC</a> provider and set appropriate RBAC rules. Task done - no need to write this.</p> BUT:</h2> Unfortunately I can’t have an OIDC provider for non-production clusters… because. Sighs…</em></p> TL;DR</summary> Use Kubernetes built-in OIDC authentication; if this is not possible for whatever reason, the GitLab Kubernetes Agent might be worth a look. </details> </p> Okay what other options are there for Kubernetes authentication:</p> Webhook Token Authentication</li> Authenticating Proxy</li> Static token file</li> X509 client certificates</li> </ul> Both webhook authentication and an authentication proxy add considerable overhead and ultimately also need to be plunged into a user registry that I can not use or have control of.</p> So what about static token files?</strong> In practice, I’ve never seen them outside of test setups. It requires placing a file with the username, secret and groups on every controlplane node. The effort required to manage and sync such a file, containing static credentials in plain text</strong>, is just nuts. Even more absurd, the api-server has to be restarted every time this file is changed! And since these tokens have no built-in expiration date, you have to delete an entry and restart the api-server if a credential needs to be invalidated. So this is a hard pass!</p> So X509 client certificates it is?</strong> The X509 client certificates approach does not require the api-server to be restarted and provides a built-in expiration mechanism. Client certificates are a common approach to establishing a secure communication between two entities, it is often also referred to as mutual TLS (mTLS), where both the client and the server have to identify themselves via a certificate. One-directional TLS (where just the server is ensuring its authenticity) is part of almost every website we visit. TLS is considered very secure since, it uses asymmetric encryption with private keys, with a complexity between 1024 and 4098 bits, depending on the algorithm. Kubernetes uses mTLS for almost all of its components to establish trusted and secure communication between every part of the cluster. Kubernetes clusters created with kubeadm also create a admin.conf</code> which also uses X509 client certificates to authenticate against the Kubernetes api-server. These X509 client certificates are the default way to authenticate against a Kubernetes cluster. For easier usage, Kubernetes base64 encodes the certificate and the private key and puts them inside the well-known YAML kubeconfig. The file is most likely located at ~/.kube/config</code>.</p> I have gone the X509 client certificates way and if you are interested, below is a rough draft on how you can automate user config creation. But there is a catch too Sighs…x2</em></p> Creating X509 client certificates</summary> Every user should have their own personal kubeconfig to ensure proper access controls and auditing capabilities. Unfortunately there is no built-in support from Kubernetes for creating these kube-client-configs. Nevertheless the process can easily be automated.</p> The user has to create a new key-pair with openssl which he will use for accessing the cluster later. This can be done with the following command:</p> 1</span>#</span> Bits for the key</span></span> 2</span>KEY_SIZE</span>=</span>4</span>0</span>9</span>6</span></span> 3</span>#</span> Username</span></span> 4</span>USERNAME</span>=</span>M</span>Y</span>_</span>U</span>S</span>E</span>R</span></span> 5</span>#</span> Groups</span></span> 6</span>GROUP_1</span>=</span>T</span>E</span>A</span>M</span>_</span>A</span></span> 7</span>GROUP_2</span>=</span>T</span>E</span>A</span>M</span>_</span>B</span></span> 8</span></span> 9</span>#</span> Creating key and certificate signing request</span></span> 10</span>openssl</span> req</span> -</span>new</span> -</span>newkey</span> rsa:</span>$</span>KEY_SIZE</span> -</span>nodes</span> -</span>keyout</span> k8s-client-1.key</span> -</span>batch</span> -</span>out</span> k8s-client-1.csr</span> -</span>subj</span> "</span>/CN=</span>$</span>{</span>USERNAME</span>}</span>/O=</span>$</span>{</span>GROUP_1</span>}</span>/O=</span>$</span>{</span>GROUP_1</span>}</span>/emailAddress=</span>$</span>{</span>USERNAME</span>}</span>@example.com</span>"</span></span></code></pre> The resulting certificate signing request (CSR) now has to be signed by the clusters Common Authority (CA). The signing request can either be done wither the Kubernetes certificate api by creating a CertificateSigningRequest</code> or manually by copying the clients CSR to the api-server and signing it with the clusters CA via openssl. The first option is clearly preferable as it does not require access to the node on which the api-server runs on and there is a traceable audit log. The resulting certificate and the key are then base64 encoded and inserted into the format of the kubeconfig. The resulting file will have this layout (without the certificate data redacted):</p> 1</span>a</span>piVersion</span>:</span> v</span>1</span></span> 2</span>k</span>ind</span>:</span> C</span>onfig</span></span> 3</span>c</span>urrent-context</span>:</span> d</span>ocker-desktop</span></span> 4</span>c</span>lusters</span>:</span></span> 5</span> -</span> c</span>luster</span>:</span></span> 6</span> c</span>ertificate-authority-data</span>:</span> L</span>S0...S0K</span></span> 7</span> s</span>erver</span>:</span> h</span>ttps://Kubernetes.docker.internal:6443</span></span> 8</span> n</span>ame</span>:</span> d</span>ocker-desktop</span></span> 9</span>c</span>ontexts</span>:</span></span> 10</span> -</span> c</span>ontext</span>:</span></span> 11</span> c</span>luster</span>:</span> d</span>ocker-desktop</span></span> 12</span> u</span>ser</span>:</span> d</span>ocker-desktop</span></span> 13</span> n</span>ame</span>:</span> d</span>ocker-desktop</span></span> 14</span>u</span>sers</span>:</span></span> 15</span> -</span> n</span>ame</span>:</span> d</span>ocker-desktop</span></span> 16</span> u</span>ser</span>:</span></span> 17</span> c</span>lient-certificate-data</span>:</span> L</span>S0...g==</span></span> 18</span> c</span>lient-key-data</span>:</span> L</span>S0...o=</span></span></code></pre> NOTE:</em> If you want a fully automated process for this just send me a massage!</p> </details> </p> The Problems with static credentials</h3> While mTLS is technical very secure, it still suffers from problems organizational standpoint. From a modern standpoint, certificates are still long-lived credentials. Thanks to the exemplary work done by the lets encrypt</a> project, most certificates are issued with shorter lifetimes compared to the previously common validity of 1+ years. Still, a default of three months is a heck longer than the one-hour default lifetime of an OIDC ID-Token. It is also possible to revoke certificates, but just like the creation process, it is not that easy to implement this process. Even when these security aspects are not considered, client certificates become hard to manage in larger or dynamic teams. You must have a way to revoke access from a retired team member. When a user needs to be added, the whole certification creation process has to be done again. Authorization should be done on group level basis, but you can’t add a group to a certificate after it was issued. This is not flexible and manageable in any larger organization. For this reason, dynamic authentication processes - like OIDC - are often preferred.</p> I have a great article about OAuth2 and OIDC coming up soon on my companies website - stay tuned</em></p> Alternatives</h2> So what are my alternatives? The said cluster in deployed in a very restrictive network environment. Ingress is only allowed from specific clients within a corporate network and only specific ports and traffic. Egress is only possible via a proxy. But we needed to deploy from our GitLab (which is on a different network) instance to our cluster. Other people solved this by creating GitLab runners within the Kubernetes cluster and saving the kube-client-credentials-conf as a CI secret. DONT DO THIS!</strong></p> You are pulling in random container images from the internet to your cluster and run arbitrary scripts from CI in your critical application environment. You are risking your cluster health, the confidently of secrets and are giving up a considerable amount of compute. And you also have to maintain the runners. This also violates the ISO 27001 norm since now access is not personalized anymore and auditing is really hard.</p> GitLab Agent</h3> Quite some time ago I stumbled upon the GitLab Agent. It’s a small application that creates a reverse tunnel between your control-plane and GitLab instance via websockets. This allows you to use any runner in any network location to securely communicate over the GitLab Instance websocket tunnel, via the GitLab agent to your Kubernetes cluster. GitLab automatically creates short-lived tokens (CI_JOB_TOKEN</code>) and only allows authorized users to use the tunnel in CI.</p> You can install the agent easily with:</p> 1</span>helm</span> repo</span> add</span> gitlab</span> https://charts.gitlab.io</span></span> 2</span>helm</span> repo</span> update</span></span> 3</span>helm</span> upgrade</span> -</span>-install</span> test</span> gitlab/gitlab-agent</span> \</span></span> 4</span> -</span>-namespace</span> gitlab-agent</span> \</span></span> 5</span> -</span>-create-namespace</span> \</span></span> 6</span> -</span>-set</span> image.tag=v17.0.0</span> \</span></span> 7</span> -</span>-set</span> config.token=glagent-xxx</span> \</span></span> 8</span> -</span>-set</span> config.kasAddress=wss://gitlab.example.com//-/Kubernetes-agent/</span></span></code></pre> Now you can create a file in you git repo with the path .gitlab/agents/<agent-name>/config.yaml</code> and even share the cluster access with other Git projects:</p> 1</span>c</span>i_access</span>:</span></span> 2</span> p</span>rojects</span>:</span></span> 3</span> -</span> i</span>d</span>:</span> m</span>y-team/serviceX</span></span> 4</span> -</span> i</span>d</span>:</span> e</span>xternal-team1/projectA</span></span> 5</span> -</span> i</span>d</span>:</span> e</span>xternal-team2/projectB</span></span> 6</span> g</span>roups</span>:</span></span> 7</span> -</span> i</span>d</span>:</span> q</span>a/loadtests</span></span></code></pre> No secrets needed, secure and controlled access to the cluster.</p> WAIT…That wasn’t the problem I wanted to solve. I need to share access with people - not with CI!</em></p> Good News:</strong> This also works for people.</p> You can just add the following to your .gitlab/agents/<agent-name>/config.yaml</code>:</p> 1</span>u</span>ser_access</span>:</span></span> 2</span> a</span>ccess_as</span>:</span></span> 3</span> u</span>ser</span>:</span> {</span>}</span></span> 4</span> p</span>rojects</span>:</span></span> 5</span> -</span> i</span>d</span>:</span> g</span>lobal/groups/admins</span></span> 6</span> g</span>roups</span>:</span></span> 7</span> -</span> i</span>d</span>:</span> m</span>y-team</span></span> 8</span> -</span> i</span>d</span>:</span> q</span>a/loadtests</span></span></code></pre> Now GitLab allows all users that are in these projects or groups to access the cluster endpoint via the agents tunnel. The important part in that config is the access_as.user</code> property. The Agent will use impersonation</a> to act like the GitLab user that uses the tunnel. That means you can easily set fine-grained permissions via RBAC to only allow the qa</code> group read access while the admins</code> have cluster-admin roles. If you are a masochist, you can also apply the RBAC rules per user.</p> The Solution</h3> Now, all users can configure their kubeconfigs to use the cluster endpoint https://gitlab.exmaple.com/-/Kubernetes-agent</code> and use their GitLab credentials to authenticate. Basic permissions are handled by GitLab and the fine-grained Kubernetes permissions are controlled via RBAC.</p> If a team member retires, just remove them from the GitLab group and the access is gone. A new team member needs onboarding, add them to GitLab and he already has access to the cluster. Easy management and even ISO 27001 compliant</em> 🙌</p> Complete Kubeconfig</summary> 1</span>a</span>piVersion</span>:</span> v</span>1</span></span> 2</span>k</span>ind</span>:</span> C</span>onfig</span></span> 3</span>c</span>urrent-context</span>:</span> g</span>itlab-cluster-via-agent-42</span></span> 4</span>c</span>lusters</span>:</span></span> 5</span> -</span> c</span>luster</span>:</span></span> 6</span> s</span>erver</span>:</span> h</span>ttps://gitlab.exmaple.com/-/Kubernetes-agent</span></span> 7</span> n</span>ame</span>:</span> g</span>itlab</span></span> 8</span>c</span>ontexts</span>:</span></span> 9</span> -</span> c</span>ontext</span>:</span></span> 10</span> c</span>luster</span>:</span> g</span>itlab</span></span> 11</span> u</span>ser</span>:</span> g</span>itlab-agent-42</span></span> 12</span> n</span>ame</span>:</span> g</span>itlab-cluster-via-agent-42</span></span> 13</span>u</span>sers</span>:</span></span> 14</span> #</span> Users can use a pat to access the cluster</span></span> 15</span> -</span> n</span>ame</span>:</span> g</span>itlab-agent-42</span></span> 16</span> u</span>ser</span>:</span></span> 17</span> t</span>oken</span>:</span> "</span>pat:42:XXX</span>"</span></span> 18</span> #</span> Or use the glab cli for just in time credentials with a 24h expiry</span></span> 19</span> -</span> n</span>ame</span>:</span> g</span>itlab-agent-42-glab</span></span> 20</span> u</span>ser</span>:</span></span> 21</span> e</span>xec</span>:</span></span> 22</span> a</span>piVersion</span>:</span> c</span>lient.authentication.k8s.io/v1</span></span> 23</span> a</span>rgs</span>:</span></span> 24</span> -</span> c</span>luster</span></span> 25</span> -</span> a</span>gent</span></span> 26</span> -</span> g</span>et-token</span></span> 27</span> -</span> --</span>agent</span></span> 28</span> -</span> "</span>42</span>"</span></span> 29</span> c</span>ommand</span>:</span> g</span>lab</span></span> 30</span> e</span>nv</span>:</span></span> 31</span> -</span> n</span>ame</span>:</span> G</span>ITLAB_HOST</span></span> 32</span> v</span>alue</span>:</span> h</span>ttps://gitlab.example.com</span></span> 33</span> -</span> n</span>ame</span>:</span> G</span>ITLAB_CLIENT_ID</span></span> 34</span> v</span>alue</span>:</span> X</span>XX</span></span> 35</span> i</span>nstallHint</span>:</span> "</span>To access the cluster install glab. Instructions at https://gitlab.com/gitlab-org/cli#installation.</span>\n</span>"</span></span> 36</span> i</span>nteractiveMode</span>:</span> N</span>ever</span></span> 37</span> p</span>rovideClusterInfo</span>:</span> false</span></span> 38</span></span></code></pre></details> Side Notes</h2> This is not the only solution to access private clusters. But this did the job and security is fine with it. Jump Hosts, VPNs and SSH tunnels are also commonly used but they are often a nightmare to manage or to set up. A notable alternative is tailscale which also creates a tunnel to you tailnet and allows for user authentication. Unfortunately, tailscale is not used at my current gig.</p> While these are solutions, I still would recommend to just hook your OIDC provider such as ActiveDirectory, LDAB or Keycloak into the Kubernetes api-server. It is also a much more standardized solution and should be the primary source for all user contexts. To my surprise not many managed Kubernetes offerings support setting the OIDC args for the api-server. So this might after all be a good solution if your Kubernetes provider is lacking.</p> The recurring problem of the Kubernetes metrics server and insecure Kubelet certificate 2024-05-18T00:00:00+00:00 The recurring problem of the Metrics Server and Insecure Kubelet certificate</h2> What is this about:</h3> Currently the kubelet, the client agent coordinating container lifecycle on each node, also runs a server, using a self signed certificate by default. Therefore any client connecting the the kubelet server, most prominent the metrics server, will not be able to verify the servers identity.</p> </p> Current state:</h3> Currently if the kubelet starts and no --tls-cert-file</code> and --tls-private-key-file</code> are specified it generated a self-signed pair and uses it for its server component according the kubelet docs</a> The is a feature feature flag called RotateKubeletServerCertificate</code> which would allow the use of serverTLSBootstrap</code>. This uses the native CertificateSigningRequest</code> (csr) resource wich is part of the certificates.k8s.io/</code> api. When serverTLSBootstrap</code> is enabled the kubelet creates a csr via the kubernetes certificates api for its server component. Users can either approve the csr manually with kubectl certificate approve <my-csr></code> or create a rolebinding wich allows for auto-approve. The resulting cert is singed by the clusters ca if the kube-controller has access to the ca cert and key (most likely it has). The process is described in the tls-bootstrapping</a> documentation, this provides the prerequisites for a trust chain verification.</p> The problem:</h3> With self-singed certificates there is no way to reliable verify the kubelet’s server identity. Even the metrics-servers docs says that passing --kubelet-insecure-tls</code> is not a solution for production. The recent addition serverTLSBootstrap</code> is an approvement but the control over the created csr is very limited to non existed. Most Kubernetes setups are not trivial. Nodes have multiple IPs and may be available over one ore move dns names, let alone IPv6 addresses. From my experience and according to other posts the created kubelet csr does not contain any SANs which could identify the server. The CN ist set to system:node:<name></code> wich is NOT a valid DNS name. This lack over the certificate details also results in the lack of the possibility to verify the identity of the kubelet server. Binging us back the the need for --kubelet-insecure-tls</code> for all clients talking to the kubelet. Even managed kubernetes offerings have this problem.</p> The Proposal:</h3> Kubernetes admins and vendors should have an easy possibility to crate trusted kubelet server certificates. This can be achieved either via the certificate generation phase of kubeadm or via passing extra args to the kubelet tls setup to include additional SANs.</p> 1</span>#</span> Existing behavior with etcd and kube-api-server</span></span> 2</span>a</span>piVersion</span>:</span> k</span>ubeadm.k8s.io/v1beta3</span></span> 3</span>k</span>ind</span>:</span> C</span>lusterConfiguration</span></span> 4</span>a</span>piServer</span>:</span></span> 5</span> c</span>ertSANs</span>:</span></span> 6</span> -</span> "</span>10.100.1.1</span>"</span></span> 7</span> -</span> "</span>ec2-10-100-0-1.compute-1.amazonaws.com</span>"</span></span> 8</span>e</span>tcd</span>:</span></span> 9</span> l</span>ocal</span>:</span></span> 10</span> i</span>mageRepository</span>:</span> "</span>registry.k8s.io</span>"</span></span> 11</span> s</span>erverCertSANs</span>:</span></span> 12</span> -</span> "</span>ec2-10-100-0-1.compute-1.amazonaws.com</span>"</span></span> 13</span>---</span></span> 14</span>a</span>piVersion</span>:</span> k</span>ubeadm.k8s.io/v1beta3</span></span> 15</span>k</span>ind</span>:</span> I</span>nitConfiguration</span></span> 16</span>n</span>odeRegistration</span>:</span></span> 17</span> n</span>ame</span>:</span> "</span>ec2-10-100-0-1</span>"</span></span> 18</span> k</span>ubeletExtraArgs</span>:</span></span> 19</span> n</span>ode-ip</span>:</span> "</span>10.100.0.1</span>"</span></span> 20</span> #</span> Proposal: ether as an arg</span></span> 21</span> s</span>erverCertSANs</span>:</span> "</span>10.100.0.1,ec2-10-100-0-1.compute-1.amazonaws.com</span>"</span></span> 22</span> #</span> Proposal: or as an own kubeadm nodeRegistration property</span></span> 23</span> s</span>erverCertSANs</span>:</span></span> 24</span> -</span> "</span>10.100.0.1</span>"</span></span> 25</span> -</span> "</span>ec2-10-100-0-1.compute-1.amazonaws.com</span>"</span></span></code></pre> Personal I’m in favor of adding an extra arg to the kubelet binary to include a list of additional SANs. This would also allow profit users not using kubeadm to have a properly configured kubelet server certificate.</p> The Workaround:</h3> For personal testing setups I use ansible to prepare my nodes and bootstrab a cluster. One the certificate generation phase is complete I can generate my onw private key and certificate request and sing it with the clusters ca. Though I don’t recommend this for production, this is why we need a more managed and automated approach.</p> References</h3> Kubernetes Docs TLS Bootstrap</a></li> Kubernetes Docs kubeadm troubleshoot tls</a></li> GitHub metrics-server error - missing SANs</a></li> GitHub kubernetes node - missing SANs</a></li> GitHub kubernetes singed serving certificates </a></li> Reddit - kubeadm not including SANs</a></li> </ul> Using AWS from GitHub without Credentials 2024-03-18T00:00:00+00:00 Using AWS from GitHub without Credentials</h1> AWS allows external services to authenticate without any secrets, which increases security and also reduces management overhead. There is no need to create, distribute, or rotate any secrets.</p> This short guide shows how to set up and use this federated identity concept.</p> Pre-Requirements</h2> To follow this guide, you need:</p> A GitHub repo with read/write access. We will use my-org/demo</code></li> A AWS account that can create roles.</li> This example will use terraform</a> for IaC deployments. Other tools or the WebUI should also work.</li> </ul> Configure AWS</h2> You need to configure your AWS account as an identity provider. This can be done via the aws_iam_openid_connect_provider</code> resource or via the AWS console. To use that identity provider, you need to create a role (or use an existing one) and establish a trust relationship with that role and your GitHub repo. This can can be archived with a iam_policy_document</code>. In terraform, this would look like this:</p> 1</span>#</span> Get the identity provider arn</span></span> 2</span>resource</span> "aws_iam_openid_connect_provider"</span> "github"</span> {</span></span> 3</span> client_id_list</span> =</span> [</span>"</span>sts.amazonaws.com</span>"</span>]</span></span> 4</span> thumbprint_list</span> =</span> [</span>"</span>1b511abead59c6ce207077c0bf0e0043b1382612</span>"</span>]</span></span> 5</span> url</span> =</span> "</span>https://token.actions.githubusercontent.com</span>"</span></span> 6</span>}</span></span> 7</span></span> 8</span>#</span> Create iam trust policy with github repo</span></span> 9</span>data</span> "aws_iam_policy_document"</span> "github_actions_assume_role"</span> {</span></span> 10</span> statement</span> {</span></span> 11</span> actions</span> =</span> [</span>"</span>sts:AssumeRoleWithWebIdentity</span>"</span>]</span></span> 12</span> principals</span> {</span></span> 13</span> #</span> GitHub identity provider</span></span> 14</span> type</span> =</span> "</span>Federated</span>"</span></span> 15</span> identifiers</span> =</span> [</span>data</span>.</span>aws_iam_openid_connect_provider</span>.</span>github</span>.</span>arn</span>]</span></span> 16</span> }</span></span> 17</span> condition</span> {</span></span> 18</span> test</span> =</span> "</span>StringLike</span>"</span></span> 19</span> variable</span> =</span> "</span>token.actions.githubusercontent.com:sub</span>"</span></span> 20</span> #</span> CHANGE ME: Set organization to my-org and to demo</span></span> 21</span> values</span> =</span> [</span>"</span>repo:</span>${</span>var</span>.</span>organization</span>}</span>/</span>${</span>var</span>.</span>git_repo_name</span>}</span>:</span>"</span>]</span></span> 22</span> }</span></span> 23</span> }</span></span> 24</span>}</span></span> 25</span></span> 26</span>#</span> Create role and assign iam_policy_document</span></span> 27</span>resource</span> "aws_iam_role"</span> "github_actions"</span> {</span></span> 28</span> name</span> =</span> "</span>github-actions-</span>${</span>lower</span>(</span>var</span>.</span>organization</span>)</span>}</span>-</span>${</span>lower</span>(</span>var</span>.</span>git_repo_name</span>)</span>}</span>"</span></span> 29</span> assume_role_policy</span> =</span> data</span>.</span>aws_iam_policy_document</span>.</span>github_actions_assume_role</span>.</span>json</span></span> 30</span> tags</span> =</span> var</span>.</span>tags</span></span> 31</span>}</span></span></code></pre> This is all that is needed to use this identity on the AWS side. Most likely, you need additional permissions attached to your roles in order to deploy lambda functions, access S3 or push containers to ECR. Just create this polices and attach them to the role as you need.</p> Configure GitHub Actions</h2> Now we need GitHub Actions to log into AWS. For this, we create a workflow with this minimal configuration:</p> 1</span>n</span>ame</span>:</span> D</span>emo</span></span> 2</span></span> 3</span>on</span>:</span></span> 4</span> p</span>ush</span>:</span></span> 5</span> b</span>ranches</span>:</span> [</span>m</span>ain</span>]</span></span> 6</span></span> 7</span>#</span> Minimal permissions needed</span></span> 8</span>p</span>ermissions</span>:</span></span> 9</span> c</span>ontents</span>:</span> r</span>ead</span></span> 10</span> i</span>d-token</span>:</span> w</span>rite</span></span> 11</span></span> 12</span>j</span>obs</span>:</span></span> 13</span> b</span>uild-image</span>:</span></span> 14</span> r</span>uns-on</span>:</span> u</span>buntu-latest</span></span> 15</span> s</span>teps</span>:</span></span> 16</span> -</span> n</span>ame</span>:</span> C</span>heckout repository</span></span> 17</span> u</span>ses</span>:</span> a</span>ctions/checkout@v4</span></span> 18</span></span> 19</span> -</span> n</span>ame</span>:</span> C</span>onfigure AWS Credentials</span></span> 20</span> u</span>ses</span>:</span> a</span>ws-actions/configure-aws-credentials@v4</span></span> 21</span> w</span>ith</span>:</span></span> 22</span> #</span> Your region. Can be a var or hardcoded</span></span> 23</span> a</span>ws-region</span>:</span> $</span>{{ vars.AWS_REGION }}</span></span> 24</span> #</span> Your role name you created previously</span></span> 25</span> r</span>ole-to-assume</span>:</span> $</span>{{ secrets.AWS_ROLE }}</span></span> 26</span></span> 27</span> -</span> n</span>ame</span>:</span> S</span>3 list</span></span> 28</span> r</span>un</span>:</span> a</span>ws s3 ls MY_BUCKET</span></span></code></pre> Note:</em> When you use repo secrets/vars you need to have maintainer or admin permissions for the GitHub repo. Yet, both values are not considered confidential.</p> Now you can run AWS commands in GitHub actions without having to set or manage any passwords or keys.</p> Comparing GitHub Actions with GitLab CI/CD - A deep dive! 2024-01-31T00:00:00+00:00 Comparing GitHub Actions with GitLab CI/CD</h1> Introduction</h2> GitLab just announced the availability of their GitLab CI/CD Catalog (2023/12),</a> which states a perfect opportunity to compare the current state of the two CI/CD systems of everyone’s favourite version control providers: GitLab CI & GitHub Actions.</p> Disclaimer:</em> While there are a lot of other CI/CD systems like Circle CI, Azure DevOps, AWS CodeBuild, Travis CI and surprisingly, even Jenkins is still around, these tools are dedicated to solving one problem and are not part of a fully integrated developer platform. I also have not used most of them to a sufficient extent to be able to provide a meaningful review. This post only focuses on GitLab CI and GitHub Actions, especially their SaaS offerings, from a technical and software engineering standpoint. The value of these tools may vary depending on your specific requirements.</p> 2024-10-12 Update:</em> GitLab is working on step-runners</a> a more modular approach of running CI jobs. The existing of this project might give some insides of what GitLab thinks for its current CI design.</p> </p> General CI setup & flow</h2> GitHub Actions and GitLab both define their CI/CD procedures using yaml</code> files which will be parsed, templated/interpolated, and processed by the CI/CD task scheduler. In general, one or more jobs are created which run in one or more stages. By default, jobs within one stage run in parallel, while stages run in sequence. Naturally, both GitHub Actions and GitLab have first-party integration support within their Git hosting platforms. While they can also be used to run tasks for Git repos hosted on a third-party service, it comes with limitations, often making it impractical. In fact, many triggers for the CI/CD jobs are directly related to the Git repositories on which they perform these actions. While GitHub evaluates each file in .github/workflows</code> individually by checking if the emitted event matches one of the many workflow triggers</a> specified via the on:</code> keyword, GitLab only has one CI file called .gitlab-ci.yml</code>. Whereas this file can include separate CI configurations, GitLab always merges all definitions into a single monolithic file and evaluates on a job-by-job base if the job should be created and run or not.</p> GitHub has a lot of triggers for every imaginable event that might accrue within a GitHub repository. Workflows can be run when an issue is created, labelled, or a comment is added. GitLab, on the other hand, only has the predefined CI_PIPELINE_SOURCE</code> variable which, in combination with other predefined variables,</a> can be used to decide if a CI job is created. It does not support events like issues or comments. However, integration with external event sources is more flexible in GitLab CI with the choice of API</code>, Triggers</code> and chat, enabled by the direct integration support of various applications such as Slack. GitHub only allows to manually create CI jobs via the workflow_dispatch</code> event, which can be triggered via the API or the WebUI. These and other differences also affect the way variables and tokens are used for these CI/CD systems. Basic event triggers such as the push</code> of a branch/tag or merge_requests</code> are supported by both systems.</p> Building the Jobs</h2> Based on the event and its values, the specified CI/CD jobs get created. While the general declaration of a job definition is quite similar, the specification and execution of the actual commands are quite different. Each job has a name, a selector to specify the runner, variables and script or step property. Both systems also support a matrix property to run one job definition multiple times for different environments, runtime versions or operating systems. Users can reference variables, secrets and event values within a job definition. GitHub even allows the usage of a limited set of expressions</a> to compare, encode or alter the content of variables. It also includes an additional property to determine if a job should be run after evaluating the on</code> property of a workflow file. GitLab only has one level of conditionals defined via it’s rules</code> property. Every job can also specify dependencies on other jobs. The actual steps that the CI should run are specified differently depending on the system used:</p> GitHub Actions</h3> GitHub Actions allows users to run an arbitrarily amount of action modules. An action is a (specific) reference to a other Git repository that performs a specific task within the CI. When a job is run, the action is checked out and runs its logic. From the users perspective, most actions are declarative, which allows users to utilize them without having to know their implementation details. Examples of actions include installing Docker</a> or setting up Minikube</a>. GitHub has a huge marketplace</a> of official and community actions. Users can customize the behavior of actions by passing input arguments to them. This approach allows for great reuse of tasks, a low entry barrier, and lean CI configurations. Besides the marketplace actions users can also use a generic run</code> action, which accepts any shell code, allowing any generic imperative commands. In addition to Bash and PowerShell, it is also possible to directly use Python, JavaScript and other interpreted languages. In addition to workflow control and job control, each step has an optional parameter to determine if the step should be executed.</p> 1</span>#</span> GitHub Action example workflow</span></span> 2</span>n</span>ame</span>:</span> E</span>xample Pipeline</span></span> 3</span></span> 4</span>on</span>:</span></span> 5</span> p</span>ull_request</span>:</span></span> 6</span> b</span>ranches</span>:</span> [</span>'</span>releases/</span>'</span>,</span> '</span>main</span>'</span>]</span></span> 7</span> p</span>ush</span>:</span></span> 8</span> b</span>ranches</span>:</span> [</span>m</span>ain</span>]</span></span> 9</span></span> 10</span>j</span>obs</span>:</span></span> 11</span> c</span>ode-style</span>:</span></span> 12</span> r</span>uns-on</span>:</span> u</span>buntu-latest</span></span> 13</span> s</span>teps</span>:</span></span> 14</span> -</span> n</span>ame</span>:</span> c</span>heckout</span></span> 15</span> u</span>ses</span>:</span> a</span>ctions/checkout@v4</span></span> 16</span></span> 17</span> -</span> n</span>ame</span>:</span> I</span>nfo for main branch</span></span> 18</span> i</span>f</span>:</span> g</span>ithub.ref == 'refs/heads/main'</span></span> 19</span> r</span>un</span>:</span> e</span>cho "Running tests in main"</span></span> 20</span></span> 21</span> -</span> u</span>ses</span>:</span> a</span>ctions/setup-python@v5</span></span> 22</span> w</span>ith</span>:</span></span> 23</span> p</span>ython-version</span>:</span> '</span>3.11</span>'</span></span> 24</span></span> 25</span> -</span> u</span>ses</span>:</span> h</span>ashicorp/setup-terraform@v3</span></span> 26</span> -</span> u</span>ses</span>:</span> t</span>erraform-linters/setup-tflint@v4.0.0</span></span> 27</span> -</span> u</span>ses</span>:</span> p</span>re-commit/action@v3.0.0</span></span></code></pre>GitLab CI</h3> GitLab has three different run stages: before_script</code>, script</code> and after_script</code>. Each stage allows a list of shell commands and can execute everything a script could. The after_script</code> is always executed, even if previous steps errored. Allowing users to send webhooks, perform error handling and print debug messages. Reusing these scripts is possible via inheriting from an existing job, extending from an existing job, or using the !reference</code> tag which allows using commands specified somewhere else. All commands have to be imperative shell instructions, which requires users to carefully construct every task they want to perform themselves.</p> 1</span>#</span> GitLab CI Pipeline example</span></span> 2</span></span> 3</span>t</span>ags</span>:</span> [</span>d</span>ocker</span>]</span></span> 4</span>v</span>ariables</span>:</span></span> 5</span> M</span>Y_GLOBAL_VAR</span>:</span> F</span>oo</span></span> 6</span> G</span>IT_DEPTH</span>:</span> 5</span></span> 7</span></span> 8</span>#</span> include:</span></span> 9</span>#</span> - template: Terraform.latest.gitlab-ci.yml</span></span> 10</span>#</span> - template: Security/SAST.gitlab-ci.yml</span></span> 11</span></span> 12</span>B</span>uild-Image-Buildah-Template</span>:</span></span> 13</span> i</span>mage</span>:</span> r</span>egistry.access.redhat.com/ubi8/buildah:latest</span></span> 14</span> s</span>tage</span>:</span> b</span>uild</span></span> 15</span> v</span>ariables</span>:</span></span> 16</span> R</span>EGISTRY</span>:</span> $</span>{CI_REGISTRY}</span></span> 17</span> R</span>EGISTRY_USER</span>:</span> $</span>{CI_REGISTRY_USER}</span></span> 18</span> R</span>EGISTRY_PASS</span>:</span> $</span>{CI_REGISTRY_PASSWORD}</span></span> 19</span> C</span>ONTEXT</span>:</span> $</span>CI_PROJECT_DIR</span></span> 20</span> B</span>UILD_IMAGE_TAG</span>:</span> $</span>CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME</span></span> 21</span> D</span>OCKERFILE</span>:</span> $</span>CONTEXT/Dockerfile</span></span> 22</span> s</span>cript</span>:</span></span> 23</span> -</span> b</span>uildah login -u "${CI_REGISTRY_USER}" -p "${CI_REGISTRY_PASSWORD}" "${CI_REGISTRY}"</span></span> 24</span> -</span> \|</span></span> 25</span> buildah bud --pull \</span></span> 26</span> --build-arg COMMIT_HASH=$CI_COMMIT_SHORT_SHA \</span></span> 27</span> --build-arg COMMIT_TAG=$CI_COMMIT_REF_NAME \</span></span> 28</span> --tag "${BUILD_IMAGE_TAG}" \</span></span> 29</span> --file ${DOCKERFILE} ${CONTEXT}</span></span> 30</span> -</span> b</span>uildah push --creds "${REGISTRY_USER}:${REGISTRY_PASS}" "${BUILD_IMAGE_TAG}"</span></span> 31</span> r</span>ules</span>:</span></span> 32</span> -</span> i</span>f</span>:</span> $</span>CI_PIPELINE_SOURCE == "schedule"</span></span> 33</span> -</span> i</span>f</span>:</span> $</span>CI_PIPELINE_SOURCE == "push"</span></span> 34</span> -</span> !reference</span> [</span>.</span>global</span>,</span> d</span>efault_rules</span>]</span></span></code></pre>Differences</h3> GitHub’s prebuild actions allow for an easy-to-use and quickly configured CI. They are great for reusability and the abstraction of low-level tasks. However, users and organizations must keep in mind that actions in the community marketplace is code written by others. It is recommended to check all actions before using them and to pin their reference to a specific commit to prevent supply chain attacks. GitLab’s approach requires more effort upfront and a deeper knowledge about every step in the setup to perform. While GitLab also allows reusing jobs, it is not as powerful as the later sections will show. The fine-grained and clear structure of GitHub Actions, that determines whether a workflow, job, or step should be executed is an advantage over the monolithic approach of GitLab.</p> Reusing CI Configurations</h2> GitHub Actions allow for great usability at the task level, as the previous section already showed. However, CI jobs are rarely constructed of just on step. Most jobs include checking out the code, installing the language runtime, downloading build dependencies and performing some checks. A Docker build job is almost always the same, just like security scans. They just need some arguments that point to the source and parameters for the output artifact. In an organization, it might be desirable to share these common build steps to keep it DRY</em> and reduce code hygiene maintenance. This would also assure that all jobs are compliant with internal standards and policies. GitHub Actions offer reusable workflows,</a> which are GitHub Action workflow files that can be called from any other project. These reusables define one or more jobs and offer inputs to customize how the included jobs are run. Being built on Git, these workflow files are versioned and can be pinned to a specific commit. For GitLab, the story is a little more complicated. GitLab uses a monolith .gitlab-ci.yml</code> file in which all jobs are defined. But the jobs don’t have to be declared within that file. They can be included from another project or any other file that is accessible via HTTP from the GitLab instance. In general, there are two types of includes</code> in GitLab, both can have multiple nested includes of their own. There is the generic include of any other job definition file that is offered by a shared repository or a third-party entity. The second type is the official GitLab CI/CD template</a> type. These templates are common CI/CD jobs that are directly provided by GitLab and come with every GitLab installation. Both types do not provide the possibility of directly passing inputs to the included jobs. All customizations must be done via GitLab’s CI/CD variables or by explicitly overriding the job to fit the desired needs. To apply these customizations, it is often necessary to look at the source of a specific job, which might be defined in multiple levels of nested includes. On December 21, 2023, GitLab announced the beta availability of the GitLab CI/CD Catalog (2023/12)</a>. This feature enables a third type of job inclusion into the .gitlab-ci.yml</code>. It provides a community store where every user can create their own definitions for one or more jobs. Anyone can include these components and customize them via a series of inputs. Consumers just need to look at the component page of the catalog item to get a list of all available inputs and further documentation. It’s a much cleaner interface between the provider and consumer of a job. Just like it is with GitHub Actions on the individual task level. The difference is that these catalog items still don’t let users fully alter any jobs on the individual task level. The components include whole jobs and are therefore more similar to GitHub’s reusable workflows than GitHub’s Action. A list of already available GitLab CI/CD Components can be found here</a>. Creating your own component is quite easy. GitLab requires a specific repository layout and a label on the project to specify it as a CI/CD Component. With GitLab releases, a new version of that component is created. To test this new functionality, I created my own component</a> that builds Docker images with Googles kaniko builder (which does not require Docker in Docker) and automatically applies some common OCI spec labels. Feel free to check it out. NOTE:</em> Just like third-party GitHub Actions, before using a third-party component, it should be verified and pinned to an exact version.</p> While the CI/CD Components are a step towards a more user-friendly and reusable CI code, they still don’t offer fine-grained control. GitHub’s conditional controls on workflow, job and step level offer a much more flexible setup and expose a cleaner solution than GitLab by including, merging, extending and overriding jobs from many sources. The complete .gitlab-ci.yml</code> file with all overrides, references and includes applied can be viewed in GitLab’s pipeline editor. However, due to the multiple inclusions, this file can quickly become a few thousand lines long and is not very user-friendly to read.</p> Runners</h2> To run the defined CI/CD jobs, both GitHub and GitLab need runners. Both services offer Hosted Runners</em> that are owned and maintained by the respective provider (SaaS offering). Each provider offers a free plan for their CI/CD runners, which includes a certain amount of compute minutes. The table below shows a brief, none complete overview of their offerings.</p> Plan</th> Included Minutes</th> Price</th> Cost per minute/per core for additional minutes</th></tr></thead> GitHub Free</td> 2000</td> 0$</td> 0.004$</td></tr> GitHub Pro</td> 3000</td> 4$</td> 0.004$</td></tr> GitLab Free</td> 400</td> 0$</td> 0.005$</td></tr> GitLab Pro</td> 10.000</td> 29$</td> 0.005$</td></tr> </tbody></table> Note:</em> Cost per minute is for extra minutes that exceed the included minutes. It is calculated based on available price information.</p> This table is not representative of real-world costs because both providers offer a variety of different runner sizes with different operating systems. Based on the size and operating system (OS), there is a cost multiplier that is factored in for every real compute minute consumed. The numbers in the table are for the default (smallest) runners based on a Linux system. For specific use cases, please refer to GitHub’s pricing docs</a> or GitLab’s pricing docs</a>. Both providers offer Linux, macOS and Windows runners. At the time of writing, GitLab’s offerings for macOS and Windows are still in beta. GitHub’s runners are hosted on the Azure Cloud using the Standard_DS2_v2</code> series VMs and GitLab’s runners are hosted on Google Compute Platform (GCP) using the n2d_machines</code> series. Both runner offerings are hosted in the US only currently, which may have an impact on latency and can interfere with privacy constraints. The architecture of both CI/CD systems envisions that the available runners continuously communicate with the GitHub/GitLab instance and process jobs from their task queue. Only the GitHub/GitLab instances has to be accessible by the runners, the runners themselves don’t need to be exposed to the internet. This can be important for certain restrictive network topologies. While the CI/CD job scheduling is quite similar, there are some differences between GitHub and GitLab in how the jobs are executed.</p> GitHub runners</h3> When an event that triggers a workflow is emitted, the defined jobs within the workflow get templated (using lazy templating) and are added to the job queue. Available GitHub runners that match the runs-on</code> tag automatically pick up queued jobs and execute them on a fresh VM provisioned just for that job. The GitHub provided VMs come preinstalled with a lot of tools</a> to allow for fast execution of the jobs. It essentially runs shell commands directly on the host, even if the actual commands are abstracted by the GitHub Action modules. Each step gets individually templated, which allows users to pass values from one step or job to another using a special GITHUB_OUTPUT</code> variable. These values do not have to be known before a job starts, they can be obtained and computed during job execution. The command outputs of jobs are streamed back to the GitHub instance, accessible via the GitHub UI in a shell-like output canvas. The output is near real-time, but the canvas can not be reliably searched via Ctrl-F</code>.</p> GitLab runners</h3> The .gitlab-ci.yml</code> gets fully templated and merged every time there is a CI triggering event. If the rules</code> section of a job matches the triggering event, it gets added to the job queue. GitLab uses tags</code> to specify an attribute or a list of attributes that the runner must have in order to run the job. An example would be tags: [linux, large]</code>, where the runner must have at least these tags. GitLab’s runners can have different executors, which determine how the user-defined job gets executed. The supported executors are Shell, SSH, VirtualBox, Docker, Kubernetes and some more</a>. Each executor comes with some advantages and disadvantages. The Shell and SSH executors directly run on the specified machine. Users must make sure that every build dependency is either already installed on the machine or is included in the job script section. These executors change the host filesystem and are not idempotent, which can lead to a lack of reproducibility and unexpected behaviour. Therefore, the use of a virtualized executor is often preferred. Docker and Kubernetes use the container image specified in the image</code> tag and run all scripts within that container. When the job is finished, the container gets deleted and the host system is in the same state as before. Currently, GitLab’s SaaS runners use the Docker+Machine</code> executor, a GitLab maintained fork</a> of the deprecated Docker Machine project. These executors spawn new Google Cloud VMs with Docker installed to run the specified container image and execute all job scripts. At the time of writing, GitLab is working on replacing this executor with an alternative</a>. While these executors are idempotent, they have some drawbacks when users want to perform some low-level system actions or need to use virtualisation themselves, resulting in nested virtualisation. Running container builds in the GitLab CI often requires the use of Docker in Docker, resulting in bind-mount of the Docker Socket and slower build performance, unless tools like kaniko</em> or buildah</em> are used. Variables between jobs can be passed to each other by writing them to a file and passing these artefacts between jobs. The script outputs are also sent back to the GitLab instance in periodic chunks. The resulting log in the UI is fully searchable. Other than GitHub, GitLab automatically downloads the Git repository at the ref specified in the event that triggered the job, unless it’s explicitly turned off. If artefacts were defined in the job, they also get automatically up and downloaded. GitLab also offers a wider set of predefined variables</a> that users can access while the jobs executes.</p> Comparison</h3> The execution of jobs is quite different between GitHub and GitLab. While the GitLab solution seems more complex at first glance, knowledge of its runner architecture is not required by developers. Only for GitLab admins who implement their own runners need to know the details in order to decide for an appropriate executor. Depending on the size of the organization, a simple Docker or Kubernetes executor should cover most use cases for a CI/CD runner. GitHub also allows users to bring their own, self-hosted runners, which just like GitLab will not be billed (at least not by GitHub/GitLab). Own runners allow for better confidentiality between own jobs and the jobs of others, have consistent performance and meet specific requirements such as CPU architecture and GPU support. It also allows overcoming network borders by placing the runner inside restricted network segments to allow for communication with services that are not publicly accessible. Own runners might also be needed to overcome latency issues, since currently both GitHub’s and GitLab’s runners are located in the US. GitHub’s runners allow for more flexible usage by just being a VM that is owned by you during the duration of the job, while GitLab’s runners might be orientated more towards one specific use case. GitLab’s runners automatically check out the code and manage artefacts distribution for the user. If the right container image is chosen, there it literally no need to install anything since all tools are already the present in container. Both runners should be able to fulfil most requirements, users should just be aware of the respective limitations regarding location, availability, CPU architecture and other feature and properties like cost. It must be mentioned that self-hosted runners can run more than one job per host concurrently, depending on the configuration. Due to privacy concerns, this feature is not enabled for hosted/SaaS runners. A performance comparison is intentionally not part of this comparison, since there are too many variables and influences present for any meaningful results.</p> Security and Secrets</h2> Both GitHub and GitLab allow users to set variables and secrets that are then available in their CI/CD runtime. Both input types can be set at the instance, group/organization, project level, and even per registered deployment environment. Secrets are special variables that are considered confidential, these will be masked by the CI system if they are found in the output log. GitLab allows users with the appropriate permissions to view the value of a secret after it has been set, while GitHub never shows the value again. Regarding security, this makes no real difference, as anyone with access to the CI files can output the secrets when executing a job. While the secret values will be masked, this safety mechanism can easily be bypassed by reversing and base64 encoding the value before they get written to the log. Therefore, the general recommendation is the usage of short-lived tokens. For project-related resources, both CI systems use such a short-lived token to access the code stored in Git, upload artifacts or create releases. The GITHUB_TOKEN</code> or GitLab’s CI_JOB_TOKEN</code> is only valid during the execution of the job. The permissions for these tokens can be limited. GitLab allows altering the permissions of users, they can be limited to a specific scope and certain actions. The CI_JOB_TOKEN</code> will inherent the permissions of the user that triggered the CI job. However, when a CI job runs, this token is accessible, with its specific permissions, for all jobs and tasks within these jobs. GitHub on the other hand, allows for a more fine-grained permission</a> model for its token. These permissions can also be defined at the workflow or job level. Unlike GitLab, GitHub does not make its token accessible for every step in the CI flow by default. Creating a release via GitHub Actions requires access to the GitHub release API with a valid token, downloading public dependencies or compiling code does not require any tokens. Only steps that request access to the token or are explicitly given access via the ${{ secrets.GITHUB_TOKEN }}</code> parameter can use that token. The default untrustworthy approach makes a lot of sense since users are much more likely to use third-party actions in their CI, which is code written by strangers. Before the introduction of the GitLab CI/CD catalog, most CI code in GitLab was owned by the project owners or their organization. GitHub defaults to the least privilege solution with less trust at every step, where access can be given only if required. Organizations can enforce an even more permissive permission scope for the GITHUB_TOKEN</code> and can also limit the allowed GitHub Actions to internal or whitelisted ones in order to meet company compliance policies. GitLab does not currently support such a feature out of the box. Instead, GitLab allows users to extend the permissions of its CI_JOB_TOKEN</code> to access additional Git repositories. In a multi repository/microservice project, this results in a smaller number of user-managed tokens. To access multiple private repos in GitHub, users need to create their own token since the GITHUB_TOKEN</code> currently does not support such a feature and is only valid for the current repository. Both services encourage the use of short-lived tokens or external secret solutions. GitLab has first-party support with HashiCorp Vault to use external secrets. Besides this, both GitHub and GitLab act as an Identity-Provider</em> (IDP) to allow the usage of OpenID Connect (OIDC)</a> to offer secure communication with the three major cloud providers using short-lived tokens. Depending on the configurations, this allows users to access and manage cloud resources in the CI/CD by using a trust relationship rather than any explicit credentials. GitLab builds this functionality directly into their CI/CD system, while GitHub relies on GitHub Actions like configure-aws-credentials</code> or azure-login</code>. A useful extra feature that GitLab does not have is the ability to add secret masks while the CI is already running. Especially when using short-lived access tokens, which the CI cannot know in advance, it is helpful to use echo "::add-mask::MY_TOKEN"</code> to prevent unwanted exposure in logs.</p> Additional Functionality & Services</h2> Running predefined tasks when a specific event is triggered is the core functionality of a CI/CD system, but requirements and competition in the CI/CD world are constantly increasing. Having functionality that makes the lives of developers easier might be a major selling point. These following functions were not considered in the previous comparison, but should be mentioned for a complete picture of GitHub Actions and GitLab CI. Both GitHub Actions and GitLab CI can run additional services besides the main CI tasks. These services are defined as containers that can offer a database or local web server within the CI job. This can allow users to run integration tests within the CI. A job can perform a database migration test on an ephemeral DB or an end-to-end test with playwright against a temporary hosted website, all within a CI job. Project documentation or a static website can be hosted on GitHub and GitLab using their Pages</em> feature. Both CI/CD systems can build and directly deploy HTML sites to their page environments without the need for any extra authentication. To speed up tests and build jobs, both systems support caches. GitHub has official actions to upload and download caches, while GitLab offers the cache</code> keyword and manages uploading and downloading caches as part of the job initialisation. When all test succeeded, software packages can be deployed to one or more environments. Both systems offer a environment</code> keyword. Every job that references an environment creates a deployment in that environment. Environments can have their own set of variables and secrets to allow stricter separation between them for increased security. A link to a successful deployment is then shown on the repository or pull request UI. GitHub requires users to create environments beforehand. GitLab can dynamically create environments based on supplied variables, like the branch name. Furthermore, GitLab also has different actions for environments to create, access and delete them on demand. Environments can even be automatically removed after a certain time or after a pull request gets merged. Compared to GitHub, GitLab’s environments are definitely more advanced and versatile. For Kubernetes users, GitLab offers its optional GitLab Agent</a> to access the insides of a cluster without any additional tokens. The agent gets deployed within a cluster and communicates with a GitLab instance, allowing it to be accessed via the CI.</p> Overall Comparison</h2> GitHub Actions and GitLab CI are two very powerful CI/CD systems that allow users to automate almost every imaginable task within the software development life cycle. At a functional level, both systems are on a similarly equal level, but the way they are implemented differs significantly. GitHub offers a stable, versatile and modular platform to run almost every kind of imaginable task. It integrates seamlessly with every one of GitHub’s features. Where needed, it allows for integration with standard solutions, such as the ability to run containers or authenticate with third-party services over OIDC. It relies heavily on its vast marketplace of official and community actions, which do most of the actual work, such as checking out code, installing software, using caches or publishing releases. GitHub also gives fine-grained control over security-related settings, which can be enforced to be significantly more permissive than its competitor, GitLab. GitLab on the other hand, provides a much more managed solution for its users. It is part of the CI/CD system to check out code, manage caches and releases. It directly integrates with Vault and major cloud providers. GitLab itself also seems to allow for easier integration with third-party systems. CI triggers from external services like issue trackers and chat tools are much more versatile, just like the CI environment setup. But GitLab requires the user to write all the CI task implementations themselves. Sharing and reusing CI definitions becomes difficult and quickly confusing with the monolithic approach of one big .gitlab-ci.yml</code> file approach. There are multiple levels of including, inherence and extending which can make it hard to get a quick oversight of what a job actually does. Eventually, this might get better with GitLab’s new CI/CD catalog, but this is not guaranteed since components compare much more to GitHub’s reusable workflows instead of individual actions. GitLab is definitely in the middle of a migration process regarding their CI, not just to allow it to be more reusable, but also to get rid of their docker+machine</code> runners. In contrast to GitHub’s modular approach, GitLab seems to have to deal with significantly more historical challenges due to its more managed approach. From a software engineering perspective, GitHub’s design might be the more beneficial long-term position because their modular approach allows them to add features without many legacy impediments. There is no clear recommendation for one or the other system. Both are more than capable of running any imaginable task. The requirements for the actual development Platform, GitHub and GitLab, should be much more important than its CI features. Costs and data privacy, as well as the ability to self-host, are often more relevant factors than the differences between the CI/CD systems.</p> </p> Building preconfigured OS images with HashiCorp Packer 2023-12-12T00:00:00+00:00 Building preconfigured OS images with Packer</h1> Introduction</h2> Hetzner Cloud provides a number of different operating system images to select from when creating a new virtual machine. These images provide the basic ground to get started with your service, but they are rarely configured according to your specific needs. Maybe you want to install a database or a web server, add a maintenance user, or just tweak system settings to improve performance and security. While there are tools to do this automatically for every new VM, this takes time and may not always be reproducible. Wouldn’t it be easier to have your own known good personal pre-configured OS image that you can use for every new VM?</p> This can be archived with HashiCorp Packer.</p> This article demonstrates how to use Packer to create custom images for your Hetzner VMs using Infrastructure as Code (IaC). It even allows users to create VMs with operating systems that are not officially supported by Hetzner.</p> Note:</em> This article was also published on Hetzner’s Community Blog</a> </p> Prerequisites</strong></p> To follow this tutorial, you need:</p> Hetzner Cloud API token</a> in the Cloud Console</a></li> Access to the Internet</li> </ul> Installing Packer</h2> Visit Hashicorp Packer</a> and install Packer according to your OS. On Debian based Linux, run:</p> 1</span>#</span> Get the latest version</span></span> 2</span>curl</span> -</span>sL</span> https://api.github.com/repos/hashicorp/packer/releases/latest</span> \|</span> jq</span> -</span>r</span> .tag_name</span></span> 3</span>#</span> Set the version and download</span></span> 4</span>PACKER_VERSION</span>=</span>1</span>.</span>1</span>1</span>.</span>2</span></span> 5</span>curl</span> -</span>sL</span> https://releases.hashicorp.com/packer/</span>$</span>{</span>PACKER_VERSION</span>}</span>/packer_</span>$</span>{</span>PACKER_VERSION</span>}</span>_linux_amd64.zip</span> -</span>o</span> /tmp/packer.zip</span></span> 6</span>#</span> Unzip - may needs sudo</span></span> 7</span>unzip</span> -</span>q</span> /tmp/packer.zip</span> -</span>d</span> /usr/local/bin</span> -</span>x</span> "</span>LICENSE.txt</span>"</span></span></code></pre> You can run packer --version</code> to view the version and check if the installation was successful.</p> Creating a custom image</h2> Create a project folder with mkdir my-hetzner-img</code> and enter the directory. Similar to Terraform, Packer uses providers to communicate with the needed build system. The usage of different providers allows Packer to offer a vast verity of different target systems. It allows users to create software or OS packages for all major cloud providers, on-prem systems and even Docker images. To use a provider, create a file called provider.pkr.hcl</code> and add the provider config:</p> 1</span>#</span> packer.pkr.hcl</span></span> 2</span>packer</span> {</span></span> 3</span> required_plugins</span> {</span></span> 4</span> hcloud</span> =</span> {</span></span> 5</span> source</span> =</span> "</span>github.com/hetznercloud/hcloud</span>"</span></span> 6</span> version</span> =</span> "</span>>= 1.2.0</span>"</span></span> 7</span> }</span></span> 8</span> }</span></span> 9</span>}</span></span></code></pre> Packer uses the HashiCorp configuration language (HCL) for declaring the desired state, just like Terraform.</p> To create a personalized custom image, Packer needs a base form where it starts from. This is called the “source”:</p> 1</span>#</span> custom-img-v1.pkr.hcl</span></span> 2</span>source</span> "hcloud"</span> "base-amd64"</span> {</span></span> 3</span> image</span> =</span> "</span>debian-12</span>"</span></span> 4</span> location</span> =</span> "</span>nbg1</span>"</span></span> 5</span> server_type</span> =</span> "</span>cx11</span>"</span></span> 6</span> ssh_keys</span> =</span> [</span>]</span></span> 7</span> user_data</span> =</span> "</span>"</span></span> 8</span> ssh_username</span> =</span> "</span>root</span>"</span></span> 9</span> snapshot_name</span> =</span> "</span>custom-img</span>"</span></span> 10</span> snapshot_labels</span> =</span> {</span></span> 11</span> base</span> =</span> "</span>debian-12</span>"</span>,</span></span> 12</span> version</span> =</span> "</span>v1.0.0</span>"</span>,</span></span> 13</span> name</span> =</span> "</span>custom-img</span>"</span></span> 14</span> }</span></span> 15</span>}</span></span></code></pre> This tells Packer where to start. In the case of Hetzner, Packer needs to know the base image, the server type to use for building, and the location of that server. It also specifies the name of the Snapshot that will be created and the tags that should be applied to that server.</p> The next step is to provide your customization. Meaning the actual build step.</p> 1</span>#</span> custom-img-v1.pkr.hcl</span></span> 2</span>build</span> {</span></span> 3</span> sources</span> =</span> [</span>"</span>source.hcloud.base-amd64</span>"</span>]</span></span> 4</span> provisioner</span> "shell"</span> {</span></span> 5</span> inline</span> =</span> [</span></span> 6</span> "</span>apt-get update</span>"</span>,</span></span> 7</span> "</span>apt-get install -y wget fail2ban cowsay</span>"</span>,</span></span> 8</span> "</span>/usr/games/cowsay 'Hi Hetzner Cloud' > /etc/motd</span>"</span>,</span></span> 9</span> ]</span></span> 10</span> env</span> =</span> {</span></span> 11</span> BUILDER</span> =</span> "</span>packer</span>"</span></span> 12</span> }</span></span> 13</span> }</span></span> 14</span>}</span></span></code></pre> It uses the source specified before and specifies a shell</code> provisioner. Every line in inline</code> will be executed on that server. In this case, it installs fail2ban</code> to harden the server by slowing down the 1000s of SSH login attacks that happen regularly and adding a little message to the motd</code> file.</p> Note:</strong> When you run packer build .</code>, Packer will automatically create a new server and a new Snapshot in your Hetzner Cloud project. Both will be charged. The server is automatically</strong> deleted immediately after either the Snapshot was created or the creation failed.</p> </blockquote> To create that image run the following commands:</p> 1</span>#</span> Set your Hetzner API Token</span></span> 2</span>export</span> HCLOUD_TOKEN</span>=</span>"</span>XXX</span>"</span></span> 3</span>#</span> Initialize the project - only needed once</span></span> 4</span>packer</span> init</span> .</span></span> 5</span>#</span> Build</span></span> 6</span>packer</span> build</span> .</span></span></code></pre> Now Packer builds that server and streams all the logs to your current terminal. Verify that the new image is available in the Hetzer Cloud Console</a>:</p> </p> Taking it to the next level</h2> Putting all commands in HCL can quickly get crowded. It is also not reasonable to copy-paste the code for every variant of an image you might want to build. To make things more generic, Packer can use variables:</p> 1</span>#</span> custom-img-v2.pkr.hcl</span></span> 2</span>variable</span> "base_image"</span> {</span></span> 3</span> type</span> =</span> string</span></span> 4</span> default</span> =</span> "</span>debian-12</span>"</span></span> 5</span>}</span></span> 6</span>variable</span> "output_name"</span> {</span></span> 7</span> type</span> =</span> string</span></span> 8</span> default</span> =</span> "</span>snapshot</span>"</span></span> 9</span>}</span></span> 10</span>variable</span> "version"</span> {</span></span> 11</span> type</span> =</span> string</span></span> 12</span> default</span> =</span> "</span>v1.0.0</span>"</span></span> 13</span>}</span></span> 14</span>variable</span> "user_data_path"</span> {</span></span> 15</span> type</span> =</span> string</span></span> 16</span> default</span> =</span> "</span>cloud-init-default.yml</span>"</span></span> 17</span>}</span></span> 18</span></span> 19</span>source</span> "hcloud"</span> "base-amd64"</span> {</span></span> 20</span> image</span> =</span> var</span>.</span>base_image</span></span> 21</span> location</span> =</span> "</span>nbg1</span>"</span></span> 22</span> server_type</span> =</span> "</span>cx11</span>"</span></span> 23</span> ssh_keys</span> =</span> [</span>]</span></span> 24</span> user_data</span> =</span> file</span>(</span>var</span>.</span>user_data_path</span>)</span></span> 25</span> ssh_username</span> =</span> "</span>root</span>"</span></span> 26</span> snapshot_name</span> =</span> "</span>${</span>var</span>.</span>output_name</span>}</span>-</span>${</span>var</span>.</span>version</span>}</span>"</span></span> 27</span> snapshot_labels</span> =</span> {</span></span> 28</span> base</span> =</span> var.base_image,</span></span> 29</span> version</span> =</span> var.version,</span></span> 30</span> name</span> =</span> "</span>${</span>var</span>.</span>output_name</span>}</span>-</span>${</span>var</span>.</span>version</span>}</span>"</span></span> 31</span> }</span></span> 32</span>}</span></span> 33</span></span> 34</span>build</span> {</span></span> 35</span> sources</span> =</span> [</span>"</span>source.hcloud.base-amd64</span>"</span>]</span></span> 36</span> provisioner</span> "shell"</span> {</span></span> 37</span> scripts</span> =</span> [</span></span> 38</span> "</span>os-setup.sh</span>"</span>,</span></span> 39</span> ]</span></span> 40</span> env</span> =</span> {</span></span> 41</span> BUILDER</span> =</span> "</span>packer</span>"</span></span> 42</span> }</span></span> 43</span> }</span></span> 44</span>}</span></span></code></pre> Now, we can specify and override the base image at runtime by using packer build -var bas_image=ubuntu-22.04 -var version=v1.1.0 .</code>. This version also uses the scripts</code> parameter rather than in-lining all the commands. This allows for much more complex setups. It also uses cloud-init</a> to configure some settings declaratively.</p> Hetzner also supports Arm64-based servers</a> which offer a great price-to-performance ratio and impressive efficiency. Due to the different nature of Arm, software needs to be compiled for this architecture and all configurations also need to be applied for this architecture. Fortunately, Packer allows to reuse any build scripts for Arm.</p> Just add another source to your code and update the build step to include that source:</p> 1</span>#</span> custom-img-v2.pkr.hcl</span></span> 2</span>source</span> "hcloud"</span> "base-arm64"</span> {</span></span> 3</span> image</span> =</span> var</span>.</span>base_image</span></span> 4</span> location</span> =</span> "</span>nbg1</span>"</span></span> 5</span> server_type</span> =</span> "</span>cax11</span>"</span></span> 6</span> ssh_keys</span> =</span> [</span>]</span></span> 7</span> user_data</span> =</span> file</span>(</span>var</span>.</span>user_data_path</span>)</span></span> 8</span> ssh_username</span> =</span> "</span>root</span>"</span></span> 9</span> snapshot_name</span> =</span> "</span>${</span>var</span>.</span>output_name</span>}</span>-</span>${</span>var</span>.</span>version</span>}</span>"</span></span> 10</span> snapshot_labels</span> =</span> {</span></span> 11</span> base</span> =</span> var.base_image,</span></span> 12</span> version</span> =</span> var.version,</span></span> 13</span> name</span> =</span> "</span>${</span>var</span>.</span>output_name</span>}</span>-</span>${</span>var</span>.</span>version</span>}</span>"</span></span> 14</span> }</span></span> 15</span>}</span></span> 16</span>...</span></span> 17</span>build</span> {</span></span> 18</span>-</span> sources</span> =</span> [</span>"</span>source.hcloud.base-amd64</span>"</span>]</span></span> 19</span>+</span> sources</span> =</span> [</span>"</span>source.hcloud.base-amd64</span>"</span>,</span> "</span>source.hcloud.base-arm64</span>"</span>]</span></span> 20</span> provisioner</span> "shell"</span> {</span></span> 21</span>...</span></span></code></pre> This version now automatically builds images for Arm64-based servers.</p> Tips and advanced usage:</h2> About disk size:</strong> You may have noticed that the code above uses the smallest instances available on Hetzner. This is not only to reduce cost but also allows for the created Snapshot to be deployed for every VM. Due to the way disks are managed at Hetzner, a new VM must have a disk that is at least the same size as the one from which the Snapshot was created. Using an eight-core server might be a little faster than using a smaller one, but it would limit the servers you can deploy form this image to VMs with at least 240GB disks.</p> About Tags:</strong> Tags are metadata that you can add to a Snapshot. These are useful to provide information about the origin of an image and what it is used for. Also, tags are the only way to select an existing Snapshot in Terraform to deploy a new VM. They should definitely be used.</p> About cloud-init:</strong> You can use cloud-init</code> to do some early setup while building images with Packer, but you might also want to use cloud-init</code> when deploying your custom image. By default, cloud-init</code> only runs once, so it needs to be reset. Consider adding the following lines to your bash script to wait and reset cloud-init</code>.</p> 1</span>#</span> os-setup.sh</span></span> 2</span>#!</span>/bin/bash</span></span> 3</span>set</span> -</span>e</span> -</span>o</span> pipefail</span></span> 4</span></span> 5</span>echo</span> "</span>Waiting for cloud-init to finish...</span>"</span></span> 6</span>cloud-init</span> status</span> -</span>-wait</span></span> 7</span></span> 8</span>echo</span> "</span>Installing packages...</span>"</span></span> 9</span>apt-get</span> update</span></span> 10</span>apt-get</span> install</span> -</span>-yes</span> -</span>-no-install-recommends</span> wget</span> fail2ban</span></span> 11</span></span> 12</span>#</span> My setup...</span></span> 13</span></span> 14</span>echo</span> "</span>Cleanup...</span>"</span></span> 15</span>cloud-init</span> clean</span> -</span>-machine-id</span> -</span>-seed</span> -</span>-logs</span></span> 16</span>rm</span> -</span>rvf</span> /var/lib/cloud/instances</span> /etc/machine-id</span> /var/lib/dbus/machine-id</span> /var/log/cloud-init</span></span></span></code></pre> This allows users to run cloud-init</code> a second time.</p> Different operating systems:</strong> You can create and boot completely different operating systems and create your own images for them using Packer. By booting into Hetzner’s rescue</em> mode, you can override the entire disk image of a VM:</p> 1</span>build</span> {</span></span> 2</span> sources</span> =</span> [</span>"</span>source.hcloud.myvm</span>"</span>]</span></span> 3</span> provisioner</span> "shell"</span> {</span></span> 4</span> inline</span> =</span> [</span></span> 5</span> "</span>apt-get install -y wget</span>"</span>,</span></span> 6</span> "</span>wget https://github.com/siderolabs/talos/releases/download/v1.5.5/hcloud-amd64.raw.xz</span>"</span>,</span></span> 7</span> "</span>xz -d -c hcloud-amd64.raw.xz \| dd of=/dev/sda && sync</span>"</span>,</span></span> 8</span> ]</span></span> 9</span> }</span></span> 10</span>}</span></span></code></pre> My personal use case for Packer is to create preconfigured Kubernetes images with fine-tuned system parameters and preloaded container images. The horizontal autoscaler can use these images to provision new worker nodes quickly and reliably. But there are plenty of other use cases.</p> </p> For further configuration, check out the Packer documentation</a> and the Hetzner Packer builder documentation</a>. You can find the entire code used in this article can be found on GitHub</a>.</p> Conclusion</h2> This article gave a comprehensive guide on how to install and use HashiCorp Packer. It provided a base project on how to use Packer to create custom OS images for Hetzner Cloud VMs, leveraging Infrastructure as Code practices. It also addressed advanced using variables and cloud-init. This approach allows for the creation of tailored, pre-configured OS images, offering a more efficient and streamlined process for deploying VMs with customized settings and applications.</p> Building and Cracking the Enigma - Part I 2023-07-09T00:00:00+00:00 Building and Cracking the Enigma - Part I</h1> As more and more parts of our lives are shifted to the internet, the subject of security is becoming increasingly relevant. Large language models (LLMs) and chat controls claim more and more personal data. A significant part of security on the internet is achieved through encryption. During the last few weeks, I have been increasingly engaged with encryption and went down the rabbit-hole of encryption history. Despite the exciting early days of message encoding and encryptions like the Caesar-Cipher, the Enigma has a very special place in history, especially from the point of a computer scientist. Thats way I decided to rebuild the enigma in software, to subsequently also try to crack it.</p> </p> What is the Enigma</h3> The Enigma is a cipher device mainly used in the second world war by the Nazis. Most of the military communication was transmitted with messages encrypted via the Enigma. Even when the allies obtained one of the Enigma devices, they were not able to decipher their messages because of its complex design created by Arthur Scherbius.</p> How it works</h3> To understand the following parts, I want to provide a quick (and none complete) overview of how the Enigma works. The Enigma is a symmetric encryption machine in the form of a typewriter. Each Enigma has at least four rotors (wheels) with about 26 steps. There are some variants of the Enigma which also include additional turnover rotors and switches, but for simplicity these will not be considered her. Common to all Enigmas however, was a plugboard where the user could electricity connect two outlets to interchange the key for the letter that was pressed via the letter input buttons. When the user pressed a key an electrical current went through the, may or may not switched, keys of the plugboard, to the entry rotor stepping it and eventually their subsequent rotors, to a final turnover rotor which ultimately highlighted the ciphered latter on the displayed alphabet. Even though the Enigma is a symmetric encryption device, its design always produced different outputs even when the same key was pressed multiple times.</p> </p> To decipher a given text, the user would need a compatible Enigma machine and must know the exact configuration for the initial rotor state and the plugboard configuration. Overall there where about 150 trillion possible configurations</strong></p> So cracking this will be quiet a challenge. A challenge even worth a hole (and fantastic) movie. The Imitation Game</a> tells the storey about cracking the Enigma and the invention of our modern day computing model by the mathematician Alan Turing.</p> Some Cryptographic Context</h3> Cryptography is the term used to describe the encryption of confidential information so that it can only be fully recovered by the intended recipient. Besides cryptography, there is stenography, which deals with hiding information and there is hashing which is also often used in the context of cryptography. With hash functions, however, information is lost, so that this technique is more suitable for unique assignments and the verification of information.</p> Cryptography and hash functions are fundamental components of our digital lives today. For example, the layers of the ISO-OSI reference model does not specify how to ensure, the information sent is identical to the information received. That’s why TCP (the most used protocol on the web) has built-in error detection, based on checksums using hash functions. As the Internet grew out of its baby shoes and spread across universities into every aspect of our lives today, it became clear that the Internet was not designed with security in mind.</p> In today’s world, TLS/SSL forms the basis of this, post-field added, security. On the web, public certificates and private keys are used to encrypt information from unauthorized entities. This is based on an asymmetric process where anyone can encrypt information with the public key, but only the owner of the private key can decrypt it. This is also the foundation for SSH keys and digital signatures. Since asymmetric encryption and decryption is quite time consuming, on the web there is a separate handshake (more about the SSL handshake</a>) negotiated per session, where both sides agree on a secret symmetric key and then use it for encryption. With a symmetrical encryption you can encrypt and decrypt information with the same key, like with the Enigma. With the Enigma, the rotor and switchboard settings are the key, on the Internet it is often a password.</p> With each and every encryption there is the possibility guess the key. Mathematicians have accordingly searched for cryptographic methods that have a high complexity, making it difficult to guess the key. Trying every possibility in the problem-space through brute-force results in complexity, often measured in bits. Here is a small comparison:</p> Encryption</th> Possibilities base 2</th> Possibilities base 10</th></tr></thead> Enigma</td> 2^68</td> ≈ 15 × 10^19</td></tr> RSA 2048</td> 2^112</td> ≈ 51 × 10^32</td></tr> AES 265</td> 2^265</td> ≈ 59 × 10^78</td></tr> </tbody></table> Just as a reminder: Every additional bit doubles the possibilities. Current computers would take about 300 trillion years to crack a RSA 2048 key. To get a glims about AES 265 check out this great video</a>.</p> Implementing the Enigma</h3> I decided to rebuild the Enigma in software, to deeper understand its working, using my go to prototyping language python. First I needed to narrow down the scope of the parts I wanted to implement. I ended up with the following plan:</p> Enigma with 3 rotors and 1 reverse rotor</li> Support for the plugboard</li> Support for the letters a-z</li> Support for choice for 3 out of 5 possible rotors</li> Support for 1 out of 3 reverse rotors</li> </ul> This is a doable scope while also keeping the core concepts of the original Enigma. Modeling the rotors in software resulted basically in one list for every rotor containing tuples for the next step-position.</p> 1</span>allRotor</span> =</span> {</span></span> 2</span> 1</span>:</span> [</span>[</span>0</span>,</span> 4</span>]</span>,</span> [</span>1</span>,</span> 10</span>]</span>,</span> [</span>2</span>,</span> 12</span>]</span>,</span> [</span>3</span>,</span> 5</span>]</span>,</span> [</span>4</span>,</span> 11</span>]</span>,</span> [</span>5</span>,</span> 6</span>]</span>,</span> [</span>6</span>,</span> 3</span>]</span>,</span> [</span>7</span>,</span> 16</span>]</span>,</span> [</span>8</span>,</span> 21</span>]</span>,</span> [</span>9</span>,</span> 25</span>]</span>,</span> [</span>10</span>,</span> 13</span>]</span>,</span> [</span>11</span>,</span> 19</span>]</span>,</span> [</span>12</span>,</span> 14</span>]</span>,</span> [</span>13</span>,</span> 22</span>]</span>,</span> [</span>14</span>,</span> 24</span>]</span>,</span> [</span>15</span>,</span> 7</span>]</span>,</span> [</span>16</span>,</span> 23</span>]</span>,</span> [</span>17</span>,</span> 20</span>]</span>,</span> [</span>18</span>,</span> 18</span>]</span>,</span> [</span>19</span>,</span> 15</span>]</span>,</span> [</span>20</span>,</span> 0</span>]</span>,</span> [</span>21</span>,</span> 8</span>]</span>,</span> [</span>22</span>,</span> 1</span>]</span>,</span> [</span>23</span>,</span> 17</span>]</span>,</span> [</span>24</span>,</span> 2</span>]</span>,</span> [</span>25</span>,</span> 9</span>]</span>]</span>,</span></span> 3</span> 2</span>:</span> [</span>[</span>0</span>,</span> 0</span>]</span>,</span> [</span>1</span>,</span> 9</span>]</span>,</span> [</span>2</span>,</span> 3</span>]</span>,</span> [</span>3</span>,</span> 10</span>]</span>,</span> [</span>4</span>,</span> 18</span>]</span>,</span> [</span>5</span>,</span> 8</span>]</span>,</span> [</span>6</span>,</span> 17</span>]</span>,</span> [</span>7</span>,</span> 20</span>]</span>,</span> [</span>8</span>,</span> 23</span>]</span>,</span> [</span>9</span>,</span> 1</span>]</span>,</span> [</span>10</span>,</span> 11</span>]</span>,</span> [</span>11</span>,</span> 7</span>]</span>,</span> [</span>12</span>,</span> 22</span>]</span>,</span> [</span>13</span>,</span> 19</span>]</span>,</span> [</span>14</span>,</span> 12</span>]</span>,</span> [</span>15</span>,</span> 2</span>]</span>,</span> [</span>16</span>,</span> 16</span>]</span>,</span> [</span>17</span>,</span> 6</span>]</span>,</span> [</span>18</span>,</span> 25</span>]</span>,</span> [</span>19</span>,</span> 13</span>]</span>,</span> [</span>20</span>,</span> 15</span>]</span>,</span> [</span>21</span>,</span> 24</span>]</span>,</span> [</span>22</span>,</span> 5</span>]</span>,</span> [</span>23</span>,</span> 21</span>]</span>,</span> [</span>24</span>,</span> 14</span>]</span>,</span> [</span>25</span>,</span> 4</span>]</span>]</span>,</span></span> 4</span> 3</span>:</span> [</span>[</span>0</span>,</span> 1</span>]</span>,</span> [</span>1</span>,</span> 3</span>]</span>,</span> [</span>2</span>,</span> 5</span>]</span>,</span> [</span>3</span>,</span> 7</span>]</span>,</span> [</span>4</span>,</span> 9</span>]</span>,</span> [</span>5</span>,</span> 11</span>]</span>,</span> [</span>6</span>,</span> 2</span>]</span>,</span> [</span>7</span>,</span> 15</span>]</span>,</span> [</span>8</span>,</span> 17</span>]</span>,</span> [</span>9</span>,</span> 19</span>]</span>,</span> [</span>10</span>,</span> 23</span>]</span>,</span> [</span>11</span>,</span> 21</span>]</span>,</span> [</span>12</span>,</span> 25</span>]</span>,</span> [</span>13</span>,</span> 13</span>]</span>,</span> [</span>14</span>,</span> 24</span>]</span>,</span> [</span>15</span>,</span> 4</span>]</span>,</span> [</span>16</span>,</span> 8</span>]</span>,</span> [</span>17</span>,</span> 22</span>]</span>,</span> [</span>18</span>,</span> 6</span>]</span>,</span> [</span>19</span>,</span> 0</span>]</span>,</span> [</span>20</span>,</span> 10</span>]</span>,</span> [</span>21</span>,</span> 12</span>]</span>,</span> [</span>22</span>,</span> 20</span>]</span>,</span> [</span>23</span>,</span> 18</span>]</span>,</span> [</span>24</span>,</span> 16</span>]</span>,</span> [</span>25</span>,</span> 14</span>]</span>]</span>,</span></span> 5</span> 4</span>:</span> [</span>[</span>0</span>,</span> 4</span>]</span>,</span> [</span>1</span>,</span> 18</span>]</span>,</span> [</span>2</span>,</span> 14</span>]</span>,</span> [</span>3</span>,</span> 21</span>]</span>,</span> [</span>4</span>,</span> 15</span>]</span>,</span> [</span>5</span>,</span> 25</span>]</span>,</span> [</span>6</span>,</span> 9</span>]</span>,</span> [</span>7</span>,</span> 0</span>]</span>,</span> [</span>8</span>,</span> 24</span>]</span>,</span> [</span>9</span>,</span> 16</span>]</span>,</span> [</span>10</span>,</span> 20</span>]</span>,</span> [</span>11</span>,</span> 8</span>]</span>,</span> [</span>12</span>,</span> 17</span>]</span>,</span> [</span>13</span>,</span> 7</span>]</span>,</span> [</span>14</span>,</span> 23</span>]</span>,</span> [</span>15</span>,</span> 11</span>]</span>,</span> [</span>16</span>,</span> 13</span>]</span>,</span> [</span>17</span>,</span> 5</span>]</span>,</span> [</span>18</span>,</span> 19</span>]</span>,</span> [</span>19</span>,</span> 6</span>]</span>,</span> [</span>20</span>,</span> 10</span>]</span>,</span> [</span>21</span>,</span> 3</span>]</span>,</span> [</span>22</span>,</span> 2</span>]</span>,</span> [</span>23</span>,</span> 12</span>]</span>,</span> [</span>24</span>,</span> 22</span>]</span>,</span> [</span>25</span>,</span> 1</span>]</span>]</span>,</span></span> 6</span> 5</span>:</span> [</span>[</span>0</span>,</span> 21</span>]</span>,</span> [</span>1</span>,</span> 25</span>]</span>,</span> [</span>2</span>,</span> 1</span>]</span>,</span> [</span>3</span>,</span> 17</span>]</span>,</span> [</span>4</span>,</span> 6</span>]</span>,</span> [</span>5</span>,</span> 8</span>]</span>,</span> [</span>6</span>,</span> 19</span>]</span>,</span> [</span>7</span>,</span> 24</span>]</span>,</span> [</span>8</span>,</span> 20</span>]</span>,</span> [</span>9</span>,</span> 15</span>]</span>,</span> [</span>10</span>,</span> 18</span>]</span>,</span> [</span>11</span>,</span> 3</span>]</span>,</span> [</span>12</span>,</span> 13</span>]</span>,</span> [</span>13</span>,</span> 7</span>]</span>,</span> [</span>14</span>,</span> 11</span>]</span>,</span> [</span>15</span>,</span> 23</span>]</span>,</span> [</span>16</span>,</span> 0</span>]</span>,</span> [</span>17</span>,</span> 22</span>]</span>,</span> [</span>18</span>,</span> 12</span>]</span>,</span> [</span>19</span>,</span> 9</span>]</span>,</span> [</span>20</span>,</span> 16</span>]</span>,</span> [</span>21</span>,</span> 14</span>]</span>,</span> [</span>22</span>,</span> 5</span>]</span>,</span> [</span>23</span>,</span> 4</span>]</span>,</span> [</span>24</span>,</span> 2</span>]</span>,</span> [</span>25</span>,</span> 10</span>]</span>]</span></span> 7</span>}</span></span></code></pre> The plugboard is only a hash map switching two arbitrary letters. The Enigma class just used these inputs as settings and passed each character passed to to the plugboard, the reverse rotor, stepping it, the rotor 1 to 3, stepping them and back to the reverse roter.</p> Due to the reverse rotor, the Enigma is symmetrical in operation. This means that users do not have to distinguish between cyphering and deciphering inputs. Using it looks something like this:</p> 1</span>></span> pyenigma.py</span></span> 2</span>Provide</span> a</span> strimg</span> to</span> encrypt/decrypt:</span> dont</span> panic</span> </span></span> 3</span>Encryped:</span> kihdqqocv</span></span></code></pre> You can checkout the code on my GitHub</a>. Disclaimer: This is a quick and dirty implementation I did in collage. Its NOT production grade!</em> The next step will be cracking it.</p> See you!</strong></p> How to send OnPrem Prometheus metrics to MS Azure 2023-05-25T00:00:00+00:00 Sending metrics to Azure - with Prometheus</h1> With this post, I want to provide a quick demonstration on how to send Prometheus metrics to Azure managed Prometheus. Let’s get stared.</p> What is Prometheus</h2> Prometheus is an open-source monitoring system that scrapes (pulls) system and application metrics from supported targets. It is an graduated project of the Cloud Native Computing Foundation (CNCF) and the de facto standard in the Kubernetes world. It also supports service discovery and alerting. The major drawback of Prometheus is, that it stores it’s metrics in block storage as local files and has no native support for clustering multiple instances. Horizontal scaling is therefore difficult to archive.</p> Why send metrics to Azure</h2> Cloud services as Azure and Grafana Cloud solve the clustering and storage issues of a single Prometheus instance by providing managed services. (Often implemented through Mimir</a>.) Companies can send metrics from multiple Prometheus instances and different locations, all to a single location, which minimizes duplicated setups (cost), the risk of a singe point of failure and provides scaling, while also providing a single location (called a single pane of glass) for service performance data. When using Azure, Prometheus also integrates nicely with AKS and other Azure services, even without the need of credentials due to service principles. Prometheus supports sending data to a different location using the remote_write</code> feature, which I will use here to sent data from any system and app to Azure.</p> Setting up Azure Prometheus</h2> For simplicity, I will demonstrate the set up process using the Azure Portal. But the process can also be automated via the CLI or terraform. You have to do the following steps:</p> Log into Azure</li> Search for Prometheus</li> Start Creating a Prometheus instance Set a name and location</li> Allow public networking</li> Create and wait</li> </ul> </li> </ul> This will create a new managed monitor resource-group with your Prometheus instance. Write down the Metrics ingestion endpoint</em></p> Next step is to create an App/User that can push to that endpoint. For this we:</p> Open the Azure Active Directory</li> Go to App Registrations</li> Click New Registration Provide a name</li> Choose single tenant mode</li> </ul> </li> Go to Certificates & Secrets</li> </ul> For simplicity we use a secret in this demo, but a certificate is safer and recommended. It is best practice to store the certificate in a vault (like Azure Keyvault or Hashicorp Vault) and only request the credentials via the vault’s API. A guide how to do this can be found here</a>.</p> Create a secret and copy it (you will not be able to see it again). Go back to the overview of the created app registration. Copy the ClientId and then press on Endpoints</em> to copy the OAuth 2.0 token endpoint (v2)</em> URL.</p> Next, we need to authorize our user to push metrics to the Metrics ingestion endpoint</em>. Go back to the created resource-groups for the Prometheus instance and go to Access Control (IMA)</em>. On the role-assignment tab, add a new role-assignment with the role of Monitoring Metrics Publisher</em> and assign it to the created user from the previous step. </p> This is it for the azure site!</p> Sending Metrics</h2> Now we can set up our local tools to send metrics.</p> The flow we created uses the OpenIdConnect (OIDC) standard which is part of the Oauth2 spec. When you want lo learn more abut the described authentication, you can check out this in depth article TBA</strong>.</p> To send metrics form on Prometheus instance we need to set up the Prometheus config like this:</p> 1</span>g</span>lobal</span>:</span></span> 2</span> s</span>crape_interval</span>:</span> 1</span>m</span></span> 3</span></span> 4</span>s</span>crape_configs</span>:</span></span> 5</span> -</span> j</span>ob_name</span>:</span> "</span>INTERNAL</span>"</span></span> 6</span> h</span>onor_labels</span>:</span> true</span></span> 7</span> s</span>tatic_configs</span>:</span></span> 8</span> -</span> t</span>argets</span>:</span> [</span>"</span>localhost:9090</span>"</span>]</span></span> 9</span></span> 10</span>r</span>emote_write</span>:</span></span> 11</span> -</span> n</span>ame</span>:</span> a</span>zure-write</span></span> 12</span> u</span>rl</span>:</span> <</span>Metrics ingestion endpoint></span></span> 13</span> h</span>eaders</span>:</span></span> 14</span> X</span>-Scope-OrgID</span>:</span> a</span>nonymous</span></span> 15</span> o</span>auth2</span>:</span></span> 16</span> c</span>lient_id</span>:</span> <</span>ClientId></span></span> 17</span> c</span>lient_secret</span>:</span> <</span>ClientSecret></span></span> 18</span> t</span>oken_url</span>:</span> <</span>OAuth 2.0 token endpoint (v2)></span></span> 19</span> s</span>copes</span>:</span> [</span> h</span>ttps://monitor.azure.com/.default</span> ]</span></span></code></pre> Thats it.</p> Start your Prometheus instance and see the logs that are coming in. You can visualize them in Azure Monitor or via the Azure Managed Grafana.</p> See you!</strong></p> The Notebook for Developers - now even better 2023-03-06T00:00:00+00:00 Joplin: The notebook for Devs</h1> How do you keep your digital life organized? How do you store and structure your knowledge in the era of information (overload)? I struggled for sooo long to find a good solution for these problems, but all the systems I tried, every app I used, felt off</em>. They didn’t let me do the things I wanted to, were slow to operate or were just too costly. I almost gave up and went back to analog notes and a mix of digital short notes/Google-Notes - until a colleague told me about Joplin. Right from the beginning, I loved the content-focused approach and the lack of the corporate business stuff that wanted to sell me things. And the best thing is: It’s extendable - so I extended it.</p> With this article, I want to give an overview of Joplin, what differentiates it from other notebook apps and the necessary information to decide if it is a tool fitting for your workflow.</p> </p> What is Joplin?</h2> Joplin is a cross-platform notebook app (Windows, Linux, macOS, iOS, Android) that is entirely based on Markdown. If you are not comfortable with Markdown, read no further - Joplin is not for you. But if you like Markdown because of its easy formatting and capabilities, use it regularly in ReadMe.md’s or GitHub/GitLab Issues, Joplin might be just the right fit.</p> Its content-focused approach is its main differentiator between other apps. There is no need to switch between heading</em> formatting and normal</em> formatting, and there is almost no need to use the mouse. This approach enables users to work incredibly fast and get their thoughts written down. From personal experience, a lot of other tools do way too much to make the content pretty and flexible instead of focusing on the actual content. More on this later. When the time comes to make it pretty, Joplin allows users to export their notes in almost every imaginable format, including PDF and HTML.</p> Another differentiator is that Joplin is open-source software. Everyone can see the sources, and personal data keeps being personal. There are no Upgrade here</em>, Premium there</em> popups, it is just focused on content. There is no big corporation behind Joplin that wants to push their subscription model. Of course, there is the option to spend money, and users can also opt-in to JoplinCloud (between $17 and $80 annually), but this is purely optional. As you might expect, you can sync your notes with Joplin Cloud, but this is not only possible with JoplinCloud. It can also be done with OneDrive, Dropbox, GoogleDrive, and NextCloud. A quick NextCloud setup allows users to store as many notes as they want. End-to-End encryption is also supported.</p> Comparison to Other Apps</h2> As I mentioned earlier, I have tried some other notebook apps, including Notion, OneNote, Evernote, and Google Notes.</p> On Evernote:</strong> Regarding Evernote, I used it way back when I was in school, so things might have changed since then. First of all, you need</strong> an account. Even when you have an account, there are artificial limitations such as a maximum upload quota of 60MB per month, a maximum of two synced devices, and no offline access to notes. Also, there is currently no Linux client, although it is in early beta now. I never got up to pace with Evernote. From my experience, it tried to solve too many tasks and fit every use case, but did not solve one thing well or fit perfectly. I was even willing to pay to try their premium features, but since they declined my student discount and I was not willing to pay $13 monthly, I turned my back on Evernote and never looked back. Taking notes and using basic features should not cost more than Spotify or Netflix.</p> Notion:</strong> As for Notion, I cannot say too many negative things about it. It just did not fit my requirements. I wanted static, easily writable notes that I can organize and search. Notion is much more dynamic and even has dynamic project boards, agendas, and timesheets. Although I think there are other services that execute project management better than Notion, I know for a fact that there are thousands of people out there who love Notion. However, they have to fix their app - copy and paste is beyond broken.</p> Others:</strong> I also tried OneNote, DesktopNotes, and Google Notes. At a certain amount of notes, the latter tools are not capable of staying organized and are too limiting. For a quick note I need to remember, sure, but nothing more. OneNote is fundamentally a good app, but Microsoft screwed up with all these different versions of OneNote (Office OneNote, OneNote WPF, OneNote Online). I still use it if I have to create technical drawings with my tablet, but writing and formatting longer texts - ain’t gonna happen.</p> On Joplin</h2> As stated before, Joplin is a cross-platform notebook app that is entirely focused on notes. There are no fancy agendas, drawing or advanced formatting support. It supports all the major Markdown features, including headings, bold and italic formatting, enumerations, and tables. Images can be included via drag and drop or Markdown image includes. Math mode is also supported. It is perfect for my workflows because I already write all my docs and notes in Markdown, so I’m quite fast. It is also easy to import text because most of the tools I use have Markdown support. It is easy to copy and paste between Github and Joplin. Even entire websites (or portions of them) can be imported (and converted to Markdown) via a web-clipper. In case users want to stop using Joplin, they can easily export their notes in various formats. But, as I mentioned above, formatting options are limited, and everything that goes beyond the mentioned methods has to be done via HTML or via Plugins. That’s right, Joplin has a significant number of plugins which can do all kinds of stuff. Users can add support for draw.io diagrams, daily agendas, calendar tabs, and much more. It is also considerably easy to build a plugin, so I did it myself to add a feature I really wanted to make Joplin my central knowledge solution.</p> My Plugin</h2> Why:</strong> I think almost everyone stumbled at least once upon one of these Awesome lists of _</em>. Notable mentions are awesome open-source alternatives to SaaS</a>, fucking-awesome-python</a> or awesome-infosec</a>. Sure, you can copy/clip them, but then you have a static version, and you don’t get the newest additions and updates. Realistically, no one regularly visits all of these cheat sheets and updates them. So I wrote a plugin for that: joplin-plugin-remote-note-pull</a>. It lets users create new notes from any publicly available website and regularly updates the note with the original upstream version.</p> How:</strong> I started with a plugin skeleton. Joplin even provides a plugin-generator</a>, which gets you set up. The docs</a> provide all the other info one might need. Plugins are written in JavaScript/TypeScript and get executed via the NodeJs runtime Joplin is built on. Developers can add new UI elements or register new functions. Basically my plugin just does is make a web-request to the newly added note and download the content. Before storing it as a new note, it fixes some relative links and paths in the gathered content and adds a custom footer to the note. Other than that, it is just a timer-function that runs every x minutes to update all notes created via my plugin. It was a fun project to play around with. Publishing plugins is also relatively easy; they just have to follow a specific naming schema and get automatically picked up by Joplin’s internal plugin browser.</p> For now, I’m quite happy with this workflow and wished I had it earlier. If anyone is interested in sharing their favorite way to stay organized in this world of information overload, feel free to reach out.</p> See you!</strong></p> Broken NodeJS Apps due to secerity dot-release - Debugging the NodeJS CVE-2022-32213 fix 2022-08-18T00:00:00+00:00 Broken NodeJS Apps due to security dot-release</h1> On July 7th the vulnerabilities CVE-2022-32213</a>, CVE-2022-32214</a> and CVE-2022-32215</a> where publicly disclosed. They affected all current NodeJS versions (v14.x, v16.x, v18.x)! The same day fixes for the vulnerabilities where released. My colleagues and I assumed a quick and easy deployment to spit these fixes onto our client’s production systems. A simple rebuild of the old code state with the new NodeJS version and replacement of the old containers. We thought wrong! 🙃</p> Our staging system showed that some of our apps didn’t get any requests anymore and clients got a HTTP 4xx error code. </p> Some Background Information</h3> For a better understanding let me give a simplified overview of our system architecture: </p> We use a reverse proxy which is the only service directly exposed to the internet. It handles TLS termination and routes the traffic to the appropriate backend service. Some services require TLS client verification (mTLS</a>), meaning the client has to send its unique certificate to the server where it gets verified and passed to the backend service if the certificate was valid.</p> We triggered a new build with the updated NodeJS version (in our case form NodeJS v14.19.3 to v14.20.0) and deployed it to our testing system. The deployment went smooth, but then we saw that every service using client verification was broken after the update.</p> Debugging time</h3> To verify the bug I deployed the same code with the two different NodeJS versions (v14.19.3 & v14.20.0). On v14.19.3 I got a HTTP 200 response, on v14.20.0 I got a HTTP 400 response indicating a bad-request. But our application never threw a 400 error - nor did it log anything. - Time to dig deeper…</em></p> I created a minimal NodeJS http-server and deployed it with the two different NodeJS versions (and even new ones</em>):</p> 1</span>const</span> port</span> =</span> 3000</span>;</span></span> 2</span>var</span> http</span> =</span> require</span>(</span>'</span>http</span>'</span>)</span>;</span></span> 3</span></span> 4</span>//</span>Dead simple http server:</span></span> 5</span>http</span>.</span>createServer</span>(</span>(</span>req</span>,</span> res</span>)</span> =></span> {</span></span> 6</span> res</span>.</span>setHeader</span>(</span>'</span>Content-Type</span>'</span>,</span> '</span>application/json</span>'</span>)</span>;</span></span> 7</span> res</span>.</span>write</span>(</span>JSON</span>.</span>stringify</span>(</span>req</span>.</span>headers</span>)</span>)</span>;</span></span> 8</span> console</span>.</span>log</span>(</span>`</span>REQ FROM: path: </span>${</span>req</span>.</span>url</span>}</span>; remote: </span>${</span>req</span>.</span>ip</span>}</span>`</span>)</span>;</span></span> 9</span> console</span>.</span>log</span>(</span>req</span>.</span>headers</span>)</span>;</span></span> 10</span> res</span>.</span>end</span>(</span>)</span>;</span></span> 11</span>}</span>)</span>.</span>listen</span>(</span>port</span>,</span> '</span>0.0.0.0</span>'</span>)</span>;</span></span></code></pre> I got the same behavior 😒. Not a solution but now I knew it wasn’t a bug in our application. Is it a bug in NodeJS? In a dot-release? For security fixes?</p> Let’s start by looking at the release notes of NodeJS 14.20.0</a></p> 1</span>###</span> Notable Changes</span></span> 2</span>[</span>8e8aef836c</span>]</span> - (SEMVER-MAJOR) src,deps,build,test: add OpenSSL config appname (Daniel Bevenius) #43124</span></span> 3</span>[</span>98965b137d</span>]</span> - deps: upgrade openssl sources to 1.1.1q (RafaelGSS) #43686</span></span> 4</span>###</span> Commits</span></span> 5</span>[</span>b93e048bf6</span>]</span> - deps: update archs files for OpenSSL-1.1.1q (RafaelGSS) #43686</span></span> 6</span>[</span>98965b137d</span>]</span> - deps: upgrade openssl sources to 1.1.1q (RafaelGSS) #43686</span></span> 7</span>[</span>837a1d803e</span>]</span> - deps: update archs files for OpenSSL-1.1.1p (RafaelGSS) #43527</span></span> 8</span>[</span>c5d9c9a49e</span>]</span> - deps: upgrade openssl sources to 1.1.1p (RafaelGSS) #43527</span></span> 9</span>[</span>da0fda0fe8</span>]</span> - http: stricter Transfer-Encoding and header separator parsing (Paolo Insogna) #315</span></span> 10</span>[</span>48c5aa5cab</span>]</span> - src: fix IPv4 validation in inspector_socket (Tobias Nießen) nodejs-private/node-private#320</span></span> 11</span>[</span>8e8aef836c</span>]</span> - (SEMVER-MAJOR) src,deps,build,test: add OpenSSL config appname (Daniel Bevenius) #43124</span></span></code></pre> Hmmm - The certificates are generated with openssl</em>, but the actual verification takes place before it reaches NodeJS. The only other notable thing is the stricter http parser. Let’s take a deeper look.</p> Encoding is everything</h3> I want to see the actual HTTP request that gets proxieed to NodeJS. So I replaced the NodeJS application with Netcat – The Swiss Army Knife of Networking</a>. With the help of @bnoordhuis</a> I found that we were getting the following:</p> 1</span># HexDump</span></span> 2</span>00000000: 4745 5420 2f20 4854 5450 2f31 2e31 0d0a 436f 6e6e 6563 7469 6f6e 3a20 7570 GET / HTTP/1.1..Connection: up</span></span> 3</span>0000001e: 6772 6164 650d 0a48 6f73 743a 2064 6d2d 7465 7374 696e 672d 7374 6167 696e grade..Host: dm-testing-stagin</span></span> 4</span>0000003c: 672d 7369 7465 2e63 632d 6973 6f62 7573 2e63 6f6d 0d0a 4d59 5f43 4552 543a g-site.exampleio.com..MY_CERT:</span></span> 5</span>0000005a: 202d 2d2d 2d2d 4245 4749 4e20 4345 5254 4946 4943 4154 452d 2d2d 2d2d 0a09 -----BEGIN CERTIFICATE-----..</span></span> 6</span>00000078: 4d49 4947 4154 4343 412b 6d67 4177 4942 4167 4942 4154 414e 4267 6b71 686b MIIGATCCA+mgAwIBAgIBATANBgkqhk</span></span> 7</span>00000096: 6947 3977 3042 4151 7346 4144 4342 6d54 4561 4d42 6747 4131 5545 4177 7752 iG9w0BAQsFADCBmTEaMBgGA1UEAwwR</span></span> 8</span>000000b4: 5132 3974 0a09 6347 4675 6553 3168 6458 526f 6233 4a70 6448 6b78 437a 414a Q29t..cGFueS1hdXRob3JpdHkxCzAJ</span></span> 9</span>000000d2: 4267 4e56 4241 5954 416b 5246 4d52 5177 4567 5944 5651 5149 4441 744d 6233 BgNVBAYTAkRFMRQwEgYDVQQIDAtMb3</span></span> 10</span>...</span></span></code></pre> Do you see these 0a09</code> sequences? That is the encoding for \n\t</code> which is NOT</strong> a valid sequence for new-lines in HTTP-Headers. But NodeJS accepted these sequences anyway - at least till version v14.20.0.</p> The correct sequence, based on the protocol, should be 0a0d09</code> which is \r\n\t</code>.</p> The Solution:</h3> The solution is easy - just change the way the certificates are encoded! - Wait! We can’t do that!</em> Due to the nature of our service we don’t have direct control over the clients connecting to our server. It would be a major inconvenience to change this on all clients. Another option is to start NodeJS with the --insecure-http-parser</code> flag - which isn’t an option either. What about our reverse proxy? Turns out NGINX</a> has a dedicated variable for client certificates in its ssl-module</a> since version 1.13.5. Instead of ssl_client_cert</code> we could use ssl_client_escaped_cert</code> which contains the url-encoded (and therefore valid encoded) certificate. A quick test with the new proxy configuration showed that the new NodeJS version was now working.</p> Some Background Information</h3> Planned was the update from NodeJS v14.19.3 to v14.20.0. The above-mentioned CVE’s are about the llhttp</em> http-parser of NodeJS. It is part of the core of NodeJS and handles http requests before</em> any custom JavaScript executed. The security release made the http parser stricter to only accept requests that follow exactly the HTTP protocol standard. The new behavior prevents attackers to use request smuggling.</p> What we learned & what we did</h3> Unfortunately deploying these security fixes wasn’t as straight forward as we originally thought. We did not expect some of our application to just completely stop working. We needed some time to debug but also didn’t want to expose the other working</em> services to unnecessary risk. So we updated every application that didn’t use client verification as soon as all our tests passed end kept back the update form those that did. Two and 1/2 days later we found a solution for the problem and also updated the application that used client verification. We learned about the importance of working within the protocol specification even though some software allows more than it has to. And even though JavaScript is a high level language, it is important to have understanding for the low level actions of the networking-stack. We also got remained that just because it is a dot-release it still can break a lot and case some quality time with your favorite rubber duck 🦆</p> See you!</strong></p> Productive Working - Be lazy and get things done - Part II 2022-02-12T00:00:00+00:00 Productive Working - Be lazy & get things done - Part II</h1> My math teacher always told me that mathematicians are lazy and so should we. Only do the minimal number of steps to reach your desired destination. Whether or not these steps are simple in themselves is another matter. But somehow this phrase stuck with me.</p> So here are my favorite shortcuts, scripts and tricks to get things done:</em> Personal Favorite:</strong> My favorite feature are the search shortcuts in the Browser ❤️ </p> Automate</h3> Are there tasks that you have to do over and over again? Like setting up your work environment? Start the same programs every time? Automate them!</strong> Some example:</p> I use SSH a lot</em>. To I created a script in my .bashrc</code> file to automaticly start the ssh-agent and add my private key. So I can run SSH commands from every environment:</p> 1</span>#</span> SSH-Agent setup</span></span> 2</span>env</span>=</span>~</span>/</span>.</span>s</span>s</span>h</span>/</span>a</span>g</span>e</span>n</span>t</span>.</span>e</span>n</span>v</span></span> 3</span>agent_load_env</span> (</span>)</span> {</span> test</span> -</span>f</span> "</span>$</span>env</span>"</span> &&</span> .</span> "</span>$</span>env</span>"</span> ></span>\|</span> /dev/null</span> ;</span> }</span></span> 4</span></span> 5</span>agent_start</span> (</span>)</span> {</span></span> 6</span> (</span>umask</span> 077</span>;</span> ssh-agent</span> ></span>\|</span> "</span>$</span>env</span>"</span>)</span></span> 7</span> .</span> "</span>$</span>env</span>"</span> ></span>\|</span> /dev/null</span> ;</span> }</span></span> 8</span></span> 9</span>agent_load_env</span></span> 10</span></span> 11</span>#</span> agent_run_state: 0=agent running w/ key; 1=agent w/o key; 2= agent not running</span></span> 12</span>agent_run_state</span>=</span>$(</span>ssh-add</span> -</span>l</span> ></span>\|</span> /dev/null</span> 2</span>>&1</span>;</span> echo</span> $</span>?</span>)</span></span> 13</span></span> 14</span>if</span> [</span> !</span> "</span>$</span>SSH_AUTH_SOCK</span>"</span> ]</span> \|</span>\|</span> [</span> $</span>agent_run_state</span> =</span> 2</span> ]</span>;</span> then</span></span> 15</span> agent_start</span></span> 16</span> ssh-add</span></span> 17</span>elif</span> [</span> "</span>$</span>SSH_AUTH_SOCK</span>"</span> ]</span> &&</span> [</span> $</span>agent_run_state</span> =</span> 1</span> ]</span>;</span> then</span></span> 18</span> ssh-add</span></span> 19</span>fi</span></span> 20</span>unset</span> env</span></span></code></pre> Working with many GIT repos and dont want to pull manually:</p> 1</span>pull_all</span>(</span>)</span> {</span></span> 2</span> FAILS</span>=</span>(</span>)</span></span> 3</span> workdir</span>=</span>$(</span>pwd</span>)</span></span> 4</span> for</span> REPO</span> in</span> </span>/</span>;</span> do</span></span> 5</span> if</span> [</span> !</span> -d</span> "</span>$</span>REPO</span>/.git</span>"</span> ]</span>;</span> then</span> continue</span>;</span> fi</span></span> 6</span> echo</span> -</span>e</span> "</span>$</span>{</span>GRN</span>}</span>Entering and performing GIT PULL on </span>$</span>{</span>REPO</span>}</span>$</span>{</span>NC</span>}</span>"</span></span> 7</span> cd</span> $</span>REPO</span> &&</span> bash</span> -</span>c</span> "</span>git pull --recurse-submodules</span>"</span></span> 8</span> if</span> [</span> $</span>?</span> -eq</span> 0</span> ]</span>;</span> then</span></span> 9</span> echo</span> -</span>e</span> "</span>$</span>{</span>GRN</span>}</span>Done</span>$</span>{</span>NC</span>}</span>"</span></span> 10</span> else</span></span> 11</span> FAILS</span>+=</span>(</span>$</span>REPO</span>)</span>;</span> echo</span> -</span>e</span> "</span>$</span>{</span>RED</span>}</span>Command faild</span>$</span>{</span>NC</span>}</span>"</span></span> 12</span> fi</span></span> 13</span> cd</span> $</span>workdir</span></span> 14</span> done</span></span> 15</span> #</span> Result</span></span> 16</span> if</span> [</span> $</span>{</span>#</span>FAILS</span>[</span>@</span>]</span>}</span> -eq</span> 0</span> ]</span>;</span> then</span></span> 17</span> echo</span> -</span>e</span> "</span>$</span>{</span>GRN</span>}</span>All commands sucseeded</span>$</span>{</span>NC</span>}</span>"</span></span> 18</span> else</span></span> 19</span> echo</span> -</span>e</span> "</span>$</span>{</span>RED</span>}</span>The following repos exited with an error:\n</span>$</span>{</span>FAILS</span>[</span>@</span>]</span>}</span>$</span>{</span>NC</span>}</span>"</span></span> 20</span> fi</span></span> 21</span>}</span></span></code></pre> Backup your Dotfiles:</p> 1</span>#</span> Create Dotfiles</span></span> 2</span>DOT_ARCHIVE</span>=</span>$</span>HOME</span>/</span>T</span>o</span>o</span>l</span>s</span>/</span>d</span>o</span>t</span>f</span>i</span>l</span>e</span>s</span>.</span>t</span>a</span>r</span>.</span>g</span>z</span></span> 3</span>if</span> [</span> !</span> -f</span> "</span>$</span>DOT_ARCHIVE</span>"</span> ]</span> \|</span>\|</span> [</span> $(</span>(</span>$(</span>date</span> +%s</span>)</span> - </span>$(</span>date</span> +%s</span> -</span>r</span> $</span>DOT_ARCHIVE</span>)</span>)</span>)</span> -gt</span> 604800</span> ]</span>;</span> then</span></span> 4</span> ORI_PWD</span>=</span>$(</span>pwd</span>)</span></span> 5</span> cd</span> $</span>HOME</span></span> 6</span> echo</span> -</span>n</span> "</span>Creating dofiles archive... </span>"</span></span> 7</span> tar</span> -</span>czf</span> $</span>DOT_ARCHIVE</span> -</span>-exclude=</span>{</span>'</span>.vscode</span>'</span>,</span>'</span>.git</span>'</span>,</span>'</span>..</span>'</span>,</span>'</span>./</span>'</span>}</span> $(</span>find</span> .</span> -</span>maxdepth</span> 1</span> -</span>name</span> "</span>.*</span>"</span> -</span>printf</span> '</span>%P\n</span>'</span>)</span> &&</span> \</span></span> 8</span> gpg</span> -</span>-yes</span> -</span>-batch</span> -</span>-passphrase=</span>$(</span>echo</span> -</span>n</span> $</span>DOTFILES_PW</span> \|</span> base64</span> -</span>-decode</span> )</span> -</span>o</span> $</span>DOT_ARCHIVE</span>.gpg</span> -</span>c</span> $</span>DOT_ARCHIVE</span> &&</span> \</span></span> 9</span> echo</span> "</span>done</span>"</span></span> 10</span> cd</span> $</span>ORI_PWD</span></span> 11</span>fi</span></span></code></pre> There are soo many more!</p> Choose your tools</h2> Take your time to find the appropriate tool for the job. And then learn it until you can use it in your sleep. Choose the editor and mail client you like and get really good at it. The same goes for your browser. All chromium based browsers support search-shortcuts in the addressbar of the browser. So you can directly search YouTube instead of googling YT, clicking the first link and then start searching form there. This saves more than 3 clicks!!!</strong> </p> Shortcuts!!!1!</h2> Not having to leave the keyboard to reach the mouse in order to click a button that is hidden behind two menues gives you a huge</strong> advantage over others. It lets you archive your goal quicker so you can focus on the next step. There are some universall shortcuts that work in almost every application that everybody shoud know:</p> Common:</em></p> 1</span>#</span> Copy # Print # Search # Swicht between windows </span></span> 2</span>Ctrl</span> +</span> c</span> Ctrl</span> +</span> p</span> Win</span> +</span> SeachWord</span> Alt</span> +</span> tab</span> </span></span> 3</span>#</span> Past # Open # Desktop # Open app x in taskbar </span></span> 4</span>Ctrl</span> +</span> v</span> Ctrl</span> +</span> o</span> Win</span> +</span> d</span> Win</span> +</span> [0-9</span>] </span></span> 5</span>#</span> Cut # New document # Snap Window # Duplicate/Extend monitor </span></span> 6</span>Ctrl</span> +</span> x</span> Ctrl</span> +</span> n</span> Win</span> +</span> any-arrow</span> Win</span> +</span> p</span> </span></span> 7</span>#</span> Search # New tab # Open explorer # Open settings </span></span> 8</span>Ctrl</span> +</span> f</span> Ctrl</span> +</span> t</span> Win</span> +</span> e</span> Win</span> +</span> i</span> </span></span> 9</span>#</span> Undo # close tab # Lock screen # Screen Record </span></span> 10</span>Ctrl</span> +</span> z</span> Ctrl</span> +</span> w</span> Win</span> +</span> l</span> Win</span> +</span> g</span> </span></span> 11</span>#</span> Redo # Select all # Emoj #Paste with history clipbord </span></span> 12</span>Ctrl</span> +</span> y</span> Ctrl</span> +</span> a</span> Win</span> +</span> .</span> Win</span> +</span> v</span> </span></span> 13</span>#</span> Save # Mark wohle word # Open Screenshot </span></span> 14</span>Ctrl</span> +</span> s</span> Ctrl</span> +</span> Shift</span> +</span> arrow</span> Ctrl</span> +</span> o</span> Win</span> +</span> Shift</span> +</span> s</span> </span></span></code></pre> Browser:</em></p> 1</span>#</span> Reload # Open history # Switch tabs # Go to tab [1-9] </span></span> 2</span>Ctrl</span> +</span> r</span> Ctrl</span> +</span> h</span> Ctrl</span> +</span> tab</span> Ctrl</span> +</span> [0-9</span>] </span></span> 3</span>#</span> Add favo # Focuse searchbar # Reopen tab # Zoom </span></span> 4</span>Ctrl</span> +</span> d</span> Ctrl</span> +</span> e</span> Ctrl</span> +</span> Shift</span> +</span> t</span> Ctrl</span> +</span> Shift</span> +</span> +/-</span> </span></span> 5</span>#</span> Go back # go forward </span></span> 6</span>alt</span> +</span> left-arrow</span> alt</span> +</span> right-arrow</span> </span></span></code></pre> VSCode:</em> </p> Note:</strong> I also created a collection with my favorite shortcuts, scripts and tricks to work efficient as a programmer. Read it here</a></p> See you!</strong></p> Productive Working - Be lazy and get things done - Part I 2022-02-09T00:00:00+00:00 Productive Working - Be lazy & get things done - Part I</h1> My math teacher always told me that mathematicians are lazy and so should we. Only do the minimal number of steps to reach your desired destination. Whether or not these steps are simple in themselves is another matter. But somehow this phrase stuck with me. As I grow older I measured more value to my time. Doing my obligatory tasks efficiently gave me more time that I could spend on hobbies. So developed certain principles to get my stuff done, some of which I want to share today. </p> Overall Principles</h2> Disclaimer:</em> The following practices have been developed with an IT background and may not be applicable to everyone. However, I have tried to make the introductory practices as general as possible so that they can be used by as many people as possible.</p> Time Managment</h3> To increase our efficiency we need to measure the amount of tasks we can complete in a certain amount of time. Split the amount of time you have in equaly sized chunks like 15mins. Prioritize your tasks according to their importance and duration. If you complete 3-4 small tasks within the first 15mins of allocated time, the amount of overall work you will have to complete looks a lot smaller, and you are more motivated. This can be things like teaching garbage, putting away clothes or taking away a letter/package. For work allocate one or two chunks of our time for replaying to emails. Collect the mails over the day and replay to them on one session, don’t</strong> let incoming mails distract you from your current task. Finish your current tasks and then start something new. Peaking at the mail or your phone will only take a second, but it will take you over 1.5 minutes to fully mentally switch back to the previous task. A wonderful article from Johnn Hari in The Gurdian</a> diecribes how our all attention span decreased how expensive</em> these context switches between tasks are. Tasks can be completed a lot quicker when you fully concentrate on one thing at a time. When the amount of tasks becomes too much to sort in your head write them down or maybe even visualize them with a method like Gnatt diagrams: These diagrams allow to visually lay out your tasks and identify blocking tasks. These are tasks that have to be done before others can start. Before you can send a package you have to pack it first. When you wait for certain things to finish you can complete other stuff. For example while waiting for your oven to heat up your food you can clean and store the dishes from last day. Combining things like putting your cloths in the washing machine while heading downstairs to buy groceries allows you to complete multiple things in one run.</p> Take breaks</h3> Keeping up with all the tasks of today’s life is exhausting. So you need time to recharge. Keep this in mind when planing you day and allocate some breaks. When taking a break really stop what you were working on, stand up and do something that lets you relax for a little time. Don’t feel bad for taking these breaks, they are in your schedule and there letting you brain relax a little. If you are disturbed by others tell them that you are having a break right now and will come back to them in 10 min.</p> Recurring Tasks</h2> Routine</h3> Certain tasks keep recurring and need to be done over and over again. Plan these tasks in advance, do not be surprised by them and try to develop regular processes so that they become a routine. Once these tasks have entered the daily routine, you can do them in your sleep. If you notice that the current task exhausts you, or you get stuck, you can take a break and while going for a walk you can complete some of these recurring tasks. The brain can relax during this time and you don’t feel guilty for interrupting your current task, because you are doing other tasks. These breaks may help you to find a new point of view to the previous task. More than once the visit to the restroom or quick shower gave me a new approach for autually solving my previos task 😁</p> Note:</strong> I also created a collection with my favorite shortcuts, scripts and tricks to work efficient as a programmer. Read it here</a></p> See you!</strong></p> LaTeX - A getting started for Devs 2022-01-22T00:00:00+00:00 Getting started with LaTex as a Dev</h1> About LaTeX</h2> LaTeX is a typesetting system that allows the user to write plain text, while the formatting is provided via markup tags, similar to HTML</code>. It differs from WYSIWYG (What You See Is What You Get</em>) applications like Word in that the final document must first be generated by a compiler. While the output of a LaTeX document is reproducible, minimal and just beautiful</em></strong>, it can be a challenge to get started for beginners. To get started users need to choose one of the multiple existing LaTeX distributions, choose an editor and setup a script to automatically build the document and preview it.</p> This article will automate and abstract most of the pain points and will let you focus on the content - like it is supposed by LaTeX. </p> Functions</h2> The setup provided by this article you will have the following features:</p> A quick, clean, working LaTeX environment setup based on TeXLive</li> Pandoc - To convert your projects between different formats</li> Have LaTeX Workshop an LaTeX Utilities preinstalled</li> GUI for math symbols and some TiKz</li> BibTeX support</li> Spell checking</li> Markdown support</li> Git support</li> Scientific project template</li> Have our environment on a remote computer</li> A platform independent setup that can be used on any OS (Win/Unix/macOS)</li> Have a setup that is faster than MiKTeX on Windows</li> Support for x86/arm/arm64 architectures</li> Persistent bash_history</li> </ul> Pre-Requirements</h2> This is a getting-started</em> guide focused on developers. Therefore, it will use tools common in the software developing world. Nevertheless, everyone can follow this guide to create a quick latex environment.</p> The following tools are needed for this guide:</p> VSCodoe</h3> VSCode is a cross-platform text editor designed for fast coding developed by Microsoft. It is heavily extendable with various extensions provided by a large community. It can be downloaded here</a></p> Docker</h3> Docker is a universal container runtime that allows users to run various software in a predefined environment isolated from the actual host system. Depending on your system you need one of the following:</p> Host</th> Version</th> Link</th></tr></thead> Linux</td> docker-ce</td> Download</a></td></tr> Windows</td> Docker Desktop</td> Download</a></td></tr> MacOS</td> Docker Desktop</td> Download</a></td></tr> </tbody></table> Note:</em> Docker builds upon functions provided by the Linux kernel. Windows and macOS need to virtualize the kernel. Therefore, the best performance is achievable on Linux.</p> Getting started</h2> To get started you have two options: Having the source files locally on host drive or in a persistent volume managed by Docker. If you pan to modify and access your project files with other tools as well I would suggest using the local setup. For slightly better disk performance or if you just want to use VSCode to edit your document you can also use a persistent volume managed by Docker.</p> Local setup</h3> The TeX source is on your host OS and gets mounted as volume</p> 1</span>#</span> Open a terminal an type:</span></span> 2</span>git</span> clone</span> https://github.com/hegerdes/VSCode-LaTeX-Container</span></span> 3</span>code</span> VSCode-LaTeX-Container</span></span> 4</span>#</span> In VSCode hit F1</span></span> 5</span>></span> Remote-Containers: Reopen in Container</span></span> 6</span>#</span> Wait for the initial pull and build</span></span> 7</span>#</span> Note: You need to have Docker and VSCode remote extentions installed</span></span> 8</span>#</span> Search for "ms-vscode-remote.vscode-remote-extensionpack"</span></span></code></pre>In a container</h3> The entire project is within the container</p> 1</span>#</span> Open VSCode</span></span> 2</span>#</span> In VSCode hit F1</span></span> 3</span>></span> Remote-Containers: Clone Repository in Container Volume</span></span> 4</span>></span> https://github.com/hegerdes/VSCode-LaTeX-Container</span></span> 5</span>#</span> Wait for the initial pull and build</span></span> 6</span>#</span> Note: You need to have Docker and VSCode remote extentions installed</span></span> 7</span>#</span> Search for "ms-vscode-remote.vscode-remote-extensionpack"</span></span></code></pre> 👉 Start typing you next big article!</p> Some background:</h2> The LaTeX template</h3> The included template was build up over the time and is designed for scientific projects. But I didn’t start from scratch either. Credit goes to:</p> 1</span>%</span> Original author:</span></span> 2</span>%</span> WikiBooks (LaTeX - Title Creation) with modifications by:</span></span> 3</span>%</span> Vel (vel@latextemplates.com)</span></span> 4</span>%</span> hegerdes (hegerdes@outlook.de)</span></span></code></pre>About the Docker image</h3> There are multiple base images debian-[bullseye\|buster (deprecated)] and ubuntu-[focal\|bionic (deprecated)]</strong>. All these images have texlive, texlive-latex-extra texlive-lang-english, texlive-luatex, texlive-xetex, texlive-pstricks, texlive-science, latexmk, cm-super, chktex</strong> with additional tools like git, zsh and pandoc(not in alpine)</strong> installed. Every image is available on x86/arm/arm64 architectures.</p> The slim images only contain texlive, texlive-latex-extra, texlive-lang-english, latexmk, cm-super, chktex</strong> If you want a minimal image use these, but this might lack common tools/packages.</p> There are two full images that contain everything</strong> in the LaTeX world except for docs. These are BIG</strong> and generally not recommended for fast startups.</p> There are a bunch of language specific images that are build up on the bullseye-base</strong> and focal-base</strong> images. Languages are: all, arabic, chinese, cjk, cyrillic, czechslovak, english, european, french, german, greek, italian, japanese, korean, other, polish, portuguese, spanish</strong>.</p> Use one of these if your work on a none English project! Simply change the VARIANT</code> arg in the devcontainer.json to bullseye-lang-<YOUR_LANGUAGE></code> or ubuntu-lang-<YOUR_LANGUAGE></code>.</p> I plan on updating these images every second month.</p> VSCode workspace</h3> I added a VSCode workspace file with sensible</em> setting. It includes some settings for Docker and the LaTeX extensions. Feel free to customize it after your own taste.</p> If you want to see what I did with the template look over to henrikgerdes.me/papers</a></p> Why all this</h2> I always liked the concept of LaTeX and its focus on content instead of the formatting. But getting started was hard and I wanted to contribute to make it a little more accessible. I first used MiKTeX and TeXworks, but I found the usage of shortcuts hard and didn’t like the PDF viewer. I switched to Notepad++, SumataPDF (both great tools) and a handy script. It was great until my projects got bigger. So I used VSCode and LaTeX Workshop and I loved it. All my shortcuts and tools I used before now applied to LaTeX. I was satisfied until I realized how slow MiKTeX on Windows is compared to Linux.</p> I love Linux, but some things are more convenient on Windows. I started my Bachelor theses about development environments and the usage of container tools. So I have further developed my setup to bring everything together.</p> I found that the tianon/latex</code> and other image were outdated and did not meet my expectations. I rather created my own image. It is own the large side, but I rather have all my tools there at any time instead of being slowed down by missing them or have to install packages manually. I hope some of you find some interesting tips and tricks in my setup.</p> If you find any issues let me know</a></p> See you!</strong></p> Reddit Video Downloader 2021-12-24T00:00:00+00:00 One Day Build: WebApp to download Videos from Reddit</h1> It has become kind of annoying to download a simple video from Reddit. My sister wanted to save a video but couldn’t do it. So, I did what all programmers do - I wrote a small app. You can try it out here</a> or use it via your terminal like described below</em>.</p> </p> Some Background</h2> The URLs of videos on Reddit are not directly visible. You have dig through some HTML tags and even more annoying, Reddit sores to video and audio files separably. So we need to find the URLs, download the content and combine video and audio. For this I wrote a small Python backend.</p> The Backend</h3> The backend requires a URL as an argument, and makes a request to that URL. The result is analyzed with BeautifulSoup</code></a>. After the video and audio URL got extracted, both files are downloaded separably and are then combined with ffmpeg</code></a>. I used Flask to handle the HTTP requests and provide routing. It is a simple-to-use minimal framework that was fast to implement. I wanted people to be able to use the backend directly, so it is possible to download videos with a terminal. The following bash function should download any Reddit video for you without the need to use any browser:</p> 1</span>#</span> Add this function to yor terminal session - or put it into your .bash_rc</span></span> 2</span>#</span> This need wget, curl and jq</span></span> 3</span>reddvid</span> (</span>)</span> {</span></span> 4</span> wget</span> https://reddvid.herokuapp.com/</span>$(</span>curl</span> -</span>X</span> GET</span> '</span>https://reddvid.herokuapp.com/</span>'</span> -</span>-form</span> "</span>url=</span>\"</span>$</span>{</span>1</span>}</span>\"</span>}</span>"</span> \|</span> jq</span> -</span>r</span> '</span>.download</span>'</span>)</span></span> 5</span>}</span></span> 6</span>#</span> Call the function</span></span> 7</span>reddvid</span> <</span>REDDIT_UR</span>L</span>></span></span></code></pre> Any non-technical person has the option to use the WebApp frontend.</p> The Frontend</h3> The WebApp is a simple VUE app that was build with nuxt.js, simply because I already know VUE and I like my WebApps to be a static page that can be served by any HTTP-Server. This allows for a broad range of deployment options.</p> Deployment</h3> The deployment strategy I chose was only based on tractability and cheep criteria. I am a student that did this project for fun and did not and couldn’t spend much on server infrastructure. The backend is deployed via a Python Docker Image</a> on Heroku</a> with a deployment option that goes to sleep after 30 min. The WebApp is hosted on Netlify</a> which is a cheap option for static apps that creates a new deployment for every commit.</p> Have Fun</h2> This is a project I build for my own usage and without much testing. I’m sure it has bugs and that it will beak on specific actions. Just use it if it is handy to you, share your thoughts, encountered issues and possible improvements. Swarm knowledge is the bast way to learn!!!</p> See you!</strong></p> Install & Configure Grafana Promtail 2021-12-21T00:00:00+00:00 Install & Configure Promtail for Grafana Loki</h1> This post will demonstrate a reliable and easy way to install Promtail to your server in order to collect all system logs in a central place for proper monitoring. NOTE:</em> This post will only cover the install and setup for Linux based systems</p> </p> About Promtail</h2> Promtail is part of the Grafana platform. The Grafana platform is increasing in popularity due to its open-source nature, easy usability and ability to plug into existing infrastructure. Grafana allows to visualize and query data from multiple sources. The data can be stored in InfluxDB</a>, ELK</a>, Prometheus</a> or the in-house Loki</a> data store. Loki is often called the Prometheus version of logs. But to query log-data in Loki over Grafana, the logs have to be shipped first. For Docker, this can easily be done with the Docker Loki Log-Driver</a>, but what about logs of none dockerized applications or logs of the host system? This is where Promtail comes in! It constantly reads log-files (e.g. in /var/log</code>) and sends them to a Loki instance.</p> Installing Promtail</h2> Promtail is available as a static binary on the official Grafana Loki GitHub</a> page. We could download it by hand, or</em> we could use the terminal like a real programmer:</p> 1</span>#</span> Download latest:</span></span> 2</span>curl</span> -</span>s</span> https://api.github.com/repos/grafana/loki/releases/latest</span> \|</span> \</span></span> 3</span> grep</span> browser_download_url</span> \|</span> \</span></span> 4</span> cut</span> -</span>d</span> '</span>"</span>'</span> -</span>f</span> 4</span> \|</span> grep</span> promtail-linux-amd64.zip</span> \|</span> \</span></span> 5</span> wget</span> -</span>i</span> -</span></span> 6</span>#</span> Unzip and</span></span> 7</span>unzip</span> promtail-linux-amd64.zip</span></span> 8</span>mv</span> promtail-linux-amd64</span> /usr/local/bin/promtail</span></span> 9</span>#</span> Verify</span></span> 10</span>promtail</span> -</span>-version</span></span> 11</span></span></code></pre> We are using the GitHub API to get the latest release of Promtail and are filtering for the linux-and64</code> binary with grep</code>. The result gets piped to wget</code> in order to download the zip file. Subsequently, we unzip the archive and move it to a place that is in our PATH</code>. Now we can test if Promtail is working.</p> Now we need a Promtail config:</p> 1</span>#</span> Promtail config</span></span> 2</span>s</span>erver</span>:</span></span> 3</span> d</span>isable</span>:</span> true</span></span> 4</span></span> 5</span>p</span>ositions</span>:</span></span> 6</span> f</span>ilename</span>:</span> /</span>tmp/promtail-positions.yaml</span></span> 7</span></span> 8</span>c</span>lients</span>:</span></span> 9</span> -</span> u</span>rl</span>:</span> h</span>ttp://<SERVER>:3100/loki/api/v1/push</span></span> 10</span></span> 11</span>s</span>crape_configs</span>:</span></span> 12</span>-</span> j</span>ob_name</span>:</span> a</span>uthlog</span></span> 13</span> s</span>tatic_configs</span>:</span></span> 14</span> -</span> t</span>argets</span>:</span></span> 15</span> -</span> l</span>ocalhost</span></span> 16</span> l</span>abels</span>:</span></span> 17</span> j</span>ob</span>:</span> a</span>uthlog</span></span> 18</span> s</span>hipper</span>:</span> p</span>romtail</span></span> 19</span> _</span>_path__</span>:</span> /</span>var/log/auth.log</span></span> 20</span></span> 21</span>-</span> j</span>ob_name</span>:</span> d</span>aemonlog</span></span> 22</span> s</span>tatic_configs</span>:</span></span> 23</span> -</span> t</span>argets</span>:</span></span> 24</span> -</span> l</span>ocalhost</span></span> 25</span> l</span>abels</span>:</span></span> 26</span> j</span>ob</span>:</span> s</span>yslog</span></span> 27</span> s</span>hipper</span>:</span> p</span>romtail</span></span> 28</span> _</span>_path__</span>:</span> /</span>var/log/daemon.log</span></span></code></pre> Our Promtail do not need to receive any logs we are just querying local files, so we can disable the server. The positions-file is a marker file for Promtail to remember where it stopped reading a log file, and the client is the destination of our logs. The scrape_configs</code> specifies what logs Promtail should send to Loki. In this example, we are using daemon.log</code> and auth.log</code>. There are tones of other possibilities, like access logs from ´nginx´ and alike, but this would be too much for this post.</p> We could start Promtail with promtail -config.file promtail-config.yml</code>, but we would have little control over the process, and it would not start again if the process fails or the host is restarted. So let’s create separate user for Promtail (optional) and a service:</p> 1</span>#</span> Create promtail user</span></span> 2</span>useradd</span> -</span>r</span> promtail</span></span> 3</span>usermod</span> -</span>a</span> -</span>G</span> adm</span> promtail</span></span> 4</span>#</span> Create a service</span></span> 5</span>tee</span> /etc/systemd/system/promtail.service</span><<</span>EOF</span> ></span> /dev/null</span></span> 6</span>[Unit]</span></span> 7</span>Description=Promtail service</span></span> 8</span>After=network.target</span></span> 9</span></span> 10</span>[Service]</span></span> 11</span>Type=simple</span></span> 12</span>User=promtail</span></span> 13</span>ExecStart=/usr/local/bin/promtail -config.file /path/to/promtail-config.yml</span></span> 14</span>Restart=always</span></span> 15</span></span> 16</span>[Install]</span></span> 17</span>WantedBy=multi-user.target</span></span> 18</span>EOF</span></span></code></pre> Now we can simply run the following and have a reliable and good Promtail setup that can easily be automated with tools like Ansible or Puppet.</p> 1</span>#</span> Start the service</span></span> 2</span>sudo</span> systemctl</span> daemon-reload</span></span> 3</span>sudo</span> systemctl</span> enable</span> promtail.service</span></span> 4</span>sudo</span> systemctl</span> start</span> promtail.service</span></span> 5</span>#</span> Verify status</span></span> 6</span>sudo</span> systemctl</span> status</span> promtail.service</span></span></code></pre> See you!</strong></p> Dockerfile for NodeJS & Typescript 2021-11-15T00:00:00+00:00 How to build a Docker image for a TypeScript based NodeJS project</h1> You just finished your TypeScript project and now you want to release it to the internet by using ECS, GKE, Azure or any other cloud service? Or maybe you just want to share it with your co-workers without having them to install and configure additional software. Docker is a great use case for this!</p> This post shows demonstrates how to create an optimized production ready Docker image for your project. </p> Pre-Requirements</h2> In order to build a Docker image you must have access to system that has the Docker engine installed. Depending on your system you need one of the following:</p> Host</th> Version</th> Link</th></tr></thead> Linux</td> docker-ce</td> Download</a></td></tr> Windows</td> Docker Desktop</td> Download</a></td></tr> MacOS</td> Docker Desktop</td> Download</a></td></tr> </tbody></table> Note:</em> Docker builds upon functions provided by the Linux kernel. Windows and macOS need to virtualize the kernel. Therefore, the best performance is achievable on linux.</p> Project setup</h2> Assuming the following file structure:</p> project/</span></span> - src/</span></span> - server.ts</span></span> - util.ts</span></span> - node_modules</span></span> - index.ts</span></span> - tsconfig.json</span></span> - package.json</span></span></code></pre> In order to run our app we need to:</p> Install our dependencies</li> Transpile TS to JS</li> Start our app</li> </ol> This needs to be specified in our Dockerfile which acts as the build instructions for our Docker image. We create a file named Dockerfile and add the following:</p> 1</span>FROM</span> node</span></span> 2</span></span> 3</span>WORKDIR</span> /app</span></span> 4</span>COPY</span> . /app</span></span> 5</span>RUN</span> npm install && npm build</span></span></code></pre> Our image is based on the official NodeJS Docker image. We copy our source code into the image and run the install and compile command.</p> This would work but is not an optimal solution. We are using Docker to get a consistent runtime that works everywhere but by not using a specific tag for our base image we are losing the consistence. If no tag is specified the latest</code> tag is used which changes often.</p> A lot of images include a lot of software that we actually do not need. These increases image size, storage consumption and network transfer times. It also increases to attack surface because any other program can have additional vulnerabilities. Let’s fix some of this:</p> 1</span>FROM</span> node:16-alpine</span></span> 2</span></span> 3</span>WORKDIR</span> /app</span></span> 4</span>COPY</span> package.json package-lock.json /app/</span></span> 5</span>RUN</span> npm install</span></span> 6</span></span> 7</span>COPY</span> . /app</span></span> 8</span>RUN</span> npm run build</span></span></code></pre> Now we are using an exact version and the much smaller alpine version of Nodes Docker image. We are also utilizing Dockers caching functionality. Docker only runs the commands that changed since the last run. Since we are not modifying the package.json</code> as often as our source code we can cache the install command and save time. This is better but not perfect yet. In order to run our app we neither need the TypeScript files nor the tsc</code> compiler.</p> This is where multi-stage build come handy. We can build our JavaScript in one stage and then copy the result to another stage that is actually used.</p> 1</span>FROM</span> node:16-alpine </span>as</span> build</span></span> 2</span></span> 3</span>WORKDIR</span> /app</span></span> 4</span>COPY</span> package.json package-lock.json /app/</span></span> 5</span>RUN</span> npm install</span></span> 6</span></span> 7</span>COPY</span> . /app</span></span> 8</span>RUN</span> npm run generate</span></span> 9</span></span> 10</span>FROM</span> node:16-alpine </span>as</span> production</span></span> 11</span></span> 12</span>USER</span> node</span></span> 13</span>WORKDIR</span> /app</span></span> 14</span>COPY</span> --chown=node:node --from=build /app/compiledJS /app</span></span> 15</span>RUN</span> npm install --only=production</span></span></code></pre> Now the code is compiled in one stage and shipped in another. We are also changing to user in order to omit running our app as root. Lower privileges are another step to increase your app’s security.</p> To increase the flexibility of our Dockerfile we can add arguments. With arguments, we can influence the building process on execution without modifying the Dockerfile. A common argument is the VARIANT</code> variable. Which allows to change the version of our base image.</p> The last step is to include an ENTRYPOINT</code> to specify what command should be executed when a container is created. I strongly recommend to directly call node index.js</code> instead of using npm start</code>. When using npm</code>, npm becomes our main process and acts as a process manager for the node process. Unfortunate npm</code> performs poorly as a process manager. It does not redirect any control or exit signals to the main node process. This can lead to a container not willing to stop or an unexpected shutdown of our node process without properly handling exit tasks.</p> The final Dockerfile should look something like this:</p> 1</span>FROM</span> node:16-alpine </span>as</span> build</span></span> 2</span></span> 3</span>WORKDIR</span> /app</span></span> 4</span>COPY</span> package.json package-lock.json /app/</span></span> 5</span>RUN</span> npm install</span></span> 6</span></span> 7</span>COPY</span> . /app</span></span> 8</span>RUN</span> npm run generate</span></span> 9</span></span> 10</span>ARG</span> VARIANT=16-alpine</span></span> 11</span>FROM</span> node:${VARIANT} </span>as</span> production</span></span> 12</span></span> 13</span>USER</span> node</span></span> 14</span>WORKDIR</span> /app</span></span> 15</span>COPY</span> --chown=node:node --from=build /app/compiledJS /app</span></span> 16</span>RUN</span> npm install --only=production</span></span> 17</span></span> 18</span>ARG</span> COMMIT_TAG=</span>"</span>none</span>"</span></span> 19</span>LABEL</span> COMMIT_TAG=$COMMIT_TAG</span></span> 20</span>ENTRYPOINT</span> [ </span>"</span>node</span>"</span> , </span>"</span>index,js</span>"</span>]</span></span></code></pre> We now can build our image with: docker build -t my-app --build-arg COMMIT_TAG="version-1.0.0" .</code></p> Placeholder 2021-11-14T00:00:00+00:00 More is comming soon</h1> Lorem ipsum dolor sit amet, consectetur adipiscing elit. Quisque consequat pretium mauris ac scelerisque. Sed semper egestas nisl. Integer tempus turpis mauris, eu dictum mauris porta id. Quisque nec ultricies erat. Mauris accumsan nisl velit, nec semper velit vehicula ornare. Pellentesque eu tortor ac ante finibus tincidunt. Phasellus tortor arcu, mattis id mauris a, fermentum pulvinar sem. Sed quis interdum velit. Vestibulum convallis arcu eget rutrum elementum. Nullam sapien orci, ullamcorper sit amet consequat at, tempus at dolor. Nunc et malesuada odio. Nam laoreet tempor metus vel vestibulum. Nullam convallis, risus eu venenatis rhoncus, sem ex bibendum enim, vel interdum dolor nisl quis urna. Curabitur in sodales enim, vel mattis velit</p> </p> See you</strong></p>

Native IPv6 Kubernetes for true edge routing

2026-04-18T00:00:00+00:00

Kubernetes & IPv6</h1>
Let’s bring some technologies with very different adoption metrics closer together:
Kubernetes is just over 11 years old, as of writing. IPv6 is from 1998, that’s more than twice the age of k8s with 27 years. Some early deployments are even older.
While Kubernetes is nothing but a success story, IPv6 has a complicated history. Based on Google data</a>, IPv6 just hit 50% of traffic - after almost 30 years. Let’s change that because it will make all our lives easier in the long run. Because IPv6 is the replacement of a legacy protocol - not an extension!
One of the major advantages of IPv6 is the much larger address space of 2^128 compared to 2^32. Instead of handing out individual IPs to each host or even sharing them behind (C)NAT, ISPs and cloud provider usually hand out entire ranges. Typically ranges between `/56</code> and /64</code> are advertised. This means that every one of these prefixes has more IPs available then the entire IPv4 space.`
`Let’s have some fun with that in Kubernetes!`

An easy IPv6 setup in Kubernetes</h2> Kubernetes has supported IPv6 since v1.6</code>, but the usability heavily depends on the CNI. The easiest way to use IPv6 in Kubernetes is to set these two args on cluster initialization with: kubeadm init --pod-network-cidr=fd00:10:32::/56 --service-cidr=fd00:10:64::/112</code> This example uses unique local addresses (ULA) IPv6 and does not need any special network requirements. Functionally it is equivalent to IPv4 clusters. These settings and the configured CNI will create an overlay network where pods can reach each other directly via the ULA addresses, but IP packets from and to the Internet need to be rewritten to reach anything. Nothing is really simplified here since we still distinguish between “private” and “public” IPs. Nevertheless, this is a great approach for testing and dev environments with tools like minikube, kind and more.
Native Routing</h3> Since IPv6 allows anyone to get large, unique global address spaces, we should use them: kubeadm init --pod-network-cidr=2a01:10:32::/56 --service-cidr=2a01:34:e393::/112</code> It uses global unicast addresses (GUA) that were allocated to you by your ISP or cloud provider. This setup requires a little more planning. We need our own IPv6 prefix that is large enough and not used by other devices. By default, Kubernetes tries to allocate a /64</code> from the pod network IPv6 CIDR to each node. In my opinion that is a very large default and can be wasteful in certain environments, since we will never have 2^64 pods on one node. It can be changed by setting --node-cidr-mask-size-ipv6</code> accordingly. Depending on your requirements, I would set a value between /64</code> and /112</code>. Even a /112</code> would allow 2^16=65536 IPs for each node to be consumed by pods. Exactly the same cidr-mask-arg can be used when you don’t have a /56</code> IPv6 prefix or do not want to allocate the entire range to a single cluster. Service CIDRs can also be adjusted according to ones need. Due to this approach being tailored to the current environment it runs in, it is a little less flexible, BUT it allows much simpler routing. As long as you don’t have any firewalls configured and all addresses are correctly advertised, every pod can directly reach the internet and can be reached from the internet. No service, loadbalancer, proxy or NAT required. Flat, native and efficient networking.
Native Routing on the edge</h3> From the network perspective, the routing became simpler, but it needs proper planning and requires you to have one large CIDR allocated to you. Sometimes this is not the case. Kubernetes is designed for distributed workloads. Maybe you want a multi-region cluster, a cluster spanning between clouds and a cluster that includes edge devices. Now every location has their own distinct address prefixes. Here IPv6 really shines. Kubernetes at its core does not really care about IPs. We can have scattered CIDRs per node, since the routing is handled by the CNI anyway, not Kubernetes. The only thing that is preventing us from doing this by default is the node-ipam-controller</code> in the kube-controller-manager</code> - which can be turned off. Now the pod-network-cidr</code> does not matter. The cluster initializes and CNI can handles the IP allocation, either via BGP or “manually” by patching each node’s .spec.podCIDR</code> and .spec.podCIDRs</code> fields. Now we can have a cluster with one node on AWS, one on GCP, Azure and one running at your home - with all pods communicating natively over IPv6.
`A few notes about Kubernetes IPv6</h2>`Within the Kubernetes network and the IPv6 Internet, all of the above approaches work without problems. Unfortunately, not everything is IPv6 and we still have to deal with legacy IPv4. The best and easiest way to deal with that is to use DNS64</a> at the CoreDNS upstream level. DNS64 essentially provides AAAA records for domains that do not have any natively by appending the target’s IPv4 A address to a predefined IPv6 prefix. The resulting address will point to an NAT64</a> implementation, which will handle IPv6 to IPv4 translation. While starting, there is no need to build this on your own, there are public and free DNS64/NAT64 services available. Since we are already building clusters how they will be run in the future, we needed to alter some Kubernetes default configurations. The same most likely applies to your CNI. I used Cilium for all my setups, some of the configuration changes includes turning off IPv4, setting the routing mode and using kube-proxy replacement. Native IPv6 routing simplifies the networking drastically but also exposes your workloads directly to the Internet if all addresses are correctly advertised and there is no filtering; a firewall is essential. Cilium provides great tooling for protecting the node, cluster and individual pods. A word about DualStack: One may ask: Why just not use IPv4 and IPv6 in the cluster and workloads would use whatever works - no need for NAT64/DNS64 DONT You will have two network stacks, loose native routing, have services that by default are SingleStack and apps that bind to the wrong IPs. Trust me, I tried it. You will not have less problems but double - IPv4 is legacy. There is a reason AWS EKS does not offer this configuration. </blockquote> Encouragement - try it</h2> While this all can sound a little complex at the beginning, it has become quite simple. Resources on the web are getting more numerous, and ISPs and clouds improved their IPv6 adoption a lot (except GitHub). I used Hetzner a lot for testing, they give you a free /64</code> for each node. Scaleway is also a good option. If you’re curious, you can also try to reverse engineer EKS and reimplement it, like I did - they are not doing anything magical. If native IPv6 is, for whatever reason, not possible, please at least create a loadbalancer that has IPv4 and IPv6. Internally all traffic can still be IPv4, but everyone on the Internet can talk to you over a modern connection without the need for DNS64/NAT64. Regarding the different adoption rates, I have a thesis: Kubernetes is something “new”, it solved a new and unsolved problem and had big names carrying it. IPv6 aims to fix issues from a previous solution that still kind of works. Unfortunately, it is often cheaper to find workarounds for old solutions instead of investing in new ones. But with the increasing price for IPv4 addresses, this finally seems to change. I hope I was able to give you a high-level overview and some approaches on how to bring IPv6 to the 90% traffic mark. Feel free to write me via Mastodon</a> or the Kubernetes Slack</a> if you have any questions. I implemented all of the above approaches and even automated most of them completely - it’s not that hard :)

Gateway API doesn't solve real problems - yet 2026-01-31T00:00:00+00:00 Gateway API doesn’t solve real problems - yet</h1> 🧾📖 PDF version</a> Disclaimer: This tells my personal experiences with the Kubernetes Gateway-API and shows the value Ingress-Nginx provided. My takeaway: Gateway API makes hard things possible - but easy things hard. </blockquote> Why the Change?</h2> The Kubernetes steering committee shocked a lot of people when they announced the retirement of the ingress-nginx-controller</a> at the 2025 Atlanta-Kubecon. After March 2026, the biggest ingress controller will not receive any more updates and patches. This change resulted in quite some angry comments towards the maintainers and the committee. But this change was foreseeable. The maintainers repeatedly reached out for more support or asked for corporate sponsors - which were unanswered. The decision is understandable, at least. A free, feature rich reverse proxy with good documentation that is used by thousands for free. Some corporations have generated billions in sales with requests that were served by ingress-nginx</code>, but the only thing that was given back were support requests, bug reports or new feature requests. I will not go into the debate of corporations exploiting open-source projects - this has been talked about a lot recently. The decision has been made, and the direction given was clear. The future is the Gateway-API</a>! Why a new API?</h2> The Ingress</code> API is a core component of the networking.k8s.io</code> Kubernetes API-Group. It defines the routing path for HTTP-Traffic (L7-Layer) via a bunch of rules with PATH</code> and Host</code>. It can also do TLS for these rules but that’s pretty much it. It does not handle headers like X-Forwarded</code>, cookies, authentication, payload size, compression, timeouts, redirects, traffic splitting and many more. All of these things can be done with ingress though, but not via its core API-Definition. Annotations are commonly used to enrich API-Objects with additional information. Ingress-Nginx had over 110 of them. A problem that comes with Annotations is that they are not validated, since they are not part of the API-Spec and that they are not vendor agnostic. So, you can’t use the same annotations from Ingress-Nginx for Kong, HAProxy, Traefik or the Apache-Webserver-Ingress. Each of these have their own feature- and annotation set, switching ingress-controller usually takes some time. Another concern is that all these settings are applied at the ingress level. Should a developer with Ingress</code> access really be able to alter the TLS config for a whole domain? Gateway-API promises to fix this. A well-defined API that separates operational tasks like TLS and proxy settings from routing configuration. Gateway-API introduces a stricter role-based approach. It has a mental model of how does what and which resources can be referenced from where. It’s HTTPRoute</code> supports native routing via hostname, header, path or query parameters. All well defined in its core API but still extendable via filters. Apart from HTTP routing, Gateway-API also proposes GRPCRoutes</code>, TLSRoutes</code>, TCPRoutes</code> and UDPRoutes</code>. While some of these are still experimental, having a well-defined, vendor-agnostic spec for all these use-cases is a killer feature. How does this affect Developers and Admins?</h2> In theory this means less error-prone routing configuration (let’s see how this will work out) and developers not having to worry about TLS. In practice, developers will need to learn a new doc-spec or wait for admins to give them templates. Admins on the other hand will have a lot more to do. Configure the Gateway, provide templates for developers and allow all necessary namespaces to reference the Gateway. They need to test and set up new integration, handle TLS and maintain a separate API that is maintained outside the Kubernetes core API. Gateway API is more complex</h3> With more features comes more complexity. The Ingress API has two resource types, Ingress</code> and the hardly noticed IngressClass</code> type. The Gateway-API has ReferenceGrant</code>, BackendTLSPolicy</code>, GRPCRoute</code>, Gateway</code>, GatewayClass</code>, HTTPRoute</code> and these are just the stable API-Objects, there are a lot more in experimental mode. Ingress is a core component of Kubernetes, the Gateway-API is not shipped with Kubernetes and needs to be installed separately via CRDs</code>. This might result in bootstrapping challenges since one cannot expect that the APIs are installed. To install them use kubectl apply --server-side -f https://github.com/kubernetes-sigs/gateway-api/releases/download/<LATEST_VERSION>/standard-install.yaml</code>. The --server-side</code> is required since the API is too big for client side apply. </blockquote> To setup a Gateway</code> cluster admins need to install CRDs</code>, choose an implementation, install an operator/controller and then create the Gateway</code>. Most Gateway-API implementations separate the controlplane from the dataplane. Most Ingress implementations used a shared approach where an agent ran as a sidecar process next to the dataplane to ensure the proper configuration. Right now, the selecting a Gateway is actually the hardest part. While the main API-Objects specs have reached GA, their features are not yet supported by all implementations. While the Gateway-API SIG tries to help with this decision by providing comparison tables</a> and conformance reports</a> it is still a lot of work to go through. These on-paper comparisons don’t replace real-word tests to catch any edge cases and say nothing about performance between all these. If you look for some real-world comparisons, you may wanna check out this benchmark</a> from John Howard</a>. Gateway API is not vendor agnostic</h3> Let’s remember the goals of Gateway API: Separate personas, strict validation and a vendor-agnostic config. Right now, I see a trend that is moving against the last goal. Yes, TLS configuration and routing might be standardized, but the nginx-gateway-fabric</a> for example already brings their own set of 10 CRDs</code> for configuring the Gateway</code> itself or for use cases that are not yet defined in the standard Gateway-API spec. And so does Kong and Envoy-Gateway with even more CRDs</code>. There is no native way to do auth or mTLS enforcement (for clients) in the Gateway-API yet, so implementations bring their own filters and CRDs</code>. Gateway API, by design, is extendable and vendors are using this extendibility to overcome the shortcomings of the core API. They are using these to separate themselves from each other and maybe - to create a vendor-lock-in. Never forget that there are companies behind these products that want and need to make money. Maybe over time more features will be covered by the core Gateway-API, but most of us know even if the spec is well defined, implementations can behave differently. My TLSRoute</code> worked perfectly with Cilium but was not working with nginx-gateway-fabric. Gateway API is not widely supported yet</h3> New technologies need time for adoption, it is totally normal that most helm charts do not yet ship with HTTPRoutes</code> alongside Ingress</code>. Unfortunately a lot of helm charts do not support a extraObjects</code> either, so you need multiple commands to deploy your app or a multi source GitOps solution. Creating a good general helm template for HTTPRoutes</code> is quite hard. I know this because I crated the HttpRoute</code> helm template that gets generated when you run helm create <my-chart></code> in recent versions of helm. </blockquote> I wanna give another example of a problem I encountered: A common scenario in highly automated clusters is having an Ingress-Controller, CertManager and ExternalDNS. These work perfectly together. The ingress is created and uses a fallback TLS certificate, ExternalDNS checks the domain records and adds a new record if not set yet and Certmanager creates a http-01-challenge</code> to obtain a valid certificate. It may take some minutes, but eventually you have a valid, working Ingress</code> with TLS. Now try the same with Gateway-API. It will not work, even with experimental flags. Why does it not work? The Gateway handles the TLS, not the HTTPRoute</code>. The default listener likely does not have a valid certificate for your new domain, so you add a new listener. The HTTPRoute</code> references that listener but never becomes ready because the listener is not healthy yet due to the missing certificate. ExternalDNS watches HTTPRoutes</code>, but only the ones that are ready and have an IP. Now the circle closes, without a valid DNS record, you can’t complete a http-01-challenge</code>. It’s a deadlock. The only way around this that I found was to switch to a dns-01 challenge</code>. The ExternalDNS project is aware of this problem, but AFAIK the solution is still work-in-progress. Ingress-Nginx will be missed</h2> I don’t want to make Gateway-API look bad; it is powerful - maybe not just ready yet. For those not ready to switch: Maybe look at Cainguards EmeritOSS</a> program. They announced</a> to provide basic security maintenance for ingress-nginx</code>. </blockquote> For these reason, Ingress-Nginx will be missed. It is a dead simple solution that just works. The battle-tested Nginx with a small agent that does config generation and reloads. Provided by a couple of individuals that did get nothing in return but provided the technology that served half of all Kubernetes-hosted HTTP traffic. Ingress-Nginx is a community project - no big company behind it. That’s something that will change with Gateway-API. Implementing all of Gateway-APIs features is a lot of work, no one can expect a couple of individuals to do that in their free time. Now it’s done by corporations that battle for the market share that Ingress-Nginx will leave behind. Once this battle is settled, I wouldn’t be surprised to see more Broadcom, HashiCorp and Docker like rug-pulls. AWS Web-Identity-Token - The free IDP for all your OnPrem solutions 2025-12-08T00:00:00+00:00 Reduce your Token-Usage for all non AWS-Apps with AWS Web-Identity-Token</h1> TLDR: AWS new STS Web-Identity-Token function allows secure, low maintenance authentication via OIDC for all none AWS Services at no additional cost. Perfect for OnPrem system or CI. </blockquote> AWS enabled a new functionality within their secret-token-service (STS) which basically is a free and managed OpenID Connect</a> (OIDC) Identity-Provider (IDP). Users and IAM-Roles with the needed permissions can create shot lived JWTs</a> and can securely authenticate against any service that supports OIDC-Token authentication. Services can validate the token via the AWS managed IDP-Issuer. This essentially makes all long lived and manually managed credentials unneeded. Use cases</h2> I found this functionality via a post from Piotr Pabis</a>. He implemented a (little over-engendered) demo to create a low maintenance authentication setup between AWS and GCP using AWS Web-Identity-Tokens, to show how to create multi-cloud secure auth setups with short lived tokens. This inspired me to test it out and try to access my private Kubernetes Cluster with AWS-Tokens, but the use cases are virtually endless! Just to give a few examples: Kubernetes</h3> If you have a hybrid infrastructure with some systems on AWS and some OnPrem this can come very handy. Kubernetes natively supports OIDC and all you need to enable AWS Web-Identity-Token authentication is to add the AWS managed IdP to your Kubernetes structured-auth-config</a>. 1# Docs: https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens 2apiVersion: apiserver.config.k8s.io/v1beta1 3kind: AuthenticationConfiguration 4jwt: 5 - issuer: 6 url: https://xxx.tokens.sts.global.api.aws 7 audiences: 8 - https://k8s.my-org.com 9 claimMappings: 10 username: 11 claim: sub 12 prefix: "aws:"</code></pre> Note: You can add as many Issuers as you want, you can also configure and limit anonymous auth to increase your Kubernetes security</a> </blockquote> Now you can use normal Kubernetes RBAC to allow the aws:<YOUR_ROLE_ARN></code> user to access the needed resources. No need to ever share a Kubeconf.yaml</code> ever again. Combine this approach with the credential less authentication between GitHub Actions or GitLab CI</a> and gone are the days of managing long lived credentials, rotating them and risking leakage just to deploy something to Kubernetes. Workflow Automation</h3> You use automated Workflow? Have any WebHooks or have Microsoft Power Automate setup? Now everything that is accessible can have low maintenance short lived token that can easily be verified via the aws managed IdP. Need some examples? Your AWS Lambda has an issue and you want it to automatically open a ticket in your support system? The lambda can use its own identity to create a token which the ticket system can validate.</li> Access an OnPrem file Share via HTTP. Just send the JWT as a Bearer, validate it and access internal documents to be processed in your AWS automation.</li> One AWS Lambda can authenticate to another without Cognito.</li> You CI assumes an AWS role and sends an authenticated request to your chat app to push notifications.</li> Update any OnPrem hosted document without pushing it to AWS.</li> </ul> How to use</h2> So how do I start using this? Easy, just enable it and you can start requesting tokens. Everything else is managed by AWS. You can just the aws cli</code> to request tokens, but also the SDK of your favorite language. Examples can be found at the AWS Python boto3 docs</a>. 1# Get your issuer config 2aws iam get-outbound-web-identity-federation-info --output json 3# if not enabled, you can enable it with 4aws iam enable-outbound-web-identity-federation --output json</code></pre> If not already enabled, it creates a aws managed private key-pair (RSA and ECDSA) and publishes the IdP-Config under a globally accessible and unique endpoint. The command then gives you an URL in the format https://xxx.tokens.sts.global.api.aws</code>. When you access https://xxx.tokens.sts.global.api.aws/.well-known/openid-configuration</code> you can see all supported claims, supported signature algorithms and the path to your JWKs config, which contains all information to validate your JWTs with it’s associated public key. 1{ 2 "Version": "2012-10-17", 3 "Statement": [ 4 { 5 "Effect": "Allow", 6 "Action": [ 7 "sts:TagGetWebIdentityToken", 8 "sts:GetWebIdentityToken", 9 "sts:SetContext" 10 ], 11 "Resource": "*" 12 } 13 ] 14}</code></pre> Note: You can also add Condition</code> to the permissions to ensure a low TTL, the used signer algorithm or any validation on tags you can think of. You also need at least the AWS-CLI version ≥2.32.5</code> </blockquote> Now every IAM-Role with these permissions can crate a JWT: 1aws sts get-web-identity-token --audience demo --signing-algorithm ES384 2# You can also add up to 50 tags which will make extra information available in the JWT 3aws sts get-web-identity-token --audience demo --signing-algorithm ES384 --tags Key=account,Value=123456</code></pre> The decoded JWT will look like this: 1{ 2 "aud": "demo", 3 "sub": "<MY_IAM_ROLE_ARN>", 4 "https://sts.amazonaws.com/": { 5 "org_id": "o-xxx", 6 "aws_account": "123456", 7 "ou_path": ["xxx"], 8 "request_tags": { 9 "account": "123456" 10 }, 11 "original_session_exp": "2025-12-07T22:17:32Z", 12 "source_region": "eu-central-1", 13 "principal_id": "<MY_IAM_ROLE_ARN>", 14 "identity_store_user_id": "xxx" 15 }, 16 "iss": "https://xxx.tokens.sts.global.api.aws", 17 "exp": 1765140780, 18 "iat": 1765140480, 19 "jti": "xxx" 20}</code></pre> For testing purposes or if you have self developed applications to you want to use with AWS web-identity-tokens yo can use the example python script below to validate the token. validate.py</summary> 1import requests 2import jwt 3 4def verifyToken( 5 token: str, 6 jwks: dict, 7 audience: str, 8 issuer: str, 9 algorithms = ["RS256", "ES384"] 10) -> dict: 11 try: 12 return jwt.decode( 13 token, jwks, algorithms=algorithms, audience=audience, issuer=issuer 14 ) 15 except Exception as e: 16 print("Token verification failed:", e) 17 return None 18 19 20def getJWKs(jwks_uri: str) -> dict: 21 resp = requests.get(jwks_uri) 22 resp.raise_for_status() 23 return resp.json() 24 25def getJWKsUri(issuer: str) -> str: 26 resp = requests.get(f"{issuer}/.well-known/openid-configuration") 27 resp.raise_for_status() 28 return resp.json()["jwks_uri"] 29 30 31if __name__ == "__main__": 32 iss = "https://xxx.tokens.sts.global.api.aws" 33 aud = "demo" 34 token = "xxx" 35 verifyToken(token, getJWKs(getJWKsUri(iss))["keys"][0], aud, iss)</code></pre></details> The Grafana trust problem 2025-11-14T00:00:00+00:00 I can’t recommend Grafana anymore</h1> Disclaimer: This tells my personal experiences with Grafana products. It also includes some facts but your experience may vary, and I would love to hear your take. </blockquote> I started my work life at a small software company near my university. They develop, run websites and operate web services for multiple clients. Everyone had multiple responsibilities, and they heavily relied on interns and freshmen—which can be both bad and good. For me it was good because I learned a lot. At some point we needed a monitoring solution, and Zabbix didn’t fit well into the new and declarative world of containers and Docker. I was tasked to find a solution. I looked at Loki/Prometheus with Grafana and Elastic with Kibana. Elastic was a beast! Heavy, hard to run, resource-hungry, and complex, Loki and Prometheus were the perfect fit back then. So, I created a docker-compose.yaml</code> with Loki, Prometheus and Grafana. Since they all were within a internal Docker network, we required no auth between them. Grafana was only exposed over an SSH tunnel. One static scrape config and the Docker Loki log plugin later, we had our observability stack. For non-Docker logs, we used Promtail. Loki and Prometheus stayed on the same machine and all we required was a local volume mount. Load was minimal. ℹ️ This is when I learned that you should not transform every log parameter to a label just to make it easier to select in the Grafana UI. Having a label for latency with basically limitless values will fill every disk inode’s, that’s just how Cortex bin-packs. </blockquote> I also found out Grafana Labs has a cloud offering with a nice free tier. I even used this for personal stuff. I had a good experience with them. Time goes on, and I switched jobs. Now we have Kubernetes. The Prometheus container was now switching nodes. Roaming storage was a problem back then, and our workload increased by a lot. We also needed long-term storage (13 months). So I looked around and found Thanos and Mimir. Previous experiences with Grafana products were good, so I chose Mimir. Should be similar to Loki since both are based on Cortex. Now we didn’t really need Prometheus anymore. We were only using remote_write</code> from Prometheus. Grafana had a solution for this. With the Grafana Agent, you can ship both logs and metrics to a remote location all in one binary. This seemed like a no-brainer. Time goes on, and Grafana changed the Grafana Agent setup to Grafana Agent Flow Mode—some adjustments, but okay - software changes. And man, did Grafana like to change things. They started to build their own observability platform to steal some of DataDog’s customers. They created Grafana OnCall their own notification system. Not only that, but they heavily invested in Helm charts and general starter templates. Basically, it took only two commands to install the metric/log shippers and use Grafana Cloud. And even when you don’t want or can’t use Grafana Cloud, here are the Helm charts to install Mimir/Loki/Tempo. To make things even easier, let’s all put it in an umbrella chart (it renders to 6k lines in the default state). Or use their Grafana Operator to manage Grafana installs - or at least parts of it. As many may have experienced, software maintenance shows with age. Grafana OnCall is deprecated, and Grafana Agent and Agent Flow were deprecated within 2-3 years of their creation. Some of the easy to use Helm charts are not maintained anymore. They also deprecated Angular within Grafana and switched to React for dashboards. This broke most existing dashboards. On the same day they deprecated the Grafana Agent, they announced Grafana Alloy. The all in one replacement. It can do Logs, Metrics, Traces (zipkin & jaeger) and OTEL. The solution for everything! The solution kind of had a rough start and was a little buggy. But it got better over time. The Alloy Operator also entered the game because why not. ℹ️ They choose to use their own configuration language for alloy. Something that looks like HCL. I can understand why they didn’t want to use YAML but I’m still not a fan of this. Not everything needs their own DSL. </blockquote> Happy End, right? - Not quite The all-in-one solution does not support everything. While Grafana built their own monitoring empire, the kube-prometheus community consistently and naturally developed. The Prometheus Operator with ServiceMonitor</code> and PodMonitor</code> CRDs became the defacto standards in Kubernetes. So Alloy also supports the monitoring.coreos.com</code> api-group CRDs, at least some parts of it. It natively works with ServiceMonitor</code> and PodMonitor</code>, but PrometheusRules</code> needs extra configuration. The AlertmanagerConfig</code> which would need to be implemented in Mimir is not supported. Because Mimir brings its own Alertmanager - at least sort of. There are version differences and small incompatibilities. But I got it all working; now I can finally stop explaining to my boss why we need to re-structure the monitoring stack every year. Grafana just released Mimir 3.0. They re-architected the ingestion logic for scalability, and now they use a message broker. Yes, Mimir in version 3.0 needs Apache Kafka to work. None of the above things alone would be a reason to ditch Grafana products. Set aside the fact that they made it incredibly difficult now to find the ingestion endpoints for Grafana Cloud since they want to push users to use their new fleet-config management service. But all this together makes me uncomfortable recommending Grafana stuff. I just don’t know what will change next. I want stability for my monitoring; I want it boring, and that’s something Grafana is not offering. It seems like the pace within Grafana is way too fast for many companies, and I know for a fact that that pace is partially driven by career-driven development. There are some smart people at Grafana but not every customer is smart nor has the capacity to make Grafana their priority number one. Complexity kills - we’ve seen this. ℹ️ Don’t get me wrong. Mimir, Loki and Grafana are technically really good software products and I (mostly) still like them but it’s the way these products are managed which makes me question them. </blockquote> Sometimes I wonder how I would see this if I had chosen the ELK stack at my first job. I also wonder if the OpenShift approach (kube-prometheus-stack) with Thanos for long-term storage is the most time-stable solution. I just hope OTEL settles, gets stable and boring fast, and just lets me pick whatever I want for my backend. Because right now I’m done with monitoring. I just want to support our application and do not want to revisit the monitoring setup every x weeks, because monitoring is a necessity—not the product. At least for most companies. Rootless GitLab Runners 2025-10-05T00:00:00+00:00 Rootless GitLab Runners</h1> TL;DR: Rootless Docker has become easy! You can also easily run the GitLab Runner binary with rootless Docker without impacting workloads with rootless-kit. Even run buildkit, dind and docker build! See the HOW-TO! </blockquote> GitLab CI is the biggest CI/CD system after GitHub Actions and is the preferred solution for self-hosted and enterprise SCM-Systems. While GitHub gives you a fresh, full-blown VM for each job, GitLab has the concept of different Executors. While the GitLab-Runner binary manages the communication and job setup, the executor does the actual work. While there are Shell</code> and SSH</code> executors, container-based executors are often preferred for flexibility, security and reproducibility. While Docker can improve security, it is not know for it in its default configuration, since everything runs as root. Attack Surface Docker</h2> By default, Docker is started as root and everyone in the docker</code> user group can communicate with the root-owned socket in /var/run/docker.sock</code>. Since the user-ids inside a container map 1:1 to the user-ids on the host, a simple docker run --it --privileged -v /:/host debian</code> can overtake the whole system. Even if the user executing it is not root. ℹ️ Docker and other containers are NOT virtualization! All processes run on the same kernel as the host and are only isolated by process namespace and cgroup capabilities. </blockquote> This alone is kind of bad. It gets even worse if you run arbitrary, potentially malicious code from unknown systems. A complete host takeover is only one services: [docker:dind]</code> or a stupid runner config</a> away. But how can you run jobs rootless and what about docker build</code> jobs that need dind? Securing Docker & CI</h2> This can be mitigated by not running Docker as root at all. Rootless Docker used to be a pain but has become surprisingly easy on modern systems. NOTE: The instructions are based on a Debian system but can be adapted to others. First install Docker according to the official docs: 1# Install pre-requirements and some extra packages 2sudo apt update 3sudo apt install curl ca-certificates curl uidmap apparmor lsb-release \ 4 slirp4netns dbus-user-session fuse-overlayfs systemd-container fuse-overlayfs cifs-utils 5 6# Add the docker apt repo 7sudo install -m 0755 -d /etc/apt/keyrings 8sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc 9sudo chmod a+r /etc/apt/keyrings/docker.asc 10echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \ 11 https://download.docker.com/linux/debian \ 12 $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ 13 sudo tee /etc/apt/sources.list.d/docker.list > /dev/null 14 15# Finally install docker 16sudo apt update 17sudo apt install docker-ce docker-ce-cli containerd.io \ 18 docker-buildx-plugin docker-compose-plugin docker-ce-rootless-extras</code></pre> Now Docker is running as root. Which we don’t want. Disable it via systemd and optionally allow normal users to open ports below 1024. 1# Disable the root docker service 2sudo systemctl restart apparmor.service 3sudo systemctl disable --now docker.service docker.socket 4sudo rm /var/run/docker.sock 5# Allow privileged ports for normal users 6sudo sysctl -w net.ipv4.ip_unprivileged_port_start=0</code></pre> Now we need the gitlab-runner binary that communicates with GitLab and our executor. By default, the official install scripts also runs gitlab-runner as root. It should also run as a normal user: 1# Create gitlab-runner user and install the GitLab Runner 2sudo useradd --create-home --shell /usr/bin/bash --user-group gitlab-runner 3curl -JOL https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh 4sudo bash script.deb.sh 5sudo apt install gitlab-runner 6sudo systemctl stop gitlab-runner 7sudo systemctl disable --now gitlab-runner</code></pre> We want to run docker rootless, but the users inside of containers are often root or require root. Since we want all our workloads to work with the new setup we use the user-namespace capability of the Linux kernel. This maps the root user form inside a container to an unprivileged user to the host. So even if an attacker manages to break out of a container, he will never have privileges higher than the newly created gitlab-runner</code> user. 1# Set user-namespace mapping 2sudo echo $(id -u gitlab-runner):100000:65536 >> /etc/subuid 3sudo echo $(id -u gitlab-runner):100000:65536 >> /etc/subgid 4 5# If you run everything headless without a login shell you need to enable lingering 6sudo loginctl enable-linger</code></pre> We can now switch to the new gitlab-runner</code> user and set up docker and the required configs: gitlab-runner.service</summary> 1# gitlab-runner.service 2[Unit] 3Description="GitLab Runner" 4ConditionFileIsExecutable="/usr/bin/gitlab-runner" 5After="network.target" 6 7[Service] 8Restart="always" 9RestartSec=120 10StartLimitInterval=5 11StartLimitBurst=10 12Environment="DOCKER_HOST=unix:///run/user/<<YOUR_GITLAB_RUNNER_USERID>>/docker.sock" 13ExecStart=/usr/bin/gitlab-runner run \ 14--config /home/gitlab-runner/gitlab-runner-config.toml \ 15--working-directory /home/gitlab-runner \ 16--service gitlab-runner --user gitlab-runner 17 18[Install] 19WantedBy="default.target"</code></pre></details> gitlab-runner-config.toml</summary> 1# gitlab-runner-config.toml 2concurrent = 4 3check_interval = 0 4shutdown_timeout = 0 5[session_server] 6 session_timeout = 1800 7[[runners]] 8 name = "Rootless Docker" 9 url = "https://gitlab.com" 10 id = 1234 11 token = "<MY_TOKEN>" 12 executor = "docker" 13 [runners.docker] 14 image = "debian" 15 allowed_pull_policies = ["always", "if-not-present"] 16 allowed_privileged_images = ["moby/buildkit:*", "docker:dind" ] 17 disable_entrypoint_overwrite = false 18 privileged = true 19 oom_kill_disable = false 20 disable_cache = false 21 volumes = ["/cache", "/certs/client"] 22 shm_size = 0 23 network_mtu = 0</code></pre></details> We now can start rootless docker and configure gitlab-runner to use it: 1sudo -i && su gitlab-runner 2mkdir -p .config/systemd/user /.config/docker 3 4# Run rootless docker 5dockerd-rootless-setuptool.sh install 6 7# You can now run docker commands 8export DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock 9docker run --rm hello-world 10 11# Setup GitLab Runner conf and systemd service with the conf above 12# Remember to adjust your user-id and set your token 13vim .config/systemd/user/gitlab-runner.service 14vim gitlab-runner-config.toml 15 16# Activate the runner 17systemctl --user daemon-reload 18systemctl --user enable gitlab-runner.service 19systemctl --user start gitlab-runner.service</code></pre> With the correct token and GitLab server configured you should now see the runner in the GitLab UI. You now can run any CI-Job with this runner - even dind works! 1rootless-job: 2 tags: [my-rootless-runner] 3 services: [docker:dind] 4 stage: test 5 scripts: 6 - echo "I am running as $(whoami)"</code></pre>What about Kubernetes</h2> What if you use the Kubernetes GitLab-Executor and run buildkit in Kubernetes? When you run modern Kubernetes with containerd</code> version >=2.x1</code> and Kubernetes >= 1.33.x</code> you can use userNamespace</code> with your pods. It is just one parameter. Just set spec.hostUsers</code> to false</code> in your Pods</code> any thats it. And now?</h2> This is one measurement to avoid security issues that may overtake your CI runners, but it is by far not the only attack that often used. Supply chain security is a wide and deep topic and the upcoming post will explain how to protect your environment from credential threats and other attacks. Follow Up: Let's talk about anonymous access to Kubernetes 2025-05-22T00:00:00+00:00 Follow Up: Let’s talk about anonymous access to Kubernetes</h1> TL;DR: Kubernetes default anonymous-auth allows unended information exposure, but disabling it was hard. Now you can limit anonymous-auth to specific paths only. </blockquote> In 2023 Rory McCune (raesene) wrote a nice blog post</a> about Kubernetes anonymous-auth</code> flag, which is enabled by default. While security mechanism like RBAC are still applied, certain endpoints in Kubernetes can be accessed without providing any form of authentication - therefore, it is called anonymous-auth</code>. He raised awareness for this flag and also did a regular Kubernetes Census</a> listing how many Kubernetes API-Servers were publicly accessible and in which version. He was able to gather that data, because the /version</code> path in one of the paths that can be accessed without any form of authentication. Rory McCune suggests to outright disable the anonyms auth flag with --anonymous-auth=false</code> to reduce the exposed information and the potential attack vector. While it is definitely a smart idea, it is often unpractical since kubeadm</code> needs certain anonymous-auth endpoints in order to join new nodes to a cluster. In long-lived static clusters where new nodes are rarely added, this would not be a large issue, but the kube-api-server itself uses anonymous-auth</code> endpoints in order to perform health checks on itself. These endpoints may also be used by loadbalancer in order to check it’s targets status. If these checks failed, due to a 401 Unauthorized</code> HTTP Error, the api-server would mark itself (actually it is the kubelet doing the check) as unhealthy and restart itself - over and over again. Making the whole cluster unaccessible. So setting --anonymous-auth=false</code> often was impractical. But Kubernetes is a fast-moving platform and has heavily increased security in recent years with features like usernamespaces</code>, appamor</code>, selinux</code> and bound-serviceaccount-tokens. They also introduced the structured-authentication-config</a> in KEP-4633</a>. A config file that allows users to declaratively configure OIDC-Auth (the only reasonable auth option for user access in larger clusters) AND a fine-grained anonymous auth configuration. It took me a few tries but with the following authentication-config you can drastically reduce the information exposure of your cluster and still be able to join new nodes via kubeadm</code> and check the cluster health. It also prevents misconfiguration with unintended RBAC bindings to the unauthenticated</code> group. 1apiVersion: apiserver.config.k8s.io/v1beta1 2kind: AuthenticationConfiguration 3anonymous: 4 enabled: true 5 conditions: 6 # kubeadm needs to read the discovery info 7 - path: /api/v1/namespaces/kube-public/configmaps/cluster-info 8 # for liveness and readiness checks: 9 - path: /healthz 10 - path: /readyz 11 - path: /livez 12jwt: 13 # Optional OIDC config</code></pre> This only gives access to the health/status endpoints and to one specific configmap. The cluster-version and other information are not accessible without authentication. The health endpoints only return 200 OK</code> and the configmap contains the IDs of bootstrap-token and the clusters CA certificate, which is accessible anyway. Thats it! Easy way to protect your clusters a little more without sacrificing functionality. P.S. If someone knows how to write CEL expressions that can validate the extra claims of a user idtoken - write me! </blockquote> Level up your Ansible Code - Creating Golden Images 2025-04-20T00:00:00+00:00 Level up your Ansible Code - Creating Golden Images</h1> TL;DR: The combination of Ansible and Packer can be a terrific solution to improve provisioning time, increase reliability and reduce stress on other infrastructure. Run Packer once with your existing Ansible-Config and reuse the resulting image a 1000 times. </blockquote> So the story usually looks like this: You became tired of doing the same web-server base configuration over and over again. Sometimes it didn’t work because you misspelled a parameter or copy-pasted the wrong command. But then someone mentioned Ansible. A tool that can run the same actions over and over again with the desired same output (when done right - which I have seen not that often). It can be lunched against 1000s of machines. It sounded great since you wanted to have more free time and are not afraid to automate yourself out of your job (don’t worry, will not happen - you will only level up). So you started to implement your tasks as Ansible roles and tasks. It took a while to get right but now you have made yourself a name by always delivering setups in such a quick time with rock solid and consistent quality. Great, enjoy your life, right? Yeah, maybe. But you seek even better outputs, even more “free” time, got a new requirement, or just want a good reason for your next raise? I don’t really care, but this is where this canticle comes in handy! My Ansible setup</h2> My story actually looked quite similar. I use Ansible to set up vanilla k8s cluster on systems. I have a common</code> role for all nodes and a cp-bootstrap</code>/worker-join</code> role. Setting up a Debian based cluster, with a lot of extra conf for extra container-runtimes and newer kernels it takes about 15min to fully bootstrap a cluster on the Hetzner cloud. That is so slow. The annoying stuff in Ansible</h2> Manly because Ansible is slow. Even with connection-pipelining</a>, multiple forks</a> and other tweaks. It gets even worse if the connection goes over VPS,JumpHosts or some Proxies. But not only Ansible is slow. Installing packages is slow, setting kernel parameters and downloading stuff is slow. And sometimes, not often, your playbooks FAIL. Because a connection timed out, you got rate limited or a package version changed. When this happens, your playbooks are not idempotent, let alone reproducible. It mostly works out, but you can not guarantee that the state will always be exactly the same when you run your playbook over and over again. I get it, reproducibility is hard, but we can make your life’s a little easier when we don’t have to run Ansible that often. And even save more time, because again, Ansible is slow. Introducing Packer & Golden Images</h2> Packer is a packaging software from HashiCorp. It uses plugins to integrate with various providers, such as AWS, Azure, GCE but also onPrem solutions like VMware, Proxmox and good old QEMU. It uses HashiCorp’s Configuration Language (HCL), just like Terraform. In Packer you usually have one or more source</code> resources with one or more build</code> steps. Packers plugin ecosystem makes it really versatile for your build configuration. You can create AMIs on AWS, but also Docker-Images, ISOs and RAW disk images. Reusability is provided by HCL by the usage of variables, inputs and basic logic functions (JSON-Encode, reverse, map, etc.) build into HCL. With Packer you can archive real reproducibility. You create a working image once and reuse it a 1000 times. All machines created from this image will have the same, known good base configuration with the same packages installed. Perfect for providing ready-to-go hardened servers. The resulting images are called golden-images. And now we will use our existing configuration automation code for Ansible with the Packers plugin system to cut down provisioning time for our k8s clusters. Merging Packer & Ansible</h2> Let’s work through reusing my existing Ansible playbooks to create golden images by using an concrete example. I want to cut down provisioning time for k8s clusters from 15 minutes to 5 minutes. For this I will use golden images created by Packer with the the Hetzner hcloud</code> Packer plugin. I need a source and some basic inputs to make it configurable: 1# hcloud.pkr.hcl 2packer { 3 required_plugins { 4 hcloud = { 5 source = "github.com/hetznercloud/hcloud" 6 version = ">= 1.6.0" 7 } 8 } 9} 10 11variable "base_image" { 12 type = string 13 default = "debian-12" 14} 15variable "k8s_version" { 16 type = string 17 default = "1.32.3" 18} 19variable "user_data_path" { 20 type = string 21 default = "cloud-init.yml" 22} 23 24locals { 25 output_name = "debian-12-k8s-v${var.k8s_version}" 26} 27 28source "hcloud" "k8s-amd64" { 29 image = var.base_image 30 location = "nbg1" 31 server_type = "cx22" 32 ssh_keys = [] 33 user_data = file(var.user_data_path) 34 ssh_username = "root" 35 snapshot_name = "${local.output_name}-amd64" 36 snapshot_labels = { 37 type = "infra", 38 base = var.base_image, 39 version = "${var.k8s_version}", 40 name = "${local.output_name}-amd64" 41 arch = "amd64" 42 } 43} 44source "hcloud" "k8s-arm64" { 45 image = var.base_image 46 location = "nbg1" 47 server_type = "cax11" 48 ssh_keys = [] 49 user_data = file(var.user_data_path) 50 ssh_username = "root" 51 snapshot_name = "${local.output_name}-arm64" 52 snapshot_labels = { 53 type = "infra", 54 base = var.base_image, 55 version = "${var.k8s_version}", 56 name = "${local.output_name}-arm64" 57 arch = "arm64" 58 } 59}</code></pre> This allows me to create images (or snapshots for Hetzner) for debian based imaged on amd64 and arm64. The image name will be set based on the input parameters. Now we need a build step. While Packer has a dedicated Ansible plugin, I choose to not use it, since it did not meet my requirements for handling restarts and other parameters. Instead I choose to use the build-in shell</code> provisioner which just runs some inline commands that are a predefined in a shell script: 1build { 2 sources = ["source.hcloud.k8s-amd64", "source.hcloud.k8s-arm64"] 3 4 provisioner "shell" { 5 expect_disconnect = true 6 env = { 7 k8s_version = "${var.k8s_version}" 8 } 9 scripts = [ 10 "ansible-setup.sh", 11 ] 12 } 13 provisioner "shell" { 14 pause_before = "30s" 15 max_retries = 1 16 env = { 17 k8s_version = "${var.k8s_version}" 18 } 19 scripts = [ 20 "ansible-setup.sh", 21 ] 22 } 23}</code></pre> This runs the same shell script two times, which is exactly what I need because my IaC code includes a restart to swap the current kernel version. All what my shell script really does is to wait for cloud-init</code> to finish, clone my Ansible git repo and run it with a bunch of predefined vars. Finally it performs some cleanups and resets cloud-init: 1#!/bin/bash 2set -e -o pipefail 3 4echo "Waiting for cloud-init to finish..." 5cloud-init status --wait 6 7# setup requirements 8echo "Installing packages..." 9apt-get update -qq 10apt-get install -qq --yes --no-install-recommends git python3-pip 11pip3 install --user --break-system-packages --no-warn-script-location --no-cache-dir ansible jmespath 12PATH=~/.local/bin:$PATH 13 14if [ ! -d "playbooks" ]; then 15 git clone --depth 1 https://github.com/hegerdes/ansible-playbooks.git playbooks 16fi 17 18# setup ansible play 19echo "Running playbook..." 20cd playbooks 21echo "Vars:" 22cat <<EOF >hostvars.yaml 23k8s_cri: crun 24k8s_containerd_variant: github 25k8s_ensure_min_kernel_version: 6.12.* 26EOF 27 28printenv | sed 's/=/\: /g' | grep k8s >>hostvars.yaml 29cat hostvars.yaml 30 31cat <<EOF >pb_k8s_local.yml 32- name: K8s-ClusterPrep 33 hosts: localhost 34 become: true 35 gather_facts: true 36 roles: 37 - k8s/common 38EOF 39 40# Run playbook 41ansible-playbook pb_k8s_local.yml --extra-vars "@hostvars.yaml" -v && cd</code></pre> I just run my existing playbooks locally on the target server to archive the desired configuration. While the reproducibility is provided by the resulting image, we should still try to make our playbooks idempotent, so we can run them over and over again. It should not perform any changes when the desired state is already reached. With this and some additional cleanup code to minimize image size, I get ready to use golden k8s images, which I can use to quickly spin up a new cluster or use as an autoscaling nodes. Every dependency I need is already in that image. Every kernel parameter is already tuned to my desire. I can just use these, and be sure that they will always act the same, since these are truly reproducible. The creation of these two images took about 12 minutes. Thats 12 minutes I have to spent once (for every k8s version) and I now save for every deployment. When I now want to create a new k8s cluster, I can just uses these images and be sure they are already configured to my desire. Bootstrapping a new cluster with these images takes actually less then 5 minutes. It saves time and I am much more constable, that my setups are always acting the same, because they are reproducible. ℹ️ You can find the complete code on my GitHub in my GitOps</a> repo. </blockquote> Conclusion</h2> Ansible, Packer and even Terraform are great tools that work together and ma a real dream team when put together. Thats one of the reason RedHat bought HashiCorp and now works on putting these even closer together. These tools are not competitors! When you want to save even more time, you can easily glue your Ansible code together with tools like Packer, to create truly reproducible deployments. Understanding and using modern day authentication frameworks to improve security, productivity and user acceptance 2025-02-12T00:00:00+00:00 Understanding modern day Authentication with OAuth2 and OIDC</h1> ℹ️ This post was written in the context of my employment at ARES. While it does not promote any software or solutions from which ARES or I will benefit off, it is written in a more cooperate tone. The german version can be found at my company’s website. </blockquote> Authentication is hard. Especially when the traditional concept of a trusted network becomes increasingly outdated with remote employees and the security aware concept of Zero Trust. Nowadays every entity composes a potential threat. Implementing authentication into every application (even into internal ones) becomes a significant overhead, consumes valuable development time, and drives up cost. We would like to give a top-level overview of modern authentication methods, where these can be used, how they can improve the security of offered services and how they can even help to improve the usability of services. The need and pains of authentication</h2> Authentication is needed everywhere. Today there are distinct accounts for every individual which hold personal information. With data becoming more and more valuable and sensitive, it’s a company’s duty to protect user data from unauthorized access. Failing this can result in a massive lost in trust and reputation while also eventually result in legal consequences. This also includes HR information and logged working hours of employees. Managing these information becomes increasingly difficult with the number of different applications deployed. Keeping employee’s data (such as address, accounting, and social information) safe, updated and in sync between multiple applications pose a considerable challenge, due to the duplicated data handling nature of these systems. When offering services for end users the same problems occur. Users nowadays have lots of different accounts. When there is a change in their life, like moving to a new city, changing banks or an update in their relationship status, they are not likely to update all their information on every platform they use. Inaccurate data lowers the quality of the accumulated information and my also introduced cost due to an incoming service ticked caused by a failed bank transaction. In a well disingenuous system, there would be a central location where each and every app that needs user data can request them or even enriches them by adding additional information. A 3rd party application could request access to a user’s posts and contacts, do analysis on them, provide users insights about their contents, and offer recommendations for expending their network. There is an increasing awareness to the value of personal data and some users might not be willing to provide these information to any app. When users are promoted with lengthy registration process, for a new app, they are much more likely to discard dealing with the app compared to a quick an easy setup where they can log in with their existing accounts, like a Google account. The lack of an uncomplicated, quick sign-up process could lead to a significant reduction in the user adoption rate, which could, depending on the business model, reduce the generated revenue stream. Beside the logical and business aspects of user authentication there is also the technical aspect. As mentioned at the beginning, authentication is hard. Building it from scratch is only recommended in the rarest cases and when having skilled and experienced developers. They have to consider hashing, salting and storing credentials in a secure manner. There a plenty of news</a> where companies, even large and reputable companies, faced data breaches, exposing credentials and sensitive personal information about their users. Developers can get 99% of the authentication process right but render their whole implementation vulnerable by making one mistake. Even when authentication is handled by an external library, developers have to consider things like security questions, password rests and 2 Factor Authentication (2FA) with services like One-Time-Passwords (OTP) and hardware tokens. They also have to constantly update and review the used library in order fix disclosed vulnerabilities and to prevent supply chain attacks. Complicating things even further, the combination of username and password is not considered best practice anymore, due to cumbersome nature and susceptibility to being stolen. That’s why companies like Google are researching and pushing new authentication methods, like passkeys, and want to get rid of the traditional password. Considering and implementing all these aspects while designing and implementing an app introduces a considerable amount of business and application logic, especially if one decides to presume the path of a microservice architecture service. The good thing is that there are a lot of people facing this problem and that there are existing solutions to streamline, enhance and simplify the authentication process for every entity involved. The most promising and widespread approach are the OAuth2 and OIDC approach. Introduction to OIDC and OAuth2</h2> Both OAuth2 and OIDC provide concepts of how one can grant controlled access to specific resources. The OIDC specification restrictions its scope to only authentication, meaning it will only guarantee the legitimacy of the user through an external identity provider. Access control for limiting theses authenticated users to specific subset of resources is explicitly not part of OIDC and must be handled by the apps logic. This is helpful if there is no need for fine grained access control, if just the membership to the organisation must be assessed or if the app already has a system for fine grained access controls implemented. A prominent example is Kubernetes. The users authenticity can be provided via OIDC the access control is handled by Kubernetes RBAC system. The OAuth2 authentication spec is mainly described in the rfc6749</a> and covers both authentication and authorisation. Therefore it is quite broad and even has some sub specification and several extensions. For simplicity, this article will only focus on the main aspects of this spec needed for understanding and implementing it. The key concept of OAuth2 is that it provides an authorization framework where the authentication is handled by an external provider, often called the authorization server, where the client (mostly an app or the browser) can request access to information of the resource owner (user) on behalf of that user to access protected resources. The important aspect in OIDC and OAuth2 is, that the user does not log himself in on the main app or website, but is instead redirected to one of the many existing authorization provides</a> like Google, Microsoft or Facebook. While logging in, the user can see the type of information that the client wants to access (called scope) and can either approve or decline them. By design the client and any external resource never gets to access the user’s credentials and the entire authentication logic is handled by the authorization server. So there is no need to store passwords or implement passwords resets and 2FA yourself. The OAuth2 spec offers different authentication flows, called “grants”, designed for different use cases. The most common flows, and the ones covered in this article, are the Authorization Code Flow and the Client Credentials Flow. The visualization below shows the Authorization Code Flow, which is the grant most used for interactive user Authentication. When the user visits a company owned Login-Page he can choose to Login with <My-Auth-Provider>. This button takes the user to the Login-Page of that provider where he can login and give consent to the requested information. On confirmation, the user is redirected back to the original site with an Authorization-Code embedded in the URL. Now the company app can use that Authorization-Code, together with other configuration parameters to request an access and refresh token from the Auth-Provider. Using that access token, the app/website can request information about that user or perform actions on behalf of that user, without ever knowing the user’s password. The allowed actions are limited to the degree that the user has agreed to in the beginning. The access token has typically has a very short lifespan and regularly expires. New access tokens can be seamlessly requested via form the authorisation server. This significantly reduces the risk of leaked or stolen credentials while also allowing to easily revoke a users-session. Even if the theoretical procedures seem complex at first glance, the implementation is relative straightforward. In addition, this setup saves all the complexity required for a user registration process, password storage and reset, mail verification, 2-factor authentication, and maintenance. The app/website does not need to store any information about the user, since it can always request up-to-date information. So it does not need to worry that the address may be outdated. Due to the short lifespan of each access token (default is 2h), the users information security is also improved as the risk of session hijacking is significantly reduced. Only the original app can request a new access token with its configuration and the refresh token. The second grant mentioned is the Client Credentials Flow. It is designed for use cases where no user interaction is needed and where both, client and resource server, is owned by the same entity. This method allows two application to communicate safely with each other, while also providing the advantages of short lived tokens and scoped access. This concept can be extended even further. With federation one can enable a trust relationship between two systems. When this is used there is no need for any credentials anymore. CI systems are ideal candidates for such a setup since it allows secure and low maintenance communication between systems like GitHub Actions to AWS or Azure. Whats in for me?</h2> The potential savings that can be achieved through OAuth2, especially in enterprises, are enormous. For example, using OAuth2 in conjunction with Microsoft 365 would allow a company to secure and manage all of its systems with one central user account, such as the existing Exchange/Microsoft account. Users would have significantly fewer passwords and techniques such as Single Sign On (SSO) could significantly reduce the average number of 12 daily logins (found out by security-insider.de</a>) and the associated time required. Workers will be more productive and less annoyed by re typing passwords all day. Upon termination of employment, the responsible parties only have to block one account instead of going through long lists of accesses and deleting them with the high risk of overlooking something. The onboarding of new employees is also significantly shortened because only one account needs to be created and the new employee can then access all the necessary systems. Because of these advantages, the reduced effort required for authentication, the additional security and the convenience for users, the use of OAuth2 continues to grow. Many applications and services such as Jira, GitLab and others already offer built-in support for OAuth2. Even if applications do not support this, such as internal wikis or applications that do not inherently provide authentication, this can be implemented with little effort. The idea to offload the entire user management might sound a little scary at fist, specially if it is an external identity provider, but since OAuth2 and OIDC are well defined public specifications with a broad adoption there is no risk of a vendor lock-in. If a Microsoft or Google controlled identity store ist not an option, there is a wide range of alternative solution to implement an identity provider. Some of them fully open-source and self-hostable. Well established solutions are KeyCloak and DEX. Especially in the cloud environment, with many microservices, where Zero Trust is becoming more and more standard, a Kubernetes application can be made OAuth2 ready with just one manifest and thus offer its employees and customers faster, more secure, and better services. The whole idea of OAuth2 is to push the One Identity concept for each user. There should be no additional accounts, the user has a central authority with secure, reliable information and this identity will be authorized to access the necessary resources. This identity concept is provided by the OpenIdConnect (OIDC) standard, which is part of the OAuth2 specification. If your company struggles with authentication and employees are frustrated by the current status reach out to one of our experts at ARES Consulting GmbH or contact info@ares-consulting.de to make your application ready and secure with the future of authorization. We simplify authorization! What is new in containerd 2.0 2024-11-10T00:00:00+00:00 What is new in containerd 2.0?</h1> Containerd recently released its first major update since its 1.0 release form over 7 years ago. It provides some exciting new features, improves compatibility, gets rid of some deprecated functions and may even improve performance. This post aims to give a high level overview of the changes made in containerd and takes a closer look into the user-namespaces functionality. ℹ️ Containerd is a Container-Runtime-Interface (CRI) compliant implementation that can manage the entire lifecycle of an container. It is used in Docker and most Kubernetes flavors (including AKS/EKS/GKE) to manage all container processes. </blockquote> How do I upgrade</h2> The containerd project is committed to keep the upgrade path as simple as possible. Most settings should work the same as containerd v1.7x</code>. When you use the deprecated aufs</code> snapshotter (most likely you are not) you have to switch to a new snapshotter. Another change is that you have to explcitly re-enable the image schema V1 if you still depend on those. Beside that there should be little changes that require attention. To upgrade to the newer config version v3</code> you can just run containerd config migrate</code>. New functions</h2> The GitHub Release</a> from containerd 2.0 gives a first overview of the changes, but does not explain the implications of each change. I piked a few highlight and will cover them in more detail: User-Namespaces This is a big one for me and will increase security of containerized workloads by a lot! Why is it important? </blockquote> If you run a container today you can set the UID</code>, the user UserID as which the main container process runs. Every single security guide regarding containers recommends to run container processes as none-root and as a unprivileged user, which IDs start at the 1000 range. For most applications this is not an issue, but there are certain processes that require root. If processes need to install packages they require root, CI-Jobs often require root and it is really hard to run an ssh</code> server as none-root. Even one of the most used containers, nginx</code>, start as root and then drop their permissions to a none privileged user. The problem with root is that if an attacker manages to escape the container sandbox he is automatically root on the host system since UserIDs in a container map one to one to the ones on the host. User-Namespaces changes this. If you ran a container with the user-namespaces feature enabled a container can run as root, but has a different, unprivileged UserID on the host. This is a great feature that drastically improves security without a performance penalty. The 2.0 release of containerd supports user-namespaces by default but you may still have to enable it in your kernel via user.max_user_namespaces=1048576</code> (Suggested values from CoreOS for CRI-O). While the feature is now shipped with containerd, it is still in beta for Kubernetes. Your can enable it by setting the featureGate</code> UserNamespacesSupport</code> to true</code> on the api-server. To start a pod which uses user-namespaces set the hostUsers</code> parameter to false</code>. Thats all you need to do. This can also be enforced via admission controllers. 1apiVersion: v1 2kind: Pod 3metadata: 4 labels: 5 run: user-namespaces-demo 6 name: user-namespaces-demo 7spec: 8 # Enables user-namespaces 9 hostUsers: false 10 containers: 11 - image: nginx 12 name: user-namespaces-demo 13 resources: {}</code></pre> Intel ISA-L’s igzip This is a nice addition which may increase your container startup times by a few hundred milliseconds. But how? Container images are just a bunch of gziped archives layered ontop of each other. A manifest specifies which archives belong to which image. When an image is pulled it gets transferred via HTTP over the network and has to be extracted and decompressed. Most of the time this is done via gzip</code>. But gzip</code> is single-threaded and does not use the full potential of modern processor. To overcome this there is also a multithreaded implementation of gzip</code> called pigz</code>. While pigz</code> is faster then the default variant, igzip</code> is even more performant. Acording to benchmark by simonis</a> igzip</code> is twice as fas as gzip</code>. If you want to profit from this improvement just install igzip</code> (debian package is called isal</code>), containerd automatically check which version is installed on the system and picks the most preformat one. Container Checkpoints This is more of a debug function, but with containerd 2.0 you now can freeze an entire container at is current state, including all memory and running processes, and export it. The container now can be restored or further inspected. You can read more about this on the criu-wiki</a>. Notable mentions The config format version changed to v3. You can migrate it with containerd config migrate</code></li> Containerd now allows container processes to bind to privileged ports by default.</li> Enables the Container-Device-Interface (CDI) by default</li> Deprecated aufs</code> snapshotter was deleted</li> </ul> Making OnPrem Kubernetes feel like AKS/EKS/GKE 2024-10-28T00:00:00+00:00 Improving OnPrem Kubernetes & making it feel like managed k8s</h1> Managed Kubernetes services like AKS, EKS or GKE are awesome. The hyperscaler’s take away so many annoying and difficult tasks. When you use a managed Kubernetes offering, you don’t have to deal with the controlplane, no etcd backups and provisioning of new nodes is fast and painless. A very nice feature of these managed Kubernetes services is that you can just use the container images from your hyperscaler’s container registries without having to create a image-pull-secret</code> in every namespace and deployment. Not having to deal with this saves hours of mindless work and saves you from stupid errors that nobody wants to deal with. But how do the hyperscaler’s do that? See, AWS, Azure and Google also just cook with water, it’s no magic. Read on how you can make your OnPrem feel a little more like your-generic-hyperscaler. The kubelet-credential-provider-api</h2> What is the kubelet-credential-provider-api? It is a very small interface where the kubelet (the agent that runs on every kubernetes node) invokes a small program and passes a well-defined CredentialProviderRequest</code> to it. The program is then expected to respond with a CredentialProviderResponse</code>. This response contains the registry credentials which the kubelet then passes to the container-runtime-interface (CRI) which pulls the desired image from the target registry. This way, users can create deployments with images from password protected registries without having to specify any secrets, since the kubelet just pulls the credentials dynamically. But how does the credential-provider</code> get the registry credentials? That actually depends on the cloud you are on. On AWS the ecr-credential-provider</a> uses the IAM role of the ec2 node it is on and then requests a shot lived token from AWS’s secret-token-services for the target registry, if the IAM policy allows it. It’s pretty much the same on Azure, while their credential-provider</code> is not open-source, it is safe to assume that it uses Azures managed identities to acquire credentials for the desired registry. You can read more about the credential-provider-api on the Kubernetes Docs</a>. How can I profit from that?</h2> It is quite easy to implement the kubernetes credential-provider-api</code> yourself. And suddenly no developer needs to provide a password anymore if they pull images for you common-internal registry or from your DockerHub mirror. Even if you use docker.io</code> directly, it is easy to reach the 100 pulls per 6h pull limit. But if you use a credential-provider, the limit is immediately doubled without any work for the devs. Since it is so easy, I did it myself. I created a the kubelet-static-credential-provider</a> which runs on every architecture that is supported by Kubernetes. All you have to do is download the binary on every kubernetes node, create a small config and add the following arguments to your kubelet: --image-credential-provider-bin-dir=/var/lib/kubelet-plugins/</code></li> --image-credential-provider-config=/srv/kscp-conf.yaml</code></li> </ul> The kscp-conf.yaml</code> should look like this and can contain as many credential providers as you want: 1# /srv/kscp-conf.yaml 2apiVersion: kubelet.config.k8s.io/v1 3kind: CredentialProviderConfig 4providers: 5 - name: static-credential-provider 6 # You can also use a config file instead of envs 7 # args:[--config, <path-to-password-conf>] 8 matchImages: [docker.io] 9 defaultCacheDuration: "12h" 10 apiVersion: credentialprovider.kubelet.k8s.io/v1 11 env: 12 - name: KSCP_REGISTRY_USERNAME 13 value: <my-user> 14 - name: KSCP_REGISTRY_PASSWORD 15 value: <my-password> 16 # Optional: More credential providers</code></pre> Now I doubled my DockerHub pull limit and can use private registries without any additional work for any devs. And let me say this, they will love you for not having to deal with these secrets. By the way, this is also a good method to bootstrap a cluster where the images are in private registries. NOTE: My static-credential-provider</code> is written in Go and uses the native Kubernetes types. But since it is such a simple interface, you can easily create the same functionality in a bash script when type safety is not so important. Don’t believe me? I also did a bash version, check it out here</a>. A word about security</h2> I already hear the screaming. THIS IS NOT SECURE! This is bad practice!!1! Yeah, my implementation expects credentials that are on every node in plain text. But if some attacker gains access to one of your Kubernetes worker nodes you have far bigger problems to deal with. When an attacker has access to a node, he can get to the information and credentials on every container on that node. He can see all traffic and export any data on that node. Your infrastructure was compromised long before the attacker got access to the registry credentials. Even more important: Container images are not considered a safe place in the first place. If there is anything sensitive in your container image you are doing things wrong! My advice is to create credentials that are read-only. So even in the case these leak, they can only be used to pull images with some application code that is to no use for anyone outside the company. It is also entirely possible to authenticate just a few registries with this method. The matchImages</code> attribute allows a list of images for which the kubelet should invoke the credential-provider</code>. You can even use patterns like docker.io/company-xyz-common/*</code> in each entry to have fine grain control over when to use this functionally. Conclusion</h2> If you have a self hosted Kubernetes cluster, this is an easy way to boos the devs productivity. It also allows pulling images from private registry in the bootstrap process when you can’t use image-pull-secrets</code> yes. Implementing this is quite easy. Feel free to use my static-credential-provider</code> or write you own version. Even a bash script can be enough and I also provided some alternatives in the Readme of my implementation. You can check it out here: kubelet-static-credential-provider</a> New Website - Abandon JavaScript Frameworks 2024-09-01T00:00:00+00:00 New Website - Abandoning JavaScript frameworks</h1> This is the third generation of my website. I went from raw handwritten HTML to falling into the JavaScript frontend framework pit, back to raw HTML with a little templating. Let me tell you about this journey and my thoughts. Be aware that I’m not a web-developer and I will never be one! My First website - Getting things out there</h2> I was still in university and wanted to get some practical experience. My university had zero lectures about web-development, but I thought having my own site would be cool. I also had never shipped any software yet, so I had no clue how to put something out there for others to use. So I found a nice template I liked and customized it to my likings. HTML is not that complicated, and I didn’t need any JavaScript for this simple static site. So building the skeleton was quite easy, unitl I wanted to style it with CSS. I have worked with various programming languages, also low level languages like C/C++ but I’ve never had to fight this much to get things working. I get the idea of classes but to this day I never fully understood the global scope of styles, overrides nor am I even close to know even a subset of all the CSS properties and transformers. Eventually I had something usable, I needed to get it out there. I bought my first domain from the small German hoster IONOS - which was exiting back then. Now I was looking for a free hosting solution and eventually found netlify</a>. I needed to put put domain and hosting together. DNS at IONOS sucks (like almost everything from their offerings) so I transferred the DNS zone to netlify and started battling a little with TLS. Yet, eventually I got to a state that was fine for me. Still the site had issues. Whenever I wanted to add a project I copied a bunch of HTML and added the new text. The header/footer was manually copied to every page and the CSS was looking okay, but was actually a mess. But it worked for back then. Check out v1</a> Getting into professionalism - Frameworks</h2> I was still in university but was working almost full-time at a local software company. My task there was to modernize and automate a lot of the deployment stack. I was given a lot of trust and I learned a lot. Things that work and things that don’t work. Occassionally I helped out on one of there products which was a NodeJS backend and a VUE.JS frontend which used nuxt to facilliate server-side-rendering (SSR). Everyone on YouTube and Medium was talking about frontend Frameworks for the web. Everything seemed to be build with Angular, React or VUE. So I belived this is the way to build the web. I fully bought into it. So I started to rebuild my site with nuxt. Being able to reuse components for header and footer was great, but the whole creation process seemed a bit off. Passing object variables to a VUE component as a string is strange. The complexity added up. I wanted to publish blog posts written in markdown - no problem with nuxt-content. To style it use bootstrap, vuetify and more. Want to add approximation for reading time? Yet another package. Which you have to integrate with webpack. I did’t want to learn webpack, I wanted to create a webpage with a blog. It all seems unnecessary complicated. Having to keep in mind which part of the app was rendered server-side and which client side added additional cognitive load. But since everyone is using the technology, it seems like this is the way. Yet, eventually I got to a state that was fine for me. Still the site had issues. Load times where a not good, the style was inconsistent, code-blocks looked awful and I never got it working that the site remembered if had dark-mode activated on refreshes. But it was good enough. I used that site for almost three years. It worked for writing articles and I prioritized other things. Check out v2</a> A fresh start - escaping the JS dept</h2> The site was automatically build from the main branch via netlify. Dependabot was setup to update my packages. But the JavaScript world is fast moving and a lot of people don’t bother with backwards compatibly. With time nuxt3 was released, but nothing worked with it. The site was using vue2, bootstrab4 and webpack4. Everything got deprecated and Dependabot kept reminding me of updates and vulnabilities. The site was slow, full of outdated packages and vulnerabilities! To this day I carefully select my JS packages and avoid dependencies when ever possible because I think the JS ecosystem is a mess. So manny packages like leftpad, is-even and more - WHY? I was looking for alternatives quite some time. I did’t want to go down the JS framework road again, mainly because I did’t need most of their functions. I wanted a static site generator. I looked into hugo which is great but I did’t find nice themes and found the themes creation process to much for my task. Zola to the rescue!</h4> Eventually I found zola</a> by browsing on the webpage of a colleague. It looked great, entirely content orientated: You can have themes, but you don’t need to</li> First party support for markdown rendering</li> Single small binary</li> Build & refresh is fast as heck</li> Support for SASS</li> Auto generation for search index</li> </ul> So I started porting over my first page as a proof of concept. And it worked great! Exactly the feature-set I wanted and nothing more. Templating is done with terra</a>, a python jinja2 like templating language which I was quite familiar with thanks to my countless hours in ansible. I had not a single issue with it. At least not with zola. CSS on the other hand reminded me why I will never be a frontend-developer. Getting things to look nice on every device is hard - even with bootstrap. If I could change one thing regarding zola, it would be better control on how to pass classed into rendered content and more built-in code syntax highlight themes. The current state</h2> I moved my back on JavaScript frontend frameworks - at least for static content. The whole site is now build with zola and adding new stuff is super easy. I also moved to site to cloudflare, better metrics, performance and I will not be billed 50k if I get DDOSed. The site works - even without JS enabled, the only thing that does not work without JS is search and the particle effect. I’m quite happy with it right now. I still need battle some fights. But for my own sanity and well bing I’m shipping early and often to keep seeing progress. Things to do: Fix sidebar layout on blog page ✅</li> Add optional Live Demo button on some project</li> Dark Theme ✅</li> Add Search ✅</li> Write more 🔄️</li> </ul> Some stats between my site: </th> RAW</th> Nuxt</th> Zola</th></tr></thead> JS</td> 140KB</td> 4.2MB</td> 156KB</td></tr> HTML</td> 544KB</td> 13MB</td> 1.4MB</td></tr> Build Time</td> 0s</td> 14s</td> 2.2s</td></tr> </tbody></table> 🧾📖 NOTE: You might want to check out v1</a>, v2</a> and v3</a> of my site. Benchmarking what actually drive our containers 2024-07-01T00:00:00+00:00 The Engines that run our Kubernetes Workloads</h1> 🧾📖 PDF version</a> Target audience: This article is a deep dive for people living the Kubernetes lifestyle, for people who know or want to know how the low level stuff works and performs. I do not explain every container or Kubernetes component as it is expected to be known. Kubernetes has become THE standard for container orchestration. It is not just a software tool, it is a framework with extensive extensibility features. Entire businesses are built on top of Kubernetes and offer essential, nice to have and abstruse services. Tools like ArgoCD and Crossplane are build for Kubernetes and changed the way how we deploy software. There is even a new category of operating systems like CoreOS, Bottlerocket and Talos that are build purposely to run containers. All that happened in just 10 years. Happy late birthday K8s! Kubernetes is dominating almost every Ted-talk and system architecture presentation, yet the actual engine that drive our workloads sometimes gets forgotten. Kubernetes is just an orchestrator, and its primary (but not only) task is to manage containers. It does not run any containers itself. That gets delegated to the container runtime interface. A Little History</h2> Initially, Kubernetes was build to manage Docker Containers. Docker is the technology that made containers accessible to the mainstream. Containers were not an entirely new technology. LXC containers existed before and Linux namespaces are a thing since 2002, but Docker made it so easy to use containers. By handling software packaging, distribution and all low level network and device configuration, Docker allowed every developer to start any application in an isolated environment. Promising to finally overcoming the famous: It works on my machine meme. Even the Kubernetes core team admits that Kubernetes would’ve been such a success without Docker. In an alternative universe, we are all using Mesos</a> 😉 At some point, a lot of different people and companies tried to solve similar problems regarding containers. Unlike some dinosaur companies with mediocre stubborn management, they agreed to build some common interfaces for their solutions so that parts of their implementations could be swapped while relying on certain standards. This led to the creation of the Open Container Initiative (OCI), Container Network Interface (CNI), Container Storage Interface (CSI) and Container Runtime Interface (CRI). Today, I want to talk about the Container Runtime Interface (CRI). With version 1.24, Kubernetes has overcome its dependence on Docker and now only relies on any CRI compliant runtime to start containers. And there are quite a few. CRIs</h2> The most common CRIs are containerd, CRI-O and Mirantis Container Runtime. Of these three, containerd is the most wildly used option. The containerd project was originally developed by Docker Inc and still is the core of Docker, but its ownership was since donated to the CNCF. When a new Pod is scheduled to a node, the kubelet communicates with containerd over the container runtime interface to actually pull the image and start the container with the configuration specified in the pod definition. Containerd is now in charge to set up the Linux namespace, mount volumes or devices and apply seccomp profiles and much more for every new container. That’s a lot of tasks, and not quite according to the UNIX philosophy to do one thing and do it well. Digging deeper</h3> So containerd doesn’t do that all on its own. It uses something I call the container engine (others also call it OCI runtime, but I believe this would cause some confusion). By default, containerd calls runc</code>. The runc binary actually spawns the container process with all its applied properties. It does not pull images, manages the containers file system, sets up networking or monitors the container process. Overly simplified it is just a spawn</code> syscall with a bunch of extra properties for some isolation, but it still runs on the same kernel since containers are NOT VMs. Most of the tooling for Kubernetes and containerd is written in go. Go is a great language, but some people argue that it is not the best choice for low level system functions. Discourse and different opinions are great and drive innovation. And since a bunch of smart people choose to standardize container interfaces, everyone can create their own OCI compliant implementation and use it with the existing CRIs and tools. And that is exactly what people did, and I will take a look at some of them and compare them. Benchmarking Container Engines</h2> Containers are created all the time. We don’t even think about them anymore when we schedule a deployment with 42 replicas. But since it happens so frequently, we might take an occasional look at what actually happens at a low level and how it impacts our cluster’s efficiency. Do container engines impact start-up time and memory consumption? Since I didn’t find any real up-to-date comparisons, I took a look for myself and ended up comparing these implementations: runc - The default OCI engine written in go that comes with containerd</li> crun - Alternative OCI implantation written in C, claiming to be -49.4% faster than runc for running /usr/bin/true</code></li> gvisor/runsc - Googles safe OCI implantation (also written in go) that aims to improve security by running many syscalls in userspace</li> youki - Quiet young alternative OCI implantation written in rust</li> </ul> Testsetup</h3> Disclaimer: This is NOT a scientific benchmark. I’m quite a noop in profiling such applications, and I cannot claim full correctness for my findings. If you know how to do it better, please show me. I created a VM on the Hetzner-Cloud using their dedicated offerings to avoid the effect of noisy neighbors. I redid the test on another instance to ensure I didn’t get a low-performing one. System Data: 1region: nbg1-dc3 2os: Linux 6.1.0-21-amd64 Debian 12 (2024-05-03) x86_64 GNU/Linux 3sku: 4 type: CCX33 5 cpu: 8 6 ram: 32</code></pre> The test environment was configured with an ansible playbook and my personal Kubernetes role</a> I use to play around. My benchmark script uses the time</code> tool to measure execution time and calculates min, max and mean for 1000 container creations for the busybox image running /usr/bin/true</code>. I use nerdctl</code> to create containers rather than calling the OCI implementation directly, since it mimics a more realistic use of an CRI. Before every test, I run a small warm-up to allow the CPU to boost clock speed. I found it hard to measure CPU and memory consumption for a short-lived process, so I installed node-exporter and have it scraped by Prometheus (running on another system) every 5 seconds. 1containerd version 1.7.18 2crun version 1.15 3runc version 1.1.13 4runsc version release-20240617.0 5youki version 0.3.3</code></pre>Results</h3> The node’s CPU and RAM were not even remotely utilized as seen in the Grafana dashboard below. In contrast to the production environment where the kubelet can create several containers simultaneously, my benchmark runs in serial. Nevertheless, some conclusions can be drawn. The test showed indeed that crun outperforms containerd’s default OCI engine, yet the difference is not as big as claimed on the crun website. However, you have to bear in mind that it was not the same test as on the crun page. The performed benchmark includes the overhead of containerd, to represent a more realistic deployment compared to calling the OCI engine directly. Given the claims from crun I was expecting a bigger difference between crun and runc. A 21% difference is still impressive. Surprisingly, the quiet new youki implementation is right up there head to head with runc when it comes to pure performance. Unfortunately, youki didn’t perform as consistent and errored out quite some times. I reported the error to the authors and hope it will be fixed soon, despite its good performance an error rate of 3.6% is not acceptable in a production environment. As expected, runsc performs worst by far. However, performance is not its primary goal, it aims to improve security by implementing syscalls in userspace. The chosen test may also not favorable for runsc since running a short-lived container with true</code> does not have many syscalls, the lower context switche are still noticeable as shown in the last dashboard below. The main findings are summarized in the table below. OCI</th> Mean</th> Min</th> Max</th> Errors</th> Total Time</th> Min Mem</th></tr></thead> crun</td> 0.28s</td> 0.13s</td> 0.41s</td> 0</td> 306s</td> 160kb</td></tr> runc</td> 0.34s</td> 0.16s</td> 0.44s</td> 1</td> 362s</td> 8mb</td></tr> runsc</td> 0.62s</td> 0.44s</td> 1.64s</td> 0</td> 642s</td> 32mb</td></tr> youki</td> 0.32s</td> 0.16s</td> 0.46s</td> 36</td> 354s</td> 2mb</td></tr> </tbody></table> Memory was hard to test, and I found no significant differences here. From the data node-exporter provided it seems like crun and runc use a little less memory compared to the others, but I can not prove that no other process caused the increase, even when I did my best to keep variations low. A claim that I was able to verify is that cruns authors say it requires far less memory to start a container than runc. I tested how low I could set the memory limit via the CRI for each engine via nerdctl run --memory 4mb ...</code>. The authors claim it requires less than 4mb, which is the lowest limit podman allows. I was able to start busybox with a memory limit of just 160kb. This is indeed impressive, even though it is not directly relevant in a production environment, since most applications within a container will need and have much higher limits. Honorable findings</h3> I wanted to compare the runtime and memory performance of these OCI implementations, but when monitoring, I also noticed some spikes in IO. I did not expect that starting a small container like busybox would cause some significant IO writes, and I don’t really know why write activity is “so high” for just starting a container. Both crun and runc caused around ≈700-800 IOPS and ≈4-4.5 MB/s in writes. Slightly less was needed for youki and runsc was using just half of IOPS and MB/s in comparison with crun and runc. According to its promises, runsc also caused fewer context switches for syscalls. I’m quite interested in learning more about the “high” writes, though. Especially where the differences come from when the same snaphotter is used across all OCI engines. The complete raw output can be found on my GitHub Gist</a>. Conclusion</h2> The performance differences between the tested OCI implementations are not as big as I expected, and even though crun performs better, I doubt that switching to it justifies the operational overhead for most companies - at least for managed Kubernetes offerings. As far as I know most managed Kubernetes offerings use containerd with runc. Being able to just use AWS managed AMIs (or Azures) is a huge time and cost saver. Unless the last percent of performance is highly important, I would just stick with runc. It would still be nice to see hyperscaler’s include crun in their AMIs (which costs 2MB of space) and give users the choice via runtimeclasses. For self-hosted and embedded Kubernetes offerings, crun might be worth a look. Even more if companies are running cutting edge technology like WASM within their clusters. Both crun and youki can be compiled with WASM support. Meaning users can run WASM applications in their clusters with the same OCI as all their other containers. Which is SICK! I’ve been using this for quite some time and it works great. The only downside is that WASM support is not enabled by default yet and you have to compile it yourself. (I’ve set up an GH Action automation just for that.) The first look of youki is promising, but without having these errors solved, I wouldn’t use it in production. When it comes to gvisor/runsc I’m a little unsure. It offers additional security, but most people run their own or trusted workloads in their clusters. If one would run random untrusted code, like someone’s leetcode or lambda code I might want to use things like kata containers or firecracker VMs which offer a higher level of isolation. If someone wants to include these in the tests, I’m very interested - I couldn’t run these since they require nested virtualization, which is not supported on Hetzners CCX VMs. This leaves gvisor in some weird middle ground, especially with Kubernetes upcoming ability to run containers in user namespaces further increasing security. Innovation in the container world has settled down a bit since a lot of the technology as reached mainstream adoption and has standardized. The existing solutions have become more accessible, and the defaults cover the needs of the majority of use cases. Yet, it is still interesting how new solutions hit the market, resulting in a drop in replacements for solving specific problems. Using GitLab to manage Kubernetes access 2024-06-12T00:00:00+00:00 Using GitLab to manage Kubernetes access</h1> Recently I have been tasked to share access to a Kubernetes test cluster with other teams. Easy: Add them to a group in your OIDC</a> provider and set appropriate RBAC rules. Task done - no need to write this. BUT:</h2> Unfortunately I can’t have an OIDC provider for non-production clusters… because. Sighs… TL;DR</summary> Use Kubernetes built-in OIDC authentication; if this is not possible for whatever reason, the GitLab Kubernetes Agent might be worth a look. </details> Okay what other options are there for Kubernetes authentication: Webhook Token Authentication</li> Authenticating Proxy</li> Static token file</li> X509 client certificates</li> </ul> Both webhook authentication and an authentication proxy add considerable overhead and ultimately also need to be plunged into a user registry that I can not use or have control of. So what about static token files? In practice, I’ve never seen them outside of test setups. It requires placing a file with the username, secret and groups on every controlplane node. The effort required to manage and sync such a file, containing static credentials in plain text, is just nuts. Even more absurd, the api-server has to be restarted every time this file is changed! And since these tokens have no built-in expiration date, you have to delete an entry and restart the api-server if a credential needs to be invalidated. So this is a hard pass! So X509 client certificates it is? The X509 client certificates approach does not require the api-server to be restarted and provides a built-in expiration mechanism. Client certificates are a common approach to establishing a secure communication between two entities, it is often also referred to as mutual TLS (mTLS), where both the client and the server have to identify themselves via a certificate. One-directional TLS (where just the server is ensuring its authenticity) is part of almost every website we visit. TLS is considered very secure since, it uses asymmetric encryption with private keys, with a complexity between 1024 and 4098 bits, depending on the algorithm. Kubernetes uses mTLS for almost all of its components to establish trusted and secure communication between every part of the cluster. Kubernetes clusters created with kubeadm also create a admin.conf</code> which also uses X509 client certificates to authenticate against the Kubernetes api-server. These X509 client certificates are the default way to authenticate against a Kubernetes cluster. For easier usage, Kubernetes base64 encodes the certificate and the private key and puts them inside the well-known YAML kubeconfig. The file is most likely located at ~/.kube/config</code>. I have gone the X509 client certificates way and if you are interested, below is a rough draft on how you can automate user config creation. But there is a catch too Sighs…x2 Creating X509 client certificates</summary> Every user should have their own personal kubeconfig to ensure proper access controls and auditing capabilities. Unfortunately there is no built-in support from Kubernetes for creating these kube-client-configs. Nevertheless the process can easily be automated. The user has to create a new key-pair with openssl which he will use for accessing the cluster later. This can be done with the following command: 1# Bits for the key 2KEY_SIZE=4096 3# Username 4USERNAME=MY_USER 5# Groups 6GROUP_1=TEAM_A 7GROUP_2=TEAM_B 8 9# Creating key and certificate signing request 10openssl req -new -newkey rsa:$KEY_SIZE -nodes -keyout k8s-client-1.key -batch -out k8s-client-1.csr -subj "/CN=${USERNAME}/O=${GROUP_1}/O=${GROUP_1}/emailAddress=${USERNAME}@example.com"</code></pre> The resulting certificate signing request (CSR) now has to be signed by the clusters Common Authority (CA). The signing request can either be done wither the Kubernetes certificate api by creating a CertificateSigningRequest</code> or manually by copying the clients CSR to the api-server and signing it with the clusters CA via openssl. The first option is clearly preferable as it does not require access to the node on which the api-server runs on and there is a traceable audit log. The resulting certificate and the key are then base64 encoded and inserted into the format of the kubeconfig. The resulting file will have this layout (without the certificate data redacted): 1apiVersion: v1 2kind: Config 3current-context: docker-desktop 4clusters: 5 - cluster: 6 certificate-authority-data: LS0...S0K 7 server: https://Kubernetes.docker.internal:6443 8 name: docker-desktop 9contexts: 10 - context: 11 cluster: docker-desktop 12 user: docker-desktop 13 name: docker-desktop 14users: 15 - name: docker-desktop 16 user: 17 client-certificate-data: LS0...g== 18 client-key-data: LS0...o=</code></pre> NOTE: If you want a fully automated process for this just send me a massage! </details> The Problems with static credentials</h3> While mTLS is technical very secure, it still suffers from problems organizational standpoint. From a modern standpoint, certificates are still long-lived credentials. Thanks to the exemplary work done by the lets encrypt</a> project, most certificates are issued with shorter lifetimes compared to the previously common validity of 1+ years. Still, a default of three months is a heck longer than the one-hour default lifetime of an OIDC ID-Token. It is also possible to revoke certificates, but just like the creation process, it is not that easy to implement this process. Even when these security aspects are not considered, client certificates become hard to manage in larger or dynamic teams. You must have a way to revoke access from a retired team member. When a user needs to be added, the whole certification creation process has to be done again. Authorization should be done on group level basis, but you can’t add a group to a certificate after it was issued. This is not flexible and manageable in any larger organization. For this reason, dynamic authentication processes - like OIDC - are often preferred. I have a great article about OAuth2 and OIDC coming up soon on my companies website - stay tuned Alternatives</h2> So what are my alternatives? The said cluster in deployed in a very restrictive network environment. Ingress is only allowed from specific clients within a corporate network and only specific ports and traffic. Egress is only possible via a proxy. But we needed to deploy from our GitLab (which is on a different network) instance to our cluster. Other people solved this by creating GitLab runners within the Kubernetes cluster and saving the kube-client-credentials-conf as a CI secret. DONT DO THIS! You are pulling in random container images from the internet to your cluster and run arbitrary scripts from CI in your critical application environment. You are risking your cluster health, the confidently of secrets and are giving up a considerable amount of compute. And you also have to maintain the runners. This also violates the ISO 27001 norm since now access is not personalized anymore and auditing is really hard. GitLab Agent</h3> Quite some time ago I stumbled upon the GitLab Agent. It’s a small application that creates a reverse tunnel between your control-plane and GitLab instance via websockets. This allows you to use any runner in any network location to securely communicate over the GitLab Instance websocket tunnel, via the GitLab agent to your Kubernetes cluster. GitLab automatically creates short-lived tokens (CI_JOB_TOKEN</code>) and only allows authorized users to use the tunnel in CI. You can install the agent easily with: 1helm repo add gitlab https://charts.gitlab.io 2helm repo update 3helm upgrade --install test gitlab/gitlab-agent \ 4 --namespace gitlab-agent \ 5 --create-namespace \ 6 --set image.tag=v17.0.0 \ 7 --set config.token=glagent-xxx \ 8 --set config.kasAddress=wss://gitlab.example.com//-/Kubernetes-agent/</code></pre> Now you can create a file in you git repo with the path .gitlab/agents/<agent-name>/config.yaml</code> and even share the cluster access with other Git projects: 1ci_access: 2 projects: 3 - id: my-team/serviceX 4 - id: external-team1/projectA 5 - id: external-team2/projectB 6 groups: 7 - id: qa/loadtests</code></pre> No secrets needed, secure and controlled access to the cluster. WAIT…That wasn’t the problem I wanted to solve. I need to share access with people - not with CI! Good News: This also works for people. You can just add the following to your .gitlab/agents/<agent-name>/config.yaml</code>: 1user_access: 2 access_as: 3 user: {} 4 projects: 5 - id: global/groups/admins 6 groups: 7 - id: my-team 8 - id: qa/loadtests</code></pre> Now GitLab allows all users that are in these projects or groups to access the cluster endpoint via the agents tunnel. The important part in that config is the access_as.user</code> property. The Agent will use impersonation</a> to act like the GitLab user that uses the tunnel. That means you can easily set fine-grained permissions via RBAC to only allow the qa</code> group read access while the admins</code> have cluster-admin roles. If you are a masochist, you can also apply the RBAC rules per user. The Solution</h3> Now, all users can configure their kubeconfigs to use the cluster endpoint https://gitlab.exmaple.com/-/Kubernetes-agent</code> and use their GitLab credentials to authenticate. Basic permissions are handled by GitLab and the fine-grained Kubernetes permissions are controlled via RBAC. If a team member retires, just remove them from the GitLab group and the access is gone. A new team member needs onboarding, add them to GitLab and he already has access to the cluster. Easy management and even ISO 27001 compliant 🙌 Complete Kubeconfig</summary> 1apiVersion: v1 2kind: Config 3current-context: gitlab-cluster-via-agent-42 4clusters: 5 - cluster: 6 server: https://gitlab.exmaple.com/-/Kubernetes-agent 7 name: gitlab 8contexts: 9 - context: 10 cluster: gitlab 11 user: gitlab-agent-42 12 name: gitlab-cluster-via-agent-42 13users: 14 # Users can use a pat to access the cluster 15 - name: gitlab-agent-42 16 user: 17 token: "pat:42:XXX" 18 # Or use the glab cli for just in time credentials with a 24h expiry 19 - name: gitlab-agent-42-glab 20 user: 21 exec: 22 apiVersion: client.authentication.k8s.io/v1 23 args: 24 - cluster 25 - agent 26 - get-token 27 - --agent 28 - "42" 29 command: glab 30 env: 31 - name: GITLAB_HOST 32 value: https://gitlab.example.com 33 - name: GITLAB_CLIENT_ID 34 value: XXX 35 installHint: "To access the cluster install glab. Instructions at https://gitlab.com/gitlab-org/cli#installation.\n" 36 interactiveMode: Never 37 provideClusterInfo: false 38</code></pre></details> Side Notes</h2> This is not the only solution to access private clusters. But this did the job and security is fine with it. Jump Hosts, VPNs and SSH tunnels are also commonly used but they are often a nightmare to manage or to set up. A notable alternative is tailscale which also creates a tunnel to you tailnet and allows for user authentication. Unfortunately, tailscale is not used at my current gig. While these are solutions, I still would recommend to just hook your OIDC provider such as ActiveDirectory, LDAB or Keycloak into the Kubernetes api-server. It is also a much more standardized solution and should be the primary source for all user contexts. To my surprise not many managed Kubernetes offerings support setting the OIDC args for the api-server. So this might after all be a good solution if your Kubernetes provider is lacking. The recurring problem of the Kubernetes metrics server and insecure Kubelet certificate 2024-05-18T00:00:00+00:00 The recurring problem of the Metrics Server and Insecure Kubelet certificate</h2> What is this about:</h3> Currently the kubelet, the client agent coordinating container lifecycle on each node, also runs a server, using a self signed certificate by default. Therefore any client connecting the the kubelet server, most prominent the metrics server, will not be able to verify the servers identity. Current state:</h3> Currently if the kubelet starts and no --tls-cert-file</code> and --tls-private-key-file</code> are specified it generated a self-signed pair and uses it for its server component according the kubelet docs</a> The is a feature feature flag called RotateKubeletServerCertificate</code> which would allow the use of serverTLSBootstrap</code>. This uses the native CertificateSigningRequest</code> (csr) resource wich is part of the certificates.k8s.io/</code> api. When serverTLSBootstrap</code> is enabled the kubelet creates a csr via the kubernetes certificates api for its server component. Users can either approve the csr manually with kubectl certificate approve <my-csr></code> or create a rolebinding wich allows for auto-approve. The resulting cert is singed by the clusters ca if the kube-controller has access to the ca cert and key (most likely it has). The process is described in the tls-bootstrapping</a> documentation, this provides the prerequisites for a trust chain verification. The problem:</h3> With self-singed certificates there is no way to reliable verify the kubelet’s server identity. Even the metrics-servers docs says that passing --kubelet-insecure-tls</code> is not a solution for production. The recent addition serverTLSBootstrap</code> is an approvement but the control over the created csr is very limited to non existed. Most Kubernetes setups are not trivial. Nodes have multiple IPs and may be available over one ore move dns names, let alone IPv6 addresses. From my experience and according to other posts the created kubelet csr does not contain any SANs which could identify the server. The CN ist set to system:node:<name></code> wich is NOT a valid DNS name. This lack over the certificate details also results in the lack of the possibility to verify the identity of the kubelet server. Binging us back the the need for --kubelet-insecure-tls</code> for all clients talking to the kubelet. Even managed kubernetes offerings have this problem. The Proposal:</h3> Kubernetes admins and vendors should have an easy possibility to crate trusted kubelet server certificates. This can be achieved either via the certificate generation phase of kubeadm or via passing extra args to the kubelet tls setup to include additional SANs. 1# Existing behavior with etcd and kube-api-server 2apiVersion: kubeadm.k8s.io/v1beta3 3kind: ClusterConfiguration 4apiServer: 5 certSANs: 6 - "10.100.1.1" 7 - "ec2-10-100-0-1.compute-1.amazonaws.com" 8etcd: 9 local: 10 imageRepository: "registry.k8s.io" 11 serverCertSANs: 12 - "ec2-10-100-0-1.compute-1.amazonaws.com" 13--- 14apiVersion: kubeadm.k8s.io/v1beta3 15kind: InitConfiguration 16nodeRegistration: 17 name: "ec2-10-100-0-1" 18 kubeletExtraArgs: 19 node-ip: "10.100.0.1" 20 # Proposal: ether as an arg 21 serverCertSANs: "10.100.0.1,ec2-10-100-0-1.compute-1.amazonaws.com" 22 # Proposal: or as an own kubeadm nodeRegistration property 23 serverCertSANs: 24 - "10.100.0.1" 25 - "ec2-10-100-0-1.compute-1.amazonaws.com"</code></pre> Personal I’m in favor of adding an extra arg to the kubelet binary to include a list of additional SANs. This would also allow profit users not using kubeadm to have a properly configured kubelet server certificate. The Workaround:</h3> For personal testing setups I use ansible to prepare my nodes and bootstrab a cluster. One the certificate generation phase is complete I can generate my onw private key and certificate request and sing it with the clusters ca. Though I don’t recommend this for production, this is why we need a more managed and automated approach. References</h3> Kubernetes Docs TLS Bootstrap</a></li> Kubernetes Docs kubeadm troubleshoot tls</a></li> GitHub metrics-server error - missing SANs</a></li> GitHub kubernetes node - missing SANs</a></li> GitHub kubernetes singed serving certificates </a></li> Reddit - kubeadm not including SANs</a></li> </ul> Using AWS from GitHub without Credentials 2024-03-18T00:00:00+00:00 Using AWS from GitHub without Credentials</h1> AWS allows external services to authenticate without any secrets, which increases security and also reduces management overhead. There is no need to create, distribute, or rotate any secrets. This short guide shows how to set up and use this federated identity concept. Pre-Requirements</h2> To follow this guide, you need: A GitHub repo with read/write access. We will use my-org/demo</code></li> A AWS account that can create roles.</li> This example will use terraform</a> for IaC deployments. Other tools or the WebUI should also work.</li> </ul> Configure AWS</h2> You need to configure your AWS account as an identity provider. This can be done via the aws_iam_openid_connect_provider</code> resource or via the AWS console. To use that identity provider, you need to create a role (or use an existing one) and establish a trust relationship with that role and your GitHub repo. This can can be archived with a iam_policy_document</code>. In terraform, this would look like this: 1# Get the identity provider arn 2resource "aws_iam_openid_connect_provider" "github" { 3 client_id_list = ["sts.amazonaws.com"] 4 thumbprint_list = ["1b511abead59c6ce207077c0bf0e0043b1382612"] 5 url = "https://token.actions.githubusercontent.com" 6} 7 8# Create iam trust policy with github repo 9data "aws_iam_policy_document" "github_actions_assume_role" { 10 statement { 11 actions = ["sts:AssumeRoleWithWebIdentity"] 12 principals { 13 # GitHub identity provider 14 type = "Federated" 15 identifiers = [data.aws_iam_openid_connect_provider.github.arn] 16 } 17 condition { 18 test = "StringLike" 19 variable = "token.actions.githubusercontent.com:sub" 20 # CHANGE ME: Set organization to my-org and to demo 21 values = ["repo:${var.organization}/${var.git_repo_name}:*"] 22 } 23 } 24} 25 26# Create role and assign iam_policy_document 27resource "aws_iam_role" "github_actions" { 28 name = "github-actions-${lower(var.organization)}-${lower(var.git_repo_name)}" 29 assume_role_policy = data.aws_iam_policy_document.github_actions_assume_role.json 30 tags = var.tags 31}</code></pre> This is all that is needed to use this identity on the AWS side. Most likely, you need additional permissions attached to your roles in order to deploy lambda functions, access S3 or push containers to ECR. Just create this polices and attach them to the role as you need. Configure GitHub Actions</h2> Now we need GitHub Actions to log into AWS. For this, we create a workflow with this minimal configuration: 1name: Demo 2 3on: 4 push: 5 branches: [main] 6 7# Minimal permissions needed 8permissions: 9 contents: read 10 id-token: write 11 12jobs: 13 build-image: 14 runs-on: ubuntu-latest 15 steps: 16 - name: Checkout repository 17 uses: actions/checkout@v4 18 19 - name: Configure AWS Credentials 20 uses: aws-actions/configure-aws-credentials@v4 21 with: 22 # Your region. Can be a var or hardcoded 23 aws-region: ${{ vars.AWS_REGION }} 24 # Your role name you created previously 25 role-to-assume: ${{ secrets.AWS_ROLE }} 26 27 - name: S3 list 28 run: aws s3 ls MY_BUCKET</code></pre> Note: When you use repo secrets/vars you need to have maintainer or admin permissions for the GitHub repo. Yet, both values are not considered confidential. Now you can run AWS commands in GitHub actions without having to set or manage any passwords or keys. Comparing GitHub Actions with GitLab CI/CD - A deep dive! 2024-01-31T00:00:00+00:00 Comparing GitHub Actions with GitLab CI/CD</h1> Introduction</h2> GitLab just announced the availability of their GitLab CI/CD Catalog (2023/12),</a> which states a perfect opportunity to compare the current state of the two CI/CD systems of everyone’s favourite version control providers: GitLab CI & GitHub Actions. Disclaimer: While there are a lot of other CI/CD systems like Circle CI, Azure DevOps, AWS CodeBuild, Travis CI and surprisingly, even Jenkins is still around, these tools are dedicated to solving one problem and are not part of a fully integrated developer platform. I also have not used most of them to a sufficient extent to be able to provide a meaningful review. This post only focuses on GitLab CI and GitHub Actions, especially their SaaS offerings, from a technical and software engineering standpoint. The value of these tools may vary depending on your specific requirements. 2024-10-12 Update: GitLab is working on step-runners</a> a more modular approach of running CI jobs. The existing of this project might give some insides of what GitLab thinks for its current CI design. General CI setup & flow</h2> GitHub Actions and GitLab both define their CI/CD procedures using yaml</code> files which will be parsed, templated/interpolated, and processed by the CI/CD task scheduler. In general, one or more jobs are created which run in one or more stages. By default, jobs within one stage run in parallel, while stages run in sequence. Naturally, both GitHub Actions and GitLab have first-party integration support within their Git hosting platforms. While they can also be used to run tasks for Git repos hosted on a third-party service, it comes with limitations, often making it impractical. In fact, many triggers for the CI/CD jobs are directly related to the Git repositories on which they perform these actions. While GitHub evaluates each file in .github/workflows</code> individually by checking if the emitted event matches one of the many workflow triggers</a> specified via the on:</code> keyword, GitLab only has one CI file called .gitlab-ci.yml</code>. Whereas this file can include separate CI configurations, GitLab always merges all definitions into a single monolithic file and evaluates on a job-by-job base if the job should be created and run or not. GitHub has a lot of triggers for every imaginable event that might accrue within a GitHub repository. Workflows can be run when an issue is created, labelled, or a comment is added. GitLab, on the other hand, only has the predefined CI_PIPELINE_SOURCE</code> variable which, in combination with other predefined variables,</a> can be used to decide if a CI job is created. It does not support events like issues or comments. However, integration with external event sources is more flexible in GitLab CI with the choice of API</code>, Triggers</code> and chat, enabled by the direct integration support of various applications such as Slack. GitHub only allows to manually create CI jobs via the workflow_dispatch</code> event, which can be triggered via the API or the WebUI. These and other differences also affect the way variables and tokens are used for these CI/CD systems. Basic event triggers such as the push</code> of a branch/tag or merge_requests</code> are supported by both systems. Building the Jobs</h2> Based on the event and its values, the specified CI/CD jobs get created. While the general declaration of a job definition is quite similar, the specification and execution of the actual commands are quite different. Each job has a name, a selector to specify the runner, variables and script or step property. Both systems also support a matrix property to run one job definition multiple times for different environments, runtime versions or operating systems. Users can reference variables, secrets and event values within a job definition. GitHub even allows the usage of a limited set of expressions</a> to compare, encode or alter the content of variables. It also includes an additional property to determine if a job should be run after evaluating the on</code> property of a workflow file. GitLab only has one level of conditionals defined via it’s rules</code> property. Every job can also specify dependencies on other jobs. The actual steps that the CI should run are specified differently depending on the system used: GitHub Actions</h3> GitHub Actions allows users to run an arbitrarily amount of action modules. An action is a (specific) reference to a other Git repository that performs a specific task within the CI. When a job is run, the action is checked out and runs its logic. From the users perspective, most actions are declarative, which allows users to utilize them without having to know their implementation details. Examples of actions include installing Docker</a> or setting up Minikube</a>. GitHub has a huge marketplace</a> of official and community actions. Users can customize the behavior of actions by passing input arguments to them. This approach allows for great reuse of tasks, a low entry barrier, and lean CI configurations. Besides the marketplace actions users can also use a generic run</code> action, which accepts any shell code, allowing any generic imperative commands. In addition to Bash and PowerShell, it is also possible to directly use Python, JavaScript and other interpreted languages. In addition to workflow control and job control, each step has an optional parameter to determine if the step should be executed. 1# GitHub Action example workflow 2name: Example Pipeline 3 4on: 5 pull_request: 6 branches: ['releases/', 'main'] 7 push: 8 branches: [main] 9 10jobs: 11 code-style: 12 runs-on: ubuntu-latest 13 steps: 14 - name: checkout 15 uses: actions/checkout@v4 16 17 - name: Info for main branch 18 if: github.ref == 'refs/heads/main' 19 run: echo "Running tests in main" 20 21 - uses: actions/setup-python@v5 22 with: 23 python-version: '3.11' 24 25 - uses: hashicorp/setup-terraform@v3 26 - uses: terraform-linters/setup-tflint@v4.0.0 27 - uses: pre-commit/action@v3.0.0</code></pre>GitLab CI</h3> GitLab has three different run stages: before_script</code>, script</code> and after_script</code>. Each stage allows a list of shell commands and can execute everything a script could. The after_script</code> is always executed, even if previous steps errored. Allowing users to send webhooks, perform error handling and print debug messages. Reusing these scripts is possible via inheriting from an existing job, extending from an existing job, or using the !reference</code> tag which allows using commands specified somewhere else. All commands have to be imperative shell instructions, which requires users to carefully construct every task they want to perform themselves. 1# GitLab CI Pipeline example 2 3tags: [docker] 4variables: 5 MY_GLOBAL_VAR: Foo 6 GIT_DEPTH: 5 7 8# include: 9# - template: Terraform.latest.gitlab-ci.yml 10# - template: Security/SAST.gitlab-ci.yml 11 12Build-Image-Buildah-Template: 13 image: registry.access.redhat.com/ubi8/buildah:latest 14 stage: build 15 variables: 16 REGISTRY: ${CI_REGISTRY} 17 REGISTRY_USER: ${CI_REGISTRY_USER} 18 REGISTRY_PASS: ${CI_REGISTRY_PASSWORD} 19 CONTEXT: $CI_PROJECT_DIR 20 BUILD_IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME 21 DOCKERFILE: $CONTEXT/Dockerfile 22 script: 23 - buildah login -u "${CI_REGISTRY_USER}" -p "${CI_REGISTRY_PASSWORD}" "${CI_REGISTRY}" 24 - | 25 buildah bud --pull \ 26 --build-arg COMMIT_HASH=$CI_COMMIT_SHORT_SHA \ 27 --build-arg COMMIT_TAG=$CI_COMMIT_REF_NAME \ 28 --tag "${BUILD_IMAGE_TAG}" \ 29 --file ${DOCKERFILE} ${CONTEXT} 30 - buildah push --creds "${REGISTRY_USER}:${REGISTRY_PASS}" "${BUILD_IMAGE_TAG}" 31 rules: 32 - if: $CI_PIPELINE_SOURCE == "schedule" 33 - if: $CI_PIPELINE_SOURCE == "push" 34 - !reference [.global, default_rules]</code></pre>Differences</h3> GitHub’s prebuild actions allow for an easy-to-use and quickly configured CI. They are great for reusability and the abstraction of low-level tasks. However, users and organizations must keep in mind that actions in the community marketplace is code written by others. It is recommended to check all actions before using them and to pin their reference to a specific commit to prevent supply chain attacks. GitLab’s approach requires more effort upfront and a deeper knowledge about every step in the setup to perform. While GitLab also allows reusing jobs, it is not as powerful as the later sections will show. The fine-grained and clear structure of GitHub Actions, that determines whether a workflow, job, or step should be executed is an advantage over the monolithic approach of GitLab. Reusing CI Configurations</h2> GitHub Actions allow for great usability at the task level, as the previous section already showed. However, CI jobs are rarely constructed of just on step. Most jobs include checking out the code, installing the language runtime, downloading build dependencies and performing some checks. A Docker build job is almost always the same, just like security scans. They just need some arguments that point to the source and parameters for the output artifact. In an organization, it might be desirable to share these common build steps to keep it DRY and reduce code hygiene maintenance. This would also assure that all jobs are compliant with internal standards and policies. GitHub Actions offer reusable workflows,</a> which are GitHub Action workflow files that can be called from any other project. These reusables define one or more jobs and offer inputs to customize how the included jobs are run. Being built on Git, these workflow files are versioned and can be pinned to a specific commit. For GitLab, the story is a little more complicated. GitLab uses a monolith .gitlab-ci.yml</code> file in which all jobs are defined. But the jobs don’t have to be declared within that file. They can be included from another project or any other file that is accessible via HTTP from the GitLab instance. In general, there are two types of includes</code> in GitLab, both can have multiple nested includes of their own. There is the generic include of any other job definition file that is offered by a shared repository or a third-party entity. The second type is the official GitLab CI/CD template</a> type. These templates are common CI/CD jobs that are directly provided by GitLab and come with every GitLab installation. Both types do not provide the possibility of directly passing inputs to the included jobs. All customizations must be done via GitLab’s CI/CD variables or by explicitly overriding the job to fit the desired needs. To apply these customizations, it is often necessary to look at the source of a specific job, which might be defined in multiple levels of nested includes. On December 21, 2023, GitLab announced the beta availability of the GitLab CI/CD Catalog (2023/12)</a>. This feature enables a third type of job inclusion into the .gitlab-ci.yml</code>. It provides a community store where every user can create their own definitions for one or more jobs. Anyone can include these components and customize them via a series of inputs. Consumers just need to look at the component page of the catalog item to get a list of all available inputs and further documentation. It’s a much cleaner interface between the provider and consumer of a job. Just like it is with GitHub Actions on the individual task level. The difference is that these catalog items still don’t let users fully alter any jobs on the individual task level. The components include whole jobs and are therefore more similar to GitHub’s reusable workflows than GitHub’s Action. A list of already available GitLab CI/CD Components can be found here</a>. Creating your own component is quite easy. GitLab requires a specific repository layout and a label on the project to specify it as a CI/CD Component. With GitLab releases, a new version of that component is created. To test this new functionality, I created my own component</a> that builds Docker images with Googles kaniko builder (which does not require Docker in Docker) and automatically applies some common OCI spec labels. Feel free to check it out. NOTE: Just like third-party GitHub Actions, before using a third-party component, it should be verified and pinned to an exact version. While the CI/CD Components are a step towards a more user-friendly and reusable CI code, they still don’t offer fine-grained control. GitHub’s conditional controls on workflow, job and step level offer a much more flexible setup and expose a cleaner solution than GitLab by including, merging, extending and overriding jobs from many sources. The complete .gitlab-ci.yml</code> file with all overrides, references and includes applied can be viewed in GitLab’s pipeline editor. However, due to the multiple inclusions, this file can quickly become a few thousand lines long and is not very user-friendly to read. Runners</h2> To run the defined CI/CD jobs, both GitHub and GitLab need runners. Both services offer Hosted Runners that are owned and maintained by the respective provider (SaaS offering). Each provider offers a free plan for their CI/CD runners, which includes a certain amount of compute minutes. The table below shows a brief, none complete overview of their offerings. Plan</th> Included Minutes</th> Price</th> Cost per minute/per core for additional minutes</th></tr></thead> GitHub Free</td> 2000</td> 0$</td> 0.004$*</td></tr> GitHub Pro</td> 3000</td> 4$</td> 0.004$*</td></tr> GitLab Free</td> 400</td> 0$</td> 0.005$*</td></tr> GitLab Pro</td> 10.000</td> 29$</td> 0.005$*</td></tr> </tbody></table> Note: Cost per minute is for extra minutes that exceed the included minutes. It is calculated based on available price information. This table is not representative of real-world costs because both providers offer a variety of different runner sizes with different operating systems. Based on the size and operating system (OS), there is a cost multiplier that is factored in for every real compute minute consumed. The numbers in the table are for the default (smallest) runners based on a Linux system. For specific use cases, please refer to GitHub’s pricing docs</a> or GitLab’s pricing docs</a>. Both providers offer Linux, macOS and Windows runners. At the time of writing, GitLab’s offerings for macOS and Windows are still in beta. GitHub’s runners are hosted on the Azure Cloud using the Standard_DS2_v2</code> series VMs and GitLab’s runners are hosted on Google Compute Platform (GCP) using the n2d_machines</code> series. Both runner offerings are hosted in the US only currently, which may have an impact on latency and can interfere with privacy constraints. The architecture of both CI/CD systems envisions that the available runners continuously communicate with the GitHub/GitLab instance and process jobs from their task queue. Only the GitHub/GitLab instances has to be accessible by the runners, the runners themselves don’t need to be exposed to the internet. This can be important for certain restrictive network topologies. While the CI/CD job scheduling is quite similar, there are some differences between GitHub and GitLab in how the jobs are executed. GitHub runners</h3> When an event that triggers a workflow is emitted, the defined jobs within the workflow get templated (using lazy templating) and are added to the job queue. Available GitHub runners that match the runs-on</code> tag automatically pick up queued jobs and execute them on a fresh VM provisioned just for that job. The GitHub provided VMs come preinstalled with a lot of tools</a> to allow for fast execution of the jobs. It essentially runs shell commands directly on the host, even if the actual commands are abstracted by the GitHub Action modules. Each step gets individually templated, which allows users to pass values from one step or job to another using a special GITHUB_OUTPUT</code> variable. These values do not have to be known before a job starts, they can be obtained and computed during job execution. The command outputs of jobs are streamed back to the GitHub instance, accessible via the GitHub UI in a shell-like output canvas. The output is near real-time, but the canvas can not be reliably searched via Ctrl-F</code>. GitLab runners</h3> The .gitlab-ci.yml</code> gets fully templated and merged every time there is a CI triggering event. If the rules</code> section of a job matches the triggering event, it gets added to the job queue. GitLab uses tags</code> to specify an attribute or a list of attributes that the runner must have in order to run the job. An example would be tags: [linux, large]</code>, where the runner must have at least these tags. GitLab’s runners can have different executors, which determine how the user-defined job gets executed. The supported executors are Shell, SSH, VirtualBox, Docker, Kubernetes and some more</a>. Each executor comes with some advantages and disadvantages. The Shell and SSH executors directly run on the specified machine. Users must make sure that every build dependency is either already installed on the machine or is included in the job script section. These executors change the host filesystem and are not idempotent, which can lead to a lack of reproducibility and unexpected behaviour. Therefore, the use of a virtualized executor is often preferred. Docker and Kubernetes use the container image specified in the image</code> tag and run all scripts within that container. When the job is finished, the container gets deleted and the host system is in the same state as before. Currently, GitLab’s SaaS runners use the Docker+Machine</code> executor, a GitLab maintained fork</a> of the deprecated Docker Machine project. These executors spawn new Google Cloud VMs with Docker installed to run the specified container image and execute all job scripts. At the time of writing, GitLab is working on replacing this executor with an alternative</a>. While these executors are idempotent, they have some drawbacks when users want to perform some low-level system actions or need to use virtualisation themselves, resulting in nested virtualisation. Running container builds in the GitLab CI often requires the use of Docker in Docker, resulting in bind-mount of the Docker Socket and slower build performance, unless tools like kaniko or buildah are used. Variables between jobs can be passed to each other by writing them to a file and passing these artefacts between jobs. The script outputs are also sent back to the GitLab instance in periodic chunks. The resulting log in the UI is fully searchable. Other than GitHub, GitLab automatically downloads the Git repository at the ref specified in the event that triggered the job, unless it’s explicitly turned off. If artefacts were defined in the job, they also get automatically up and downloaded. GitLab also offers a wider set of predefined variables</a> that users can access while the jobs executes. Comparison</h3> The execution of jobs is quite different between GitHub and GitLab. While the GitLab solution seems more complex at first glance, knowledge of its runner architecture is not required by developers. Only for GitLab admins who implement their own runners need to know the details in order to decide for an appropriate executor. Depending on the size of the organization, a simple Docker or Kubernetes executor should cover most use cases for a CI/CD runner. GitHub also allows users to bring their own, self-hosted runners, which just like GitLab will not be billed (at least not by GitHub/GitLab). Own runners allow for better confidentiality between own jobs and the jobs of others, have consistent performance and meet specific requirements such as CPU architecture and GPU support. It also allows overcoming network borders by placing the runner inside restricted network segments to allow for communication with services that are not publicly accessible. Own runners might also be needed to overcome latency issues, since currently both GitHub’s and GitLab’s runners are located in the US. GitHub’s runners allow for more flexible usage by just being a VM that is owned by you during the duration of the job, while GitLab’s runners might be orientated more towards one specific use case. GitLab’s runners automatically check out the code and manage artefacts distribution for the user. If the right container image is chosen, there it literally no need to install anything since all tools are already the present in container. Both runners should be able to fulfil most requirements, users should just be aware of the respective limitations regarding location, availability, CPU architecture and other feature and properties like cost. It must be mentioned that self-hosted runners can run more than one job per host concurrently, depending on the configuration. Due to privacy concerns, this feature is not enabled for hosted/SaaS runners. A performance comparison is intentionally not part of this comparison, since there are too many variables and influences present for any meaningful results. Security and Secrets</h2> Both GitHub and GitLab allow users to set variables and secrets that are then available in their CI/CD runtime. Both input types can be set at the instance, group/organization, project level, and even per registered deployment environment. Secrets are special variables that are considered confidential, these will be masked by the CI system if they are found in the output log. GitLab allows users with the appropriate permissions to view the value of a secret after it has been set, while GitHub never shows the value again. Regarding security, this makes no real difference, as anyone with access to the CI files can output the secrets when executing a job. While the secret values will be masked, this safety mechanism can easily be bypassed by reversing and base64 encoding the value before they get written to the log. Therefore, the general recommendation is the usage of short-lived tokens. For project-related resources, both CI systems use such a short-lived token to access the code stored in Git, upload artifacts or create releases. The GITHUB_TOKEN</code> or GitLab’s CI_JOB_TOKEN</code> is only valid during the execution of the job. The permissions for these tokens can be limited. GitLab allows altering the permissions of users, they can be limited to a specific scope and certain actions. The CI_JOB_TOKEN</code> will inherent the permissions of the user that triggered the CI job. However, when a CI job runs, this token is accessible, with its specific permissions, for all jobs and tasks within these jobs. GitHub on the other hand, allows for a more fine-grained permission</a> model for its token. These permissions can also be defined at the workflow or job level. Unlike GitLab, GitHub does not make its token accessible for every step in the CI flow by default. Creating a release via GitHub Actions requires access to the GitHub release API with a valid token, downloading public dependencies or compiling code does not require any tokens. Only steps that request access to the token or are explicitly given access via the ${{ secrets.GITHUB_TOKEN }}</code> parameter can use that token. The default untrustworthy approach makes a lot of sense since users are much more likely to use third-party actions in their CI, which is code written by strangers. Before the introduction of the GitLab CI/CD catalog, most CI code in GitLab was owned by the project owners or their organization. GitHub defaults to the least privilege solution with less trust at every step, where access can be given only if required. Organizations can enforce an even more permissive permission scope for the GITHUB_TOKEN</code> and can also limit the allowed GitHub Actions to internal or whitelisted ones in order to meet company compliance policies. GitLab does not currently support such a feature out of the box. Instead, GitLab allows users to extend the permissions of its CI_JOB_TOKEN</code> to access additional Git repositories. In a multi repository/microservice project, this results in a smaller number of user-managed tokens. To access multiple private repos in GitHub, users need to create their own token since the GITHUB_TOKEN</code> currently does not support such a feature and is only valid for the current repository. Both services encourage the use of short-lived tokens or external secret solutions. GitLab has first-party support with HashiCorp Vault to use external secrets. Besides this, both GitHub and GitLab act as an Identity-Provider (IDP) to allow the usage of OpenID Connect (OIDC)</a> to offer secure communication with the three major cloud providers using short-lived tokens. Depending on the configurations, this allows users to access and manage cloud resources in the CI/CD by using a trust relationship rather than any explicit credentials. GitLab builds this functionality directly into their CI/CD system, while GitHub relies on GitHub Actions like configure-aws-credentials</code> or azure-login</code>. A useful extra feature that GitLab does not have is the ability to add secret masks while the CI is already running. Especially when using short-lived access tokens, which the CI cannot know in advance, it is helpful to use echo "::add-mask::MY_TOKEN"</code> to prevent unwanted exposure in logs. Additional Functionality & Services</h2> Running predefined tasks when a specific event is triggered is the core functionality of a CI/CD system, but requirements and competition in the CI/CD world are constantly increasing. Having functionality that makes the lives of developers easier might be a major selling point. These following functions were not considered in the previous comparison, but should be mentioned for a complete picture of GitHub Actions and GitLab CI. Both GitHub Actions and GitLab CI can run additional services besides the main CI tasks. These services are defined as containers that can offer a database or local web server within the CI job. This can allow users to run integration tests within the CI. A job can perform a database migration test on an ephemeral DB or an end-to-end test with playwright against a temporary hosted website, all within a CI job. Project documentation or a static website can be hosted on GitHub and GitLab using their Pages feature. Both CI/CD systems can build and directly deploy HTML sites to their page environments without the need for any extra authentication. To speed up tests and build jobs, both systems support caches. GitHub has official actions to upload and download caches, while GitLab offers the cache</code> keyword and manages uploading and downloading caches as part of the job initialisation. When all test succeeded, software packages can be deployed to one or more environments. Both systems offer a environment</code> keyword. Every job that references an environment creates a deployment in that environment. Environments can have their own set of variables and secrets to allow stricter separation between them for increased security. A link to a successful deployment is then shown on the repository or pull request UI. GitHub requires users to create environments beforehand. GitLab can dynamically create environments based on supplied variables, like the branch name. Furthermore, GitLab also has different actions for environments to create, access and delete them on demand. Environments can even be automatically removed after a certain time or after a pull request gets merged. Compared to GitHub, GitLab’s environments are definitely more advanced and versatile. For Kubernetes users, GitLab offers its optional GitLab Agent</a> to access the insides of a cluster without any additional tokens. The agent gets deployed within a cluster and communicates with a GitLab instance, allowing it to be accessed via the CI. Overall Comparison</h2> GitHub Actions and GitLab CI are two very powerful CI/CD systems that allow users to automate almost every imaginable task within the software development life cycle. At a functional level, both systems are on a similarly equal level, but the way they are implemented differs significantly. GitHub offers a stable, versatile and modular platform to run almost every kind of imaginable task. It integrates seamlessly with every one of GitHub’s features. Where needed, it allows for integration with standard solutions, such as the ability to run containers or authenticate with third-party services over OIDC. It relies heavily on its vast marketplace of official and community actions, which do most of the actual work, such as checking out code, installing software, using caches or publishing releases. GitHub also gives fine-grained control over security-related settings, which can be enforced to be significantly more permissive than its competitor, GitLab. GitLab on the other hand, provides a much more managed solution for its users. It is part of the CI/CD system to check out code, manage caches and releases. It directly integrates with Vault and major cloud providers. GitLab itself also seems to allow for easier integration with third-party systems. CI triggers from external services like issue trackers and chat tools are much more versatile, just like the CI environment setup. But GitLab requires the user to write all the CI task implementations themselves. Sharing and reusing CI definitions becomes difficult and quickly confusing with the monolithic approach of one big .gitlab-ci.yml</code> file approach. There are multiple levels of including, inherence and extending which can make it hard to get a quick oversight of what a job actually does. Eventually, this might get better with GitLab’s new CI/CD catalog, but this is not guaranteed since components compare much more to GitHub’s reusable workflows instead of individual actions. GitLab is definitely in the middle of a migration process regarding their CI, not just to allow it to be more reusable, but also to get rid of their docker+machine</code> runners. In contrast to GitHub’s modular approach, GitLab seems to have to deal with significantly more historical challenges due to its more managed approach. From a software engineering perspective, GitHub’s design might be the more beneficial long-term position because their modular approach allows them to add features without many legacy impediments. There is no clear recommendation for one or the other system. Both are more than capable of running any imaginable task. The requirements for the actual development Platform, GitHub and GitLab, should be much more important than its CI features. Costs and data privacy, as well as the ability to self-host, are often more relevant factors than the differences between the CI/CD systems. Building preconfigured OS images with HashiCorp Packer 2023-12-12T00:00:00+00:00 Building preconfigured OS images with Packer</h1> Introduction</h2> Hetzner Cloud provides a number of different operating system images to select from when creating a new virtual machine. These images provide the basic ground to get started with your service, but they are rarely configured according to your specific needs. Maybe you want to install a database or a web server, add a maintenance user, or just tweak system settings to improve performance and security. While there are tools to do this automatically for every new VM, this takes time and may not always be reproducible. Wouldn’t it be easier to have your own known good personal pre-configured OS image that you can use for every new VM? This can be archived with HashiCorp Packer. This article demonstrates how to use Packer to create custom images for your Hetzner VMs using Infrastructure as Code (IaC). It even allows users to create VMs with operating systems that are not officially supported by Hetzner. Note: This article was also published on Hetzner’s Community Blog</a> Prerequisites To follow this tutorial, you need: Hetzner Cloud API token</a> in the Cloud Console</a></li> Access to the Internet</li> </ul> Installing Packer</h2> Visit Hashicorp Packer</a> and install Packer according to your OS. On Debian based Linux, run: 1# Get the latest version 2curl -sL https://api.github.com/repos/hashicorp/packer/releases/latest | jq -r .tag_name 3# Set the version and download 4PACKER_VERSION=1.11.2 5curl -sL https://releases.hashicorp.com/packer/${PACKER_VERSION}/packer_${PACKER_VERSION}_linux_amd64.zip -o /tmp/packer.zip 6# Unzip - may needs sudo 7unzip -q /tmp/packer.zip -d /usr/local/bin -x "LICENSE.txt"</code></pre> You can run packer --version</code> to view the version and check if the installation was successful. Creating a custom image</h2> Create a project folder with mkdir my-hetzner-img</code> and enter the directory. Similar to Terraform, Packer uses providers to communicate with the needed build system. The usage of different providers allows Packer to offer a vast verity of different target systems. It allows users to create software or OS packages for all major cloud providers, on-prem systems and even Docker images. To use a provider, create a file called provider.pkr.hcl</code> and add the provider config: 1# packer.pkr.hcl 2packer { 3 required_plugins { 4 hcloud = { 5 source = "github.com/hetznercloud/hcloud" 6 version = ">= 1.2.0" 7 } 8 } 9}</code></pre> Packer uses the HashiCorp configuration language (HCL) for declaring the desired state, just like Terraform. To create a personalized custom image, Packer needs a base form where it starts from. This is called the “source”: 1# custom-img-v1.pkr.hcl 2source "hcloud" "base-amd64" { 3 image = "debian-12" 4 location = "nbg1" 5 server_type = "cx11" 6 ssh_keys = [] 7 user_data = "" 8 ssh_username = "root" 9 snapshot_name = "custom-img" 10 snapshot_labels = { 11 base = "debian-12", 12 version = "v1.0.0", 13 name = "custom-img" 14 } 15}</code></pre> This tells Packer where to start. In the case of Hetzner, Packer needs to know the base image, the server type to use for building, and the location of that server. It also specifies the name of the Snapshot that will be created and the tags that should be applied to that server. The next step is to provide your customization. Meaning the actual build step. 1# custom-img-v1.pkr.hcl 2build { 3 sources = ["source.hcloud.base-amd64"] 4 provisioner "shell" { 5 inline = [ 6 "apt-get update", 7 "apt-get install -y wget fail2ban cowsay", 8 "/usr/games/cowsay 'Hi Hetzner Cloud' > /etc/motd", 9 ] 10 env = { 11 BUILDER = "packer" 12 } 13 } 14}</code></pre> It uses the source specified before and specifies a shell</code> provisioner. Every line in inline</code> will be executed on that server. In this case, it installs fail2ban</code> to harden the server by slowing down the 1000s of SSH login attacks that happen regularly and adding a little message to the motd</code> file. Note: When you run packer build .</code>, Packer will automatically create a new server and a new Snapshot in your Hetzner Cloud project. Both will be charged. The server is automatically deleted immediately after either the Snapshot was created or the creation failed. </blockquote> To create that image run the following commands: 1# Set your Hetzner API Token 2export HCLOUD_TOKEN="XXX" 3# Initialize the project - only needed once 4packer init . 5# Build 6packer build .</code></pre> Now Packer builds that server and streams all the logs to your current terminal. Verify that the new image is available in the Hetzer Cloud Console</a>: Taking it to the next level</h2> Putting all commands in HCL can quickly get crowded. It is also not reasonable to copy-paste the code for every variant of an image you might want to build. To make things more generic, Packer can use variables: 1# custom-img-v2.pkr.hcl 2variable "base_image" { 3 type = string 4 default = "debian-12" 5} 6variable "output_name" { 7 type = string 8 default = "snapshot" 9} 10variable "version" { 11 type = string 12 default = "v1.0.0" 13} 14variable "user_data_path" { 15 type = string 16 default = "cloud-init-default.yml" 17} 18 19source "hcloud" "base-amd64" { 20 image = var.base_image 21 location = "nbg1" 22 server_type = "cx11" 23 ssh_keys = [] 24 user_data = file(var.user_data_path) 25 ssh_username = "root" 26 snapshot_name = "${var.output_name}-${var.version}" 27 snapshot_labels = { 28 base = var.base_image, 29 version = var.version, 30 name = "${var.output_name}-${var.version}" 31 } 32} 33 34build { 35 sources = ["source.hcloud.base-amd64"] 36 provisioner "shell" { 37 scripts = [ 38 "os-setup.sh", 39 ] 40 env = { 41 BUILDER = "packer" 42 } 43 } 44}</code></pre> Now, we can specify and override the base image at runtime by using packer build -var bas_image=ubuntu-22.04 -var version=v1.1.0 .</code>. This version also uses the scripts</code> parameter rather than in-lining all the commands. This allows for much more complex setups. It also uses cloud-init</a> to configure some settings declaratively. Hetzner also supports Arm64-based servers</a> which offer a great price-to-performance ratio and impressive efficiency. Due to the different nature of Arm, software needs to be compiled for this architecture and all configurations also need to be applied for this architecture. Fortunately, Packer allows to reuse any build scripts for Arm. Just add another source to your code and update the build step to include that source: 1# custom-img-v2.pkr.hcl 2source "hcloud" "base-arm64" { 3 image = var.base_image 4 location = "nbg1" 5 server_type = "cax11" 6 ssh_keys = [] 7 user_data = file(var.user_data_path) 8 ssh_username = "root" 9 snapshot_name = "${var.output_name}-${var.version}" 10 snapshot_labels = { 11 base = var.base_image, 12 version = var.version, 13 name = "${var.output_name}-${var.version}" 14 } 15} 16... 17build { 18- sources = ["source.hcloud.base-amd64"] 19+ sources = ["source.hcloud.base-amd64", "source.hcloud.base-arm64"] 20 provisioner "shell" { 21...</code></pre> This version now automatically builds images for Arm64-based servers. Tips and advanced usage:</h2> About disk size: You may have noticed that the code above uses the smallest instances available on Hetzner. This is not only to reduce cost but also allows for the created Snapshot to be deployed for every VM. Due to the way disks are managed at Hetzner, a new VM must have a disk that is at least the same size as the one from which the Snapshot was created. Using an eight-core server might be a little faster than using a smaller one, but it would limit the servers you can deploy form this image to VMs with at least 240GB disks. About Tags: Tags are metadata that you can add to a Snapshot. These are useful to provide information about the origin of an image and what it is used for. Also, tags are the only way to select an existing Snapshot in Terraform to deploy a new VM. They should definitely be used. About cloud-init: You can use cloud-init</code> to do some early setup while building images with Packer, but you might also want to use cloud-init</code> when deploying your custom image. By default, cloud-init</code> only runs once, so it needs to be reset. Consider adding the following lines to your bash script to wait and reset cloud-init</code>. 1# os-setup.sh 2#!/bin/bash 3set -e -o pipefail 4 5echo "Waiting for cloud-init to finish..." 6cloud-init status --wait 7 8echo "Installing packages..." 9apt-get update 10apt-get install --yes --no-install-recommends wget fail2ban 11 12# My setup... 13 14echo "Cleanup..." 15cloud-init clean --machine-id --seed --logs 16rm -rvf /var/lib/cloud/instances /etc/machine-id /var/lib/dbus/machine-id /var/log/cloud-init*</code></pre> This allows users to run cloud-init</code> a second time. Different operating systems: You can create and boot completely different operating systems and create your own images for them using Packer. By booting into Hetzner’s rescue mode, you can override the entire disk image of a VM: 1build { 2 sources = ["source.hcloud.myvm"] 3 provisioner "shell" { 4 inline = [ 5 "apt-get install -y wget", 6 "wget https://github.com/siderolabs/talos/releases/download/v1.5.5/hcloud-amd64.raw.xz", 7 "xz -d -c hcloud-amd64.raw.xz | dd of=/dev/sda && sync", 8 ] 9 } 10}</code></pre> My personal use case for Packer is to create preconfigured Kubernetes images with fine-tuned system parameters and preloaded container images. The horizontal autoscaler can use these images to provision new worker nodes quickly and reliably. But there are plenty of other use cases. For further configuration, check out the Packer documentation</a> and the Hetzner Packer builder documentation</a>. You can find the entire code used in this article can be found on GitHub</a>. Conclusion</h2> This article gave a comprehensive guide on how to install and use HashiCorp Packer. It provided a base project on how to use Packer to create custom OS images for Hetzner Cloud VMs, leveraging Infrastructure as Code practices. It also addressed advanced using variables and cloud-init. This approach allows for the creation of tailored, pre-configured OS images, offering a more efficient and streamlined process for deploying VMs with customized settings and applications. Building and Cracking the Enigma - Part I 2023-07-09T00:00:00+00:00 Building and Cracking the Enigma - Part I</h1> As more and more parts of our lives are shifted to the internet, the subject of security is becoming increasingly relevant. Large language models (LLMs) and chat controls claim more and more personal data. A significant part of security on the internet is achieved through encryption. During the last few weeks, I have been increasingly engaged with encryption and went down the rabbit-hole of encryption history. Despite the exciting early days of message encoding and encryptions like the Caesar-Cipher, the Enigma has a very special place in history, especially from the point of a computer scientist. Thats way I decided to rebuild the enigma in software, to subsequently also try to crack it. What is the Enigma</h3> The Enigma is a cipher device mainly used in the second world war by the Nazis. Most of the military communication was transmitted with messages encrypted via the Enigma. Even when the allies obtained one of the Enigma devices, they were not able to decipher their messages because of its complex design created by Arthur Scherbius. How it works</h3> To understand the following parts, I want to provide a quick (and none complete) overview of how the Enigma works. The Enigma is a symmetric encryption machine in the form of a typewriter. Each Enigma has at least four rotors (wheels) with about 26 steps. There are some variants of the Enigma which also include additional turnover rotors and switches, but for simplicity these will not be considered her. Common to all Enigmas however, was a plugboard where the user could electricity connect two outlets to interchange the key for the letter that was pressed via the letter input buttons. When the user pressed a key an electrical current went through the, may or may not switched, keys of the plugboard, to the entry rotor stepping it and eventually their subsequent rotors, to a final turnover rotor which ultimately highlighted the ciphered latter on the displayed alphabet. Even though the Enigma is a symmetric encryption device, its design always produced different outputs even when the same key was pressed multiple times. To decipher a given text, the user would need a compatible Enigma machine and must know the exact configuration for the initial rotor state and the plugboard configuration. Overall there where about 150 trillion possible configurations So cracking this will be quiet a challenge. A challenge even worth a hole (and fantastic) movie. The Imitation Game</a> tells the storey about cracking the Enigma and the invention of our modern day computing model by the mathematician Alan Turing. Some Cryptographic Context</h3> Cryptography is the term used to describe the encryption of confidential information so that it can only be fully recovered by the intended recipient. Besides cryptography, there is stenography, which deals with hiding information and there is hashing which is also often used in the context of cryptography. With hash functions, however, information is lost, so that this technique is more suitable for unique assignments and the verification of information. Cryptography and hash functions are fundamental components of our digital lives today. For example, the layers of the ISO-OSI reference model does not specify how to ensure, the information sent is identical to the information received. That’s why TCP (the most used protocol on the web) has built-in error detection, based on checksums using hash functions. As the Internet grew out of its baby shoes and spread across universities into every aspect of our lives today, it became clear that the Internet was not designed with security in mind. In today’s world, TLS/SSL forms the basis of this, post-field added, security. On the web, public certificates and private keys are used to encrypt information from unauthorized entities. This is based on an asymmetric process where anyone can encrypt information with the public key, but only the owner of the private key can decrypt it. This is also the foundation for SSH keys and digital signatures. Since asymmetric encryption and decryption is quite time consuming, on the web there is a separate handshake (more about the SSL handshake</a>) negotiated per session, where both sides agree on a secret symmetric key and then use it for encryption. With a symmetrical encryption you can encrypt and decrypt information with the same key, like with the Enigma. With the Enigma, the rotor and switchboard settings are the key, on the Internet it is often a password. With each and every encryption there is the possibility guess the key. Mathematicians have accordingly searched for cryptographic methods that have a high complexity, making it difficult to guess the key. Trying every possibility in the problem-space through brute-force results in complexity, often measured in bits. Here is a small comparison: Encryption</th> Possibilities base 2</th> Possibilities base 10</th></tr></thead> Enigma</td> 2^68</td> ≈ 15 × 10^19</td></tr> RSA 2048</td> 2^112</td> ≈ 51 × 10^32</td></tr> AES 265</td> 2^265</td> ≈ 59 × 10^78</td></tr> </tbody></table> Just as a reminder: Every additional bit doubles the possibilities. Current computers would take about 300 trillion years to crack a RSA 2048 key. To get a glims about AES 265 check out this great video</a>. Implementing the Enigma</h3> I decided to rebuild the Enigma in software, to deeper understand its working, using my go to prototyping language python. First I needed to narrow down the scope of the parts I wanted to implement. I ended up with the following plan: Enigma with 3 rotors and 1 reverse rotor</li> Support for the plugboard</li> Support for the letters a-z</li> Support for choice for 3 out of 5 possible rotors</li> Support for 1 out of 3 reverse rotors</li> </ul> This is a doable scope while also keeping the core concepts of the original Enigma. Modeling the rotors in software resulted basically in one list for every rotor containing tuples for the next step-position. 1allRotor = { 2 1: [[0, 4], [1, 10], [2, 12], [3, 5], [4, 11], [5, 6], [6, 3], [7, 16], [8, 21], [9, 25], [10, 13], [11, 19], [12, 14], [13, 22], [14, 24], [15, 7], [16, 23], [17, 20], [18, 18], [19, 15], [20, 0], [21, 8], [22, 1], [23, 17], [24, 2], [25, 9]], 3 2: [[0, 0], [1, 9], [2, 3], [3, 10], [4, 18], [5, 8], [6, 17], [7, 20], [8, 23], [9, 1], [10, 11], [11, 7], [12, 22], [13, 19], [14, 12], [15, 2], [16, 16], [17, 6], [18, 25], [19, 13], [20, 15], [21, 24], [22, 5], [23, 21], [24, 14], [25, 4]], 4 3: [[0, 1], [1, 3], [2, 5], [3, 7], [4, 9], [5, 11], [6, 2], [7, 15], [8, 17], [9, 19], [10, 23], [11, 21], [12, 25], [13, 13], [14, 24], [15, 4], [16, 8], [17, 22], [18, 6], [19, 0], [20, 10], [21, 12], [22, 20], [23, 18], [24, 16], [25, 14]], 5 4: [[0, 4], [1, 18], [2, 14], [3, 21], [4, 15], [5, 25], [6, 9], [7, 0], [8, 24], [9, 16], [10, 20], [11, 8], [12, 17], [13, 7], [14, 23], [15, 11], [16, 13], [17, 5], [18, 19], [19, 6], [20, 10], [21, 3], [22, 2], [23, 12], [24, 22], [25, 1]], 6 5: [[0, 21], [1, 25], [2, 1], [3, 17], [4, 6], [5, 8], [6, 19], [7, 24], [8, 20], [9, 15], [10, 18], [11, 3], [12, 13], [13, 7], [14, 11], [15, 23], [16, 0], [17, 22], [18, 12], [19, 9], [20, 16], [21, 14], [22, 5], [23, 4], [24, 2], [25, 10]] 7}</code></pre> The plugboard is only a hash map switching two arbitrary letters. The Enigma class just used these inputs as settings and passed each character passed to to the plugboard, the reverse rotor, stepping it, the rotor 1 to 3, stepping them and back to the reverse roter. Due to the reverse rotor, the Enigma is symmetrical in operation. This means that users do not have to distinguish between cyphering and deciphering inputs. Using it looks something like this: 1> pyenigma.py 2Provide a strimg to encrypt/decrypt: dont panic 3Encryped: kihdqqocv</code></pre> You can checkout the code on my GitHub</a>. Disclaimer: This is a quick and dirty implementation I did in collage. Its NOT production grade! The next step will be cracking it. See you! How to send OnPrem Prometheus metrics to MS Azure 2023-05-25T00:00:00+00:00 Sending metrics to Azure - with Prometheus</h1> With this post, I want to provide a quick demonstration on how to send Prometheus metrics to Azure managed Prometheus. Let’s get stared. What is Prometheus</h2> Prometheus is an open-source monitoring system that scrapes (pulls) system and application metrics from supported targets. It is an graduated project of the Cloud Native Computing Foundation (CNCF) and the de facto standard in the Kubernetes world. It also supports service discovery and alerting. The major drawback of Prometheus is, that it stores it’s metrics in block storage as local files and has no native support for clustering multiple instances. Horizontal scaling is therefore difficult to archive. Why send metrics to Azure</h2> Cloud services as Azure and Grafana Cloud solve the clustering and storage issues of a single Prometheus instance by providing managed services. (Often implemented through Mimir</a>.) Companies can send metrics from multiple Prometheus instances and different locations, all to a single location, which minimizes duplicated setups (cost), the risk of a singe point of failure and provides scaling, while also providing a single location (called a single pane of glass) for service performance data. When using Azure, Prometheus also integrates nicely with AKS and other Azure services, even without the need of credentials due to service principles. Prometheus supports sending data to a different location using the remote_write</code> feature, which I will use here to sent data from any system and app to Azure. Setting up Azure Prometheus</h2> For simplicity, I will demonstrate the set up process using the Azure Portal. But the process can also be automated via the CLI or terraform. You have to do the following steps: Log into Azure</li> Search for Prometheus</li> Start Creating a Prometheus instance Set a name and location</li> Allow public networking</li> Create and wait</li> </ul> </li> </ul> This will create a new managed monitor resource-group with your Prometheus instance. Write down the Metrics ingestion endpoint Next step is to create an App/User that can push to that endpoint. For this we: Open the Azure Active Directory</li> Go to App Registrations</li> Click New Registration Provide a name</li> Choose single tenant mode</li> </ul> </li> Go to Certificates & Secrets</li> </ul> For simplicity we use a secret in this demo, but a certificate is safer and recommended. It is best practice to store the certificate in a vault (like Azure Keyvault or Hashicorp Vault) and only request the credentials via the vault’s API. A guide how to do this can be found here</a>. Create a secret and copy it (you will not be able to see it again). Go back to the overview of the created app registration. Copy the ClientId and then press on Endpoints to copy the OAuth 2.0 token endpoint (v2) URL. Next, we need to authorize our user to push metrics to the Metrics ingestion endpoint. Go back to the created resource-groups for the Prometheus instance and go to Access Control (IMA). On the role-assignment tab, add a new role-assignment with the role of Monitoring Metrics Publisher and assign it to the created user from the previous step. This is it for the azure site! Sending Metrics</h2> Now we can set up our local tools to send metrics. The flow we created uses the OpenIdConnect (OIDC) standard which is part of the Oauth2 spec. When you want lo learn more abut the described authentication, you can check out this in depth article TBA. To send metrics form on Prometheus instance we need to set up the Prometheus config like this: 1global: 2 scrape_interval: 1m 3 4scrape_configs: 5 - job_name: "INTERNAL" 6 honor_labels: true 7 static_configs: 8 - targets: ["localhost:9090"] 9 10remote_write: 11 - name: azure-write 12 url: <Metrics ingestion endpoint> 13 headers: 14 X-Scope-OrgID: anonymous 15 oauth2: 16 client_id: <ClientId> 17 client_secret: <ClientSecret> 18 token_url: <OAuth 2.0 token endpoint (v2)> 19 scopes: [ https://monitor.azure.com/.default ]</code></pre> Thats it. Start your Prometheus instance and see the logs that are coming in. You can visualize them in Azure Monitor or via the Azure Managed Grafana. See you! The Notebook for Developers - now even better 2023-03-06T00:00:00+00:00 Joplin: The notebook for Devs</h1> How do you keep your digital life organized? How do you store and structure your knowledge in the era of information (overload)? I struggled for sooo long to find a good solution for these problems, but all the systems I tried, every app I used, felt off. They didn’t let me do the things I wanted to, were slow to operate or were just too costly. I almost gave up and went back to analog notes and a mix of digital short notes/Google-Notes - until a colleague told me about Joplin. Right from the beginning, I loved the content-focused approach and the lack of the corporate business stuff that wanted to sell me things. And the best thing is: It’s extendable - so I extended it. With this article, I want to give an overview of Joplin, what differentiates it from other notebook apps and the necessary information to decide if it is a tool fitting for your workflow. What is Joplin?</h2> Joplin is a cross-platform notebook app (Windows, Linux, macOS, iOS, Android) that is entirely based on Markdown. If you are not comfortable with Markdown, read no further - Joplin is not for you. But if you like Markdown because of its easy formatting and capabilities, use it regularly in ReadMe.md’s or GitHub/GitLab Issues, Joplin might be just the right fit. Its content-focused approach is its main differentiator between other apps. There is no need to switch between heading formatting and normal formatting, and there is almost no need to use the mouse. This approach enables users to work incredibly fast and get their thoughts written down. From personal experience, a lot of other tools do way too much to make the content pretty and flexible instead of focusing on the actual content. More on this later. When the time comes to make it pretty, Joplin allows users to export their notes in almost every imaginable format, including PDF and HTML. Another differentiator is that Joplin is open-source software. Everyone can see the sources, and personal data keeps being personal. There are no Upgrade here, Premium there popups, it is just focused on content. There is no big corporation behind Joplin that wants to push their subscription model. Of course, there is the option to spend money, and users can also opt-in to JoplinCloud (between $17 and $80 annually), but this is purely optional. As you might expect, you can sync your notes with Joplin Cloud, but this is not only possible with JoplinCloud. It can also be done with OneDrive, Dropbox, GoogleDrive, and NextCloud. A quick NextCloud setup allows users to store as many notes as they want. End-to-End encryption is also supported. Comparison to Other Apps</h2> As I mentioned earlier, I have tried some other notebook apps, including Notion, OneNote, Evernote, and Google Notes. On Evernote: Regarding Evernote, I used it way back when I was in school, so things might have changed since then. First of all, you need an account. Even when you have an account, there are artificial limitations such as a maximum upload quota of 60MB per month, a maximum of two synced devices, and no offline access to notes. Also, there is currently no Linux client, although it is in early beta now. I never got up to pace with Evernote. From my experience, it tried to solve too many tasks and fit every use case, but did not solve one thing well or fit perfectly. I was even willing to pay to try their premium features, but since they declined my student discount and I was not willing to pay $13 monthly, I turned my back on Evernote and never looked back. Taking notes and using basic features should not cost more than Spotify or Netflix. Notion: As for Notion, I cannot say too many negative things about it. It just did not fit my requirements. I wanted static, easily writable notes that I can organize and search. Notion is much more dynamic and even has dynamic project boards, agendas, and timesheets. Although I think there are other services that execute project management better than Notion, I know for a fact that there are thousands of people out there who love Notion. However, they have to fix their app - copy and paste is beyond broken. Others: I also tried OneNote, DesktopNotes, and Google Notes. At a certain amount of notes, the latter tools are not capable of staying organized and are too limiting. For a quick note I need to remember, sure, but nothing more. OneNote is fundamentally a good app, but Microsoft screwed up with all these different versions of OneNote (Office OneNote, OneNote WPF, OneNote Online). I still use it if I have to create technical drawings with my tablet, but writing and formatting longer texts - ain’t gonna happen. On Joplin</h2> As stated before, Joplin is a cross-platform notebook app that is entirely focused on notes. There are no fancy agendas, drawing or advanced formatting support. It supports all the major Markdown features, including headings, bold and italic formatting, enumerations, and tables. Images can be included via drag and drop or Markdown image includes. Math mode is also supported. It is perfect for my workflows because I already write all my docs and notes in Markdown, so I’m quite fast. It is also easy to import text because most of the tools I use have Markdown support. It is easy to copy and paste between Github and Joplin. Even entire websites (or portions of them) can be imported (and converted to Markdown) via a web-clipper. In case users want to stop using Joplin, they can easily export their notes in various formats. But, as I mentioned above, formatting options are limited, and everything that goes beyond the mentioned methods has to be done via HTML or via Plugins. That’s right, Joplin has a significant number of plugins which can do all kinds of stuff. Users can add support for draw.io diagrams, daily agendas, calendar tabs, and much more. It is also considerably easy to build a plugin, so I did it myself to add a feature I really wanted to make Joplin my central knowledge solution. My Plugin</h2> Why: I think almost everyone stumbled at least once upon one of these Awesome lists of _. Notable mentions are awesome open-source alternatives to SaaS</a>, fucking-awesome-python</a> or awesome-infosec</a>. Sure, you can copy/clip them, but then you have a static version, and you don’t get the newest additions and updates. Realistically, no one regularly visits all of these cheat sheets and updates them. So I wrote a plugin for that: joplin-plugin-remote-note-pull</a>. It lets users create new notes from any publicly available website and regularly updates the note with the original upstream version. How: I started with a plugin skeleton. Joplin even provides a plugin-generator</a>, which gets you set up. The docs</a> provide all the other info one might need. Plugins are written in JavaScript/TypeScript and get executed via the NodeJs runtime Joplin is built on. Developers can add new UI elements or register new functions. Basically my plugin just does is make a web-request to the newly added note and download the content. Before storing it as a new note, it fixes some relative links and paths in the gathered content and adds a custom footer to the note. Other than that, it is just a timer-function that runs every x minutes to update all notes created via my plugin. It was a fun project to play around with. Publishing plugins is also relatively easy; they just have to follow a specific naming schema and get automatically picked up by Joplin’s internal plugin browser. For now, I’m quite happy with this workflow and wished I had it earlier. If anyone is interested in sharing their favorite way to stay organized in this world of information overload, feel free to reach out. See you! Broken NodeJS Apps due to secerity dot-release - Debugging the NodeJS CVE-2022-32213 fix 2022-08-18T00:00:00+00:00 Broken NodeJS Apps due to security dot-release</h1> On July 7th the vulnerabilities CVE-2022-32213</a>, CVE-2022-32214</a> and CVE-2022-32215</a> where publicly disclosed. They affected all current NodeJS versions (v14.x, v16.x, v18.x)! The same day fixes for the vulnerabilities where released. My colleagues and I assumed a quick and easy deployment to spit these fixes onto our client’s production systems. A simple rebuild of the old code state with the new NodeJS version and replacement of the old containers. We thought wrong! 🙃 Our staging system showed that some of our apps didn’t get any requests anymore and clients got a HTTP 4xx error code. Some Background Information</h3> For a better understanding let me give a simplified overview of our system architecture: We use a reverse proxy which is the only service directly exposed to the internet. It handles TLS termination and routes the traffic to the appropriate backend service. Some services require TLS client verification (mTLS</a>), meaning the client has to send its unique certificate to the server where it gets verified and passed to the backend service if the certificate was valid. We triggered a new build with the updated NodeJS version (in our case form NodeJS v14.19.3 to v14.20.0) and deployed it to our testing system. The deployment went smooth, but then we saw that every service using client verification was broken after the update. Debugging time</h3> To verify the bug I deployed the same code with the two different NodeJS versions (v14.19.3 & v14.20.0). On v14.19.3 I got a HTTP 200 response, on v14.20.0 I got a HTTP 400 response indicating a bad-request. But our application never threw a 400 error - nor did it log anything. - Time to dig deeper… I created a minimal NodeJS http-server and deployed it with the two different NodeJS versions (and even new ones): 1const port = 3000; 2var http = require('http'); 3 4//Dead simple http server: 5http.createServer((req, res) => { 6 res.setHeader('Content-Type', 'application/json'); 7 res.write(JSON.stringify(req.headers)); 8 console.log(`REQ FROM: path: ${req.url}; remote: ${req.ip}`); 9 console.log(req.headers); 10 res.end(); 11}).listen(port, '0.0.0.0');</code></pre> I got the same behavior 😒. Not a solution but now I knew it wasn’t a bug in our application. Is it a bug in NodeJS? In a dot-release? For security fixes? Let’s start by looking at the release notes of NodeJS 14.20.0</a> 1### Notable Changes 2[8e8aef836c] - (SEMVER-MAJOR) src,deps,build,test: add OpenSSL config appname (Daniel Bevenius) #43124 3[98965b137d] - deps: upgrade openssl sources to 1.1.1q (RafaelGSS) #43686 4### Commits 5[b93e048bf6] - deps: update archs files for OpenSSL-1.1.1q (RafaelGSS) #43686 6[98965b137d] - deps: upgrade openssl sources to 1.1.1q (RafaelGSS) #43686 7[837a1d803e] - deps: update archs files for OpenSSL-1.1.1p (RafaelGSS) #43527 8[c5d9c9a49e] - deps: upgrade openssl sources to 1.1.1p (RafaelGSS) #43527 9[da0fda0fe8] - http: stricter Transfer-Encoding and header separator parsing (Paolo Insogna) #315 10[48c5aa5cab] - src: fix IPv4 validation in inspector_socket (Tobias Nießen) nodejs-private/node-private#320 11[8e8aef836c] - (SEMVER-MAJOR) src,deps,build,test: add OpenSSL config appname (Daniel Bevenius) #43124</code></pre> Hmmm - The certificates are generated with openssl, but the actual verification takes place before it reaches NodeJS. The only other notable thing is the stricter http parser. Let’s take a deeper look. Encoding is everything</h3> I want to see the actual HTTP request that gets proxieed to NodeJS. So I replaced the NodeJS application with Netcat – The Swiss Army Knife of Networking</a>. With the help of @bnoordhuis</a> I found that we were getting the following: 1# HexDump 200000000: 4745 5420 2f20 4854 5450 2f31 2e31 0d0a 436f 6e6e 6563 7469 6f6e 3a20 7570 GET / HTTP/1.1..Connection: up 30000001e: 6772 6164 650d 0a48 6f73 743a 2064 6d2d 7465 7374 696e 672d 7374 6167 696e grade..Host: dm-testing-stagin 40000003c: 672d 7369 7465 2e63 632d 6973 6f62 7573 2e63 6f6d 0d0a 4d59 5f43 4552 543a g-site.exampleio.com..MY_CERT: 50000005a: 202d 2d2d 2d2d 4245 4749 4e20 4345 5254 4946 4943 4154 452d 2d2d 2d2d 0a09 -----BEGIN CERTIFICATE-----.. 600000078: 4d49 4947 4154 4343 412b 6d67 4177 4942 4167 4942 4154 414e 4267 6b71 686b MIIGATCCA+mgAwIBAgIBATANBgkqhk 700000096: 6947 3977 3042 4151 7346 4144 4342 6d54 4561 4d42 6747 4131 5545 4177 7752 iG9w0BAQsFADCBmTEaMBgGA1UEAwwR 8000000b4: 5132 3974 0a09 6347 4675 6553 3168 6458 526f 6233 4a70 6448 6b78 437a 414a Q29t..cGFueS1hdXRob3JpdHkxCzAJ 9000000d2: 4267 4e56 4241 5954 416b 5246 4d52 5177 4567 5944 5651 5149 4441 744d 6233 BgNVBAYTAkRFMRQwEgYDVQQIDAtMb3 10...</code></pre> Do you see these 0a09</code> sequences? That is the encoding for \n\t</code> which is NOT a valid sequence for new-lines in HTTP-Headers. But NodeJS accepted these sequences anyway - at least till version v14.20.0. The correct sequence, based on the protocol, should be 0a0d09</code> which is \r\n\t</code>. The Solution:</h3> The solution is easy - just change the way the certificates are encoded! - Wait! We can’t do that! Due to the nature of our service we don’t have direct control over the clients connecting to our server. It would be a major inconvenience to change this on all clients. Another option is to start NodeJS with the --insecure-http-parser</code> flag - which isn’t an option either. What about our reverse proxy? Turns out NGINX</a> has a dedicated variable for client certificates in its ssl-module</a> since version 1.13.5. Instead of ssl_client_cert</code> we could use ssl_client_escaped_cert</code> which contains the url-encoded (and therefore valid encoded) certificate. A quick test with the new proxy configuration showed that the new NodeJS version was now working. Some Background Information</h3> Planned was the update from NodeJS v14.19.3 to v14.20.0. The above-mentioned CVE’s are about the llhttp http-parser of NodeJS. It is part of the core of NodeJS and handles http requests before any custom JavaScript executed. The security release made the http parser stricter to only accept requests that follow exactly the HTTP protocol standard. The new behavior prevents attackers to use request smuggling. What we learned & what we did</h3> Unfortunately deploying these security fixes wasn’t as straight forward as we originally thought. We did not expect some of our application to just completely stop working. We needed some time to debug but also didn’t want to expose the other working services to unnecessary risk. So we updated every application that didn’t use client verification as soon as all our tests passed end kept back the update form those that did. Two and 1/2 days later we found a solution for the problem and also updated the application that used client verification. We learned about the importance of working within the protocol specification even though some software allows more than it has to. And even though JavaScript is a high level language, it is important to have understanding for the low level actions of the networking-stack. We also got remained that just because it is a dot-release it still can break a lot and case some quality time with your favorite rubber duck 🦆 See you! Productive Working - Be lazy and get things done - Part II 2022-02-12T00:00:00+00:00 Productive Working - Be lazy & get things done - Part II</h1> My math teacher always told me that mathematicians are lazy and so should we. Only do the minimal number of steps to reach your desired destination. Whether or not these steps are simple in themselves is another matter. But somehow this phrase stuck with me. So here are my favorite shortcuts, scripts and tricks to get things done: Personal Favorite: My favorite feature are the search shortcuts in the Browser ❤️ Automate</h3> Are there tasks that you have to do over and over again? Like setting up your work environment? Start the same programs every time? Automate them! Some example: I use SSH a lot. To I created a script in my .bashrc</code> file to automaticly start the ssh-agent and add my private key. So I can run SSH commands from every environment: 1# SSH-Agent setup 2env=~/.ssh/agent.env 3agent_load_env () { test -f "$env" && . "$env" >| /dev/null ; } 4 5agent_start () { 6 (umask 077; ssh-agent >| "$env") 7 . "$env" >| /dev/null ; } 8 9agent_load_env 10 11# agent_run_state: 0=agent running w/ key; 1=agent w/o key; 2= agent not running 12agent_run_state=$(ssh-add -l >| /dev/null 2>&1; echo $?) 13 14if [ ! "$SSH_AUTH_SOCK" ] || [ $agent_run_state = 2 ]; then 15 agent_start 16 ssh-add 17elif [ "$SSH_AUTH_SOCK" ] && [ $agent_run_state = 1 ]; then 18 ssh-add 19fi 20unset env</code></pre> Working with many GIT repos and dont want to pull manually: 1pull_all() { 2 FAILS=() 3 workdir=$(pwd) 4 for REPO in */; do 5 if [ ! -d "$REPO/.git" ]; then continue; fi 6 echo -e "${GRN}Entering and performing GIT PULL on ${REPO}${NC}" 7 cd $REPO && bash -c "git pull --recurse-submodules" 8 if [ $? -eq 0 ]; then 9 echo -e "${GRN}Done${NC}" 10 else 11 FAILS+=($REPO); echo -e "${RED}Command faild${NC}" 12 fi 13 cd $workdir 14 done 15 # Result 16 if [ ${#FAILS[@]} -eq 0 ]; then 17 echo -e "${GRN}All commands sucseeded${NC}" 18 else 19 echo -e "${RED}The following repos exited with an error:\n${FAILS[@]}${NC}" 20 fi 21}</code></pre> Backup your Dotfiles: 1# Create Dotfiles 2DOT_ARCHIVE=$HOME/Tools/dotfiles.tar.gz 3if [ ! -f "$DOT_ARCHIVE" ] || [ $(($(date +%s) - $(date +%s -r $DOT_ARCHIVE))) -gt 604800 ]; then 4 ORI_PWD=$(pwd) 5 cd $HOME 6 echo -n "Creating dofiles archive... " 7 tar -czf $DOT_ARCHIVE --exclude={'.vscode','.git','..','.*/'} $(find . -maxdepth 1 -name ".*" -printf '%P\n') && \ 8 gpg --yes --batch --passphrase=$(echo -n $DOTFILES_PW | base64 --decode ) -o $DOT_ARCHIVE.gpg -c $DOT_ARCHIVE && \ 9 echo "done" 10 cd $ORI_PWD 11fi</code></pre> There are soo many more! Choose your tools</h2> Take your time to find the appropriate tool for the job. And then learn it until you can use it in your sleep. Choose the editor and mail client you like and get really good at it. The same goes for your browser. All chromium based browsers support search-shortcuts in the addressbar of the browser. So you can directly search YouTube instead of googling YT, clicking the first link and then start searching form there. This saves more than 3 clicks!!! Shortcuts!!!1!</h2> Not having to leave the keyboard to reach the mouse in order to click a button that is hidden behind two menues gives you a huge advantage over others. It lets you archive your goal quicker so you can focus on the next step. There are some universall shortcuts that work in almost every application that everybody shoud know: Common: 1# Copy # Print # Search # Swicht between windows 2Ctrl + c Ctrl + p Win + SeachWord Alt + tab 3# Past # Open # Desktop # Open app x in taskbar 4Ctrl + v Ctrl + o Win + d Win + [0-9] 5# Cut # New document # Snap Window # Duplicate/Extend monitor 6Ctrl + x Ctrl + n Win + any-arrow Win + p 7# Search # New tab # Open explorer # Open settings 8Ctrl + f Ctrl + t Win + e Win + i 9# Undo # close tab # Lock screen # Screen Record 10Ctrl + z Ctrl + w Win + l Win + g 11# Redo # Select all # Emoj #Paste with history clipbord 12Ctrl + y Ctrl + a Win + . Win + v 13# Save # Mark wohle word # Open Screenshot 14Ctrl + s Ctrl + Shift + arrow Ctrl + o Win + Shift + s </code></pre> Browser: 1# Reload # Open history # Switch tabs # Go to tab [1-9] 2Ctrl + r Ctrl + h Ctrl + tab Ctrl + [0-9] 3# Add favo # Focuse searchbar # Reopen tab # Zoom 4Ctrl + d Ctrl + e Ctrl + Shift + t Ctrl + Shift + +/- 5# Go back # go forward 6alt + left-arrow alt + right-arrow </code></pre> VSCode: Note: I also created a collection with my favorite shortcuts, scripts and tricks to work efficient as a programmer. Read it here</a> See you! Productive Working - Be lazy and get things done - Part I 2022-02-09T00:00:00+00:00 Productive Working - Be lazy & get things done - Part I</h1> My math teacher always told me that mathematicians are lazy and so should we. Only do the minimal number of steps to reach your desired destination. Whether or not these steps are simple in themselves is another matter. But somehow this phrase stuck with me. As I grow older I measured more value to my time. Doing my obligatory tasks efficiently gave me more time that I could spend on hobbies. So developed certain principles to get my stuff done, some of which I want to share today. Overall Principles</h2> Disclaimer: The following practices have been developed with an IT background and may not be applicable to everyone. However, I have tried to make the introductory practices as general as possible so that they can be used by as many people as possible. Time Managment</h3> To increase our efficiency we need to measure the amount of tasks we can complete in a certain amount of time. Split the amount of time you have in equaly sized chunks like 15mins. Prioritize your tasks according to their importance and duration. If you complete 3-4 small tasks within the first 15mins of allocated time, the amount of overall work you will have to complete looks a lot smaller, and you are more motivated. This can be things like teaching garbage, putting away clothes or taking away a letter/package. For work allocate one or two chunks of our time for replaying to emails. Collect the mails over the day and replay to them on one session, don’t let incoming mails distract you from your current task. Finish your current tasks and then start something new. Peaking at the mail or your phone will only take a second, but it will take you over 1.5 minutes to fully mentally switch back to the previous task. A wonderful article from Johnn Hari in The Gurdian</a> diecribes how our all attention span decreased how expensive these context switches between tasks are. Tasks can be completed a lot quicker when you fully concentrate on one thing at a time. When the amount of tasks becomes too much to sort in your head write them down or maybe even visualize them with a method like Gnatt diagrams: These diagrams allow to visually lay out your tasks and identify blocking tasks. These are tasks that have to be done before others can start. Before you can send a package you have to pack it first. When you wait for certain things to finish you can complete other stuff. For example while waiting for your oven to heat up your food you can clean and store the dishes from last day. Combining things like putting your cloths in the washing machine while heading downstairs to buy groceries allows you to complete multiple things in one run. Take breaks</h3> Keeping up with all the tasks of today’s life is exhausting. So you need time to recharge. Keep this in mind when planing you day and allocate some breaks. When taking a break really stop what you were working on, stand up and do something that lets you relax for a little time. Don’t feel bad for taking these breaks, they are in your schedule and there letting you brain relax a little. If you are disturbed by others tell them that you are having a break right now and will come back to them in 10 min. Recurring Tasks</h2> Routine</h3> Certain tasks keep recurring and need to be done over and over again. Plan these tasks in advance, do not be surprised by them and try to develop regular processes so that they become a routine. Once these tasks have entered the daily routine, you can do them in your sleep. If you notice that the current task exhausts you, or you get stuck, you can take a break and while going for a walk you can complete some of these recurring tasks. The brain can relax during this time and you don’t feel guilty for interrupting your current task, because you are doing other tasks. These breaks may help you to find a new point of view to the previous task. More than once the visit to the restroom or quick shower gave me a new approach for autually solving my previos task 😁 Note: I also created a collection with my favorite shortcuts, scripts and tricks to work efficient as a programmer. Read it here</a> See you! LaTeX - A getting started for Devs 2022-01-22T00:00:00+00:00 Getting started with LaTex as a Dev</h1> About LaTeX</h2> LaTeX is a typesetting system that allows the user to write plain text, while the formatting is provided via markup tags, similar to HTML</code>. It differs from WYSIWYG (What You See Is What You Get) applications like Word in that the final document must first be generated by a compiler. While the output of a LaTeX document is reproducible, minimal and just beautiful, it can be a challenge to get started for beginners. To get started users need to choose one of the multiple existing LaTeX distributions, choose an editor and setup a script to automatically build the document and preview it. This article will automate and abstract most of the pain points and will let you focus on the content - like it is supposed by LaTeX. Functions</h2> The setup provided by this article you will have the following features: A quick, clean, working LaTeX environment setup based on TeXLive</li> Pandoc - To convert your projects between different formats</li> Have LaTeX Workshop an LaTeX Utilities preinstalled</li> GUI for math symbols and some TiKz</li> BibTeX support</li> Spell checking</li> Markdown support</li> Git support</li> Scientific project template</li> Have our environment on a remote computer</li> A platform independent setup that can be used on any OS (Win/Unix/macOS)</li> Have a setup that is faster than MiKTeX on Windows</li> Support for x86/arm/arm64 architectures</li> Persistent bash_history</li> </ul> Pre-Requirements</h2> This is a getting-started guide focused on developers. Therefore, it will use tools common in the software developing world. Nevertheless, everyone can follow this guide to create a quick latex environment. The following tools are needed for this guide: VSCodoe</h3> VSCode is a cross-platform text editor designed for fast coding developed by Microsoft. It is heavily extendable with various extensions provided by a large community. It can be downloaded here</a> Docker</h3> Docker is a universal container runtime that allows users to run various software in a predefined environment isolated from the actual host system. Depending on your system you need one of the following: Host</th> Version</th> Link</th></tr></thead> Linux</td> docker-ce</td> Download</a></td></tr> Windows</td> Docker Desktop</td> Download</a></td></tr> MacOS</td> Docker Desktop</td> Download</a></td></tr> </tbody></table> Note: Docker builds upon functions provided by the Linux kernel. Windows and macOS need to virtualize the kernel. Therefore, the best performance is achievable on Linux. Getting started</h2> To get started you have two options: Having the source files locally on host drive or in a persistent volume managed by Docker. If you pan to modify and access your project files with other tools as well I would suggest using the local setup. For slightly better disk performance or if you just want to use VSCode to edit your document you can also use a persistent volume managed by Docker. Local setup</h3> The TeX source is on your host OS and gets mounted as volume 1# Open a terminal an type: 2git clone https://github.com/hegerdes/VSCode-LaTeX-Container 3code VSCode-LaTeX-Container 4# In VSCode hit F1 5> Remote-Containers: Reopen in Container 6# Wait for the initial pull and build 7# Note: You need to have Docker and VSCode remote extentions installed 8# Search for "ms-vscode-remote.vscode-remote-extensionpack"</code></pre>In a container</h3> The entire project is within the container 1# Open VSCode 2# In VSCode hit F1 3> Remote-Containers: Clone Repository in Container Volume 4> https://github.com/hegerdes/VSCode-LaTeX-Container 5# Wait for the initial pull and build 6# Note: You need to have Docker and VSCode remote extentions installed 7# Search for "ms-vscode-remote.vscode-remote-extensionpack"</code></pre> 👉 Start typing you next big article! Some background:</h2> The LaTeX template</h3> The included template was build up over the time and is designed for scientific projects. But I didn’t start from scratch either. Credit goes to: 1% Original author: 2% WikiBooks (LaTeX - Title Creation) with modifications by: 3% Vel (vel@latextemplates.com) 4% hegerdes (hegerdes@outlook.de)</code></pre>About the Docker image</h3> There are multiple base images debian-[bullseye|buster (deprecated)] and ubuntu-[focal|bionic (deprecated)]. All these images have texlive, texlive-latex-extra texlive-lang-english, texlive-luatex, texlive-xetex, texlive-pstricks, texlive-science, latexmk, cm-super, chktex with additional tools like git, zsh and pandoc(not in alpine) installed. Every image is available on x86/arm/arm64 architectures. The slim images only contain texlive, texlive-latex-extra, texlive-lang-english, latexmk, cm-super, chktex If you want a minimal image use these, but this might lack common tools/packages. There are two full images that contain everything in the LaTeX world except for docs. These are BIG and generally not recommended for fast startups. There are a bunch of language specific images that are build up on the bullseye-base and focal-base images. Languages are: all, arabic, chinese, cjk, cyrillic, czechslovak, english, european, french, german, greek, italian, japanese, korean, other, polish, portuguese, spanish. Use one of these if your work on a none English project! Simply change the VARIANT</code> arg in the devcontainer.json to bullseye-lang-<YOUR_LANGUAGE></code> or ubuntu-lang-<YOUR_LANGUAGE></code>. I plan on updating these images every second month. VSCode workspace</h3> I added a VSCode workspace file with sensible setting. It includes some settings for Docker and the LaTeX extensions. Feel free to customize it after your own taste. If you want to see what I did with the template look over to henrikgerdes.me/papers</a> Why all this</h2> I always liked the concept of LaTeX and its focus on content instead of the formatting. But getting started was hard and I wanted to contribute to make it a little more accessible. I first used MiKTeX and TeXworks, but I found the usage of shortcuts hard and didn’t like the PDF viewer. I switched to Notepad++, SumataPDF (both great tools) and a handy script. It was great until my projects got bigger. So I used VSCode and LaTeX Workshop and I loved it. All my shortcuts and tools I used before now applied to LaTeX. I was satisfied until I realized how slow MiKTeX on Windows is compared to Linux. I love Linux, but some things are more convenient on Windows. I started my Bachelor theses about development environments and the usage of container tools. So I have further developed my setup to bring everything together. I found that the tianon/latex</code> and other image were outdated and did not meet my expectations. I rather created my own image. It is own the large side, but I rather have all my tools there at any time instead of being slowed down by missing them or have to install packages manually. I hope some of you find some interesting tips and tricks in my setup. If you find any issues let me know</a> See you! Reddit Video Downloader 2021-12-24T00:00:00+00:00 One Day Build: WebApp to download Videos from Reddit</h1> It has become kind of annoying to download a simple video from Reddit. My sister wanted to save a video but couldn’t do it. So, I did what all programmers do - I wrote a small app. You can try it out here</a> or use it via your terminal like described below. Some Background</h2> The URLs of videos on Reddit are not directly visible. You have dig through some HTML tags and even more annoying, Reddit sores to video and audio files separably. So we need to find the URLs, download the content and combine video and audio. For this I wrote a small Python backend. The Backend</h3> The backend requires a URL as an argument, and makes a request to that URL. The result is analyzed with BeautifulSoup</code></a>. After the video and audio URL got extracted, both files are downloaded separably and are then combined with ffmpeg</code></a>. I used Flask to handle the HTTP requests and provide routing. It is a simple-to-use minimal framework that was fast to implement. I wanted people to be able to use the backend directly, so it is possible to download videos with a terminal. The following bash function should download any Reddit video for you without the need to use any browser: 1# Add this function to yor terminal session - or put it into your .bash_rc 2# This need wget, curl and jq 3reddvid () { 4 wget https://reddvid.herokuapp.com/$(curl -X GET 'https://reddvid.herokuapp.com/' --form "url=\"${1}\"}" | jq -r '.download') 5} 6# Call the function 7reddvid <REDDIT_URL></code></pre> Any non-technical person has the option to use the WebApp frontend. The Frontend</h3> The WebApp is a simple VUE app that was build with nuxt.js, simply because I already know VUE and I like my WebApps to be a static page that can be served by any HTTP-Server. This allows for a broad range of deployment options. Deployment</h3> The deployment strategy I chose was only based on tractability and cheep criteria. I am a student that did this project for fun and did not and couldn’t spend much on server infrastructure. The backend is deployed via a Python Docker Image</a> on Heroku</a> with a deployment option that goes to sleep after 30 min. The WebApp is hosted on Netlify</a> which is a cheap option for static apps that creates a new deployment for every commit. Have Fun</h2> This is a project I build for my own usage and without much testing. I’m sure it has bugs and that it will beak on specific actions. Just use it if it is handy to you, share your thoughts, encountered issues and possible improvements. Swarm knowledge is the bast way to learn!!! See you! Install & Configure Grafana Promtail 2021-12-21T00:00:00+00:00 Install & Configure Promtail for Grafana Loki</h1> This post will demonstrate a reliable and easy way to install Promtail to your server in order to collect all system logs in a central place for proper monitoring. NOTE: This post will only cover the install and setup for Linux based systems About Promtail</h2> Promtail is part of the Grafana platform. The Grafana platform is increasing in popularity due to its open-source nature, easy usability and ability to plug into existing infrastructure. Grafana allows to visualize and query data from multiple sources. The data can be stored in InfluxDB</a>, ELK</a>, Prometheus</a> or the in-house Loki</a> data store. Loki is often called the Prometheus version of logs. But to query log-data in Loki over Grafana, the logs have to be shipped first. For Docker, this can easily be done with the Docker Loki Log-Driver</a>, but what about logs of none dockerized applications or logs of the host system? This is where Promtail comes in! It constantly reads log-files (e.g. in /var/log</code>) and sends them to a Loki instance. Installing Promtail</h2> Promtail is available as a static binary on the official Grafana Loki GitHub</a> page. We could download it by hand, or we could use the terminal like a real programmer: 1# Download latest: 2curl -s https://api.github.com/repos/grafana/loki/releases/latest | \ 3 grep browser_download_url | \ 4 cut -d '"' -f 4 | grep promtail-linux-amd64.zip | \ 5 wget -i - 6# Unzip and 7unzip promtail-linux-amd64.zip 8mv promtail-linux-amd64 /usr/local/bin/promtail 9# Verify 10promtail --version 11</code></pre> We are using the GitHub API to get the latest release of Promtail and are filtering for the linux-and64</code> binary with grep</code>. The result gets piped to wget</code> in order to download the zip file. Subsequently, we unzip the archive and move it to a place that is in our PATH</code>. Now we can test if Promtail is working. Now we need a Promtail config: 1# Promtail config 2server: 3 disable: true 4 5positions: 6 filename: /tmp/promtail-positions.yaml 7 8clients: 9 - url: http://<SERVER>:3100/loki/api/v1/push 10 11scrape_configs: 12- job_name: authlog 13 static_configs: 14 - targets: 15 - localhost 16 labels: 17 job: authlog 18 shipper: promtail 19 __path__: /var/log/auth.log 20 21- job_name: daemonlog 22 static_configs: 23 - targets: 24 - localhost 25 labels: 26 job: syslog 27 shipper: promtail 28 __path__: /var/log/daemon.log</code></pre> Our Promtail do not need to receive any logs we are just querying local files, so we can disable the server. The positions-file is a marker file for Promtail to remember where it stopped reading a log file, and the client is the destination of our logs. The scrape_configs</code> specifies what logs Promtail should send to Loki. In this example, we are using daemon.log</code> and auth.log</code>. There are tones of other possibilities, like access logs from ´nginx´ and alike, but this would be too much for this post. We could start Promtail with promtail -config.file promtail-config.yml</code>, but we would have little control over the process, and it would not start again if the process fails or the host is restarted. So let’s create separate user for Promtail (optional) and a service: 1# Create promtail user 2useradd -r promtail 3usermod -a -G adm promtail 4# Create a service 5tee /etc/systemd/system/promtail.service<<EOF > /dev/null 6[Unit] 7Description=Promtail service 8After=network.target 9 10[Service] 11Type=simple 12User=promtail 13ExecStart=/usr/local/bin/promtail -config.file /path/to/promtail-config.yml 14Restart=always 15 16[Install] 17WantedBy=multi-user.target 18EOF</code></pre> Now we can simply run the following and have a reliable and good Promtail setup that can easily be automated with tools like Ansible or Puppet. 1# Start the service 2sudo systemctl daemon-reload 3sudo systemctl enable promtail.service 4sudo systemctl start promtail.service 5# Verify status 6sudo systemctl status promtail.service</code></pre> See you! Dockerfile for NodeJS & Typescript 2021-11-15T00:00:00+00:00 How to build a Docker image for a TypeScript based NodeJS project</h1> You just finished your TypeScript project and now you want to release it to the internet by using ECS, GKE, Azure or any other cloud service? Or maybe you just want to share it with your co-workers without having them to install and configure additional software. Docker is a great use case for this! This post shows demonstrates how to create an optimized production ready Docker image for your project. Pre-Requirements</h2> In order to build a Docker image you must have access to system that has the Docker engine installed. Depending on your system you need one of the following: Host</th> Version</th> Link</th></tr></thead> Linux</td> docker-ce</td> Download</a></td></tr> Windows</td> Docker Desktop</td> Download</a></td></tr> MacOS</td> Docker Desktop</td> Download</a></td></tr> </tbody></table> Note: Docker builds upon functions provided by the Linux kernel. Windows and macOS need to virtualize the kernel. Therefore, the best performance is achievable on linux. Project setup</h2> Assuming the following file structure: project/ - src/ - server.ts - util.ts - node_modules - index.ts - tsconfig.json - package.json</code></pre> In order to run our app we need to: Install our dependencies</li> Transpile TS to JS</li> Start our app</li> </ol> This needs to be specified in our Dockerfile which acts as the build instructions for our Docker image. We create a file named Dockerfile and add the following: 1FROM node 2 3WORKDIR /app 4COPY . /app 5RUN npm install && npm build</code></pre> Our image is based on the official NodeJS Docker image. We copy our source code into the image and run the install and compile command. This would work but is not an optimal solution. We are using Docker to get a consistent runtime that works everywhere but by not using a specific tag for our base image we are losing the consistence. If no tag is specified the latest</code> tag is used which changes often. A lot of images include a lot of software that we actually do not need. These increases image size, storage consumption and network transfer times. It also increases to attack surface because any other program can have additional vulnerabilities. Let’s fix some of this: 1FROM node:16-alpine 2 3WORKDIR /app 4COPY package.json package-lock.json /app/ 5RUN npm install 6 7COPY . /app 8RUN npm run build</code></pre> Now we are using an exact version and the much smaller alpine version of Nodes Docker image. We are also utilizing Dockers caching functionality. Docker only runs the commands that changed since the last run. Since we are not modifying the package.json</code> as often as our source code we can cache the install command and save time. This is better but not perfect yet. In order to run our app we neither need the TypeScript files nor the tsc</code> compiler. This is where multi-stage build come handy. We can build our JavaScript in one stage and then copy the result to another stage that is actually used. 1FROM node:16-alpine as build 2 3WORKDIR /app 4COPY package.json package-lock.json /app/ 5RUN npm install 6 7COPY . /app 8RUN npm run generate 9 10FROM node:16-alpine as production 11 12USER node 13WORKDIR /app 14COPY --chown=node:node --from=build /app/compiledJS /app 15RUN npm install --only=production</code></pre> Now the code is compiled in one stage and shipped in another. We are also changing to user in order to omit running our app as root. Lower privileges are another step to increase your app’s security. To increase the flexibility of our Dockerfile we can add arguments. With arguments, we can influence the building process on execution without modifying the Dockerfile. A common argument is the VARIANT</code> variable. Which allows to change the version of our base image. The last step is to include an ENTRYPOINT</code> to specify what command should be executed when a container is created. I strongly recommend to directly call node index.js</code> instead of using npm start</code>. When using npm</code>, npm becomes our main process and acts as a process manager for the node process. Unfortunate npm</code> performs poorly as a process manager. It does not redirect any control or exit signals to the main node process. This can lead to a container not willing to stop or an unexpected shutdown of our node process without properly handling exit tasks. The final Dockerfile should look something like this: 1FROM node:16-alpine as build 2 3WORKDIR /app 4COPY package.json package-lock.json /app/ 5RUN npm install 6 7COPY . /app 8RUN npm run generate 9 10ARG VARIANT=16-alpine 11FROM node:${VARIANT} as production 12 13USER node 14WORKDIR /app 15COPY --chown=node:node --from=build /app/compiledJS /app 16RUN npm install --only=production 17 18ARG COMMIT_TAG="none" 19LABEL COMMIT_TAG=$COMMIT_TAG 20ENTRYPOINT [ "node" , "index,js"]</code></pre> We now can build our image with: docker build -t my-app --build-arg COMMIT_TAG="version-1.0.0" .</code> Placeholder 2021-11-14T00:00:00+00:00 More is comming soon</h1> Lorem ipsum dolor sit amet, consectetur adipiscing elit. Quisque consequat pretium mauris ac scelerisque. Sed semper egestas nisl. Integer tempus turpis mauris, eu dictum mauris porta id. Quisque nec ultricies erat. Mauris accumsan nisl velit, nec semper velit vehicula ornare. Pellentesque eu tortor ac ante finibus tincidunt. Phasellus tortor arcu, mattis id mauris a, fermentum pulvinar sem. Sed quis interdum velit. Vestibulum convallis arcu eget rutrum elementum. Nullam sapien orci, ullamcorper sit amet consequat at, tempus at dolor. Nunc et malesuada odio. Nam laoreet tempor metus vel vestibulum. Nullam convallis, risus eu venenatis rhoncus, sem ex bibendum enim, vel interdum dolor nisl quis urna. Curabitur in sodales enim, vel mattis velit See you

Native IPv6 Kubernetes for true edge routing

Gateway API doesn't solve real problems - yet

AWS Web-Identity-Token - The free IDP for all your OnPrem solutions

Reduce your Token-Usage for all non AWS-Apps with AWS Web-Identity-Token</h1>

How to use</h2> So how do I start using this?</p>

The Grafana trust problem

Rootless GitLab Runners

Rootless GitLab Runners</h1>

Follow Up: Let's talk about anonymous access to Kubernetes

Follow Up: Let’s talk about anonymous access to Kubernetes</h1>

Level up your Ansible Code - Creating Golden Images

Level up your Ansible Code - Creating Golden Images</h1>

Understanding and using modern day authentication frameworks to improve security, productivity and user acceptance

Understanding modern day Authentication with OAuth2 and OIDC</h1>

What is new in containerd 2.0

Making OnPrem Kubernetes feel like AKS/EKS/GKE

New Website - Abandon JavaScript Frameworks

Benchmarking what actually drive our containers

The recurring problem of the Kubernetes metrics server and insecure Kubelet certificate

The recurring problem of the Metrics Server and Insecure Kubelet certificate</h2>

Using AWS from GitHub without Credentials

Pre-Requirements</h2> To follow this guide, you need:</p>

Comparing GitHub Actions with GitLab CI/CD - A deep dive!

Comparing GitHub Actions with GitLab CI/CD</h1>

Building preconfigured OS images with HashiCorp Packer

Building preconfigured OS images with Packer</h1>

Building and Cracking the Enigma - Part I

How to send OnPrem Prometheus metrics to MS Azure

Sending metrics to Azure - with Prometheus</h1> With this post, I want to provide a quick demonstration on how to send Prometheus metrics to Azure managed Prometheus. Let’s get stared.</p>

Setting up Azure Prometheus</h2> For simplicity, I will demonstrate the set up process using the Azure Portal. But the process can also be automated via the CLI or terraform. You have to do the following steps:</p> Log into Azure</li> Search for Prometheus</li>

The Notebook for Developers - now even better

Broken NodeJS Apps due to secerity dot-release - Debugging the NodeJS CVE-2022-32213 fix

Broken NodeJS Apps due to security dot-release</h1>

Some Background Information</h3> For a better understanding let me give a simplified overview of our system architecture: </p>

Productive Working - Be lazy and get things done - Part II

Productive Working - Be lazy and get things done - Part I

Sending metrics to Azure - with Prometheus</h1>

With this post, I want to provide a quick demonstration on how to send Prometheus metrics to Azure managed Prometheus. Let’s get stared.</p>