Bilingual

Fixing a kubelet memory leak in Kubernetes 1.36
21 days ago
- The author discovered a memory leak in Kubernetes 1.36 on a small test cluster, revealed due to high memory pressure.
- Investigation showed the kubelet process itself was growing in memory usage, not the application pods.
- Heap profiling using Go's pprof identified nearly a million leaked contexts as the primary cause, consuming most memory.
- A code change in Kubernetes 1.36 introduced a context leak by overwriting cancel functions without canceling old contexts in the pod reconciliation loop.
- AI tooling (Codex) helped pinpoint the regression in the codebase, specifically in the startPodSync function.
- A patch was developed and submitted; it was simplified to revert the immediate leak after integration test failures revealed additional context issues.
- The Kubernetes team was responsive, with the fix merged into the master branch for v1.37 and backported to v1.36.
- Restarting kubelet temporarily resolved memory pressure, but the underlying leak persisted until patched.
- Lessons include the value of heap profiling, the time dimension in long-running systems, and infrastructure-level issues.
- A one-liner command to monitor kubelet memory usage showed a significant drop from 974 MiB to 110 MiB after restart.
I ported Kubernetes to the browser
21 days ago
- Webernetes is a partial port of Kubernetes to TypeScript, enabling clusters to run in the browser.
- It is about 140KiB gzipped and avoids WebAssembly due to size constraints, instead being a from-scratch TypeScript implementation.
- The project includes features like pod lifecycles, DNS, networking, and deployments, with a browser-based image registry.
- Almost all code was generated by LLMs, but rigorous manual review and integration testing against k3s were used to ensure correctness.
- It includes 204 integration tests and 1,855 unit tests to validate behavior matches real Kubernetes.
- The author spent two months on the project, with significant LLM token usage, costing over $1,800 in the final week.
- Webernetes is intended for interactive educational content, not production, and future expansions are planned based on user needs.
Netflix Simplified Batch Compute with Kueue
24 days ago
- Netflix transitioned from its custom managed batch solution (CMB) to the Kubernetes-native Kueue for batch workload queuing and scheduling.
- CMB managed batch jobs using a tenant hierarchy with reserved and shared capacity, but lacked features like preemption and faced development challenges.
- Kueue was chosen due to its compatibility with Titus scheduling, multi-tenant quota support, and native features like preemption and fair sharing.
- The migration to Netflix Batch (using Kueue) was transparent to users, maintained throughput, and involved converting tenants to Kueue primitives like ClusterQueue and LocalQueue.
- Key lessons included maintaining API parity, migrating complex use cases early, and adjusting Kueue configurations for high throughput.
- Kueue is now fully productionized at Netflix, managing millions of batch jobs, improving resource utilization, and enabling fair sharing with preemption.
- Future plans include enrolling more Titus workloads and leveraging Kueue for internal Kubernetes-native training infrastructure.
Federating Clusters for Zero-Downtime Kubernetes
a month ago
- Linkerd's multicluster extension enables three modes: federated, flat mirroring, and gateway mirroring, which can coexist on the same set of linked clusters.
- Federated mode allows services with the same name across clusters to be unioned into a single load-balanced endpoint, providing automatic failover with no application changes.
- Flat mirroring enables explicit routing to remote services by name, with traffic going directly to remote pods, requiring a flat network with pod IPs routable.
- Gateway mirroring routes traffic through a gateway IP, allowing cross-cluster connectivity without requiring pod-to-pod network reachability.
- A demo setup across three GKE clusters demonstrates all three modes, with VPC peering for flat networking and a shared trust anchor for cross-cluster mTLS.
- Chaos testing shows that federated services automatically redistribute traffic when a cluster fails, while mirrored services require client-side failover handling.
- Key operational considerations include non-overlapping CIDRs, proper VPC peering route exchange, controller count per link, and careful cluster node configuration to control costs.
- A decision framework helps choose the appropriate mode based on client needs, network constraints, and failover requirements.
FluxCD turns 10 today, stats and nostalgia
14 days ago
- Flux marks its 10th anniversary, having started with an initial commit on July 7, 2016 by Peter Bourgon.
- Flux pioneered GitOps, a term coined in a Weaveworks blog post, which emphasizes operations via pull requests and disaster recovery.
- The project evolved from a monolithic binary to Flux 2, utilizing the GitOps Toolkit with CRDs and controller-runtime for better modularity and security.
- Over the decade, Flux has seen significant contributions: 1,076 contributors, 44 repos, 17,946 pull requests, and 7,474 issues, with 30.2 billion container image downloads for Flux 2 alone.
- Flux is trusted in demanding environments like 5G towers, retail, cloud control-planes, and air-gapped networks due to its reliability, security, and robust machinery for Kubernetes deployments.
- The community is highlighted as integral to Flux's success, with maintainers emphasizing collaboration, technical excellence, and a shared commitment to improvement.
- Future directions include enhancing AI integration with tools like fluxcd/agent-skills and Flux Operator MCP, while prioritizing security, open APIs, and fast adaptation to changing infrastructure.
I Revived My Broken MacBook into My Own Personal Cloud
2 days ago
- Revived a broken 2017 MacBook Pro with a broken keyboard into a personal private cloud.
- Installed Ubuntu 24.04 Desktop using an old Raspberry Pi SD card as installer.
- Fixed Broadcom Wi-Fi drivers with a script that runs on every restart.
- Deployed k3s and Flux CD for GitOps-style Kubernetes orchestration.
- Disabled lid sleep to keep server running continuously.
- Solved network issues: ISP used CG-NAT, requested a public IP and kept IP-update script running.
- Used a local DNS provider (Domene.shop) for SSL certificates via Let's Encrypt.
- Deployed OpenCloud (file storage) and Open WebUI (AI chat) on the cluster.
- Managed Kubernetes manifests with takk, an infrastructure-as-code framework in Python.
- Built a FastAPI bridge to connect Obsidian notes to the AI for querying files.
- Added monitoring with Prometheus MCP and alerting via takk (email on crash loops).
- Plan to add local LLM, self-healing agents, and hardened backups.
Cheap Self-Hosted Kubernetes on Hetzner Cloud
3 hours ago
- terraform-hcloud-kube-hetzner is an open-source Terraform project that simplifies deploying a production-ready K3s cluster on Hetzner Cloud.
- Hetzner is a German infrastructure provider offering high-performance hardware at reasonable prices, with a solid API and features like private networking, load balancers, and VMs.
- Kube-Hetzner codifies best practices for running K3s reliably on Hetzner Cloud, handling provisioning, networking, load balancers, node pools, and automated rolling upgrades.
- It uses openSUSE MicroOS, an immutable, container-centric OS with atomic updates and a read-only root filesystem for security and reliability.
- K3s is a lightweight, conformant Kubernetes distribution that packages components into a single binary, simplifying cluster operations.
- The blog plans a highly available cluster with 3 control plane and 3 worker nodes across three Hetzner zones, enabling local-path provisioning and optional Longhorn for replicated storage.
- Monthly cost estimate: ~€58 with VAT for the described setup, which is cheaper than typical managed Kubernetes services (e.g., $70-$75 on GCP/AWS).
- Cost-saving options include using a single load balancer, MetalLB, or removing public IPs via a bastion host or VPN.
- Deployment steps: clone starter project, install Terraform, hcloud CLI, kubectl; run terraform init, plan, apply; retrieve kubeconfig and test connectivity.
- Backups use K3s' etcd snapshots to S3, and node backups via Hetzner's built-in backup mechanism.
- Upgrades are handled by Rancher's System Upgrade Controller (SUC) and kured for automated node reboots with configurable maintenance windows.

Hasty Briefsbeta

#kubernetes

Fixing a kubelet memory leak in Kubernetes 1.36

I ported Kubernetes to the browser

Netflix Simplified Batch Compute with Kueue

Federating Clusters for Zero-Downtime Kubernetes

FluxCD turns 10 today, stats and nostalgia

I Revived My Broken MacBook into My Own Personal Cloud

Cheap Self-Hosted Kubernetes on Hetzner Cloud