Bilingual

Fixing a kubelet memory leak in Kubernetes 1.36
5 days ago
- The author discovered a memory leak in Kubernetes 1.36 on a small test cluster, revealed due to high memory pressure.
- Investigation showed the kubelet process itself was growing in memory usage, not the application pods.
- Heap profiling using Go's pprof identified nearly a million leaked contexts as the primary cause, consuming most memory.
- A code change in Kubernetes 1.36 introduced a context leak by overwriting cancel functions without canceling old contexts in the pod reconciliation loop.
- AI tooling (Codex) helped pinpoint the regression in the codebase, specifically in the startPodSync function.
- A patch was developed and submitted; it was simplified to revert the immediate leak after integration test failures revealed additional context issues.
- The Kubernetes team was responsive, with the fix merged into the master branch for v1.37 and backported to v1.36.
- Restarting kubelet temporarily resolved memory pressure, but the underlying leak persisted until patched.
- Lessons include the value of heap profiling, the time dimension in long-running systems, and infrastructure-level issues.
- A one-liner command to monitor kubelet memory usage showed a significant drop from 974 MiB to 110 MiB after restart.
I ported Kubernetes to the browser
5 days ago
- Webernetes is a partial port of Kubernetes to TypeScript, enabling clusters to run in the browser.
- It is about 140KiB gzipped and avoids WebAssembly due to size constraints, instead being a from-scratch TypeScript implementation.
- The project includes features like pod lifecycles, DNS, networking, and deployments, with a browser-based image registry.
- Almost all code was generated by LLMs, but rigorous manual review and integration testing against k3s were used to ensure correctness.
- It includes 204 integration tests and 1,855 unit tests to validate behavior matches real Kubernetes.
- The author spent two months on the project, with significant LLM token usage, costing over $1,800 in the final week.
- Webernetes is intended for interactive educational content, not production, and future expansions are planned based on user needs.
Netflix Simplified Batch Compute with Kueue
8 days ago
- Netflix transitioned from its custom managed batch solution (CMB) to the Kubernetes-native Kueue for batch workload queuing and scheduling.
- CMB managed batch jobs using a tenant hierarchy with reserved and shared capacity, but lacked features like preemption and faced development challenges.
- Kueue was chosen due to its compatibility with Titus scheduling, multi-tenant quota support, and native features like preemption and fair sharing.
- The migration to Netflix Batch (using Kueue) was transparent to users, maintained throughput, and involved converting tenants to Kueue primitives like ClusterQueue and LocalQueue.
- Key lessons included maintaining API parity, migrating complex use cases early, and adjusting Kueue configurations for high throughput.
- Kueue is now fully productionized at Netflix, managing millions of batch jobs, improving resource utilization, and enabling fair sharing with preemption.
- Future plans include enrolling more Titus workloads and leveraging Kueue for internal Kubernetes-native training infrastructure.
Federating Clusters for Zero-Downtime Kubernetes
10 days ago
- Linkerd's multicluster extension enables three modes: federated, flat mirroring, and gateway mirroring, which can coexist on the same set of linked clusters.
- Federated mode allows services with the same name across clusters to be unioned into a single load-balanced endpoint, providing automatic failover with no application changes.
- Flat mirroring enables explicit routing to remote services by name, with traffic going directly to remote pods, requiring a flat network with pod IPs routable.
- Gateway mirroring routes traffic through a gateway IP, allowing cross-cluster connectivity without requiring pod-to-pod network reachability.
- A demo setup across three GKE clusters demonstrates all three modes, with VPC peering for flat networking and a shared trust anchor for cross-cluster mTLS.
- Chaos testing shows that federated services automatically redistribute traffic when a cluster fails, while mirrored services require client-side failover handling.
- Key operational considerations include non-overlapping CIDRs, proper VPC peering route exchange, controller count per link, and careful cluster node configuration to control costs.
- A decision framework helps choose the appropriate mode based on client needs, network constraints, and failover requirements.

Hasty Briefsbeta

#kubernetes

Fixing a kubelet memory leak in Kubernetes 1.36

I ported Kubernetes to the browser

Netflix Simplified Batch Compute with Kueue

Federating Clusters for Zero-Downtime Kubernetes