Bilingual

Fixing a kubelet memory leak in Kubernetes 1.36
a month ago
- The author discovered a memory leak in Kubernetes 1.36 on a small test cluster, revealed due to high memory pressure.
- Investigation showed the kubelet process itself was growing in memory usage, not the application pods.
- Heap profiling using Go's pprof identified nearly a million leaked contexts as the primary cause, consuming most memory.
- A code change in Kubernetes 1.36 introduced a context leak by overwriting cancel functions without canceling old contexts in the pod reconciliation loop.
- AI tooling (Codex) helped pinpoint the regression in the codebase, specifically in the startPodSync function.
- A patch was developed and submitted; it was simplified to revert the immediate leak after integration test failures revealed additional context issues.
- The Kubernetes team was responsive, with the fix merged into the master branch for v1.37 and backported to v1.36.
- Restarting kubelet temporarily resolved memory pressure, but the underlying leak persisted until patched.
- Lessons include the value of heap profiling, the time dimension in long-running systems, and infrastructure-level issues.
- A one-liner command to monitor kubelet memory usage showed a significant drop from 974 MiB to 110 MiB after restart.
DLL that was not present in memory despite not being formally unloaded
a month ago
- A bug reported that shell32.dll was causing crashes in a third-party program, but investigation revealed shell32 was a victim.
- Crash dumps showed a stack overflow due to a recursive exception handling death spiral, starting from an access violation in combase!CoTaskMemFree.
- Analysis found that combase.dll had been forcibly unloaded from memory (likely via VirtualFree), not by FreeLibrary, causing subsequent calls to fail.
- Shell32 was blamed as it was the first to call into the unloaded combase.dll during process termination, triggering the crash loop.
- Reviewing 100 recent crashes showed 46% were due to similar rogue DLL force-unloads, with different DLLs affected, indicating a bucket spray issue.
- The root cause is unknown, but theories point to memory corruption or uninitialized variables in the third-party program or other components.
- Wil comments suggest avoiding static object destruction during shutdown and note existing shutdown-aware types in Microsoft code.
Underappreciated Builtin: Grand Unified Debugger
a month ago
- The Grand Unified Debugger (GUD) in Emacs is an underappreciated built-in feature that supports multiple debuggers like GDB, LLDB, Perl, Java, Python, and others.
- A demo using GDB with a Fibonacci program shows how to set breakpoints via fringe clicks, use toolbar buttons (Run, Next Line, Step Line, Up/Down Stack), and manage breakpoints in a dedicated window.
- GUD provides features like an I/O window for output, variable inspection by hovering, a locals window, and variable watching for automatic updates when values change.
It's not me, it's the compiler
a month ago
- A programmer discovered a compiler bug in Rust while refactoring a JavaScript engine's parser.
- The bug was in a function where using 'x as u32' for boolean addition caused incorrect behavior, but using an explicit if-else worked.
- Assembly inspection confirmed the compiler omitted an increment instruction, leading to a miscompilation.
- The issue was reported as 'Suspected miscompilation' and quickly fixed within 18 hours by the Rust community.
- The bug was classified as p-critical and i-miscompile, highlighting rare but serious compiler errors.
No laptop? No problem. Code on the Go brings the debugger to your phone
a month ago
- Code on the Go (CoGo™) enables debugging on a single Android device without a PC, cable, or external connections.
- It uses a local socket-based approach with Java Debug Wire Protocol (JDWP) instead of the traditional ADB bridge model.
- Privileged access is bootstrapped via a modified version of Shizuku, leveraging wireless debugging pairing for shell-level access.
- The target app is launched in debug mode using the --attach-agent flag to inject libjdwp.so, establishing a debug connection.
- The IDE communicates via JDI using sun-jdi.jar from AOSP, enabling breakpoints, stepping, variable inspection, and editing.
- Current limitations include Java-only debugging, primitive variable editing only, user-dependent bootstrapping, and reliance on system JDWP.
- Future plans include Kotlin support, conditional breakpoints, expression evaluation, and attaching to running processes.
The Safari MCP server for web developers
25 days ago
- Safari MCP server introduced in Safari Technology Preview 247.
- Enables agents to access browser data (DOM, network requests, screenshots, console logs).
- Speeds up debugging by reducing context switching between browser and terminal.
- Use cases include checking rendering, improving Safari compatibility, analyzing performance, and verifying accessibility.
- Available tools include console messages, network requests, page interactions, screenshots, and JavaScript evaluation.
- Setup requires installing Safari Technology Preview and enabling developer features.
- Local-only operation ensures data privacy and no external network calls.
- Goal is to enhance AI-assisted web development workflows for Safari.
Game Boy Advance Dev: Logging to the Console
24 days ago
- mGBA allows GBA developers to use printf-like functionality for debugging by writing to specific memory-mapped registers.
- Logging is enabled by writing 0xC0DE to REG_LOG_ENABLE, writing the message to REG_LOG_BUFFER, and then sending a log level to REG_LOG_SEND.
- The log levels include ERROR, WARNING, INFO, and DEBUG, with each corresponding to a specific value for the REG_LOG_SEND register.
- Logs can be viewed in mGBA's Tools > View logs... window or output to the terminal if launched from the command line with --log-level.
- Advanced logging features include printf-style formatting using vsnprintf and conditional compilation with a macro to exclude logging from release builds.
Where did my segfault go?
16 days ago
- A segmentation fault message is printed by the shell when it reaps a child process that died from SIGSEGV, not by the crashing program itself.
- When using commands like 'bash -c', bash may exec directly into the command without forking, preventing the shell from being the parent and printing the error.
- Using a wrapper script or forcing a subshell (e.g., with parentheses) ensures a shell is present as the parent to detect and report the segmentation fault.
- Adding a command after the crashing one (e.g., 'true') also prevents exec optimization, allowing the shell to print the error message.
Show HN: Web-based Jujutsu revset explorer and debugger
7 days ago
- The page provides a playground for exploring and debugging jj revsets.
- Users can edit the graph and evaluate revsets.
- Setups can be shared via a permalink.
- Nodes can be selected with Enter/Space and moved with arrow keys.
- Nodes can be deleted with Delete, and Escape cancels selection.
- Edges can be selected and deleted similarly.
- A TOML 'revset-aliases' table can be specified.
A flaky test exposed a Redis client use-after-free
6 days ago
- Non-deterministic bugs are harder to debug because they require repeated execution and often involve data corruption that occurs before the crash.
- The bug was discovered in the redis-client library after a Redis gem upgrade caused flaky tests in Buildkite's CI system.
- The team used Test Engine to correlate test reliability, core dumps to inspect crash states, and ASAN to identify a heap use-after-free in hiredis C extension.
- The fix involved disabling the C extension for Redis connections and reporting the issue upstream.
- The debugging process combined skill, teamwork, and luck, with tools enabling detection of memory corruption that would otherwise take months to fix.
We audited 1,751 "relationship milestones" our RAG extractor wrote 62% were junk
6 days ago
- A conversational AI with long-term memory stores (vector and fact) had an extractor that over-produced junk milestones (~6.5/day), causing prompt monoculture with forced romantic declarations.
- Diagnosis used a trace inspector and golden set; a keyword-as-necessary-condition rule blocked 94% of historical junk while passing 100% of positive controls.
- Fix A stopped the tap: keyword gate, roleplay guard, tightened dictionaries, anti-multi-label, deployed behind a test suite.
- Fix B cleaned existing data via reclassification (never delete) using an LLM judge, after avoiding prompt contamination and unsound shortcuts.
- Measurement showed new milestones/day dropped 92% (6.5 to 0.5), entity labels per message reduced to 1, and forced milestones per prompt dropped from 2.0 to 0.65.
- A post-fix incident revealed a relevance failure: retyped facts were force-injected unconditionally, causing accurate but irrelevant memories, highlighting a gap in context-aware injection.
- Qualitative contrast: before fix, assistant drifted romantic; after fix, it stayed on-topic, recalled specifics correctly, and admitted gaps instead of fabricating.
- Remaining issues include always-include inflation, extractor gate leaks, date over-extraction, parallel junk populations, debugger persona limitations, and untested scale/volume degradation.
- Key takeaways: memory can be accurate but wrong (relevance vs truth), instrument before tuning, and reversibility enables honest shipping and measurement.
Pinball on 64-Bit Alpha AXP Windows NT
6 days ago
- Pinball (3D Pinball for Windows – Space Cadet) was removed from Windows Vista due to a reported collision detection bug in 64-bit versions.
- NCommander challenged the removal in 2021, finding 64-bit versions playable on XP and pre-release Vista, suggesting UI incompatibility with Vista design.
- Raymond Chen later clarified the bug was specific to the 64-bit Alpha AXP version, which had a severe collision detection bug.
- The author debugged Pinball on Alpha NT emulators (QEMU/ES40) after fixes enabled booting the 64-bit build 2210.
- Two bugs were found: a WNDCLASSA struct alignment issue causing crashes in RegisterClassA, and pointer truncation in mciseq.dll due to ZAPNOT instruction misuse.
- Patching these bugs made Pinball work flawlessly on 64-bit Alpha NT, without encountering the original collision detector bug.
- Author speculates the collision bug might have been introduced after build 2210 or only manifests in release builds.
<antirez>
8 hours ago
- A long-running process crash is problematic, especially when it holds significant memory state; web programming frameworks that create new interpreters per page view offer a more reliable paradigm.
- Redis experienced a crash in its radix tree implementation, with error addresses zero-padded at the end of the heap, indicating a read overflow beyond memory bounds.
- Extensive fuzz testing failed to reproduce the crash because the bug was already fixed in a newer version of the code, but not backported to Redis 4.0 due to being bundled in a Streams commit.
- The crash analysis involved examining CPU registers and disassembly, revealing that the compiler-generated code did not match the C source due to version mismatch.
- Key lessons include the value of detailed crash reports for debugging system software, the necessity of understanding AMD64 assembler, the power of fuzz testing with reference comparisons, and careful management of bug fixes across branches.
- Seeking help from knowledgeable peers (e.g., Fedor Indutny) was crucial in resolving the mystery.
<antirez>
8 hours ago
- Lua scripting is the most successful Redis feature, enabling programmable and fast scripts that are cached and executed using the same interpreter.
- Two key features requested by users are a proper Lua debugger and script effects replication, both implemented by the author.
- The debugger is a remote, interactive, client-server tool that uses fork() to avoid blocking the server and enables rollback of dataset changes.
- Key debugger features include breakpoints, stepping, variable inspection, backtrace, and custom Lua functions like redis.breakpoint() and redis.debug().
- Script effects replication allows replicating only the write commands resulting from a script, improving efficiency and enabling use of non-deterministic commands like TIME.
- The feature supports opt-in via redis.replicate_commands(), with advanced control using redis.set_repl() to exclude certain commands from replication.
- Both features will be available in Redis 3.2, scheduled for release in December 2015.

debugging

Fixing a kubelet memory leak in Kubernetes 1.36

DLL that was not present in memory despite not being formally unloaded

Underappreciated Builtin: Grand Unified Debugger

It's not me, it's the compiler

No laptop? No problem. Code on the Go brings the debugger to your phone

The Safari MCP server for web developers

Game Boy Advance Dev: Logging to the Console

Where did my segfault go?

Show HN: Web-based Jujutsu revset explorer and debugger

A flaky test exposed a Redis client use-after-free

We audited 1,751 "relationship milestones" our RAG extractor wrote 62% were junk

Pinball on 64-Bit Alpha AXP Windows NT

<antirez>

<antirez>