Hasty Briefsbeta

All tags

#debugging

163 stories total

Bilingual

DuckGPT – agent for solving complex coding problems
5 months ago
- DuckGPT is introduced as a helpful duck assistant for coding problems.
- Users are encouraged to describe their coding issues line by line for assistance.
- The service promises responses with the intelligence of a rubber duck, implying a focus on debugging through explanation.
Lot of software engineers don't understand basic networking
5 months ago
- Most software engineers lack deep networking knowledge, relying on abstractions.
- Abstractions fail during debugging, leading to blind troubleshooting without underlying knowledge.
- The gap in networking knowledge causes longer debugging, fragile architectures, security issues, and missed performance optimizations.
- Key networking concepts to learn include OSI model layers, IP addressing, TCP/UDP, DNS, and HTTP/HTTPS.
- Practical debugging tools include ping, traceroute, dig, curl, netstat, ss, and tcpdump.
- Understanding distributed systems patterns like timeouts, retries, circuit breakers, and load balancing is crucial.
- Resources for learning include textbooks, zines, guides, and cloud provider documentation.
- Engineers with networking knowledge debug faster, design resilient systems, and have a competitive edge.
Don't Gobble Up Errors
5 months ago
- Avoid silently gobbling up errors; ensure errors are logged and handled appropriately.
- Letting an error bubble up is often better than incorrect error handling in a try/catch block.
- Example of bad practice: returning null for a missing trace_id, which can lead to undefined behavior.
- Good practice: throw an error with context when a required trace_id is missing.
- Best practice: sometimes, omitting try/catch entirely and letting the error fail naturally is acceptable.
- Silently handling errors can hide issues, making debugging harder due to lack of logs or error visibility.
A kernel bug froze my machine: Debugging an async-profiler deadlock
5 months ago
- A Linux user encountered a machine freeze when using async-profiler with QuestDB, traced to a kernel bug in version 6.17.
- The bug was a deadlock in the perf_events subsystem, specifically in the hrtimer_cancel function, causing the system to freeze.
- A workaround was found by starting the profiler with the -e ctimer option, avoiding the problematic kernel feature.
- The user set up QEMU to debug the kernel, reproducing the freeze and using GDB to inspect the deadlock in real-time.
- The debugging revealed that the deadlock occurred when hrtimer_cancel was called from within its own callback, leading to an infinite loop.
- Attempts to manually fix the deadlock by altering return values in GDB were partially successful but not reliable for production use.
- The conclusion was to use the workaround or wait for a kernel fix, with insights gained into kernel debugging and perf_events.
Debug Mode for LLMs in vLLora
5 months ago
- vLLora introduces Debug Mode for LLM requests to inspect and edit requests before execution.
- Debug Mode allows users to pause, inspect, edit, and continue LLM requests, similar to software debugging.
- Key features include inspecting the exact request, editing messages, parameters, and tool definitions, and continuing execution.
- Debug Mode addresses common issues like silent tool-call failures, corrupted context, and lack of visibility in LLM workflows.
- When a request pauses, users can view the model, message array, parameters, tool definitions, and any injected fields.
- Edits in Debug Mode affect only the current request, allowing for quick validation and testing without changing application code.
- Debug Mode is particularly useful for agents, helping catch drift early and avoid rerunning long workflows.
- The feature aims to make debugging LLM systems less tedious and more efficient.
Hacking LLDB for a great Zig debugging experience
5 months ago
- Zig programming language debugging in LLDB is challenging due to unreadable slices, optionals, and error unions.
- zdb is introduced as an LLDB plugin to improve Zig debugging without rebuilding LLDB or using Python scripts.
- The plugin provides readable output for Zig types and enables Zig-like expressions in LLDB.
- zdb works by accessing LLDB's internal APIs, rewriting expressions, and using offset tables for compatibility across LLDB versions.
- Limitations include lack of GUI tree expansion, but CLI expressions work well.
- Comparison shows zdb offers a balance between usability and practicality compared to stock LLDB, Python scripts, and zig-lldb.
Kernighan's Lever
4 months ago
- Debugging is twice as hard as writing the code initially.
- Writing clever code may make debugging more challenging, but it also fosters skill growth.
- Kernighan's lever: Motivation from debugging challenges drives long-term programmer improvement.
- Avoiding clever techniques to ease debugging can hinder personal development.
- Flow state in programming occurs when challenge matches skill level.
- Debugging pushes programmers to expand their skills beyond current capabilities.
Logging Sucks
4 months ago
- Logging is fundamentally broken in modern distributed systems.
- Traditional logs lack context and are optimized for writing, not querying.
- String search in logs is inefficient due to inconsistent formats and missing relationships.
- Structured logging is necessary but not sufficient for effective debugging.
- Wide events (canonical log lines) provide comprehensive context per request.
- OpenTelemetry is a delivery mechanism but doesn't solve what to log or add business context.
- High-cardinality and high-dimensionality data make logs useful for debugging.
- Tail sampling helps manage costs by keeping critical events (errors, slow requests, VIP users).
- Wide events transform debugging from archaeology to analytics.
- Modern tools support high-cardinality data, making wide events practical.
Finding Jingle Town: Debugging an N64 Game Without Symbols
4 months ago
- Ares is a cycle-accurate N64 emulator that supports remote debugging with GDB.
- Ares can run original ROMs or those built from decompilation projects, ensuring byte-for-byte accuracy.
- Setting up GDB for remote debugging involves creating a reverse tunnel and configuring the architecture to MIPS:4000.
- GDB can connect to Ares and use the ELF file from the decompilation project for debugging.
- Breakpoints can be set on functions by name, allowing for detailed debugging sessions.
- A function in Snowboard Kids 2 loads level-specific overlays, with a global variable indicating the current level index.
- Testing confirmed that the level index changes predictably as different levels are loaded.
- The debugging process is manual but effective for understanding game behavior like level initialization.
- The article mentions interest in visual debugging tools and externally generated debug symbols.
- The Snowboard Kids 2 decompilation project is open for contributors, with updates shared on Bluesky and Discord.
Parsing Advances
4 months ago
- Author writes a toy parser during Christmas break, inspired by Resilient LL Parsing Tutorial.
- Focus on producing syntax trees and diagnostics rather than bailing on first error.
- Pitfall: Infinite loops/recursion due to resilience (not consuming tokens).
- Example: Parsing function argument list may loop if expression doesn't consume tokens.
- Traditional solution: 'Fuel' mechanism and mental mapping of token-consuming functions.
- New solution: Assert parser advancement to materialize mental map in code.
- Benefits: Immediate error on non-advancement and explicit token-consuming function tracking.
- Base parser API introduced to implement this solution.
- Buggy function example provided to illustrate the initial problem.
The GDB JIT Interface
4 months ago
- GDB uses debug information (DWARF format) from compilers like Clang, GCC, and rustc to provide debugging features.
- GDB struggles with JIT-compiled functions because the instruction pointer isn't in the debug info tables, leading to '???' in the output.
- GDB provides a JIT interface that requires implementing functions like `__jit_debug_register_code` and managing a linked list (`jit_code_entry`).
- The JIT interface involves creating ELF/Mach-O objects with DWARF info, which is complex and requires stable memory pointers.
- GDB also supports a newer, more flexible interface where custom binary formats can be used, requiring a reader plugin to be loaded into GDB.
- The newer interface involves implementing functions like `gdb_read_debug_info` and `gdb_unwind_frame`, but many runtimes leave these stubbed.
- There's a suggestion to reuse the Linux perf map interface in GDB for JIT symbols, though it would offer limited functionality.
- V8 notes performance issues with the JIT interface due to O(n²) behavior in the linked list and the need to disable moving GC for stability.
- ART handles dead code entries by treating the GDB JIT linked list as a weak reference and periodically cleaning it up.
Profiling with Ctrl-C (2024)
4 months ago
- Ctrl-C profiling is a simple yet effective method for identifying performance issues in programs, especially for 'stupid problems' encountered by lazy people in unfriendly environments.
- The author shares personal experiences where Ctrl-C profiling helped identify issues like slow JSON parsing in debug builds and slow GDB performance with LLD linker, leading to quick fixes.
- Ctrl-C profiling is praised for its simplicity, requiring less setup than traditional profilers, and being applicable in environments where standard profilers might not work, such as on chips without standard OS support.
- Despite its advantages, Ctrl-C profiling has limitations, such as being a sampling profiler with low frequency, making it unsuitable for detecting very small changes across a program or tail latency issues.
- The article contrasts Ctrl-C profiling with more sophisticated profiling methods, highlighting scenarios where traditional profilers are necessary, such as identifying small performance regressions or analyzing multi-threaded systems.
- The author concludes by endorsing Ctrl-C profiling as a primitive yet often surprisingly effective tool, especially when compared to more complex and sometimes overkill profiling solutions.
Determining Current Arm Cortex-M Security State with GDB
4 months ago
- Armv8-M microcontrollers often include the Cortex-M Security Extension (CMSE), known as TrustZone.
- CMSE divides memory into Secure and Non-Secure regions, with the processor state changing based on the executing region.
- The stack pointer (SP) can match either the Main Stack Pointer (MSP) or Process Stack Pointer (PSP), depending on the operating mode and CONTROL.SPSEL field.
- With CMSE, there are Secure (*_S) and Non-Secure (*_NS) variants of registers like MSP and PSP.
- The current security state can be determined by checking which stack pointer variant the SP matches.
- GDB commands can be used to inspect SP, PSP_NS, MSP_NS, PSP_S, and MSP_S to quickly determine the security state.
- Example GDB output shows SP matching MSP_S in Secure state and PSP_NS in Non-Secure state.
- The CONTROL.SPSEL field can be checked to determine if the PSP is active in Thread mode.
Stop Forwarding Errors, Start Designing Them
4 months ago
- Current error handling practices often just forward errors without preserving meaningful context.
- Rust's `std::error::Error` trait is flawed as it assumes a linear chain of errors, excluding scenarios with multiple sources.
- Backtraces in async code are often useless and expensive, failing to show the logical path of errors.
- The `Provide/Request` API introduces unpredictability and complexity without clear benefits.
- `thiserror` categorizes errors by origin rather than actionable responses, making recovery unclear.
- `anyhow` is too convenient, often leading to missed opportunities to add context.
- Errors should be designed with two audiences in mind: machines (for automated recovery) and humans (for debugging).
- For machines, errors should be flat, actionable, and kind-based, enabling clear decision-making.
- For humans, errors should capture rich context automatically and ergonomically.
- The `exn` library demonstrates a better approach by enforcing context at module boundaries using the type system.
- Combining machine-readable errors with human-readable context provides the best of both worlds.
- Errors are communication tools and should be designed thoughtfully, not just forwarded.
Show HN: Live VNC for web agents – debugging native captcha on Cloud Run
4 months ago
- Live VNC view and takeover for ephemeral cloud browsers improves debugging for web agents.
- Web agents fail due to the web's distributed system nature, not just UI issues.
- DOM-native automation avoids screenshots/vision and CDP-based tools like Playwright/Puppeteer.
- Live VNC helps debug real-time issues like captcha solving and iframe targeting.
- Cloud Run constraints require a relay-based architecture for VNC to avoid sticky routing issues.
- The solution involves a runner, relay, and signed pairing tokens for secure, scalable VNC access.
- Minimal display stack (Xvfb + x11vnc + xsetroot) replaces heavier solutions like Fluxbox.
- Security model ensures no public VNC ports, short-lived tokens, and role separation.
- Live VNC observability improves debugging for captchas, iframes, shadow DOM, and parallel execution.
- Performance metrics show stable relay service but some errors in web agent runners.
- Bandwidth is a critical factor for scaling VNC sessions.
- Lessons learned include avoiding VNC in runners and unnecessary complexity like Fluxbox.
Investigating and fixing a nasty clone bug
4 months ago
- The author encountered a bug in the bors GitHub merge bot where HTTP PATCH requests sometimes had empty bodies during retries.
- The issue was traced to the octocrab crate's retry mechanism, which performed a shallow clone of the request body instead of a deep copy.
- The bug was fixed by modifying the OctoBody struct to include a buffered copy of the body for retries, ensuring the body is not empty when retried.
- The author also explored whether LLMs like Claude could have found the bug sooner, with mixed results.
- The fix was merged into octocrab and released in version 0.49.1.
Finding and Fixing Ghostty's Largest Memory Leak
4 months ago
- Ghostty was consuming absurd amounts of memory, with reports of up to 37 GB after 10 days of uptime.
- The memory leak was present since Ghostty 1.0 but became noticeable with the rise of CLI applications like Claude Code.
- Ghostty uses a PageList data structure for terminal memory management, with standard and non-standard pages allocated via mmap.
- A scrollback pruning optimization reused the oldest page as the newest without properly resizing non-standard pages, leading to memory leaks.
- The fix involves never reusing non-standard pages during scrollback pruning, ensuring proper memory deallocation.
- Virtual memory tags on macOS were added to help identify and debug memory allocations related to the PageList.
- Ghostty employs various leak detection methods, including debug builds, unit tests, CI runs with valgrind, and macOS Instruments.
- The community's detailed diagnostics and reproduction efforts were crucial in identifying and fixing the leak.
Launch a Debugging Terminal into GitHub Actions
4 months ago
- WebRTC is used to create a debugging terminal for GitHub Actions, allowing interactive access when builds fail.
- The solution leverages Peer-to-Peer (P2P) connections via WebRTC to avoid high data transfer costs.
- Security is ensured using OAuth for browser authentication and OIDC tokens for Actions VM identity verification.
- A signaling server facilitates the introduction of peers (Actions VM and user's browser) without handling terminal data.
- The terminal session is streamed over WebRTC's data channel, with Ghostty library used for browser-side terminal display.
- Additional security measures include One-Time Passwords (OTP) for peer validation, independent of the signaling server.
- The signaling server is hosted on Railway.com, utilizing its pay-for-use model to minimize costs, with features like automatic sleeping when idle.
Finding and Fixing a 50k Goroutine Leak That Nearly Killed Production
4 months ago
- A production API service experienced a severe goroutine leak, growing from 1,200 to 50,847 goroutines over six weeks, causing memory usage to spike to 47GB and response times to 32 seconds.
- The leak originated in a WebSocket notification system due to three critical bugs: not calling cancel() on context, not stopping tickers, and not closing channels.
- Uber's LeakProf tool was instrumental in identifying the leak, revealing that goroutines for dead WebSocket connections were not being cleaned up.
- The fix involved proper cleanup of resources: calling cancel() on context, stopping tickers, closing channels, and implementing a connection monitor to detect and clean up dead connections.
- Recovery was gradual, with emergency measures to stop the bleeding, cleanup scripts to remove existing leaks, and new monitoring to prevent future occurrences.
- New monitoring and alerting were added, including Prometheus metrics for goroutine count, WebSocket subscriptions, and active connections.
- Enhanced testing strategies were implemented, including leak detection tests, load testing for leaks, and benchmarks with goroutine tracking.
- Key lessons learned include the importance of exit strategies for goroutines, proper resource cleanup, monitoring goroutine counts, and testing for leaks.
- The cost of the bug included degraded performance, customer complaints, engineering hours, extra AWS costs, and reputation damage.
- Prevention measures now include pre-commit hooks to catch missing ticker.Stop() calls and the use of goleak in tests.
The fix for a segfault that never shipped
3 months ago
- Recall.ai processes millions of hours of meetings monthly, normalizing diverse audio formats into a consistent one.
- The company encountered a rare segfault in an ancient AAC encoder, leading to data loss in 1 out of 36 million meeting bot instances.
- The issue was traced to a fixed-point math bug in the VisualOn AAC encoder, which was patched over a decade ago but never distributed downstream.
- To diagnose the issue, Recall.ai developed 'Garbage Truck', a Rust binary to capture and upload core dumps from ephemeral EC2 instances to S3.
- Analysis revealed the crash occurred in `voAACEnc_pow2_xy` due to an out-of-bounds access caused by incorrect fixed-point arithmetic leading to `avgEn < minEn`.
- The team implemented a workaround by enforcing `avgEn >= minEn` and ultimately replaced the outdated encoder with a modern AAC encoder to prevent future crashes.

first prev6next

About|Login

#debugging

DuckGPT – agent for solving complex coding problems

Lot of software engineers don't understand basic networking

Don't Gobble Up Errors

A kernel bug froze my machine: Debugging an async-profiler deadlock

Debug Mode for LLMs in vLLora

Hacking LLDB for a great Zig debugging experience

Kernighan's Lever

Logging Sucks

Finding Jingle Town: Debugging an N64 Game Without Symbols

Parsing Advances

The GDB JIT Interface

Profiling with Ctrl-C (2024)

Determining Current Arm Cortex-M Security State with GDB

Stop Forwarding Errors, Start Designing Them

Show HN: Live VNC for web agents – debugging native captcha on Cloud Run

Investigating and fixing a nasty clone bug

Finding and Fixing Ghostty's Largest Memory Leak

Launch a Debugging Terminal into GitHub Actions

Finding and Fixing a 50k Goroutine Leak That Nearly Killed Production

The fix for a segfault that never shipped