Hasty Briefsbeta

All tags

#debugging

163 stories total

Bilingual

Debugging Lotus 1-2-3 by fax
a year ago
- In 1990, the author helped debug a Lotus 1-2-3 setup via fax due to expensive transatlantic calls.
- The Zeroth Law of Debugging was highlighted as crucial in this scenario.
- The debug involved checking the right margin setting in Lotus 1-2-3 using the command '/ppomr'.
- Fax was a cost-effective alternative to voice calls for technical support across continents.
- The author found the actual fax later, reminiscing about the debugging process.
Bringing Record and Replay debugging everywhere on Linux
a year ago
- Modified rr debugger to run without CPU Hardware Performance counters, enabling use in cloud VMs and containers.
- Introduces Software Counters mode rr, using lightweight dynamic and static instrumentation.
- Record and Replay (rr) allows programs to run identically across replays by logging non-deterministic aspects.
- Useful for debugging as it captures exact program state, including user inputs, network calls, and thread interleaving.
- rr is the only open-source system meeting criteria like broad usability, performance, and compatibility with gdb/lldb.
- Software Counters mode rr addresses the limitation of requiring CPU HW performance counters, expanding rr's accessibility.
- Project inspired by the challenges of porting rr to Rust and the need for rr in restricted environments.
- Details on building and running Software Counters mode rr available on GitHub.
War story: the hardest bug I ever debugged
a year ago
- A sudden surge of fatal errors in Google Docs without any apparent cause or user complaints.
- The bug was Chrome-specific, starting at a particular release, making it hard to diagnose.
- Reproduction involved creating a large document and repeatedly bolding/unbolding text, revealing the bug was nondeterministic.
- Investigation pointed to a layout engine issue where subpixel rendering errors accumulated over time.
- Debugging was complicated by the bug's nondeterministic nature and slow reproduction steps.
- Collaboration with a coworker and a tech lead was crucial in identifying the root cause.
- The issue was traced back to a V8 optimization bug where Math.abs() incorrectly returned negative values.
- A temporary fix was implemented with a browser version check and a manual workaround for Math.abs().
- The V8 team had already fixed the issue, highlighting the challenge of debugging external dependencies.
- The experience underscored the unpredictable and sometimes frustrating nature of debugging complex software issues.
Show HN: Await-Tree – Visualize Async Rust Task Execution in Real-Time
a year ago
- Async Rust Futures can be composed or nested for various control flows.
- The execution of async tasks can be visualized as a logical tree that changes over time.
- `await-tree` allows runtime dumping of the execution tree with annotated spans.
- Supports dynamic span customization with runtime `String` for better debugging.
- Handles complex async control flows and arbitrary `Future` topologies.
- Uses an arena-based data structure with zero extra `unsafe` code.
- Optional features include `serde` for serialization and `tokio` for runtime integration.
- Compared to `async-backtrace`, `await-tree` offers more flexibility and safety.
- Used in production by RisingWave, a distributed streaming database.
- Licensed under Apache License 2.0.
Diagnosing bugs preventing sleep on Windows
a year ago
- Diagnosing programs preventing Windows auto-lock and sleep is straightforward using breakpoints and Windows APIs.
- The issue was traced to a hidden dialog in a CEF-based application that kept a power request active, preventing the lock screen and sleep mode.
- Tools like `powercfg`, `pwrtest`, and ETW with the Microsoft-Windows-Kernel-Power provider can help monitor and diagnose power requests.
- The bug was fixed after identifying the hidden dialog as the root cause.
Pdeathsig is almost never what you want
a year ago
- The author was tasked with optimizing Output Media start latency at Recall.ai.
- Output Media renders customer-supplied web pages into audio and video for bots, using Chromium in a sandboxed environment with Bubblewrap.
- Initial latency was 12 seconds due to Chromium's resource-intensive startup when Output Media was activated.
- The plan was to pre-load Chromium at bot startup to reduce latency.
- Testing revealed Chromium terminated unexpectedly, despite the parent process still running.
- Debugging identified the issue with Bubblewrap's --die-with-parent flag, which uses Linux's PR_SET_PDEATHSIG.
- PR_SET_PDEATHSIG tracks the parent thread, not the process, causing issues with Tokio's thread management.
- Tokio's dynamic thread parking and reaping led to premature termination of Chromium.
- Removing --die-with-parent resolved the issue, reducing latency to 2-3 seconds.
- The solution improved customer experience by significantly cutting wait times.
AI isn't ready to replace human coders for debugging, researchers say
a year ago
- AI is widely used in software development, from coding assistance to application building.
- AI agents are not yet capable of replacing most programmers due to poor debugging skills.
- Microsoft Research developed debug-gym to improve AI's debugging capabilities.
- Debug-gym provides AI models with tools like setting breakpoints and navigating code.
- AI models perform better with debug-gym but still lag behind human developers.
- Interactive debugging tools can help AI agents tackle real-world software tasks more effectively.
- Human oversight is needed to approve AI-proposed fixes, ensuring they are contextually grounded.
Peering into the Linux Kernel with Trace
a year ago
- The author encountered intermittent test failures due to unexpected changes in file access times.
- Using strace didn't reveal the cause, leading to suspicion of another process or a kernel bug.
- The BCC tools, specifically trace, were used to monitor kernel function calls in real-time.
- trace was configured to monitor the touch_atime function, revealing Sublime Text's background thread was accessing files.
- The issue was traced back to Sublime Text's git integration feature updating access times.
- The author highlights the power of trace for debugging and its addition to their toolkit.
- The process involved parsing probe specifications, converting them to eBPF bytecode, and using kprobes for monitoring.
- The article underscores the importance of transparency in system design for easier debugging.
The case of the UI thread that hung in a kernel call
a year ago
- A customer reported a UI thread hang that couldn't be diagnosed due to the stack being paged out.
- The thread was suspended for over five hours, but no debugger was attached to explain the suspension.
- A watchdog thread within the same process was found to suspend the UI thread to capture stack traces, leading to a deadlock.
- The deadlock occurred because the UI thread held a lock needed by the watchdog thread to capture the stack trace.
- Suspending threads within the same process risks deadlock if the suspended thread holds resources needed by others.
- The solution is to use an external process for watchdog functionality to avoid deadlocks.
- The kernel delays thread suspension to avoid interrupting critical operations, but this doesn't prevent user-mode deadlocks.
- Microsoft's design choices around thread suspension and loader locks were criticized, but the root issue was the in-process watchdog.
Using Python's .__dict__ to Work with Attributes – Real Python
a year ago
- Python's .__dict__ is a special attribute in classes and instances that maps attribute names to their values.
- .__dict__ allows dynamic inspection, modification, addition, or deletion of attributes, making it useful for metaprogramming and debugging.
- Both vars() and .__dict__ can inspect an object's attributes, with .__dict__ providing direct access to the object's namespace.
- Common use cases for .__dict__ include dynamic attribute management, introspection, serialization, and debugging.
- .__dict__ is fundamental to Python's data model and is used internally by the interpreter for dynamic attribute access.
- Classes and instances have separate .__dict__ attributes, with instance .__dict__ containing only instance-specific attributes.
- Functions also have a .__dict__ attribute, which can be used to attach metadata or for caching purposes.
- Built-in functions and data types may not have a .__dict__ attribute, as they are implemented in C for efficiency.
- .__dict__ can be used for memoization, serializing objects to JSON, and customizing attribute access with special methods like __getattribute__ and __setattr__.
- Descriptors can use .__dict__ to avoid recursion issues when implementing __get__ and __set__ methods.
Debugging Nginx: why my SSL certificates don't apply after reload?
a year ago
- An automation for creating new nginx hosts stopped working, redirecting to another host instead of the intended site.
- Certificate renewals faced similar issues, with sites only working correctly after a full nginx restart.
- Debugging revealed Nginx was hitting its maximum file descriptor limit, preventing proper reloading.
- Solution involves increasing worker_rlimit_nofile in nginx.conf and setting a systemd override for the service limit.
- Difference between 'reload' and 'restart' in Nginx explained: reload gracefully updates configuration, restart closes all connections first.
- Diagnosis steps include checking process limits and current file descriptor usage.
- Fixing requires updating Nginx configuration and creating a systemd override, followed by applying changes and verifying the fix.
Being a top programmer: Don't get stuck
a year ago
- Programmers face daily problems, some trivial and some complex, which can consume hours if not addressed systematically.
- A systematic approach to problem-solving, akin to methods used by detectives and scientists, can significantly enhance productivity.
- Key strategies include avoiding immediate reliance on ChatGPT, utilizing error messages effectively, and replicating issues locally to narrow down causes.
- Reading documentation, checking for typos, and ensuring correct permissions and environment variables are crucial steps in debugging.
- Advanced techniques involve atomic tests, ruling out cached results, and understanding the scientific method to systematically eliminate potential causes.
- When stuck, seeking help from colleagues or more experienced developers can provide new insights, but it's important to avoid the XY Problem when asking for help.
- Ultimately, if all else fails, revisiting the problem with fresh knowledge or designing an alternative solution may be necessary.
Show HN: A VS Code extension to visualise Rust logs in the context of your code
a year ago
- VS Code extension for integrating telemetry data (traces, logs, metrics) into your code.
- Install the extension and import logs via copy/paste, file, web, or Axiom.co.
- Debug code with logs by selecting a repository and setting the Claude API Key.
- View logs in context by clicking on log lines and navigating the call stack.
- Clone the demo repository 'hyperdrive-eng/playground' to test functionality.
- Load demo logs from the repository and explore logs and their parent contexts.
Kmemdump Proposed for the Linux Kernel to Help with Memory Dumping and Debugging
a year ago
- Eugen Hristev of Linaro proposed kmemdump for the Linux kernel to assist in debugging by dumping specific memory regions.
- Kmemdump allows kernel drivers to register memory chunks for easy dumping during system problems or debugging.
- The infrastructure avoids dumping entire RAM by focusing on flagged memory regions important for debugging.
- Kmemdump can save monitored regions even if the kernel crashes or freezes, provided appropriate hardware support.
- Memory regions can be assembled into a coredump ELF file for analysis with tools like GDB.
- Initial focus is on Qualcomm hardware, with Qualcomm Minidump as a backend for kmemdump.
- Kmemdump works independently of pstore, kdump, or kexec, making it viable for devices where these mechanisms fail.
- It creates a core image similar to /proc/vmcore, containing only registered regions for analysis.
- Kmemdump registers necessary kernel information at init time, requiring no additional user intervention.
Searching for the cause of hung tasks in the Linux kernel
a year ago
- Linux kernel produces hung task warnings when processes are stuck in an uninterruptible state (TASK_UNINTERRUPTIBLE) for too long.
- The hung task warning is triggered by the khungtaskd daemon, which checks processes in the D state (TASK_UNINTERRUPTIBLE).
- Processes in the D state cannot be interrupted by signals and may indicate system resource issues or overwhelmed subsystems.
- The TASK_KILLABLE state was introduced to allow termination with fatal signals while still protecting process memory.
- Example #1: XFS file system slowdown caused hung task warnings due to missing no_read_workqueue/no_write_workqueue flags.
- Example #2: A process generating a core dump can trigger hung task warnings as the kernel preserves memory in the D state.
- Example #3: The rtnl_mutex lock in the kernel networking subsystem caused multiple processes to hang, identified via BPF tracing.
- Debugging hung tasks involves analyzing stack traces, checking system metrics, and using tools like bpftrace and drgn.
- Hung task warnings are valuable for identifying system issues but may point to victims rather than the root cause.
Writing a Linux Debugger (2017)
a year ago
- Debuggers are essential tools for developers, yet there's limited information on how they work or how to write one.
- This tutorial series aims to explain debugger mechanics and guide through writing a Linux debugger, covering features like execution control, breakpoints, memory access, and stepping.
- The debugger will support C and C++ programs that output standard DWARF debug information, focusing on functionality over robust error handling.
- Setup involves dependencies like Linenoise for command-line input and libelfin for parsing debug information, with instructions for installation and building.
- The debugger launches the debugee program using fork/exec, utilizing ptrace for process control and observation.
- A debugger loop listens for user input, supporting commands similar to gdb and lldb, with initial support for continuing execution.
- Future parts will expand on setting breakpoints, handling registers and memory, and more advanced debugging features.
Differential Coverage for Debugging
a year ago
- Differential coverage compares code coverage of passing and failing tests to identify problematic code sections.
- The technique involves generating coverage profiles for both passing and failing tests, then diffing them to find unique code paths.
- In the example, a bug in `natmul.go` was identified by highlighting code unique to the failing test.
- Differential coverage is efficient, pinpointing relevant code sections out of thousands of lines.
- The method isn't foolproof but is effective when bugs are failure-triggered.
- Basic coverage profiles for single failing tests can also guide debugging by showing executed code paths.
- The technique can also be applied to passing tests to identify specific functionalities, like SOCKS5 proxy implementation in `net/http`.
Show HN: Web-eval-agent – Let the coding agent debug itself
a year ago
- operative.sh's MCP Server launches a browser-use powered agent for autonomous web app debugging in your code editor.
- Features include BrowserUse navigation (2x faster), network traffic capture, console error collection, and autonomous debugging.
- Pre-requisites include Homebrew, npm, and jq installations.
- Installation involves downloading and running a script from operative.sh after obtaining a free API key.
- Steps to set up include installing uv, the web-eval-agent, and playwright for testing.
- Example task: Testing API-key deletion flow with detailed steps and successful UX feedback.
- Console logs and network requests are captured and displayed for debugging purposes.
- Chronological timeline of events during the debugging process is provided.
- Built with love by operative.sh, offering a dashboard for live logs.
Recognizing Patterns in Memory (2022)
a year ago
- Debugging skills require experience and pattern recognition.
- Memory patterns can hint at corruption issues.
- Aligned 32/64-bit data is common in file formats and memory structures.
- Pointers in memory often follow specific address patterns.
- UTF-16 text encoding is recognizable by alternating zeros and ASCII values.
- x64 code often contains sequences of 'CC' bytes and 'C3' (ret instruction).
- High entropy data usually indicates compressed or encrypted content.
Bambu Lab added anti-debugger code in their BBL network plugin
a year ago
- BBL network plugin includes debugger detection since version 1.10.2, terminating the app if a debugger is attached.
- This prevents development using Visual Studio or other dev tools with a debugger, hindering contributions to the open-source project.
- Attempts to hide information (certificate, network protocol) have already been bypassed, making the debugger restriction unnecessary.
- The restriction forces developers to crack the network plugin just to contribute, which is counterproductive.
- The author expresses frustration over the unnecessary obstacles to development, emphasizing no interest in the hidden aspects of the DLL.

firstprev0next

About|Login

#debugging

Debugging Lotus 1-2-3 by fax

Bringing Record and Replay debugging everywhere on Linux

War story: the hardest bug I ever debugged

Show HN: Await-Tree – Visualize Async Rust Task Execution in Real-Time

Diagnosing bugs preventing sleep on Windows

Pdeathsig is almost never what you want

AI isn't ready to replace human coders for debugging, researchers say

Peering into the Linux Kernel with Trace

The case of the UI thread that hung in a kernel call

Using Python's .__dict__ to Work with Attributes – Real Python

Debugging Nginx: why my SSL certificates don't apply after reload?

Being a top programmer: Don't get stuck

Show HN: A VS Code extension to visualise Rust logs in the context of your code

Kmemdump Proposed for the Linux Kernel to Help with Memory Dumping and Debugging

Searching for the cause of hung tasks in the Linux kernel

Writing a Linux Debugger (2017)

Differential Coverage for Debugging

Show HN: Web-eval-agent – Let the coding agent debug itself

Recognizing Patterns in Memory (2022)

Bambu Lab added anti-debugger code in their BBL network plugin

Using Python's .dict to Work with Attributes – Real Python