Everything around LLMs is still magical and wishful thinking

10 months ago

Criticism of AI often comes from developers who haven't deeply engaged with current tools like MCP, beyond making API calls to LLMs.
There's a significant divide between users who find LLMs helpful some/most of the time and those who dismiss them as entirely ineffective.
The gap in perception stems from a lack of quantified, detailed descriptions of how LLMs are used across different contexts.
Key unknowns include the nature of projects, codebases, user expertise, and the extent of additional work required post-LLM use.
LLMs are non-deterministic; their effectiveness can vary even for the same problem over time.
Comparisons between different users' experiences with LLMs are challenging due to varying contexts and non-deterministic outcomes.
The industry's hype around LLMs often overlooks critical evaluation, leading to polarized views without substantive evidence.
Anecdotal success stories with LLMs lack crucial details, such as codebase size, bug specifics, and the level of manual oversight required.
Despite skepticism, the author uses LLM tools extensively, acknowledging their partial effectiveness but rejecting the notion of them being magical or strictly engineering solutions.
The discourse around LLMs is overly simplistic, framing them as either magical or engineered, ignoring their statistical, non-deterministic nature.

Hasty Briefsbeta