The LLM Is Not a Junior Engineer
14 hours ago
- #productivity metrics
- #AI risks and ethics
- #LLMs in software development
- The essay discusses thoughtful incorporation of LLMs into software development, acknowledging their utility for hobbyists or low-stakes projects but warns against assuming the same benefits for large teams or sensitive applications.
- It criticizes anthropomorphizing LLMs as 'junior engineers,' emphasizing that AI lacks memory, learning, morality, and accountability unlike real junior engineers who grow and bear consequences.
- The author highlights the nondeterministic nature of LLMs, comparing their randomness to a pachinko machine, which makes debugging and reliability challenging compared to deterministic software.
- There's a call for clarifying risks associated with GenAI, advocating for structured risk assessment processes to evaluate potential failures and impacts, given the industry's still-evolving understanding.
- The concept of 'human in the loop' is explored, stressing the need for clear definitions and safeguards to prevent cognitive surrender and ensure meaningful human oversight, especially under pressure to adopt AI.
- Budget concerns are raised regarding LLM usage costs, noting unpredictable token-based pricing, potential rate hikes, and the difficulty in budgeting due to variable token consumption and hidden expenses.
- Productivity metrics for AI-assisted development are criticized as unreliable or manipulable, with examples from Amazon and Google showing how metrics like estimation points or developer hours can be misleading.
- The essay concludes by questioning the sustainability of heavy AI reliance, suggesting teams consider scenarios where AI tools might become unavailable or too costly, and advocating for ethical, open-source alternatives.