13 days ago
- Small local models for agentic coding were evaluated using a viability funnel covering RAM, speed, tool calling, functional correctness, conversation handling, task complexity, and code quality.
- The evaluation process went through manual evals, automated evals, and day-to-day use, with tasks including sorting/cumulating a bar chart and creating a bar chart from access log data in JavaScript/TypeScript, plus Bash and Python scripts.
- Results varied: some models like Qwen Coder Next 80B MoE produced functional code but crashed in extended conversations, while others like Gemma 4 26B succeeded manually but failed in automated tests, with performance differing between machines.
- Day-to-day use with Qwen3.6 35B MoE showed success for small, well-defined tasks (e.g., Bash/Python scripts, website updates) but struggled with complex logic, requiring careful task selection and more code review.
- Key factors affecting viability include task complexity, number of files to edit, specificity of instructions, and tech stack, with local models offering a 'detox' by encouraging slower, more thoughtful coding but lagging behind larger models in autonomy.