3 months ago
- Self-hosted LLMs offer privacy, cost savings, and no reliance on external companies, with local data and no subscription fees.
- The author built an AI Slop Machine using a Radeon AI Pro R9700 GPU in an eGPU dock connected to an old Thinkpad via Thunderbolt 3 for running Qwen 3.6 27B.
- Performance with llama.cpp includes prefilling at 507-814 TPS and decoding at 28-33 TPS, supporting large context windows up to 131k tokens.
- The system is used as a server accessible remotely via SSH, keeping the client laptop cool and portable, though it requires a stable internet connection.
- An AI agent collaborates on tuning and configuration, managing tasks in a shared workspace with ongoing improvements and experimentation.
- OpenRouter data privacy issues highlight that some providers may retain and sell user data, prompting the author to use more expensive 'Zero Data Retention' options.