Show HN: Building a web search engine from scratch with 3B neural embeddings
12 days ago
- #machine-learning
- #search-engine
- #neural-embeddings
- The author built a web search engine from scratch in two months, focusing on improving search quality and relevance using neural embeddings.
- Key motivations included addressing the decline in search engine quality due to SEO spam and leveraging transformer-based text embedding models for better natural language comprehension.
- The search engine was designed to understand intent rather than keywords, enabling it to handle complex, nuanced queries effectively.
- Technical highlights include generating 3 billion SBERT embeddings with 200 GPUs, crawling 50K pages per second, and achieving end-to-end query latency of 500 ms.
- The system architecture involved components like normalization, chunking, a crawler, pipeline, storage, service mesh, GPU buildout, sharded HNSW, and knowledge graph integration.
- The search engine demonstrated superior performance in understanding and answering complex queries compared to traditional keyword-based search engines.
- Cost optimization was a significant focus, with the author leveraging lesser-known, cost-efficient services to manage expenses.
- The project concluded with insights into the potential of neural embeddings for search and plans for future improvements, including better quality filtering and leveraging LLMs for reranking.