2 months ago
- Migrated from Docker Compose to bare systemd, reducing operational headaches and improving stability.
- Introduced an unranked query endpoint that performs simple term intersections, reducing query execution time to ~5ms and freeing up threads.
- Added a separate crawler partition for wide domains (e.g., tumblr.com, substack.com) to run parallel time-based crawls, cutting main crawl time from two weeks to five days.
- Overall, these changes halved crawl times, lowered computational costs, and made deployments faster and more reliable.