Hasty Briefsbeta

LF AI and Data Hosts Vortex Project for Data Access for AI and Analytics

18 days ago
https://www.linuxfoundation.org/press/lf-ai-data-foundation-hosts-vortex-project-to-power-high-performance-data-access-for-ai-and-analytics
Copy Link
  • #AI
  • #Data Storage
  • #Open Source
  • The LF AI & Data Foundation announced the launch of the Vortex Project, an open, extensible columnar storage format designed for high-performance data systems.
  • Vortex is contributed by SpiralDB and supported by industry leaders like Microsoft, Snowflake, Palantir, and NVIDIA.
  • Vortex offers state-of-the-art performance, including 100x faster random access reads and 5x faster writes compared to Apache Parquet.
  • The format is optimized for modern workloads, including GPU-based training and cloud object storage like S3 and GCS.
  • Vortex features an extensible architecture to incorporate new compression techniques and integrates with tools like Apache Arrow and Apache Spark.
  • The project aims to eliminate CPU bottlenecks by enabling direct GPU decompression, improving AI training data access.
  • Industry leaders highlight Vortex's potential to address storage bottlenecks and advance open-source data processing for AI.
AboutLogin