Hasty Briefsbeta

All tags

#database

200 stories total

Bilingual

Mumps (Programming Language)
a year ago
- MUMPS (Massachusetts General Hospital Utility Multi-Programming System) is an imperative, high-level programming language with an integrated key-value database.
- Originally developed in 1966 at Massachusetts General Hospital for managing patient records and laboratory systems.
- MUMPS is widely used in healthcare IT, including electronic health records (EHRs), with systems like Epic serving 78% of U.S. patients.
- Unique feature: Integrated database language for direct, high-speed disk storage access.
- History: Developed by Neil Pappalardo, Robert Greenes, and Curt Marble; first implemented on DEC PDP-7.
- Standardized in 1977 (ANSI X11.1-1977) and later adopted as ISO standard (ISO 11756:1999).
- Key implementations include GT.M (FIS) and Caché (InterSystems), with modern successors like IRIS.
- MUMPS uses hierarchical sparse arrays (globals) for persistent storage, supporting strings and numeric subscripts.
- Criticisms: Terse syntax, lack of data types, and unconventional operators (e.g., '> for ≤).
- Major users: U.S. Department of Veterans Affairs (VistA), Epic, MEDITECH, and Barclays Bank.
Filedb: Disk Based Key-Value Store Inspired by Bitcask
a year ago
- FileDB is a Zig-implementation of Bitcask, inspired by the Riak1 paper.
- It stores record metadata in a log-structured hashtable and keeps one disk file open for appending records.
- On restarts or when MAX_FILE_REACHED, the disk file is rotated, with old files kept open for reading only.
- Compaction runs periodically to combine disk files into one and update the metadata hashtable.
- Sync process syncs open disk files at configurable intervals or on every request if configured.
- Metadata allows O(1) record fetching by tracking exact file and position.
- In-memory metadata is constant size, regardless of record value size.
- High throughput is achieved via append-only mode.
- Provides functions like init, deinit, put, get, delete, list, sync, storeHashMap, and loadKeyDir.
- Includes a Redis-compatible client with demonstrated performance benchmarks.
- Benchmarks show high throughput for set and get operations.
- References to Zig programming resources are provided.
Show HN: gRPSQLite – A SQLite VFS for remote databases via gRPC
a year ago
- gRPSQLite enables remote SQLite databases via gRPC, allowing any storage system to act as a backend.
- Features include real SQLite compatibility, support for any storage system, instant availability, atomic transactions, and read replicas.
- Quick start involves running a container, starting a memory server, and using SQLite normally with data persistence.
- The system converts SQLite file operations into gRPC calls, requiring a simple gRPC server implementation for storage handling.
- Early-stage software with limited testing, not recommended for production use.
- Use cases include multi-tenant SaaS, AI workflows, immutable databases, and read fanout scenarios.
- Examples provided in Rust demonstrate memory server implementations with various features.
- Two methods of using the SQLite VFS: dynamically loading or statically compiling into a SQLite binary.
- Performance benefits include atomic batch commits and read replicas for scaling reads.
- Contributions require discussion before pull requests to ensure alignment with project goals.
Show HN: SnapQL – Desktop app to query Postgres with AI
10 months ago
- Generate schema-aware queries in seconds with AI
- Supports any PostgreSQL database
- Fully local desktop app
- Use your own OpenAI key
- Steps to build a local copy: clone the repo, run npm install, install XCode for MacOS, run npm build:mac or npm build:win, install the binary from ./dist
Interesting Bits of Postgres Grammar
10 months ago
- Postgres allows custom operators, which can be prefix or infix but not postfix.
- Custom operators enable unique syntax like lambda expressions in array_filter.
- Compound SELECT statements apply ORDER BY to the entire result set.
- CREATE FUNCTION can use column types via %TYPE syntax.
- Adjacent string literals merge unless separated by a comment.
- Quoted identifiers support escaping double quotes and Unicode via U&.
- Operators can be called as functions, including schema specification.
ZeQLplus: Terminal SQLite Database Browser
10 months ago
- Open any SQLite database file quickly and efficiently.
- Operates in Terminal/CMD with a tiny, dependency-free executable.
- Features include listing tables, paginated row views, and custom SQL queries.
- Cross-platform support for macOS, Linux, and Windows.
- Pre-built binaries available for easy download and use without installation.
- Instructions for running from command line and building from source provided.
- Requires Vlang v0.4.10 or above for source builds.
- Licensed under MIT.
Show HN: ToplingDB - A Persistent Key-Value Store for External Storage
10 months ago
- ToplingDB is developed and maintained by Topling Inc. and built with RocksDB.
- ToplingDB's submodule rockside serves as the entry point.
- Key features include SidePlugin for DB configs via JSON/YAML, Embedded Http Server for DB info and online changes, and performance improvements over RocksDB.
- ToplingDB offers faster transaction lock management, MultiGet with concurrent IO, and de-virtualization for performance.
- Features like zero copy for searches, memtable as log index, and builtin SidePlugins enhance functionality.
- Includes Prometheus metrics support and bugfixes for RocksDB.
- Supports MyTopling (MySQL on ToplingDB) and Todis (Redis on ToplingDB).
- SidePlugin allows plugins to be compiled as dynamic libs and loaded at runtime without code changes.
- ToplingDB's structure includes public and private repositories for community and enterprise versions.
- Compilation requires C++17, with gcc 8.3 or newer recommended.
- Installation and setup instructions provided for Linux systems.
- Performance benchmarks show ToplingDB is faster than RocksDB, with options for community and enterprise configurations.
- Disabled RocksDB features by default for simplicity, with macros to enable them.
- License terms updated to remove restrictions on bytedance usage.
- RocksDB, the foundation of ToplingDB, is developed by Facebook and built on LevelDB, offering a fast key-value server for flash storage.
Introducing the first alpha of Turso: The next evolution of SQLite
10 months ago
- Announcement of 'Project Limbo', a full rewrite of SQLite, now named 'Turso'.
- First alpha release of Turso, aiming to match SQLite's reliability with advanced testing techniques.
- Offering $1,000 for finding data corruption bugs missed by their testing.
- Turso addresses modern application needs that SQLite can't handle, with over 115 contributors.
- Uses Deterministic Simulation Testing (DST) and partnership with Antithesis for robustness.
- Alpha features include basic SQLite operations and an asynchronous interface; some features still in development.
- Turso is already being used in projects like Spice.ai, showing performance improvements.
- Focus on stability and reliability achieved through partnerships with Antithesis and Blacksmith.
- Open invitation for community involvement and design partners.
What 'Project Hail Mary' teaches us about the PlanetScale vs. Neon debate
10 months ago
- The article discusses the ongoing debate between Planetscale and Neon, highlighting their different strengths.
- It references a quote from 'Project Hail Mary' to illustrate the trade-off between efficiency and scalability in system design.
- The author emphasizes that both Planetscale and Neon excel in their respective niches and use cases.
- A reminder is given that there is no universal optimal solution, and negativity on social media should not overshadow this understanding.
aileaks.dev - Community driven database of AI related security incidents
10 months ago
- A community-driven database of AI-related security incidents.
- Allows users to submit incidents for review.
- Displays loading status while fetching incidents.
Show HN: DataRamen, a Fast SQL Explorer with Automatic Joins and Data Navigation
10 months ago
- DataRamen is a web GUI for MySQL and PostgreSQL designed for developers.
- First-time installation requires Node.js v22 or above.
- Install DataRamen CLI tool via npm with: npm i -g @dataramen/cli.
- Start the local server with the command: dataramen start.
- Access the app at app.dataramen.xyz in your browser.
- Features include connecting to PostgreSQL & MySQL with credentials.
- Explore schema: view tables, columns, types, and relationships.
- Query data visually without writing SQL.
- Follow relationships by clicking to expand related records.
- Insert & update data visually without syntax errors.
- Save queries for later use.
- Option to write and run raw SQL anytime.
Piccolo: Powerful async ORM, query builder, and admin GUI
10 months ago
- A powerful async ORM, query builder, and admin GUI.
- Supports Postgres, SQLite, and CockroachDB.
- Offers both async and sync support.
- Includes a built-in playground for easy learning.
- Features tab completion support for iPython and VSCode.
- Comes with batteries included: User model, authentication, migrations, admin, etc.
- Fully type annotated, modern Python.
- Designed for web developers and data scientists.
- Provides examples for Select, Join, Delete, and Update operations.
- Quick start guide available in the docs.
QuestDB (YC S20) Is Hiring a Technical Content Lead
9 months ago
- QuestDB is a specialized database for real-time time series data analysis, focusing on performance and simplicity.
- The company is remote-first, globally distributed, and backed by leading venture capital firms and Y Combinator.
- QuestDB is used by leading companies in industries like Financial Services, Energy, and Space Exploration.
- The Technical Content Lead will own QuestDB’s technical content strategy, including documentation, articles, and newsletters.
- Responsibilities include creating engaging technical content, optimizing SEO, and engaging with the developer community.
- Requirements include 5+ years of experience in technical content creation and familiarity with git and markdown.
- Nice-to-have skills include a background in software engineering and experience with time series databases.
- Benefits include flexible working hours, 100% remote work, and generous equity options.
- QuestDB values transparency, autonomy, and strategic input from employees.
Manticore Search: Fast, efficient, drop-in replacement for Elasticsearch
9 months ago
- Manticore Search is an open-source, fast database designed for search, serving as a great alternative to Elasticsearch.
- It is significantly faster than alternatives like MySQL and Elasticsearch, with reproducible benchmarks showing up to 182x faster performance in some cases.
- Features include modern multithreading architecture, efficient query parallelization, powerful full-text search, and support for both row-wise and columnar storage.
- Manticore is SQL-first, compatible with MySQL protocol, and offers clients in multiple programming languages including PHP, Python, Java, and Rust.
- Built-in tools for high availability, data backup, and recovery, along with support for synchronous replication and load balancing.
- Supports a wide range of search capabilities like fuzzy search, faceted search, geo-spatial search, and vector search.
- Easy to integrate with Docker, and installation is supported on various platforms including Linux and macOS.
- Manticore Search is used by companies like Craigslist, Socialgist, and PubChem for efficient searching and stream filtering.
- Community and enterprise support options are available, including audit, consulting, development, and training services.
nullable but not null
9 months ago
- Adding a new field to a model often starts as nullable to avoid table locks during migration.
- Application logic is updated to populate the field, followed by a backfill job for existing records.
- The final step of making the field non-nullable is often forgotten, leading to schema-data mismatch.
- Leaving fields nullable when they shouldn't be creates confusion, missed constraints, and unnecessary complexity.
- A script is provided to identify nullable fields that could be made non-nullable by checking actual null percentages.
- The script scans models for nullable fields and calculates the percentage of rows with null values.
- Fields with 0% nulls should be updated to non-nullable to improve schema accuracy and simplify code.
- Example output shows fields with their null percentages, helping identify candidates for schema cleanup.
- Fixing these issues ensures the data model is honest, improves validation, and keeps the application healthier.
How Turso made connections to SQLite databases 575x faster
9 months ago
- Turso is a full rewrite of SQLite in Rust, designed for modern applications.
- It addresses SQLite's architectural limitations, like improving write concurrency and eliminating the 'database is locked' issue.
- Turso significantly speeds up connection opening, achieving a 500x improvement for databases with many tables (e.g., 10,000 tables).
- SQLite's connection opening slows down with more tables due to schema parsing.
- Turso optimizes this by sharing schema state among connections, reducing opening time to 40 microseconds.
- Turso embraces parallelism and multithreading, unlike SQLite's single-threaded origins.
- Turso is open for contributions, with over 130 contributors, unlike SQLite's closed community.
Long Term Support
9 months ago
- SQLite developers aim to support SQLite until 2050, ensuring long-term viability.
- SQLite is highly portable, running on any platform with an 8-bit byte, 32/64-bit integers, and a C compiler.
- SQLite database files are identical across platforms, ensuring compatibility without conversion.
- Aviation-grade testing ensures robustness, with every machine-code branch tested on multiple platforms.
- Extensive, detailed documentation and heavily commented source code aid new developers.
- Disaster planning includes cryptographic protection and global replication of source-code history.
- SQLite focuses on timeless code, avoiding trends to ensure future readability and maintainability.
- Backwards compatibility of the C-language API and on-disk format is promised for future versions.
- SQLite is recommended by the US Library of Congress for digital content preservation.
New Compilation Methods for Complex User-Defined Functions
9 months ago
- User-defined functions (UDFs) often have poor performance, limiting their use in performance-critical applications.
- This thesis leverages programming language techniques to improve UDF performance by compiling them to pure SQL queries.
- For iterative UDFs, a novel trampolined-style compilation approach eliminates the impedance mismatch between PL/SQL and SQL, improving performance by up to 3 times.
- The approach allows the entire computation to be expressed in SQL, enabling optimization and execution on any system supporting contemporary SQL.
- For recursive UDFs, compilation to pure SQL queries eliminates overhead, improving performance by up to 180 times.
- The thesis demonstrates that SQL is expressive enough for complex computations and achieves excellent performance.
- Detailed rules are provided for implementing a UDF compiler in any database system supporting LATERAL joins and recursive CTEs.
HorizonDB, a geocoding engine in Rust that replaces Elasticsearch
9 months ago
- Radar processes over 1 billion API calls per day, serving hundreds of millions of devices globally.
- HorizonDB, a geospatial database written in Rust, was developed to consolidate multiple location services into a single, high-performance binary.
- Prior to HorizonDB, geocoding was split across Elasticsearch, microservices, and MongoDB, leading to operational and scaling challenges.
- HorizonDB utilizes technologies like RocksDB, S2, Tantivy, FSTs, LightGBM, and FastText to enhance performance and efficiency.
- Rust was chosen for its safety, performance, and concurrency features, making it ideal for systems programming.
- RocksDB provides microsecond response times, serving as the primary record store with high efficiency.
- S2 enables efficient spatial indexing, converting O(n) point-in-polygon lookups into constant time operations.
- FSTs offer efficient string compression and prefix queries, caching millions of 'happy paths' with minimal memory usage.
- Tantivy, an in-process inverted index, was preferred over external services like Elasticsearch for its simplicity and performance.
- FastText improves search precision by representing words as numeric vectors, handling typos and out-of-vocabulary words effectively.
- LightGBM models classify query intent, structuring queries to enhance search performance and precision.
- Apache Spark processes hundreds of millions of data points quickly, with near-linear scalability.
- HorizonDB has significantly improved cost, performance, and scalability for Radar's geolocation services.
- Radar is hiring engineering talent and invites interested candidates to check their jobs page.
Build durable workflows with Postgres
9 months ago
- The critical architectural decision for building a durable workflows library was choosing the right data store for workflow metadata.
- Postgres was chosen for its technical features, including concurrency control, relational data model, and transaction support.
- Postgres concurrency control, especially locking clauses, enables scalable distributed queues by reducing contention among workers.
- The relational data model and secondary indexes in Postgres allow for performant observability tooling over workflow metadata.
- Postgres transactions enable exactly-once execution guarantees for steps performing database operations, improving reliability.
- Building scalable queues in Postgres involves using locking clauses to prevent contention among workers, allowing high throughput.
- Observability in durable workflows is enhanced by Postgres's ability to run complex queries over workflow metadata efficiently.
- Secondary indexes in Postgres are used selectively to balance query performance and runtime overhead for observability.
- Exactly-once semantics are achieved by executing steps and their checkpoints within the same Postgres transaction.
- DBOS offers resources for those interested in durable workflows, including a quickstart guide and GitHub repository.

first prev2next

About|Login

#database

Mumps (Programming Language)

Filedb: Disk Based Key-Value Store Inspired by Bitcask

Show HN: gRPSQLite – A SQLite VFS for remote databases via gRPC

Show HN: SnapQL – Desktop app to query Postgres with AI

Interesting Bits of Postgres Grammar

ZeQLplus: Terminal SQLite Database Browser

Show HN: ToplingDB - A Persistent Key-Value Store for External Storage

Introducing the first alpha of Turso: The next evolution of SQLite

What 'Project Hail Mary' teaches us about the PlanetScale vs. Neon debate

aileaks.dev - Community driven database of AI related security incidents

Show HN: DataRamen, a Fast SQL Explorer with Automatic Joins and Data Navigation

Piccolo: Powerful async ORM, query builder, and admin GUI

QuestDB (YC S20) Is Hiring a Technical Content Lead

Manticore Search: Fast, efficient, drop-in replacement for Elasticsearch

nullable but not null

How Turso made connections to SQLite databases 575x faster

Long Term Support

New Compilation Methods for Complex User-Defined Functions

HorizonDB, a geocoding engine in Rust that replaces Elasticsearch

Build durable workflows with Postgres