data visualization

Bilingual

The mean means nothing: data visualization to debug a latency problem
2 months ago
- Performance improvements in benchmarks may not reflect live production due to noise in metrics.
- Multiple statistics (mean, median, p95, p99) can tell contradictory stories about the same data change.
- Cumulative distribution functions (CDFs) visualize the entire distribution, revealing shifts and crossing points.
- Rollout data over time shows emerging regression patterns that aggregates hide.
- Bimodality in latency often has a root cause, such as cache hits vs. misses based on response size.
- Jointplots can isolate populations by variables like response size to identify actionable causes.
- Multiple visualizations are needed to fully understand data instead of relying on a single number.
From gnuplot to Matplotlib & Pandas - Bert Hubert's writings
2 months ago
- The author transitions from gnuplot to Matplotlib and Pandas for data plotting and analysis, citing gnuplot's limitations outside its core strengths.
- Pandas and Matplotlib offer integrated data labeling and manipulation, contrasting gnuplot's need for external preprocessing with awk and shell scripts.
- The article demonstrates practical examples: loading vmstat data into Pandas, plotting context switches vs. interrupts, and handling time windows with rolling statistics.
- Pandas supports advanced features like stacked graphs, derivatives using diff() or np.gradient(), and resampling to combine datasets or reduce noise.
- Data selection in Pandas is flexible using boolean indexing or query() methods, enabling precise filtering for analysis.
- Jupyter notebook provides an accessible interface for Matplotlib and Pandas, with setup instructions and a GitHub repository of sample data and code.
Pointing machines, population pyramids, post office scandal, type species, and horse urine
2 months ago
- Michelangelo likely used a 'compass method' for sculpting marble, not a pointing machine, which was invented later.
- Population pyramids, invented by Francis Amasa Walker in the 1874 US atlas, are powerful data visualizations showing past and future trends.
- In the British Post Office scandal, buggy software led to nearly 1,000 wrongful prosecutions due to arrogance, stupidity, and negligence.
- Miscarriages of justice may be common but underreported, as prosecutors have wide latitude and can convict innocents with vague evidence.
- Type species definitions in zoological nomenclature are complex, with post office scandal and biological terms highlighting obscure technical language.
- The hormonal drug Premarin is named after pregnant mare urine (PREgnant MARe's urINe), reflecting historical pharmaceutical practices.
Datasette-plot – Datasette Plugin for building data visualizations
2 months ago
- datasette-plot is a new Datasette plugin for creating visualizations from tables and SQL queries, serving as a successor to datasette-vega with more chart types and better integration.
- It leverages Observable Plot, a JavaScript visualization framework known for simplicity and beauty, allowing complex charts like stacked area charts with minimal code.
- After installation, users can access 'Show plot' on tables or SQL queries, customize visualizations with multiple marks, and share them via encoded URLs.
- Future plans include adding more visualization marks, improving sharing options for reusable charts, and supporting richer data types like dates.
The Largest Vocabulary in Hip Hop (2017)
2 months ago
- The analysis includes vocabulary size of 85 rappers based on 35,000 words from albums/EPs/mixtapes, with Aesop Rock having the largest vocabulary.
- Wu-Tang Clan ranks high due to diverse member contributions; GZA, Ghostface, Raekwon, and Method Man also appear in top 20.
- Regional breakdown shows East Coast has highest average vocabulary (4,804 words), while South has lowest (4,268), linked to crunk music style.
- Top artists include less-known names like Kool Keith, Canibus, and CunninLyguists, with E-40 and Outkast noted for slang and portmanteaus.
- Bottom 20% features big names like Lil Wayne, 2Pac, Snoop Dogg, Kanye West, and Drake, with DMX in last place, highlighting vocabulary diversity doesn't equate to success.
We rewrote our custom visualisation renderers from SVG to be in Canvas
2 months ago
- The new profiler dashboard replaces SVG with Canvas for rendering flame graphs, eliminating the need for DOM nodes per frame and reducing browser workload.
- Canvas uses an immediate-mode API, where drawing commands create pixels directly, avoiding style, layout, and garbage collection overhead associated with retained-mode SVG.
- Data is handled via columnar Arrow format over Arrow Flight, enabling zero-copy reads and efficient rendering by accessing arrays for frame attributes like x-positions, widths, and depths.
- Rendering optimizations include viewport culling to draw only visible frames and precomputed positions, making zooming, scrolling, and resetting interactions feel instant.
- Performance improvements show the Canvas renderer is about 3.6× faster in isolated tests, with live interactions like zooming and scrolling becoming 4-5 times quicker, though accessibility and text selection remain trade-offs.
An Interactive Map of AI
2 months ago
- The map of AI visually represents 56 concepts and 150 connections from Artifipedia, growing as new entries are added.
- It includes key AI concepts like AI Agent, Chain-of-Thought, and Transformers, covering topics from model training to ethics.
- The map is embeddable on any site, ensuring it stays current with the encyclopedia's updates, and is free to use.
Evilcharts
2 months ago
- Sankey charts visualize flow data using nodes and links with gradient colors and glow effects.
- The chart is a composable compound component with <EvilSankeyChart /> as the container and <Node />, <Link />, and <Tooltip /> as children.
- Interactive selection can be enabled by setting isClickable on <Node /> and using onSelectionChange callback.
- Loading state is supported via an isLoading prop that shows placeholder animations.
- Examples demonstrate customization of link variants, nodeWidth, nodePadding, and other properties.
- Labels can be displayed on nodes using <NodeLabel /> inside <Node />, with options for inside or outside placement.
- Link variants include "solid" for a single color and "source" for coloring based on source nodes.
- Glowing nodes can be added using the glow prop on <Node /> with an array of node names.
Sentiment analysis of Reddit threads using BERT models
3 months ago
- The interface indicates no positive standout or analyses have been conducted yet.
- Users are instructed to paste a public Reddit post link and press Analyze to see live emotion streams for comments.
- The thread currently reads as mixed with a net sentiment of 0.00, and 0 comments have been analyzed.
- No strong positive or negative keywords, and no standout comments are present in the current data.
- A call to action from Josip Bartulović of Sibence Digital invites teams to discuss building similar streaming data products.
Crossword Heatmap
3 months ago
- Analysis focuses on NYT crossword layouts: 15x15 daily and 21x21 Sunday versions.
- Visualizations show character distribution likelihoods per cell, not overall frequency.
- Characters grouped by placement patterns, e.g., 'Great starters' vs. 'Great enders'.
- Highlighted distribution extremes, like A's outliers vs. G's even spread.
- Identified rare characters (J, Q, X, Z) and their limited placement roles.

data visualization

The mean means nothing: data visualization to debug a latency problem

From gnuplot to Matplotlib & Pandas - Bert Hubert's writings

Pointing machines, population pyramids, post office scandal, type species, and horse urine

Datasette-plot – Datasette Plugin for building data visualizations

The Largest Vocabulary in Hip Hop (2017)

We rewrote our custom visualisation renderers from SVG to be in Canvas

An Interactive Map of AI

Evilcharts

Sentiment analysis of Reddit threads using BERT models

Crossword Heatmap