data analysis

Bilingual

Some animated Dutch COVID-19 Figures - Bert Hubert's writings
2 months ago
- The author created an animation of Dutch COVID-19 figures using the 'date of onset' field to reduce the impact of testing delays.
- The data uses 7-day smoothed daily positive test numbers to effectively remove weekend effects.
- Week-on-week growth was 60% at the end of September, after which the daily growth rate dropped.
- Part of the decrease is attributed to the latency between symptom onset and positive test results, with the point shifting right over nine days of data.
- Another part of the decrease may be real, and the animation shows how the growth rate develops over time as more data arrives.
- When the three lines (current, one-day-old, two-day-old) are close together, the growth rate for that date is considered settled.
TeleMessage Explorer: a new open source research tool
2 months ago
- Micah Lee released TeleMessage Explorer, an open-source research tool for analyzing the hacked TeleMessage dataset.
- The tool requires the TeleMessage dataset from DDoSecrets and about 530 GB of free space, along with Docker and Python/Poetry.
- Users extract strings from 2,729 heap dump files, then crunch the data into a PostgreSQL database.
- The tool includes a backend API, a Vue.js frontend, and a cruncher script to insert JSON objects into the database.
- TeleMessage Explorer allows searching and browsing groups, users, messages, and attachments, with examples including Signal messages and OAuth2 validations.
- The dataset contains data from various customers, including a White House staffer and messages from groups like 'POTUS | ROME-VATICAN | PRESS GC'.
Forensic statistics in Python
2 months ago
- The author describes a project that required reverse engineering a data analysis with ambiguous variable choices.
- An example is provided: three numbers with possible values (first: 31,41,59; second: 26,53; third: 58,97,93,23) and code enumerates all 24 combinations to print sums.
- All sums in the example are unique, so a known sum of 187 identifies the numbers as 41, 53, and 93, while a sum of 200 indicates no valid combination.
Could a Neuroscientist Understand a Microprocessor?(2017)
2 months ago
- Neuroscience methods, tested on a known microprocessor, fail to reveal meaningful understanding of its hierarchical information processing.
- Current analytic approaches in neuroscience may not produce insights into neural systems, even with unlimited data.
- Complex artificial systems like microprocessors can serve as validation platforms for time-series and structure discovery methods.
- Standard techniques (tuning curves, connectomics, lesions, Granger causality) show structure but not true functional comprehension.
- The processor's simple components (transistors) produce complex brain-like signals (oscillations, power laws), but these are epiphenomena.
- Existing methods often reveal correlations but lack causal or hierarchical explanations due to limitations in experimental design and theory.
Scrabble tile values vs. actual letter frequency across 172,835 words
2 months ago
- Scrabble tile values were set in the 1930s by counting letters on a newspaper page.
- A study of all 172,835 words in the ENABLE list found that tile values are accurate at extremes (E cheap, Q/Z expensive) but have 11 inversions where a higher-value tile is actually more common.
- Example: H is worth 4 points but appears more often than B (worth 3 points).
- The longest isograms (no repeated letters) are 'dermatoglyphics' and 'uncopyrightable' at 15 letters.
- The five letters A, E, P, R, S form the most anagrams: 12 different words.
- Letter S starts the most words (18,988), while X starts the fewest (136).
Exploring Gymflation with AI
2 months ago
- Project investigates 'gymflation' by analyzing deadlift PRs on YouTube over time.
- Uses YouTube API (via httr2) to collect video data from 2015 to 2025.
- Employs a local LLM (Ministral 3B) via Ellmer to extract weight, unit, and reps from video metadata.
- Improves extraction accuracy (92% on test set) using in-context learning with few-shot examples.
- Applies vision-language models (Gemini, MiniMax, Qwen) to classify lifter gender from thumbnails, achieving ~80% accuracy.
- Finds no evidence of gymflation: average deadlift weight has not increased over time; heavy lifts may even be decreasing.
- Highlights practical tips: caching API requests, using structured outputs, parallel processing, cost management, and manual annotation.
Funny item co-occurrences in 3.2M Instacart orders
2 months ago
- The Instacart dataset from 3 million orders was analyzed to find funny item co-occurrences.
- Initial analysis using product-level data yielded uninteresting results due to too many unique combinations.
- Using 'lift' (actual frequency vs. expected frequency) improved ranking but still lacked humor.
- Products were reclassified into 1,000 GPC bricks for better granularity, reducing complexity by 50x.
- A humor score (0-1) was assigned to each brick based on taboo or comedic potential.
- Funniest rare combos emerged when combining humor scores with lift, such as 'Garlic and Diarrhoea Remedies' or 'Kale and Enemas/Douches'.
- Small carts (2-3 items) also produced humorous pairs like 'Oreo Cookies and Personal Lubricant'.
What's the most popular number in Hacker News titles?
2 months ago
- The most popular number in Hacker News titles is initially found to be 2 using a naive regex query on the HN dataset, which counts digit sequences regardless of context.
- A smarter regex pattern that captures whole comma-grouped numbers, treats decimals as single tokens, and filters years reveals that 1 becomes the most popular number, followed by 2 and 3.
- Small numbers dominate due to three main reasons: listicles (e.g., '3 ways to do X'), version numbers (e.g., software releases like Rust 1.0), and round numbers used as thresholds or rankings (e.g., 'Top 10').
- External events affect number popularity, such as '19' spiking in 2020 due to COVID-19 and '4' jumping in 2023 because of GPT-4's launch.
- The analysis concludes that the prominence of small numbers in titles is not due to Benford's Law but rather to common content patterns and specific historical events.
Estimating the heights of New Yorkers from their scuff marks
2 months ago
- The wall at Smith–Ninth Streets has rough concrete, retaining scuff marks from commuters leaning against it.
- A darkened band about 18 inches above the floor is formed by aggregated shoe marks.
- Using a photo, the author isolated the scuff region, converted pixel coordinates to inches via a keypad reference.
- Relative luminance analysis showed darkness levels against height, indicating scuff frequency.
- Experimentally, a scuff mark was found to be about 27% up the body height, yielding a height-to-scuff ratio.
- Applying this ratio to the data estimated commuter heights, but variance was too wide due to inconsistent leg angles.
- Future work involves Bayesian correction using angle distributions and encourages collaboration on wear pattern studies.
How ChatGPT Picks Sources (I Read the Network Traffic, Not the Outputs)
3 months ago
- ChatGPT uses different data pipelines: serp for open web, labrador for licensed publishers, bright for Bright Data scraper, oxylabs for Oxylabs scraper.
- Some queries (like 'text' type) skip web searches and use training data, affecting SEO opportunities.
- The 'fan-out' queries reveal ChatGPT runs multiple sub-queries, including site-specific probes, for complex tasks.
- There is a difference between fetched, cited, and mentioned sources; citations require plain text and favor platforms like Reddit over YouTube.
- Ranking logic is server-side and not visible; personalization can influence answers based on user history.
- Local results are limited to two, and shopping queries show variability in sourcing behavior.
- To optimize for ChatGPT, ensure content is scrapable in plain HTML, avoid JavaScript for key data, and focus on third-party coverage and strong pages per claim.
Show HN: I measured the half-life of 41,301 Show HN launches. It's 7 hours
3 months ago
- The median Show HN launch gets 2 points and 0 comments, with 61.7% receiving no comments and 78.9% getting one or none.
- The half-life of a launch's comment thread is 7.2 hours; half of all comments arrive before the first workday ends.
- Success in terms of points or comments does not extend the half-life; even top launches decay within similar timeframes.
- By hour 48, 4.2% of comments remain for median launches, with 71% of launches nearly finished; traffic drops faster.
- Launch spikes are brief, so distribution plans should focus on long-term engagement beyond the initial 48-hour window.
- Shipyard is being built as an alternative to HN's ephemeral attention, offering sustained review collection post-launch.

data analysis

Some animated Dutch COVID-19 Figures - Bert Hubert's writings

TeleMessage Explorer: a new open source research tool

Forensic statistics in Python

Could a Neuroscientist Understand a Microprocessor?(2017)

Scrabble tile values vs. actual letter frequency across 172,835 words

Exploring Gymflation with AI

Funny item co-occurrences in 3.2M Instacart orders

What's the most popular number in Hacker News titles?

Estimating the heights of New Yorkers from their scuff marks

How ChatGPT Picks Sources (I Read the Network Traffic, Not the Outputs)

Show HN: I measured the half-life of 41,301 Show HN launches. It's 7 hours