AI Measurement Science
a day ago
- AI system decisions depend on measurements of reasoning, planning, and safety, but measurement quality is rarely examined.
- The book frames evaluation as an inference problem: estimating latent properties from observed responses.
- It covers validity, data structure, probabilistic models (IRT, Bradley-Terry), estimation, reliability, efficiency, generalizability, and strategic design.
- Target audience includes researchers, practitioners, and students with backgrounds in probability, statistics, and machine learning.
- The goal is to move from ad-hoc benchmarking to principled measurement grounded in solid science.