Validity, Reliability, and Significance: Empirical Methods for NLP and Data Science - Riezler, Stefan (Synthesis lectures on human language technologies, 2022)
Cover -- Copyright -- Title Page -- Contents -- Preface -- Acknowledgments -- Introduction -- Empirical Methods in Machine Learning -- Scope and Outline of this Book -- Intended Readership -- Validity -- Validity Problems in NLP and Data Science -- Bias Features -- Illegitimate Features -- Circular Features -- Theories of Measurement and Validity -- The Concept of Validity in Psychometrics -- The Theory of Scales of Measurement -- Theories of Measurement in Philosophy of Science -- Prediction as Measurement -- Feature Representations -- Measurement Data -- Descriptive and Model-Based Validity Tests -- Dataset Bias Test -- Transformation Invariance Test -- A Model-Based Test for Circularity -- Notes on Practical Usage -- Reliability -- Untangling Terminology: Reliability, Agreement, and Others -- Performance Evaluation as Measurement -- Descriptive and Model-Based Reliability Tests -- Agreement Coefficients for Data Annotation -- Bootstrap Confidence Intervals for Model Evaluation -- Model-Based Reliability Testing -- Notes on Practical Usage -- Significance -- Parametric Significance Tests -- Sampling-Based Significance Tests -- Bootstrap Resampling -- Permutation Tests -- Model-Based Significance Testing -- The Generalized Likelihood Ratio Test -- Likelihood Ratio Tests using LMEMs -- Notes on Practical Usage -- Mathematical Background -- Generalized Additive Models -- General Form of Model -- Example -- Parameter Estimation -- Linear Mixed Effects Models -- General Form of Model -- Example -- Parameter Optimization -- The Distribution of the Likelihood Ratio Statistic -- Score Function and Fisher Information -- Taylor Expansion and Asymptotic Distribution -- Bibliography -- Authors' Biographies.
Description based on publisher supplied metadata and other sources