Test-Driven Data Analysis (Chapman & Hall/CRC Data Science Series)

Test-driven data analysis is the synthesis of ideas from test-driven development of software to data-intensive work including data science, data analysis, and data engineering.It is a methodology for improving the quality of data and of analytical pipelines and processes.It can be thought of as data analysis as if the answers actually matter. Part I of the book describes the use of constraints for data validation and shows how tools can use a machine-learning-style approach to infer suitable constraints from data.This is illustrated using the command-line tools in the open-source Python tdda library, as well as alternatives.These can be used regardless of the programming language in use.Part II covers reference testing of analytical pipelines and processes.Again, there is support in the tdda library for this, including automatic test-generation capabilities for testing software in any language.There are specific discussions of testing models and modeling, including the large language m