model validation

model validation

by Dr. Eitan Farchi

Formal & Physical Sciences Computing & Cybersecurity UMZSoftware EngineeringUMZTSoftware testing and verification
🎙 Dr. Eitan Farchi 👥 46 📅 August 11, 2020 ⏱ 24 min 👁 3 📄 tutorial 🧭 2026-08-18
Available in: English (current) Français

Keywords

validationHoeffdingsample sizeerror probabilitymachine learning

Summary

The video is a lecture on model validation in machine learning, presented by Dr. Eitan Farchi. The speaker introduces the concept of validating a learned model h against an unknown target function f by estimating the probability of error. He explains that this probability can be estimated using a fresh validation set, and the average of an indicator variable provides an unbiased estimate. The variance of this estimate is shown to be p(1-p)/v, where p is the true error probability and v is the validation set size. To determine the required validation set size, the speaker uses Hoeffding’s inequality, which provides a bound that depends on v, unlike Chebyshev’s inequality. Solving the inequality yields v > 2/epsilon^2 * log(2/delta), where epsilon is the desired accuracy and delta is the confidence level. The lecture also touches on the connection to concept drift, noting that the assumption of a representative sample may not hold in the presence of drift. The presentation is informal but mathematically rigorous, suitable for an audience with some background in probability and machine learning.

176 words

Critical Evaluation

Value of the Information & Strength of the Argument

The video provides a clear and rigorous explanation of model validation, focusing on the statistical foundations. The argumentation is solid: the speaker derives the sample size formula using Hoeffding’s inequality, which is appropriate for bounded random variables. The explanation of why Chebyshev’s inequality is insufficient is insightful. The connection to concept drift is briefly mentioned, adding practical relevance. However, the lecture lacks concrete examples or visual aids, which might make it less accessible to beginners. The value lies in its theoretical clarity and the step-by-step derivation, which is valuable for understanding the underlying principles.

Scientific Rigor, Source Quality, Title Accuracy

The scientific rigor is high: the mathematical derivations are correct and well-explained. However, the video does not cite any external sources, and the description only mentions the speaker’s name. The title ‘model validation’ is appropriate and accurately reflects the content. There are no comments provided, so no analysis of public reception is possible.

162 words

Title / Content Match

The title 'model validation' accurately reflects the content, which focuses on the statistical validation of machine learning models.

Quality & Reliability

7/10

The content is a clear, mathematically grounded tutorial on model validation, using probability inequalities (Hoeffding) to derive sample size bounds. The reasoning is sound, but the video is a lecture without citations or references to external sources, and the presentation is informal.

Key Moments

Markers derived by PSI from the transcript: the creator did not define chapters.

Contribution & Novelties

The video offers a clear, self-contained derivation of the sample size needed for model validation, using Hoeffding’s inequality. It emphasizes the importance of a fresh validation set and the pitfalls of using the validation set for training. The connection to concept drift is a valuable addition, highlighting the assumptions underlying the validation process.

Pour aller plus loin :

97 words

Radar Profile

The radar profile shows high scores in quality of information and technical level, with moderate scores in quantity and reliability. This indicates a focused, mathematically rigorous tutorial that may lack breadth and external validation.

Reliability 7/10