∅no free lunch.
Menu / 🥫 The Pantry

The Pantry:
what every model cooks with.

Models are the dishes. These are the ingredients on the shelf — run out of one and every recipe tastes off.

📖 A companion to the model guide, not a stop on the Route or the Menu: each model page links here when it leans on one of these ideas. Read it front to back, or dip in from a model and head back to the Menu.

Interviewers rarely ask “explain random forest” in a vacuum. They ask why it beats a single tree (it averages away the wobble, called variance), how you'd evaluate it when only 0.3% of payments are fraud (precision–recall curves, calibrated probabilities), and what broke when it looked brilliant offline (leakage: the model peeked at the answers). Pick your depth below (you can flip any single card too). The Refresher view of each card gives you:

one-line definitionintuitionthe formula🎤 interview one-liner⚠ common trapused by → models🧪 5 live widgets
PANTRY STOCK28 ingredients · 8 shelves
  • Generalization5
  • Training & optimization5
  • Evaluation4
  • Data prep4
  • Geometry & distance3
  • Trees & ensembles3
  • Probability & lin. alg.2
  • Unsupervised & time2
SHELF LIFEevery interview
*** no free lunch ***
01

Generalization

The only thing that matters is how you do on data you haven't seen.

1.1

The bias–variance trade-off

#bias-variance

Think of two darts players. One always lands in the same spot, but it's the wrong spot. That's bias: the model is too simple, like drawing a straight line through lemonade sales that rise with temperature and then level off. The other player's darts are centred on the bullseye but scattered everywhere. That's variance: the model is so flexible that it changes wildly whenever it sees slightly different data.

Making a model more flexible fixes the first player and creates the second. Your job is the middle: flexible enough to follow the real pattern, calm enough not to chase noise.

🧪 Try it · slide from stiff to wiggly

Test error splits into what your model is too rigid to capture (bias), how much it wobbles with the training sample (variance), and noise nobody can remove.

Imagine retraining the same model on a hundred different samples of the same world. Bias is how far the average of those hundred models sits from the truth: a straight line fitted to a curved relationship is wrong in the same way every time. Variance is how much the hundred models disagree with each other: a 15-degree polynomial chases each sample's quirks and looks different every time.

Flexibility trades one for the other. More capacity (deeper trees, a smaller $k$ in k-nearest neighbours (kNN), weaker regularisation) cuts bias and raises variance. Averaging many models (bagging, short for bootstrap aggregating), shrinkage (regularisation) and more data all cut variance. The best model sits at the bottom of the U-shaped test-error curve, and that bottom can never go below the irreducible noise.

$$\mathbb{E}\big[(y-\hat f(x))^2\big] = \underbrace{\big(\mathbb{E}[\hat f(x)] - f(x)\big)^2}_{\text{bias}^2} + \underbrace{\operatorname{Var}\big[\hat f(x)\big]}_{\text{variance}} + \underbrace{\sigma^2}_{\text{noise}}$$

The decomposition is exact for squared loss. For 0-1 loss (simply counting right vs wrong) it's messier, but the intuition carries over. Modern caveat: very over-parameterised models can show “double descent”, where test error falls again past the interpolation point (the size at which the model can fit every training row exactly). Know the phrase; don't lead with it.

🧪 Try it · fit a curve, watch the U
🎤 Interview one-liner
“Bias is being consistently wrong; variance is being inconsistently right. I tell which one I have by comparing the error on the data the model learnt from with the error on data it has never seen, then I either give the model more flexibility, or rein it in and feed it more data.”
“Bias is being consistently wrong, variance is being inconsistently right. I diagnose which one I have by comparing training error with validation error (error on held-out data), then add capacity, or add constraints and data.”
⚠ Common trapSaying “more data fixes overfitting and underfitting.” More data reduces variance only. A high-bias model stays wrong with a billion rows: train and validation error both plateau high and close together.
1.2

Overfitting & underfitting

#overfitting

A student who memorises last year's exam paper word for word scores 100% on that paper, then flunks the real exam because the questions changed. That's overfitting: a delivery-time model that learnt every quirk of last month's orders (including that one rainy Tuesday) but can't predict next week.

The opposite, underfitting, is the student who only learnt “the answer is usually C”, which is wrong everywhere. The tell: an overfit model looks brilliant on the data it trained on and disappointing on new data. An underfit one is mediocre on both.

Overfitting is learning the training set's noise as if it were signal. Underfitting is failing to learn the signal at all.

The tell-tale signature is in the learning curves (training and validation error plotted against the amount of training data). An overfit model has low training error and much higher validation error, and the gap shrinks as you add data. An underfit model has both errors high and close together, and adding data doesn't help.

Each model family has its own overfitting knob: tree depth and the minimum rows per leaf (scikit-learn: min_samples_leaf); the number of boosting rounds without early stopping (halting once validation error stops improving); $k=1$ in k-nearest neighbours (kNN); a very narrow radial-basis-function (RBF) kernel, i.e. a large $\gamma$, in support vector machines (SVMs); too many polynomial features. The cure is the same idea every time: constrain capacity, add data, or average many models, and pick the setting on held-out data rather than training data.

Overfitting has a quieter cousin: overfitting the validation set. Try 500 hyperparameter combinations on one validation split and the winner is partly lucky. Nested cross-validation (CV: an inner loop of splits to tune, an outer loop to score), or a final untouched test set, keeps you honest.

🎤 Interview one-liner
“I compare how the model does on the data it learnt from with how it does on fresh data. A big gap means it memorised (overfitting), so I simplify it or get more data; poor on both means it's too simple (underfitting), so I give it more to work with.”
“I look at the train–validation gap and the learning curve: a big gap that closes with more data means overfitting, so I regularise or get data; both errors high means underfitting, so I add features or capacity.”
⚠ Common trapCalling 99% training accuracy “great”. For a 1-nearest-neighbour model or a fully grown decision tree, 100% training accuracy is guaranteed and tells you nothing. Only out-of-sample numbers count.
1.3

Regularization (L1, L2 & friends)

#regularization

Imagine packing for a trip with a strict luggage limit. You can still bring what matters, but you think twice before adding a third pair of shoes. Regularization is a luggage limit for a model: it can still use features, but every big weight costs something.

In a demand forecast for a dark store (a delivery-only grocery warehouse) with 60 signals, it stops the model from leaning hard on a signal that only looked useful by luck. There are two flavours (the “L1” and “L2” in the title are their maths names). Ridge (L2) makes every feature pack lighter. Lasso (L1) makes you leave some items at home entirely, so it picks features for you. The widget below shows a model predicting orders from just two signals: app push notifications and discount depth.

🧪 Try it · pack the model's luggage

Add a cost for complexity to the objective, so the model only uses big weights when the data really pays for them.

Fitting the data perfectly is cheap; fitting it with small, simple parameters is what generalises. Regularisation adds a penalty term with strength $\lambda$ (lambda): you trade a bit of training fit (bias ↑) for stability across samples (variance ↓).

$$\min_{\beta}\; \underbrace{\sum_i \big(y_i - x_i^\top\beta\big)^2}_{\text{fit}} \;+\; \lambda\,\underbrace{\Big(\alpha\lVert\beta\rVert_1 + \tfrac{1-\alpha}{2}\lVert\beta\rVert_2^2\Big)}_{\text{elastic-net penalty}}$$

$\lVert\beta\rVert_1$ is the sum of absolute weights and $\lVert\beta\rVert_2^2$ the sum of squared weights. $\alpha$ mixes the two: $\alpha=1$ is pure lasso (L1), $\alpha=0$ pure ridge (L2), anything in between is elastic net. Naming clash: scikit-learn calls the strength λ alpha and the mix l1_ratio.

L2 · RidgeShrinks all weights smoothly towards 0 and never exactly to 0. Shares weight across correlated features. Has a closed-form solution; equivalent to a Gaussian (bell-curve) prior (see MLE & MAP below).
L1 · LassoDrives many weights exactly to 0, so it does feature selection. Arbitrarily picks one feature from a correlated group. Equivalent to a Laplace (sharp-peaked) prior.
Beyond linearTree depth / min leaf size, learning rate & subsampling in boosting, $C$ in support vector machines (SVMs; an inverse strength!), dropout in neural nets, early stopping, and weight decay in matrix factorisation.

The geometry explains the sparsity. Rewrite the penalty as a budget $\lVert\beta\rVert \le t$. The solution is where the loss contours first touch the budget region, and the L1 diamond has corners sitting on the axes.

🧪 Try it · why Lasso zeroes coefficients
🎤 Interview one-liner
“Regularization accepts a slightly worse fit on the training data in exchange for a model that doesn't swing wildly. Ridge shrinks every weight a little; lasso pushes some weights to exactly zero, so it also picks features for you.”
“Regularisation trades a little bias for a lot less variance. L2 (ridge) shrinks everything and copes with correlated features; L1 (lasso) zeroes coefficients because its diamond-shaped constraint has corners on the axes; elastic net gets both.”
⚠ Common trapRegularising unscaled features. The penalty treats every coefficient equally, so a feature in rupees (tiny β) gets almost no penalty while one in lakhs gets crushed. Standardise first. Also: in scikit-learn (sklearn), the C parameter of logistic regression and SVMs is the inverse of λ, so a smaller C means a stronger penalty.
1.4

Train / validation / test splits & cross-validation

#train-test

You don't let students see the final exam while they're studying, and the same goes for models. Split your data three ways: a training set to learn from (the textbook), a validation set to try out settings (mock tests), and a test set you open only once, at the very end (the real exam).

Cross-validation just rotates the mock-test role across several slices, so one lucky slice can't fool you. Golden rule: split the way the model will be used. If you're forecasting next month's orders, train on the past and test on the future. Never shuffle them together.

Train on one slice, choose settings on a second, and report on a third that you only touch once.

The training set fits the parameters. The validation set (or the cross-validation folds described below) picks hyperparameters and features. The test set is a sealed envelope that estimates real-world performance. Use the test set to make any decision and it quietly becomes a validation set, and your reported number becomes optimistic.

k-fold cross-validation (CV) cuts the data into $k$ parts (folds), rotates the validation role across them and averages, which gives a lower-variance estimate on small data. The split must mirror how the model will be used:

  • Stratified folds for imbalanced classes, so each fold keeps the positive rate.
  • Grouped folds (scikit-learn: GroupKFold) when rows share an entity (the same user, store or patient), so you don't test on someone you trained on.
  • Time-based folds (scikit-learn: TimeSeriesSplit; also called rolling-origin evaluation) when the future must not leak into the past.
$$\widehat{\text{Err}}_{\text{CV}} = \frac{1}{k}\sum_{j=1}^{k} \frac{1}{|F_j|}\sum_{i\in F_j} L\big(y_i,\, \hat f^{(-j)}(x_i)\big)$$

$F_j$ is fold $j$, $\hat f^{(-j)}$ is the model trained on every fold except $j$, and $L$ is the loss.

🎤 Interview one-liner
“I split the data the way the model will be used: if we predict next month, I train on earlier months and test on later ones, and I keep each customer in just one split. I tune settings on the validation data and open the test set only once, at the end.”
“My split mimics production: if we predict next month for existing customers, I split by time and group by customer. Cross-validation picks hyperparameters; the test set is touched once, at the end.”
⚠ Common trapA random split (scikit-learn's train_test_split shuffles rows by default) on time-ordered or multi-row-per-user data. Validation scores look superb because the model has already seen tomorrow, or the same user, and they collapse in production.
1.5

Data leakage

#data-leakage

Leakage is the model secretly peeking at the answer key. Say you're predicting whether a food order will arrive late, and one column is the customer's delivery rating. Ratings are only given after delivery, and late orders get low ratings. So in testing the model “predicts” lateness perfectly, but live it's useless, because the rating doesn't exist yet when the order is placed.

Leakage also sneaks in when you prepare the data using the whole dataset before splitting it. A simple test for every column: would I actually know this value at the moment I make the prediction?

Information reaches training that won't be available at prediction time, so validation scores lie.

Leakage is the number-one reason a model is “amazing offline, useless online”. It comes in two flavours:

  • Target / feature leakage: a feature that is a consequence of the label. A “refund issued” flag predicts fraud perfectly. So does the delivery partner's rating when predicting late delivery: it is only filled in after the order lands.
  • Train–test contamination: fitting any data-preparation step on the full dataset before splitting: feature scaling, filling in missing values (imputation), principal component analysis (PCA), target encoding, or SMOTE (Synthetic Minority Over-sampling Technique, which invents new rare-class rows). Duplicate or near-duplicate rows across splits. Random splits on time series.

The defence is procedural. Every feature must answer: “would I know this value at the exact moment I make the prediction?” Put all preprocessing inside a scikit-learn Pipeline (one object that chains the preparation steps and the model) so it is re-fit on the training part of each cross-validation fold. Use point-in-time joins (look up each feature as it was at prediction time) from the feature store, the shared database of model inputs. Be suspicious of any single feature with near-perfect importance.

🎤 Interview one-liner
“If results look too good to be true, I suspect leakage first: I check that every column would really be known when the prediction is made, and that all data preparation is learnt from the training data only.”
“If validation looks too good, I assume leakage first: I check feature timestamps against the prediction time, fit all preprocessing inside the cross-validation loop (one scikit-learn Pipeline), and audit the top features for anything that's a consequence of the label.”
⚠ Common trapOversampling the rare class (e.g. with SMOTE, Synthetic Minority Over-sampling Technique) or target-encoding before the cross-validation split. Synthetic neighbours of validation rows end up in training, and the AUC (area under the ROC curve, a 0.5–1 ranking score) jumps by 0.1 for no real reason.
02

Training & Optimization

What the model is trying to minimise, and how it gets there.

2.1

Loss functions

#loss-functions

A loss function is the model's scorecard for mistakes: it decides how much each mistake hurts. When predicting delivery time, being 2 minutes off and 30 minutes off are very different. Squared error treats the 30-minute miss as 225 times worse, so a few disasters pull the model around. Absolute error treats it as just 15 times worse, so it stays calmer about outliers.

For yes/no problems like fraud, log-loss punishes the model most when it was confidently wrong. Choosing the loss is choosing what “good” means, so pick the one that matches what actually costs the business.

The loss is how the model is told what a mistake costs. Pick it, and you've picked what “best” means.

Squared error punishes big misses quadratically, so its optimum is the conditional mean, and a few outliers drag it around. Absolute error's optimum is the median, which makes it robust but non-smooth at zero. Huber loss is squared near zero and linear in the tails. Quantile (pinball) loss targets any percentile you choose, which is great for “90% of deliveries arrive within X minutes”.

For classification, log-loss (cross-entropy) rewards calibrated probabilities and punishes confident mistakes without limit. Hinge loss (support vector machines, SVMs) stops caring once a point is safely beyond the margin, so only the points near the boundary (the “support vectors”) shape the model, and you get no probabilities. Exponential loss (AdaBoost, adaptive boosting) punishes misclassified points very hard, so it is sensitive to label noise.

$$\underbrace{(y-\hat y)^2}_{\text{squared (MSE)}}\quad \underbrace{|y-\hat y|}_{\text{absolute (MAE)}}\quad \underbrace{-\big[y\log p + (1-y)\log(1-p)\big]}_{\text{log-loss}}\quad \underbrace{\max(0,\,1 - y f(x))}_{\text{hinge},\; y\in\{\pm1\}}$$

(Averaged over rows, the first two are the mean squared error, MSE, and the mean absolute error, MAE; $p$ is the predicted probability and $f(x)$ the raw score.) The training loss and the business metric often differ. You train on a smooth, differentiable proxy (log-loss) and then choose thresholds or report the metric the business cares about (e.g. recall, the share of frauds caught, at a 1% false-positive rate).

🎤 Interview one-liner
“The loss decides what the model aims for: squared error chases the average, absolute error the middle value (the median), log-loss honest probabilities. I pick the one that matches what a mistake really costs, then judge the model on the business's own measure.”
“The loss defines what the model estimates: mean squared error (MSE) gives the mean, mean absolute error (MAE) the median, pinball loss a quantile, log-loss calibrated probabilities. I pick the loss to match the decision, then evaluate on the business metric.”
⚠ Common trapTraining with MSE on a heavy-tailed target (delivery times, order value) and then wondering why predictions are biased high for typical cases. Use Huber or MAE, a log-transform, or model the quantile directly.
2.2

Gradient descent

#gradient-descent

You're on a foggy hillside and want to reach the lowest point. You can't see the valley, but you can feel which way the ground slopes under your feet. So you take a step downhill, feel again, and step again. That's gradient descent: the model nudges its settings a little in whichever direction reduces its error most, over and over.

The learning rate is your step size. With tiny steps you'll get there eventually. With giant leaps you overshoot, land on the opposite slope, and can bounce higher and higher until training blows up.

🧪 Try it · help the hiker down the valley

Repeatedly take a small step downhill on the loss surface, in the direction of steepest descent.

The gradient $\nabla L$ points uphill, so you step the other way, scaled by the learning rate $\eta$. Batch gradient descent uses all rows per step. Stochastic / mini-batch gradient descent (SGD) uses a small random sample: noisier, but far cheaper per step, and the noise even helps escape bad spots. Momentum keeps part of the previous step to build speed; RMSProp (root-mean-square propagation) and Adam (adaptive moment estimation) also adapt the step size per direction.

$$w_{t+1} = w_t - \eta\,\nabla L(w_t) \qquad\text{(on a quadratic bowl it converges if and only if } 0 \lt \eta \lt 2/\lambda_{\max}\text{)}$$

The shape of the bowl matters as much as the step size. If one direction is steep (large curvature $\lambda_{\max}$; here λ means curvature, an eigenvalue of the matrix of second derivatives, not the regularisation strength) and another is flat ($\lambda_{\min}$), the learning rate is capped by the steep wall but progress is limited by the flat floor. The ratio $\kappa = \lambda_{\max}/\lambda_{\min}$ is the condition number, and unscaled features are the classic way to make κ huge.

Gradient boosting is the same idea in function space: each new tree is a step along the negative gradient of the loss with respect to the current predictions.

🧪 Try it · learning rate vs. a narrow canyon
🎤 Interview one-liner
“Gradient descent repeatedly takes a small step downhill on the error. The learning rate is the step size: too big and it overshoots and blows up, too small and it crawls. Putting features on similar scales makes the valley rounder and the walk much easier.”
“Gradient descent steps against the gradient, scaled by the learning rate. Too large overshoots and diverges past 2/λmax (λmax = the steepest curvature); too small crawls. Ill-conditioning from unscaled features makes it zig-zag, so I standardise and use momentum or Adam.”
⚠ Common trapBlaming the model when the loss explodes to NaN (“not a number”). Nine times out of ten the learning rate is too high or the features are on wildly different scales. Lower η, standardise, then debug further.
2.3

Convexity

#convexity

Picture two landscapes. One is a single smooth bowl: wherever you drop a marble, it rolls to the same bottom. The other is a bumpy mountain range with many little dips, and the marble gets stuck in whichever dip is closest.

A convex problem is the bowl. Logistic regression (a standard model for yes/no predictions) is one: train it ten times and you get the same answer every time. k-means clustering (which sorts points into k groups around k centres) is the bumpy kind: different starting points can give different clusters. That's why it's run several times and the best result is kept.

A convex loss is a single bowl: any local minimum is the global minimum, so the optimiser can't get trapped.

Formally, the chord between any two points on the curve lies on or above it. Practically, convexity means reproducible training: different seeds and initialisations reach the same answer, and you can trust that a bad model comes from the features, not from the optimiser.

$$f\big(\theta x + (1-\theta) y\big) \le \theta f(x) + (1-\theta) f(y), \qquad \theta\in[0,1]$$
Convex ✓Ordinary least squares (OLS), ridge, lasso, logistic regression, the linear support vector machine (SVM, hinge loss) and the SVM's dual problem. Ridge is strictly convex, so its optimum is unique.
Non-convex ✗k-means, Gaussian mixture models (GMMs) fitted by expectation–maximisation (EM), matrix factorisation (bi-convex: convex in the user factors or the item factors alone, not both), t-SNE and UMAP (t-distributed stochastic neighbour embedding; uniform manifold approximation and projection), neural nets, and greedy tree building (not even an optimiser of a global loss).

Non-convex models need restarts and good initialisation: several k-means restarts (scikit-learn: n_init), k-means++ seeding (spread-out starting centres), multiple EM runs, and alternating least squares (ALS), which fixes items and solves for users, then swaps, so each half-step is convex.

🎤 Interview one-liner
“Convex means a single bowl, so training lands on the same best answer every time. Logistic regression is convex; k-means clustering isn't, so we run it several times from different starts and keep the best.”
“Convex means one bowl, so any minimum is global and results don't depend on initialisation. Logistic regression and support vector machines (SVMs) are convex; k-means, Gaussian mixture models (GMMs) and matrix factorisation aren't, which is why we use multiple restarts.”
⚠ Common trapAssuming convex means unique. Lasso with correlated features, or ordinary least squares (OLS) with more features than rows ($p \gt n$), has a convex loss but a whole set of equally optimal solutions. Strict convexity (ridge) is what pins down a single answer.
2.4

Maximum likelihood (MLE) & maximum a posteriori (MAP)

#mle-map

Maximum likelihood is detective work: “which explanation makes the evidence I see most likely?” If 30 of 100 orders were late, the most likely late-rate is 30%. That's MLE (maximum likelihood estimation), and most training losses are secretly this idea.

But what if you saw only 3 orders and all were late? MLE says “100% late”, which is clearly overconfident. MAP (maximum a posteriori: the best guess after combining evidence with a prior belief) adds common sense from before you saw the data (a prior, such as “late rates are usually 5–20%”) and blends it with the evidence. With little data the prior matters a lot; with lots of data the evidence wins. Regularization is exactly this kind of prior.

MLE picks the parameters that make the observed data most probable. MAP does the same but also weighs in a prior belief about the parameters.

Write down a story for how the data was generated, e.g. “each label is a coin flip with probability $\sigma(x^\top\beta)$”, where $\sigma$ is the sigmoid that squashes any number into 0–1. Then find the $\theta$ under which the data you actually saw is most likely. Taking logs turns products into sums, and the negative log-likelihood is the familiar loss:

$$\hat\theta_{\text{MLE}} = \arg\max_\theta \sum_i \log p(y_i \mid x_i, \theta) \qquad \hat\theta_{\text{MAP}} = \arg\max_\theta \Big[\sum_i \log p(y_i\mid x_i,\theta) + \log p(\theta)\Big]$$
  • Gaussian noise ⇒ MLE = least squares. Laplace (sharp-peaked, heavy-tailed) noise ⇒ least absolute error (MAE, mean absolute error).
  • Bernoulli (0/1 coin-flip) labels ⇒ MLE = log-loss (logistic regression).
  • Gaussian prior on β ⇒ MAP = ridge. Laplace prior ⇒ lasso. Regularisation is a prior in disguise.
  • Laplace smoothing in Naive Bayes (adding 1 to every count) is MAP with a Dirichlet prior. Expectation–maximisation (EM, used to fit Gaussian mixture models) is MLE when some variables (cluster labels) are hidden.

MAP is still a point estimate. Full Bayesian inference keeps the whole posterior and gives you uncertainty, at a computational cost.

🎤 Interview one-liner
“Maximum likelihood picks the settings that make the observed data most probable. MAP also mixes in a prior belief, which stops the model overreacting to small samples, and regularization is exactly such a prior.”
“Most losses are negative log-likelihoods: squared error is Gaussian maximum likelihood (MLE) and log-loss is Bernoulli MLE. Adding a prior gives maximum a posteriori (MAP), which is exactly L2 or L1 regularisation for Gaussian or Laplace priors.”
⚠ Common trapMLE on tiny or perfectly separable data. Unregularised logistic regression on separable classes sends weights to infinity, and Naive Bayes without smoothing assigns probability 0 to an unseen word. A prior (MAP) fixes both.
2.5

Hyperparameter tuning

#hyperparameter-tuning

Hyperparameters are the oven settings, not the recipe. The model learns its own ingredients (weights) from data, but you have to choose things like “how deep can a tree grow?” or “how big are the learning steps?”.

Tuning means baking several test batches with different settings and tasting each one on held-out data. Grid search tries every combination. Random search tries a random sample and usually finds good settings faster. Smarter tools learn which settings look promising. Just never taste-test on the final exam data, or you'll fool yourself about how good the dish is.

Choosing the settings the model can't learn itself (depth, λ, k, learning rate) by measuring held-out performance.

Grid search tries every combination: fine for 1–2 knobs, wasteful beyond that. Random search samples combinations and, for the same budget, explores many more values of the knobs that actually matter (Bergstra & Bengio). Bayesian optimisation (e.g. Optuna's tree-structured Parzen estimator, TPE, or Gaussian-process-based tools) fits a model of “settings → score” and proposes promising settings next. Successive halving / Hyperband kills bad configurations early, on small budgets.

Practical rules: search scale-type knobs (λ, C, learning rate) on a log scale; tune the few knobs that matter most first (learning rate × number of trees, depth, regularisation); use early stopping instead of tuning the number of rounds; and keep the test set out of it. If you need an unbiased estimate of the tuned pipeline, use nested cross-validation (CV): an inner loop tunes, an outer loop scores.

🎤 Interview one-liner
“I try settings on held-out validation data, using random search or smarter tools over the few settings that matter most, and keep the final test set untouched so my reported score stays honest.”
“I use random or Bayesian search on log scales over the few high-impact knobs, with early stopping and cross-validation splits that mirror production, and report on an untouched test set (or nested cross-validation) so the tuning itself doesn't inflate the score.”
⚠ Common trapReporting the best cross-validation score from a 1,000-trial search as your expected performance. It's the maximum of noisy estimates, so it's biased upwards. Re-evaluate the winner on fresh data.