Decision Trees

Building interpretable hierarchical decision boundaries by recursively splitting feature spaces.

🟢 beginner5 min readsupervisedtrees

Decision Trees are non parametric supervised learning algorithms that partition data into hierarchical axis aligned decision regions. Starting from a root node, the tree greedily splits features at thresholds that maximize impurity reduction (Gini Impurity or Information Gain Entropy). Decision Trees handle non linear patterns and mixed feature types without feature scaling, but overfit easily if unpruned.

What is a Decision Tree?

A Decision Tree is an intuitive hierarchical model that makes predictions by asking a series of yes/no questions about feature values.

                         [ Credit Score > 700? ]
                         /                     \
                      YES                       NO
                     /                            \
           [ Income > $50k? ]               [ Approved: FALSE ]
           /                \
        YES                  NO
       /                       \
 [ Approved: TRUE ]     [ Approved: FALSE ]

Decision trees partition feature space into axis aligned hyper-rectangles:

  Feature X2
     ▲
     │  Region A (Class 0) │  Region B (Class 1)
  c2 ┼─────────────────────┼─────────────────────
     │  Region C (Class 1) │  Region D (Class 0)
     └─────────────────────┴─────────────────────► Feature X1
                          c1

How Decision Trees Are Built (CART Algorithm)

Classification and Regression Trees (CART) build trees top-down using a Greedy Recursive Binary Splitting approach:

Evaluate Candidate Splits: For every feature $j$ and every possible threshold value $t$, evaluate splitting data into left node $D_L$ and right node $D_R$.
Select Best Split: Choose the feature $j^$ and threshold $t^$ that minimizes impurity or maximizes Information Gain.
Recurse: Repeat splitting on child nodes until a stopping criterion is met (e.g. maximum depth reached or minimum samples per leaf).

Key Advantages

High Interpretability: Decision trees can be visualized and explained easily to non technical stakeholders.
Zero Feature Scaling Needed: Invariant to monotonic scaling. StandardScaler has zero impact on split choices.
Handles Mixed Data Types: Manages numerical and categorical features naturally.

Key Disadvantages

High Variance & Overfitting: Deep unconstrained trees grow complex leaves that memorize training noise.
Axis Aligned Limits: Struggles to capture smooth diagonal decision boundaries because splits are always orthogonal (perpendicular to axes).
Instability: Small changes in training data can produce a completely different tree hierarchy.

Say this out loud

Decision Trees split data recursively into axis aligned decision regions using greedy threshold splits that maximize impurity reduction. They are highly interpretable and require zero feature scaling. However, unconstrained deep trees suffer from high variance and severe overfitting, which is fixed by pruning or ensembling into Random Forests.

Followups to expect

How does a Decision Tree make predictions for Regression tasks? The tree predicts the mean target value of all training samples falling into that specific leaf node, using Mean Squared Error to select splits.
What is Cost Complexity Pruning (α pruning)? Pruning subtrees that add excessive leaf nodes unless they reduce total impurity by more than penalty factor $\alpha$.

Check yourself

Question 1 of 3

Why do unconstrained deep Decision Trees suffer from severe overfitting on training data?

More in Classical ML

See all →

Bias–Variance Tradeoff4 min Overfitting vs Underfitting3 min Linear Regression4 min