Gini Impurity vs Entropy

Measuring Node Impurity

When a Decision Tree evaluates candidate feature splits, it looks for splits that make child nodes as pure as possible (containing samples from a single class).

Two primary metrics measure node impurity:

┌──────────────────────────┬──────────────────────────┐
│ 1. GINI IMPURITY         │ 2. SHANNON ENTROPY       │
├──────────────────────────┼──────────────────────────┤
│ Measures probability of  │ Measures Information     │
│ misclassifying a random  │ Disorder in bits using   │
│ sample.                  │ logarithms.              │
│ Faster (No logarithms!). │ Slightly stronger penalty│
│ Default in scikit-learn. │ for impure nodes.        │
└──────────────────────────┴──────────────────────────┘

1. Gini Impurity

Gini Impurity measures the probability that a randomly chosen element from the set would be incorrectly labeled if it were randomly labeled according to class distribution:

$$G = 1 - \sum_{i=1}^C p_i^2$$

• $p_i$: Probability of a sample belonging to class $i$.
• $C$: Total number of classes.

Example Calculations (2 Classes)

1. Completely Pure Node ($100%$ Class A, $0%$ Class B):

$$G = 1 - (1.0^2 + 0.0^2) = \mathbf{0.0}$$

2. Completely Impure Node ($50%$ Class A, $50%$ Class B):

$$G = 1 - (0.5^2 + 0.5^2) = 1 - 0.5 = \mathbf{0.5}$$

2. Entropy (Information Gain)

Shannon Entropy measures information disorder in bits:

$$H = -\sum_{i=1}^C p_i \log_2(p_i)$$

Example Calculations (2 Classes)

1. Completely Pure Node ($100%$ Class A):

$$H = -(1 \cdot \log_2(1) + 0) = \mathbf{0.0 \text{ bits}}$$

2. Completely Impure Node ($50%$ Class A, $50%$ Class B):

$$H = -(0.5 \log_2(0.5) + 0.5 \log_2(0.5)) = \mathbf{1.0 \text{ bit}}$$

Information Gain Ratio

Information Gain measures the reduction in entropy before and after a split:

$$\text{Gain}(S, A) = H(S) - \sum_{v} \frac{|S_v|}{|S|} H(S_v)$$

Detailed Metric Comparison

Say this out loud

Gini Impurity and Entropy both measure node impurity in Decision Trees, outputting 0.0 for pure single class nodes. Gini measures misclassification probability using squared class proportions, while Entropy measures information disorder in bits using logarithms. Gini is the default choice in scikit-learn because avoiding logarithms makes training faster with identical tree performance.

Followups to expect

1. What is Information Gain Bias toward high cardinality features? Pure Information Gain favors features with many unique values (like ID numbers), creating useless pure leaves. Use Gain Ratio (C4.5 algorithm) to normalize gain by split information.
2. Can Gini or Entropy be used for Regression Trees? No. Regression trees use Variance Reduction (minimizing Mean Squared Error or Mean Absolute Error) to evaluate splits on continuous target values.