Decision Trees and Ensemble Methods

Imagine you’re deciding whether to buy a house. You might ask:

• Is the price affordable?
• Is the location safe?
• Is the commute short?

Each question leads to a decision. If yes, you go one way; if no, you go another. By the end of the process, you’ve decided: buy or don’t buy.

That’s exactly how Decision Trees work in machine learning, they split data into smaller and smaller pieces by asking yes/no questions until they reach a decision.

But real-world data is messy. A single tree can easily overfit. That’s where Ensemble Methods like Random Forests and Gradient Boosting come in. Instead of relying on one tree, they combine many trees to build stronger, more accurate models.

I once built a loan approval model using decision trees. At first, accuracy looked great on training data, but on new applicants, the model failed badly, it had memorized quirks of the old data. Switching to a Random Forest instantly improved generalization, and later, Gradient Boosting gave the best balance between accuracy and interpretability.

What is a Decision Tree?

A Decision Tree is one of the simplest and most intuitive algorithms in machine learning. Imagine a flowchart, like the kind you might use for troubleshooting a problem. At each step, the tree asks a yes/no question, branches based on the answer, and eventually lands on an outcome.

• Internal nodes → Questions (e.g., “Is income > $50,000?”).
• Branches → Paths determined by yes/no answers.
• Leaves → Final outcomes (e.g., “Approve loan” or “Reject loan”).

Why are Decision Trees popular?

• They’re interpretable. You can explain them to someone with no ML background.
• They mimic human decision-making. Most people make decisions step by step, just like a tree.

Example: Will a Student Pass or Fail?

Let’s say we want to predict whether a student passes (1) or fails (0) an exam, based on hours studied and attendance percentage.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
from sklearn.tree import DecisionTreeClassifier
from sklearn import tree
import pandas as pd

# Dataset
data = {
    "hours_studied": [2, 10, 4, 15, 1, 8, 12, 3],
    "attendance": [60, 95, 70, 98, 40, 85, 90, 55],
    "passed": [0, 1, 0, 1, 0, 1, 1, 0]
}

df = pd.DataFrame(data)

X = df[["hours_studied", "attendance"]]
y = df["passed"]

# Train Decision Tree
clf = DecisionTreeClassifier(max_depth=3, random_state=42)
clf.fit(X, y)

# Visualize Tree
tree.plot_tree(
    clf, 
    feature_names=["hours_studied", "attendance"], 
    class_names=["Fail", "Pass"], 
    filled=True
)

Interpretation:

• The root node might split on “hours_studied > 6.”
• If yes, the model predicts “Pass.”
• If no, it checks attendance: if attendance < 70%, predict “Fail,” otherwise “Pass.”

This is very similar to how a teacher might think about predicting exam performance.

Advantages

• Easy to explain → Great for communicating with non-technical stakeholders.
• Works on different data types → Handles both numbers (hours) and categories (yes/no).
• Low preprocessing → No need to normalize or scale features.

Disadvantages

• Overfitting → Trees can grow too deep and memorize quirks of training data.
• Instability → Small changes in data can completely change the tree structure.

💡 Real-world mistake I made:
In one of my early churn prediction projects for a telecom company, a single decision tree gave me 99% accuracy on the training data. I was thrilled… until I tested it on new customers and accuracy dropped to 65%. The issue? My tree had grown too deep and was memorizing the training data. Setting a maximum depth fixed the problem.

Why Random Forests?

A single decision tree is like asking one friend for advice. What if that friend is wrong or biased?

A Random Forest is like asking 100 different friends, each with a slightly different perspective, and then averaging their opinions.

It builds many decision trees on:

1. Random subsets of the data.
2. Random subsets of features.

By averaging predictions, Random Forests reduce the weakness of any single tree.

Example: Predicting House Prices

We’ll use the same house pricing dataset we explored earlier.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
import numpy as np
import pandas as pd

# Dataset
data = {
    "sqft": [1400, 1600, 1700, 1875, 1100, 1550, 2350, 2450],
    "bedrooms": [3, 3, 4, 4, 2, 3, 4, 5],
    "bathrooms": [2, 2, 3, 2, 1, 2, 3, 4],
    "location_score": [8, 7, 9, 6, 5, 7, 9, 10],
    "price": [200000, 240000, 280000, 340000, 150000, 250000, 400000, 475000]
}

df = pd.DataFrame(data)
X = df.drop(columns=["price"])
y = df["price"]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Random Forest model
rf = RandomForestRegressor(n_estimators=100, random_state=42)
rf.fit(X_train, y_train)

# Predictions
y_pred = rf.predict(X_test)
rmse = np.sqrt(mean_squared_error(y_test, y_pred))
print("RMSE:", rmse)

Interpretation:
Instead of trusting one tree, the Random Forest relies on many. Each tree makes a prediction, and the forest averages them. This gives stable, robust predictions.

Advantages

• High accuracy in most cases.
• Resistant to overfitting compared to a single tree.
• Good defaults → works well even without heavy tuning.

Disadvantages

• Harder to interpret than a single tree (though feature importance can help).
• Slower when building many trees.

When I was working on predicting insurance claim amounts, I first tried a single tree. Results were inconsistent — even small tweaks in data changed predictions. Switching to a Random Forest gave consistent, trustworthy predictions, which made managers comfortable enough to use the model in real decision-making.

Why Gradient Boosting?

Random Forests combine trees independently. Gradient Boosting takes a different approach:

• It builds trees sequentially.
• Each new tree focuses on the errors made by the previous trees.

Think of it like a tutor:

• A student fails algebra questions.
• Next lesson focuses on algebra.
• Then geometry mistakes are corrected.

Over time, the student improves. That’s what Gradient Boosting does for models.

Example: Gradient Boosting Regressor

1
2
3
4
5
6
7
8
9
10
11
12
13
from sklearn.ensemble import GradientBoostingRegressor

gbr = GradientBoostingRegressor(
    n_estimators=100, 
    learning_rate=0.1, 
    max_depth=3, 
    random_state=42
)
gbr.fit(X_train, y_train)

y_pred = gbr.predict(X_test)
rmse = np.sqrt(mean_squared_error(y_test, y_pred))
print("RMSE:", rmse)

Interpretation:

• Each tree is correcting the mistakes of the last.
• The learning rate controls how much each tree contributes. A small learning rate (0.05–0.1) makes the model slower but more stable.

XGBoost and LightGBM Basics

Both are specialized libraries for gradient boosting, popular in industry and Kaggle competitions.

• XGBoost (Extreme Gradient Boosting):
  • Adds regularization to prevent overfitting.
  • Highly optimized with parallel processing.
• LightGBM (Light Gradient Boosting Machine):
  • Trains much faster on large datasets.
  • Uses histogram-based splitting to reduce memory usage.

In a credit risk project, XGBoost outperformed Random Forests in accuracy and speed. Later, when scaling up to millions of rows, LightGBM handled the data in minutes where XGBoost took hours.

Advantages

• Extremely accurate for structured/tabular data.
• Handles non-linear relationships effectively.
• Flexible with many parameters to fine-tune.

Disadvantages

• Complex to tune requires experimentation.
• Slower than Random Forests.
• Prone to overfitting if the learning rate is too high.

Mistake I’ve seen:
A student once set the learning rate to 1.0 thinking “faster learning = better.” Instead, the model wildly overfit. Reducing it to 0.1 stabilized the training and gave excellent results.

Conclusion

From this module, you should now understand:

• Decision Trees: Great for interpretation but fragile and prone to overfitting.
• Random Forests: Stronger, more stable by averaging many trees.
• Gradient Boosting (XGBoost, LightGBM): Sequentially improves on errors, delivering state-of-the-art performance in many real-world tasks.

Final thought from my experience:
In practice, I usually follow a progression:

• Start with a Decision Tree to understand patterns.
• Move to a Random Forest for stability and decent accuracy.
• Try XGBoost or LightGBM when maximum accuracy is needed.

This combination balances interpretability, speed, and predictive power exactly what businesses want.