Neural Networks

So far, we’ve explored models like Linear Regression, Logistic Regression, and Decision Trees. These models are powerful, they can predict sales, classify emails, or segment customers.

But they have limits:

• Linear regression only captures straight-line relationships.
• Logistic regression struggles when patterns aren’t clearly separable.
• Decision Trees can overfit and become unstable.

Now imagine data that is messy, curved, or full of complex patterns like handwriting recognition, voice commands, or detecting objects in photos. These problems require models that can handle non-linearity and abstraction.

That’s where Neural Networks come in.

Neural networks are inspired by the brain. They are built from simple units called neurons, which pass information forward, adjust based on mistakes, and gradually learn patterns from data.

When I trained my first neural network, I gave it the classic MNIST dataset, images of handwritten digits (0–9). Logistic regression only reached ~80% accuracy. The neural network, however, could recognize curved strokes, slanted writing, and pixel variations. It achieved 95%+ accuracy. It felt magical at first, but it wasn’t magic, just math layered in a way that could learn better.

What is a Perceptron and Activation Functions?

Think of a perceptron as the simplest brain cell in a machine.

• It takes inputs, multiplies them by weights (importance), adds a bias (offset), and produces an output after passing through an activation function.

Formula:

y=f(w1​x1​+w2​x2​+...+b)

Where:

• x1 ,x2​ = inputs (like “hours studied” or “attendance”)
• w1, w2​ = weights (how important each input is)
• b = bias term (adjustment factor)
• f = activation function

Example:
Suppose a perceptron predicts if a student will pass based on hours studied. If studying more hours increases passing chance, the weight for “hours” will be positive.

Why Do We Need Activation Functions?

If we just use raw sums, perceptrons behave like linear regression — limited to straight lines. The real world isn’t always linear.

Activation functions introduce non-linearity, allowing neural networks to learn curves, thresholds, and complex patterns.

Common Activations:

• Sigmoid: Outputs between 0 and 1 → good for probabilities.
• ReLU (Rectified Linear Unit): Turns negatives into 0, keeps positives → fast and widely used.
• Softmax: Used for multi-class problems, gives probability distribution across categories.

From my experience:
In my first deep network, I used sigmoid everywhere. Training was slow, and accuracy stalled. Switching to ReLU instantly sped up training and solved the vanishing gradient problem. It was a game-changer.

Why Add Multiple Layers?

One perceptron is like a simple rule. But stacking many perceptrons into layers creates a Multi-Layer Perceptron (MLP) that can model complex patterns.

• The input layer takes features.
• Hidden layers learn abstract patterns.
• The output layer gives predictions.

Example: Student Pass/Fail Prediction

Let’s build a neural network in TensorFlow/Keras.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
import numpy as np

# Dataset: [hours studied, attendance]
X = np.array([
    [2, 60], [10, 95], [4, 70], [15, 98], 
    [1, 40], [8, 85], [12, 90], [3, 55]
])
y = np.array([0, 1, 0, 1, 0, 1, 1, 0])  # 0=Fail, 1=Pass

# Build model
model = Sequential([
    Dense(8, activation='relu', input_shape=(2,)),   # Hidden layer
    Dense(4, activation='relu'),                     # Hidden layer
    Dense(1, activation='sigmoid')                   # Output (binary)
])

# Compile
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# Train
history = model.fit(X, y, epochs=100, verbose=0)

# Evaluate
loss, acc = model.evaluate(X, y, verbose=0)
print(f"Accuracy: {acc:.2f}")

Explanation

• import tensorflow as tf ... → Brings in TensorFlow and Keras, which contain ready-made tools for building and training neural networks. Without this, we’d have to code all math (matrix multiplication, backpropagation) by hand.
• X = np.array([...]) → Our input features: hours studied and attendance. We need to turn them into numbers so the model can learn from them.
• y = np.array([...]) → Our labels: 0 = Fail, 1 = Pass. This gives the model the “right answers” to learn from.
• model = Sequential([...]) → Creates a model that stacks layers in a straight line. Sequential is chosen because this is a simple left-to-right architecture (no fancy branches).
• Dense(8, activation='relu', input_shape=(2,)) → First hidden layer.
  • Why 8 neurons? It gives the model “space” to learn multiple patterns.
  • Why relu? It introduces non-linearity so the model can learn curves, not just straight lines.
  • Why input_shape=(2,)? Because each student has exactly 2 features (hours, attendance). Without this, the model wouldn’t know the input size.
• Dense(4, activation='relu') → Second hidden layer.
  • Why another layer? Stacking layers allows the network to combine patterns from the previous layer into more complex rules.
  • Why fewer neurons (4)? To gradually compress information, making the model simpler and less likely to memorize noise.
• Dense(1, activation='sigmoid') → Output layer.
  • Why 1 neuron? Because we only need one number (probability of passing).
  • Why sigmoid? It squashes output into [0,1], which is exactly what we want for probabilities.
• model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy']) →
  • Why Adam? It automatically adapts the learning rate for each weight, making training faster and more stable.
  • Why Binary Crossentropy? It’s the best loss function for yes/no problems because it punishes wrong confident predictions strongly.
  • Why Accuracy? Because in a binary task, accuracy is the simplest metric to judge how well the model is doing.
• model.fit(X, y, epochs=100) →
  • Why fit? This is the training loop, the heart of learning. Each epoch updates weights to make predictions closer to the true labels.
  • Why 100 epochs? Enough iterations to let the model learn patterns, but not too many to overfit (memorize).
• model.evaluate(X, y) → Tests the model after training. If accuracy is high, the model has learned the relationship between hours, attendance, and passing.
• print(f"Accuracy: {acc:.2f}") → Shows the impact of everything: how well our network predicts Pass vs Fail.

Field story:
When I built a student exam prediction system, logistic regression worked okay but ignored interactions between attendance and hours. Neural networks captured that students with moderate study hours but high attendance often passed. That insight was valuable for schools.

What is Multi-Class Classification?

So far, we predicted binary outcomes (yes/no). But many problems involve multiple categories.

Examples:

• Handwritten digit recognition (0–9).
• Fruit classification (apple, mango, orange).
• News categorization (sports, politics, tech).

Example: Classifying Fruits by Weight & Sweetness

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# Dataset: [weight (grams), sweetness score]
X = np.array([
    [150, 7], [120, 6],  # Apples
    [200, 9], [220, 10], # Mangoes
    [100, 4], [90, 3]    # Oranges
])
y = np.array([0, 0, 1, 1, 2, 2])  # 0=Apple, 1=Mango, 2=Orange

# Model
model = Sequential([
    Dense(8, activation='relu', input_shape=(2,)),   
    Dense(6, activation='relu'),                     
    Dense(3, activation='softmax')   # 3 classes
])

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

# Train
history = model.fit(X, y, epochs=200, verbose=0)

# Evaluate
loss, acc = model.evaluate(X, y, verbose=0)
print(f"Accuracy: {acc:.2f}")

Explanation with Why and Impact

• X = np.array([...]) → Each fruit is described by weight and sweetness. Choosing simple numeric features makes it easy to train.
• y = np.array([0,0,1,1,2,2]) → Class IDs: Apple=0, Mango=1, Orange=2. These IDs guide the model on which class each fruit belongs to.
• model = Sequential([...]) → Same as before: a stack of layers is enough for this simple task.
• Dense(8, activation='relu', input_shape=(2,)) →
  • Why 8 neurons? To capture multiple patterns (e.g., light but sweet vs heavy and sweet).
  • Why relu? To add non-linearity; fruits are not linearly separable by weight and sweetness.
  • Why input_shape=2? Because each fruit has exactly 2 features.
• Dense(6, activation='relu') → A second hidden layer. This compresses and refines the learned features, like combining “big size” and “sweetness” into a single concept of “likely Mango.”
• Dense(3, activation='softmax') → Output layer.
  • Why 3 neurons? Because we have 3 classes. Each neuron outputs probability for one fruit.
  • Why Softmax? Because it ensures the outputs are probabilities that sum to 1, making it easy to interpret (e.g., 80% Mango, 15% Apple, 5% Orange).
• model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) →
  • Why Adam again? Fast and reliable.
  • Why Sparse Categorical Crossentropy? Because labels are integers (0,1,2). If labels were one-hot vectors, we’d use categorical_crossentropy.
  • Why Accuracy? Multi-class accuracy is intuitive — what % of fruits were classified correctly.
• model.fit(X, y, epochs=200) →
  • Why 200 epochs? Multi-class problems often need more training because the model must learn multiple decision boundaries.
  • Each epoch helps the network refine probabilities for each fruit.
• model.evaluate(X, y) → Measures how well the final trained network predicts fruit types.
• print(f"Accuracy: {acc:.2f}") → Tells us the impact: whether the network can successfully distinguish apples, mangoes, and oranges.

Example:
I once helped a support team build a model to categorize customer tickets (billing, technical issue, general feedback). Before automation, agents manually sorted tickets. After deploying a Softmax-based classifier, urgent tech issues were routed instantly, cutting response times in half.

Advantages of Neural Networks

• Can model complex, non-linear patterns.
• Flexible: works for images, speech, text, numbers.
• With enough data, can outperform traditional ML models.

Disadvantages

• Needs lots of data to work well.
• Harder to explain than Decision Trees.
• Can overfit if network is too deep without safeguards.

Mistake I’ve seen:
A student once trained a deep network on just 100 samples. Accuracy showed 100%, but it was pure memorization. Adding more data and using dropout layers fixed it.

By now, you’ve learned:

• Perceptrons are the basic building blocks.
• Activation functions add the non-linearity needed for real problems.
• Multi-layer networks in TensorFlow/Keras are powerful for binary and multi-class classification.
• Softmax enables multi-class predictions, a key tool in many real-world applications.

Final thought from my journey:
The first time I deployed a neural network in production, I realized the value wasn’t in being “fancy.” It was in solving real-world problems others thought were impossible. Neural networks are just layers of math, but when applied correctly, they unlock possibilities, from self-driving cars to medical diagnostics.