Interactive Decision Boundary Explorer

Watch in real-time as different ML models warp the decision space to separate classes through Gradient Descent. Experiment with Logistic Regression's linear constraints or a Neural Network's hyper-dimensional folding.

Click to add points, or generate a dataset

Data Setup

0.20

Model & Training

Epoch
0
Loss
0.0000
0.100
0.0010

Decision Boundary - The Explanation

Math Explorer

Logistic Regression

Models the probability of a point belonging to class 1 using a linear combination of features passed through a Sigmoid activation function.

z=w1x1+w2x2+bz = w_1x_1 + w_2x_2 + b
P(y=1∣x)=11+e−zP(y=1|x) = \frac{1}{1 + e^{-z}}

This algorithm mathematically can only draw a straight straight line through the 2D plane. It fails on non-linear datasets (like Moons or Circles).

1. Formal Definition

A decision boundary is a function (or set of points) that partitions the feature space into regions assigned to different classes.

Mathematically, for a classifier:

f(x)=0f(x) = 0

This defines the boundary. The classification follows:

  • If f(x)>0f(x) > 0 → Class 1
  • If f(x)<0f(x) < 0 → Class 2
Conclusion: So the model is not just predicting—it is dividing space.

2. Geometric Interpretation

Let's think geometrically. Depending on the dimensions of your input data:

  • In 2D: The boundary is a line
  • In 3D: The boundary is a plane
  • In higher dimensions: The boundary is a hyperplane

For a linear classifier, this geometric separator is captured by the equation:

wTx+b=0w^T x + b = 0

Where ww = weight vector (direction), xx = input features, and bb = bias (shift).

Conclusion: This is not just an equation—it is a geometric separator.

3. Visual Intuition (Critical)

Imagine plotting your data. Each point is a student: Red = Fail, and Blue = Pass.

The decision boundary is the theoretical line where the model is fundamentally uncertain.

Important Insight: The boundary is the exact threshold where the model's confidence switches.

4. Linear vs Non-Linear Boundaries

Linear Boundary

Assumes classes are separable by a perfectly straight feature line.

Common Models:
  • Logistic Regression
  • Linear SVM
  • Perceptron
Limitation: Cannot capture complex patterns.

Non-Linear Boundary

Uses curved, flexible, and adaptive contours to separate data.

Common Models:
  • Neural Networks
  • Kernel SVM
  • Decision Trees
Advantage: These models reshape space itself to separate data.

5. Learning the Boundary (Optimization Perspective)

Now, this is where theory meets practice. A model does not "draw" a boundary directly. Instead, it:

  1. Defines an objective function f(x)f(x)
  2. Uses a loss function to measure the cost of misclassification
  3. Optimizes parameters w,bw, b iteratively
Conclusion: The boundary emerges naturally as a consequence of minimizing mathematical error.

6. Margin — A Subtle but Powerful Idea

In Support Vector Machines, just having a boundary is not enough; we want to maximize the margin.

Margin = Distance to nearest points

Why? Because a larger margin leads to drastically better generalization.

Deep Principle: The goal is not just separating data, but separating it robustly.

7. Overfitting vs Underfitting (Critical Insight)

Let me challenge you to think visually:

  • A straight line might be too simple → Underfitting
  • A highly wiggly curve wrapping every point → Overfitting

Fact: The goal is not perfect separation of training data.

Goal: The goal is pure generalization to unseen data.

8. Probabilistic Interpretation

Especially in architectures like Logistic Regression, the decision boundary is exactly where:

P(y=1∣x)=0.5P(y=1 \mid x) = 0.5
  • Boundary = Maximum uncertanity space.
  • Away from boundary = Higher confidence spaces.

9. High-Dimensional Reality

In real ML systems, you rarely work in 2D or 3D. You work in hundreds or thousands of dimensions. Yet the same core idea holds:

Insight: A decision boundary is always a hyperplane or hypersurface slicing through N-dimensional space.

10. Final Conceptual Insight

"A machine learning model is nothing but a system that learns how to carve space into meaningful regions."

  • Data lives in space.
  • The model draws boundaries.
  • Prediction = finding which region a point belongs to.

Summary (Professor's Closing Notes)

  • * Decision boundary = separating surface in feature space.
  • * Defined formally by f(x)=0f(x) = 0.
  • * Linear boundaries are simple and interpretable.
  • * Non-linear boundaries are powerful and flexible.
  • * Boundaries are learned via iterative optimization.
  • * Margin strictly improves robustness.
  • * Complexity must always be controlled.