In 1943, Warren McCulloch and Walter Pitts published the first concept of a simplified brain cell, the McCulloch-Pitts (MCP) neuron. They were trying to understand how the biological brain works in order to design artificial intelligence. Their model described a nerve cell as a simple logic gate with binary outputs: multiple signals arrive, they get integrated, and if the accumulated signal exceeds a certain threshold, an output signal is generated.

Only a few years later, in 1957, Frank Rosenblatt at the Cornell Aeronautical Laboratory published the first concept of the perceptron learning rule based on the MCP neuron model. Rosenblatt proposed an algorithm that would automatically learn the optimal weight coefficients that would then be multiplied with the input features in order to make the decision of whether a neuron fires or not.

The New York Times announced that the Navy expected this machine would soon "walk, talk, see, write, reproduce itself and be conscious of its existence." That did not happen. But what Rosenblatt created was something arguably more important: the first algorithm that could learn from examples. Every neural network, every LLM, every AI system you interact with today traces its lineage back to this single idea.

What Machine Learning Actually Is

Traditional programming works like this: you write rules, feed in data, and get output. If you want a spam filter, you write rules like "if email contains Nigerian prince, mark as spam."

Machine learning flips this around. You provide examples (emails labeled as spam or not spam), and the algorithm figures out the rules itself. The output is not a prediction. It is a model that can make predictions.

The perceptron was the first practical implementation of this idea.

The Biological Inspiration

Biological neurons are interconnected nerve cells in the brain involved in processing and transmitting chemical and electrical signals. McCulloch and Pitts described such a nerve cell as a simple logic gate with binary outputs: multiple signals arrive at the dendrites, they are then integrated into the cell body, and if the accumulated signal exceeds a certain threshold, an output signal is generated that will be passed on by the axon.

The perceptron mimics this with mathematics:

1. Take some inputs (numbers)
2. Multiply each input by a weight (how important is this input?)
3. Add them all up, plus a bias term
4. If the sum is zero or above, output 1. Otherwise, output 0.

That is the entire forward pass.

The Formal Definition

More formally, we can put the idea behind artificial neurons into the context of a binary classification task with two classes: 0 and 1. We define a decision function that takes a linear combination of input values x and a corresponding weight vector w, where z is the net input:

z = w1*x1 + w2*x2 + ... + wm*xm + b

Or in vector notation:

z = w · x + b

The decision function is a unit step function:

output = 1 if z >= 0, else 0

The weights control the orientation of the decision boundary. The bias controls its position, shifting it up or down. Together, they define a hyperplane that separates the two classes.

The Learning Algorithm

The whole idea behind the perceptron is to use a reductionist approach to mimic how a single neuron in the brain works: it either fires or it does not. The perceptron algorithm can be summarized by the following steps:

1. Initialize the weights and bias unit to 0 or small random numbers
2. For each training example:
   • Compute the output value (predicted class)
   • Update the weights and bias unit

The update rule is beautifully simple:

w = w + eta * (y - predicted) * x
b = b + eta * (y - predicted)

Here, eta is the learning rate (typically between 0.0 and 1.0), y is the true class label, and predicted is what the perceptron guessed.

Why the Update Rule Works

In the two scenarios where the perceptron predicts correctly, the weights remain unchanged since the update values are zero:

• True label = 0, Predicted = 0: update = eta * (0 - 0) * x = 0
• True label = 1, Predicted = 1: update = eta * (1 - 1) * x = 0

However, in the case of a wrong prediction, the weights are pushed toward the correct direction:

• True label = 1, Predicted = 0: update = eta * (1 - 0) * x = eta * x (weights move toward the input)
• True label = 0, Predicted = 1: update = eta * (0 - 1) * x = -eta * x (weights move away from the input)

Each update tilts the decision boundary slightly, trying to get the misclassified point on the correct side.

A Complete Implementation

Here is a clean implementation in Python:

import numpy as np

class Perceptron:
    def __init__(self, eta=0.01, n_iter=50, random_state=1):
        self.eta = eta
        self.n_iter = n_iter
        self.random_state = random_state
    
    def fit(self, X, y):
        rgen = np.random.RandomState(self.random_state)
        self.w_ = rgen.normal(loc=0.0, scale=0.01, size=X.shape[1])
        self.b_ = np.float_(0.)
        self.errors_ = []
        
        for _ in range(self.n_iter):
            errors = 0
            for xi, target in zip(X, y):
                update = self.eta * (target - self.predict(xi))
                self.w_ += update * xi
                self.b_ += update
                errors += int(update != 0.0)
            self.errors_.append(errors)
        return self
    
    def net_input(self, X):
        return np.dot(X, self.w_) + self.b_
    
    def predict(self, X):
        return np.where(self.net_input(X) >= 0.0, 1, 0)

We initialize weights to small random numbers rather than zeros. If all weights start at zero, the learning rate only affects the scale of the weight vector, not its direction. Small random values avoid this problem.

Training on Real Data

The classic demonstration uses the Iris dataset, classifying setosa versus versicolor flowers using sepal length and petal length as features. With just two features, we can visualize the decision boundary.

import pandas as pd

# Load data
s = 'https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data'
df = pd.read_csv(s, header=None, encoding='utf-8')

# Extract setosa and versicolor (first 100 samples)
y = df.iloc[0:100, 4].values
y = np.where(y == 'Iris-setosa', 0, 1)

# Extract sepal length and petal length
X = df.iloc[0:100, [0, 2]].values

# Train
ppn = Perceptron(eta=0.1, n_iter=10)
ppn.fit(X, y)

The perceptron converges after about six epochs and classifies all training examples perfectly. This works because setosa and versicolor are linearly separable in this feature space.

The Limitation

The convergence of the perceptron is only guaranteed if the two classes are linearly separable, meaning they can be perfectly separated by a linear decision boundary.

The classic counterexample is XOR:

Input (0,0) -> Output 0
Input (0,1) -> Output 1
Input (1,0) -> Output 1
Input (1,1) -> Output 0

Try drawing a single straight line to separate the 0s from the 1s. You cannot.

In 1969, Marvin Minsky and Seymour Papert published a critique pointing this out. The AI community panicked. Funding dried up. The AI Winter began.

If the two classes cannot be separated by a linear decision boundary, the perceptron will never stop updating the weights unless you set a maximum number of epochs.

The Solution

Add more layers. A multi-layer perceptron with a hidden layer can solve XOR. And with enough layers, it can approximate any function.

It took until 1986 for researchers to popularize backpropagation, the algorithm that makes training deep networks practical. This insight eventually led to the deep learning revolution we are living through now.

Why This Matters

The perceptron introduced every fundamental concept we still use:

• Weighted sums: the basic operation of every neural network
• Activation functions: the step function here, ReLU and others in modern networks
• Bias terms: allowing the decision boundary to shift
• Iterative learning: updating weights based on errors
• The idea that machines can learn from examples

Understanding the perceptron means understanding the atom of neural networks. Every layer of every deep network does essentially the same thing: weighted sum, activation, repeat.

The difference is scale and composition. Stack enough of these simple operations together, and you get GPT-4.

Try It Yourself

I built an interactive demo where you can place points and watch a perceptron learn to separate them in real-time: i33ym.cc/demo-perceptron

What is Next

In the next essay, we will look at Adaline, the adaptive linear neuron. It introduces a crucial upgrade: gradient descent. Instead of the perceptron's all-or-nothing update rule, Adaline uses calculus to find the optimal direction to adjust weights. This is the foundation of how all modern neural networks learn.