neural-networks.io

neural-networks.io

Linear regression example

In this example, we want to approximate the following scatter plot with a single layer neural network. Blue points are the training set given by an input \( x_i \) and an expected output \( y'_i \). The red line is the output of the network \( y=f(x) \) after training.

The following perceptron will be used for the single layer network:

\( x \) is the input. The activation function is given by \( f(x)=x \).

Update rule

As explained on the previous page, the weights will be updated according to this formula:

$$ w_i'= w_i + \eta(y'-y)x_i $$

Let's detail for each weight \( w_1 \) and \( w_2 \):

$$ w_1'= w_1 + \eta(y'-y)x $$ $$ w_2'= w_2 + \eta(y'-y) $$

Source code

Click one of the language below to display the source code of this example:

close all;
clear all;
clc;


%% Parameters
% Model parameters
a=0.6;
b=-2;
NoiseStd=0.5;
% Training set size
N=1000;
% Learning rate
Eta=0.003;

%% Create trainning set
% X is the input
X=20*rand(N,1)-10;
% Y_ is the expected output (+ noise)
Y_=a*X+b +normrnd(0,NoiseStd,N,1);


%% Initialize weight
W=[0;0];

%% Trainig loop
for i = 1:numel(X)
    % Forward
    Y=W'*[X(i);1];
    W=W+Eta*(Y_(i) - Y)*[X(i);1];
end

%% Plots
% Training set
plot (X,Y_,'.');
hold on;

% Network output
Xt=[-10:10];
Yt=W(1)*Xt + W(2);
plot (Xt,Yt,'r','LineWidth',4);
xlabel ('X');
ylabel ('Y');
title ('Single Layer perceptron');
axis square equal;
grid on;

%% Display results
Model=[a,b]
Weights=W'

Output :

Model =

    0.6000   -2.0000


Weights =

    0.5954   -1.8923

#include <iostream>

// Parameters of the line (y=ax+b)
#define     a       0.6
#define     b       -2

// Size fo training set
#define     N       1000

// Learning rate
#define     ETA     0.03

// Generate random double
double rand_double(double fMin, double fMax)
{
    double f = (double)rand() / RAND_MAX;
    return fMin + f * (fMax - fMin);
}

int main(void)
{
    // Create training set
    double X[N],Y_[N];
    for (int i=0;i$lt;N;i++)
    {
        X[i]=rand_double(-10,10);
        Y_[i]=a*X[i]+b;
    }


    // Training
    double w1=0,w2=0;
    for (int i=0;i<N;i++)
    {
        // Forward
        double Y=w1*X[i]+w2;
        // Update weights (backward)
        w1=w1+ETA*(Y_[i]-Y)*X[i];
        w2=w2+ETA*(Y_[i]-Y);
    }

    std::cout << "::: Results :::" << std::endl;

    // Display a and b
    std::cout << std::endl;
    std::cout << "** Model **" << std::endl;
    std::cout << "a=" << a << std::endl;
    std::cout << "b=" << b << std::endl;

    // Display weights (w1 and w2)
    std::cout << std::endl;
    std::cout << "** Network weights **" << std::endl;
    std::cout << "w1=" << w1 << std::endl;
    std::cout << "w2=" << w2 << std::endl;


    return 0;
}

Output :

::: Results :::

** Model **
a=0.6
b=-2

** Network weights **
w1=0.6
w2=-2