NeuralNetworks icon indicating copy to clipboard operation
NeuralNetworks copied to clipboard

Published 20 hours ago • verified publisher icon GiorgosXou
→

Metadata

A resource-conscious neural network implementation for MCUs

Simple MLP - NeuralNetwork Library For Microcontrollers

Nothing "Import ant", just a simple library for implementing Neural-Networks(NNs) easily and effectively on any Arduino board and other microcontrollers.

Summary

NN Functions Input Type (x) Output Type (Y) Action
getMeanSqrdError(x) Unsigned Int float Calculates & Returns the MSE (Mean Squared Error) = [ SSE (Sum Squared Error) divided by the Product of number-οf-οutputs and inputs-per-epoch aka batch-size].
*FeedForward(x) 1D Float Array 1D Float Array "Feeds" the NN with X-input values and returns Y-Output Values, If needed.
BackProp(x) 1D Float Array - "Tells" to the NN if the outputs-Y were correct/the-expected/X-inputs and then, "teaches" it.

Examples: Backpropagation_Single_Xor_Gate.ino
Understanding the Basics of a Neural Network:
0 1 2 3 4 5 6 7 8 9 10 11 12 .

Examples

* Example Files (.ino) About
1 Backpropagation_Single_Xor_Gate Training a NN to behave as a xor-gate
2 Backpropagation_double_Xor Training a NN to behave as a 3-input-xor circuit
3 FeedForward_double_Xor Pre-trained NN behaving as a 3-input-xor circuit
4 Run_In_Loop Example of using a NN inside the loop
5 FeedForward_double_Xor_PROGMEM Weights & biases are stored in the programmable memmory
6 FeedForward_Individual_MNIST_PROGMEM Recognizing handwritten digits [proof of concept]
7 Any_Activation_Function_Per_Layer Example of using multiple Activation Functions per Layer.

Important

Features

  • Current
    • + Use of activation-functions per layer-to-layer.
    • + Optimizations based on user's preference.
    • + MSE/BCE/CCE loss-functions.
    • + Many activation-functions.
    • + Simplicity!
  • In the future, i want
    • - Better code.
    • - Use of custom functions.
    • - More Activation Functions.
    • - Different weight initialization methods.
    • - Even more properties, for many different needs.

Tested On

  • Arduino Uno
  • ATtiny85
    • doesn't have FPU that makes Maths on it, "difficult" for the SRAM (i think..?)
    • If you want to Use "Serial" On An ATtiny85 Click Here (Be Careful SoftwareSerial Uses A lot of SRAM)
    • Backprop maths on An ATtiny85 doesn't work properly for some reasons, though Feed Forword maths Work! [...]
    • (since the first release I haven't tested it again on the ATtiny85 at least yet, so I am not sure)

Functions, Variables & [...]

(NN) Neural-Network's Constructors
NeuralNetwork()
NeuralNetwork(const unsigned int *_layer, const unsigned int &NumberOflayers, byte* _ActFunctionPerLayer=nullptr)
NeuralNetwork(const unsigned int *layer_, const unsigned int &NumberOflayers, const float &LRw, const float &LRb, byte *_ActFunctionPerLayer=nullptr)
NeuralNetwork(const unsigned int *layer_, float *default_Weights, float *default_Bias, const unsigned int &NumberOflayers, byte *_ActFunctionPerLayer=nullptr);

Type [Main Functions]

NN Functions Input Type (x) Output Type (Y) Action
FeedForward_Individual(x) Float 1D Float Array "Feeds" the NN with each one X-input Individually until it returns Y-Output Values, If needed. (Way Less RAM too)
*FeedForword(x) 1D Float Array 1D Float Array "Feeds" the NN with X-input values and returns Y-Output Values, If needed.
BackProp(x) 1D Float Array - Tells to the NN if the outputs-Y were right/the-expected-X-inputs and then, teaches it.
print() - String Serial.Prints the weights and biases of NN. (If B10000000 prints from PROGMEM)

float [Loss Functions]

No need for #define MEAN_SQUARED_ERROR, MSE is the default loss and it is always enabled. The only case in which you will also need to define the MSE in your sketch, is only if you want to use it in relation with another loss-function. To use any other loss-function except from MSE just define it as seen below.

Loss Functions Enabling MACRO
NN.getMeanSqrdError(unsigned int batch_size) #define MEAN_SQUARED_ERROR
NN.getBinaryCrossEntropy(unsigned int batch_size) #define BINARY_CROSS_ENTROPY
NN.getCategoricalCrossEntropy(unsigned int batch_size) #define CATEGORICAL_CROSS_ENTROPY

float [Loss Variables]

To use any of the variables below, you first need to #define a loss function as said above too.

Loss variables Sum variables
NN.MeanSqrdError NN.sumSquaredError
NN.BinaryCrossEntropy NN.sumOfBinaryCrossEntropy
NN.CategoricalCrossEntropy NN.sumOfCategoricalCrossEntropy

float [Activation Functions]

Because of (my uncertainty and) the strict RAM optimization that allows the library to use one array that stores only the values after the activation instead of two arrays storing values before and after the activation, the use of some derivative functions in backpropagation are not supported by this library at this moment, as also seen by the MACRO NO_BACKPROP below. This means that if you want to use and #define any function from 8-13 under the section "NO_BACKPROP support" , you won't be able to use backpropagation.

Enabling MACRO Activation Functions Action
0 #define Sigmoid NN.layers->Sigmoid(&x) Returns 1/(1+e^(-x))
1 #define Tanh NN.layers->Tanh(&x) Returns (e^(2*x)-1)/(e^(2*x)+1)
2 #define ReLU NN.layers->ReLU(&x) Returns (x>0)?x:0
3 #define LeakyELU NN.layers->LeakyELU(&x) Returns (x>0)?x:AlphaLeaky*x
4 #define ELU NN.layers->ELU(&x) Returns (x>0)?x:AlphaELU*(e^(x)-1)
5 #define SELU NN.layers->SELU(&x) Returns (x>0)?x:AlphaSELU*(e^(x)-1)
6 #define Softmax NN.layers->Softmax(&x) void "complicated implementation"
7 #define Identity NN.layers->Identity(&x) Returns x
NO_BACKPROP support
8 #define BinaryStep NN.layers->BinaryStep(&x) Returns (x < 0) ? 0 : 1
9 #define Softplus NN.layers->Softplus(&x) Returns log(1 + exp(x))
10 #define SiLU NN.layers->SiLU(&x) Returns x / (1 + exp(-x))
11 #define GELU NN.layers->GELU(&x) Returns (1/2) * x * (1 + erf(x / sqrt(x)))
12 #define Mish NN.layers->Mish(&x) Returns x * Tanh(log(1 + exp(x)))
13 #define Gaussian NN.layers->Gaussian(&x) Returns exp(-(x*x))
Derivative Functions
0 #define Sigmoid NN.layers->SigmoidDer(&fx) Returns fx-fx*fx
1 #define Tanh NN.layers->TanhDer(&fx) Returns 1-fx*fx
2 #define ReLU NN.layers->ReLUDer(&fx) Returns (fx>0)?1:0
3 #define LeakyELU NN.layers->LeakyELUDer(&fx) Returns (fx>0)?1:AlphaLeaky
4 #define ELU NN.layers->ELUDer(&fx) Returns (fx>0)?1:fx+AlphaELU
5 #define SELU NN.layers->SELUDer(&fx) Returns (fx>0)?LamdaSELU:fx+AlphaSELU*LamdaSELU
6 #define Softmax NN.layers->SoftmaxDer(&fx) Returns fx * (1 - fx)
7 #define Identity NN.layers->IdentityDer(&x) Returns x

if you want to use other activation function from the default one, just define one other:

#define Sigmoid //[default] No need definition, for single activation across network
#define Tanh
#define ReLU
#define LeakyELU
#define ELU
#define SELU
...

Use any activation function per layer-to-layer, like :

#define ACTIVATION__PER_LAYER
#include <NeuralNetwork.h>

unsigned int layers[] = {3, 4, ..., 2, 1};
byte Actv_Functions[] = {   1, ..., 2, 0};

// Tanh > ... > ReLU > Sigmoid

If you want to drastically reduce ROM & slightly RAM size you can Define which Functions to use/compile, like:

#define ACTIVATION__PER_LAYER
		#define Sigmoid // 0
		//#define Tanh
		//#define ReLU
		//#define LeakyELU
		#define ELU     // 1
		#define SELU    // 2
		...
#include <NeuralNetwork.h>

unsigned int layers[] = {3, 4, ..., 2, 1};
byte Actv_Functions[] = {   1, ..., 2, 0};

// ELU > ... > SELU > Sigmoid

⚠️ have in mind that because I used f(x) for derivatives there might be chances of mistakes (?)


float [Variables Of Activation Functions]

Enabling MACRO Activation Variables Default Explenation
#define LeakyELU NN.AlphaLeaky 0.01 the α of Leaky
#define ELU NN.AlphaELU 1 the α of ELU
#define SELU NN.AlphaSELU 1.6733 the α of SELU
#define SELU NN.LamdaSELU 1.0507 the λ of SELU

Type [Other Variables]

Type NN's Variables Explenation
byte* NN.ActFunctionPerLayer if ACTIVATION__PER_LAYER defined
float NN.LearningRateOfWeights The Learning-Rate-Of-Weights
float NN.LearningRateOfBiases The Learning-Rate-Of-Biases
float* NN.weights If B00010000 defined
Layer* NN.layers Layers of NN
Layer's Variables
float* NN.layers[i].bias The bias of an individual layer[i]
float* NN.layers[i].outputs[] The Output array of an individual layer[i]
float** NN.layers[i].weights[][] if not B00010000 defined
float* NN.layers[i].preLgamma[] The γ-error of previous layer[i-1]
unsigned int NN.layers[i]._numberOfInputs The number-Of-Inputs for an individual layer[i]
unsigned int NN.layers[i]._numberOfOutputs The number-Of-Outputs for an individual layer[i]

#define [MACRO Properties]

#define _1_OPTIMIZE B00000000
_1_OPTIMIZE Action
B00000000 Nothing
B10000000 ⚠️ Enables the use of PROGMEM" instead of RAM
B01000000 ⚠️ Deletes previous layer's Outputs: Reduces RAM by a factor of ((the_sum_of_each_layer'_s _numberOfOutputs) - (_numberOfOutputs of_biggest_layer) *4 ) approximately i think ?
B00100000 🔴 Partialy Reduces RAM for Weights, level 1
B00010000 Reduces RAM for Weights, level 2 by a factor of number_of_layers*2 - 2 bytes
B00001000 🟢 Deletes previous layer's Gamma
B00000100 Reduce RAM by a factor of 2*(number_of_layers - 1 or 2)bytes slightly different usage with multiple NNs
B00000010 Recomended | Disables default MSE loss function | Reduces ROM, RAM & CPU consumption


⚠️ = Backpropagation is not allowed
🔴 = Not yet implimented
🟢 = Always enabled (not switchable yet.)


Searches Across Internet

Here most of the resources I came across the internet, I recomend you to have a look if you want to (but please stay aware of the fact that for some of those sites, I had only opened them checked something and then closed them in a matter of seconds [so, please don't get them all siriously])

THANK YOU!

I also want especially to thanks Underpower Jet for his amazing tutorial, by bringing it more to the surface. Because after all the videos and links I came across, he was the one that made the most significant difference to my understanding of neural networks. Plus, I would like to thanks: giant_neural_network for this and this, 3Blue1Brown for this, the Arduino community and everyone else who gave me the oportunity to learn and make this library possible to exist [...]

New Searches:

12\08\2021


xx\xx\202x

Old Searches:

| | Intresting |NN.| Neural Network(s) |A.| Arduino etc. |-| Mostly .NET & Other |*| Maybe Intresting?

NNs PROGMEM Define directive Other & "Random"
Playlist Arduino Tutorial A. Initialize Array Values
Playlist Manual Arduino Define A. Inheritance,destructors?
BackPropagation Examples Determining board A. Identifying Arduino type?
Math Chain Rule + Post define extern? A. Create compile error?
Getting Started + Pointers .ino Determining Board A. Measuring Memory Usage
+ BackProp Tutorial + Double Info Understanding #if ? A. External Memory
+ BackProp Tutorial read-only? Random Defined Site? A. ATtiny85 Math Issues?
+ Complete NN chart ! flash to RAM? Loading local libraries? A. Attiny85 External Mem.?
+ MIT Deep RL Info Near Far? A. Splitting Array?
+ MIT Deep Learning Example A. Importing Loads Everything?
- .NET Framework What is PGM_P? NN. Backprop. For Dummies YT
- .NET ! 1,2,3,4,5,6 Passing Array? NN. Convolutional (Math- code)
- C# Implementation, 2 Passing Array? NN. In 4 minutes - Python
- .NET Deep NN Reading? NN. Quick Intro
* Predicting Languages Easy data handling? NN. -
* MIT Recurrent NNs Reading Long? NN. PyConSG 2016
- 2007 .NET Img Recognition Multi-D Array? NN. Simple 9 lines of Py-code
- C# Handwritten Recognition Attiny85 mem? NN. MIT Press book
Youtube Chanel Prob Array use? NN. A Beginner's Guide
+ Recurrent Explained Double or float? NN. MIT courses?
- .NET 1,2,3 NN.Back Propagation
- C# Handwritten Recognition NN. MLP Maths?
+ Python NN From Scratch NN. Math. Notations Into Code
* How Backpropagation! NN. Maths Into Code
linear Regression NN. (TAA),(BDI) Architecture
+ MLP NN. fast.ai ?
+ MLP NN. deeplearning.net
🇬🇷 Νευρωνικά Δίκτυα NN. BackProp Python
🇬🇷 Γενετικός Αλγόριθμος NN. C# Code
- MS NN Classification NN. Implement BackProp Python
- VB ML Q-Learning NN. Java Tut Playlist ?
* handwritten Recognition NN. BackProp for Dummies
* Deep Learning in 6 weeks? NN. Wiki Backprop Math
* Playlist NN. Looking Inside
- SciBasic

Outro

Forgive me if I have made any mistakes and please don't take me seriously with every claim i made, I am mainly "self taught" in this field of NeuralNetworks, I am not a professional programmer nor do I have a good knowledge in many of the fields used to create this library [...]

I need help, I need to learn,
I want to learn, I want to help [...]
I wish love and happiness to Everyone! 🤍
Σημασία έχει, ότι ως αποτέλεσμα έχει την αγάπη.

if you want to help me and others to educate ourselves better and if you have a love and passion for sharing and helping others, then I highly suggest you to join our discord server 🤍

My Instagram account is: giorgos.xou ;) feel free to ask me anything

Donation

if you want me to continue support this library, learn and bring more functionality to it, it would be really helpfull if you would consider donating, even the least amount of 0.01$ would be really appreciated!

Bitcoin Address:

image

1N8niMND7SqE8B7rsE7Mx7gPctkx2rUMTf

Donate Donate Donate

(I need money, else I might gonna kill myself for real). stackedit.io have been used for the editing of this MD file

← Metadata

66
Stars
21
Forks
Watchers

Owner

verified publisher icon GiorgosXou

Metadata

A resource-conscious neural network implementation for MCUs

Back