Course: Applied Machine Learning

This course covers the applied/coding side of algorithmics in machine learning, with some deep learning and evolutionary algorithms thrown in as well.

(The course is now in its third iteration, and I update this repo continually)

Moshe Sipper’s Cat-a-log of Writings

Some Pros and Cons of Basic ML Algorithms, in 2 Minutes

Course Topics ✧ Resources

Syllabus

❖ Python ❖ Artificial Intelligence ❖ Date Science ❖ Machine Learning Intro ❖ Scikit-learn ❖ ML Models ❖ Decision Trees ❖ Random Forest ❖ Linear Regression ❖ Logistic Regression ❖ Linear Models ❖ Regularization: Ridge & Lasso ❖ AdaBoost ❖ Gradient Boosting ❖ AddGBoost ❖ Ensembles ❖ XGBoost ❖ Comparing ML algorithms ❖ Gradient Descent ❖ SVM ❖ Bayesian ❖ Metrics ❖ Data Leakage ❖ Dimensionality Reduction ❖ Clustering ❖ Hyperparameters ❖ Some Topics in Probability ❖ Feature Importances ❖ Semi-Supervised Learning ❖ Neural Networks ❖ Deep Learning ❖ Large Language Models ❖ DL and AI ❖ Evolutionary Algorithms: Basics ❖ Evolutionary Algorithms: Advanced

Topics (according to order of instruction)

(: my colab notebooks, : my medium articles)

• Python

  • Python Programming
  • Python
  • PyCharm
  • Pandas
  • NumPy
  • NumPy
  • Numba
  • np.dot vs loop example

• Artificial Intelligence

  • Computing Machinery and Intelligence
  • Four Takeaways on the Race to Amass Data for A.I.

• Date Science

  • Data Science Landscape
  • Building the Machine Learning Model
  • 21 Most Important (and Must-know) Mathematical Equations in Data Science

• Machine Learning Intro

  • Machine Learning: history, applications, recent successes
  • How to avoid machine learning pitfalls
  • Top 10 machine learning algorithms with their use-cases
  • Introduction to machine learning
  • Train/Val/Test
  • Simple weather example
  • Iris knn (map)
  • kfold

• Scikit-learn

  • Machine learning with scikit-learn
  • A Minimal Example of Machine Learning (with scikit-learn)
  • 19 Most Elegant Sklearn Tricks I Found After 3 Years of Use
  • Toy datasets (sklearn)

• ML Models

  • Machine learning models

• Decision Trees

  • Decision trees
  • Decision trees

• Random Forest

  • Random Forests
  • Random Forests

• Linear Regression

  • Linear Regression
  • LinearReg
  • A Quick Summary of Linear Regression

• Logistic Regression

  • Logistic Regression
  • Logistic Regression
  • Cross-Entropy Loss
  • Linear Regression vs Logistic Regression in a Nutshell
  • LinVsLog
  • PolynomialFeatures

• Linear Models

  • Optimization of linear models

• Regularization: Ridge & Lasso

  • Ridge vs. Lasso
  • notes on regularization in ML

• AdaBoost

  • Adaptive Boosting

• Gradient Boosting

  • Gradient Boosting
  • Why tree gradients give you a boost

• AddGBoost

  • AddGBoost
  • Strong(er) Gradient Boosting

• Ensembles

  • Two’s Company, Three’s an Ensemble

• XGBoost

  • XGBoost

• Comparing ML algorithms

  • Comparing supervised learning algorithms
  • How to find the best performing Machine Learning algorithm
  • load_boston removal
  • Choosing the right estimator

• Gradient Descent

  • Gradient Descent
  • Least Squares
  • Least Squares
  • Stochastic Gradient Descent
  • Stochastic Gradient Descent Algorithm With Python and NumPy
  • Gradient Descent With Momentum

• SVM

  • Support Vector Machine
  • Plot different SVM classifiers in the iris dataset

• Bayesian

  • Multinomial Naive Bayes

• Metrics

  • sklearn.metrics
  • Confusion Matrix
  • Confusion Matrix & Metrics in a minute
  • Sensitivity and Specificity
  • Sensitivity and specificity
  • ROC and AUC
  • balanced accuracy
  • balanced accuracy
  • various metrics from CM

• Data Leakage

  • Data Leakage Basics, with Examples in Scikit-Learn
  • Could machine learning fuel a reproducibility crisis in science?

• Dimensionality Reduction

  • 11 Different Uses of Dimensionality Reduction
  • PCA
  • PCA
  • PCA vs LR
  • pca
  • t-SNE
  • tsne

• Clustering

  • Clustering
  • Hierarchical clustering
  • K-means clustering
  • kmeans
  • From Data to Clusters: When is Your Clustering Good Enough?

• Hyperparameters

  • Model Parameter vs. Hyperparameter
  • Hyperparameter tuning
  • Hyperparameter tuning
  • Optuna
  • optuna
  • Evaluating Hyperparameters in Machine Learning

• Some Topics in Probability

  • p-values
  • how to calculate p-values
  • p-hacking
  • Probability is not Likelihood
  • t-test
  • t-test
  • t-test vs p-value
  • scpipy ttest
  • chi-square
  • 17 Statistical Hypothesis Tests in Python
  • permutation test (AddGBoost + code)
  • The Permutation Test: A Data Scientist’s BFF

• Feature Importances

  • Decision Tree
  • Explainable AI explained! | #4 SHAP
  • SHAP Values Explained

• Semi-Supervised Learning

  • Boosting vs. semi-supervised learning
  • LabelPropagation

• Neural Networks

  • Neural networks
  • Neural networks
  • Tinker With a Neural Network in Your Browser

• Deep Learning

  • Neural Networks with À La Carte Selection of Activation Functions
  • PyTorch
  • PyTorch
  • Double Descent
  • Overparameterization, Backpropagation, Alimentation: Them and Us
  • conv demo
  • convolution
  • A simple image convolution
  • Implementing Image Processing Kernels from scratch using Convolution in Python
  • Introduction to image generation (diffusion)
  • Loss is Boss and other articles in the DL section

• Large Language Models

  • Introduction to large language models
  • Scikit-LLM
  • A Tiny Large Language Model (LLM), Coded, and Hallucinating

• DL and AI

  • Growth of AI computing
  • AI move from Academia
  • Artificial General Intelligence Is Not as Imminent as You Might Think

• Evolutionary Algorithms: Basics

  • Evolutionary Computation
  • How to Build a Genetic Algorithm from Scratch in Python with Just 33 Lines of Code
  • Evolutionary Algorithms, Genetic Programming, and Learning
  • Tiny GA
  • EC-KitY
  • Genetic Programming slides
  • Tiny GP
  • GP Tutorial
  • GP vids
  • GP slides
  • Natural Computing

• Evolutionary Algorithms: Advanced

  • Multi-Objective Optimization
  • Schema theorem
  • Linear GP
  • Cartesian GP
  • Grammatical Evolution
  • Coevolutionary Computation
  • New Pathways in Coevolutionary Computation
  • Novelty search
  • Humies
  • Evolutionary Art
  • Building Activation Functions for Deep Networks
  • Evolutionary Adversarial Attacks on Deep Networks

Resources

Cheat Sheets

• Machine Learning Glossary
• Some Pros and Cons of Basic ML Algorithms, in 2 Minutes
• Cheat Sheets for Machine Learning and Data Science
• The Illustrated Machine Learning Website

Vids

• John Koza Genetic Programming (YouTube)
• גיא כתבי - אלגוריתמים אבולוציוניים (YouTube) [גיא בוגר הקורס שלי: אלגוריתמים אבולוציוניים וחיים מלאכותיים]
• StatQuest with Josh Starmer
• ML YouTube Courses
• Machine Learning Essentials for Biomedical Data Science: Introduction and ML Basics
• Artificial Intelligence Under Fire: Attacking and Defending Deep Neural Networks

Basic Reads

• Genetic and Evolutionary Algorithms and Programming
• Choosing Representation, Mutation, and Crossover in Genetic Algorithms
• Introduction to Evolutionary Computing (course/book slides)
• 26 Top Machine Learning Interview Questions and Answers: Theory Edition
• 10 Popular Machine Learning Algorithms In A Nutshell
• Machine learning preparatory week @PSL
• Neural Networks and Deep Learning (coursera)
• Tinker With a Neural Network in Your Browser
• Common Machine Learning Algorithms for Beginners

Advanced Reads

• What can LLMs never do?
• Foundational Challenges in Assuring Alignment and Safety of Large Language Models
• “Explainability” Is a Poor Band-Aid for Biased AI in Medicine
• Some Techniques To Make Your PyTorch Models Train (Much) Faster
• GPT in 60 Lines of NumPy
• ROC-AUC
• Why video games are essential for inventing artificial intelligence

Books (🡇 means free to download)

• M. Sipper, Evolved to Win, Lulu, 2011 🡇
• M. Sipper, Machine Nature: The Coming Age of Bio-Inspired Computing, McGraw-Hill, New York, 2002
• A.E. Eiben and J.E. Smith, Introduction to Evolutionary Computing, Springer, 1st edition, 2003, Corr. 2nd printing, 2007
• R. Poli, B. Langdon, & N. McPhee, A Field Guide to Genetic Programming, 2008 🡇
• J. Koza, Genetic Programming: On the Programming of Computers by Means of Natural Selection, MIT Press, Cambridge, MA, 1992.
• S. Luke, Essentials of Metaheuristics, 2013 🡇
• A. Geron, Hands On Machine Learning with Scikit Learn and TensorFlow, 2017 🡇
• G. James, D. Witten, T. Hastie, R. Tibshirani, An Introduction to Statistical Learning, 2nd edition, 2021 🡇
• J. VanderPlas, Python Data Science Handbook
• K. Reitz, The Hitchhiker’s Guide to Python
• M. Nielsen, Neural Networks and Deep Learning
• Z. Michalewicz & D.B. Fogel, How to Solve It: Modern Heuristics, 2nd ed. Revised and Extended, 2004
• Z. Michalewicz. Genetic Algorithms + Data Structures = Evolution Programs. Springer-Verlag, Berlin, 3rd edition, 1996
• D. Floreano & C. Mattiussi, Bio-Inspired Artificial Intelligence: Theories, Methods, and Technologies, MIT Press, 2008
• A. Tettamanzi & M. Tomassini, Soft Computing: Integrating Evolutionary, Neural, and Fuzzy Systems, Springer-Verlag, Heidelberg, 2001
• M. Mohri, A. Rostamizadeh, and A. Talwalka, Foundations of Machine Learning, MIT Press, 2012 🡇
• Simon J.D. Prince, Understanding Deep Learning, MIT Press, 2023 🡇

Software

• EC-KitY: Evolutionary Computation Tool Kit in Python with Seamless Machine Learning Integration
• gplearn: Genetic Programming in Python, with a scikit-learn inspired and compatible API
• LEAP: Library for Evolutionary Algorithms in Python
• DEAP: Distributed Evolutionary Algorithms in Python
• Swarm Intelligence in Python (Genetic Algorithm, Particle Swarm Optimization, Simulated Annealing, Ant Colony Algorithm, Immune Algorithm, Artificial Fish Swarm Algorithm in Python)
• Scikit-learn: Machine Learning in Python
• Mlxtend (machine learning extensions)
• PyTorch (deep networks)
• Best-of Machine Learning with Python
• Fundamental concepts of PyTorch through self-contained examples
• Faster Python calculations with Numba

Datasets

• Tabular & cleaned (PMLB)
• By domain
• By application
• Search engine
• Kaggle competitions
• OpenML
• UCI Machine Learning Repository
• Image Databases
• AWS Open Data Registry
• Wikipedia ML Datasets
• The Big Bad NLP Database
• Datasets for Machine Learning and Deep Learning
• Browse State-of-the-Art