notes icon indicating copy to clipboard operation
notes copied to clipboard

Published 20 hours ago • verified publisher icon jph00

Full stack dev: topics to teach

Open jph00 opened this issue 4 years ago • 20 comments

This is the early planning of a new course "uncool full stack development" or something similar. Basically, I want to show people how to use simple broad-purpose tools, rather than SAAS and complex stuff like docker. So we'll be looking at cron, bash, perl, simple VPS servers, etc.

Why? Because: it's cheaper (use a single cheap server to do loads of stuff, rather than signing up for heaps of SAAS that do one little thing); you learn the foundations of everything else, which is knowledge that never goes out of date; rather than learning lots of domain specific languages, config files, and APIs, for every different service, you learn a small number of powerful foundations you can re-use everywhere; when you're working with SAAS or higher-level frameworks, you can debug your problems because you know what's happening underneath; lots of interviews ask about lower-level foundations, so it'll help you get recruited; for building a startup or hobby project, it's far cheaper, easier, and more scalable when you're in control of what you're doing and you understand how it all works.

Contents (links to sections)

Other similar syllabi:

  • Startup Engineering by Balaji S. Srinivasan and Vijay Pande; notes on first few lectures
  • CS50 and CS50W
  • https://wizardzines.com (we can use these in the course, if we link to the relevant page on wizardzines.com for any of the images we show in the video. For example https://wizardzines.com/comics/bash-tricks/)

jph00 avatar Sep 20 '20 15:09 jph00

CLI tools

  • perl: -p -i -n -e -l -a
  • git / gh
  • apt / apt-fast
  • sudo
  • ack
  • cron
  • ssh & tunneling
  • shell
    • scripts / #!
    • pipes
    • variables, variable substitution; !! , !$
    • pushd / popd / cd -
    • bashrc
    • completions
    • subshells; source
  • npm (global vs local)
  • jq (e.g automate pypi/github api)
  • parallel (e.g git_pull_all)
  • rsync (local; ssh; delete)
  • tmux (as nohup; panes)
  • vnc

jph00 avatar Sep 20 '20 16:09 jph00

Concepts

  • Public/private keys (ssh; https; dkim)
  • IPs/subnets/ports (and secgroups)
  • DNS / coredns / dig / PTR
    • Secondary: zoneedit / puck.nether.net / buddydns
  • fork / prefork / async
  • reverse proxy / fastcgi / wsgi / asgi
  • Env vars / PATH
  • VPNs / wireguard

jph00 avatar Sep 20 '20 16:09 jph00

Protocols/foundations

  • TCP / UDP / unix sockets
  • HTTP
  • SMTP
  • SSL / letsencrypt
  • Web scraping (selenium etc)
  • supervisor/systemd
  • Package management (conda, apt, pip)
  • Documentation
  • openstack / aws
  • Cloud block storage
  • CRUD
  • Rest
  • Authentication
  • Server setup, security, and firewalls
  • Deployment: standalone; web-based
  • Automation

jph00 avatar Sep 20 '20 16:09 jph00

Programming tools

  • Python
  • regex
    • Use to find stuff in web page, e.g cheap TLDs at https://www.sav.com/domains, using Chrome regex search
  • glob
  • HTTP/ASGI servers, cookies (same-site)
  • HTML
  • CSS: grid, flexbox, ?tailwind
  • JS / Typescript: modern JS without libs
  • Editors: vscode; vim; emmet
  • ?golang ?julia (implement protocols e.g POP3, SMTP, DNS)
  • sqlite (other DBs too?)
  • SQL
  • Debugging and debuggers
  • Profiling
  • Testing (unit / integration / continuous)
  • REPLs and notebooks
  • Chrome dev tools
  • Simple queues

jph00 avatar Sep 20 '20 16:09 jph00

Basic services

  • Stripe
  • Mail delivery (AWS SES; elasticemail; sendpulse); mail-tester; Postmark DMARC; DKIM validator
  • VPS (DreamCompute; OVH; EthernetServers; Vultr; Genesis Hosting; Kamatera; ssdnodes; AWS LightSail; DigitalOcean; vpsdime)
    • VPS Benchmarks and LowEndBox
    • Openstack / AWS CLI / etc
    • snapshots
    • load balancing, DNS round robin (NB: CloudFlare with proxy effectively low TTL), host redirect
  • Cloudflare
  • GitHub: Actions; Codespaces; Issues/PRs; Releases
  • Building your own server

jph00 avatar Sep 20 '20 16:09 jph00

Things to implement

  • Best is projects in your subject matter domain
    • Or reach out to subject matter experts and offer to help them (e.g. data journalists; non-profits; clubs...)
    • Make sure if you build something for someone else, you're willing to maintain it as long as it's needed!
  • zapier stuff
  • fut.io
  • guerrillamail.com; develmail.com
  • status monitor
    • email; dns; RAM; uptime; df
  • AWS services?
  • wireguard VPN (access your home PC from anywhere)
  • pinboard.in
  • pagerduty
  • google analytics
  • RSS reader
  • twitter thread
  • basic web / tcp server
  • turn a notebook into a web app

jph00 avatar Sep 20 '20 16:09 jph00

Startup skills

  • MVP (smallest complete thing people would love)
  • Product/market fit
  • What's it like to be in a startup
  • Iterations
  • Branding (incl Google search; Twitter cards; Analytics; Rachel's post)
  • Support
  • Book-keeping
  • Build vs buy
  • Scaling
  • Hiring
  • Funding
  • Doing good, and avoiding harm (including to yourself, your staff, your users, society...)
  • Creating a company (when/why?)
  • Competitive analysis
  • Pricing
  • Things to not waste time on

jph00 avatar Sep 20 '20 16:09 jph00

Little VPS projects

Full list

  • VPN
  • Web crawling
  • Domain/service pings...
  • Find deals on #lowendbox
  • "a DokuWiki for keeping my thoughts organised. Because it has permission access control, I can make user accounts for friends/families/lab mates for specific categories so we can share data"
  • https://expiring.link/ (made in a weekend)
  • cleaver.cloud imitation
  • https://coronavirus.dunderdata.com/
  • backup storage
  • web ebook library
  • Minecraft server
  • video transcoder
  • personal SVOD (“Netflix”)
  • intrusion detection
  • video streaming + chat room
  • web server for staging/testing
  • Apache Superset dashboards
  • "a cron that sends me a reminder email about work emails that I should've responded to - but didn't" ("created model that "detect" emails I should care about (Uvicorn server hosted on AWS); created a Node app based outlook add-in that checks with the model and saves the email id/subject; node-cron send a reminder every other day")
  • pi.hole
  • receive/act on stats from home automation sensors, notifies if batteries run low, plants need to be watered...
  • "bots (Go and Ruby) are used to provide personal notification on Telegram and Discord, to quickly perform simple trades via API on crypto exchanges and to provide services (like price queries etc.) to crypto communities"
  • home automation and video server; object detection on video feeds to minimise false alerts
  • nextcloud
  • http://notify.run
  • "A Twitter bot that runs people's tweets on a 1980s home computer emulation and posts a video reply"
  • "https://graphly.snrrrub.com to track npm dependencies and associated open source licenses"
  • "https://nomly.snrrrub.com as a startup name generator (also a great drinking game)"
  • "I host http://aiquizzes.com on a $5 per month droplet"
  • "Habit tracking: Update streaks of habits on @todoist"
  • "Investing: Track currently open IPOs and add to to-do list on last day of deadline."
  • "Use @rescuetime API to send daily % change in time spent on social media"
  • "Fast Reverse Proxy so that I can tunnel into any of my machines"
  • "I have a server with a cron job that sends a small group of family and friends a daily cute animal picture. Uses PRAW to scrape the top entries of http://reddit.com/r/aww for the source image, downloads it, and then uses gmail API to send the email."
  • curl http://conda.ml gives you an install script for the latest Anaconda (Linux & Mac)
  • "Jenkins job that auto compiles a selection of papers/drafts in GitHub repo with papers and posts them to slack"
  • "http://RemoteOK.io is a single PHP file called "index.php" generating $65,651 this month. No frameworks. No libraries (except jQuery)"
  • "post a piece of content on your own server with your own URL. You do not have to host it externally."
  • proxy server to anonymize location information including VPN
  • TOR remote connection
  • "a cron job that checks for arXiv paper updates, downloads them to my Dropbox folder, and emails me so I remember to read the update"

jph00 avatar Sep 22 '20 13:09 jph00

potential concepts to cover

  • how to copy/paste in a terminal
  • if I write a short script / program / little tool and want it available anywhere from the command line, where do I put it? (this falls under PATH but would be great to explicitly discuss this scenario)
    • do such scripts need to be written in bash or can I use something else, for instance Python?
    • how do I run something when my machine starts? (/etc/fstab for mounting, where do I put one off scripts I'd like run, understanding run levels and troubleshooting when the boot sequence doesn't complete)
  • how can I tell what's running on my system?
  • what are PIDs? how are they useful?
  • how is LOCALE important? how do I change keyboard layout (for typing in another language) without changing locale (setxkbmap)
  • mapping caps to ctrl - great for ergonomics, any other key mappings that are helpful?
  • how can I create an IOT device / remote sensor, what platform do I use?
  • if I create an API and expose it, can I let only authenticated users use it? how? how do I allow only my application to use the api (for instance, to communicate with my mobile app)
  • fail2ban
  • what is OpenStack, how can it be of use to us
  • in the context of a shell, what are the differences between interactive, non-interactive, login, non-login flavors and how do you arrive at each?
  • what is tty?
  • many technological stacks have interprocess communication, what it is and in what situations it is useful? what is the simplest way to implement it?
  • how do PPAs work? how do you add one? can you trust a PPA?
  • using a password manager, sharing the encrypted db within a team to facilitate work, what do you do with passwords when someone leaves the team?
  • file permissions, ownership
  • symlinking (hard, soft)
  • how do you decide what is worth to work on? how do you share work in a small team? is a situation where you have 3 equal parties even feasible or should the model always be one lead facilitating collaboration
  • ctrl-r to search for command in history
  • file permissions, file ownership
  • minimal css reset 
* {
  margin: 0;
  padding: 0;
  box-sizing: border-box;
}

section, article, aside, footer, header, nav {
  display: block;
}

radekosmulski avatar Sep 23 '20 17:09 radekosmulski

Hi @jph00

Stumbled upon this issue and glad you're taking a look at Julia.

Here are some resources that might be helpful in developing content:

In particular, the above blogposts describe Julia's philosophy behind its unique facilities for running external commands, and the issues with python, perl and bash that motivated it.

Also some fast and composable text processing libraries:

  • https://github.com/gkappler/CombinedParsers.jl
  • https://github.com/BioJulia/Automa.jl

These are in addition to Julia's native regex capabilities.

Also, I know some individuals who are using Julia for web, sysadmin and general scripting/automation tasks and might be able to help with developing a potential Julia side of the course, I'll reach out to them.

Edit: Some web scraping libraries:

  • https://github.com/bicycle1885/EzXML.jl
  • https://github.com/JuliaWeb/Gumbo.jl
  • https://github.com/Algocircle/Cascadia.jl
  • https://github.com/Nosferican/WebDriver.jl

AriMKatz avatar Oct 01 '20 19:10 AriMKatz

@jph00 This repo Link has a comprehensive list and could give you some information regarding various self-hosted (alternatives) tools available.

Gokkulnath avatar Oct 03 '20 12:10 Gokkulnath

One thing that I always struggle with is the topic of asynchronous programming, and dealing with promises and/or Futures.

I never feel like I truly understand how to use it correctly. I also have some kind of frustration whenever I'm forced to use it, because it seems to me that there is a fair amount of boilerplate required (but perhaps I just don't know the best way to do things). Just wanted to throw that out there

hamelsmu avatar Oct 07 '20 04:10 hamelsmu

One thing that I always struggle with is the topic of asynchronous programming, and dealing with promises and/or Futures.

I never feel like I truly understand how to use it correctly. I also have some kind of frustration whenever I'm forced to use it, because it seems to me that there is a fair amount of boilerplate required (but perhaps I just don't know the best way to do things). Just wanted to throw that out there

In what situations have you felt forced to use it, specifically?

jph00 avatar Oct 07 '20 16:10 jph00

In what situations have you felt forced to use it, specifically?

Thanks for asking :). Situations where I have felt forced to use it:

  • Using javascript clients or libraries that have async functions in them for example octokit.rest.js.
  • When I'm using an API of some kind that takes a long time to return results, like machine learning as a service APIs. Sometimes I want to make many of these requests but don't want to block on their responses and not necessarily to use separate threads for that ( although thats what I often do out of laziness and because that is what I can more clearly reason about in my head).

P.S. one beautiful thing about fastai is allowed me to really understand the concept of callbacks more deeply, which are often entangled or a required way of thinking in async programming (at least with the programming languages and libraries I have tried to use).

hamelsmu avatar Oct 07 '20 17:10 hamelsmu

This is a great topic ! I'm really keen on following your new course (or following how it is built). Can I ask where the repository of it ? Thank you for always helping people learning interesting things :)

dienhoa avatar Oct 16 '20 07:10 dienhoa

Wow this is a fantastic endeavor! I have always wished that Startup Engineering could still be accessed. Here is another class you can check out - MIT's 'Missing Semester'

gojira avatar Oct 24 '20 22:10 gojira

I came to also point to https://missing.csail.mit.edu/

tcapelle avatar Oct 26 '20 09:10 tcapelle

@seem @radekosmulski we should start teaching this course. Between us we have lots of stuff we have done already here.

hamelsmu avatar Feb 24 '23 22:02 hamelsmu

@seeM @radekosmulski we should start teaching this course. Between us we have lots of stuff we have done already here.

Yes! I'm very excited about all of this stuff!

seeM avatar Feb 25 '23 07:02 seeM

Love that you're picking this up!

I'd love to see coverage on app deployment / development. I'm thinking of stuff like the folloiwing that I'm having to do / learn about more and more without any real background.

  • TypeScript / node frameworks - Express etc.
  • Modify somebody's web app to what I need
  • Deploy on Vercel / Netlify

If somebody could distill the most valuable information about getting going with these in typical fastai fashion, I think it would be really valuable vs. spending 10 hours on some bootcamp with low information density.

gojira avatar Feb 25 '23 19:02 gojira