proposals
proposals copied to clipboard
carpentries-incubator
How The Web Works
- What is the topic of your lesson or lesson proposal?
"How The Web Works" - many researchers at some point have to start making use of HTTP-based APIs, and unless they are very technically-minded then this involves a huge dump of concepts all at once. It would be good to have a gentle introduction to the Web to get to the point that learners can interact with Web APIs confidently.
Concepts that are important to understand to be able to do this (this is opinion, and relatively off the top of my head):
- What is a socket and a port? (Usually HTTP is on port 80, HTTPS on port 443; some information on what a port is is needed to understand that)
- What is HTTP? What are requests, responses, headers? Some discussion of statelessness would be important here.
- How can you authenticate to an HTTP server? E.g. with authentication tokens, or API keys, or usernames and passwords.
- Testing the above with curl
- How to make HTTP requests from e.g. Python/R?
- What are JSON and XML? How do you parse them in e.g. Python/R?
- Since this is talking about the Web, a little on HTML (especially as you'll frequently see HTML in responses if you make an invalid request to an API), possibly touching on how AJAX adds interactivity
Things that, while interesting and related, would probably be out of scope:
- Writing your own HTTP API, running a server
- Web scraping
- Do you already have a draft of your lesson? You're welcome to share materials at any stage of development. If you already have drafted materials, please include a link.
Nothing yet. We're considering writing something if we don't find any existing material that achieves the above goals.
Thank you for submitting this lesson idea @edbennett. It sounds like a very good concept for a lesson, and certainly one I would have found useful in previous years. It would also be great to have a resource like this to point people to from the introduction to our lesson on Jekyll & GitHub Pages, which sacrifices a great deal of detail and accuracy in favour of brevity in its description of how a webpage is loaded.
It sounds like you are only testing the water with this idea right now? But, if and when you would like me to create a repository for you to begin working on this lesson, please post back here and I will be happy to help.
Useful resource for getting started working with APIs in Python: https://www.dataquest.io/blog/python-api-tutorial/
This all looks good! I'm not sure I'd concentrate too much on point 1 tho actually - do people really need to understand sockets to be able to consume web-services? (I'm not sure I'd be able to describe them well!).
I'm wondering if an explanation of what an IP address is that the internet is packet switched is worth including or is that un-necessary detail?
I think this would be a great addition as a lesson. I'm actually getting ready to teach on this topic today (in thirty minutes!) for a text analysis course and have taught it a few different ways; I would be happy to contribute/collaborate.
FWIW, I've found this set of articles really useful as introductory pieces (although they are a bit intimidating for the absolute beginner): An illustrated introduction to APIs API Whispering 101
I'm glad to see there's interest in this!
@wood-chris: You're right; I think at least a superficial understanding of what a port is is useful, but you probably don't need to use the word socket to explain that.
@colinsauze: One point I had intended to include on the list and that slipped my mind was the distinction between a domain name and an IP address and how they relate to each other (with or without mentioning DNS). I'm not sure that discussion fo packet switching is needed though.
@mfall3, @burrowsdt Thanks for the links to resources; I'll have a read through them.
We've also been talking about an "API's for scientists" course at WHOI. I was thinking of starting with reading csvs from urls with dplyr::read_csv() in R or pandas.read_csv() in python, moving through requests/httr for requesting and receiving data, JSON, and culminating with things like ERDDAP and/or using server requests to generate files for download. This is a lot, so a subset is probably reasonable to start with. Happy to help/contribute once your lesson starts taking shape!
@blongworth this sounds good! As a former oceanographer, I'm always interested in working out how to help environmental scientists in general - some of whom might be pretty mathematical and have no issues with Fortran / C / Matlab, but don't realise, without some prompting, how higher level libraries might help them and make their lives easier. (I also used to collaborate with some of the data folk at WHOI!)
@edbennett, do you have a timeline in mind for lesson development? My group at WHOI is considering a half day workshop on API's sometime before the end of the year.
The lesson already got developed and ran once, https://github.com/edbennett/web-novice
Thanks for signposting the link to the lesson Colin. The material as exists was developed quite rapidly (given time constraints of needing to run it on a specific date), and for a specific audience, so couldn't include everything discussed here, and what is there still has some scope for improvement. We've made some issues highlighting areas that would benefit from work.