peterportal-public-api icon indicating copy to clipboard operation
peterportal-public-api copied to clipboard

Dump old WebSOC data into sqlite db

Open ramanxg opened this issue 3 years ago • 2 comments

In reference to this comment, we could put old websoc data into the sqlite so that the query time is faster. This should prevent us from having to make a new request to websoc for data that will not change.

We can start by doing this with the course offerings from 2019-20 and earlier. It might be helpful to also keep track of query times so we know how much our api is improving for speed. These requests might not occur very often, but in the case that they do, we won't have to make more requests to websoc.

Todo

  • Request and dump the old data into a csv file. We can do this with our own api.
  • Create a table in our sqlite with appropriate fields and put csv file data into it.
  • Restructure both rest endpoints and graphql so that for any requests for older years, the sqlite file is queried instead of the websoc-api.
  • Test and compare speeds

ramanxg avatar Apr 19 '21 00:04 ramanxg