Chris Muir
Chris Muir
One question: Is it important that the deidentified string be identical for identical raw string values? Like if "cats" appears 12 times, should the strings post-anonymization be identical? i.e. all...
I feel like removing proper names is a good idea. Even though all of this data is public, it feels weird to leave in people's names and make it all...
@soodoku I got started on this, I've written a script that will get all of the district links for each year, and then for each set of district links it...
Yeah I have calls to `Sys.sleep` between each request. The robots.txt file doesn't mention `/ftf/`....I guess it's fine, as long as we go easy on them. It'll take some time...
So I worked on this some today, quick update....I let the script run overnight to scrape all of the teacher links from each of the district links. There's a total...
Cool, yeah I'm letting it run on the 2012 teacher links for now. Once that's done, I'll write those results to csv and upload to the repo. We can take...
Just pushed the 2012 IL teacher salaries to the repo. The data came out very clean from the website, there were over 162K records scraped and every single one returned...
No unfortunately this isn't done, I'm slowly working through each year. The number of records per year is around 160k, and each record requires a single request to the website,...
Cool, got it. I'll keep adding data to the repo as each year finishes.
Quick update, the site has been completely down for the last ~48 hours. No "site maintenance" screen or anything, just a blank white page. I'll keep checking it.