censusreporter Embedding returns 404

Hello,

I've tested this multiple times, and am a bit confused about the chart embedding functionality. When embedding a chart iframe for a Census Tract that I have already visited via the Census Reporter website, there is no issue - however, if the chart hasn't had 'embed' clicked via the website a 404 error is returned on a query for the chart's raw data.

For example, the following works (I have visited it on Census Reporter): <iframe id="cr-embed-14000US29149480200-demographics-age-distribution_by_decade-total" class="census-reporter-embed" src="https://s3.amazonaws.com/embed.censusreporter.org/1.0/iframe.html?geoID=14000US29149480200&chartDataID=demographics-age-distribution_by_decade-total&dataYear=2016&releaseID=ACS_2016_5-year&chartType=histogram&chartHeight=200&chartQualifier=&chartTitle=Population+by+age+range&initialSort=&statType=scaled-percentage" frameborder="0" width="100%" height="300" style="margin: 1em; max-width: 720px;"></iframe>

YET, this does not (I haven't visited it via Census Reporter):

This issue also appears to be affecting the "Embed Charts" documentation: https://censusreporter.org/examples/embed-charts/

Jan 20 '18 00:01 jacksonvoelkel

Hi, this is a great question! Yep, you are correct that programatically (or arbitrarily) created embed charts won't work if no one has ever clicked "Embed" for that chart on Census Reporter. That's expected behavior based on the way we've built the embed system.

When someone embeds one of our charts, we really don't have a way to know how high traffic might be on the page where it lands. We host all the js, html, and data for those embeds, though, so we've taken some steps to protect our bandwidth. One of those steps is storing chart data in tiny individual JSON files, so embeds never touch our database (or even the larger JSON files underneath our profile pages).

Here's an example of one of those JSON files for a chart embed. Nearly all these files are under 1kb, so naturally that's SUPER helpful in terms of our bandwidth, and also for performance on the pages where charts are embedded.

We don't really want to create every possible piece of chart data for every possible geography in advance, though—that would be an incredible number of files to store. And then multiply that number of files by the number of ACS releases we've supported (nine at this point), because we lock chart data by release so people's embeds don't unexpectedly change numbers in place.

So our solution is to create that JSON fragment the first time someone clicks "Embed" on any chart on Census Reporter. That click fires off a task that checks our S3 bucket to see if the data file exists, and if not, generates it and saves it for anyone else who comes along. If someone grabs embed code and changes something like the geoID variable in the querystring (which looks like what you're trying here), the chart's javascript expects to find the corresponding JSON file for that place/chart data in our S3 bucket. Which, unless another person just happens to have clicked "Embed" on that chart at some previous point, won't exist and will end up throwing a 404.

Sorry about the lengthy description here, but I wanted to explain our thinking behind building the embed system this way. I definitely understand the approach you're taking to modify embed code in place—I've done plenty of things just like that myself! Sorry it won't work in this case.

Jan 20 '18 01:01 ryanpitts

Thank you for the detailed answer!

Do you have any suggestions for creating this programatically, so as to achieve the functionality described in "Can I automate these embeds?" as described at https://censusreporter.org/examples/embed-charts/?

Thanks, Jackson

On Fri, Jan 19, 2018 at 5:08 PM, Ryan Pitts [email protected] wrote:

Hi, this is a great question! Yep, you are correct that programatically (or arbitrarily) created embed charts won't work if no one has ever clicked "Embed" for that chart on Census Reporter. That's expected behavior based on the way we've built the embed system.

When someone embeds one of our charts, we really don't have a way to know how high traffic might be on the page where it lands. We host all the js, html, and data for those embeds, though, so we've taken some steps to protect our bandwidth. One of those steps is storing chart data in tiny individual JSON files, so embeds never touch our database (or even the larger JSON files underneath our profile pages).

Here's an example of one of those JSON files for a chart embed http://embed.censusreporter.org.s3.amazonaws.com/1.0/data/charts/ACS_2016_1-year/04000US06-economics-poverty-children.json. Nearly all these files are under 1kb, so naturally that's SUPER helpful in terms of our bandwidth, and also for performance on the pages where charts are embedded.

We don't really want to create every possible piece of chart data for every possible geography in advance, though—that would be an incredible number of files to store. And then multiply that number of files by the number of ACS releases we've supported (nine at this point), because we lock chart data by release so people's embeds don't unexpectedly change numbers in place.

So our solution is to create that JSON fragment the first time someone clicks "Embed" on any chart on Census Reporter. That click fires off a task that checks our S3 bucket to see if the data file exists, and if not, generates it and saves it for anyone else who comes along. If someone grabs embed code and changes something like the geoID variable in the querystring (which looks like what you're trying here), the chart's javascript expects to find the corresponding JSON file for that place/chart data in our S3 bucket. Which, unless another person just happens to have clicked "Embed" on that chart at some previous point, won't exist and will end up throwing a 404.

Sorry about the lengthy description here, but I wanted to explain our thinking behind building the embed system this way. I definitely understand the approach you're taking to modify embed code in place—I've done plenty of things just like that myself! Sorry it won't work in this case.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/censusreporter/censusreporter/issues/231#issuecomment-359131492, or mute the thread https://github.com/notifications/unsubscribe-auth/AJsLsxatiTFPE-1xXJqorJVqNvO2-KFpks5tMTx1gaJpZM4RlOCX .

Jan 20 '18 01:01 jacksonvoelkel

Unfortunately, that piece of documentation is outdated and I need to change it. Our original embed strategy worked just as described there, until we had to refactor it after stumbling into the bandwidth- and release-related issues I hinted at above. (That refactor broke the embed chart examples on that page as well—you'll notice that the querystring now include some additional variables like dataYear and releaseID that aren't present in that outdated example.)

My HUGE apologies for having misleading documentation there on that aspect of embed charts.

I think it might be possible to make a script that would load Census Reporter pages based on a set of geoIDs, and fake a click on the "Embed" link of the chart you care about. But it actually does require that click. Here's the bit of code that triggers the task that creates the JSON file—it relies on the fact that each chart instance knows the data object it needs to create itself, and uses that to populate the static data file.

Jan 20 '18 01:01 ryanpitts

Dang, what a shame!! I'll try to figure something out... I definitely can't click 'embed' on 5 different charts for every tract in the State of Oregon. I appreciate your time. If you think of anything that might work, please let me know!

Jackson

On Fri, Jan 19, 2018 at 5:31 PM, Ryan Pitts [email protected] wrote:

Unfortunately, that piece of documentation is outdated and I need to change it. Our original embed strategy worked just as described there, until we had to refactor it after stumbling into the bandwidth- and release-related issues I hinted at above. (That refactor broke the embed chart examples on that page as well—you'll notice that the querystring now include some additional variables like dataYear and releaseID that aren't present in that outdated example.)

My HUGE apologies for having misleading documentation there on that aspect of embed charts.

I think it might be possible to make a script that would load Census Reporter pages based on a set of geoIDs, and fake a click on the "Embed" link of the chart you care about. But it actually does require that click. Here's the bit of code https://github.com/censusreporter/censusreporter/blob/master/censusreporter/apps/census/static/js/charts.js#L780-L785 that triggers the task that creates the JSON file—it relies on the fact that each chart instance knows the data object it needs to create itself, and uses that to populate the static data file.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/censusreporter/censusreporter/issues/231#issuecomment-359133765, or mute the thread https://github.com/notifications/unsubscribe-auth/AJsLs95VzHfoz1aQcEOyylm0JtTQTMVBks5tMUH0gaJpZM4RlOCX .

Jan 20 '18 01:01 jacksonvoelkel

hmm. Out of curiosity, what's the use case you're wanting to satisfy?

Jan 22 '18 17:01 ryanpitts

I'm building a map of Oregon, showing conservation areas. The conservative group would like to see Census tracts and ZIPs (a toggleable layer) and demographic information. I would like to arrange it so that when a user clicks on a tract/ZIP, a sidebar with a few charts is displayed. When I mentioned to a friend that I would be doing this with chart.js, they mentioned that Census Reporter already had embedable charts!

On Jan 22, 2018, at 9:19 AM, Ryan Pitts [email protected] wrote:

hmm. Out of curiosity, what's the use case you're wanting to satisfy?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

Jan 22 '18 17:01 jacksonvoelkel

censusreporter censusreporter copied to clipboard

Embedding returns 404

censusreporter
censusreporter copied to clipboard