Upper limit to cloropleth areas?

[DataStrategist]

(might be related to https://github.com/rstudio/leaflet/issues/246, but perhaps different)

I am trying to render a cloropleth of 1673 areas. The map renders fine locally, but when I deploy it to my website (github -> netlify), it seems to be broken when I use that many areas, BUT NOT when the number of areas is smaller. Take a look at this:

https://www.amitkohli.com/out_manchester/viz/manchester_small.html (<- renders fine, with 30 points) https://www.amitkohli.com/out_manchester/viz/manchester_med.html (<- renders fine, with 300 points) https://www.amitkohli.com/out_manchester/viz/manchester600.html (<- renders fine, with 600 points) https://www.amitkohli.com/out_manchester/viz/manchester.html (<- 1673 points, doesn't render, providing this error in the debug console)

(also tried w/ 1000 points... that's too big too, so it seems the limit is somewhere b/w 600-1000 points)

Details: I'm saving each map using htmlwidgets::saveWidget(map_leaflet, file="x.html",selfcontained = FALSE).

• I've tried selfcontained = TRUE to no avail.
• I've tried using the same folder of "guts" since I know in some cases there's some shenanigans associated with https, but the only difference between these examples posted above is the number of rows passed into the head command.

In case it's useful for anything, here's the code I'm using:

## make map
finished <- WGS84 %>% 
  mutate(col = case_when(
    value == "doublepeak" ~ "red",
    value == "peak" | value == "lastpeak" ~ "orange",
    value == "nothing" ~ "forestgreen",
  )) %>% 
  head(600) %>%
  leaflet() %>%
  addTiles() %>% 
  addPolygons(
    stroke = FALSE, # remove polygon borders
    fillColor = ~col,
    # fillColor = ~pal(val), # set fill color with function from above and value
    fillOpacity = 0.6, color = "white", smoothFactor = 0.5, # make it nicer
    popup = ~popupImage(fn)  ## from the library leafpop
    # popup = popupImage(images)
    # popup = p_popup
    )  


# ## add legend
# finished <- finished %>% 
#   addLegend(pal = pal, values = ~value, opacity = 0.7, title = NULL,
#             position = "bottomright")

## add title and explanation
rr <- tags$div(
  HTML(paste0('<h1> Growth trend in COVID19 cases in Manchester</h1>
        <h2> As of ', Sys.Date(), '</h2>', 
       '<p>The colors represent new cases in the past 2 weeks: Green means "no new cases", 
       orange means "new cases either last week or this week" and red means "new cases both 
       last week and this week"</p>
       <p>Clicking on a LSOA will bring up the cumulative count, the number of new
       cases this week, and will show the curve of cumulative cases to date (each point represents a week)</p>'
)))


## Combine the two
map_leaflet <- finished %>%
  addControl(rr, position = "bottomleft")

saveWidget(map_leaflet, file="manchester600.html",selfcontained = FALSE)

I'm using the latest dev version of leaflet (from today).

Is there anything I can do to show all the points on one map?

[jcheng5]

The HTML file is being truncated somewhere between your local machine (since it works fine when you load it locally) and when it ends up at https://www.amitkohli.com/out_manchester/viz/manchester.html. I'll bet if you look at the file sizes, they're different; what I just downloaded from that URL is 41226240 bytes, what is it locally for you?

40MB is a huge amount of data for a choropleth of that size, have you tried polygon simplification? http://rstudio.github.io/leaflet/shapes.html#simplifying-complex-polygonspolylines

[DataStrategist]

You are right Joe, the files ARE being truncated!

On my local it's 47,426 KB On github it's 46,3xx KB on netlify it's 41,226 KB

That's crazy business! I readded the file from the console to see all the bits and bobs to try to explain file-size-change b/w local and GH:

Could it be the delta compression? And then b/w GH and netlify:

11:46:02 AM: Build ready to start
11:46:05 AM: build-image version: 8e315e54bc4032a32e73290be556cde4f8348c12
11:46:05 AM: build-image tag: v2.8.2
11:46:05 AM: buildbot version: 45cd00068410d70db72c9672cdd014995488e12a
11:46:05 AM: Fetching cached dependencies
11:46:06 AM: Starting to download cache of 727.8MB
11:46:19 AM: Finished downloading cache in 13.544706791s
11:46:19 AM: Starting to extract cache
11:46:31 AM: Finished extracting cache in 12.190903626s
11:46:31 AM: Finished fetching cache in 25.910612505s
11:46:31 AM: Starting to prepare the repo for build
11:46:32 AM: Preparing Git Reference refs/heads/master
11:46:44 AM: Starting build script
11:46:44 AM: Installing dependencies
11:46:44 AM: Started restoring cached node version
11:46:47 AM: Finished restoring cached node version
11:46:48 AM: v8.17.0 is already installed.
11:46:48 AM: Now using node v8.17.0 (npm v6.13.4)
11:46:48 AM: Attempting ruby version 2.3.6, read from environment
11:46:50 AM: Using ruby version 2.3.6
11:46:50 AM: Using PHP version 5.6
11:46:50 AM: Installing Hugo 0.42
11:46:50 AM: Hugo Static Site Generator v0.42 linux/amd64 BuildDate: 2018-06-12T06:37:22Z
11:46:50 AM: Started restoring cached go cache
11:46:51 AM: Finished restoring cached go cache
11:46:51 AM: Installing Go version 1.10
11:46:51 AM: unset GOOS;
11:46:51 AM: unset GOARCH;
11:46:51 AM: export GOROOT='/opt/buildhome/.gimme_cache/versions/go1.10.linux.amd64';
11:46:51 AM: export PATH="/opt/buildhome/.gimme_cache/versions/go1.10.linux.amd64/bin:${PATH}";
11:46:51 AM: go version >&2;
11:46:51 AM: export GIMME_ENV='/opt/buildhome/.gimme_cache/env/go1.10.linux.amd64.env';
11:46:51 AM: go version go1.10 linux/amd64
11:46:51 AM: Installing missing commands
11:46:51 AM: Verify run directory
11:46:51 AM: Executing user command: hugo
11:46:52 AM: Building sites …
11:46:53 AM: 

11:46:53 AM:                    |  EN
11:46:53 AM: +------------------+------+
11:46:53 AM: Pages            |  423
11:46:53 AM:   Paginator pages  |   32
11:46:53 AM:   Non-page files   | 1039
11:46:53 AM:   Static files     |   50
11:46:53 AM:   Processed images |    0
11:46:53 AM: Aliases          |  181
11:46:53 AM:   Sitemaps
11:46:53 AM: |    1
11:46:53 AM:   Cleaned          |
11:46:53 AM:    0
11:46:53 AM: Total in 1811 ms
11:46:53 AM: Skipping functions preparation step: no functions directory set
11:46:53 AM: Caching artifacts
11:46:53 AM: Started saving pip cache
11:46:53 AM: Finished saving pip cache
11:46:53 AM: Started saving emacs cask dependencies
11:46:53 AM: Finished saving emacs cask dependencies
11:46:53 AM: Started saving maven dependencies
11:46:53 AM: Finished saving maven dependencies
11:46:53 AM: Started saving boot dependencies
11:46:53 AM: Finished saving boot dependencies
11:46:54 AM: Started saving go dependencies
11:46:55 AM: Finished saving go dependencies
11:46:55 AM: Build script success
11:46:55 AM: Starting to deploy site from 'public'
11:46:56 AM: Creating deploy tree asynchronously
11:46:56 AM: Creating deploy upload records
11:46:58 AM: 1 new files to upload
11:46:58 AM: 0 new functions to upload
11:47:01 AM: Starting post processing
11:47:02 AM: Post processing - HTML
11:47:12 AM: Minifying css bundle
11:47:13 AM: Minifying css bundle
11:47:16 AM: Post processing - header rules
11:47:16 AM: Post processing - redirect rules
11:47:16 AM: Post processing done
11:47:16 AM: Site is live
11:48:09 AM: Finished processing build request in 2m3.378211949s

I see the CSS being minified, but nothing that would explain the maaaassive truncation!

Anyway, in terms of polygon simplification:

> simplified <- rmapshaper::ms_simplify(WGS84)
> ## Check size and simplify:
> object.size(WGS84)
22032824 bytes
> object.size(simplified)
2865856 bytes

I would say that worked! 10x improvement! Indeed, the final map is only 6MB. That's smaller than even the 300 points one! The link now works! https://www.amitkohli.com/out_manchester/viz/manchester.html

(and by the way, this is a map tracking new COVID cases and it WILL be used by people in the government, so feel extra good tonight).

This is solved! Thanks! Not closing in case you want to comment on the github-netlify disparity, but feel free to close!