Improve map performance over long date ranges (month+)

[nichhk]

Overview

Currently, the frontend will get up to 10k 311 requests/day for each day in the selected date range. If we select the past month, our server does not even work reliably--our frontend makes ~30 requests in parallel, one for each day, and 5-6 requests fail on the server side.

This is to say nothing of a browser's ability to render 100k+ requests in the first place. We'll need to measure that once we get the map to work reliably over a month's worth of request data.

Action Items

• [ ] We are not caching the results for these API calls. We can try to implement caching by adding a cached decorator on get_filtered_requests
• [ ] We can try limiting the max number of requests that we pull for each day; currently, we limit it to 10k, but there are around 4k requests per day, on average. We could try to set the limit to something like 1k.
• [ ] We could throw more resources at the server--more threads, or other things
• [ ] Look into server code to see if anything is particularly slow there

Resources

• https://github.com/hackforla/311-data/issues/1396

[nichhk]

@EchoProject what do you think about the first checkbox in the comment above? How important is it to render all the requests for a particular date range in the map?

[nichhk]

Another idea is to retrieve a limited number of requests on a city-level, but get all NC-level requests when a user selects an NC.

[edwinjue]

Here's some reading material as promised from today's meeting.

To improve map performance, in theory, we could combine webworkers with mapbox's real-time data-join capability. We could try some branches in dev, collaborate and see if it is a solution that ultimately suits our needs. It's pretty neat to learn about nonetheless

• Mapbox Real-time mapping
• How to run javascript function in "background" / without freezing UI
• Using Web Workers
• RxJS

[nichhk]

@hqasmei, I'd encourage you to read Edwin's post above and see if any of those ideas seem useful in improving the map's load time. But for now, this is a good way to improve the user experience: #1408. Let me know if you have any questions!

[mc759]

Hey @edwinjue and @hqasmei Do you have an update for us on this issue?

Please update:

• Progress:
• Blockers:
• Availability:
• ETA:

Thanks!

[hqasmei]

Edwin correct me if I'm wrong

Progress: Hasn't been worked on yet, because first wanted to fix the #1408 issue first. Personally my option the code should be refactored to composition rather than inheritance to be able to handle this easier. Not sure where this stands in priority, but probably would take longer than #1408 issue

Blockers: Availability, not knowing the priority on this, and discussions on frontend infrastructure change for the Map.jsx and any other file to handle the requests.

Availability: a couple hours a week (~6 hours), evenings and weekends

ETA: Not sure

[hqasmei]

Had a meeting with Eric on 12/30/2022. Here are some of my notes:

Problem

• The way the code is architected, has to wait until it completes getting the data before displaying the data

Potential Solution One (quickest):

• Have the initial loading screen only display one days worth of data
• Fast and needs less time - Issues: Still will take a while to load if you change the data range to a longer range

Potential Solution Two (longer):

• Rearchitect the current structure to handle load as response as they come in rather than waiting to all the responses to complete

Updates

• I emailed Mapbox to see if they have any examples with handling large data loading
• Copied the section that needs refactoring and renamed it
• Started refactoring the class component (legacy) into functional component

Questions:

• What is the expectation?
• Do you want one weeks of data to load as the data comes in?
• Do you want to limit the range that can be selected by the user?