Bug: Saved Properties Disappear When Zoomed Out

[tcmoorman]

Describe the bug When viewing the 'Saved' tab, zooming the property map out past a certain point will cause some properties in the Saved tab to disappear. Some properties are not affected, it is not clear yet what criteria causes properties to remain or disappear. The properties disappear as the map hits different points of downscaling as it is zoomed out

To Reproduce

1. Go to Find Properties
2. Save 3-4 properties
3. Zoom out until the level of detail on the map downscales
4. Properties will disappear from the Saved list
5. Zooming back in causes the properties to show again

Screenshots

Desktop (please complete the following information):

• OS: Mac OS 11.2.3
• Browser: Chrome
• Version 127.0.6533.120

[github-actions[bot]]

This issue has been marked as stale because it has been open for 30 days with no activity.

[HeyZoos]

Hello! I'd like to tackle this next

[nlebovits]

@HeyZoos assigned! thanks

[HeyZoos]

It seems to be because of the clustering behavior of MapLibre at certain zoom levels.

If I zoom out the feature count drops, I added a conditional which will log the feature if it's clustered

Note these values, so two of the features got grouped together into 1:

    "clustered": true,
    "point_count": 2,

{
    "address": "2011 N 22ND ST",
    "owner_1": "CITY OF PHILA",
    "owner_2": "DEPT OF PUBLIC PROP",
    "council_district": "5",
    "zoning_base_district": "RM-1",
    "zipcode": "19121",
    "opa_id": "322086001",
    "parcel_type": "Land",
    "city_owner_agency": "DPP",
    "side_yard_eligible": "No",
    "program": "",
    "phs_care_program": "No",
    "all_violations_past_year": 0,
    "open_violations_past_year": 0,
    "li_complaints": "",
    "rco_info": "32nd Ward RCO; 2231 W. Fontain st \nPhiladelphia, P \n19121; [email protected]; 2158068891|Strawberry Mansion Community Concern; 2640 N. Myrtlewood Street\r\nPhiladelphia, PA 19132; [email protected]; 2678479383|MAP Holistic CDC; 1510 W Stiles Street Philadelphia, PA 19121; [email protected]; 2159836894|The Good News Community Organization; 1631 C.B. Moore Ave 19121; [email protected]; 2673126508|St. Elizabeth's RCO; 1845 N. 23rd Street; [email protected]; 2672536178",
    "rco_names": "32nd Ward RCO|Strawberry Mansion Community Concern|MAP Holistic CDC|The Good News Community Organization|St. Elizabeth's RCO",
    "tree_canopy_gap": 0.153728083895667,
    "neighborhood": "North Central",
    "gun_crimes_density_percentile": 96,
    "gun_crimes_density_label": "96th Percentile",
    "drug_crimes_density_percentile": 86,
    "drug_crimes_density_label": "86th Percentile",
    "sheriff_sale": "N",
    "market_value": 112300,
    "sale_date": "1982-02-22T05:00:00Z",
    "sale_price": 500,
    "mailing_address_1": "MUNICIPAL SERVICES BLDG",
    "mailing_city_state": "PHILADELPHIA PA",
    "mailing_street": "1401 JOHN F KENNEDY BLVD",
    "mailing_zip": "19107",
    "unsafe_building": "N",
    "imm_dang_building": "N",
    "tactical_urbanism": "Yes",
    "conservatorship": "No",
    "owner_type": "Public",
    "park_priority": 3,
    "n_contiguous": 7,
    "permit_count": 46,
    "dev_rank": "Medium",
    "n_properties_owned": 962,
    "negligent_dev": false,
    "priority_level": "High",
    "access_process": "Private Land Use Agreement",
    "clustered": true,
    "point_count": 2,
    "sqrt_point_count": 1.41,
    "point_count_abbreviated": "2"
}

[HeyZoos]

This is a bit tricky, I think the ideal solution would be to somehow turn off clustering when a user is in that "Saved" view.

[HeyZoos]

We'd need to configure the layers / make a different non-clustered layer, this is the source of the features that are rendered and displayed on the side

// src/components/PropertyMap.tsx
    const features = map.queryRenderedFeatures(bbox, { layers });

[HeyZoos]

Hmm. So we can toggle clustering, but only for Sources with a geojson type (missing from VectorSourceSpecification)

export type GeoJSONSourceSpecification = {
	"type": "geojson";
         ...
	"cluster"?: boolean;

        <Source
          type="geojson"   // was vector
          url={`pmtiles://https://storage.googleapis.com/${googleCloudBucketName}/vacant_properties_tiles${
            useStagingTiles ? '_staging' : ''
          }.pmtiles`}  // So these tiles need to be regenerated for GeoJSON
          id="vacant_properties_tiles" 
          cluster={false}  // Then this is controllable, we could use a state variable
        >

[nlebovits]

@HeyZoos we're using the pmtiles for responsive rendering of the large geospatial dataset when so many features are in view. But, if the user is only looking at saved properties (which likely won't be more than a handful at any given time), could we just use the polygon geojson data even when zoomed out, provided that the user is in the saved property view?

@CodeWritingCow , thoughts?

[CodeWritingCow]

@HeyZoos @nlebovits If we just use the polygon geojson data, would there be any potential performance issues? Would it slow down our map's load/rerender time?

[nlebovits]

@CodeWritingCow I can't say for sure--I'm not familiar enough with this kind of scenario. I could generate a copy in the cloud for us to use if needed. Alternatively, we could store them as a geoparquet file, which would likely be more performant. @HeyZoos what do you think?

[nlebovits]

@HeyZoos maybe we use a parquet file of vector data for the point layer? do you think that would allow us to toggle clustering while also keeping it performant in the browser?

[HeyZoos]

@HeyZoos @nlebovits If we just use the polygon geojson data, would there be any potential performance issues? Would it slow down our map's load/rerender time?

Technically yes, because with a vector tile source the features are pre-clustered. Using a geojson source would result in clustering at runtime.

But! Since the number of properties in the saved view is relatively very small, and since we would disable clustering in the saved view, I don't think it would have an impact.

@HeyZoos maybe we use a parquet file of vector data for the point layer? do you think that would allow us to toggle clustering while also keeping it performant in the browser?

I'm betting on it not being a performance issue if we use the GeoJSON source but will definitely revisit if there's an impact.

[brandonfcohen1]

I haven't been following closely gave this a quick read- if you're just using geojson for saved properties you should have no performance issues.

[HeyZoos]

Hey gang, so it seems like the GeoJSON approach is a no-go. With the vector tile format, requests can be made to fetch a subset of the data resulting in a nice snappy initial page load

But it seems like with the GeoJSON, even if we just use a subset of the data, it loads the entire dataset in one go

I think the next thing to try is to generate an un-clustered vector tile layer

[HeyZoos]

Ok gang I tried it with an un-clustered version of the pmtiles and that did not work either

tippecanoe \
  --force \
  --no-feature-limit \
  --no-tile-size-limit \
  --drop-rate=0 \
  -z14 -Z0 \
  -o output.mbtiles \
  -l vacant_properties_tiles_points temp_vacant_properties_tiles_points.geojson \
  -l vacant_properties_tiles_polygons temp_vacant_properties_tiles_polygons.geojson

./go-pmtiles_1.22.2_Windows_x86_64/pmtiles.exe convert unclustered.mbtiles unclustered.pmtiles

        <Source
          type="vector"
          url={"pmtiles://http://localhost:8080/unclustered.pmtiles"}
          id="vacant_properties_tiles"
        >

npx http-server . -p 8080 --cors

[HeyZoos]

Features would still be dropped based on the zoom level, I think what it comes down to is that we can't display saved properties based on just the tile data and filtering with propertyIds

map.queryRenderedFeatures(bbox, { layers });

...

    } else {
      setShouldFilterSavedProperties(true);
      dispatch({
        type: 'SET_DIMENSIONS',
        property: 'opa_id',
        dimensions: [...propertyIds],
      });
    }

I think we'd actually have to start caching the saved properties data in its entirety (maybe this could be geoJSON) and rendering that instead)

[github-actions[bot]]

This issue has been marked as stale because it has been open for 30 days with no activity.

[tcmoorman]

It was observed that the property count will sometimes change depending on the map zoom level, even if all properties in the search are within the viewport. Screenshots attached showing the property count changing for the same search. Note: this is not the property count within the Saved tab but rather the total property count

[nlebovits]

@tcmoorman I think that our discussed switch to storing data in parquet files might allow us to get around this, but we may have to switch to using duckdb in browser with WASM (https://duckdb.org/2021/10/29/duckdb-wasm.html). This may require a more substantive ticket to resolve interactivity oveall.