api icon indicating copy to clipboard operation
api copied to clipboard

Route API using old cached data?

Open addd45 opened this issue 1 year ago • 2 comments

Sorry in advance if I missed this in the docs... I check the API doc and swagger page, but I was only able to get some clue via looking at the API code.

It seems the route API fetches the csv data and imports it into a redis db, which is what the route api then uses. However, the caching period seems very slow. I frequently am seeing old / outdated routes, despite the csv source being updated.

I understand this is a free service so I don't want to give the impression that I am complaining :). However, I am wondering if this intentional behavior or perhaps a bug? If intentional, would it be unreasonable to request a daily refresh? Does the refresh but a heavy strain on resources? Do feeders get more up-to-date data?

More details below--

For example, as of today the route API for callsign "JIA5248" returns the previous route of SYR -> PHL. It was updated 5 days ago (10/20/24) to STL -> PHL.

route api response:

[
  {
    "_airport_codes_iata": "SYR-PHL",
    "_airports": [
      {
        "alt_feet": 421,
        "alt_meters": 128.32,
        "countryiso2": "US",
        "iata": "SYR",
        "icao": "KSYR",
        "lat": 43.111198,
        "location": "Syracuse",
        "lon": -76.1063,
        "name": "Syracuse Hancock International Airport"
      },
      {
        "alt_feet": 36,
        "alt_meters": 10.97,
        "countryiso2": "US",
        "iata": "PHL",
        "icao": "KPHL",
        "lat": 39.871899,
        "location": "Philadelphia",
        "lon": -75.241096,
        "name": "Philadelphia International Airport"
      }
    ],
    "airline_code": "JIA",
    "airport_codes": "KSYR-KPHL",
    "callsign": "JIA5248",
    "number": "5248",
    "plausible": 0
  }
]

you can see that the route was updated 5 days ago @ https://github.com/vradarserver/standing-data/blob/main/routes/schema-01/J/JIA-all.csv

addd45 avatar Oct 25 '24 19:10 addd45

the api gets the csv from https://vrs-standing-data.adsb.lol/ so if that update breaks it will get outdated data. perhaps that is what you saw.

iakat avatar Feb 03 '25 18:02 iakat

The API does not pull from that csv real time - which indeed makes sense because that would be incredibly resource intensive to parse a csv upon each request. Thus it is cached into a redis(?) cache on certain intervals so that the APIs pull data from this cache.

My conern here is the frequency of the cache updates. If I understand correctly, the csv source is updated at least daily as new routes are constantly updated. Realistically, for this API, once daily would seem like a good balance or starting point, but any longer than that will surely yield invalid/expired data.

In my original post the API was returning results from a csv file over 5 days old, which means the API is not properly reloading the cache timely enough for the overall goal that the API is designed for.

addd45 avatar Feb 03 '25 18:02 addd45