New maps missing from 2.6 map server downloads list

[asvitkine]

New maps missing from 2.6 downloads list.

For example, "UHD World War II Global" is available for download from 2.5, but not from 2.6. It's listed in yaml here: https://github.com/triplea-game/triplea/blob/25234345e2d7afaf7c94ca1faea8ab5c69c3321a/triplea_maps.yaml#L2803C12-L2804C1

According to MapListingDao, it seems 2.6 fetches them from a database?

  @SqlQuery(
      "select"
          + "    m.map_name,"
          + "    m.download_url,"
          + "    m.download_size_bytes,"
          + "    m.preview_image_url,"
          + "    m.description,"
          + "    m.last_commit_date"
          + "  from map_index m"
          + "  order by m.map_name")
  List<MapListingRecord> fetchMapListings();

Are we missing automation to update that DB when new maps are added?

[DanVanAtta]

The thing that does the indexing is here: https://github.com/triplea-game/triplea/blob/master/spitfire-server/maps-module/src/main/java/org/triplea/maps/indexing/MapIndexingTaskRunner.java

The maps-server is not running right now though. The fetch to get a listing of maps should be broken as well. If that is working.. then it likely means the poller thread died.

There were issues with the poller thread dying and the maps server would stop indexing. That has been a task to figure out and splitting it from the regular lobby server was hopefully going to help figure that out.

[asvitkine]

@DanVanAtta what it take to get this fixed up?

Do we still think it's the issue with the poller thread? If so, did you consider adding a mechanism to check if it's running and if not, to start it again?

[asvitkine]

@DanVanAtta what should we do here? Do we have logs we can look at from the poller thread? If not, can we add logging?

[DanVanAtta]

Currently the 'poller' thread is just a background job that is running as part of the lobby. It emits logs just as much as the rest of the lobby does. There might be an exact indication of why it halts somewhere in there (or maybe that messaging is missing entirely). Gathering this information is certainly pretty key.

If so, did you consider adding a mechanism to check if it's running and if not, to start it again?

I fear that might be a bit too band-aid and might not actually help if it promptly crashes again. We arguably should hook up a metrics system like 'prometheus' and then emit a heartbeat metric that the poller thread is still running. In this way we would have a graph that we could look at to know its performance. We could potentially add some alerting to such a graph too.

Overall though, the poller thread ought to be pretty robust and simply stay running.

what should we do here?

Splitting the map server from the lobby arguably can help by making the problem space a bit smaller.

I'm starting to feel we are pinned to a very old version of dropwizard that has not aged very well. 7 years ago dropwizard was an excellent choice. As is, I would favor Micronaut or SpringBoot over drop wizard. Springboot I know has more solid management of background threads. Switching the platform we use is an option that could potentially moot this problem.

Fundamentally, we just need to dig in a bit to discover when and then why the thread stops.

[asvitkine]

@DanVanAtta

So I'd like us to get a handle on this, since from my perspective this is basically the main issue with 2.6 right now. Everything else should be good to go for a 2.6 release to make it the "stable" version. I believe gameplay wise, it should actually be more stable than 2.5 since a lot of 2.5-era bugs have been fixed.

So back to this issue, I think we need to confirm if the issue is that the map is missing from the DB (it was never indexed), or something else. Is it something you can check? The map mentioned in the OP is still missing from the download list for 2.6.

Or if you don't have time to look into this, can you give me the necessary access so I can do it? Would also like to be able to grep server logs - e.g. everything that has been logged by MapIndexingTaskRunner.java.

... and getting back for me to understand the issue at hand, when you say the "poller thread", do you mean MapIndexingTaskRunner or something else?

[DanVanAtta]

2.6 is a bigger topic, let's move that discussion to: https://github.com/triplea-game/triplea/discussions/12114

I gave that discussion a bit of a seed with the different topics that we need to address.

Or if you don't have time to look into this, can you give me the necessary access so I can do it?

If you can generate an ed25519 ssh-key and add the public key here: https://github.com/triplea-game/triplea/blob/master/infrastructure/ansible/roles/system/admin_user/files/authorized_keys

I run the deployment of that config and you'll then be able to SSH to the servers.

... and getting back for me to understand the issue at hand, when you say the "poller thread", do you mean MapIndexingTaskRunner or something else?

IIRC, yes. Essentially it stops running. I somewhat wonder if it is missing some sort of try-catch. Overall I'm not thrilled with the lack of robustness around this mechanism and somewhat blame dropwizard. I'd be pretty interested in micronaut as a replacement for dropwizard. But... likely we just need to add a try-catch somewhere high level wrapping the scheduled job

[DanVanAtta]

PS: @asvitkine FYI & FWIW: I'm hoping to get some time for some of these things in the coming weeks. I'm more willing to spend some time on TripleA, but it's been a slow ramp-up and time has been sparse.

[asvitkine]

@DanVanAtta Looking at the code of MapIndexingTaskRunner it seems the way it's set up now, it will just finish eventually, instead of running indefinitely to process new maps.

In it's run() method, it calls indexNextMapRepo() at the end.

That function will keep processing all the entries until reposToIndex is empty. But once it is, it won't reschedule itself.

So unless there's some higher-level code that's supposed to be recreating these objects, I think that's the problem. It just runs until it's done, then it never runs again. Or am I missing something, @DanVanAtta ?

[asvitkine]

Oh, I guess it is supposed to be scheduled at a higher level, via the code in buildMapsIndexingSchedule(), which creates a ScheduledTask that theoretically would be calling run() every once in a while. Although there's a lot of abstraction, so hard to follow exactly where it's started.

Do we know what configuration.getMapIndexingPeriodMinutes() is set to?

[DanVanAtta]

Do we know what configuration.getMapIndexingPeriodMinutes() is set to?

2 hours

The *Config.java interface classes like that are mapped to properties in the configuration.yml file of the server: https://github.com/triplea-game/triplea/blob/master/spitfire-server/dropwizard-server/configuration.yml#L18

The value is updated via environment variable: https://github.com/triplea-game/triplea/blob/master/infrastructure/ansible/roles/lobby_server/templates/lobby_server.service.j2#L14C11-L14C39

The final value used in production is defined here: https://github.com/triplea-game/triplea/blob/master/infrastructure/ansible/roles/lobby_server/defaults/main.yml

[asvitkine]

@DanVanAtta Did I see right that you did rework the code in question and split it out to a separate server or something?

Is that running now in production? I still don't see "UHD World War II Global" show up in 2.6. Perhaps there's some other issue with that item?

[asvitkine]

Currently, I see 2.6 show an error when trying to download maps:

Failed to download the list of available maps from TripleA servers. You can download the needed maps manually into your TripleA maps subfolder from https://github.com/triplea-maps/. FeignException$Unauthorized: [401 Unauthorized] during [GET] to [https://prod.triplea-game.org/maps/listing] [MapsClient#fetchMapListing()]: [Invalid request] If this problem happens frequently and is something you cannot fix, please report it to TripleA

[asvitkine]

I'm wondering if we should just revert to how 2.5 does downloads if we can't get 2.6 to work properly...

[asvitkine]

2.5 reads the map listing directly from:

public static final String MAP_DOWNLOAD_LIST =
      "https://raw.githubusercontent.com/triplea-game/triplea/master/triplea_maps.yaml";

[DanVanAtta]

Could make more sense to have the maps server read the static file rather than parse files live from repository.

We haven't really tried to get the maps server back up and running in a correct configuration. Currently, I believe it's more a lack of time & effort preventing us here rather than "we can't get it work properly."

On Wed, Feb 21, 2024 at 12:10 PM asvitkine @.***> wrote:

2.5 reads the map listing directly from:

public static final String MAP_DOWNLOAD_LIST = "https://raw.githubusercontent.com/triplea-game/triplea/master/triplea_maps.yaml";

— Reply to this email directly, view it on GitHub https://github.com/triplea-game/triplea/issues/11768#issuecomment-1957820278, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC6SZOMZ2Z7HAIH4CAMSSCDYUZIB7AVCNFSM6AAAAAA2HM5XCSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJXHAZDAMRXHA . You are receiving this because you were mentioned.Message ID: @.***>

[asvitkine]

I landed https://github.com/triplea-game/triplea/pull/12353, restoring the 2.5 codepath and keeping the new one behind a flag. With that change, I'm removing the Major and Regression labels from this issue. It's still a bug with the new codepath, but not critical given we have the old codepath back.

[Cernelius]

So... Should this be removed from the "2.6 Release" project list?

[Cernelius]

And maybe removing "map tags" too?