Add region field to server entry

[GeckoEidechse]

The idea behind this PR is to add information about geographic location of a gameserver the serverbrowser. This could be combined with a PR to the client that replaces the currently unused latency column with a region column.

Currently the accuracy of the region is limited to continent, e.g. "Europe", "North America", ...

It gets populated via response from 3rd-party GeoIP API service (https://ip-api.com).

I'm making this a draft for now cause there's a bunch of open questions:

• Do we want to use this 3rd-party ip-api.com (they offer 45 requests a minute for free) or another?
• Do we want to add an env variable that allows disabling GeoIP stuff?
• What do we want to show if GeoIP was unsuccessful? Empty string "" or no field at all?
• Given the current accuracy of geographic location, is region the right name for the field?

[GeckoEidechse]

Sample output, note the region field at the end:

[
  {
    "lastHeartbeat": 1659486968464,
    "id": "2e19b8eaae850b0a16f45397bece3c3a",
    "name": "Unnamed Northstar Server",
    "description": "Default server description",
    "playerCount": 1,
    "maxPlayers": 16,
    "map": "mp_lobby",
    "playlist": "private_match",
    "hasPassword": false,
    "modInfo": {
      "Mods": [
        {
          "Name": "Northstar.Custom",
          "Version": "1.9.2",
          "RequiredOnClient": true,
          "Pdiff": "// this is just an empty pdiff file so that if people roll back an update the pdiff file will be overwritten\n"
        }
      ]
    },
    "region": "Europe"
  }
]

[Alystrasz]

Good idea, this would allow server owners not to put their server's location in the server name. Actually, if the continent field doesn't fit server browser, we could even use continentCode field (uses two characters). 😃

Do we want to use this 3rd-party ip-api.com (they offer 45 requests a minute for free) or another?

This API is simple and does the job.

I saw that you updated the TryReviveServer method, is it called often? If it is, rate limit would be rapidly reached... Maybe it's not necessary for this method to update server's region? (servers don't move between regions, do they?)

Do we want to add an env variable that allows disabling GeoIP stuff?

I don't think so; since GeoIP does apply rate limit per IP, this shouldn't be a problem during development (in the worst case, if you spammed the API too much, you'd just have to wait a few minutes for the limit to be reset).

What do we want to show if GeoIP was unsuccessful?

Do you have examples of IPs that GeoIP cannot solve? In the worst case (when it's offline for instance), we still could display something like "Unknown".

Is region the right name for the field?

I do think so; even if big actors like Google and Amazon have smaller regions, the "continent" information is enough for us.

[uniboi]

I'd prefer a npm package like this over a third party api tbh.

[GeckoEidechse]

Do we want to add an env variable that allows disabling GeoIP stuff?

I don't think so; since GeoIP does apply rate limit per IP, this shouldn't be a problem during development (in the worst case, if you spammed the API too much, you'd just have to wait a few minutes for the limit to be reset).

My main train of thought was that one might not want to have to interact with a 3rd when hosting/developing masterserver for whatever reason.

[GeckoEidechse]

I'd prefer a npm package like this over a third party api tbh.

That specific library does not seem to support granularity at a continent level which is what I'm aiming for though that could be pulled from timezone info.

Anyway, the reason I went with API was that (apart from being too lazy to look for NPM geoip packages)

1. API doesn't need updating when GeoIP allocations change (that still happens)
2. Doesn't add another dependency (I'm scared of running npm install on masterserver ^^)
3. Probably is easier to work with when we eventually add IPv6 support

On the other side library means

1. No rate-limiting
2. No reliance on 3rd-party

I guess the best way to decide which one to pick would be a poll.

For now I guess I'm just gonna try to write everything in a generic way to allow easy switching between API and library ^^

[GeckoEidechse]

Poll: For GeoIP, use

• :tada: 3rd-party API
• :rocket: NPM library

[Alystrasz]

Moreover, I said some stupid things this morning: continent granularity is not enough to be used for region information, since you can have big latency differences within the same continent (US-West VS US-East, for instance). Maybe we'll have to define regions ourselves 😭

[Alystrasz]

I'd prefer a npm package like this over a third party api tbh.

I saw that an account key is somehow required in the process of using this library? Even if we use it, just like the API, we'd still rely on software we don't develop: this doesn't solve third-party dependency ;)

[GeckoEidechse]

Maybe we'll have to define regions ourselves :sob:

Unless someone knows a library/API that is more coarse than countries and more precise than continents you might be right :/

Just for completeness sake, the reason we cannot use countries is due to

• maybe completely giving away pretty exact server position for certain countries e.g. Lichtenstein
• being terrible to filter by, e.g if you're in Germany, you don't really care if server is in France, Poland, Italy, etc but you'd still have to still filter for each area individually.
• some countries being to big for it to be precise enough, e.g. USA and Russia
• some country locations not actually being known by player (tell me where "New Caledonia" is located without looking at map first :P)

[ASpoonPlaysGames]

• some country locations not actually being known by player (tell me where "New Caledonia" is located without looking at map first :P)

uhhh, sounds like a european country of sorts, but I'm like 95% sure it's not mainland europe, so maybe one of those overseas territories? In which case I have no idea and I'm just gonna guess somewhere around south america?

Edit: damn, right train of thought but wrong place

[ASpoonPlaysGames]

But uhh, more on topic, yeah using continent/country borders isn't really going to work. There are other options, but all of the ones that i can think of sacrifice human readability, which I don't think is a good idea. Defining things ourselves might be the only good option? We could potentially do something like having 2 region tags: Continent, and cardinal location (North, East, South, West, Central), which would allow users to filter by only the continent, or by both for more precision

[Alystrasz]

We could potentially do something like having 2 region tags: Continent, and cardinal location (North, East, South, West, Central)

Sounds harder to put in server browser, compared to a single region tag (e.g. "europe-southwest").

I think we should just manually map country codes to Google regions:

• northamerica-northeast
• us-central
• us-east
• us-west
• us-south
• southamerica-east
• southamerica-west
• asia-east
• asia-northeast
• asia-south
• asia-southeast
• australia-southeast
• europe-north
• europe-central
• europe-southwest
• europe-west

We should however agree on regions list (should we add or remove some from this list) before I start assigning countries to regions (maybe ask servers owners about that).

[RoyalBlue1]

I think we should just manually map country codes to Google regions:

I like the idea of using google regions but mapping country codes might not work, because unless we divide big countrys like the us to states (or whatever else that country uses) we cant devide them into these regions

[Alystrasz]

I think we should just manually map country codes to Google regions:

I like the idea of using google regions but mapping country codes might not work, because unless we divide big countrys like the us to states (or whatever else that country uses) we cant devide them into these regions

Regarding the US, the API provides a "region" tag we could use to assign US states to regions:

I don't know if there are other countries behaving the same way as United States though.

[uniboi]

What's blocking this?

[GeckoEidechse]

What's blocking this?

https://github.com/R2Northstar/NorthstarMasterServer/pull/86 and me changing API and checking for ratelimiting.

[GeckoEidechse]

I'd prefer a npm package like this over a third party api tbh.

Aight I think you were right, pg independently told me to do use the same one as well ._.

[GeckoEidechse]

I'd prefer a npm package like this over a third party api tbh.

Aight I think you were right, pg independently told me to do use the same one as well ._.

Ok, so switching to https://github.com/geoip-lite/node-geoip for getting GeoIP is trivial due to the way I wrote the getGeoIp() function.

Issue is the coarsest scale it supports is countries which is a bit too accurate for my taste. Also because there's a lot of countries out there and so some person check server location might not even know where it's located based on the countries' name. See also: https://github.com/R2Northstar/NorthstarMasterServer/pull/84#issuecomment-1207474147

[Alystrasz]

Issue is the coarsest scale it supports is countries which is a bit too accurate for my taste.

Didn't we say we need to manually assign countries to regions? (see https://github.com/R2Northstar/NorthstarMasterServer/pull/84#issuecomment-1207494604)

[uniboi]

Didn't we say we need to manually assign countries to regions? (see #84 (comment))

We could just yoink a file like this https://github.com/lukes/ISO-3166-Countries-with-Regional-Codes/blob/master/all/all.csv

[Alystrasz]

We could just yoink a file like this https://github.com/lukes/ISO-3166-Countries-with-Regional-Codes/blob/master/all/all.csv

Good idea, that would spare us a lot of work :) I like the varying accuracy of region information of this document (region-code / sub-region-code / intermediate-region-code).

Region codes are available here btw: https://en.wikipedia.org/wiki/UN_M49#Code_lists

[Alystrasz]

Actually, with this file, all United States would be mapped as "Northern America", which might be a bit too coarse.

[GeckoEidechse]

Actually, with this file, all United States would be mapped as "Northern America", which might be a bit too coarse.

Not sure if I mentioned it before but for the first implementation, I rather be to coarse and keep the initial PR smaller and simple, and then later in a second PR refine and add a custom region mapping or whatever, than do it all at once ^^

[Alystrasz]

Not sure if I mentioned it before but for the first implementation, I rather be to coarse

If you want to use this, I can give you a hand and code the "country" => "region" association :)

[Alystrasz]

Not sure if I mentioned it before but for the first implementation, I rather be to coarse and keep the initial PR smaller and simple

Got bored in the train back home, here is an implementation: https://gist.github.com/Alystrasz/e79e568fc623a27daaf277128e784d52

[GeckoEidechse]

Got bored in the train back home, here is an implementation: https://gist.github.com/Alystrasz/e79e568fc623a27daaf277128e784d52

Oh nice :D

Question is, where do I find the file mentioned here? ^^ https://gist.github.com/Alystrasz/e79e568fc623a27daaf277128e784d52#file-countrycode_to_region-js-L48

[Alystrasz]

Question is, where do I find the file mentioned here?

It's on the repo @uniboi pointed out above: https://github.com/lukes/ISO-3166-Countries-with-Regional-Codes/tree/master/all

[pg9182]

Will be implemented in R2Northstar/Atlas.

I am most likely going to use IP2Location's free databases for the geoip lookup, then store it internally (and expose it in the metrics) as a 3-chracter geohash, which we can then map to a region name for the server list.

@Alystrasz, if you could find or create geohash -> region name mappings, it'd be greatly appreciated.

[Alystrasz]

@Alystrasz, if you could find or create geohash -> region name mappings, it'd be greatly appreciated.

1. I didn't know Atlas was a thing, congrats on the release :)
2. Does the script mentioned here do what you wanna achieve? I can rewrite it in Go if yes.

[Alystrasz]

Ok, I might have to rework previous script a bit.