Geograpy3 for other languages

[MattMacs]

Is your feature request related to a problem? Please describe. I would like to use geograpy3 in my current project, but as I'm working with sentences in German, it doesn't work...

Describe the solution you'd like Support other languages as well.

[WolfgangFahl]

As of Version 0.2.0 other language support is prepared. Which language(s) would you like to see supported and what is your usecase? It would be very helpful if you specify a concrete scenario in which your need is shown.

[JakobMiksch]

Hi, what is the state for supporting other languages?

I would be interested in German as well.

[WolfgangFahl]

@JakobMiksch - you might want to first try out to create the locations.db using a wikidata query with a german query. Also it might be possible to switch to a "dynamic" wikidata query for the resolution of locations. It's a trade off for the cache size or the query speed. The data is in principle available via the "label" field of the location hierarchy within wikidata.

Again - what is your usecase - e.g. how often do you need to lookup data - what is the expected response time and so on. Designing a solution will depend on knowing what your needs are.

[astuanax]

Need any help on this?

This is what I am looking for

• Language support: English, German, Dutch, French, Spanish, Danish, Portuguese, Finish, Swedish, Italian, Slovene.
• Download size isn't an issue, as long as we can control when data is downloaded (and to the exact path)
• To be used in an API: a pretty fast response time is needed.
• Being able to parse whole HTML body content without using too much memory

Hugginface NER models are pretty slow for this task. No experience with Spacy.

[WolfgangFahl]

@astuanax could you please create a pull request for language specific wikidata queries. We also might try using a different endpoint such as QLever for make sure we don't run into timeouts.

[WolfgangFahl]

A first step would be to externalize the queries to yaml files using the named queries feature of the pyLodStorage QueryManager https://github.com/WolfgangFahl/pyLoDStorage/blob/ef3ff1148a5addb6974d2bc0eac1004a1ba37beb/lodstorage/query.py#L498

[astuanax]

Ok, I'll have a look at the specific wikidata queries to start with.

[WolfgangFahl]

the tool https://www.npmjs.com/package/wikidata-taxonomy might be helpful: human settlement is currently our base class which might be far to broad for most real-life applications.

wdtaxonomy Q486972 -c -d -m =

will show your all specialization classes

the following queries will show the top 200 classes with more than 700 instances:

# relevant human settlement classes in wikidata
# based on wdtaxonomy Q486972 -c -d -m = -s
# WF 2021-08-25
SELECT ?item ?itemLabel ?itemDescription ?instances ?sites  WITH {
    SELECT DISTINCT ?item { ?item wdt:P279* wd:Q486972 }
  } AS %items WHERE { 
    INCLUDE %items .
    {
      SELECT ?item (count(distinct ?element) as ?instances) {
        INCLUDE %items.
        OPTIONAL { ?element wdt:P31 ?item }
      } GROUP BY ?item
    }
    {
      SELECT ?item (count(distinct ?site) as ?sites) {
        INCLUDE %items.
        OPTIONAL { ?site schema:about ?item }
      } GROUP BY ?item
    }
    SERVICE wikibase:label {
      bd:serviceParam wikibase:language "en"
    }

    FILTER(?instances>700)
  } ORDER BY DESC (?instances)
LIMIT 200

try it!

[WolfgangFahl]

sparqlquery -qp wikidata.yaml -qn Top200HumanSettlementClasses -f github

top 200 human settlement classes https://www.wikidata.org/wiki/Q486972

based on wdtaxonomy Q486972 -c -d -m = using https://www.npmjs.com/package/wikidata-taxonomy

query

# relevant human settlement classes in wikidata
# based on wdtaxonomy Q486972 -c -d -m = -s
# WF 2021-08-25
SELECT ?item ?itemLabel ?itemDescription ?instances ?sites  WITH {
    SELECT DISTINCT ?item { ?item wdt:P279* wd:Q486972 }
  } AS %items WHERE {
    INCLUDE %items .
    {
      SELECT ?item (count(distinct ?element) as ?instances) {
        INCLUDE %items.
        OPTIONAL { ?element wdt:P31 ?item }
      } GROUP BY ?item
    }
    {
      SELECT ?item (count(distinct ?site) as ?sites) {
        INCLUDE %items.
        OPTIONAL { ?site schema:about ?item }
      } GROUP BY ?item
    }
    SERVICE wikibase:label {
      bd:serviceParam wikibase:language "en"
    }

    FILTER(?instances>700)
  } ORDER BY DESC (?instances)
LIMIT 200

try it!

result

item	itemLabel	itemDescription	instances	sites
Q13100073	village-level division in China	People's Republic of China subdivision level	588606	11
Q486972	human settlement	community of any size, in which people live	570340	80
Q532	village	small clustered human settlement smaller than a town	272870	168
Q56436498	village in India	administrative division below the subdistrict level in the South Asian country	145541	0
Q5084	hamlet	small settlement in a rural area	117270	57
Q985488	residential community of the People's Republic of China	administrative territorial entity of the People's Republic of China	90953	10
Q2225692	fourth-level administrative division in Indonesia	Indonesian fourth administrative division, part of kecamatan, with a limited level of government administration. It has various national (wilayah) and local (including gampong, kampung, desa, pekon, kelurahan, nagari, etc.) designations	81294	0
Q3558970	village of Poland	type of populated place	56082	0
Q484170	commune of France	France territorial subdivision for municipalities	45582	75
Q253019	Ortsteil	named subdivision or section of a human settlement in Germany, Austria or Switzerland	41777	11
Q26211545	desa	type of administrative district in Indonesia	41655	5
Q3257686	locality	place of human settlement	40859	7
Q17343829	unincorporated community in the United States	settlement in the United States without its own local government	39123	0
Q23925393	Douar	administrative division of Morocco	32026	7
Q123705	neighborhood	geographically localised community within a larger city, town or suburb	27091	53
Q21672098	village of Ukraine	type of Ukrainian administrative division	26523	2
Q735428	town in China	People's Republic of China township-level subdivision	21612	21
Q1529096	village in Turkey	rural administrative division in Turkey	19894	8
Q2116450	manor estate	estate in land to which is incident the right to hold a manorial court	19255	17
Q634099	rural settlement of Russia	type of municipal unit in Russia	19121	19
Q1134686	frazione	territorial subdivision of a comune in Italy	19093	32
Q2514025	posyolok	type of settlement in Belarus, Kazakhstan, Russia and Ukraine	18720	20
Q15303838	municipality seat	location of the main elements of the government of a municipality	18219	0
Q65661087	village in Belarus	type of rural settlement in Belarus	17320	0
Q18663566	dissolved municipality of Japan	former municipality in Japan	15246	0
Q3957	town	settlement that is smaller than a city but bigger than a large village (or borough in some areas)	14609	85
Q20541692	settlement in Galicia, Spain		14591	0
Q20202352	locality of Mexico	entity below a municipality, identified by INEGI	12680	2
Q61089180	part of municipality in the Czech Republic	municipes of Czech Republic	11509	2
Q17051044	mahalle	type of subdivision in Turkey	11022	3
Q498162	census-designated place	statistical concentration of population defined by the United States Census Bureau	10733	42
Q131596	farm	area of land for farming, or, for aquaculture, lake, river or sea, including various structures	10110	95
Q1093829	city of the United States	administrative division at the local government level in the United States	9732	7
Q262166	municipality of Germany	lowest official level of territorial division in Germany	9608	35
Q515	city	large permanent human settlement	9168	235
Q23012917	village of Burkina Faso	Situé dans la région de la boucle du mouhoun, sapala est un gros village qui est à 32km de la commune rurale de yaba et à 25km de Toma.	8369	0
Q7930614	village of Taiwan	fourth level administrative subdivisions of Taiwan	7996	7
Q747074	commune of Italy	municipal divisions of Italy	7875	54
Q2074737	municipality of Spain	administrative territorial entity of Spain	6986	27
Q188509	suburb	residential or mixed use area, either existing as part of a city or urban area or as a separate area	6737	78
Q4285979	gampong	Indonesian village administrative division in Aceh province, equivalent of "desa" in the rest of Indonesia	6694	4
Q350895	abandoned village	village that has been deserted	6614	15
Q15127012	town of the United States	administrative division at the local government level in the United States	6471	4
Q5153359	municipality of the Czech Republic	administrative territorial division of the Czech Republic	6260	6
Q12488913	kampung of Papua	type of village in Papua	6248	2
Q2023000	khutor	type of hamlet	6181	29
Q22674925	former populated place	settlement that no longer exists, e.g. because it was abandoned, destroyed or it became part of another settlement	5698	1
Q3184121	municipality of Brazil	administrative division of the states in Brazil	5568	25
Q3055118	single entity of population	Spanish statistical unit of human settlements	5293	3
Q19730508	former municipality	municipality that no longer exists	5248	0
Q15630849	village of Bulgaria	type of village	5059	0
Q106505070	Settlement of protohistoric times	settlement of cultures that have knowledge of writing (protohistory)	5021	0
Q22580836	village in Latvia		4824	2
Q51049922	village in Estonia	type of settlement in Estonia	4458	0
Q14839548	minor locality in Sweden	small place with 50-199 inhabitants in Sweden	4358	8
Q7551933	mahallah	city subdivision or neighbourhood in parts of the Arab world, Balkans, Western and South Asia and nearby countries	4222	25
Q15284	municipality	administrative division having corporate status and usually powers of self-government or jurisdiction	4212	106
Q5327369	chōchō	subdivision of a city in Japan	4090	3
Q1484611	buurtschap	small inhabited place with its own name but often without a clear centre such as a church or market square	3628	8
Q61878	barangay	administrative division in the Philippines	3524	39
Q19658107	neighborhood of Brazil	administrative subdivision of municipalities in Brazil	3417	0
Q70208	municipality of Switzerland	smallest government division in Switzerland	3360	27
Q152081	concentration camp	camp in which people are imprisoned or confined, commonly in large groups, without trial	3335	73
Q1549591	big city	city with a population of at least 100,000	3231	20
Q74047	ghost town	city depopulated of inhabitants and that often stays practically intact	3158	55
Q751708	village in the United States	administrative division at the local government level in the United States	3148	10
Q490329	dong of South Korea	lowest administrative unit of districts throughout Souh Korea	2998	0
Q6784672	municipality of Slovakia	municipality in Slovakia	2884	9
Q34841063	seat of the local council	location of the local council of a rural locality of Romania	2861	1
Q2082774	shieling	mountain pasture	2839	20
Q2590631	municipality of Hungary		2813	4
Q12813115	urban area in Sweden	Def: population >= 200, < 200 m between buildings, etc	2713	12
Q11722303	rural municipality of Sweden and Finland	former type of municipality in Sweden (1863–1970) and Finland (1865–1977)	2628	6
Q695850	airbase	aerodrome used by a military force for the operation of military aircraft	2613	40
Q17540983	open air site		2548	2
Q56398194	core municipal part with the same name as the municipality in the Czech Republic		2480	0
Q1852859	populated place in the Netherlands	Officially designated place ("woonplaats") (city/town/village) in the Netherlands	2467	2
Q21869758	delegated commune		2409	8
Q15661340	ancient city	city in historical records	2383	0
Q667509	municipality of Austria	local administrative unit in Austria	2366	9
Q1952852	municipality of Mexico	municipio: mexican administrative country subdivisions	2320	24
Q106492558	Settlement of prehistoric times		2288	0
Q72030539	farmstead	the buildings and service areas associated with a farm	2194	0
Q16352482	dispersed settlement in Latvia	type of rural settlement in Latvia	2104	1
Q106491339	Neolithic settlement		2101	0
Q42744322	urban municipality of Germany	municipality with town privileges in Germany (may e.g. legally prepend "Stadt" to its name)	2065	2
Q106491386	Q106491386		2021	0
Q188913	plantation	long artificially established forest, farm or estate, where crops are grown for sale	1994	58
Q12225020	uzlah	local administrative division of Yemen	1982	4
Q1138414	hermitage	an isolated building in Europe	1925	4
Q2066754	manor	historical manor estate in the Baltic area	1849	7
Q3624938	city district of Brazil	geographical and administrative region in Brazil	1726	4
Q62049	county seat	administrative center or seat of government for a county or civil parish	1691	34
Q42211429	township of Minnesota	administrative territorial entity of Minnesota	1664	0
Q20757176	parish municipality of Denmark		1649	1
Q17205621	township of Iowa	township in Iowa, United States	1647	0
Q21507948	former village	village that no longer exists, e.g. because it was deserted, destroyed or it became part of another settlement	1626	1
Q9035798	township of Pennsylvania	type of municipality in the U.S. state of Pennsylvania	1588	2
Q3504085	rural municipality of Poland	administrative division of Poland	1586	6
Q2661988	urban settlement in Russia	type of a municipal formation in Russia	1583	14
Q2989398	commune of Algeria	administrative division of Algeria	1548	12
Q2039348	municipality of the Netherlands	second-level administrative division in the Netherlands	1522	22
Q104841013	hromada	administrative unit in Ukraine	1513	13
Q902814	border town	town or city on a border between larger political areas	1494	10
Q24764	municipality of the Philippines	administrative division of the Philippines	1492	17
Q2583015	ghetto in Nazi-occupied Europe	settlement set up in Nazi-occupied Europe by Nazi Germany for Jews and sometimes Romani	1481	18
Q16363214	small village in Latvia	type of rural settlement in Latvia	1457	1
Q17198545	township of Illinois	civil township subdivision of most Illinois counties	1442	0
Q4632675	dwelling place		1400	2
Q17205735	township of Kansas		1399	0
Q17318027	rural commune of Morocco	administrative division of Morocco	1352	0
Q1802801	rural municipality of Austria	type of Austrian rural municipality	1349	1
Q28371991	village of Yemen	type of populated place	1342	1
Q15092344	urban area in Norway	urban unit definition in Norway issued by Statistics Norway (SSB): population at least 200, less than 50 m between buildings, etc	1339	1
Q245016	military base	facility directly owned and operated by or for the military	1327	48
Q1348006	city block	central element of urban planning and urban design; smallest area that is surrounded by streets	1320	37
Q17198620	township of Ohio		1311	0
Q1989945	agrotown	type of rural settlement in Belarus	1306	11
Q685309	former municipality of Switzerland	type of municipality of Switzerland	1305	0
Q56557504	city of Iran	city and type of administrative subdivision of Iran	1271	0
Q7930989	city/town	human settlement	1264	0
Q15975450	smallholding	type of rural locality	1145	9
Q17205774	township of Michigan		1111	0
Q2555896	municipality of Colombia	decentralized subdivisions of the Republic of Colombia	1108	11
Q27654220	abandoned hamlet	ghost town	1096	0
Q16830604	census town of India	administrative territorial entity of India	1076	0
Q17201685	township of Indiana		1009	0
Q777120	borough of Pennsylvania	type of municipality in the U.S. state of Pennsylvania	989	7
Q4291381	amalgamated hromada	former administrative unit in Ukraine	970	2
Q190928	shipyard	place where ships are repaired and built	966	51
Q33146843	municipality of Catalonia	municipality of Catalonia, Spain	947	2
Q1985797	municipal unit of Greece	municipality subdivision in Greece	935	3
Q106071004	town of New York	form of local government in the state of New York	921	0
Q15141625	subdistrict municipality	type of municipality in Thailand	915	1
Q28328984	village in Armenia		912	0
Q148837	polis	ancient Greek social and political organisation	901	71
Q57058	municipality of Croatia	second-level administrative subdivision in Croatia	895	16
Q12015918	colony	administrative unit of Poland	885	3
Q7216840	urban-type settlement in Ukraine	official type of settlement between a town and a agricultural village	878	7
Q2989400	commune of Morocco	administrative territorial entity in Morocco	869	5
Q106505045	Q106505045		867	0
Q12161100	cortijo		866	5
Q26742250	former municipality of Estonia	former administrative division of Estonia	856	0
Q106504960	Q106504960		835	0
Q16355778	average size village in Latvia	type of rural settlement in Latvia	813	1
Q4379239	Q4379239		809	7
Q815324	town municipality of Turkey	type of administrative region in Turkey, large village with a municipality	803	7
Q15921247	constituent locality	level 3 type of administrative entity of the SIRUTA system issued by the Romanian National Institute for Statistics	790	0
Q666943	associated commune	France territorial subdivision of a larger commune but retaining certain autonomy	787	8
Q1708422	settlement site	site of a former settlement of archaeological interest	786	8
Q562061	market municipality	municipality in some German-speaking regions; usually has a historical legal right to hold markets and fairs	777	17
Q882149	nagari	semi-autonomous Minangkabau people regional administrative unit in West Sumatra, Indonesia	769	6
Q424857	ōaza	subdivision of a village or town in Japan	758	3
Q1059478	town of Japan	administrative division of Japan	757	23
Q88205	castrum	ancient Roman fortification	749	38
Q2989457	urban-type settlement	type of urban populated place	734	41
Q494721	city of Japan	administrative division of Japan	731	39
Q329683	industrial zone	area for development of industry	729	29
Q19886692	neighbourhood of São Paulo		716	0
Q3559024	village of Senegal		713	2
Q1743100	church town of Sweden	district of cottages around a church, traditionally used by visiting churchgoers	712	5

[WolfgangFahl]

yaml modification for this example