Adaptation of sparql_dataframe to Wikidata

[lbocken]

Hello,

I am trying to extract dataframes from queries in Wikidata.

For instance, this code from an example in Wikidata works to extract dictionary of countries:

`# pip install sparqlwrapper # https://rdflib.github.io/sparqlwrapper/ import sparql_dataframe import sys from SPARQLWrapper import SPARQLWrapper, JSON

endpoint_url = "https://query.wikidata.org/sparql"

query = """#Countries SELECT ?item ?itemLabel WHERE { ?item wdt:P31 wd:Q6256. SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". } }"""

def get_results(endpoint_url, query): user_agent = "WDQS-example Python/%s.%s" % (sys.version_info[0], sys.version_info[1]) # TODO adjust user agent; see https://w.wiki/CX6 sparql = SPARQLWrapper(endpoint_url, agent=user_agent) sparql.setQuery(query) sparql.setReturnFormat(JSON) return sparql.query().convert()

results = get_results(endpoint_url, query)

for result in results["results"]["bindings"]: print(result) `

When I do that : df = sparql_dataframe.get(endpoint_url, query)

I receive this error:

C:\ProgramData\Anaconda3\lib\site-packages\SPARQLWrapper\Wrapper.py:1315: RuntimeWarning: Format requested was CSV, but XML (application/sparql-results+xml;charset=utf-8) has been returned by the endpoint warnings.warn(message % (requested.upper(), format_name, mime), RuntimeWarning)

---

AttributeError Traceback (most recent call last) in ----> 1 df = sparql_dataframe.get(endpoint_url, query)

C:\ProgramData\Anaconda3\lib\site-packages\sparql_dataframe\sparql_dataframe.py in get_sparql_dataframe(endpoint, query, post) 28 sparql.setReturnFormat(CSV) 29 results = sparql.query().convert() ---> 30 _csv = StringIO(results.decode('utf-8')) 31 return pd.read_csv(_csv, sep=",")

AttributeError: 'Document' object has no attribute 'decode'

[lawlesst]

Hello,

Try passing post=True. E.g.:

sparql_dataframe.get(endpoint_url, query, post=True)

You can see in the unit tests that queries against Wikidata should work fine with post=True: https://github.com/lawlesst/sparql-dataframe/blob/master/tests/test_sparql_dataframe.py#L65

[lbocken]

---

HTTPError Traceback (most recent call last) in ----> 1 df = sparql_dataframe.get(endpoint_url, query, post = True) 2 df

C:\ProgramData\Anaconda3\lib\site-packages\sparql_dataframe\sparql_dataframe.py in get_sparql_dataframe(endpoint, query, post) 27 28 sparql.setReturnFormat(CSV) ---> 29 results = sparql.query().convert() 30 _csv = StringIO(results.decode('utf-8')) 31 return pd.read_csv(_csv, sep=",")

C:\ProgramData\Anaconda3\lib\site-packages\SPARQLWrapper\Wrapper.py in query(self) 1105 :rtype: :class:QueryResult instance 1106 """ -> 1107 return QueryResult(self._query()) 1108 1109 def queryAndConvert(self):

C:\ProgramData\Anaconda3\lib\site-packages\SPARQLWrapper\Wrapper.py in _query(self) 1085 raise EndPointInternalError(e.read()) 1086 else: -> 1087 raise e 1088 1089 def query(self):

C:\ProgramData\Anaconda3\lib\site-packages\SPARQLWrapper\Wrapper.py in _query(self) 1071 response = urlopener(request, timeout=self.timeout) 1072 else: -> 1073 response = urlopener(request) 1074 return response, self.returnFormat 1075 except urllib.error.HTTPError as e:

C:\ProgramData\Anaconda3\lib\urllib\request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context) 220 else: 221 opener = _opener --> 222 return opener.open(url, data, timeout) 223 224 def install_opener(opener):

C:\ProgramData\Anaconda3\lib\urllib\request.py in open(self, fullurl, data, timeout) 529 for processor in self.process_response.get(protocol, []): 530 meth = getattr(processor, meth_name) --> 531 response = meth(req, response) 532 533 return response

C:\ProgramData\Anaconda3\lib\urllib\request.py in http_response(self, request, response) 638 # request was successfully received, understood, and accepted. 639 if not (200 <= code < 300): --> 640 response = self.parent.error( 641 'http', request, response, code, msg, hdrs) 642

C:\ProgramData\Anaconda3\lib\urllib\request.py in error(self, proto, *args) 567 if http_err: 568 args = (dict, 'default', 'http_error_default') + orig_args --> 569 return self._call_chain(*args) 570 571 # XXX probably also want an abstract factory that knows when it makes

C:\ProgramData\Anaconda3\lib\urllib\request.py in _call_chain(self, chain, kind, meth_name, *args) 500 for handler in handlers: 501 func = getattr(handler, meth_name) --> 502 result = func(*args) 503 if result is not None: 504 return result

C:\ProgramData\Anaconda3\lib\urllib\request.py in http_error_default(self, req, fp, code, msg, hdrs) 647 class HTTPDefaultErrorHandler(BaseHandler): 648 def http_error_default(self, req, fp, code, msg, hdrs): --> 649 raise HTTPError(req.full_url, code, msg, hdrs, fp) 650 651 class HTTPRedirectHandler(BaseHandler):

HTTPError: HTTP Error 403: Forbidden

[lawlesst]

I think that's an error returned by the actual Wikidata SPARQL endpoint. It aggressively rate limits.

[lbockenrs]

Hello,

Try passing post=True. E.g.:

sparql_dataframe.get(endpoint_url, query, post=True)

You can see in the unit tests that queries against Wikidata should work fine with post=True: https://github.com/lawlesst/sparql-dataframe/blob/master/tests/test_sparql_dataframe.py#L65

How would you read the query saved into a separate file? Thanks for your help !

[lawlesst]

If your queries are saved in a text file, then you would just read them in like any other text file in Python and save them to a query variable that you would use with sparql_dataframe.get.

Here's a tutorial on reading and writing files in Python: https://realpython.com/read-write-files-python/#reading-and-writing-opened-files

[lbockenrs]

This works :

import sparql_dataframe endpoint_url = "https://query.wikidata.org/sparql" with open('query.rq', 'r') as file: query = file.read() df = sparql_dataframe.get(endpoint_url, query, post = True) df

[hbruch]

Just had the same issue issue querying wikidata. First thought, it might be caused by a version change (SPARQLWrapper was installed in version 2.0.0). It now already contains get_sparql_dataframe, so the code below was successful.

Nevertheless, thanks for creating this lib which made it directly into the wrapper!

from SPARQLWrapper import get_sparql_dataframe

endpoint = "https://query.wikidata.org/sparql"

query = """#Countries
SELECT ?item ?itemLabel
WHERE {
  ?item wdt:P31 wd:Q6256.
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". }
}
"""
df = get_sparql_dataframe(endpoint, query)