fulltext
fulltext copied to clipboard
ropensci-archive
Question: Are you planning supporting Web of Science ?
Sometimes as well called "Web Of Knowledge".
http://wokinfo.com/
They have an API, so it should be possible to support it for "search" and "abstracts".
Thanks for your question. I looked into it a bit, decided that not at this time. WOS is very complicated from a access perspective - tried using wosr and I couldn't figure out how to access the data through it. If access gets easier we can integrate here.
We are very much interested in having WoS support and might be able to contribute something. Our goal is to be able to use a single R package for searching and accessing abstracts in:
- Scopus
- Pubmed
- Web of Science
mainly and maybe some more in the future.
I think wosr is oudated and is not updated for the latest (big) changes in the WoS API: https://github.com/vt-arc/wosr/issues/11
Same for rwos: https://github.com/juba/rwos
The new WoS API itself looks rather clean and modern. (Json based, Swagger definition available, token based authentication ) Maybe it has changed a lot since you looked last time.
I found a rather new python package, this might be an option. (But is probbaly outdated as well, as it is as well Soap based)
We will probbaly do firstl a little inventory of existing packages, and see to get them working or start something from scratch.
So would you suggest in any case, to work first on a "working R package foccusing on WoS" and then later on eventualy accept a contribution to integrate it in here ?
I saw that you have already a clean way to integrate plugins, so that would make it easy.
Can you share a link to the API documentation that you're referring to?
Probably yes we could do an integration if you work on a package for WOS, but that may take longer than putting some code in here (getting the package created, then to CRAN). An example of not wrapping a package, but just including code in fulltext is for scopus - see scpous_utils.R. But some folks may want to use WOS by itself, so its definitely useful to have a separate package too.
Here is the "start" of a package supporting the new WoS APIs: It does so far not have support for "search", but it seems that a lot of basic stuff for "all APIs" of Clarivate is there. (list APIs, list endpoints, something for the keys)
https://github.com/muschellij2/webofscience
The docu I refer is here, https://developer.clarivate.com/apis/wos
I downloaded and imported the provided swagger file. (which is only accessible with a devlopper account) https://app.swaggerhub.com/apis-docs/EFSA/web-of_science_api_expanded/1.0.0
Probably yes we could do an integration if you work on a package for WOS, but that may take longer than putting some code in here (getting the package created, then to CRAN). An example of not wrapping a package, but just including code in fulltext is for scopus - see scpous_utils.R. But some folks may want to use WOS by itself, so its definitely useful to have a separate package too.
I will think about both options. We have a very narrow use case (systematic reviews of articles), in that we only need "title" and "abstract" and no other metadata. So we could eventually be much faster to add the minimal needed support for WoS here.
Should we leave this issue open for further discussion later (in a couple of weeks) or would you prefer some other form ?
Thanks. Checked, and I don't have access to that newer WOS API. For now, probably best to develop what you need in a separate repo. I'll see if I can get access. If I can't, then WOS will have to stay separate as I don't want to include data sources to which I do not have access.
Let's leave this open for now
@behrica i can't figure it out actually. I register an application on https://developer.clarivate.com/, then the application says "Your application does not have any API subscriptions currently. You may add subscriptions on the API pages.", but then I go to the APIs page https://developer.clarivate.com/apis and WOS isn't there. any ideas?
They are definitely "protecting" the APIs... (and information about them) You should see 2 extra APIs here: https://developer.clarivate.com/apis after logging in.
- web of science - expanded
- web of science - lite (less metadata, no abstracts)
The second one can be used for free, after registration The first definitely needs a paid subscriptions (not sure, if there is a trial mode or similar).
My organisation does currently have a WoS subscription, but not for the API yet. But we will purchase it "this year". I am still trying, if I can do "something" with it already now, without waiting for the subscription to be active. I requested access to Clarivant and is currently "pending"
If a user has an API token for the "expanded API" (and therefore a working subscription) the authentication should be very simple (putting a token in the http call), which needed to be passed in, similar then you do with Scopus already.
I finally got "fulltext" with Scopus working yesterday... happy about that.
But is was very painfull, as I need to be in the right IP range in order to see the abstracts (and configure correctly our corporate proxy including proxy authentication) I am happy that fulltext allows detailed configuration of "curl" .... If you want, I can document this somewhere... might help others And be on the VPN ....
I attached the swagger 2.0 definition of the WoS expanded API wos.swagger.gz
Maybe useful for you. I think there are tools to view it nicely.
The API looks, as I said, "well designed" on first sight. Including as well documentation how "quotas" will be returned, allowing json + xml.
But they definitely protect the metadata, and "advanced" metadata need to be paid. There is as well a detailed policy of "what is allowed to do " with each metadata element. (storing, showing to public, ....)
So far away from "Open Science".
But limited, basic metadata, is for free.
Thanks for all the details.
For now I don't have access, so I can't support WOS here - i'll look into access more this week
Still dont have access. So likely WOS will have to stay as a separate package unless access becomes significantly easier
Ok, thanks for the info.
Maybe we come back to this in couple of weeks, once we start working on the WOS. We definitely have the goal to make the access to WebOfScience articles "as easy as possible" from R.
Starting this will maybe help us as well to figure out the exact procedure to get access.
We will try to follow what is done here as closely as possible, in order to ease a potential future integration.
@sckott I'm currently taking on a project using the WoS lite API. I have a working script in Python that accesses WoS, scrapes it, and outputs a .RIS file of all the output.
I have access to WoS through my University and registered an application that way. All a user needs is an API key they get from WoS.
If it sounds worth it, I'm open to setting up a meeting to discuss potentially implementing an R version?
This repository is about to be archived. If you develop a related package, it might be in scope for https://ropensci.org/software-review/
