tfonteyn
Feature Request - Improved Author Metadata
Hi
I have recently started using this to store my classical arabic book collection.
One thing that i would like is to be able to able to add some important "metadata" for authors.
Some examples:
- [ ] Place of birth
- [ ] Place of death
- [ ] Date of birth
- [ ] Date of death
- [ ] Interest/specialisation/what the author wrote about (possibly allow this to be the same as the categories assigned to books, and allow multiple?)
- [ ] List of titles/honorifics
- [ ] Either multiple pen names, or (preferably) allow multiple "known as" (where the distinction being made is a pen name is a name chosen by the author, and a "known as" is a name not chosen by the author, but chosen by e.g. those around him, his students, or otherwise.
As a not so important but definitely useful second part to this, I would also like for it to be possible to fetch the author’s metadata from Wikipedia (generally all of the above fields are within Wikipedia).
Thanks
Hello,
At least some of those fields have been on my own wish-list as well. What stopped me until now was (firstly 'time', I have books to read ..) that the sites I used at the time for metadata were not satisfactory to get that info (aside of ISFDB). But as I added multiple sites over the last couple of years, this has become feasible now I think.
I've been working hard on a new major version (7.0) which, among other things, will bring:
- a full fledged support for adding tags to Books as a replacement for the single-entry 'genre'. Fetching tags/categories/genres from most of the sites is of course included.
- Goodreads as a meta-source (yes, I said I wasn't going to do it... well, I changed my mind) and they have extensive author info that could be used.
So back to your request: it won't happen until after 7.0 is ready and published. But in subsequent version I think it's at least partially possible to:
- birth/death date & place: if available, sure. These are the ones on my own list
- a "Notes" field.
- Interest/specialisation: using the same tags (7.0!) as for the books should work just fine. From sites: not sure, depends on availability of course.
- List of titles/honorifics: a bit beyond the scope I fear. Firstly, I can only deal with structured data from the internet (ISFDB being perfect, but obviously only one particular genre so not suited for all). I think this won't/can't be done. But I can put it on my "to look at later" list.
- multiple pen names: already possible. You can define a "Real author"(i.e. actual name) for any author defined as a pen-name. Let me know if you have questions on this. I can/should probably document this better. Getting this as metadata is difficult, as sites most often do not provide this (at least not in a structured way). The one exception being franco/belgian comics using bedetheque.
- multiple "known as": would need to done as pen-name, no special distinction. Aside of which... no sites offer this I think. Using a notes field for this is probably better.
All this will require some planning obviously so please don't expect it any time soon. but I'll see what I can do.
I did some checks on what's possible in the short term, and in the long term.
I think it's sensible to start with storing Author identifiers from the existing sites (if they provide such) and offer a "View Author on.." functionality. That could get implemented fairly easily and could form a basis for more later on.
I can confirm that Author identifiers with a "View Author on.." menu will be available in 7.2. Later versions will expand on this.
btw, easiest (and only) way to get Identifiers on authors is to do a "Update from internet" for a book, making sure the author field is not set to "skip"
btw2: not all sites provide author-identifiers of course, forget about amazon/bol/.. (shopping sites). Most of the others, but not all, do however
It's a bit disappointing so far. I've now implemented (in 7.3.0) what I call "AuthorResolver"'s for
- Bedetheque
- Databaze Knih
- DNB
- ISFDB
- OpenLibrary
Current functionality is to resolve author names and their potential pseudonyms.
Which means there is only OpenLibrary for the general public, while the others are either language or genre specific. And ironically, OpenLibrary is (as usual... grumble) the worst at providing good data. Goodreads proved to be even worse (a bio being the only thing available), so I did not even bother for now.
Of those 5, we can get:
- birth/death dates 5/5
- a picture 3/5
- bio 2/5 (+ goodreads)
- links to dedicated author websites 3/5
And not every author has in fact 2-4.
As to Wikipedia, you are obviously correct that it will have the best info, but it's difficult to link/fetch to as it does not provide unique-id's. Instead I'd need to do a search by author name to get to the webpage... which is NOT structured, i.e. is very hard to impossible to parse reliably.
I'll have a think how much effort (i.e. performance) it is worth, to get which data.
progress: Birth/death dates have been added:
- fetching from the aforementioned 5 sites + kb.nl
- Stripinfo/Lastdodo/Webstrips don't need it as they already access Bedetheque author info anyhow.
- Goodreads will use OpenLibrary
- so that makes 10 sites which can now resolve author pen-names and fetch birth/death date data.
- I might add douban (Chinese) to that list, but there are complications for that site.
Birthplace is not available from most supported sites, so that's out for now. Place of death is even worse available.., so out as well.
Photo: I'm looking into that.
Tags: I'd like to add but can't promise I will. OpenLibrary has "top_subjects" which quite like, but none of the other sites provides something like this.
Bio: is out, it would become quickly outdated (at least for living authors) and with the "View-on" links not actually needed imho.
I also came across OpenLibrary actually providing "List of titles/honorifics", e.g. "Ph.D." etc... but again, none of the other sites provide this.
Short term plan:
- see what I can do on a photo
- release a new version with the extended "resolver" capacities.
Thanks very much @tfonteyn for the updates and continued progress.
I do want to take a look (when i have time) at how the resolvers are written. A really good source for Arabic books (which is the bulk of what I own) is Shamela. It does certainly have a good structure for author information, including books written, and for each book the release dates, publisher information, "investigator" (i.e. the person who checks all the different manuscripts), alongside obviously the book's contents.
Unfortunately, they dont have an API endpoint, so would need to parse the HTML. I'm also not the most familar with java (i slightly touched it over 5 years ago and decided on using Flutter), but I will definitely try to contribute what i can.
- An author photo has now been added.
- Goodreads now has an Author resolver as well; can be combined with the OpenLibrary one via settings. (All sites can use all/multiple author resolvers, but it does not make sense to do this in every combo).
Would you be interested in providing an Arabic translation? If so, I can give you some pointers. It would be a new challenge for me to support Right-To-Left script, but I like a challenge :)
A very brief summary what is needed to add sites.
Searching for books needs a "SearchEngine" which is one or more interfaces from https://github.com/tfonteyn/NeverTooManyBooks/blob/dev/app/src/main/java/com/hardbacknutter/nevertoomanybooks/searchengines/SearchEngine.java
Resolving Authors is really the same logic, but kept in a separate class/interface so cross-use between sites can be done. https://github.com/tfonteyn/NeverTooManyBooks/blob/dev/app/src/main/java/com/hardbacknutter/nevertoomanybooks/searchengines/AuthorResolver.java
I have a list of steps documented to add a "SearchEngine" in the class docs here: https://github.com/tfonteyn/NeverTooManyBooks/blob/dev/app/src/main/java/com/hardbacknutter/nevertoomanybooks/searchengines/EngineId.java
A very simple example of a "SearchEngine" is BookFinder. The site is not enabled in app releases because it's too limited, but I use it for testing. https://github.com/tfonteyn/NeverTooManyBooks/blob/dev/app/src/main/java/com/hardbacknutter/nevertoomanybooks/searchengines/bookfinder/BookFinderSearchEngine.java
An example of an Author resolver: https://github.com/tfonteyn/NeverTooManyBooks/blob/dev/app/src/main/java/com/hardbacknutter/nevertoomanybooks/searchengines/databazeknih/DatabazeKnihAuthorResolver.java
As for parsing the html, that's provided as a base class for the "SearchEngine" implementation: extends JsoupSearchEngineBase
But... the real work start using the "selectors" from the JSoup library: https://jsoup.org/cookbook/extracting-data/selector-syntax
May look daunting :/ A lot of the code is similar/same between the engines thought. The jsoup selector parsing and extracting (often with regex's) is the tedious work. Firefox/Chrome "Inspect page" is your first stop obviously to understand the page you want to parse.
I'm happy to help obviously, but non-latin scripts it not easy for me. When I did the Chinese Douban site, I had help from a Chinese speaker who did the first investigation, and frankly speaking the douban site was fairly well structured. Other sites are not.
here you go 7.4.0
I think that's the major author-metadata enhancements done for now. But as usual, I will make improvements if/when needed.
I'm still leaving this issue open, as it's a good discussion.
Appreciate it.
Have played around with it; does indeed look very promising, many thanks for that.
Regarding the arabic translation, i can definitely help there.
What i will do is give a run down of the different endpoints, whats returned, the selectors, and how to parse what we need.
All of the below ive taken from devtools, so you should be able to just copy paste the js direct into console and replicate/play around with it. The website homepage is https://shamela.ws/.
Searching authors:
fetch("https://shamela.ws/ajax/author/?term=ابن+هشام&_type=query&q=ابن+هشام", {
method: "GET",
});
Three query params: _type which will equal query, then term and q, both of which will equal the author name.
Response:
{
"results": {
"items": [
{
"id": "893",
"text": "ابن هشام النحوي"
},
{
"id": "2881",
"text": "اللخمي، ابن هشام"
}
]
}
}
The text in each item is the authors name, and the associated ID.
To resolve the IDs, this is where we receive html instead of a JSON response.
Its a simple GET:
fetch("https://shamela.ws/author/893", {
"method": "GET",
});
Which returns a massive html blob.
Relevant bits from it: Author's description:
document.querySelector('#wrapper > section:nth-child(4) > div > div > div > div.alert')
This is unfortunately not really that structured, although it does contain some good parseable information. I will come back to this and show how it can be parsed once ive sampled a few different ones.
And his writings:
document.querySelector('#cat_books')
Which returns a list of divs, each of which is like so:
<div class="book_item margin-top-10">
<a href="https://shamela.ws/book/7501">
<b>
<span class="text-primary book_title">أسئلة وأجوبة في إعراب القرآن</span>
</b>
</a>
<p class="margin-top-3 margin-bottom-6 des">الكتاب: أسئلة وأجوبة في إعراب القرآن
المؤلف: عبد الله بن يوسف بن أحمد بن عبد الله ابن يوسف، أبو محمد، جمال الدين، ابن هشام (ت ٧٦١هـ)
المحقق: محمد نغش
الناشر: عمادة البحث العلمي بالجامعة الإسلامية، المدينة المنورة، المملكة العربية السعودية
الطبعة: الأولى، ١٤٠٣هـ/١٩٨٣م
عدد الصفحات: ٦٨
[ترقيم الكتاب موافق للمطبوع]
</p>
</div>
A list of books can be returned like so:
[...document.querySelector('#cat_books').querySelectorAll('#cat_books > div > a > b > span')].map(x => x.innerText)
returns:
[
"أسئلة وأجوبة في إعراب القرآن",
"أوضح المسالك إلى ألفية ابن مالك - ت هبود",
"اعتراض الشرط على الشرط",
"المسائل السفرية",
"تخليص الشواهد وتلخيص الفوائد",
"حاشيتان لابن هشام على ألفية ابن مالك",
"رسالة المباحث المرضية",
"شرح شذور الذهب لابن هشام",
"شرح قطر الندى وبل الصدى",
"متن شذور الذهب",
"متن قطر الندى وبل الصدى",
"مغني اللبيب عن كتب الأعاريب",
"نكت الإعراب = قواعد الإعراب ونزهة الطلاب"
]
A list of the urls to get the books, which contains the book id:
[...document.querySelector('#cat_books').querySelectorAll('#cat_books > div > a')].map(x => x.href)
Returns:
[
"https://shamela.ws/book/7501",
"https://shamela.ws/book/11825",
"https://shamela.ws/book/6978",
"https://shamela.ws/book/6988",
"https://shamela.ws/book/17916",
"https://shamela.ws/book/560",
"https://shamela.ws/book/6991",
"https://shamela.ws/book/6969",
"https://shamela.ws/book/6970",
"https://shamela.ws/book/96770",
"https://shamela.ws/book/11376",
"https://shamela.ws/book/6972",
"https://shamela.ws/book/5352"
]
And finally, a list of descriptions for each book:
[...document.querySelector('#cat_books').querySelectorAll('#cat_books > div > p')].map(x => x.innerText)
Returns:
[
"الكتاب: أسئلة وأجوبة في إعراب القرآن المؤلف: عبد الله بن يوسف بن أحمد بن عبد الله ابن يوسف، أبو محمد، جمال الدين، ابن هشام (ت ٧٦١هـ) المحقق: محمد نغش الناشر: عمادة البحث العلمي بالجامعة الإسلامية، المدينة المنورة، المملكة العربية السعودية الطبعة: الأولى، ١٤٠٣هـ/١٩٨٣م عدد الصفحات: ٦٨ [ترقيم الكتاب موافق للمطبوع]",
"الكتاب: أوضح المسالك إلى ألفية ابن مالك المؤلف: جمال الدين، أبو محمد، عبد الله بن يوسف بن أحمد بن عبد الله بن يوسف، ابن هشام (ت ٧٦١ هـ) حققه وعلّق عليه: بركات يوسف هبود وسَمّى عَمَلَه: مصباح السالك إلى أوضح المسالك راجعه: يوسف الشيخ محمد البقاعي الناشر: دار الفكر للطباعة والنشر والتوزيع - بيروت الطبعة: - عدد الأجزاء: ٤ [ترقيم الكتاب موافق للمطبوع]",
"الكتاب: اعتراض الشرط على الشرط المؤلف: عبد الله بن يوسف بن أحمد بن عبد الله ابن يوسف، أبو محمد، جمال الدين، ابن هشام (ت ٧٦١هـ) المحقق: د. عبد الفتاح الحموز الناشر: دار عمار - الأردن الطبعة: الأولى، ١٤٠٦هـ ١٩٨٦م عدد الصفحات: ٥٤ [ترقيم الكتاب موافق للمطبوع]",
"الكتاب: المسائل السفرية في النحو المؤلف: عبد الله بن يوسف بن أحمد بن عبد الله ابن يوسف، أبو محمد، جمال الدين، ابن هشام (ت ٧٦١ هـ) المحقق: د. حاتم صالح الضامن [ت ١٤٣٤ هـ] الناشر: مؤسسة الرسالة - بيروت الطبعة: الأولى، ١٤٠٣ هـ - ١٩٨٣ م عدد الصفحات: ٤٠ [ترقيم الكتاب موافق للمطبوع]",
"الكتاب: تخليص الشواهد وتلخيص الفوائد المؤلف: جمال الدين أبو محمد عبد الله بن يوسف بن هشام الأنصاري (ت: ٧٦١ هـ) المحقق: د. عباس مصطفى الصالحي (كلية التربية - بغداد) الناشر: دار الكتاب العربي الطبعة: الأولى، ١٤٠٦ هـ - ١٩٨٦ م أعده للشاملة: فريق رابطة النساخ برعاية (مركز النخب العلمية) [ترقيم الكتاب موافق للمطبوع] عدد الصفحات: ٥١٥",
"الكتاب: حاشيتان من حواشي ابن هشام على ألفيَّة ابن مالك، دراسةً وتحقيقًا المؤلف: ابن هشام الأنصاري (ت ٧٦١ هـ) تحقيق: جابر بن عبد الله بن سريِّع السريِّع رسالة: دكتوراه، قسم اللُّغويَّات - كليَّة اللُّغة العربيَّة - الجامعة الإسلاميَّة بالمدينة المنوَّرة إشراف: د إبراهيم بن صالح العوفي العام الجامعي: ١٤٣٩ - ١٤٤٠ هـ عدد الأجزاء: ٢ [ترقيم الكتاب موافق للرسالة الجامعية]",
"الكتاب: المباحث المرضية المتعلقة بـ (من) الشرطية المؤلف: عبد الله بن يوسف بن أحمد بن عبد الله ابن يوسف، أبو محمد، جمال الدين، ابن هشام (ت ٧٦١هـ) المحقق: الدكتور مازن المبارك الناشر: دار ابن كثير - دمشق / بيروت الطبعة: الأولى، ١٤٠٨هـ - ١٩٨٧م عدد الصفحات: ٦٦ [ترقيم الكتاب موافق للمطبوع]",
"الكتاب: شرح شذور الذهب في معرفة كلام العرب المؤلف: عبد الله بن يوسف بن أحمد بن عبد الله ابن يوسف، أبو محمد، جمال الدين، ابن هشام (ت ٧٦١هـ) المحقق: عبد الغني الدقر الناشر: الشركة المتحدة للتوزيع - سوريا سنة النشر: عدد الصفحات: ٦٠٢ [ترقيم الكتاب موافق للمطبوع]",
"الكتاب: شرح قطر الندى وبل الصدى المؤلف: أبو محمد، عبد الله، جمال الدين بن هشام الأنصاري (ت ٧٦١ هـ) المحقق: محمد محيى الدين عبد الحميد [ت ١٣٩٢ هـ] الطبعة: الحادية عشرة للمحقق ١٣٨٣ هـ- ١٩٦٣ م تمتاز بدقة الضبط والزيادة في الشروح والتحقيقات الناشر: المكتبة التجارية الكبرى بمصر طبع: مطبعة السعادة بمصر عدد الصفحات: ٣٣٥ [ترقيم الكتاب موافق للمطبوع]",
"الكتاب: متن شذور الذهب المؤلف: عبد الله بن يوسف بن أحمد بن عبد الله ابن يوسف، أبو محمد، جمال الدين، ابن هشام (ت ٧٦١هـ) الناشر: مطبعة مصطفى البابي الحلبي الطبعة: الأخيرة عدد الصفحات: ٣٤ أعده للشاملة: أبو ياسر الجزائري [ترقيم الكتاب موافق للمطبوع وهو مشكول]",
"الكتاب: متن قطر الندى وبل الصدى المؤلف: جمال الدين، عبد الله بن يوسف بن هشام النحوي (ت ٧٦١ هـ) الناشر: دار الصميعي للنشر والتوزيع الطبعة: الأولى ١٤٢٠ هـ - ١٩٩٩ م عدد الصفحات: ٢٦ [ترقيم الكتاب موافق للمطبوع]",
"الكتاب: مغني اللبيب عن كتب الأعاريب المؤلف: عبد الله بن يوسف بن أحمد بن عبد الله ابن يوسف، أبو محمد، جمال الدين، ابن هشام (ت ٧٦١هـ) المحقق: د. مازن المبارك / محمد علي حمد الله الناشر: دار الفكر - دمشق الطبعة: السادسة، ١٩٨٥ عدد الصفحات: ٩١٨ [ترقيم الكتاب موافق للمطبوع]",
"الكتاب: قواعد الإعراب ونزهة الطلاب المؤلف: جمال الدين أبو محمد عبد الله بن يوسف بن هشام الأنصاري (ت ٧٦١ هـ) المحقق: الشبراوي بن أبي المعاطي المصري الحسني الناشر: دار الريادة للنشر والتوزيع الطبعة: الأولى، ١٤٤٢ هـ - ٢٠٢١ م عدد الصفحات: ٦٦ [ترقيم الكتاب موافق للمطبوع]"
]
The above descriptions are indeed structured, and contain a lot of data. Its pretty much in the format key: value, except the issue is that theres nothing separating each key value pair.
Instead, this detail can be returned a bit better when querying a specific book id (which i assume would be what you will do anyway).
To query for a book name:
fetch("https://shamela.ws/ajax/book/?term=شرح+قطر+الن&_type=query&q=شرح+قطر+الن", {
method: "GET",
headers: {}
});
Similar to the author querying, the query params term and q are both identical and contain the book name you want to search, and _type is query.
Response:
{
"results": {
"items": [
{
"id": "2135",
"text": "تعجيل الندى بشرح قطر الندى"
},
{
"id": "6970",
"text": "شرح قطر الندى وبل الصدى"
}
]
}
}
To query a book id (which can be returned either from the author's page or the above book name query):
fetch("https://shamela.ws/book/6970", {
"method": "GET"
});
Which similarly returns a html blob.
The main info is accessible from here:
document.querySelector('#wrapper > section.padding-top-20 > div > div > div > div')
To get the author:
document.querySelector('#wrapper > section.padding-top-20 > div > div > div > div').querySelector('div > a')
Which returns:
<a href="https://shamela.ws/author/893">ابن هشام النحوي</a>
The href contains a url to get the author (with the author id at the end), and the innerText contains the author's name.
Now this part is really dirty. It returns it in a not so structured way at all unfortunately. Its basically a h3, followed by some text sandwiched between line breaks (which is the text we need), followed by some divs.
Probably the best and most reliable way to get things is something like the following:
[...document.querySelector('#wrapper > section.padding-top-20 > div > div > div > div').childNodes]
.filter(x => x.nodeType === Node.TEXT_NODE)
.map(x => x.textContent.trim())
.filter(x => x.includes(':'))
.reduce((acc, line) => (([key, value] = line.split(':').map(s => s.trim())), acc[key] = value, acc), {})
To explain, i return a list of all the text nodes, trim them, check that they do contain a colon (which will also exclude empty lines), then split into a key value using the colon as a delimeter.
This returns:
{
"الكتاب": "شرح قطر الندى وبل الصدى",
"المؤلف": "أبو محمد، عبد الله، جمال الدين بن هشام الأنصاري (ت ٧٦١ هـ)",
"المحقق": "محمد محيى الدين عبد الحميد [ت ١٣٩٢ هـ]",
"الطبعة": "الحادية عشرة للمحقق ١٣٨٣ هـ- ١٩٦٣ م تمتاز بدقة الضبط والزيادة في الشروح والتحقيقات",
"الناشر": "المكتبة التجارية الكبرى بمصر",
"طبع": "مطبعة السعادة بمصر",
"عدد الصفحات": "٣٣٥"
}
Translation of the keys:
{
"الكتاب": "Book",
"المؤلف": "Author",
"المحقق": "Editor",
"الطبعة": "Edition",
"الناشر": "Publisher",
"طبع": "Printed By",
"عدد الصفحات": "Number of Pages"
}
And for the values for author and editor, we have the dates of death within the (round and/or square) brackets (only if the person has indeed died). It is only the year (i would assume because theres usually a fair bit of uncertainty when it comes to month and day). The year is also only in Hijri, so would need converting (I did come across https://docs.oracle.com/javase/8/docs/api/java/time/chrono/HijrahDate.html which looks like its supported by default).
And finally, we have the category:
document.querySelector('#wrapper > section.page-header.page-header-sm > div > ol > li:nth-child(3) > a').innerText
Any questions or parts you would like further information on do let me know.
a number of bugfixes in the author-metadata: 7.5.0
Thanks for all your docs. At first glance it's looking very difficult for me, not knowing Arabic obviously.
As that site supports searching authors by name, and by id, a better starting point might be:
https://github.com/tfonteyn/NeverTooManyBooks/blob/dev/app/src/main/java/com/hardbacknutter/nevertoomanybooks/searchengines/openlibrary/OpenLibraryAuthorResolver.java
That one supports both as well, and uses an external parser. It might be easier for you to copy the OL resolver and as a first step make it retrieve the actual page(s) with author data. As for the SearchEngine that class uses, I think that for first tests you can just (ab)use the OL one, as all the resolver uses is "createGetDocumentRequest" which is generic anyhow (can later b replaced by a direct invocation)
I have some probably unexpected good news.
I've been able to implement an "AuthorResolver" which can get its data from WikiData, which is basically the same data as Wikipedia. I'll do a proper documentation piece on this later, but in a nutshell: I will expand on the above and the concept of "Identifiers", i.e. author-id's as used by several different websites, to link them up with WikiData. And WikiData already keeps tracks of "Shamela" author id's!
Any user can already define new identifiers without me having to do anything, but if I also allow hooking those up with wikidata automatically, then you'll actually get Shamela author info linked up without the need of parsing the Shamela site.
It'll take a couple of weeks probably, I might be away later this month.
Here you go: 7.6.0 - this adds the WikiData author-resolver.
I'll close this one now as I think I've covered all I needed to cover. Feel free to log a new issue if you want to go ahead with an actual Shamela engine and/or an Arabian translation. For the latter, I've done some quick tests for R2L script, and it will involve quite some work. I'm willing to tackle it if a translation is a certainty only of course.