Getting Zotero collection names for "smarter" bibliography

[MattHeffron]

Per the discussion in today's meeting, I did some digging around and experimenting and was able to get the Collection ID and name for each of the collection. I was able manually to organize them into the correct hierarchical structure. (I may be able to get that structure from Zotero. TBD.)

I also verified that the JSON file used to generate the Bibliography web page, already includes the Collection IDs for every entry. (It's just that we don't do anything with that, currently.)

Here's the hierarchical set of collection names:

exported to interlisp.org/bibliography
    Other Lisps
        Lisp Machines
            LispMachine (LMI)
            Symbolics
            TexasInstruments
    unsorted documents from current repos(google drive)
    Videos
    Hyperlinks
    Common Lisp
    See Also (articles relevant to Medley Interlisp)
    About Lisp
    About Interlisp
        Interlisp Documentation
            Older releases
                medley 1.0
                harmony
                fugue
                carol
                koto
                lyric
        About Interlisp-D historic
    About Applications written in Interlisp
        Trillium
        Notecards
        Rooms
        Loops
  mention/dIscusses Interlisp

They should match what's in Zotero.

Matt

[masinter]

Thanks; this looks like a good match for what we have. My hope is that for an "annotated bibliography" it should be possible to answer the question "Why is this here?". Only a few categories might require an additional note

The citations in the "unsorted" category need to be sorted. Do they fall into obvious categories? What about things in the CHM PARC archives?

Constraints:

The name of the entire collection of citations:

This is currently the "Interlisp" collection. Even though it has citations that have little to do with Interlisp, but we might want to cite. It isn't possible to "change" the name of the entire collection to be something else. It names the group of people involved, the annual billing for document storage and creating a new collection to changing it is not worth it. Please Live With calling the whole thing "Interlisp" even though there are sub-categories.

Access

There are several ways to access the collection of citations:

1. through web browsing Zotero.org,  This gives you access to the entire collection, with some limitations
2. running the Zotero desktop client. This gives you more capabilities than (1), including a browser "extension" that lets you quickly add entries. but you have to download and install it.
3. For a subset of the collection, we provide a HTML rendition of the bibliography which matches the style of the web site and the ability to present it as a timeline. This is what is served by https://interlisp.org/bibliograhy

There is some uncertainty about what should be in (3) -- the entire collection? Zotero collections are what is called "smart folders": they're just a database query for citations that match a tag, with those citations that are part of more than 1 collection appearing in "Duplicates" and those that are not in any collection appearing as "Unrfiled"

Copyright

My concern about putting copyright material not just as a link to someone else but serving the documents directly is more of a 'perception' than a direct risk. Some in the archival community will not accept works or archives of collections where the organization might find itself liable for copyright violation.  We need to be careful to distinguish the material which we  have license to distribute (medley interlisp source code and documentation) from those were we do not have such permission.  We need to be careful about labelling the source. This goes for the entire project, the content of the various repositories (which have different status than medley, maiko, and Interlisp.github.repos do). This is not something I don't see much value in our trying to defend.

"Fair Use" allows one to make copies for personal use of copyright material, but not to distribute to others. Copyright questions come up often -- many are upset, for example, that other sources aren't available as readily, and the permission to release the sources was an important milestone in the project. We don't have a general solution to the industry problem om of "abandonware". Let's not make ourselves an attractive target by not being careful about what we publish.

I'm working on rewriting the Medley sourcefiles ... another issue. ##vcoordination

Zotero isn't GitHub. There are no locks. It is possible to wipe out someone else's work accidentally, or leave behind a mess without knowing it, or leave behind extra folders.

Multiple people working on the same bibliography with background 'sync' could lead to disaster.

The "collection" folders have a "key" and it is possible to rename the folder -- except for the one at the very top which must rename "Interlisp". I don't think many want to look at the entire collection.

I think we need a plan that we all agree to.

[MattHeffron]

The "Interlisp" level isn't a Collection, it is the Group, and the collections that I listed are below it; so there's no issue with that name. This discussion about getting the collection IDs and names via the Zotero API was really only relating to your point #3 above (the possibility of user-filtering the Bibliography on the Interlisp.org website). Discussion of copyright concerns seems to be orthogonal to this discussion.

[masinter]

Let me propose the following top level collections:

1. Documentation. blue: Interlisp/Medley documentation.
2. About. green: Articles, announcements, videos, papers, primarily
3. Discusses. yellow: contains substantial information about Interlisp in relation to other systems.
4. Applications. orange: Major applications developed in Interlisp.
5. Everything else. red: about other systems, Lisp in general, Common Lisp, other Lisp Machines. Can be sub-collections.

Within each category, each citation should have:

1. Gemre: article, blog post, video, book, presentation slides, recordings. Source code, emulators, compiled applications.
2. Year: (normally derived from date of publication but can be overridden by a 'year of appplicability' So a 'history of interlisp' might have an earlier year. Documentation about earlier releases can be determined by the year of release.
3. "Why is this here?": Every entry in category "Other" should have a note with the reason why it is in the bibliography.

We can catalog current documentation by pointing to the repository where the document or implementation is maintained. We can include sub-collections of other implementations as long as there is a plausible "why is this here".

[masinter]

For a UI to access this, I imagine either checkboxes for each collection or a drop down? The default might be 1-4 or 1-5. You can tell apart genre: documentation year: . I would leave the display to be sorted by (topic-) year, but add the genre name. So Steve's volume 1 of Interlisp: The Language and it's Uses would have a year of 1986, genre book. There might be a pointer into the CHM PARC archives of genre "source code" and the date of the fugue release date.

[pamoroso]

In your earlier post, by category do you mean Zotero collections? I'm not sure what the "Other" category refers to as it's not in the 5 listed collections.

[MattHeffron]

@pamoroso, I interpreted "category" to mean "collection", and "Other" as the collection 5 "Everything else"

[masinter]

Yes, that was my idea, to put everything that doesn't fit into the first 4 collections in a collection "Everything Else". Richard Gabriel's book "Performance of Lisp Systems" would go in collection 3, "Discisses". At least my memory of what I've seen of the bibliography is that mostly what we have will fit into the first 4 collections. Outside of that, we should have a reason for inclusion. If there are many more than a dozen we can make subcollections under "Everything Else" -- but I think the casual viewer will be confused if we don't supply a reason why we are citing something that has little relation to the main topic of the site.

[MattHeffron]

OK, I just created a collection in Zotero: "Alternate Bibliography Hierarchy" with the 5 subcollections as above. (I numbered the subcollections, so they'd be in Larry's ordering.) We can add sub-subcollections as appropriate; or change the whole hierarchy if we want! Since Zotero allows items to be in multiple collections at once, we can use the "Add to Collection >" right mouse menu item to put items into the appropriate subcollection in the new hierarchy without disrupting the existing organization. Then, when we're ready, all we should need to do is to change the collection ID that's used as the root_collection in the scripts/update_bibliography.sh and do the updating. With no other change to the existing update_bibliography.sh or website, that should give essentially the same Bibliography as now. This should allow for incremental development and testing of website changes without "going live".

[MattHeffron]

For adding the "each citation should have" items we need to standardize our way of adding this metadata so it can be pulled apart for building the Bibliography web page.

Also, I do not understand the assignment of colors to the collections... Why?

[MattHeffron]

There was a bit of concurrent discussion about this in the pull request: Interlisp.github.io: Gather the Zotero collections information during scripts/update_bibliography.sh

[masinter]

@ekaltman pls note

[MattHeffron]

@Catillamen & @abhikhasnain1 (& @hjellinek ) This sounds like it evolved into (some of) what you're working on. If you agree, please rename this to more closely match your plan. If not, (i.e., too great a difference) please add a comment below, select the "Close as not planned" in the "Close with comment" down-arrow, and then select the "Close with comment" button itself.