Make quote renotes work with other Fediverse software

[tastytea]

Summary

Quote-renotes are shown as a reply to the quoted note in Misskey, but users of Mastodon and Pleroma don't see the reply and don't get a notification. One of the Mastodon devs suggested that Misskey should change the activity it sends. I don't know much about this stuff, so I'll just link the post here: https://github.com/mastodon/mastodon/issues/18473#issuecomment-1133799674

[tamaina]

Quote is already implemented in Mastodon forks such as Fedibird, which can be federated with Misskey. However, Fedibird does not notify quotes.

[tamaina]

I wonder if inReplyTo can't be an array?

[tamaina]

I'm against implementing this because I think there will definitely be complaints about notifying quotes.

[acid-chicken]

there's some incorrect stuff going on with the JSON-LD bits (for example, quoteUrl is not adopted as part of the activtiypub namespace) but again, i think the onus should be on Misskey^ to add a Mention tag of the quoted post's author or something similar if they expect it to generate a notification.

The quoteUrl is not an extension property invented by Misskey, but was added for compatibility with other Mastodon forks (previously only _misskey_quote in the misskey namespace was used).

https://github.com/misskey-dev/misskey/pull/5632

it might also be a good idea for misskey to consider using inReplyTo so that it actually creates a thread and not an onion of nested links, but that's another topic...

Since quote renote is not to be used with the primary intention of throwing a remark at the original user (unless the system should inform the original user that they was quoted), the original user's Actor Object should not be put into to in Activity (cc would be more appropriate), and using inReplyTo may deliver the Activity to other software with the wrong intent. I think there should be some other property, even if the behavior is adjusted for better standardization.

[Johann150]

I wonder if inReplyTo can't be an array?

According to the ActivityStreams 2.0 specification:

inReplyTo Indicates one or more entities for which this object is considered a response. Range: Object | Link

• https://www.w3.org/TR/activitystreams-vocabulary/#dfn-inreplyto

So on the one hand it says that it may contain multiple elements, but on the other hand Range says it may contain an object or a link. For other values it is explicitly stated that arrays are allowed so I think arrays are not allowed here. We would have to use a collection or something like that and I doubt that it would be well supported.

[Johann150]

In my opinion, acid-chicken's proposal is reasonable to put the quoted user as cc, because if another user is on Misskey, they also receive a notification about the quote.

In the Mastodon issue, it was proposed to use a Mention in the tag field but I'm concerned that if we implement that, it would become unclear to Misskey itself whether the mention was an actual mention or "just" a quote. From my testing of quoting and mentioning the quoted user at the same time, the notification is displayed as if it was a mention. The duplicate quote notification is not created.

[rinsuki]

FYI, Mastodon's REST API seems doesn't support multiple inReplyTo... https://docs.joinmastodon.org/entities/status/#in_reply_to_id

also I think it would be good to notify as boost/renote instead of reply. (but also I think it is hard...)

[tamaina]

also I think it would be good to notify as boost/renote instead of reply. (but also I think it is hard...)

→ #2264

[syuilo]

The way to make Misskey quotes work in other software is for the other software to support the concept of quotes.

Until other software supports this, there is an option to send the activity as a Renote treatment, but it is wrong to send it as a reply. This is because a quote is not a reply.

[acid-chicken]

FYI Also; fedibird uses quoteUri (not quoteUrl) and _misskey_quote: https://github.com/fedibird/mastodon/blob/398931ef101670902e47062c2e0451f0bba139d6/app/lib/activitypub/activity/create.rb#L568

[trwnh]

The way to make Misskey quotes work in other software is for the other software to support the concept of quotes.

@syuilo How do you define the concept of quotes? I can agree a quote is not a reply -- the point I was trying to make is that replies are better than quotes in most situations because they keep the same context

I think the real issue is whether Misskey quotes should send a notification or generate one in other software. There are some options I cover in https://github.com/mastodon/mastodon/issues/18473#issuecomment-1133976969

1. Put Mention in tag: this is most compatible with existing software to generate a notification, but is maybe not semantically correct?

2. Put Link in tag: this is semantically more correct if the link continues to be attached in content, similar to a Mention or Hashtag

3. Put Note in attachment: this would be an alternative to option 2, basically treating the quoted post the same as an attached image or video or poll

4. Add content to Announce: this would emphasize the boosting aspect of it, with the issue of the fallback being a silent boost/renote

[Johann150]

How do you define the concept of quotes?

In Misskey's implementation a quote is an ordinary Note that displays an embed of the quoted Note.

4. Add content to Announce: this would emphasize the boosting aspect of it, with the issue of the fallback being a silent boost/renote

I think this is not acceptable because it degrades worse than the current approach of embedding a link in the content property other ActivityPub implementations will normally use. In my opinion, the emphasis of a quote is not the renote aspect. Even the current Pleroma and Mastodon renderings are already kind of implementing quotes this way with link previews. There was an issue (#8392) raised about this when the link previews broke a little while back, so people do "use" it this way. This also makes option 2 seem the most attractive to me out of the ones you presented.

I'm not sure why the current implementation seems to be completely out of the picture, seeing that it is seemingly starting to be adopted by other ActivityPub implementations (Fedibird, Pleroma, as I also laid out on the Mastodon issue).

[trwnh]

Even the current Pleroma and Mastodon renderings are already kind of implementing quotes this way with link previews

I don't think quoting and link previews are the same thing at all -- the point of a preview is to show a little bit of information before you click a link, but the point of a quote seems to be to have some external content embedded in your object (so that you don't have to click a link)

I'm not sure why the current implementation seems to be completely out of the picture

Because the current implementation (introducing a new quoteUrl property) already has several problems with it:

• It is yet another property being added onto an object, instead of using existing properties
  • There are no expected or clearly-defined semantics for this property
• It is not even agreed upon what the property name should be, as Fedibird uses quoteUri instead of quoteUrl, and Pleroma doesn't seem keen to adopt that feature in its current form (lain expressed that they wish it used Announce instead)
• It is tacked onto the ActivityStreams namespace despite not actually being adopted as part of the ActivityStreams namespace

The 4 options I laid out above are not the only options, of course, but they are the ones that make sense for software other than Misskey. Additionally, choosing one of those 4 options has the inherent benefit of establishing semantics simply in the way the existing properties are already defined -- we already know and agree what content, tag, attachment mean. We know what an Announce is versus a Create. it is not immediately clear what a quoteUrl or quoteUri means, or what side effects it should have.

Beyond the points above, it is up to Misskey as a project to determine what is the best approach to follow. It is then up to other projects to determine how they want to handle the received activities that are being generated by Misskey.

If you want my personal opinion, I think all 4 options are ultimately fine, but they express different meanings. Therefore, it is important to determine what exactly a "quote" means, in order to choose the option that best fits that meaning.

[rinsuki]

FYI: as:quoteUrl is compatibility for some Mastodon forks; we have a modern properties https://misskey-hub.net/ns#_misskey_quote.

I thinking about new option: Create Note (like current), but also Announce original note with link to created Note; If implementation supports quote note, treat Announce as Create Note.

but it makes twice notification on old Misskey servers.

like:

[{
    "type": "Announce", 
    "id": "https://misskey.example/notes/bbbbb/activity/announce",
    "_misskey_note": "https://misskey.example/notes/bbbbb",
    "object": "https://misskey.example/notes/aaaaa",
}, {
    "type": "Create",
    "id": "https://misskey.example/notes/bbbbb/activity",
    "object": "https://misskey.example/notes/bbbbb",
}]

[trwnh]

It should be possible to use type: [Create, Announce], no need to make it two activities

In such a case, the Announce will generate a notification for a boost/renote, and the Create will generate a status in the timeline if the recipient is following the author

[rinsuki]

but "object" property will be conflict... also I'm not sure current servers implementation is supporting multiple types.

[trwnh]

anything other than Mention in tag will probably be unsupported by current servers, but i see what you were suggesting now... i'm not sure if it's better or worse than what is currently done, though

[Johann150]

I think as a transition measure we could use option 3 of putting the note as an attachment in addition to the current fields. Then when there is enough support for that we could remove the quoteUrl and/or _misskey_quote fields.

[trwnh]

on topic of attachment vs tag i think tag is more appropriate after thinking more about this. attachment is supposed to be inclusion, whereas tag is a reference.

since the fallback content literally uses RE: then i assume the intention of a quote is to make a reference to the original link

the payload would probably look something like this

{
"@context": [...],
"type": "Note",
"content": "this has a @mention and a #hashtag but is also a quote RE: http://domain.tld/some-note",
"tag": [
  {
    "type": "Mention",
    "href": "http://domain.tld/some-person",
    "name": "@mention"
  },
  {
    "type": "Hashtag",
    "href": "http://domain.tld/some-hashtag",
    "name": "#hashtag"
  },
  {
    "type": "Link",
    "href": "http://domain.tld/some-note",
    "name": "RE: http://domain.tld/some-note"
  }
],
...
}

the general idea of a tag is you look for name within content and link it to something else. there's probably a few extra steps involved for html content instead of plain-text content

there is also the option of an alternative representation referring to the id of the object directly, which may be better since you can access properties of the original object like attributedTo for addressing

"tag": [
  {
    "id": "http://domain.tld/objects/47309817392013",
    "type": "Note"
  }
],
...

or even

"tag": [ "http://domain.tld/objects/47309817392013"],
...

i can imagine this would work for non-Note objects, too? although that's a bit more fuzzy...

[Johann150]

In the Misskey web client, a quote goes in exactly the same place an image goes, is thus "included" and could be seen as an attachment from that perspective. Going by ActivityStreams 2.0 § 5.6 Mentions, Tags and Other Common Social Microsyntaxes, even though non-normative the section make use of tag extensively so I guess it would make sense too in this case.

The only concern with the Link from my side is that a quote is not just any kind of link and processors have to pay special attention (if they support quotes) so I would prefer the type: Note style of tag.

[Johann150]

Hmm, using type: Note would imply embedding the whole note in there, if that is also a quote it gets kind of recursive and complicated. So maybe using a Link is better. To easily detect that this is a special kind of link we could use the rel attribute. Taking your example above we would end up with something like this:

{
  "type": "Note",
  "id": "https://misskey.example/note/bbbbb",
  "content": "this has a @mention and a #hashtag but is also a quote\n\nRE: http://misskey.example/note/aaaaa",
  "tag": [
    {
      "type": "Mention",
      "href": "http://misskey.example/@mention",
      "name": "@mention"
    },
    {
      "type": "Hashtag",
      "href": "http://misskey.example/tag/hashtag",
      "name": "#hashtag"
    },
    {
      "type": "Link",
      "rel": "https://misskey-hub.net/ns#_misskey_quote",
      "mediaType": "application/activity+json",
      "href": "https://misskey.example/notes/aaaaa",
      "name": "RE: https://misskey.example/notes/aaaaa"
    }
  ]
}

[macgirvin]

It seems I'm late to this discussion. FWIW my projects use an attachment of type application/activity+json. It is treated exactly the same as image and video attachments. In my view this is absolutely no different than adding an inline attachment for a photo or even more appropriately - an opengraph or OEmbed representation of an an external webpage. Plaintext projects render the attachments on the receiving end and add this rendering to the displayed post. HTML projects include it already and only provide the attachment for the benefit of the plaintext platforms, so they can render it appropriate to their needs/requirements (since these projects generally remove it from the HTML content). I suppose this is yet another place where every project has to support an ever increasing number of completely different ways of doing every little thing.

[Johann150]

As presented above the difference is in the semantics of attachment and tag: If I quote something that does not mean I "made" it, I just refer to it.

Could you maybe give an example of how your representation would look for the example I gave above?

[macgirvin]

I don't always make the photos or videos I attach to a post. I just refer to them in a manner that the plaintext platforms normally use to generate the img or video link which they stripped out of the rendered HTML that I already provided.

Here is one example...

{
    "type": "Create",
    "id": "https://streamz.netzone.cc/item/87f39845-4138-411a-8b74-f9faf69f8d04",
    "published": "2022-06-23T19:51:01Z",
    "context": "https://streamz.netzone.cc/item/87f39845-4138-411a-8b74-f9faf69f8d04",
    "actor": "https://streamz.netzone.cc/channel/dirty",
    "replyTo": "https://streamz.netzone.cc/channel/dirty",
    "url": "https://streamz.netzone.cc/item/87f39845-4138-411a-8b74-f9faf69f8d04",
    "object": {
        "type": "Note",
        "id": "https://streamz.netzone.cc/item/87f39845-4138-411a-8b74-f9faf69f8d04",
        "published": "2022-06-23T19:51:01Z",
        "commentPolicy": "contacts",
        "attributedTo": "https://streamz.netzone.cc/channel/dirty",
        "context": "https://streamz.netzone.cc/conversation/87f39845-4138-411a-8b74-f9faf69f8d04",
        "content": "Awesome to see the fediverse bring people together. Community in the true sense of the word. @<span class=\"h-card\"><a class=\"zrl u-url mention\" href=\"https://unfediverse.com/channel/mike\"  target=\"_blank\"  rel=\"nofollow noopener\" >mike</a></span> that has to bee cool to see your services are instrumental in building relationships. This is why I love tech! <br><br><div class=\"shared_container\"> <div class=\"shared_header\"><a href=\"https://zap.dog/channel/haakon\" ><img src=\"https://streamz.netzone.cc/xp/3763fb59b15d76169d288ace23892ebbf9d0b7f061c292ec96e22b2e17190be7-6\" alt=\"Haakon Meland Eriksen (Zap Dog)\" height=\"32\" width=\"32\" /></a><span><a href=\"https://zap.dog/channel/haakon\" ><bdi>Haakon Meland Eriksen (Zap Dog)</bdi></a> wrote the following <a href=\"https://zap.dog/item/481556c7-3e20-4993-aea6-f15cd4ca94f2\" >post</a> <span class=\"autotime\" title=\"2022-06-23T14:32:36-04:00\" >Thu, 23 Jun 2022 14:32:36 -0400</span></span></div><div class=\"reshared-content\"><strong>Metal, oil and BBQ - the usual suspects</strong><br><a class=\"zrl\" href=\"https://zap.dog/photos/haakon/image/2d8321f3-0d8b-42eb-ae78-d856d0e0f8d2\"  target=\"_blank\"  rel=\"nofollow noopener\" ><img class=\"zrl\"  style=\"width: 100%; max-width: 1024px; \" alt=\"20220623_201206.jpg\" title=\"20220623_201206.jpg\" src=\"https://zap.dog/photo/2d8321f3-0d8b-42eb-ae78-d856d0e0f8d2-1.jpg\" ></a><br><br>We had a great afternoon here in Norway, thanks to @<span class=\"h-card\"><a class=\"zrl u-url mention\" href=\"https://hub.volse.no/channel/harald\"  target=\"_blank\"  rel=\"nofollow noopener\" >Harald Eilertsen</a></span>, who took the time to visit us here on this elevated plane of existence, or Konnerud in Drammen as some people call it. :-) We had a BBQ with my two boys, talking about olive oil from @<span class=\"h-card\"><a class=\"zrl u-url mention\" href=\"https://elsmussols.net/channel/elmussol\"  target=\"_blank\"  rel=\"nofollow noopener\" >elmussol</a></span> , metal, Fediverse, privacy and more. Hopefully, we will be able to this again. :-)</div></div>",
        "source": {
            "content": "Awesome to see the fediverse bring people together. Community in the true sense of the word. @[zrl=https://unfediverse.com/channel/mike]mike[/zrl] that has to bee cool to see your services are instrumental in building relationships. This is why I love tech! \r\n\r\n[share author='Haakon+Meland+Eriksen+%28Zap+Dog%29' profile='https://zap.dog/channel/haakon' portable_id='vajCAm0tMADbx39LxS2rfGc1OoSwoCu-3h5wnJLSewC5oUr7-n94h9rdMcUwhC_tFdB66gnsC00Zjk6ZaHc6Xg' avatar='https://streamz.netzone.cc/xp/3763fb59b15d76169d288ace23892ebbf9d0b7f061c292ec96e22b2e17190be7-6' link='https://zap.dog/item/481556c7-3e20-4993-aea6-f15cd4ca94f2' auth='false' posted='2022-06-23 18:32:36' message_id='https://zap.dog/item/481556c7-3e20-4993-aea6-f15cd4ca94f2'][b]Metal, oil and BBQ - the usual suspects[/b]\r\n[zrl=https://zap.dog/photos/haakon/image/2d8321f3-0d8b-42eb-ae78-d856d0e0f8d2][zmg width=\"1024\" height=\"1018\" alt=\"20220623_201206.jpg\"]https://zap.dog/photo/2d8321f3-0d8b-42eb-ae78-d856d0e0f8d2-1.jpg[/zmg][/zrl]\n\nWe had a great afternoon here in Norway, thanks to @[zrl=https://hub.volse.no/channel/harald]Harald Eilertsen[/zrl], who took the time to visit us here on this elevated plane of existence, or Konnerud in Drammen as some people call it. :-) We had a BBQ with my two boys, talking about olive oil from @[zrl=https://elsmussols.net/channel/elmussol]elmussol[/zrl] , metal, Fediverse, privacy and more. Hopefully, we will be able to this again. :-)[/share]",
            "mediaType": "text/x-multicode"
        },
        "replyTo": "https://streamz.netzone.cc/channel/dirty",
        "url": "https://streamz.netzone.cc/item/87f39845-4138-411a-8b74-f9faf69f8d04",
        "tag": [
            {
                "type": "Mention",
                "href": "https://unfediverse.com/channel/mike",
                "name": "@[email protected]"
            }
        ],
        "attachment": [
            {
                "type": "Image",
                "url": "https://zap.dog/photo/2d8321f3-0d8b-42eb-ae78-d856d0e0f8d2-1.jpg",
                "name": "20220623_201206.jpg"
            },
            {
                "type": "Link",
                "mediaType": "application/activity+json",
                "href": "https://zap.dog/item/481556c7-3e20-4993-aea6-f15cd4ca94f2"
            }
        ],
        "to": [
            "https://www.w3.org/ns/activitystreams#Public",
            "https://unfediverse.com/channel/mike"
        ],
        "cc": [
            "https://streamz.netzone.cc/followers/dirty"
        ]
    },
    "tag": [
        {
            "type": "Mention",
            "href": "https://unfediverse.com/channel/mike",
            "name": "@[email protected]"
        }
    ],
    "attachment": [
        {
            "type": "Link",
            "mediaType": "image/jpeg",
            "href": "https://zap.dog/photo/2d8321f3-0d8b-42eb-ae78-d856d0e0f8d2-1.jpg"
        },
        {
            "type": "Link",
            "mediaType": "application/activity+json",
            "href": "https://zap.dog/item/481556c7-3e20-4993-aea6-f15cd4ca94f2"
        }
    ],
    "to": [
        "https://www.w3.org/ns/activitystreams#Public",
        "https://unfediverse.com/channel/mike"
    ],
    "cc": []
}

[Johann150]

So it seems we do agree in that it should be a Link with mediaType: "application/activity+json". I do not have strong opinions of whether it should be attachment or tag, so I would not mind moving it to attachment, if that means better compatibility with existing software.

[macgirvin]

Reviewing that after posting did point out a couple of minor issues in our attachment encoding, which I've been working on (notably the structural difference in image attachments between the activity and the underlying object).

We also treat the issue of notification as orthagonal; so it's probably appropriate to add a mention in the tag element for the original author if your platform requires this for notifications. We currently do this in some but not all cases where we attach an existing post and didn't do so in this case. I'll need to add this to the remaining use cases for consistency.

[trwnh]

I do not have strong opinions of whether it should be attachment or tag

i do :)

to clarify, attachment does not mean you "made" it, only that it is included in your authored post and not referred to. the reason i used tag above and advocate for the usage of tag is because of the apparent intention behind misskey quotes, which is evident in the use of RE: in the fallback, literally referring to some other post.

i'm not sure what the semantics of "attaching" a quote is supposed to imply in zap's case. perhaps that the content is not to be marked up, but the attachment is to be shown separately? if that is indeed the case, then perhaps zap and misskey have two different concepts of what a "quote" is. although, seeing the generated html in content, i think zap should also use tag since it appears to be tagging the quote in the same way you would tag a person or a concept.

i'm willing to have my mind changed along some practical lines, though, e.g. "attachment and tag have taken on different meanings and should be used in a different way than defined" (maybe attachment is for objects and tag is for substrings of content? but we still have the issue of the pseudo-Mention tag sometimes being necessary for notifying someone, which could be said to be purely a workaround for Mastodon's current notification behavior?)

[silverpill]

If quotes look like attachments in UI, it makes sense to use attachment property. On the other hand, that's just graphical representation: some implementations may show quotes as "attachments", others could do something else. I agree that quotes are a special case of "object mentions". The same mechanism can be used to implement native links/backlinks in fediverse: when you click on such link, it opens in the instance UI instead of a new browser tab, just like clicking on a mention usually opens user's profile and clicking on hashtag usually opens the list of tagged posts.

[silverpill]

The desired graphical representation of a quote can be defined with css classses in post's content

[macgirvin]

As mentioned earlier, we use attachment for inline media attachments of all forms, because plaintext microblogs remove the inline attachments from the HTML content and use the attachment property to add them back to the end. I'm thinking of one plaintext microblog in particular because this is the only way we can federate rich media content to said project , and it is a dominant force in the fediverse which steadfastly refuses to support HTML content.

I don't use misskey so I do not know how it handles HTML. The content was already provided in HTML. If your platform doesn't require the use of attachment to indicate the presence of HTML that was removed from the content by your purifier, you can simply display the HTML and be done with it.

But unfortunately, due to the existence of the afore-mentioned project, this means if you do support HTML content - internally your software also probably requires code to check whether or not an attachment was already referenced in the content so as not to display it twice.

In any event, these are the reasons my software does what it does. Since we support HTML but also have to live with the elephant in the room, this turned out to be the most sensible way forward. Since we already use this mechanism for audio, video, and img media it made the most sense to extend this facility to inline Activity objects because conceptually they are treated exactly the same as the other media objects; the only difference being that they don't already have an existing HTML tag dedicated to them. If these were provided as an 'object' HTML tag inside the content, there would be no difference at all. I personally prefer to embed them server-side due to our platform requirements, though perhaps I should wrap them in an 'article' tag.