[Enhancement] Expand scope so that it can take notes from video and audio, both online and offline

[auroranil]

I had this idea where you can use org-noter to make notes from videos and audios played by a media player. It would be able to create notes and extract current timestamp of media playing in another window. I am considering making this feature, but it would take me weeks to get to speed with Emacs Lisp, and was wondering if someone else have any ideas in making this possible.

Online media content

For online media content, we could consider interacting with HTMLMediaElement, and use these particular properties/methods/events:

• HTMLMediaElement.currentTime property
• HTMLMediaElement.paused property
• HTMLMediaElement.fastSeek() method
• HTMLMediaElement.pause() method
• HTMLMediaElement.play() method
• timeupdate event

A browser extension would probably be needed so that we can interact with it from outside. I can think of two approaches which I can use for passing messages b/w Emacs and browser:

1. Native messaging
2. WebSockets

Offline media content

For offline media content, we could consider using an existing media player such as VLC

1. via dbus. VLC conforms to MPRIS specification.
2. via RC interface: https://github.com/mohkale/vlc-rc

Commands

Org Noter commands

We can utilise these following existing commands:

• org-noter opens media player at timestamp where user viewed video/audio last
• org-noter-insert-note inserts heading including timestamp in :PROPERTIES: drawer, down to seconds resolution
• org-noter-insert-precise-note inserts heading including timestamp but in milliseconds resolution
• org-noter-sync-current-note jumps to timestamp at current heading
• org-noter-sync-next-note jumps to timestamp at next heading
• org-noter-sync-previous-note jumps to timestamp at previous heading
• org-noter-kill-session kills session, which also closes media player

All other commands are unsupported for this MVP.

Media player commands

Additional commands are needed for interacting with media player within Emacs. I can think of a few so far:

• org-noter-media-jump-to-timestamp Prompts user to type in timestamp to jump to
• org-noter-media-toggle-play toggles play/pause media
• seek back or forward slightly

Does this software already exist?

To my knowledge, there is no org note taker for videos/audios. This section from "How to annotate literally everything" article says that no such note taking software exists.

These links are relevant and may be able to get this feature working:

• Bi-directional communication using WebSockets: https://github.com/alpha22jp/atomic-chrome
• Edit subtitles for video: https://github.com/rndusr/subed
• Video command primitives: https://github.com/skeeto/transcription-mode
• Video editor for Emacs: https://www.emacswiki.org/emacs/GNEVE
• org-protocol: https://orgmode.org/worg/org-contrib/org-protocol.html

[auroranil]

Example

* Big Buck Bunny - Blender Short Film - YouTube video
  :PROPERTIES:
    :NOTER_LINK: https://www.youtube.com/watch?v=aqz-KE-bpKQ
    :NOTER_CSS_SELECTOR: .video-stream
    :NOTER_TIMESTAMP: 2:40
  :END:
** Bunny picks up fallen apple
   :PROPERTIES:
     :NOTER_TIMESTAMP: 1:28
   :END:
** Credits
   :PROPERTIES:
     :NOTER_TIMESTAMP: 8:15
   :END:

[MinmoTech]

I had almost the same idea last week, though I'd focus on offline video

I'd use the mpv player with some modification, so you can press a keybind to cature the position/area of a video and annotate it in emacs

Since mpv is easily scriptable, this should be the easiest way.

I guess as you can open web videos in mpv as well, online functionality comes for free as well :)

[strikerlulu]

https://github.com/kljohann/mpv.el/wiki

[japhir]

see also https://github.com/yuchen-lea/org-media-note

[auroranil]

MPV sounds like the best media player to use for this use case.

From https://github.com/weirdNox/org-noter/issues/82#issuecomment-542424247, I don't know about hooks, but it seems like if I can use these hooks to add support for video and audio files, then I can build on top of it and create a new repo called org-noter-media which will incorporate ideas I have above as well as a synthesis of ideas coming from alternative packages. If this is not a good approach (like what the author said regarding content that does not have pages), then a completely separate package will be better, and this issue will be closed.

Hooks can be found here: https://github.com/weirdNox/org-noter/blob/9ead81d42dd4dd5074782d239b2efddf9b8b7b3d/org-noter.el#L207-L210 https://github.com/weirdNox/org-noter/blob/9ead81d42dd4dd5074782d239b2efddf9b8b7b3d/org-noter.el#L226-L271

Hooks are being used from org-pdftools package: https://github.com/fuxialexander/org-pdftools

https://github.com/fuxialexander/org-pdftools/blob/a5b61bca3f8c91b0859bb0df1a929f9a31a57b99/org-noter-pdftools.el#L264-L274

[c1-g]

I'd like to inform everyone that I implemented a working prototype which depends on org-media-note and mpv.el borrowing from @auroranil's idea here. For it to work, I have to modify org-noter to accept a link as a document source so it only works with my fork of org-noter, org-noter-plus-djvu.

For how to install read its README here.

[daraul]

May be a good idea to migrate this to https://github.com/org-noter/org-noter

• https://github.com/weirdNox/org-noter/issues/174

[auroranil]

I have no plans for this project. Feel free to fork this project and make changes to it.