Audio visualizer

[Megidd]

Description

To address this issue: https://github.com/mediacms-io/mediacms/issues/109#issuecomment-1022880199

[Megidd]

Try

It's tried to use the videojs-wavesurfer plugin.

https://collab-project.github.io/videojs-wavesurfer/#/usage

https://collab-project.github.io/videojs-wavesurfer/#/react

Approach

I have tried this approach to build and test the changes:

### CMS has developed a custom NPM package based on videojs.
### Maybe we can add `videojs-wavesurfer` plugin to this custom NPM package.

cd /home/mediacms.io/mediacms/frontend/packages/player

### https://stackoverflow.com/a/61103357/3405291
rm package-lock.json

### For some reason, a symbolic link is corrupt.
### Has to be fixed:
cd /home/mediacms.io/mediacms/frontend/packages/vjs-plugin/node_modules
ln -svf ../../vjs-plugin-font-icons mediacms-vjs-plugin-font-icons

cd /home/mediacms.io/mediacms/frontend/packages/player
npm14 install videojs-wavesurfer

### Import videojs-wavesurfer inside the custom package.
### Add wavesurfer as a plugin.
### The exact commit to be looked at.

cd /home/mediacms.io/mediacms/frontend/packages/player
npm14 run build


### See the results by Docker:
### https://github.com/mediacms-io/mediacms/blob/main/docs/developers_docs.md#frontend-application-changes
cd /home/mediacms.io/mediacms
docker-compose -f docker-compose-dev.yaml exec -T frontend npm run dist
cp -r frontend/dist/static/* static/

Implication so far

For some reason, videojs-wavesurfer plugin is not enabled. This PR is a work-in-progress.

[Megidd]

Try another approach

Commit efcebfc4164d7c5eddf6f1ce6ac7919b0a93efda tries modifying AudioViewer class.

How to test:

cd /home/mediacms.io/mediacms/frontend
npm14 install --save videojs-wavesurfer

### Commit uses videojs-wavesurfer

cd /home/mediacms.io/mediacms
docker-compose -f docker-compose-dev.yaml exec -T frontend npm run dist
cp -r frontend/dist/static/* static/

Implication

For some reason, the videojs-wavesurfer is still not working.

[Megidd]

Try integrating video.js only first

1. As suggested, it's tried to use video.js only by commit 038b846117ac2ee8a9b68de27f03e2f08048579b.
   1. The test audio is played fine.
2. Then, the videojs-wavesurfer plugin is tried by commit f486d036c1028aef06aab72d9e9e70c04ca54a05.
   1. The JavaScript console throws an error.

Error

However, I'm receiving this error:

Uncaught TypeError: Cannot read properties of undefined (reading 'params')

At this line:

// videojs.wavesurfer.js:148 

          var WavesurferMiddleware = {
            setSource: function setSource(srcObj, next) {
              if (this.player.usingPlugin('wavesurfer')) {
                var backend = this.player.wavesurfer().surfer.params.backend; // ** error line.
                var src = srcObj.src;
                var peaks = srcObj.peaks;

                switch (backend) {
                  case 'WebAudio':

Error screenshots

Debug

Maybe the videojs-wavesurfer plugin error is due to another error which is thrown right before and is due to invalid assumption about video.js API:

Uncaught TypeError: this.AudioPlayerData.instance.isEnded is not a function

I need to modify the code according to the video.js API.

Debug update

Even after resolving all other errors, the videojs-wavesurfer error remains:

[Megidd]

Implication so far

Initializing videojs instance without the plugin is fine and audio is played. But as soon as the plugin is added, the audio is not played :

              plugins: {
                // enable videojs-wavesurfer plugin
                wavesurfer: {
                  // configure videojs-wavesurfer
                  backend: 'MediaElement',
                  displayMilliseconds: true,
                  debug: true,
                  waveColor: '#163b5b',
                  progressColor: 'black',
                  cursorColor: 'black',
                  hideScrollbar: true
                }
              }

With the plugin, this error is thrown:

Uncaught TypeError: Cannot read properties of undefined (reading 'params')

videojs.wavesurfer.js:148

[Megidd]

Replacing "video.js": "^7.12.3" with the latest "video.js": "^7.17.0" didn't help.

[Megidd]

By commit e12367edc523accf8768d28c45ae350eb97c3830, audio is finally visualized, but for some reason it isn't played:

Implication

Now, the proof-of-concept is done for videojs-wavesurfer plugin. So, it's almost verified that videojs-wavesurfer plugin could be used to visualize the audio.

Next

I feel like a better design would be to change the code of frontend/packages/player/src/MediaPlayer.js in order to make use of videojs-wavesurfer plugin. Since, that code is directly calling videojs constructor:

this.player = videojs(domPlayer, passOptions);

[mgogoulos]

awesome work so far! @styiannis what do you think?

[Megidd]

Commit 0840fb3f112456b34a6c908a6193b232c5c28793 tries to use the videojs-wavesurfer plugin by MediaPlayer.js. But for some reason the audio is not visualized:

Guess

mediacms-vjs-plugin is a custom package used by MediaPlayer.js. My guess is that I need to understand how to use videojs-wavesurfer plugin alongside another mediacms-vjs-plugin custom package :question:

  "dependencies": {
    "mediacms-vjs-plugin": "file:../vjs-plugin",

[Megidd]

Commit 40a45a7b70ddfc3d0c9cabc5858426921b86dc3e tries to debug. The audio is not visualized and an error is thrown:

Uncaught TypeError: this.player.mediaCmsVjsPlugin is not a function

[Megidd]

Next

Using videojs-wavesurfer plugin by MediaPlayer.js didn't work.

Next is to try to use it by this custom package:

  "dependencies": {
    "mediacms-vjs-plugin": "file:../vjs-plugin"
  },

Which is actually implemented by frontend/packages/vjs-plugin/src/MediaCmsVjsPlugin.js.

[Megidd]

Plugin with another plugin

There is a complexity which is the fact that frontend/packages/vjs-plugin/src/MediaCmsVjsPlugin.js is itself a videojs plugin.

We intend to use a videojs plugin i.e. videojs-wavesurfer alongside another videojs plugin. That's the complexity.

Trying to figure out how to resolve...

[Megidd]

Other way of initializing videojs-wavesurfer plugin

Commit f768206fa4106cb47e5186905edc92baaf217051 implements another way of registering and initializing videojs-wavesurfer plugin.

But an error is thrown:

Uncaught TypeError: this.player.wavesurfer is not a function

At this line:

Audio is played, but it's not visualized:

Implication

Implication is that the videojs-wavesurfer plugin didn't get registered properly.

[Megidd]

To resolve the previous error, commit b0b14e30a713e3e37ce012d5bd36aff8fe90989e tries to register the videojs-wavesurfer plugin manually. But a new error is thrown:

Uncaught Error: Illegal plugin for "wavesurfer", must be a function, was object.

[mgogoulos]

Closing, if this is something you are still willing to contribute please open a separate PR. Thanks!