API-based approach

[haudiobe]

We have the conformance software https://conformance.dashif.org/, but this is currently a web-based environment. This requires that you need to add an MPD and then your run the software and get the conformance report.

We discussed today and there are cases where you would want to run this more as a service with APIs. Examples:

1. When you do an upload of test vectors, you wanna run this through conformance automatically?
2. When you generate a new build of an packager, you can do some quick regression tests?
3. And and and

I wanted to check if you have any comments on this, if you have applications for such an API based approach. Also, it would be good if you see any concerns on doing so (may increase load) and so on.

[dsilhavy]

To add another use case: we could enable a tighter integration of dash.js and the conformance tools. For instance add an option to the reference UI labeled "Check conformance". Then we would not link to the webpage, but instead query the API and show the results either in the reference UI or a separate window.

Or we check by default while starting the stream. When the check is done we show a simple [x] or [√] with a link to the detailed test results.

This behavior could also be triggered whenever we encounter an issue in the stream. We show an error message and add the option to check the MPD for conformance.

On the load aspect: Maybe we should consider a token based approach to avoid people spaming the service. Downside of this is we need to maintain and hand out tokens.

[RufaelDev]

I think before progressing the work on the validator I think it is important to check were we are and what the way forward is.

What I would expect is that with the use of streaming increasing the demand for running the validator will increase,

And we have seen (apart from errors in the validation), problems with hosting like the server running out of disk space etc, Incorrect state (returning results from a previous validation).

So moving forward we would want

1. Be able to deploy at “scale” handling large load
2. Being able to debug and fix errors without “”breaking the validator” and updating (this is currently hard)

A paradigm that is very well fit for that is the devops and microservice based approach,

In this approach the larger system (e.g. the validator) is broken up in smaller pieces with well defined Api between them (often http based). then the smaller pieces can be connected through a front end or load balancer, In fact many of the large cloud apps today use this architecture and it fits well with cloud computing. In this approach the smaller components can be updated independently and swiftly.

So looking at the validator, in fact there are several “micro” components and there is an integration through the Apache server and php. The micro components are

1. Segment validator (C++)
2. DASH MPD validator (Java)
3. Front end (javascript + php)
4. HbbTV/DVB php
5. Bit-rate calculation (python)
6. HLS Etc. .......

In fact these are already hosted in separate repositories on dash github so the state is there! Apart from improving and fixing individual components (and updating), The integration and deployment should be improved a bit ( a lot), once you have this You can deploy using kubernetes or using cloud functions that would allow you to scale To offer an api.

Once that is in a better state, it would be possible to offer API based validation, But in current state that would be bad idea as a single web server instance Would not be able to handle the load.

to see were we are now I think

1. The individual components are ranging from ok to good and should be improved, maybe have unit tests etc, This is especially important to stay up to date with the specification (validator lacking behind Published spec is not good)
2. Integration of the components should be reconsidered, this is important I think if you want to scale
3. Deployment at scale will be possible, how to do it can be discussed

As the architecture is already modular, luckily these 3 can largely happen in parallel! In fact the first step would be to improve and test the individual components. When These are better tested and defined the integration can be discussed more naturally.

I do think it is urgent as people do want to use dash, but this may decline in the future when This does not get addressed well!

I do think it is urgent as people do want to use dash, but this may decline in the future when This does not get addressed well! We constantly get questions about validator crashing breaking etc. and this does not look good at all!

[jpiesing]

I can certainly see use-cases for running the validator automatically as part of other workflows. I'm not sure how common these would be and whether someone could just take the current code and host it themselves.

Wearing a DVB hat, I could see DVB-I service list providers (service aggregators) automatically running the validator against all services they aggregate as part of routine checking / monitoring. I believe TV broadcasters automatically run QA tools against their TV broadcasts today for similar reasons.

Wearing an HbbTV hat, it would be nice to validate all DASH MPDs that are checked in to HbbTV's test suite repository and reject attempts to check-in any that fail the validator.

I would regard both of these as advanced use-cases.

[nicoweilelemental]

We certainly would be happy to use an API-based version of the solution for our packager non-regressions tests. We have other use cases where we would probably prefer to host the solution ourselves, in order to avoid throttling.

[rbouqueau]

This is solved by https://github.com/Dash-Industry-Forum/DASH-IF-Conformance/wiki/REST-Interface. Closing.