Implement DRM auto-selection

[CastagnaIT]

Description

This is the last big step of my drm rework

This PR completely rewrites the code of how DRM info provided by manifests are handled, and how DRM/CDM are initialized and managed.

The main goal is to introduce a common place where manage DRM info and CDM so in the DRMEngine, this avoids the need for each manifest parser to have the task of choosing and managing DRM info (which parsers should not do, as is self-explained by the name) and along this introduce the automatic selection of DRM.

To reach this point, i started working on this 2 years ago, initially with the RFC test PR #1415, then with PR #1672 and in between other minor PRs not mentioned here, thousands of changes were needed to achieve something of acceptable quality with this PR, although there are some poor code parts that require more attention and much more time.

The amount of work to achieve this is very high, and despite my efforts to do dozens of tests, i will never have the opportunity to verify every single use case, so it is possible that this imlementation introduces some bugs that do not allow correct playback. It would be oppurtune to have some feedback from some users who use ISA with some video add-ons, if you read this please give your feedback.

Main changes:

DRM auto-selection (new)

This requires a manifest that supports multiple DRMs, in this case you will have to provide each DRM configuration by using the new property: inputstream.adaptive.drm as explained on Wiki https://github.com/xbmc/inputstream.adaptive/wiki/Integration-DRM (that i will update later time) therefore ISA will automatically determine which DRM to use, but you will also be able to give some sort of priority to the type of DRM to be used by making use of the priority parameter.

HDCP (breaking changes)

In my research it seems to me (but i could be wrong since i haven't found comprehensive docs) that a media player should not have a way to handle HDCP, take task is inherent possibly in the DRM (CDM), which should block non-usable Keys and nothing more. The possibility of managing the rejection of keys seem not so good handled on ISA, and cannot be done on this PR, also because there is a problem in stream ID selection in Kodi core, that prevent a fallback an appropriate stream ID when one fails, which should be fixed before that.

Breaking changes 1: The ISA setting "Ignore HDCP status" is now removed, the current behavior is that the HDCP "status" is never verified, but can be re-enabled by using the new check_hdcp, read below.

Breaking changes 2: In the unwrapper_params, child parameter of the license parameter, the path_hdcp and path_hdcp_traverse has been renamed into path_hdcp_res and path_hdcp_res_traverse (that handle resolution limit), and there is a new path_hdcp_ver and path_hdcp_ver_traverse, that handle an HDCP Version comparison with the manifest value. Details on wiki page https://github.com/xbmc/inputstream.adaptive/wiki/HDCP-resolution-limit

New property parameters

The inputstream.adaptive.config have two new parameters:

• resolution_limit - Allow to limit the playable resolutions provided by manifests. This is a kind of workaround since currently we cant handle appropriately HDCP use cases. Just set the max resolution that can be used e.g. "1024x768"
• check_hdcp - Allows to enable the check of the HDCP values provided in the license response. Details on wiki page https://github.com/xbmc/inputstream.adaptive/wiki/HDCP-resolution-limit

DRM configuration changes

1. When a manifest include DRM protections that dont require additional configurations, it will be played directly, so there is no need to set inputstream.adaptive.drm or inputstream.adaptive.drm_legacy property. This is the case for example for PlayReady, or ClearKey case that embed its license url in the manifest.

2. In the ClearKey configuration with inputstream.adaptive.drm or inputstream.adaptive.drm_legacy, its not only supported to set the license URL, but now its also supported to set a URI with "data" scheme. The data follow the W3C standard, so: data:application/json;base64,REPLACE_WITH_CK_LICENSE_RESPONSE_AS_BASE64

Stream selector type: Ask quality (improvement)

In multi-codec manifests, all codec variants will now be shown. I had to include this change here out of test necessity.

Deprecated old DRM properties

All following old DRM properties:

inputstream.adaptive.license_type
inputstream.adaptive.license_key
inputstream.adaptive.license_data
inputstream.adaptive.server_certificate
inputstream.adaptive.license_flags
inputstream.adaptive.pre_init_data

Are now --officially-- deprecated, in favor of the more advanced methods: inputstream.adaptive.drm_legacy and inputstream.adaptive.drm Its recommended to use the new properties from Kodi v22, see Wiki for more details. Old properties still work, for now, but will be removed on future Kodi versions.

Other changes under the hood

• Now the initialization of DRM/CDM Sessions are done on streams to be played only. where previously sessions were created for all streams even if unused, this improve also the playback start speed (with the exception of HDCP license based is enabled).
• Removed HLS parser hack that used DRM objects to store AES data, now the AES data is stored on each segment, with an appropriate cache for downloads
• Fix JSON traverse feature used on JSON license response to find data by path, now correctly traverse also json array type cases

To do / tests

[x] Rework drm info on Dash parser [x] Rework drm info on HLS parser [x] Rework drm info on Smoothstreaming parser [x] Reimplement CheckHDCP [x] Adapt test project, if needed [x] Enable multi-drm support to "drm" property [x] Deprecate old properties [x] Test stream with single drm widevine [x] Test stream with single drm playready [x] Test stream with clearkey [x] Test stream with widevine "to" clearkey [x] Test multi drm's stream widevine+playready, between windows and android [x] Test HLS AES-128 [x] Test HLS AES-128 - key on each segment [x] Test HLS multi-period, encryption transitions: unencrypted->AES-128->unencrypted [x] Some tests on android (tested SS+PR, some widevine) [x] Some tests on linux (under Ubuntu widevine)

To study / unsolved

currently drm engine can store sessions over the time, but there is no implementation for possible cleanups for example in a multiperiod stream, when an old period is expired (and should be removed from memory) then cleanup old drm sessions. in theory drm sessions on multiperiods should be reused, but idk...

fix #1585 about to separate audio/video sessions

Motivation and context

This PR will finish the multi-drm configuration support for inputstream.adaptive.drm property

What is auto selection of DRM?

in this case we are talking about manifests (usually HLS/DASH) that support multiple DRMs at same time. Until now, every python video add-on was forced to manually set the appropriate DRM for the type of operating system in use by setting appropriate inputstream.adaptive.license_type and inputstream.adaptive.license_key (and all other properties). for example a manifest that support playready and widevine, on Windows OS is needed that the python addon set the drm configuration for widevine, and on android could be for playready.

This implementation allows a video add-ons to provide all the DRM configurations supported by the provider in a distinct way, so that ISA can automatically select a DRM by finding a match between DRM supported in the operating system in use and the manifest, so a video add-on will no longer have to check if a DRM is compatible with the operating system in use, to configure ISA.

This will be even more useful in the future when a support to new DRMs can be implemented.

Types of change

• [x] Bug fix (non-breaking change which fixes an issue)
• [x] Clean up (non-breaking change which removes non-working, unmaintained functionality)
• [ ] Improvement (non-breaking change which improves existing functionality)
• [x] New feature (non-breaking change which adds functionality)
• [x] Breaking change (fix or feature that will cause existing functionality to change)
• [ ] Cosmetic change (non-breaking change that doesn't touch code)
• [ ] None of the above (please explain below)

Checklist:

• [ ] I have read the Contributing document
• [ ] My code follows the Code Guidelines of this project
• [ ] My change requires a change to the Wiki documentation
• [x] I have updated the documentation accordingly

[CastagnaIT]

@glennguy are you still here? please let me know i need help about CBCS, is broken from Kodi 21 and i have no clear idea

for now it would be enough to have it working by using this PR (Kodi 22) have you time to check the problem?

by using "InputStream adaptive samples" addon, im pretty sure that following stream was working: List CBCS streams menu --> Google Tears [CBCS]

EDIT: fixed by #1790

[Uukrull]

I did a quick test and the only regression I noticed is that UrnToKeySystem returns "" if
<ContentProtection schemeIdUri="urn:uuid:edef8ba9-79d6-4ace-a3c8-27dcd51d21ed"> doesn't exist, but <ContentProtection schemeIdUri="urn:mpeg:dash:mp4protection:2011" value="cenc" cenc:default_KID="xxxxxxxxx" /> exists so it should support those cases, otherwise it won't work because keySystem is empty.

[CastagnaIT]

in this case the target DRM is unknown because "urn:mpeg:dash:mp4protection:2011" means only CENC protection that should be compatible with all drm's i think the better thing could be add an additional check that when there is a single DRMInfo with no KS match to the first DRM (in priority order), otherwise another idea is check the initial segment, but this require much more changes, maybe later time

[CastagnaIT]

@Uukrull fixup added let me know

[Uukrull]

Working fine here. Many thanks!

[CastagnaIT]

if no other problems arise i think that next week i will merge and make a new release

CC @matthuisman if you want take a look and/or test about the changes

CC @Sandmann79 am@zon addon use JSON license, since i have made many changes, i recalled that i recently saw am@zon logs highlighting problems on JSON HDCP value, i think you need to fix or remove the HDCP configuration although it is set up here: https://github.com/Sandmann79/xbmc/blob/0f2c30a8c02bdea32e151916e51990455fa547fd/plugin.video.amazon-test/resources/lib/playback.py#L480 i suspect that is not used properly by ISA, since in order to work the MPD manifest must also provide the hdcp attribute to each Representation tag, i cannot verify since i havent an account, anyway i add a new wiki page about this: https://github.com/xbmc/inputstream.adaptive/wiki/HDCP-resolution-limit

[matthuisman]

@CastagnaIT this sounds really good

I will try to test out various addons of mine with this new version tonight and report back

UPDATE: Tried a few addons and everything worked as expected :)

[CastagnaIT]

good thank you

[Sandmann79]

@CastagnaIT you're right, the HDCP key is probably no longer needed. Especially as there are no HDCP versions in representation tags in the manifest.

However, in my short tests with the new DRM implementation I could not play any video, even with the deprecated method and ISA >= 22.1.5. I will create an issue for this in the next few days.

[CastagnaIT]

@Sandmann79 yesterday afternoon i have updated commit of this PR have you tested with this updated build? just to know because i have add a fix on dash

[CastagnaIT]

@matthuisman I recalled this old problem on android 32bit from issue #1147 should be about "Kayo" Do you have a chance to check if the problem still occurs with this PR?

[matthuisman]

was that problem fixed with https://github.com/xbmc/inputstream.adaptive/pull/1165 ? Kayo no longer has a Kodi addon to test with unfortunately.

[CastagnaIT]

ah ok then nothing, yes has been fixed, but now the license is requested by stream type (audio/video) in a separate way should not cause any problems because it is how sessions are generally recommended to be handled but there may be some special video service case that complain about this will be addressed when the time comes...

[CastagnaIT]

@Sandmann79 i would merge this tomorrow or sunday Do you have time to take a test?

[Sandmann79]

I tested it with the latest commits and unfortunately still get the following error on Windows OS: error <general>: AddOnLog: inputstream.adaptive: CWVCencSingleSampleDecrypter::CWVCencSingleSampleDecrypter: Cannot perform License update, no session available On Android, playback works fine.

Edit - complete Log: kodi.log

[CastagnaIT]

i found the problem

      <ContentProtection schemeIdUri="urn:mpeg:dash:mp4protection:2011" value="cenc" cenc:default_KID="94D9AACF-783B-4AAB-8340-8AF4679E6606"/>
      <ContentProtection schemeIdUri="urn:uuid:9A04F079-9840-4286-AB92-E65BE0885F95" value="MSPR 2.0">
        <cenc:pssh>bAIAAAEAAQBiAjwAVwBSAE0ASABFAEEARABFAFIAIAB4AG0AbABuAHMAPQAiAGgAdAB0AHAAOgAvAC8AcwBjAGgAZQBtAGEAcwAuAG0AaQBjAHIAbwBzAG8AZgB0AC4AYwBvAG0ALwBEAFIATQAvADIAMAAwADcALwAwADMALwBQAGwAYQB5AFIAZQBhAGQAeQBIAGUAYQBkAGUAcgAiACAAdgBlAHIAcwBpAG8AbgA9ACIANAAuADAALgAwAC4AMAAiAD4APABEAEEAVABBAD4APABQAFIATwBUAEUAQwBUAEkATgBGAE8APgA8AEsARQBZAEwARQBOAD4AMQA2ADwALwBLAEUAWQBMAEUATgA+ADwAQQBMAEcASQBEAD4AQQBFAFMAQwBUAFIAPAAvAEEATABHAEkARAA+ADwALwBQAFIATwBUAEUAQwBUAEkATgBGAE8APgA8AEsASQBEAD4AegA2AHIAWgBsAEQAdAA0AHEAMABxAEQAUQBJAHIAMABaADUANQBtAEIAZwA9AD0APAAvAEsASQBEAD4APABDAEgARQBDAEsAUwBVAE0APgBtAFcARwBBAFMAVgB1AHAAMAA2AHcAPQA8AC8AQwBIAEUAQwBLAFMAVQBNAD4APABMAEEAXwBVAFIATAA+AGgAdAB0AHAAcwA6AC8ALwBwAHIAbABzAC4AYQB0AHYALQBwAHMALgBhAG0AYQB6AG8AbgAuAGMAbwBtAC8AYwBkAHAAPAAvAEwAQQBfAFUAUgBMAD4APAAvAEQAQQBUAEEAPgA8AC8AVwBSAE0ASABFAEEARABFAFIAPgA=</cenc:pssh>
      </ContentProtection>
      <ContentProtection schemeIdUri="urn:uuid:EDEF8BA9-79D6-4ACE-A3C8-27DCD51D21ED">
        <cenc:pssh>CAESEJTZqs94O0qrg0CK9GeeZgYaBmFtYXpvbiI1Y2lkOjU4alZySENPUWZXOGZwbDFxM0pHRnc9PSxsTm1xejNnN1NxdURRSXIwWjU1bUJnPT0qAlNEMgA=</cenc:pssh>
      </ContentProtection>

on am@zon manifest there are the pssh that are not in standard CENC PSSH format they directly set the pssh data (that should be contained in a PSSH box field) fwik doing this, is out-of-spec that is the reason why on Android works and not on Windows

to fix this i need to add an additional check in order to manually construct the missing PSSH box

[CastagnaIT]

@Sandmann79 fix added if you can confirm thanks

[Sandmann79]

Thanks, the initial error has now disappeared. But now it takes more than one try to start the video and rewinding during playback causes Kodi to hang. For comparison, I just tested with ISA 22.1.4 and there are no such errors.

kodi.log

[CastagnaIT]

i see closed audio session, suggest that DecrypterCapabilites::SSD_INVALID is set to audio stream that make fails not full sure since i cant test, i tried remove it when you have chance let me know

[Sandmann79]

Now the streams start at the first attempt, but some need more time to start (video start at lines: 1171 and 5107). Seeking is still not possible, see last video in the log.

kodi.log

[CastagnaIT]

code changed hoping works

[Sandmann79]

The streams now starts without delay. However, seeking still causes Kodi to crash.

kodi.log

[CastagnaIT]

can you provide one log with last working isa version? because otherwise im not able to understand the log differences with same behaviour start playback and after video seek

edit: i suspect the problem is due to new drm multisessions (per a/v mediatype) I may have forgotten a piece as explained here https://players.castlabs.com/android/latest/docs/build/html/drm.html#multi-key-playbacks i need to check also if there are same KID on a/v, this should solve the problem

[CastagnaIT]

@Sandmann79 ok pushed changes, this should work something like Omega

[Sandmann79]

Unfortunately, seeking is still not working. After seeking, there is an infinite buffering, the playback time continues, but the video and sound freeze. As soon as playback is stopped, Kodi hangs up. The logs are attached for comparison.

kodi-ISA_22_1_4.log kodi-ISA_22_1_13.log

[CastagnaIT]

isa has opened again two drm sessions due to different kids, i think i need to add a new DRM config to force the single session, such as the link i posted above

[CastagnaIT]

@Sandmann79 i have done some changes in hope i have right things this time for test i have hardcoded the single drm session if works, please provide a new log for Windows, and try test if works also on android

[Sandmann79]

On Windows it is the same behavior as before and on Android there are no problems with seeking.

kodi-Win.log kodi-A14.log

[CastagnaIT]

its weird i will need to get an am@zon account to try understand better

[Uukrull]

You can get Amazon Prime for less than 5€ per year on the egyptian Amazon (amazon.eg) You only need an address and phone number from Egypt and you can use your existing Amazon account if you have it. The only thing I noticed when using it from Europe if that with the official app the message "It seems you are travelling" appears from time to time. Other than that, no problems at all.