Library Cleanup removes all Jellyfin entries from the Library

[gersilex]

Describe the bug

A Kodi library cleanup removes all Jellyfin library elements.

Excerpt of other reports this issue might replace or complement:

• https://github.com/jellyfin/jellyfin-kodi/issues/407
• https://github.com/jellyfin/jellyfin-kodi/issues/375
• https://github.com/CastagnaIT/plugin.video.netflix/issues/888

To Reproduce

1. Have a working Kodi library with Jellyfin set up
2. Run Clean Library from the Kodi Settings

Expected behavior

The library should remain populated with the Jellyfin media entries together with local files and entries of other addons.

Actual behavior All Jellyfin entries are gone. Local files and entries of other addons remain populated.

Logs

N/A

Screenshots

N/A (see https://kodi.wiki/view/Settings/Media/Library#Video_Library for Library Cleanup Function)

System (please complete the following information):

• OS: Windows, openELEC
• Jellyfin Version: 10.6.4
• Kodi Version: 18.8, 19-ALPHA
• Addon Version: 0.6.2+py2
• Playback Mode: Add-On

Additional context

Jellyfin writes directly to the Kodi database. Direct access to the Kodi database is not allowed according to the official Kodi Add-on rules. Maybe this relates to the issue.

The issue has been reported as different issues but it all boils down to the library entries and is not related to any other addon. Thanks to @CastagnaIT from the Netflix Addon for finding this out.

[CastagnaIT]

I think you made some confusion with what I was trying to explain, but I can understand, it is not always easy to understand the complexities of kodi...

kodi video library supports various type of sources that can be: -sources as real directories that can cointains: STRM or all video files extensions like MKV and so on, these organized for tvshws or movies -sources of network type, like Samba, DLNA etc... -source of "plugin://" type, where the kodi library browse an addon to add all movies/tvshow to kodi library -and other types...

here jellyfin try to use "plugin://" type path and it injiect the paths directly to the kodi library but here something is not done right, and then all the paths added will be deleted at kodi library Cleaning

I believe that the source paths (that are hidden) that jellyfin inserts manually are not full browseable by Kodi's library, where it should be instead, and so the files are recognized as broken (only my opinion).

it wouldn't be possible to use STRM files?

so if you use the "plugin://" as source, you could avoid to use STRM (thing that netflix addon actually not support, but there is a feature requested here https://github.com/CastagnaIT/plugin.video.netflix/issues/877) but unfortunately the real use of "plugin" as source, leads to some compromises of working not always easy to do (like you browse the addon instead kodi library also when you are in Kodi library, can sound strange but it is really so, tested personally)

I think for Kodi 18 (and so relative 19) the only praticable things are:

• implement to jf a different system to use/sync Kodi library (but this takes a long time of dev)
• try to understand the root cause of why the Kodi library cleaning remove all jellyfin library items and this will solve the problems at the root

For Kodi 19 I can officially publish the changes on Kodi 19 having a partial VideoLibrary.Clean based on a directory is also intended to speed up a lot the operation on a big database (of course if kodi dev accept this change)

[gersilex]

Oh sorry, I looked into the STRM files you generate and thought this is the way to go. I think I should rewrite the issue to just report the problem instead of proposing design changes I have no clue about :sweat_smile:

[oddstr13]

I would love to completely rip out all the current database interaction code, but the addon API is too limited in this regard from what I've found so far.

We could go the STRM file way, and there are JSON-RPC calls to trigger updates of specific directories, but there's no way to actually set up a library as far as I can tell.

It may still be desirable to use STRM + nfo where we can, and only interact directly with the database when absolutely required, such as configuring the media libraries and the paths they point at.

I'm open to suggestions and documentation links to how this could be done – any info on how to replace the raw SQL we're doing right now, really.

https://kodi.wiki/view/Internet_video_and_audio_streams#The_.STRM_file_method: https://kodi.wiki/view/NFO_files#Movie_nfo https://kodi.wiki/view/JSON-RPC_API/v12#VideoLibrary.Scan

In any case, testability (unit tests) should be keept in mind when reworking these parts.

[mcarlton00]

I'd looked into this before because I don't like modifying the database directly (albeit not involving strm files). I don't see any options for adding folders through the JSON RPC api. Probably equally frowned upon by the Kodi folks as modifying the database, but we could likely edit the $KODI_HOME/userdata/sources.xml file to add a directory of our choosing. It might even help with our handling of mixed content to drop them as "Files" instead of guessing Movie vs TV Show.

<sources>
    <programs>
        <default pathversion="1"></default>
    </programs>
    <video>
        <default pathversion="1"></default>
        <source>
            <name>Movies</name>
            <path pathversion="1">/home/matt/NAS/JF-Testing/Movies/</path>
            <allowsharing>true</allowsharing>
        </source>
    </video>
    <music>
        <default pathversion="1"></default>
    </music>
    <pictures>
        <default pathversion="1"></default>
    </pictures>
    <files>
        <default pathversion="1"></default>
    </files>
    <games>
        <default pathversion="1"></default>
    </games>
</sources>

And then populate that directory with strm files. If we do get that working, we should be able to trigger more targeted content updates judging by this thread: https://stackoverflow.com/questions/46765815/how-to-scan-only-a-particular-directory-in-kodi-via-json-rpc

Might have to watch out when using embedded devices though. We've already had reports of users on FireTV and similar devices running out of storage space trying to handle images, dropping a few thousand strm files on the disk won't help that

[CastagnaIT]

For STRM i can give some tips (i use it for netflix addon)

Might have to watch out when using embedded devices though. We've already had reports of users on FireTV and similar devices running out of storage space trying to handle images, dropping a few thousand strm files on the disk won't help that

an example: Agent of shield 6 seasons are 123 strm files, for a total of 10,4KB

so take in account about 10KB every 120 files

For the NFO files, are useful only if you want to use the "Local information only" screaper in the folder NFO files can take up a lot more space depending on how much information you want to contains

But you could avoid writing NFO files if you set a Kodi screaper (e.g. TMDB)

For the STRM management:

• When you add some STRM files You can use JsonRPC VideoLibrary.Scan to update the Kodi library by set your folder

• When you remove some STRM files You can use JsonRPC to update kodi library: VideoLibrary.RemoveMovie VideoLibrary.RemoveEpisode VideoLibrary.RemoveTVShow But these require the kodi library "id" then first you have to search the kodi library id (e.g. for an episode) e.g. by using VideoLibrary.GetEpisodes with a file filters Or you can use VideoLibrary.Clean, this is more easy but: -For Kodi 18 this clean all the database -For Kodi 19 i am currently adding the directory param https://github.com/xbmc/xbmc/pull/18562 to allow cleaning only in the chosen folder

Problems/Side effects:

• When you use JsonRPC in background (e.g. while watching a video) the loading screen could appears and annoying (I opted to use only Scan/Clean at end of addon sync operations)
• When you play a STRM the video resume is broken (due to a bug of Kodi) so video start every time from beginning, then need a workaround to fix this, if needed i can explain separately
• STRM not allow to include video info like length of video or format of audio/video, need to find an alternative method to set this
• When you are about to start a Scan/Clean operation, you should verify that there is not already a Scan/Clean operation in progress, and delay it, otherwise it will be discarded

---

sidenote: there is also the MediaImport kodi project, I am not sure if it would help, i heard a rumor of the devs will be probably integrated in Kodi 19, but you will have to wait...

[mcarlton00]

I think we'd have to do NFO files. Some people have spent a decent chunk of time getting their metadata in the server adjusted just right, and wouldn't appreciate getting different metadata in Kodi vs other clients. In addition to people that have custom shows/data for items. Possible alternative is writing our own scraper to query JF instead, but that sounds like another can of worms.

Sounds like there's plenty of other extra quirks involved in strm files that need to be thought through. I appreciate you sharing what you've found so far, definitely gives us a good heads up of things we'll surely run into, so big thanks for that.

Either way, it sounds like this path is going to slow down the sync speeds since we have to go through the api layer (and write extra files) instead of writing to a database. In my mind the ideal solution is to store everything we want in our own local database (jellyfin.db, netflix.db, etc) without touching Kodi's built in libraries. This would give us the speed benefits of database queries but stops conflicts with other addons/local libraries. I just haven't found a good way of doing that at this point in time and have it populate into Kodi's internal library. We can have write our own databases and have our addons query that, but then users have to customize their home views or go through the addon menu which is less than ideal, and something I'm currently looking for a solution for with with JellyCon and the Embuary skin.

---

We've been contacted about the MediaImport project a few times, but it's been several months since I've heard any updates on it. I haven't personally had much success with it, but my testing was a good while ago. In theory they're supporting Plex and Emby, and with our api 99% the same as Emby it should just work. However since our api is likely to be changing more over time I do have some concerns about it in the future. Especially if our api changes in the middle of a Kodi release cycle, I imagine that's not going to go over super well. User "CopperYang" in our matrix chat has been keeping tabs on it and was the one to bring it to our attention.

[gersilex]

I don't see any options for adding folders through the JSON RPC api

Would this be a thing we should open an issue for at the Kodi project? Editing an XML sounds doable with some classic CRUD operations. There seems to be a clear demand for this and it would even help the Netflix and other addons.

[mcarlton00]

I've posted a message to their dev slack asking about adding directories or importing libraries, so we'll see if we get a response that way. But yes, it would definitely need to be an upstream thing, and with Kodi 19 coming it's probably a good time to broach that subject so it can get into a release.

---

I've been playing with the strm files and working through the beginnings of a plan to move to them instead of hitting the database directly, but there's still several questions and a lot more testing to do. Some in response to @CastagnaIT, some are just my own findings and ramblings.

When you play a STRM the video resume is broken (due to a bug of Kodi) so video start every time from beginning

From my testing tonight, this is super inconsistent. If I play a video halfway and stop, then try to play again, it will let me resume. However, if I play a video halfway and stop, then restart Kodi, when I try to play again it will start from the beginning. Seems like a fairly easy workaround (just catch the play action, query the server for userdata and ask the user if they want to jump to previous position), but very odd overall. I'm guessing this is the mentioned bug.

STRM not allow to include video info like length of video or format of audio/video

I'm not quite sure where this comes into play. On the Amber skin at least it shows up fine when accessing with a strm file, both in the skin and during playback. I wonder if JF is providing a different payload than Netflix that includes that info. More testing needed.

As for the new things I found or things that need more research:

• Embedded subs are super hit or miss. Some worked, some didn't. This may be a continuation of our never ending subtitle issues in the server, though. I've seen it in our existing playback methods.
• External subs are right out. Again, should be relatively straightforward fix. On playback start, query the server for external subs and pull them separately.
• Direct Play vs Transcoding. Absolutely no idea how to handle this here. I guess we'd have to hijack the play command and change the target URL based on if it should transcode or not. This might be tricky.
• Playback reporting to the server. Currently not working, not surprising, relatively easy blah blah blah.
• Sometimes after stopping playback the item name in the menu changes from the movie name to Download until the view refreshes, which is the end of the URL path. Annoying quirk, possibly another kodi bug.

---

The ramblings of a madman: How this could work

• Transition from modifying the Kodi DBs directly to generating a file tree of strm files (which is dead easy: https://{server_url}/Items/{item_id}/Download?api_key={access_token}
• Create library folders in $KODI_HOME/userdata/addon_data/plugin.video.jellyfin/ to hold the strm files.
• Add these new library folders to Kodi. Ideally with the json rpc, if not by editing the sources.xml (seriously, why does everything have to be xml? blergh)
• Stop conflicting with other addons/local libraries. Possibly work on MySQL too? (don't really care on that one, tbh).
• The sync would hit a separate jellyfin.db instead of the Kodi DBs. This basically just serves as an index of item_id to file_path so we don't have to try to parse a file tree constantly. Sqlite is way faster than that.

Now, the giant hurdle for this plan (outside of the things i mentioned in the previous section): Metadata.

We want to match what's in the server, so scanning against built in providers in Kodi won't work. The way I see it, we have 2 options: NFO files, or write our own metadata provider for JF.

• NFO Files. Browsing my library, these seem to range from 1.3KB to 5KB. For my library, that's anywhere from 27MB to 105MB for just my TV episodes (not including show/season metadata). Storage is cheap, sounds like nothing, but if we're talking stuff like FireTV or other embedded devices it's sure to be an issue for those with larger libraries. I'm not sure how much storage they typically have, but it'd nice for us to be good denizens and not consume every bit of storage on the device.
• Metadata provider. I .... don't hate it? Which I guess is saying something considering my feelings on the rest of this codebase sometimes. Our addon.xml is already kicking off like 3 separate things, one more won't kill us. Maybe. Probably. It would need a bit of finesse though. Don't want to be slamming the server with tens of thousands of requests whenever we run a library scan. Api calls are easy enough (https://{server_url}/Users/{user_id}/Items/{item_id}

There has to be holes here. It sounds far too straightforward. Not an insignificant amount of code, but straightforward. @oddstr13 tell me I'm wrong. Find the holes, before I start exploring this rabbit hole further.

[CastagnaIT]

I'm not quite sure where this comes into play. On the Amber skin at least it shows up fine when accessing with a strm file, both in the skin and during playback. I wonder if JF is providing a different payload than Netflix that includes that info. More testing needed.

you have to take into account only the media-flag of the skin directory, because when you play the video the media-flags can be updated/replaced automatically according to the played streams

External subs are right out. Again, should be relatively straightforward fix. On playback start, query the server for external subs and pull them separately.

not sure what you mean exactly but for Kodi19 there is also a new jsonRPC in progress to add on the fly the subs to the player: https://github.com/xbmc/xbmc/pull/18445 perhaps you could be interested

The sync would hit a separate jellyfin.db instead of the Kodi DBs. This basically just serves as an index of item_id to file_path so we don't have to try to parse a file tree constantly. Sqlite is way faster than that.

maybe you already knew it but i found out recently.......so pay attention to how many operations at the db do sqlite is fast, but on low-performance sdcards or mechanical harddisk is extremely slow

e.g. if you have to do 100 sqlite same similar "insert" op

This take a lot of time: open db > write "insert" > close .... open db > write "insert" > close ... and so on

This take less than 1/10 of time of single insert method: BEGIN TRANSACTION; + executemany + COMMIT;

so with my addon i have improved the situation from about 20secs to less than 1secs! to do same operations, but of course it depends on the type of implementation and what you have to do...just a suggestion

[oddstr13]

So, one wild idea I just had is the possibility of using the http network source combined with a server-side addon in Jellyfin, presenting a browsable http tree with strm, nfo files and artwork.

The strm would still be pointing to plugin://plugin.video.jellyfin/, (probably, hopefullt) giving the addon control over things such as transcoding and playback.

This way, we could leave syncing and cleaning to the standard Kodi way of doing it, or trigger partial rescan via the JSON-RPC on initial connection or websocket updates.

Keep in mind that the file and directory naming must be proper for TV series & episodes to be picked up properly by Kodi.

This would avoid the problem of a on-filesystem directory structure bloating used storage space.

[mcarlton00]

An http directory isn't a bad idea, but that will largely fall on you and the server team. My C# is .... shall we say "not good." First thought is that there might be access speed issues when scanning. Typically any http browsing I've done has had some delay, but that may be more a side effect of like nginx file indexing over the actual technology. The server should be faster in that regard. Of course, it also largely renders the addon ineffective unless a user needs transcoding. It would basically exist purely for playback reporting and transcoding.

[gersilex]

I looked over some HTTP server access logs I found on the kodi.tv forums and it looks like at least Kodi 18.3 does not use HTTP/2 will be somewhat inefficient scanning huge libraries. That's not meant to be negative. Not at all. This means if Kodi decides to go and use a HTTP/2 http client for the HTTP sources, the connections can be re-used and will get a lot faster. Probably somewhere between 10-100x times faster (with a large number of small files).

The HTTP index, that has to be implemented in Jellyfin can be generated as part of the Scan all libraries scheduled task or a new one. This is sometime referred to as a "Static API". It behaves like an API, but it doesn't hit an underlying database, as all the data is pre-rendered like a pre-rendered static website. If the webserver hands out these pre-rendered files, requests from Kodi will be ansered extremely fast, even with HTTP/1.1. Another way would be to have the webserver cache the results, but that brings all the pros and cons of caching with it as you never truly know what the client will receive. This can be combined with refreshing the data shortly after delivering the cached data, so the HTTP responses are always cached AND exactly one call behind the truth. I'm getting way to deep into this, sorry.

However. I really like idea idea and I think we have a lot off leverage to optimize if we see that the concept works out but performs badly.

[oddstr13]

@mcarlton00 well, it doesn't really do that much more now either. Making sure the library is synchronized and up to date (could still trigger re-scans of affected files), making sure watched state is syncronized between clients, and transcoding if needed (and hopefully soon also syncplay).

@gersilex HTTP/1.1 can reuse connections, but I think what HTTP/2 brings is multiple concurrent requests over the same socket (several transport streams encapsulated in one connection or something).

The structure could be dynamically created, but the nfo cached (the server already has the capability of generating nfos) – This shouldn't be any slower than the current API requests.

I think @mcarlton00's idea of possibly also combining the nfo with a metadata provider that queries the Jellyfin server should be considered, altho this could complicate implementing multi-server functionality – tho, xbmc/xbmc#14972 would be desirable for this.

[Long-Man]

Hi All. It would seem to me that most of problems been tackled here are solved with MediaImport. My question, why not pool your efforts with Montellese to get MediaImport merged rather than pursuing alternate solutions. https://github.com/Montellese/xbmc/wiki/Media-Import. Please engage Montellese. He has encountered most if not all the problems you are trying to solve.

[mcarlton00]

MediaImport both is and isn't a solution, and only partly helps with the big picture here.

It does help with:

• Adding items to the library
• Metadata

It doesn't help with:

• Transcoding
• Remote control sessions

Unknowns:

• External Subtitles
• Playback reporting

I've had reports of some users already using it and having good results. I was never able to get it working personally, but the Emby importer works for us. https://github.com/Montellese/xbmc/wiki/Emby-or-Jellyfin-Media-Importer. At some point the APIs will diverge more and we'll have to write our own but for now users do have a path forward with that if they'd like.

It seems like MediaImport will be a good direction in the future. However, (my opinion), at this point we can't rely on it. Until it's been merged into Kodi proper, it's effectively a hard fork with an extra feature. And if we put a bunch of our effort into refactoring planning on MediaImport and then it doesn't get merged before Kodi 19 releases for whatever reason, we're kinda up shit creek. So my personal stance is that until it's merged and included in Team Kodi's official builds, it can't be the central piece of our official plan for the future. Not to mention that we're sure to have Kodi 18 users for years to come still, though we haven't fully fleshed out how to handle that once 19 releases with whatever other breaking changes get thrown our way between now and then.

[Long-Man]

It does support External Subtitles and Playback Reporting. Could you expand on transcoding and remote control sessions?

What problem are you having getting it to work. Maybe I can help. Please bear in mind that the current solution still need effort to get it to work. Why not engage @Montellese to find out what is required to get it into v19. There are four skilled programmers here. Why not put the effort there?

[mcarlton00]

Transcoding is the process of re-encoding the files on the fly so that they play on the device better. Often caused by the file being a codec that's not supported by the client or bandwidth limits. Trying to pump a 50GB 4k bluray remux through a 3Mbps pipe, for example. And since the URLs and various parameters are often determined at the time you press play, I really can't think of any good way to handle that at the database layer. So it works great if your server is on your lan, but if it's remote it increases the chance of issues.

The remote control is something I don't anticipate mediaimport ever supporting, but it's something we support now. You're able to control Kodi (or other clients') playback through the Jellyfin web interface or mobile apps. It has nothing to do with getting the content into Kodi, which is why I don't see it being included in mediaimport. This is why I say it is and isn't a solution. It solves some things for us, but not others.

As for helping, not all programmers are equal. Kodi and MediaImport are both written in C++. We don't know C++. Me trying to help work on it would be roughly equivalent to trying to read Shakespeare written in Wingdings. Between not knowing the language and just not having enough free time to get into another project, options are kinda limited.

[Long-Man]

MediaImport itself does not support Remote control but it doesn't prevent it. I install the Jellyfin Kodi addon and control playback from the web client. I have been doing that for all of this year. I have done the same with Emby for Kodi and EmbyCon. But I have kept Jellyfin Kodi. I chose what tto play and add to the playqueue.

Technically I cannot see why it would not support transcoding. MediaImport stores a playback url as the path to the file. If it is currently supported in Jellyfin Kodi it should be possible.

Help can be defined in many ways. You understand the server, you know what is the experience that users desire. And finally you have maintained Jellyfin Kodi. I am sure that as a programmer you understand that testing and feedback from a person who understand can be like a boost of nitro to a programmer.

Please give it spin.

[oddstr13]

Our addon currently stores the following URI in the files table in the video database; plugin://plugin.video.jellyfin/f137a2dd21bbc1b99aa5c0f6bf02a805/?filename=Big+Buck+Bunny+%282008%29+%28HDTV-2160p+-+x264+MP3%29.mp4&id=4d67615aa1fbd62293d1b6e08c82b809&dbid=36&mode=play.

And, as @mcarlton00 said, I'm not particularly good at C++ either – can piece together some Arduino code, and understand the basic syntax, but that's it.

We still have a fair number of users on Kodi 17 (I'd love to know why), so I don't see us dropping Kodi 18 support anytime soon. (even though I'd like to drop Python 2)

[Long-Man]

Please give it a spin. I promise It won't kill you. https://forum.kodi.tv/showthread.php?tid=224794&pid=2980146#pid2980146 Please give it a spin. The repo is included but it is disabled. Enable it and install the mediaimpoter addon.

[Montellese]

Hey guys

I can fully understand that you currently can't rely on my media import work because it's not part of Kodi mainline and I can't provide any realistic timeline because I'm very busy with RL. Even if media import would be part of Kodi mainline if you have to support a large number of Kodi v17 / v18 users it's not really an option.

Nonetheless I want to provide some input and feedback on what is possible with media import:

• For python add-ons media import is just another extension in Kodi's add-on system. That means that you can combine it with all kinds of other add-on features.
  • The maintainer of the plugin.video.waipu.tv add-on added experimental support for media importing to his add-on without dropping any other features of his add-on. He also provides plugin://plugin.video.waipu.tv based URLs for imported items so playback is still handled through the add-on.
  • In the mediaimporter.emby add-on I have also integrated custom context menu options for items imported through the add-on which e.g. provide additional playback options (direct play vs. streaming) or allow to manually refresh the metadata of an imported item.
• Based on the previous point transcoding should be possible by using plugin:// based URLs for imported items instead of direct paths or HTTP-based URLs. I'm just not familiar enough with the Kodi's plugin system which is why I haven't adopted it yet. Pull requests are always welcome :-)
• As @Long-Man has pointed out external subtitles and playback reporting are already fully implemented. To improve the support for external subtitles I even extended the existing python API to be able to provide additional information like the name and language of the subtitle. See https://github.com/xbmc/xbmc/pull/17218 for the respective pull request for Kodi mainline.
• I'm not really sure what remote control sessions can do and how they work but if you can do it with your current add-on I'm pretty sure you would be able to use the same or similar code combined with media import. Again media import ist just another extension capable of directly importing items from other sources into Kodi's library.

As @Long-Man has pointed out I'm always happy for new ideas, feature requests, bug reports and other feedback from testers. I'm also not a very good python programmer and don't know Kodi's add-on system that well. Therefore progress on the media import add-ons is slow. I'm currently working on providing a version of media import for Matrix Alpha 2.

[CastagnaIT]

Thanks for this explaination, in the future I will take a look at plugin.video.waipu.tv to understand better how it is implemented, i would like try to adopt this system

[mcarlton00]

Thanks for the info @Montellese. I'm cautiously excited to see where it goes over time when other addons start implementing it. It sounds like that will significantly simplify things in the future, though given the amount of our users on legacy Kodi versions I'm not sure how soon we'll be able to take advantage of it. We could implement it but route code differently for older Kodi versions, but that sounds less than fun to maintain (though possibly still easier than our database code).

On topic for this issue as a whole, I'm feeling like the strm file route may be the answer for now, but more testing is certainly needed before we can forge full steam ahead.

My reasoning against the http directory route:

• The primary maintainers here not knowing C#, which shifts the maintenance burden to other people who are busy with their own projects
• The http directory route effectively makes the Kodi addon fully dependent on having a given plugin installed in the server (it could be used fully in dynamic mode, but in that particular use case JellyCon is a better alternative)

It's not a flawless solution, and @oddstr13 is correct that it does complicate how we can ever implement multi-server functionality. I do have the beginnings of an idea for that, but it's not going to get fully fleshed out until we can confirm that other things will work as expected.

I've also realized that our playback functionality shouldn't have to change much. Instead of the strm file containing a link directly to the server (https://{server_url}/Items/{item_id}/Download?api_key={access_token}), we could likely have the strm file point to the addon itself with something along the lines of plugin://plugin.video.jellyfin/{item_id}?mode=play. Still some questions overall, but I think it's worth investigating more at least.

[mcarlton00]

As usual, there's another hangup. It doesn't seem possible to do this in a way that allows us to get away from modifying the database at all. We can add a bunch of strm files and include that in sources.xml, but the method to tell Kodi what metadata provider to use for a given directory is in the database.

sqlite> select * from path;
idPath|strPath|strContent|strScraper|strHash|scanRecursive|useFolderNames|strSettings|noUpdate|exclude|dateAdded|idParentPath
1|/home/matt/Movies/|movies|metadata.themoviedb.org|28d9889cec58db2c060b82b6be2b4429|2147483647|0|<settings version="2"><setting id="certprefix" default="true">Rated </setting><setting id="fanart">true</setting><setting id="imdbanyway" default="true">false</setting><setting id="keeporiginaltitle" default="true">false</setting><setting id="landscape" default="true">false</setting><setting id="language" default="true">en</setting><setting id="RatingS" default="true">TMDb</setting><setting id="tmdbcertcountry" default="true">us</setting><setting id="trailer">true</setting></settings>|0|0||

This seems nicer than writing all of the metadata to the database directly, but still not a particularly great solution

[mcarlton00]

After lots of monkeying around in the database and experimenting, I can run a clean library and not purge my library contents. It's not going to get us away from the modifying the database though, and the longer I spend on this the more I think we may need to give up on that particular dream.

Changes required for us to survive a clean library operation:

1. Modify ~/.kodi/userdata/sources.xml to include a local file path.

<source>
    <name>Jellyfin Movies</name>
    <path pathversion="1">/home/matt/.kodi/userdata/addon_data/plugin.video.jellyfin/libraries/f137a2dd-21bb-c1b9-9aa5-c0f6bf02a805/</path>
    <allowsharing>true</allowsharing>
</source>

2. Update the strPath column in the path table in the database to point to this directory instead of the plugin directly. old: plugin://plugin.video.jellyfin/f137a2dd-21bb-c1b9-9aa5-c0f6bf02a805/ new: /home/matt/.kodi/userdata/addon_data/plugin.video.jellyfin/libraries/f137a2dd-21bb-c1b9-9aa5-c0f6bf02a805
3. Update the strFilename column in the files table to use a strm filename instead of the plugin path example old: plugin://plugin.video.jellyfin/f137a2dd-21bb-c1b9-9aa5-c0f6bf02a805/?dbid=3&mode=play&id=dc3a0d09-9368-0538-46d7-0d8e50ac63a0&filename=V+for+Vendetta+%282005%29.mkv example new: dc3a0d09-9368-0538-46d7-0d8e50ac63a0.strm
4. Create strm file mentioned above for each item in the library with a link to the old plugin:// path from the database example name: /home/matt/.kodi/userdata/addon_data/plugin.video.jellyfin/libraries/f137a2dd-21bb-c1b9-9aa5-c0f6bf02a805/dc3a0d09-9368-0538-46d7-0d8e50ac63a0.strm example contents: plugin://plugin.video.jellyfin/f137a2dd-21bb-c1b9-9aa5-c0f6bf02a805/?dbid=3&mode=play&id=dc3a0d09-9368-0538-46d7-0d8e50ac63a0&filename=V+for+Vendetta+%282005%29.mkv

Pros:

• We don't lose our libraries if the user/another addon/evil pixies run a clean library on Kodi
• No need to change playback flow or sync logic
• We get full metadata instead of being limited to what Kodi picks up from strm files by default

Cons:

• We're still modifying the database like the filthy heathens we are
• Potential to get more complicated if we need more structure. Movies aren't awful, but if we dump a strm file for every episode of every tv show in a single directory Kodi might be unhappy with us. And it'll be super ugly. We're talking 10s of thousands of files here
• The migration from old database to strm file + database will probably not be super quick since we have to loop through every object in the database.

Conclusion: This adventure is going to make me pull my hair out. But at this moment I think this is the most sane route forward.

[mcarlton00]

While discussing this offline, oddstr brought up a good point that writing thousands and thousands of strm files to the disk is likely to wear out sd card based systems and nuke any longevity of those systems

[Long-Man]

TBH. Jellyfin Kodi is not meant for the use case in the topic. Jellyfin Kodi makes Kodi a client for the exclusive use with Jellyfin Server. In such a case you should not be using Clean Library.

You should be using Perform local database reset or the entries under Manage Libraries on the settings section of the addon. If you want to use other Kodi Addons alongside your Jellyfin server items, MediaImport is what you should be using. Don't try to fit a square peg in a round hole.

Use the right tool for the job, you will keep your hair.

[mcarlton00]

You're missing the point. We are not clicking clean library. Other things are. We are fully at the mercy of other addons, Kodi updates, or any number of other situations that could trigger the clean library function, like the netflix one mentioned above. That's not good. It's annoying. It causes needless re-syncs, and annoys users.

I'm going to say this as nice as I can, and for the last time. MediaImport is not the solution. Full stop. It may become the solution in the future. It certainly has the promise to do so. But right now, and for the foreseeable future, it is not. It's not been merged into mainline Kodi yet. Given that we're at beta releases of Matrix, the odds of it being merged in now are slim to none (and if it does get merged, I'll have to seriously question the Kodi devs' opinions on what alpha and beta are for if you merge a large feature after theym). Not usable in vanilla Kodi = not an option here. I'm not telling thousands of users "oh we don't support vanilla Kodi anymore, go use this soft fork." Not to mention we have the large majority of our users on older versions that MediaImport is never going to support. It's not an option. I mean absolutely no disrespect to Montellese when I say this, but it's effectively vaporware. They've done an immense amount of work and have impressive results right now, and I can only imagine it's going to get better over time. But not being in vanilla Kodi and not being able to support the large majority of our users means that it's effectively useless to me. End of story.

We have options on how to make this work now, for all of our existing users. It's just exploring through those and deciding what makes sense. It will not be the most elegant solution, because we're still tricking Kodi into doing things it was not designed to do. But we can make it work. And that's what all of my comments in this thread have been, exploring those options. Now I could be doing those in private and just announce a decision at the end and be done with it, but I feel like the users who are interested in this issue would like to see the thought process and reasoning about why things are the way they are and some of the issues for why it's taking so long to find a solution. If you're tired of watching my ramblings as I continue to explore how to break Kodi in new and creative ways, the unsubscribe button is right over there to the right under "Notifications". You won't hurt my feelings, promise.

MediaImport is not the right tool for the job now, and doesn't appear to be in the near future. Stop bringing it back into a conversation where it can't be used.

[Long-Man]

It is not about being nice or not being nice. Emby for Kodi, the addon from which Jellyfin Kodi is derived was not designed to be a regular Kodi Addon. In fact it Kodi does not have a category for such an addon. It was never designed to exist with other addons adding entries to the library. The design is to take over the Kodi library. Other Video addons could coexist for browsing and playing. Granted you and others may have different ideas today, However, that was not the design. In fact the plugin entries to the library that are mentioned were added to Kodi after Emby for kodi had matured. Even strm files from Addons wasn't a thing at that time. strm files were primarily for used for internet streams. That fact in of of itself is a part of the reason for some of the playlist problems in Kodi today. The strm files were treated as playlists as some of them were exactly that.

Content aggregation from multiple sources is what you are dealing with. If you decide to use strm files then you would be changing the design of Addon. That's not a problem. Situation changes and we and our solution evolve to accommodate.

With regards to MediaImport, as you say it is not mainline, true. That does not affect the fact that in it's current implementation of taking over the Kodi database, Jellyfin kodi does not allow for media aggregation. That limitation was seen by the original developers, hence Manage Libraries and reset kodi database in the addon settings. That's not my opinion, That is just the fact.

I can see that you are annoyed with my bringing up MediaImport. My argument is not so much about MediaImport as it is about Montellese. Here is a Core Kodi Dev who is willing to help to solve these problems. IIRC, he is one of the persons with the most commits to the Kodi repo and that does not include MediaImport. Whatever road you choose to travel is going to require some changes in Kodi. He has made many changes to Kodi which from which your version of Jellyfin Kodi has benefited. In your refactoring you may have encountered some code that seems, at best, inefficient now, but was the only way possible at that time. He will not be around forever. His life has changed. A lot of his contemporaries are gone. It is going to be extremely difficult to find someone with the knowledge, skill and time to make the changes. He has very little time now. In a short while he will not have any. Don't lose the opportunity to work with someone who can help. I could put names like JMarshall, Spiff etc but they would probably not mean much to you. Kodi is a beast and you need a BeastMaster to tame it. He did not spend his time on MediaImport because he has no better use for it. He did because you foresaw the problems that you and many other addon devs are having.

It would be sad if the devs commenting in this thread realize after it is too late.

Just My 2 cents.

[oddstr13]

@Long-Man if MediaImport gets merged into Kodi 19, it would be of great help when we decide we're tired of supporting Python 2, and go full Python 3. Unless and until both of those conditions are met, MediaImport doesn't provide much if any value for us (no offense to @Montellese).

Don't try to fit a square peg in a round hole.

https://youtu.be/lPooEK-whTE?t=195