droidshows icon indicating copy to clipboard operation
droidshows copied to clipboard

[FR] Wishlist, New Fork

Open warren-bank opened this issue 2 years ago β€’ 144 comments

I've only just recently found this app. I'm absolutely loving it. Great job!

Here are a few suggestions for improvements:

  • add "FLAG_ACTIVITY_NEW_TASK" flag to Intent when launching an "external resource" URL, so DroidShows remains visible in the recents menu as the top-most Activity in the history stack for its task
  • add menu item to main menu:
    • "Sort by.." opens a dialog with options:
      • Sort by name
      • Sort by unseen (oldest first)
        • note: this ascending order is currently available
      • Sort by unseen (newest first)
        • note: this descending order would be very useful
  • add menu item to season context menu:
    • "Seasons after here are unseen" (because I sometimes like to rewatch a series from the beginning)
  • add option to configure the background color
    • default:
      • System Wallpaper
    • solid color:
      • White
      • Black
  • add option to disable the pull-to-update feature (because I often trigger it by accident)
  • add option to manage labels (ex: similar to Gmail) to categorize shows
    • default:
      • Active
      • Archive
    • use case: multi-user
      • John
      • Jane
    • use case: genre
      • Reality TV
      • British detectives
    • notes:
      • this would require additional UI to manage label associations per show
      • this would effect "update" behavior
        • currently, only "Active" shows are update (ie: not "Archive")
        • the menu item "Update shows" (ie: pull-to-update behavior) should only update the shows associated with the currently selected label
        • a new menu item "Update all" could be added that would trigger an update to shows in all labels

warren-bank avatar Mar 21 '22 10:03 warren-bank

Wow, that's a lot... Let's start with number

  1. (FLAG_ACTIVITY_NEW_TASK). Please have a look if this works: I've added it to every intent for IMDb, Wikipedia and custom external resources.

DroidShows_7.11.5_FLAG_ACTIVITY_NEW_TASK.zip

  1. "Sort by unseen" does not do what you think it does. It sort by amount of unseen aired episodes. So if you want to binge a lot, you'll see the shows in the order of available episodes. I won't change the behavior, but I'm open to a name change in order to make this clearer. I don't know what use it would be to have a "Sort by unseen (newest first)" feature t.b.h.

ltguillaume avatar Apr 16 '22 18:04 ltguillaume

  1. works great..
    • one-line change that makes a huge UX improvement, imho
    • I love having the "external resources" feature to cross-reference each show with any number of URLs w/ one as default
    • I link each show to webpages that have VOD for episodes in the given series
    • without this flag, I have a hard time remembering which season/episode is next by the time the site loads (and userscripts run)
    • with this flag, it's easy to jump back and forth between DroidShows and the app used to browse the webpage URL
    • also..
      • since I was using an apk signed by F-droid, I needed to uninstall and do a fresh reinstall.. rather than an update.. because the author signatures are different
      • I was worried whether your backup/restore would lose any data.. because I've accumulated a long list of shows (mostly archived)
      • everything worked great.. so thanks for that :)
  2. regarding sort order.. I see that this code is how "sort by unseen" is processed
    • basically: order by count_of_unseen_episodes DESC, timestamp_of_release_of_next_unseen_episode ASC
    • imho, a chronological sort would be very useful:
      • shows with 1-or-more released episodes that have not yet been seen should be grouped first
        • order by timestamp_of_release_of_next_unseen_episode DESC
        • ie: order from newest release first.. to oldest release last
      • shows without any unwatched episodes should be grouped afterward
        • order by timestamp_of_release_of_next_unseen_episode ASC
        • ie: order by date that next episode will be released.. from soonest to farthest into the future

warren-bank avatar Apr 17 '22 08:04 warren-bank

@warren-bank 2. Sorry I fail to see te use in that. Why would you want to have a sorting order from newest release to oldest when new aired episodes are available? The intent of the current sorting order is to know for which shows it's safest to start binging without having to wait long for new episodes. What issue would your proposed sorting order solve?

ltguillaume avatar Apr 17 '22 11:04 ltguillaume

In contrast to binge-watching, when a series airs a new episode periodically (lets say one per week).. it's nice to have a sort option to see newest episodes at the top.

warren-bank avatar Apr 17 '22 22:04 warren-bank

Hmz, I always use the Pin feature for that, together with Exclude seen. It's much easier that way: the shows you're actively following will pop up at the top as soon as there are new episodes available. They will show up on the day of the broadcast.

ltguillaume avatar Apr 21 '22 13:04 ltguillaume

I understand what @warren-bank is referring to, but as an app that provides this type/way of tracking (show,, episode, etc...), I (personally) think this is just right.

It's simplicity (without the Bling, Banners, etc...) is awesome for its functional role.

This isn't to mean that there isn't room for improvement (as any app isn't perfect in on itself). I just believe that some major/minor features could (potentially) over complicate it's overall functionality as it was meant to.

This is just a personal opinion, suggestion, etc and I hope I had expressed it okay via text. πŸ‘

~Ibuprophen

Ibuprophen avatar Apr 21 '22 16:04 Ibuprophen

Well, I'm still willing to look into small improvements. Bigger ones are just not really in the cards, because frankly, it's sort of a miracle that DroidShows still works, it using the APIv1 of TheTVDB and all...

ltguillaume avatar Apr 25 '22 14:04 ltguillaume

I understand @ltGuillaume. πŸ’―

I did notice this too and I think that I read that they were continuing the backwards compatibility for the API based upon various feedback from the issues experienced by others with the newer API.

I just figured that once something big goes wrong, it'll probably/might be something tied to the API... LOL!

~Ibuprophen

Ibuprophen avatar Apr 25 '22 15:04 Ibuprophen

@warren-bank Just wanted to say that I'm following your fork's development and it's pretty great! πŸ˜ƒ

ltguillaume avatar May 22 '22 20:05 ltguillaume

Thanks.. really appreciate it! I was going to say something once I finished the set of features that I'm actively working on.

In a nutshell.. right now I'm just ironing out some features that aren't related to the API. Once that's done, I'll start another branch to perform migration from thetvdb to tmdb. That should ensure long-term survival.

warren-bank avatar May 22 '22 22:05 warren-bank

Sounds great! I hope you find a way to reliably migrate the shows. Obviously the name only is not enough, but most shows have an IMDB id and a Zap2it id, that could help out. For example, in my database there are only two shows that don't have an IMDB id.

I also have a few fixes that I did only locally (shame on me), so I'll figure out what's worth implementing and let you know.

ltguillaume avatar May 22 '22 22:05 ltguillaume

By any chance.. does one of your fixes include figuring out the reason that the app crashes when scrolling/flinging through the list and it reaches the last entry? That's a major annoyance.. and I was just about to look into it.

Without having started looking (ie: intentionally causing the crash and then inspecting logcat).. I'm guessing it's a memory issue that would be solved by using a RecyclerView (if we were using support libs.. which I totally agree would add some unwanted bloat). But who knows.. since each list item is so lightweight.. maybe it's something else..

update: I'm not sure what to make of this.. I'm testing the release thetvdb/007.11.05-09API.. and can no-longer reproduce this kind of crash. I have no idea if it was inadvertently fixed (in my forked repo) somewhere along the way. Profit?!

warren-bank avatar May 23 '22 03:05 warren-bank

Do you have this on multiple devices? Because I've never experienced this. Perhaps reinstalling and restoring the database with my latest build also fixes it on your device?

ltguillaume avatar May 23 '22 06:05 ltguillaume

The Android TV Box that's attached to my bedroom TV.. currently using.. is still running your build; the one you uploaded to this issue w/ the added "FLAG_ACTIVITY_NEW_TASK" flag when starting Intents. The box is old.. MXIII running Android 4.4 w/ 2GB RAM. Still works great.. so I haven't bothered to replace it.

Anyway.. I just ran a quick test on it (while a video player continued to stream video in the background).. and the crash happened.. and logcat shows (abridged):

  • OutOfMemoryError at:
    • android.graphics.BitmapFactory.decodeFile()
    • android.graphics.drawable.Drawable.createFromPath()
    • nl.asymmetrics.droidshows.DroidShows$SeriesAdapter.getView (line 1897)

where the originating code is here

just to be on the safe side.. I think I'll add a try/catch to prevent this type of crash on low-memory devices.

update: here is the relevant commit, which is included in release thetvdb/007.11.06-09API

update: OutOfMemoryError is a fatal exception.. catching it doesn't prevent a crash.

update: commits 1 and 2 work to conserve memory by resampling every thumbnail and only caching/displaying bitmaps that are 100px x 100px. I don't know how long a list needs to be for this to consume enough heap to lead to a crash.. but my list is 140 long, I'm using a fairly ancient device, and it's working great.. no lag at all.. and more-importantly.. no crashing!

room for improvement: my methodology for cropping the posters is open for discussion. The posters appear much taller than they are wide. After resampling, I'm taking the full width and cropping height (to make a square) equally from top and bottom to take from the center. It looks fine to me, but there is definite crop loss.

warren-bank avatar May 23 '22 08:05 warren-bank

I had done a fresh install test on both versions (individually) and, thus far, saw a minor issue between the two.

The following is a Screenshot of the latest version from @ltGuillaume (version 7.11.5) which shows the artwork/cover images as it should be.

Screenshot_ver 7 11 5

The following is a Screenshot of the latest version from @warren-bank (version 007.11.07-09API) and (as you can see) had noticed the artwork/cover images not sized correctly.

Screenshot_ver 007 11 07-09API

Other than the artwork, I didn't notice (personally) anything that jumped out at me, but I would definitely let you know. :-))

I've gotta state that I'm also very impressed with the contribution that @warren-bank had done too.

Great Job!!!

~Ibuprophen

Ibuprophen avatar May 24 '22 19:05 Ibuprophen

  1. I added the 3 commits I only had locally (one of them is adding FLAG_ACTIVITY_NEW_TASK), so you can cherry-pick if you need them in your fork, too.

  2. I never furthered the implementation of the optional DVD order. If you think it'd be a useful feature, too, would you like to have a look at it? Otherwise, I probably will, but not in the near future.

About the OutOfMemoryError, that's so odd, really. I've been using DroidShows with lists of at least 80 shows on an HTC Desire (2010) until 2020 (yes, it's been my daily driver for 10 years πŸ˜›). I thought that would be a very good benchmark for memory issues in any case. I did some testing and found the current way to be very responsive and not too hungry. Android does moest of the management in lists anyway.

That being said, and thinking about this some more, the Desire doesn't have a high resolution (800x480). I don't even remember anymore how the image sizes are calculated before saving them, but it's possible that the image sizes explode on high-res displays.

Either way, if the issue @Ibuprophen explained is to be fixed and considering high-res displays, I guess there's something to say for using a different approach for showing the posters in the ListView.

ltguillaume avatar May 25 '22 14:05 ltguillaume

Also, I probably see the effect of https://github.com/warren-bank/fork-Android-DroidShows/commit/891a1ba9eddee5eb64e970bdb591392f20e6de39 on @Ibuprophen's screenshot https://user-images.githubusercontent.com/18060271/170116416-5204ca7a-fa05-4b75-9d6a-7a331cd9527b.jpg and I'd say that's a definite regression, too. What was the idea exactly?

ltguillaume avatar May 25 '22 14:05 ltguillaume

I'll re-address the icon cropping. I'm currently in the weeds with respect to data API migration. But before committing any code for that, I'll change the icon methodology: only resize such that width is always 100dp.. allow height to maintain aspect ratio.. update layout as such. Shouldn't be a problem... image size will still be very small.. aspect ratio will be maintained.. no cropping will occur. TBA..

PS: the methodology in the old code was to: resize by height (25% of screen) and maintain aspect ratio for width. Using your phone's specs as an example: screen height is 800px.. height of images after resize are 200px. Larger (or higher density) screens produce larger images that are accumulated in memory. With regard to memory, not horrible.. but we can do better. With regard for layout, you were lucky that thetvdb must pre-process all of its posters to a normalized aspect ratio; otherwise the icon widths could've been wildly inconsistent.. and the list would've looked a complete mess.

update: it just occurred to me.. since my Android box is connected to my TV over HDMI.. it's probably seeing that my screen dimensions are 4K.. 3840px width x 2160px height.. so if "icons" are resized to 25% of this height.. that's 540px each.. ouch! That could probably lead to a memory-related crash with only a moderately sized list.

warren-bank avatar May 25 '22 17:05 warren-bank

With regard for layout, you were lucky that thetvdb must pre-process all of its posters to a normalized aspect ratio; otherwise the icon widths could've been wildly inconsistent.. and the list would've looked a complete mess.

That's true. but I knew about this, so I could make use of it πŸ˜‰

ltguillaume avatar May 25 '22 18:05 ltguillaume

release: 'thetvdb/007.11.09-09API'

  • fixes the icon issue
  • fixes #106 by adding the requested sort option
  • fixes the fact that the episode details page was able to add calendar events for the air date of episodes that had already been aired
    • now that feature is only enabled for episodes that have not yet been aired (ie: the date is at some time in the future)

warren-bank avatar May 26 '22 11:05 warren-bank

Impressive! I think maybe we should look into how you could co-manage this very repo, so releases to F-Droid could start up again, while users can seemlessly update? What do you think?

I could build & sign the APK's here, but the signing key for DroidShows is used only for DroidShows, so I can eventually share it with you if you'd like.

ltguillaume avatar May 26 '22 12:05 ltguillaume

10 days later.. and version tmdb/008.00.00-09API is born.

I could use some testers.. because about 85% to 90% of the code in the entire app has been either replaced or rewritten.

notes:

  • the API is now using TMDB v3 (rather than TheTVDB v1)
    • which is entirely different
    • TheTVDB v1
      • a request for a TV series returns a huge monolithic response that contains all related data
    • TMDB v3
      • a request for a TV series returns only data about the series and a minimal amount of data about the number and size of its seasons
      • a request for a season returns a limited amount of data for the episodes in the season
        • this data isn't sufficient for our needs.. so this API endpoint isn't used
      • a request for an episode returns only data about the individual episode
  • databases can be updated from older releases, which involves:
    • migrating to an entirely new database schema
    • re-adding all TV series
      • which can take a while
      • a progress dialog shows.. progress
    • updating the new DB with data from the old (ex: archived, pinned, external resources, seen, ...)
      • data about which TV series are "pinned" has been moved from shared preferences into the DB
  • updates to each TV series involves:
    • updating data about the series
    • replacing data about the seasons (which is included in the response for data about a series)
    • requesting only the new episodes
      • old episodes are left intact, and no API requests are made to modify them

warren-bank avatar Jun 05 '22 02:06 warren-bank

This is great news @warren-bank!

I'll test it out and report back anything that I personally notice.

Is there anything specific on your mind that you would like me to keep an eye on or just in general?

~Ibuprophen

Ibuprophen avatar Jun 05 '22 02:06 Ibuprophen

nope.. just anything that's not working right, or could be improved upon.

As for speed.. there's nothing I can do. This API requires many more requests to accumulate the same quantity of information. Plus, it uses a library to convert JSON to Object models.. rather than extracting fields directly from raw XML.

In any case, now we have an option..

  • the old version of the app will continue to work great.. so long as TheTVDB v1 remains alive
  • the new version (ie: v8.0.0+) is available.. and should continue to work for a long time to come

warren-bank avatar Jun 05 '22 02:06 warren-bank

@ltGuillaume ..quick question

I just noticed your commit..

  • where the underlying issue is that when the name of a series is passed as a querystring parameter in a URL.. it should be encoded.. but hadn't been
    • in Javascipt.. we have: encodeURIComponent(name)
    • in Java.. we have: URLEncoder.encode(name, "UTF-8")
  • when looking at the code where this occurs.. I also notice that:
    • name = name.replaceAll(" \\(....\\)", "")
    • what is the intended purpose for this?
      • are the 4 characters inside the parenthesis assumed to be a year? (ex: " (2022)")
    • looking at a random sampling of series names returned in search results.. none include a matching substring
    • and considering that the TMDB branch is using names from an entirely different API.. I'm very tempted to get rid of these regex search/replace operations.. but figured I'd ask first

warren-bank avatar Jun 05 '22 10:06 warren-bank

@ltGuillaume ..quick question

I just noticed your commit..

* where the underlying issue is that when the name of a series is passed as a querystring parameter in a URL.. it should be encoded.. but hadn't been
  
  * in Javascipt.. we have: `encodeURIComponent(name)`
  * in Java.. we have: `URLEncoder.encode(name, "UTF-8")`

I ran into new problems with .encode, but I agree this is a hacky solution (as said in the commit message).

* when looking at the code where this occurs.. I also notice that:
  
  * `name = name.replaceAll(" \\(....\\)", "")`
  * what is the intended purpose for this?
    
    * are the 4 characters inside the parenthesis assumed to be a year? (ex: " (2022)")
  * looking at a random sampling of series names returned in search results.. none include a matching substring
  * and considering that the TMDB branch is using names from an entirely different API.. I'm very tempted to get rid of these regex search/replace operations.. but figured I'd ask first

This is because (at least in my collection) many show names have their year appended (e.g. Archer (2009)). I found that IMDB did not appreciate it if (2009) was included in the search term, so I decided to strip it from the show name.

As you said, TMDB might not append the year to show names.

ltguillaume avatar Jun 05 '22 13:06 ltguillaume

10 days later.. and version tmdb/008.00.00-09API is born.

I could use some testers.. because about 85% to 90% of the code in the entire app has been either replaced or rewritten.

notes:

* the API is now using TMDB v3 (rather than TheTVDB v1)
  
  * which is entirely different
  * TheTVDB v1
    
    * a request for a TV series returns a huge monolithic response that contains _all_ related data
  * TMDB v3
    
    * a request for a TV series returns only data about the series and a minimal amount of data about the number and size of its seasons
    * a request for a season returns a limited amount of data for the episodes in the season
      
      * this data isn't sufficient for our needs.. so this API endpoint isn't used
    * a request for an episode returns only data about the individual episode

* databases can be updated from older releases, which involves:
  
  * migrating to an entirely new database schema
  * re-adding all TV series
    
    * which can take a while
    * a progress dialog shows.. progress
  * updating the new DB with data from the old (ex: archived, pinned, external resources, seen, ...)
    
    * data about which TV series are "pinned" has been moved from shared preferences into the DB

* updates to each TV series involves:
  
  * updating data about the series
  * replacing data about the seasons (which is included in the response for data about a series)
  * requesting only the new episodes
    
    * old episodes are left intact, and no API requests are made to modify them

Wow! Did I miss a few commits somehow, or is basically everything you describe here in https://github.com/warren-bank/fork-Android-DroidShows/commit/e237faeba00e42605cf33538d9c5ff206e997e36 and https://github.com/warren-bank/fork-Android-DroidShows/commit/8cb1a56463b2db77464d1f0c0c25c26aae15f9e4 ?

Either way, impressive work! I'll be sure to take a better look at all the changes soon, just wanted to thank you first πŸ˜ƒ

ltguillaume avatar Jun 05 '22 13:06 ltguillaume

Nope. You didn't miss anything; I was working offline. Maybe it's just me, but I don't like to commit code in a state that can't compile. And to do this.. I needed to tear everything apart and break it.. in order to put it back together again.. differently. Once I had a version that compiled and was ready to show.. then I commited the end result. Now, as I make minor tweaks and improvements.. I'll make small individual commits.

Thanks for confirming that the regex was to strip off a year. Offhand, do you remember why URLEncoder.encode didn't work when you tested it? Encoding querystring values seems to be its raison d'Γͺtre. I could include a custom function that percent encodes any character that isn't [a-zA-Z0-9].. but it would be easier to use what's already provided.

warren-bank avatar Jun 05 '22 19:06 warren-bank

Nope. You didn't miss anything; I was working offline. Maybe it's just me, but I don't like to commit code in a state that can't compile. And to do this.. I needed to tear everything apart and break it.. in order to put it back together again.. differently. Once I had a version that compiled and was ready to show.. then I commited the end result. Now, as I make minor tweaks and improvements.. I'll make small individual commits.

Makes sense!

Thanks for confirming that the regex was to strip off a year. Offhand, do you remember why URLEncoder.encode didn't work when you tested it? Encoding querystring values seems to be its raison d'Γͺtre. I could include a custom function that percent encodes any character that isn't [a-zA-Z0-9].. but it would be easier to use what's already provided.

I just replaced every %26 .replaceAll() with Uri.encode() for the parameter strings and that works fine now. So I might have just screwed up before. I can't really remember, as I was feverish when I made the change, but I just wanted to help out @Ibuprophen as quick as possible.

ltguillaume avatar Jun 05 '22 20:06 ltguillaume

@warren-bank, I did notice a little something...

I had first (of course) backed up the DroidShows folder to another location to keep a copy of my shows database.

I had decided to test your app as a fresh install where I had cleaned the cache, uninstalled the app and removed the DroidShows folder and such.

It installed fine, I added 1 show (as I did notice the slight slowness) and then when I selected to backup, I did quickly notice that it didn't create the backup directory as reflected in an error that it couldn't do so because it couldn't locate the directory.

I'll work with it some more and let you know of anything else.

Thanks a Bunch! πŸ‘

~Ibuprophen

Ibuprophen avatar Jun 05 '22 20:06 Ibuprophen

@warren-bank, I did notice a little something...

I had first (of course) backed up the DroidShows folder to another location to keep a copy of my shows database.

I had decided to test your app as a fresh install where I had cleaned the cache, uninstalled the app and removed the DroidShows folder and such.

It installed fine, I added 1 show (as I did notice the slight slowness) and then when I selected to backup, I did quickly notice that it didn't create the backup directory as reflected in an error that it couldn't do so because it couldn't locate the directory.

I'll work with it some more and let you know of anything else.

Thanks a Bunch! πŸ‘

~Ibuprophen

This might actually be something related to DroidShows in general and Android's increasingly restrictive storage policies? Are you sure you're not getting the same thing with a clean install of one of my latest releases?

ltguillaume avatar Jun 05 '22 21:06 ltguillaume

That is odd.. this is all done pretty much identical to the way it was done upstream..

  • code
      backupFolder = sharedPrefs.getString(BACKUP_FOLDER_PREF_NAME, (Environment.getExternalStorageDirectory() + "/" + getString(R.string.layout_app_name)));
    
  • code
      File destination = new File(backupFolder, FileUtils.getDatabaseFileName(DroidShows.this));
    
  • code
      File folder = new File(backupFolder);
      if (!folder.isDirectory())
        folder.mkdir();
    

warren-bank avatar Jun 05 '22 22:06 warren-bank

it would be helpful when odd behavior is described.. to also include the version of Android in which the behavior is observed.

as Guillaume said, Android consistently breaks user space with each update.

warren-bank avatar Jun 05 '22 22:06 warren-bank

I see the problem (that causes an error when performing a backup on newer devices)..

on Android 6.0+:

  • need to request permissions at runtime
    • Settings > Apps > DroidShows > Permissions
      • Storage = true
  • this is probably only impacting my recent build, because I increased the target SDK
    • didn't see any reason for it to be set so low
    • obviously, I kept the min SDK alone.. to support old devices
  • I'll need to add a runtime check and ask for permission if not already granted

warren-bank avatar Jun 05 '22 22:06 warren-bank

@warren-bank & @ltGuillaume, that was a great suggestion that prompted me to take a look and found that it requires the user to enable the Storage Permissions in the System Settings.

I enabled it and it was all set. πŸ‘

Apps that requires certain permissions are typically set up to ask for those permissions via a pop-up screen (I hope I explained that okay... LOL!). This was why it didn't occur to off hand.

~Ibuprophen

Ibuprophen avatar Jun 05 '22 23:06 Ibuprophen

@warren-bank & @ltGuillaume, that was a great suggestion that prompted me to take a look and found that it requires the user to enable the Storage Permissions in the System Settings.

I enabled it and it was all set. πŸ‘

Apps that requires certain permissions are typically set up to ask for those permissions via a pop-up screen (I hope I explained that okay... LOL!). This was why it didn't occur to off hand.

~Ibuprophen

Great! FYI, @warren-bank has addressed this in https://github.com/warren-bank/fork-Android-DroidShows/commit/99e36efa77a3eecb522ba233038bb637400344f2

ltguillaume avatar Jun 07 '22 21:06 ltguillaume

@warren-bank, I just tested a fresh install of DroidShows-008.00.02-09API-release and the permissions pop-up worked great.

But, restoring the database from the one by @ltGuillaume didn't work out well (for me anyways).

1 - The following image reflects the file backup placed in the backup directory (note the file size).

1 - Copied Backup

2 - The following reflects 1 (of 2) toast messages.

2 - Restore Toast 1

3 - The following reflects 2 (of 2) toast messages and you'll see nothing restored.

3 - Restore Toast 2

4 - The following reflects the results of the database files in the backup directory after performing the restoration (please note the file sizes).

4 - Restore Results

5 - Just as a reference, the following is what's reflected in the latest version released by @ltGuillaume.

5 - DroidShows by ltGuillaume

... on another note...

The following reflects the huge add symbol after searching for a show, but that's minor in on itself.

6 - Large Add Symbol on Right

I hope I had explained the above okay via text.

Thanks a Bunch!

~Ibuprophen

Ibuprophen avatar Jun 08 '22 16:06 Ibuprophen

did the DB migration open a progress dialog.. for a long/slow update?

if you want to share a DB backup file.. I can test it here. the one I'm testing is small, but migration works well.

warren-bank avatar Jun 08 '22 17:06 warren-bank

No. I waited a few minutes and the progress indicator didn't pop-up.

Afterwards, I had performed a fresh install of the latest version by @ltGuillaume to perform a restore to be sure that the backup was a good one and it imported without an issue.

The following is a copy of my backup database for you to try.

DroidShows.zip

I'm not in a rush/race. Please take your time as I do understand that it can be challenging in application developments (as I have a few of my own). πŸ‘

~Ibuprophen

Ibuprophen avatar Jun 08 '22 19:06 Ibuprophen

@warren-bank Perhaps it's an idea to create the releases with a different package name (e.g. bank.warren.droidshows) and app name (DroidShows Ξ²?) for as longa as your fork is in alpha/beta? Could be useful for testing purposes.

ltguillaume avatar Jun 08 '22 20:06 ltguillaume

I agree with you @ltGuillaume regarding the temporary change in the package name during the Alpha/Beta testing.

Also a temporary app name as well so others will be able to identify which DroidShows app is which (for example "DroidShows 2" or something).

~Ibuprophen

Ibuprophen avatar Jun 08 '22 20:06 Ibuprophen

@Ibuprophen

  • I'll look at your DB now.. and see if I can reproduce a problem
  • PS: regarding the file size of the files in your backups directory..
    • .db is the backup created of the empty database that was created when you installed the app
    • .db1 is the backup that you were attempting to restore
      • you should probably rename this file so it doesn't get accidentally deleted
      • only 3 backups are saved.. having rotating filename extensions (.db > .db1 > .db2 > [deleted])
    • if you want to see the file size of the dabase after having been restored.. after having been updated (w/ data being completely replaced by the migration process from thetvdb to tmdb), then you should make another backup

@ltGuillaume

  • regarding package name.. that's entirely your call
    • as a rule, I always leave the package name of apps that I fork intact
    • I feel to do otherwise is akin to theft
      • ..or at the very least is highly disrespectful to the original author
    • if you'd like me to rename the app and package.. so the two can be installed at the same time, then I'll go ahead and do so

warren-bank avatar Jun 08 '22 20:06 warren-bank

@warren-bank, what I meant by the size was for identifying it for comparison.

I deleted the initial (blank) database, placed a copy of my backup database.

Then after the restore was performed (as I had reported), the backup directory looked like the app had taken my copy and renamed the extension to "db1" and created an empty one with the "db" extension.

~Ibuprophen

Ibuprophen avatar Jun 08 '22 20:06 Ibuprophen

ok, well.. idk.. but what I can say.. is that the update process began after I restored your DB file.. and it's currently running; I'll upload the resulting .db file after it completes.

but.. I have no idea why it didn't start for you.

PS: testing on Android 8.1

warren-bank avatar Jun 08 '22 20:06 warren-bank

considering how long this process can take.. I'm thinking that I should add a wakelock.. for cpu and wifi, then release it after the migration finishes.

warren-bank avatar Jun 08 '22 20:06 warren-bank

yep.. sure enough i turned my screen off.. and the phone went to a low power state that killed the update.. resulting in an incomplete migration that only has the shows that were updated before the process was killed.

that being said.. the process was working; I have a few Star Trek shows in my list now; but I definitely need a wakelock.

I hope that's sufficient.. and that I won't need to use a foreground service.

warren-bank avatar Jun 08 '22 20:06 warren-bank

Yeah, I didn't necessarily mean permanently, I just saw @Ibuprophen going back and forth between the two by uninstalling, and I would like to test it myself, ideally without touching the current install until it's stable. It'd also be nice to have both installed at the same time and then be able to compare the info of TMDB and TVDB.

ltguillaume avatar Jun 08 '22 20:06 ltguillaume

As for the process of restoring/updating, I found that on my devices DroidShows continued either while the screen was turned off or at least after the screen was turned back on again, but I was always on a custom ROM, so hard to say how all those stock ROMs will react to it.

ltguillaume avatar Jun 08 '22 20:06 ltguillaume

status update..

the following are all done:

  • renaming of app and package
  • adding cpu and wifi wakelocks
  • release an updated apk

the result:

  • the wakelocks make all the difference..
    • I've been restoring Ibuprophen's .db backup for a while now
    • screen is off and I'm checking on its progress every 10 minutes or so
    • the update is running and the progress is increasing.. slowly
      • sufficed to say that this is a one-time thing.. during migration from an older .db
      • not something a new user would experience
    • but most importantly.. the wakelocks are keeping the update going while the screen is off
      • they'll release once the update completes

warren-bank avatar Jun 09 '22 01:06 warren-bank

status update:

  • the code to manage the wakelocks needs work..
    • I forget that turning off the display screen calls an Activity's onStop() hook
    • I was killing the wakelocks every time the screen turned off
    • what's interesting is that the behavior of the app during DB update was better in this version than the previous version..
      • no idea why
    • but I'm testing an improved version right now...
      • it's updating much faster
      • so I'll be pushing an update soon

warren-bank avatar Jun 09 '22 04:06 warren-bank

Thx! I just installed the latest release and tried to restore a database from DroidShows v7.11.5.

  • TV Tracker created a backup DroidShows.db (empty, so it's not useful)
  • It renamed the previous DroidShows.db to DroidShows.db0
  • It says it successfully restored the backup, but the list remains empty
  1. Does backing up before restoring a database really make sense?
  2. It certainly makes no sense to back up an empty database
  3. Also, versioning is used in this process even though it's disabled. You might call it good that it does so in this particular case, because otherwise the backup you want to restore would have been gone before it even tries to restore it, but as said, I think it shouldn't automatically create a backup before restoring at all.
  4. I think that due to this versioned backup beforehand, TV Tracker restores the empty database it just backed up, not the file you actually picked, because that got renamed to DroidShows.db0

Edit: I can't add any show manually either. No matter what the search terms are (even "the"), no shows are found. So (3) might also have been caused by this, but the toast states the restore operation was successful.

  1. The spinner doesn't show Active/Archived/Log, it's just black.
  2. The spinner's icon is (still) botched (too small, not sharp anymore). Not sure what caused your decision to resize it?

what's interesting is that the behavior of the app during DB update was better in this version than the previous version...

Not sure. Perhaps you've been testing with manually turning off the screen instead of the "screen-on" timing out by itself? There might be a difference between these two situations considering wake state? Just brainstorming here.

ltguillaume avatar Jun 09 '22 04:06 ltguillaume

@Ibuprophen

using release tmdb/008.00.04-09API, migration of your DB took me almost exactly 30 minutes:

9:15 0% 9:20 20% 9:25 33% 9:30 43% 9:35 57% 9:40 87% 9:45 100%

here is the updated DB file.. DroidShows-Ibuprophen-v8.zip

warren-bank avatar Jun 09 '22 05:06 warren-bank

@ltGuillaume

  • something you just said is interesting..
    • when restoring any of the default backup filenames.. and a backup occurs before the restore.. the filepath no-longer points to the desired DB file
  • that's something I hadn't considered.. or tested for.. since I always rename my backups
    • I'll need to think about this..
  • regarding taking an automatic backup prior to a restore..
    • I wasn't aware that I had changed the behavior from the upstream app
    • I'll need to look into this, as well..
    • if this was simply a mistake, then I guess I can kill 2x birds..

warren-bank avatar Jun 09 '22 05:06 warren-bank

@ltGuillaume dumb question.. but which commit corresponds to: DroidShows v7.11.5

the most recent release is titled 7.11.4 but tagged 7.11.2 ..so I assume that this is in line with F-Droid v7.11.2

the reason I ask.. is that if you made any changes to your "current" database version number.. then our two builds would be in conflict

warren-bank avatar Jun 09 '22 05:06 warren-bank

I pretty much screwed up ages ago by committing https://github.com/ltGuillaume/DroidShows/commit/238f5f6998859963d4cecf98a684c1e036fe2b4a, because I never really committed those changes to any of the released APK's afterwards. 7.11.5 is just an internal version that has the commits I recently added (so still without that DVD Order commit, but with the Revert backups being triggered by OnStop() instead of onDestroy() (https://github.com/ltGuillaume/DroidShows/commit/e24885b3129766baab606d29353da011abc2c1a1) change). I believe you forked before that DVD Order commit, so there's no change in database version number (I just checked the exported database itself to be sure, it's 0.1.5-7G3, not 0.1.5-7G4).

ltguillaume avatar Jun 09 '22 06:06 ltguillaume

ok, good.. so that's not it; all of my DB updates use 0.1.5-7G3 as the starting point for making various incremental changes.

so.. I have no idea why nothing in the app is working for you; I'm almost wondering whether my TMDB API key is only authorizing requests coming from my IP address. I can't imagine that could be the case.. but why else is everything working for me.. on every device I try.. and nothing working for either of you? I'll test the API through a VPN and see if I can make requests.

warren-bank avatar Jun 09 '22 06:06 warren-bank

I'll check the logs for clues later today.

ltguillaume avatar Jun 09 '22 06:06 ltguillaume

nope.. IP whitelisting isn't the problem

warren-bank avatar Jun 09 '22 06:06 warren-bank

06-09 15:13:32.377 D/themoviedbapi(27127): Method: 'search', Sub-method: 'tv', Params: TmdbParameters[parameters={LANGUAGE=en, PAGE=1, QUERY=wire}] 06-09 15:13:32.378 D/themoviedbapi(27127): URL: http://api.themoviedb.org/3/search/tv?api_key=c9eb196aaf70baf91d4ce4f0fea6a360&query=wire&language=en&page=1 06-09 15:13:32.385 W/Adreno-EGL(27127): <qeglDrvAPI_eglGetConfigAttrib:607>: EGL_BAD_ATTRIBUTE 06-09 15:13:32.389 E/DroidShows(27127): ExceptionType=HTTP_503_ERROR, ResponseCode=503, URL=http://api.themoviedb.org/3/search/tv?api_key=c9eb196aaf70baf91d4ce4f0fea6a360&query=wire&language=en&page=1

503 (Service unavailable). Strange stuff. While I am using a VPN, it's not that: I can open the URL above in a browser on the same device without any problems.

ltguillaume avatar Jun 09 '22 13:06 ltguillaume

That's interesting..

Here are the relevent lines of code:

  • code
      private static final int SC_INTERNAL_SERVER_ERROR = 500;
      private static final int SC_SERVICE_UNAVAILABLE = 503;
    
  • code
      if (response.getStatusCode() >= SC_INTERNAL_SERVER_ERROR) {
        throw new MovieDbException(ApiExceptionType.HTTP_503_ERROR, response.getContent(), response.getStatusCode(), url, null);
      }
    
  • code
      private DigestedResponse requestContent(...) {
        try {...}
        catch(Exception e) {
          int statusCode = SC_SERVICE_UNAVAILABLE;
        }
      }
    

requestContent(...) must be throwing, causing the method to return SC_SERVICE_UNAVAILABLE.

The problem is, I can't reproduce an error that would allow me to debug.

Full disclosure, the library originally used Apache HttpClient to implement this class; I rewrite it and removed all external dependencies.

warren-bank avatar Jun 09 '22 18:06 warren-bank

The connection timeout and read timeout are both set to 15 seconds.. do you think there's any chance that your requests could be timing out? ..maybe I should increase these 2x values.

warren-bank avatar Jun 09 '22 18:06 warren-bank

If you could set a break point at line 191 ..in the catch block.. I'm really curious what that Exception is.

warren-bank avatar Jun 09 '22 18:06 warren-bank

No way it's a timeout, the result is instant.

ltguillaume avatar Jun 09 '22 18:06 ltguillaume

I don't even have Android Studio, so debugging this project is a bit too much work for me right now, sorry.

ltguillaume avatar Jun 09 '22 18:06 ltguillaume

Just a quick FYI (in case @ltGuillaume & @warren-bank weren't aware of this)...

The following link is to the Github page for TheTVDB (Official) Repositories that may hopefully be of some help.

https://www.github.com/thetvdb

~Ibuprophen

Ibuprophen avatar Jun 09 '22 18:06 Ibuprophen

no worries.. I figured that you already had it fired up :smile: ..didn't intend to ask you to go out of your way.

I just have no idea why I can't reproduce the problem.. you said that you were using a custom ROM.. is it LineageOS? ..I haven't tested that yet; only stock ROMs (on various devices) so far.

warren-bank avatar Jun 09 '22 18:06 warren-bank

Nah, I started out in Eclipse and never switched, because I'm stubborn and because I hate all the default dependency stuff of it. I thought I might switch as soon as I got involved in other Android projects, but I lost interest because of Google's inherent tendency of destroying everything good, especially when it comes to the Android ecosystem.

Yeah, LineageOS 18.1 (= Android 10) on a Samsung Galaxy S5.

ltguillaume avatar Jun 09 '22 18:06 ltguillaume

oh perfect.. I have a Galaxy S5 with LineageOS around here somewhere. hopefully I can fix this issue by testing with that.

warren-bank avatar Jun 09 '22 18:06 warren-bank

regarding Google/Android.. the way I feel is that so long as we continue to be able to side-load our own APKs.. I'm not going to be negative towards them.

As ridiculous as it is to say.. "so long as".. I'm allowed by my computer to install applications..

But this is the direction everything else has gone.. Firefox.. Chrome.. walled gardens everywhere; now that really pisses me off..

Indirectly related to this.. is the fact that Android developers have this tendency that makes absolutely no sense to me.. to intentionally increase the min SDK of apps for no good reason; that also really pisses me off.. but we can't blame Google for that.

warren-bank avatar Jun 09 '22 18:06 warren-bank

Well, do they increase the minSDK "because they can" or simply because they've got 1024 dependencies that would otherwise stop working?

ltguillaume avatar Jun 09 '22 19:06 ltguillaume

very true..

although.. I'll use Google Voice as an example.. because it's a utilitarian app with requires almost no UI.. and lots of people have come to depend on it..

  • the backend service hasn't changed
    • the oldest possible version of the app still works on phones that installed it back in the day
    • user authentication doesn't work in this version any longer, so if you log out (or clear data).. you're SOL
  • a few years ago, they decided to cripple all but the most recent version with a version check at startup that shows a modal dialog that forces an upgrade
  • the most recent version has a high min SDK
    • so older devices are simply SOL

the whole situation is dumb..

warren-bank avatar Jun 09 '22 19:06 warren-bank

And I think being able to sideload apps isn't enough. Much of the core OS features, previously open-source, have been disappearing for years now into a closed-source abomination of a black hole called Google Play Services. Apps simply have to use it for specific device features, and no-one knows what data is sent through it. I think we can safely say by now: everything. And not just the stuff that actually needs a part of the Play Services is victim of this hoarding, even SMS message contents are sent directly to Google servers. Basically everything with a screen is a honeypot now, it's almost a crime we even have to pay for the hardware πŸ˜›

ltguillaume avatar Jun 09 '22 19:06 ltguillaume

I'm definitely not disagreeing with you.. you're not wrong!

idk enough about building ROMs to really speak to this.. but it seems that projects like LineageOS are still able to take the good bit that are open sourced.. and build a pretty incredible OS that's free from Google's spying eye.

warren-bank avatar Jun 09 '22 19:06 warren-bank

hashtag: #RichardStallmanWasRightAllAlong

too funny.. there's actually a Reddit thread specifically for that

warren-bank avatar Jun 09 '22 19:06 warren-bank

hashtag: #RichardStallmanWasRightAllAlong

too funny.. there's actually a Reddit thread specifically for that

Hah, no surprises there πŸ˜›

I'm looking into https://divestos.org, a soft fork of LineageOS that tries to remove as many proprietary blobs (e.g. binaries of hardware manufacturers), offers a more privacy-oriented System WebView by default, includes Mull (Firefox+arkenfox, which I already use) and some other interesting apps and features.

ltguillaume avatar Jun 09 '22 19:06 ltguillaume

sir, I like the way you think :smiley:

warren-bank avatar Jun 09 '22 19:06 warren-bank

I have a handful of other Android devices I use for testing purposes. They all have the TWRP custom recovery and various LineageOS Android Versions (I've been using those Custom Firmwares since CyanogenMOD).

I'm also one of the Developmental Team Members for the Classic Shell (Windows Start Menu) Software.

With those and the Ancestry/Genealogy things I love doing keeps me very busy while Retired.

LOL! πŸ‘

~Ibuprophen

Ibuprophen avatar Jun 10 '22 00:06 Ibuprophen

easy fix.. coming soon..

java.io.IOException: Cleartext HTTP traffic to api.themoviedb.org not permitted

update:

  • release tmdb/008.00.05-09API adds the flag to allow unprotected http traffic
  • tested (quickly) on a Galaxy S5 with LineageOS (Android 11)
  • ..all good :)

PS:

  • it wasn't a stock vs. custom ROM.. thing
  • it was a "security feature" that was introduced in Android 9.. thing
    • and I had only tested on Android 4.4 and 8.1

warren-bank avatar Jun 10 '22 01:06 warren-bank

@Ibuprophen ..had to scroll up a ways to find the link ..don't forget that I've already prepared your DB for this version ..so you should be good to go

warren-bank avatar Jun 10 '22 02:06 warren-bank

Adding shows now works in the latest builds!

  1. Restoring a DroidShows database, however, returns the following:
D/DroidShows(12742):  Database update routine
D/DroidShows(12742):  Current database version: 0.1.5-7G5
D/DroidShows(12742):  Current database version: 0.1.5-7G5
E/SQLiteLog(12742):  (1) no such column: archived
D/DroidShows(12742):  Database update routine
D/DroidShows(12742):  Current database version: 0.1.5-7G5
D/DroidShows(12742):  Current database version: 0.1.5-7G5
D/DroidShows(12742):  Database needs update
D/DroidShows(12742):  Current database version: 0.1.5-7G5
D/DroidShows(12742):  UPDATING TO VERSION 0.1.5-7G6
D/themoviedbapi(12742):  Method: 'find', Sub-method: '', Params: TmdbParameters[parameters={EXTERNAL_SOURCE=tvdb_id, ID=110381}]
D/themoviedbapi(12742):  URL: http://api.themoviedb.org/3/find/110381?api_key=c9eb196aaf70baf91d4ce4f0fea6a360&external_source=tvdb_id
I/Choreographer(12742):  Skipped 49 frames!  The application may be doing too much work on its main thread.
I/ActivityTaskManager(844):  Displayed com.github.warren_bank.tiny_television_time_tracker/.DroidShows: +1s731ms
D/themoviedbapi(12742):  Method: 'find', Sub-method: '', Params: TmdbParameters[parameters={EXTERNAL_SOURCE=tvdb_id, ID=121361}]
D/themoviedbapi(12742):  URL: http://api.themoviedb.org/3/find/121361?api_key=c9eb196aaf70baf91d4ce4f0fea6a360&external_source=tvdb_id
D/themoviedbapi(12742):  TVEpisodeBasic: Unknown property='production_code' value=''
D/themoviedbapi(12742):  TVSeasonBasic: Unknown property='show_id' value='19239'
D/themoviedbapi(12742):  Method: 'find', Sub-method: '', Params: TmdbParameters[parameters={EXTERNAL_SOURCE=tvdb_id, ID=123581}]
D/themoviedbapi(12742):  URL: http://api.themoviedb.org/3/find/123581?api_key=c9eb196aaf70baf91d4ce4f0fea6a360&external_source=tvdb_id
W/System  (12742):  A resource failed to call end. 
W/System  (12742):  A resource failed to call close. 
I/chatty  (12742):  uid=10137(com.github.warren_bank.tiny_television_time_tracker) identical 3 lines
W/System  (12742):  A resource failed to call close. 
D/themoviedbapi(12742):  TVEpisodeBasic: Unknown property='production_code' value=''
D/themoviedbapi(12742):  Method: 'find', Sub-method: '', Params: TmdbParameters[parameters={EXTERNAL_SOURCE=tvdb_id, ID=134241}]
D/themoviedbapi(12742):  URL: http://api.themoviedb.org/3/find/134241?api_key=c9eb196aaf70baf91d4ce4f0fea6a360&external_source=tvdb_id
(...)
E/WindowManager(844):  App trying to use insecure INPUT_FEATURE_NO_INPUT_CHANNEL flag. Ignoring
D/themoviedbapi(12742):  TVEpisodeBasic: Unknown property='production_code' value=''
D/themoviedbapi(12742):  TVSeasonBasic: Unknown property='show_id' value='60622'
D/themoviedbapi(12742):  TVSeasonBasic: Unknown property='networks' value='[{id=88, logo={path=/aexGjtcs42DgRtZh7zOxayiry4J.png, aspect_ratio=1.677852348993289}, name=FX, origin_country=US}]'
D/themoviedbapi(12742):  Method: 'find', Sub-method: '', Params: TmdbParameters[parameters={EXTERNAL_SOURCE=tvdb_id, ID=269689}]
D/themoviedbapi(12742):  URL: http://api.themoviedb.org/3/find/269689?api_key=c9eb196aaf70baf91d4ce4f0fea6a360&external_source=tvdb_id
(...)
D/themoviedbapi(12742):  TVEpisodeBasic: Unknown property='production_code' value=''
D/themoviedbapi(12742):  Method: 'find', Sub-method: '', Params: TmdbParameters[parameters={EXTERNAL_SOURCE=tvdb_id, ID=277964}]
D/themoviedbapi(12742):  URL: http://api.themoviedb.org/3/find/277964?api_key=c9eb196aaf70baf91d4ce4f0fea6a360&external_source=tvdb_id
E/DroidShows(12742):  no results
E/DroidShows(12742):  Error creating a backup of user metadata to temporary tables.
E/DroidShows(12742):  java.lang.Exception: TMDB API failed to return the (internal) TMDB ID for a TV series that is identified by the (external) TVDB ID: 277964
E/DroidShows(12742):  	at com.github.warren_bank.tiny_television_time_tracker.database.Update.doTmdbApiMigration(Update.java:486)
E/DroidShows(12742):  	at com.github.warren_bank.tiny_television_time_tracker.database.Update.update_version_007(Update.java:301)
E/DroidShows(12742):  	at com.github.warren_bank.tiny_television_time_tracker.database.Update.updateDatabaseVersion(Update.java:203)
E/DroidShows(12742):  	at com.github.warren_bank.tiny_television_time_tracker.database.Update.access$500(Update.java:28)
E/DroidShows(12742):  	at com.github.warren_bank.tiny_television_time_tracker.database.Update$1.run(Update.java:84)
E/DroidShows(12742):  	at java.lang.Thread.run(Thread.java:919)
D/DroidShows(12742):  Current database version: 0.1.5-7G5
E/DroidShows(12742):  Attempt to update version of database schema has failed
  1. ... which makes me wonder what happens to shows that can't be found on TMDB? Seems like for now it just silently fails to add it?
  2. Every time you restart TV Tracker, it backs up the database before updating, eventually overwriting the backup(s) actually made by the user. I think it might be a good idea to have a separate filename pattern for backups before updating the database.

ltguillaume avatar Jun 10 '22 11:06 ltguillaume

  1. Some UI glitches:
TV Tracker DroidShows
Icon + text color
Odd unsharp shadows (also context menus)

Language special characters

ltguillaume avatar Jun 10 '22 11:06 ltguillaume

My apologies for the delay @warren-bank...

I downloaded the most recent app and the database file, but I hadn't tested it out yet (as I do plan to).

I was just curious about the successful ability to change the backup location to another internal directory.

For example...

@ltGuillaume uses the "/storage/emulated/0/DroidShows", as I was hoping to successfully add another directory manually like "/storage/emulated/0/DroidShows2" or similar.

This would only be during the time your app is in Alpha/Beta testing and I can test it without affecting the one from @ltGuillaume.

I'm actually thinking about installing a tracer like app for debugging purposes to provide you with the results file if necessary. I haven't used one in a little while since I haven't needed to.

Thanks a Bunch!

~Ibuprophen

Ibuprophen avatar Jun 10 '22 17:06 Ibuprophen

@Ibuprophen No need for @warren-bank to change anything, DroidShows already supports that if you create the folder with a file manager app first. You can then create a manual backup to that new folder, after which DS will ask which folder to use for daily backups from now on.

ltguillaume avatar Jun 10 '22 17:06 ltguillaume

@ltGuillaume

great feedback! ..really appreciate it

  • interesting that there are log messages from Choreographer about skipping frames because of work being done on the UI thread
    • all code pertaining to DB update happens in a separate thread
    • is this something that I need to investigate?
  • themoviedbapi is the library that's converting JSON received from the API's HTTP endpoints to Java Objects that model the data
    • it's a bit verbose, but that's a good thing
  • the big question.. what's the sane thing to do?
    • when migrating a DB from a previous version that contains data from TheTVDB
    • when one or more of its series cannot be found by TMDB's 'find' API endpoint?
      • this endpoint allows a lookup to query the TMDB id of a series by giving its unique id on one of several supported platforms; one of which is TheTVDB
    • the current behavior is to prevent performing any DB migration and to leave the old DB intact.
    • I agree.. this isn't ideal
      • considering that the app has given the user every opportunity to create a backup of the old DB, there's very little chance of losing any data by allowing the migration to proceed with the subset of series that it can identify
      • to your point, this chance would be farther reduced by giving the DB backup (in this particular case) a different filename
        • ex: DroidShows-preupdate.db
        • ex: DroidShows-preupdate.db1, DroidShows-preupdate.db2 ..with optional versioning
      • would it also be useful to keep a list of the TheTVDB id values that failed to resolve?
        • what could we do with this?
          • a Toast would be ugly and disappear too quickly to be of any use
          • save it to a text log file in the default backupFolder?
            • this could also include a list of URLs for the user to be able to easily lookup each id value at the TheTVDB API endpoint

warren-bank avatar Jun 10 '22 18:06 warren-bank

@warren-bank Your posts are difficult to reply to point-by-point. Perhaps it's easier to stick to numbered lists for this.

Skipping frames

I'm not sure if this is something problematic tbh. I see it all the time, this is an old device (2014), and it was during updating the database. If something like this would happen consistently e.g. during scrolling, then it would be something to look into.

Switching to TMDB

Ok, so basically the database update failed solely because there were shows that could not be found in TMDB? I personally think that for the switch to TMDB to be realistic, it is imperative that (1) no data (especially "seen progress") is lost and (2) the user is properly informed if problems arise.

Best action plan imho upon switching:

  1. Try using TheTVDB's ID to find the show

  2. Otherwise, if possible, try using IMDB's ID to find the show

  3. If none are found, offer the user a search panel, pre-filled with the show's title and allow them to select the proper show

  4. If it can't be found manually, keep the current show data and move it to the archived shows list

    I haven't looked through the code for this particular setting, but I assume the switch to TMDB is currently a one-time process? This last step (4) could then cause problems later on. I think this could best be tackled by a per-show check on every update whether the show has already been switched to TMDB. That way, the switch can be tried again by the user from the archived shows list (those shows can be added to TMDB later on).

Backups

  1. considering that the app has given the user every opportunity to create a backup of the old DB, there's very little chance of losing any data by allowing the migration to proceed with the subset of series that it can identify

    I don't think that's the way most users would look at it: they would lose the ability to track the show from this app and are stuck with a file holding their data that only some people know what to do with. Even if they were smart enough to make a copy of the database file, as far as they're concerned, they in fact lost that data. That's why I proposed to at least archive the show using the known data, as described in (4) above.

  2. I think appending preUpdate is a good idea.

ltguillaume avatar Jun 10 '22 19:06 ltguillaume

I just ran a quick test.. I used the series that caused your migration to fail as a test case..

If its IMDb id was either unavailable or unhelpful, then I was tempted to be lazy..

If it got me somewhere, then I'd agree that it makes a lot of sense to do a 2nd find API request to try to lookup the TMDB id by this external id value.

In summary, you win.. the IMDb lookup was successful
  http://api.themoviedb.org/3/find/277964?api_key=c9eb196aaf70baf91d4ce4f0fea6a360&external_source=tvdb_id
    not found
    TheTVDB id: 277964

  https://thetvdb.com/api/8AC675886350B3C3/series/277964/all/en.xml
    <IMDB_ID>tt3487478</IMDB_ID>

  http://api.themoviedb.org/3/find/tt3487478?api_key=c9eb196aaf70baf91d4ce4f0fea6a360&external_source=imdb_id
    {"movie_results":[],"person_results":[],"tv_results":[{"original_name":"Smeris","first_air_date":"2014-03-17","overview":"","name":"Smeris","backdrop_path":"/vLA701CBif3SvTEp6CvNloajjEG.jpg","genre_ids":[80],"vote_count":7,"original_language":"nl","vote_average":7.7,"id":62637,"poster_path":"/2K2sLtMM5L8VBjSmZ5paQDO7Nkj.jpg","origin_country":["NL"],"popularity":8.229}],"tv_episode_results":[],"tv_season_results":[]}
    TMDB id: 62637

  https://api.themoviedb.org/3/tv/62637?api_key=1553d2e4fa2912fc0953305d4d3e7c44&language=en-US&append_to_response=external_ids,credits,content_ratings
  {
      "adult": false,
      "backdrop_path": "/vLA701CBif3SvTEp6CvNloajjEG.jpg",
      "created_by": [],
      "episode_run_time": [45],
      "first_air_date": "2014-03-17",
      "genres": [{
          "id": 80,
          "name": "Crime"
      }],
      "homepage": "",
      "id": 62637,
      "in_production": false,
      "languages": ["nl"],
      "last_air_date": "2020-02-02",
      "last_episode_to_air": {
          "air_date": "2020-02-02",
          "episode_number": 3,
          "id": 2046541,
          "name": "",
          "overview": "",
          "production_code": "",
          "runtime": 45,
          "season_number": 5,
          "still_path": "/alKhRaC4Aea6Gxy3de5TO6dMxd9.jpg",
          "vote_average": 0.0,
          "vote_count": 0
      },
      "name": "Smeris",
      "next_episode_to_air": null,
      "networks": [{
          "name": "BNN",
          "id": 311,
          "logo_path": "/2RG1bG0viFfNR5K6IzacRgccKXf.png",
          "origin_country": "NL"
      }, {
          "name": "BNNVARA",
          "id": 2161,
          "logo_path": "/tiBiiquRvGhxXteossLAWQkaFA1.png",
          "origin_country": "NL"
      }],
      "number_of_episodes": 43,
      "number_of_seasons": 5,
      "origin_country": ["NL"],
      "original_language": "nl",
      "original_name": "Smeris",
      "overview": "",
      "popularity": 8.229,
      "poster_path": "/2K2sLtMM5L8VBjSmZ5paQDO7Nkj.jpg",
      "production_companies": [{
          "id": 2198,
          "logo_path": null,
          "name": "Pupkin",
          "origin_country": "NL"
      }],
      "production_countries": [{
          "iso_3166_1": "NL",
          "name": "Netherlands"
      }],
      "seasons": [{
          "air_date": "2014-03-17",
          "episode_count": 10,
          "id": 66579,
          "name": "Season 1",
          "overview": "",
          "poster_path": "/mzMzBMeup0L7v2Ef1DHlLC46Sak.jpg",
          "season_number": 1
      }, {
          "air_date": "2015-03-29",
          "episode_count": 10,
          "id": 66583,
          "name": "Season 2",
          "overview": "",
          "poster_path": "/vQpZJtr8TVCXPz0oacY3LiqiWjM.jpg",
          "season_number": 2
      }, {
          "air_date": "2017-01-22",
          "episode_count": 10,
          "id": 97579,
          "name": "Season 3",
          "overview": "",
          "poster_path": "/eVu0JLqaieISKXR8u0ZUJuofelZ.jpg",
          "season_number": 3
      }, {
          "air_date": "2018-01-14",
          "episode_count": 10,
          "id": 97820,
          "name": "Season 4",
          "overview": "",
          "poster_path": "/uOMlwp6lEGTtr4PForKfGcAda8w.jpg",
          "season_number": 4
      }, {
          "air_date": "2020-01-19",
          "episode_count": 3,
          "id": 140705,
          "name": "Season 5",
          "overview": "",
          "poster_path": "/wPUPVvbW2tzT242NaxFFmWXmmx8.jpg",
          "season_number": 5
      }],
      "spoken_languages": [{
          "english_name": "Dutch",
          "iso_639_1": "nl",
          "name": "Nederlands"
      }],
      "status": "Ended",
      "tagline": "",
      "type": "Scripted",
      "vote_average": 7.7,
      "vote_count": 7,
      "external_ids": {
          "imdb_id": "tt3487478",
          "freebase_mid": null,
          "freebase_id": null,
          "tvdb_id": null,
          "tvrage_id": null,
          "facebook_id": null,
          "instagram_id": null,
          "twitter_id": null
      },
      "credits": {
          "cast": [{
              "adult": false,
              "gender": 2,
              "id": 1233325,
              "known_for_department": "Acting",
              "name": "Jeroen van Koningsbrugge",
              "original_name": "Jeroen van Koningsbrugge",
              "popularity": 2.377,
              "profile_path": "/atgQvtWOysAEIJHInCb7o0biZHx.jpg",
              "character": "Theo Kamp",
              "credit_id": "59b1736d925141078a08ce38",
              "order": 0
          }, {
              "adult": false,
              "gender": 0,
              "id": 929018,
              "known_for_department": "Acting",
              "name": "Dennis van de Ven",
              "original_name": "Dennis van de Ven",
              "popularity": 2.283,
              "profile_path": "/2eVYUTbgezw27RIdRC7P6pSgy9J.jpg",
              "character": "Willem Niessen",
              "credit_id": "5a515c0ac3a368751f012e8f",
              "order": 1
          }, {
              "adult": false,
              "gender": 1,
              "id": 1045527,
              "known_for_department": "Acting",
              "name": "Kiki van Deursen",
              "original_name": "Kiki van Deursen",
              "popularity": 2.143,
              "profile_path": "/pek00rKtJcvPYGiMQh2fCdbknmB.jpg",
              "character": "Maartje van Vught",
              "credit_id": "5a515c7dc3a368751c012723",
              "order": 2
          }, {
              "adult": false,
              "gender": 0,
              "id": 1102355,
              "known_for_department": "Acting",
              "name": "Daan van Dijsseldonk",
              "original_name": "Daan van Dijsseldonk",
              "popularity": 2.267,
              "profile_path": "/kLpAl5LIAVfnwbMWKHplBMO4n8E.jpg",
              "character": "Harold Bergkamp",
              "credit_id": "5a515ca192514113300125da",
              "order": 3
          }, {
              "adult": false,
              "gender": 0,
              "id": 34870,
              "known_for_department": "Acting",
              "name": "Susan Visser",
              "original_name": "Susan Visser",
              "popularity": 4.385,
              "profile_path": "/twOIgVIsieX8ttHoK8MoJ8UdBB4.jpg",
              "character": "Laura den Dooyer",
              "credit_id": "5a57c16c0e0a2669d4004228",
              "order": 5
          }, {
              "adult": false,
              "gender": 1,
              "id": 1102349,
              "known_for_department": "Acting",
              "name": "Juliette van Ardenne",
              "original_name": "Juliette van Ardenne",
              "popularity": 2.908,
              "profile_path": "/3NrqkQC4utT2PJW3zpRt83JiIBi.jpg",
              "character": "Lotte van Hees",
              "credit_id": "5a5a74039251413f6e0118dc",
              "order": 6
          }],
          "crew": [{
              "adult": false,
              "gender": 0,
              "id": 1737870,
              "known_for_department": "Crew",
              "name": "Johan Knibbe",
              "original_name": "Johan Knibbe",
              "popularity": 0.6,
              "profile_path": null,
              "credit_id": "5e24c88f397df000129dda02",
              "department": "Art",
              "job": "Production Design"
          }, {
              "adult": false,
              "gender": 2,
              "id": 1077763,
              "known_for_department": "Production",
              "name": "Job Castelijn",
              "original_name": "Job Castelijn",
              "popularity": 0.982,
              "profile_path": null,
              "credit_id": "5e24c8611dcb770012ca2f62",
              "department": "Production",
              "job": "Casting Director"
          }, {
              "adult": false,
              "gender": 0,
              "id": 1412882,
              "known_for_department": "Production",
              "name": "Steven Berendsen",
              "original_name": "Steven Berendsen",
              "popularity": 0.6,
              "profile_path": null,
              "credit_id": "5e24c9651bf2660012fbb46d",
              "department": "Production",
              "job": "Executive Producer"
          }, {
              "adult": false,
              "gender": 0,
              "id": 2995518,
              "known_for_department": "Sound",
              "name": "Peter Strijbos",
              "original_name": "Peter Strijbos",
              "popularity": 0.6,
              "profile_path": null,
              "credit_id": "613c64fc9653f60043e7567a",
              "department": "Sound",
              "job": "Sound Designer"
          }, {
              "adult": false,
              "gender": 0,
              "id": 31087,
              "known_for_department": "Sound",
              "name": "Hein Verhoeven",
              "original_name": "Hein Verhoeven",
              "popularity": 0.6,
              "profile_path": null,
              "credit_id": "613c6509d4cc8e008755c1f1",
              "department": "Sound",
              "job": "Sound Designer"
          }, {
              "adult": false,
              "gender": 0,
              "id": 2895238,
              "known_for_department": "Production",
              "name": "Felix van Gisbergen",
              "original_name": "Felix van Gisbergen",
              "popularity": 0.6,
              "profile_path": null,
              "credit_id": "613c654e6af9dd0027b610e0",
              "department": "Writing",
              "job": "Creative Producer"
          }, {
              "adult": false,
              "gender": 0,
              "id": 1996716,
              "known_for_department": "Crew",
              "name": "Mirande de Jong",
              "original_name": "Mirande de Jong",
              "popularity": 0.6,
              "profile_path": null,
              "credit_id": "613c6561e27260002a975aa4",
              "department": "Writing",
              "job": "Script Editor"
          }, {
              "adult": false,
              "gender": 0,
              "id": 1338200,
              "known_for_department": "Directing",
              "name": "Joost Reijmers",
              "original_name": "Joost Reijmers",
              "popularity": 0.6,
              "profile_path": null,
              "credit_id": "5a9d92140e0a2671f2001d54",
              "department": "Writing",
              "job": "Scenario Writer"
          }, {
              "adult": false,
              "gender": 2,
              "id": 23579,
              "known_for_department": "Editing",
              "name": "Herman P. Koerts",
              "original_name": "Herman P. Koerts",
              "popularity": 1.094,
              "profile_path": null,
              "credit_id": "5b6da14c0e0a267eeb15cdc5",
              "department": "Editing",
              "job": "Editor"
          }, {
              "adult": false,
              "gender": 0,
              "id": 2514996,
              "known_for_department": "Sound",
              "name": "David van der Heijden",
              "original_name": "David van der Heijden",
              "popularity": 0.6,
              "profile_path": null,
              "credit_id": "5e24c8e98f26bc001179e572",
              "department": "Sound",
              "job": "Original Music Composer"
          }, {
              "adult": false,
              "gender": 0,
              "id": 2713401,
              "known_for_department": "Crew",
              "name": "Tjitte Jan Nieuwkoop",
              "original_name": "Tjitte Jan Nieuwkoop",
              "popularity": 0.6,
              "profile_path": null,
              "credit_id": "613c64cf60c7510062d9023f",
              "department": "Camera",
              "job": "Director of Photography"
          }, {
              "adult": false,
              "gender": 0,
              "id": 1108178,
              "known_for_department": "Writing",
              "name": "Michael Leendertse",
              "original_name": "Michael Leendertse",
              "popularity": 0.6,
              "profile_path": null,
              "credit_id": "5a9d92060e0a2672050020fc",
              "department": "Writing",
              "job": "Scenario Writer"
          }, {
              "adult": false,
              "gender": 2,
              "id": 1233325,
              "known_for_department": "Acting",
              "name": "Jeroen van Koningsbrugge",
              "original_name": "Jeroen van Koningsbrugge",
              "popularity": 2.377,
              "profile_path": "/atgQvtWOysAEIJHInCb7o0biZHx.jpg",
              "credit_id": "5a9d9261925141101f001dcb",
              "department": "Writing",
              "job": "Creative Producer"
          }, {
              "adult": false,
              "gender": 0,
              "id": 1645803,
              "known_for_department": "Production",
              "name": "Iris Otten",
              "original_name": "Iris Otten",
              "popularity": 0.732,
              "profile_path": null,
              "credit_id": "5a9d91ef925141102b001ce7",
              "department": "Production",
              "job": "Producer"
          }, {
              "adult": false,
              "gender": 2,
              "id": 52169,
              "known_for_department": "Directing",
              "name": "Pieter Kuijpers",
              "original_name": "Pieter Kuijpers",
              "popularity": 1.103,
              "profile_path": "/gf4kBCMZXAFsQQRiKYHfPcrprjR.jpg",
              "credit_id": "5a9d91d40e0a2671f9001ef1",
              "department": "Production",
              "job": "Producer"
          }, {
              "adult": false,
              "gender": 0,
              "id": 1126741,
              "known_for_department": "Production",
              "name": "Sander van Meurs",
              "original_name": "Sander van Meurs",
              "popularity": 0.98,
              "profile_path": null,
              "credit_id": "5a9d91e50e0a2672050020c4",
              "department": "Production",
              "job": "Producer"
          }, {
              "adult": false,
              "gender": 0,
              "id": 929018,
              "known_for_department": "Acting",
              "name": "Dennis van de Ven",
              "original_name": "Dennis van de Ven",
              "popularity": 2.283,
              "profile_path": "/2eVYUTbgezw27RIdRC7P6pSgy9J.jpg",
              "credit_id": "5a9d92459251411021001ca2",
              "department": "Writing",
              "job": "Creative Producer"
          }]
      },
      "content_ratings": {
          "results": [{
              "iso_3166_1": "BR",
              "rating": "16"
          }, {
              "iso_3166_1": "NL",
              "rating": "12"
          }]
      }
  }

warren-bank avatar Jun 10 '22 19:06 warren-bank

Wait what, I won something?? πŸ˜‹ Haha, very cool 😁

ltguillaume avatar Jun 10 '22 19:06 ltguillaume

regarding your suggestions:

  1. to perform an interactive search for each missing series
  2. to store missing series as 'archived'

my thoughts are that:

  1. to be perfectly honest..
    that's way more development work than I'm willing to devote to migration..
    to handle a few edge cases
    • I will add a secondary find by IMDb id
    • but if that were to fail.. I'm prepared to call the missing series a lost cause
      • as with all open-source projects..
        if somebody doesn't like that decision,
        then they're perfectly free to do it differently
  2. that's simply not possible.. because
    • the TMDB id is a primary key in the DB for all tables
    • all queries depend on it

warren-bank avatar Jun 10 '22 20:06 warren-bank

  1. Understood. I think in that case a dialog with the failed shows is the bare minimum. Evidently migration should still be possible, even if a show fails.
  2. I'd say there should be a way around that, but as you rightfully note, edge cases, potentially a lot of work.

ltguillaume avatar Jun 10 '22 21:06 ltguillaume

Just checked your new commits, real nice work, great description in the commit message! πŸ˜ƒ

ltguillaume avatar Jun 11 '22 20:06 ltguillaume

My apologies @ltGuillaume...

I just remembered that it was changing the backup directory to the External Storage that wasn't working. I just changed it to an alternative folder (within the same directory) without any issues. πŸ‘

@warren-bank, I installed your latest version and copied the database file you provided.

It restored great and took abt a minute to restore and I experienced no problems.

Also, when I performed an update to the shows, it was really lightning fast (maybe because it was up to date already), but no issues experienced with that.

When I get a little more time, I'll be checking out the other features and report my feedback to you.

You've really gone above and beyond with your work and, I agree with @ltGuillaume, that your commits are very impressive. πŸ‘ πŸ‘

~Ibuprophen

Ibuprophen avatar Jun 11 '22 20:06 Ibuprophen

Thanks you two.. I appreciate the kind words.

I just really love the app.. and didn't want to see it become obsolete. Hopefully, we can all enjoy it for many years to come.

warren-bank avatar Jun 11 '22 21:06 warren-bank

  1. @Ibuprophen yes I completely agree. The source, when I started my fork, was messy at best. I started out wanting to make small changes, adding as little code as possible for every new feature, but in doing so, I only added to the spaghetti code state it was already in πŸ˜‹ Nevertheless, things worked fine for the end user and the app stayed small. It's also because I never looked into how to properly structure code (I'm more of a scripting guy and I bet @warren-bank has recognized that already). But rewriting and restructuring the code like this is definitely cool, allows for easier reading and it makes it more inviting for outside contributions.

  2. I do have a question about the new line in the list, Last unseen. Personally I think there's no reason to include it.

    • You already see how many unseen aired episides there are
    • If you use it to "calculate" when the next episode will probably be aired, then that makes no sense, because (1) there's an option to just show the next airing, and (2) there might be a hiatus.

So I guess I just don't understand why more text should be added (I've always found this a weak point of the shows list t.b.h., if I could have thought of another way to show the info, with less text, I would've done it, but more importantly, I fail to see the use of showing the info).

ltguillaume avatar Jun 11 '22 21:06 ltguillaume

re: "last unseen"..

  • one of my original requests was to add the sort option: "sort by date of last unseen episode (descending: newest first)"
    • now that there is one.. this field makes the result of using that sort order easier to understand
    • it shows the date of the most recently aired episode (that hasn't yet been seen),
      which is helpful when tracking several shows that you stay up-to-date with..
      and want to easily see when new episodes are available

warren-bank avatar Jun 11 '22 21:06 warren-bank

Personally, I love the app itself because it's simple and to the point. Much better than the alternatives that's full of images and such (not even referring to advertising as well).

Just curious @warren-bank, it there a reason why you had removed the Plex alternative option in menu?

Not a big deal with me, but just curious as I'm thinking that you couldn't utilize the new API with it.

I remember helping @ltGuillaume out with that Plex option a while ago too. πŸ‘

~Ibuprophen

Ibuprophen avatar Jun 11 '22 22:06 Ibuprophen

I still don't understand tbh, help me out here. You said it would somehow help seeing when new episodes are available. But as I said, there might be a hiatus for the show, or the next episode has been moved a day, or a week, so showing the latest unseen episode doesn't in any way indicate when a new episode is out. It's an indirect way to determine that at best, and probably totally wrong a lot of times. The option to show the next airing, however, does exactly that what you seem to actually want to know.

@Ibuprophen That makes perfect sense, does it not? That was an alternative TheTVDB mirror. @warren-bank's fork doesn't use TheTVDB anymore.

ltguillaume avatar Jun 11 '22 22:06 ltguillaume

@ltGuillaume let me illustrate by example..

  • if the season finale of a series airs tonight, then:
    • last unseen episode = today
    • next unaired episode = 6 months from today
  • if I want to be aware of this new episode, then:
    • which date is more helpful?

warren-bank avatar Jun 11 '22 23:06 warren-bank

off-hand, I believe this summarizes the various sort options..

DroidShows (upstream):

  • provides the sort options:
    • alphabetic
    • by count of unseen episodes
  • doesn't provide any sort options for date fields
    • available date fields:
      • oldest unseen
      • newest unseen
      • next to air

TV-Tracker:

  • adds sort options:
    • oldest unseen ascending
    • newest unseen descending

common to both:

  • all sort options subsort groups of series that are equal using the primary sort:
    • next to air ascending

warren-bank avatar Jun 11 '22 23:06 warren-bank

@ltGuillaume let me illustrate by example..

* if the season finale of a series airs tonight, then:
  
  * last unseen episode = today
  * next unaired episode = 6 months from today

* if I want to be aware of this new episode, then:
  
  * which date is more helpful?

Aha! Well, my anwer would be that neither is more helpful, because (1) next airing episode would actually show today's episode, not the next, and (2) Last unseen episode doesn't give you the information that it's the last episode of the season, either.

As such, to remedy (2) I have been wanting to modify the "middot appendix", for lack of a better word. Right now, your show will get a middot appended to episodes aired if all episodes with an air date have been aired. I actually wanted it to give you the info that all episodes of the last season you've watched episodes of have now been aired. This would solve the issue where you're in, say, season 6 and some overachiever on TVDB/TMDB already added next season's new episodes with an air date in about a year πŸ˜› In order to give you the info you want for today's episode, instead it could show the middot if the air date of the last episode of the last season you've watched episodes of is before or at the current date.

(I have edited this post 13 times, I really need to go to bed πŸ™ˆ)

ltguillaume avatar Jun 11 '22 23:06 ltguillaume

This was weird @warren-bank...

The following reflects the what your app looked like after I had successfully restored the database (as I had mentioned before)...

TV Tracker - Restored Database

... The following reflects what your app looked like after just updating it to the "TV-Tracker-008.00.08-09API-release" you just released.

TV Tracker - Post-Update (008 00 08-09API-release)

It did successfully restore again without any problems, but I didn't expect an update to wipe out the existing shows.

Just wanted to provide you with a heads up.

~Ibuprophen

Ibuprophen avatar Jun 12 '22 20:06 Ibuprophen