TVKILL
TVKILL copied to clipboard
42SK
JSON "database" for brands and patterns - TV-B-Gone support
Hi, here are some fixes and updates for a new version 1.4.0. Mainly:
- Ability to load patterns from a JSON file to bypass the limit of 64kB of methods (See #46)
- All TV-B-Gone NA/EU codes are supported (Thanks to glaukon-ariston for its conversion program)
- Add some custom codes for Europe
- Add codes for projectors (Epson for example), camcorders, HiFi devices (mainly via universal remotes) (Thanks to Furrtek and its modification of the TV-B-Gone firmware)
- Translation updates
JSON file is just a binding over Brand and Pattern classes. Example for a Brand:
{
"designation": "projectors EU",
"patterns": [
{
"comment": "sony SX263U T=26.315789us",
"frequency": 38000,
"pattern": [...]
},
"mute": {
"frequency": 38000,
"pattern": [...]
}
}
It should be ok for merging; then, once done, someone should update the Google Store version and contact the F-Droid packager for a release. I can try to ping the F-Droid part.
This is awesome! I'm currently a bit busy, but I'll try to review, test, and merge your changes within the next couple of weeks. I've already skimmed through some of you code and I really like what I've seen it so far; this truly brings the app to a new level. It's great to see someone put so much effort into TVKILL, and I'm very exited to try this stuff out. Huge thanks for your PR, I might get back to you with some questions and comments once I had the chance to check things out in some more detail. Once again, thanks a lot!
Thanks to you, take your time for the rest; I am very happy to see your response.
OK, I've managed to check out your changes. Let me give you a "brief" overview of my findings. I tried out the new TV-B-Gone patterns and they seemed to work fine on the TVs I had access to, which is quite nice. I also like the new JSON pattern "database"; it really does clean things up quite a bit.
I have merged the relevant changes from this PR into a new new_pattern_db branch where I have already made some minor adjustments.
For one thing, the Epson pattern you added form #40 was identical to the AOC pattern that was already in the app, so I removed it again to avoid redundant patterns.
Speaking of redundancy, you also added some additional custom patterns, many of which were featured in multiple "brands", so I also cleaned those up a bit by getting rid of the duplicates to speed up the overall transmission time.
Regarding the custom patterns you added, it was usually easy for me to guess what kind of device they belonged to, with one notable exception: I googled the device you named sony SX263U an found out that there is indeed a remote control called SX263U, which -- however -- is from JVC, not Sony. I therefore have two questions remaining regarding your custom patterns:
- What kind of device is the "
sony SX263U"? - Did you try out any of the patterns to see if they worked before adding them to the app?
Furthermore, I also made some minor improvements to better accommodate the new TV-B-Gone patterns (e.g. a better progress dialog when transmitting individual brands and a new option to only transmit brands when additional patterns are enabled), which you can check out in the commit history.
While I was reviewing and amending your changes, I also noticed a very peculiar bug which would sporadically occur and which I have not managed to properly reproduce thus far: Sometimes when I opened the app, the transmission time would increase significantly when transmitting patterns (probably by a factor of two or larger). I don't know what caused this and if this is going to be a persistent problem; however, I suggest that we try to spend some more time using and testing the app (preferably on multiple device) before we publish a new version to F-Droid / Google Play.
I should also point out that I omitted your last two commits (61eed6c and 4f8a9c0) when merging your PR into the new branch, since they where not directly relevant to the rest of your changes and I didn't want things to get to cluttered.
To summarize: We should be mostly good to go, the the major issues remaining are:
- [ ] Further clean up the new custom patterns (especially "mystery pattern"
sony SX263U). - [ ] [New] Confirm that Furrtek patterns actually work.
- [ ] Try to reproduce and fix the peculiar increased transmission time bug.
Finlay, two short comment on your second PR (#50): I briefly tried out the new feature for uploading a custom JSON file; however, I was not able to select a file on my device (you might have to walk me through the individual steps of adding a custom DB later). Also, I saw that added a couple of new strings to the app, including the respective translations into all of the app's 5 (!) supported languages. Since I doubt that many people are such distinguished polyglots as to be proficient in all of these languages, I would assume that you probably used some kind of online translation service for this (no offense :wink:). Adding translations into language that you don't really know is something that I would generally advise against. It certainly better to ignore the translations for the time being, so that our more linguistically experienced contributors can add them later on.
Either way, I would suggest that we first try to get this PR done, so that we can then later on add all there relevant features from PR 50 on top of the adjustments we might still have to make on the new branch. Overall, I'm very exited about these new features and I hope we can soon publish them in a new version.
First of all, thank you for the full review. Also, I agree with the todolist.
Epson pattern you added form #40 was identical to the AOC pattern that was already in the app
Sorry for that; indeed, many codes are reused by multiple brands and types of devices; especially in "Universal" remotes.
With a Generic list like the TV-B-Gone one, multiple devices and brands are shuffled. I think that "Generic" lists like this one should be used only from the "Brands" tab to avoid redundancy in the transmission.
In fact, TV-B-Gone and projectors lists are more lists of devices than lists of brands...
About additonnal patterns. I only mentioned the name of the author but I should have mentioned the site on which I took the codes.
Furrtek is a French hacker who made his own version of the TV-B-Gone, with a custom firmware few years ago. He has a small community that bought the hardware and used the firmware at the time. I have good reasons to believe that the codes were added in the latest versions for a justified purpose.
It is from this page: http://furrtek.free.fr/?a=tvbgkit
I used the following version because it's the only one with a source code:
EU Firmware v0.7 pour le kit TV-B-Gone PCB V1.2
Pas de mode chiant.
124 codes en mode euro (la liste US n'a pas changé depuis la version 0.6, ne pas l'utiliser).
Tous les projecteurs Epson Tri-LCD, certains projecteurs NEC et Sanyo, certaines chaînes HiFi Panasonic et JVC, certains caméscopes Sony.
Téléviseurs Viera, NEC Multisync, Macbooks première generation et Furby (sommeil).
I used the converter of glaukon-ariston to get the codes; and from the name of the variables I tried to recover the concerned device...
That being said, like you, I had a hard time identifying the brands.
For SX263U: after a new search it seems indeed related to JVC.
The following list is ok with that: http://ftp.uhulinux.hu/uhubuild/LOGPACK-full/dev/lirc.log
But I also see other brands on this link:
https://webcache.googleusercontent.com/search?q=cache:9SvpjlI5sNMJ:https://adroe.top/tech/%3Fheader%3DAmplificator%2BSta%25C8%259Bie%2BTEAC%2BA-R610,%2Bnu%2Bsony,%2Bakai,%2Bpioneer,%2Bjvc,%2Bonkyo+&cd=15&hl=fr&ct=clnk&gl=fr&client=firefox-b-d
Telecomandă JVC RM-SX263U ( sony, marantz, pioneer, onkyo)
=> maybe a universal JVC remote (?).
Sometimes when I opened the app, the transmission time would increase significantly when transmitting patterns (probably by a factor of two or larger)
Wow, i never noticed this yet despite many tests. I will try again, maybe knowing what I am looking for I would notice the problem.
you might have to walk me through the individual steps of adding a custom DB later
Yes, you have to download a JSON file (like the one already embedded in the application) somewhere in your phone.
Use cases:
"Use a custom DB" unchecked:
- First use: Check the "Use a custom DB" option, the button below "Open file..." is then enabled, allowing you to open a file via a "browser". Once the file is selected, the app restarts and loads it.
- External file already configured in the "Open file..." field: Check the "Use a custom DB" option; the app restarts and loads the file. Obviously, if the file is no longer present or if a problem appears when loading it, then it is removed from the field and the default DB is used.
"Use a custom DB" checked:
- Uncheck the "Use a custom DB" option, the app restarts and loads the default DB.
I could have implemented it to not restart the application automatically if a file is already known, or not restart as soon as a new file is selected. However, during use I found that inconvenient. An alternative would be to update the singleton, then reload the GUI as soon as the option is changed. However, the singleton methods loading the JSON are declared lazy (it's justified) and I don't know how to force their full execution again.
Also, I saw that added a couple of new strings to the app, including the respective translations into all of the app's 5 (!) supported languages.
The meaning/phrasing of some strings are indeed modified; I prefer to update all the languages via a cross validation between GoogleTrad and DeepL than fully delete them for a few changed words.
I also prefer this solution rather than leaving the old translations which are no longer valid. It's better to have an obvious mistake (than a valid string grammatically and orthographically speaking) but which is more in agreement with the described functionality...
For the modified sentences, you can be 100% certain for French and English. Spanish is theoretically valid too. For German and languages of eastern countries it is a cross validation between the translators, helped by the original translated strings.
Here is the list of commits:
61eed6c 52e9544 0d333b6 9d92b5c
For the new sentences without previous translation, I actually used the 2 translators at 100% and I admit that it is a bit "violent" :p. This is the case for e0f1869. For this commit, again, French, English and Spanish are ok.
If you want, for this commit and/or for the others I can purge old translations while waiting for someone to redo them.
I told myself I was going to be short in my answers because I know how time consuming it is to manage projects in parallel. But I see I failed...
So just a note but for a big thing.
I spent the previous week looking for additional IR codes and came across the Xiaomi Mi Remote app. This app works very well and allows a user to find the right codes for their hardware (multiple devices are supported, from simple fans, to air conditioners, to audio/video equipment).
After many hours of reverse engineering on the application and its API, I managed to quickly set up a project that works to retrieve and exploit the codes.
This represents several thousands of distinct extinction codes (2984) (the uniqueness is for the moment based on the tuple (frequency, code)). The database is probably more consequent than that of IRdb.tk.
I demonstrated the possibility to export the codes in any format using the "new" TVKILL format as an example.
Here is its GitHub link and the write-up done (in french): https://github.com/ysard/mi_remote_database/ https://pro-domo.ddns.net/blog/retro-ingenierie-dune-application-android-xiaomi-mi-remote-partie-1.html
TVKILL compatible codes are in the assets (they are usable with the PR #50): https://github.com/ysard/mi_remote_database/releases
Thanks for your reply, that did clear things up a lot.
For SX263U: after a new search it seems indeed related to JVC.
OK, I guess that solves that; I'll simply move the pattern to the JVC codes. I'm still a bit uncomfortable about the new Furrtek patterns; we obviously don't know which devices exactly these patterns belong to, let alone if they actually work (judging form my experience, since you had to convert the patterns before adding them to the app, it is also not at all unlikely that the patterns might simply be broken and therefore useless). I would feel much easier about this if we knew that at least some of them can turn off a certain device. Sadly, I currently don't have access to any of the devices in question; if you therefore happen to find a device that the new patterns can turn off, please let me know!
Regrading there weird transmission delay, I've done a bit more testing and made an interesting observation:
I build the app from the latest version of the new_pattern_db branch (commit 2018eea) and timed how long the transmission of all patterns would take when using the universal mode with the "additional patterns" option enabled.
The first time I did this, it took about 118s --- the second time, however, it took only 106s.
There is obviously a discrepancy between the two insances; however, I do not know how much the transmission time can actually fluctuate on different occasions.
I also compared the time it took to transmit only the patterns from the older versions with the new (json) and the old (kotlin) DB: in the former case, the overall transmission time was mostly close to 24s --- in the latter, it was always around 20s.
There is definitely a minor performance regression here. Alas, I don't know why exactly this is happening and if/how we can fix it.
I will probably try to test this on another device soon and see how the result are there.
Again, if you make any observations with regard to this, let me know!
A brief comment on the custom json file feature:
Check the "Use a custom DB" option, the button below "Open file..." is then enabled, allowing you to open a file via a "browser".
I can select the "Open file…" option. However, in the file browser, my json file is grayed out and I can't select it; any idea why this might be happening?
For the modified sentences, you can be 100% certain for French and English.
OK, that's good to know. I'll make sure to also copy the french translation when I (eventually) merge this.
If you want, for this commit and/or for the others I can purge old translations while waiting for someone to redo them.
No need to do that; I'll probably simply omit the other translations when I do the merge, that'll make it easier for everyone.
After many hours of reverse engineering on the application and its API, I managed to quickly set up a project that works to retrieve and exploit the codes.
That does sound very cool! Again, I'd love to have a feature that lets people use their own patterns in TV KILL. I guess this gives us even more reason to make sure that this feature gets released soon!
Hi, this PR is not forgotten, but time is missing :/
I would feel much easier about this if we knew that at least some of them can turn off a certain device. Sadly, I currently don't have access to any of the devices in question; if you therefore happen to find a device that the new patterns can turn off, please let me know!
After testing in supermarkets where TV equipment is sailed, yes these codes work but as you can guess, it is difficult to know which ones worked. Toshiba, LG, Philips, Samsung are a priori good. Yours too ...
I think for now we should remove for the TVBgone list from the default app and let users load it if they want as part of PR #50.
I will try to valdiate the patterns on my own BDD coming from Xiaomi and from IrDB.
It should also be taken into account that the codes are sometimes localized. Some codes may not appear to work in certain regions of the world. Hence the TVBgone EU list.
There is definitely a minor performance regression here. Alas, I don't know why exactly this is happening and if/how we can fix it.
To have tested on my version with 297 codes, the time is quasi non-fluctuating (58-59 seconds). On the other hand I note a significant CPU consumption in kernel time (> 80%) on a core of the phone + ~ 25% of CPU on 3 other cores, of which 2/3 of this cost for each of them for the user time (including htop cost; I'll spare you the htop screenshot).
Memory allocation or I/O due to the IR API? I do not know. The CPU is an ARM cortex-A53, (6 cores?), 1.4GHz / 1.1GHz;
I also tested on a Redmi Note 7 with similar timing results. The CPU usage has not been measured but the device is more powerful (8 cores / 2.2GHz).
PS: Each pattern is converted during the first send and not during the following. My measurements show that this conversion does not cost more than 200ms on average.
I think the fact that you measured more than 100 seconds demonstrates a CPU limitation.
Even if I expected it, I'm still quite surprised at such consumption due to both Java code and Android API calls. However, I do not see any particularly consuming area in the code of this app which remains very "simple" :o Apart from perhaps the progression indicators during the sending of the codes?
I can select the "Open file…" option. However, in the file browser, my json file is grayed out and I can't select it; any idea why this might be happening?
The filter on the mimetype "application/json" is in fact not supposed to work on Android which does not have this entry in its database. Yet it works on my phone, hence the error. I think it is very dependent on the file picker installed by a possible operator/constructor code.
To my knowledge, there is no way to detect this because the method allowing to test the support of this mimetype does not return anything positive for me:
MimeTypeMap.getSingleton().HasMimeType("application/json") // false
It is therefore necessary to let the user choose any file and then perform the usual input verification operations to be sure to have a valid file ...
See https://github.com/ysard/TVKILL/blob/a5f6b6da16623978533943e73a1da6f9178da57f/app/src/main/java/com/redirectapps/tvkill/Preferences.java#L85-L87
Still on the filepicker I got lost in implementation details, which I feel, vary greatly from platform to platform.
See https://github.com/ysard/TVKILL/blob/a5f6b6da16623978533943e73a1da6f9178da57f/app/src/main/java/com/redirectapps/tvkill/Preferences.java#L82
An Intent initialized with ACTION_OPEN_DOCUMENT does not authorize me to exit from the downloads directory ... Bonus: if I have the misfortune to rename a json file downloaded in this folder, it then disappears from the filepicker in TVKILL; until I rename it with the original name. Surrealist.
Its ACTION_GET_CONTENT alternative allows you to exit the downloads folder via the file browser installed on the system but the opening rights are refused regardless of the location of the files ...
I tried so hard to correct these problems that I came to question the Xiaomi overlay ...
Stackoverflow is full of conditional tips to adapt to versions of Android but also to manufacturers. I don't know if there is a place where these tips are centralized ...
However, despite these remarks and limitations the problem is fixed now.
Also, I am having a problem highlighted with your code regarding the sendbydefault.
The "additional codes" parameter is not always taken into account instantly. When starting the app, select this parameter, return to the universal remote control tab and send codes: Depending to the initial state, the patterns are neither added (number remains at 20), nor deleted (number remains at 287). It goes both ways almost unpredictably. Even if you go back in preferences and change the setting.
Restarting the application (exiting the app from the list of active apps) seems to apply the new setting.
Sometimes it works the first time.
Tested on 2 phones.
Hi, one additional remark: Some patterns have a leading couple of 1 values.
For having worked on the subject in my reversing of the Xiaomi API and on the Pronto standard, I do not see any justification for this. This is not a preamble to wake up / calibrate the receiver, usually clearly longer. I don't know why these values were added and in think they can be removed...
Hello again,
its great to hear that you're still working on this PR!
I've just reviewed the changes you've made regarding the json file import. Overall, I'm quite happy with it; the import now also works on my device and the code looks good too. I think the solution you came up with solves the problem quite well.
I think for now we should remove for the TVBgone list from the default app and let users load it if they want as part of PR #50.
Personally, I don't mind leaving the TV-B-Gone codes in by default. I guess this is a feature that a lot of users probably want anyway so I guess we might as well spare them the hassle of having to load the patterns manually.
However, I might considering leaving out the Furrtek codes by default as I am still uncertain to what extent they actually work.
After testing in supermarkets where TV equipment is sailed, yes these codes work but as you can guess, it is difficult to know which ones worked. Toshiba, LG, Philips, Samsung are a priori good. Yours too ...
Just to make sure I understand you correctly here: did you find any device that could be turned off with one of the Furrtek patterns? (And can you can you be sure that it was in fact turned of by a Furrtek pattern and not accidentally by one of the other patterns in the app?)
To have tested on my version with 297 codes, the time is quasi non-fluctuating (58-59 seconds). On the other hand I note a significant CPU consumption in kernel time (> 80%) on a core of the phone + ~ 25% of CPU on 3 other cores, of which 2/3 of this cost for each of them for the user time (including htop cost; I'll spare you the htop screenshot). [...] I also tested on a Redmi Note 7 with similar timing results. The CPU usage has not been measured but the device is more powerful (8 cores / 2.2GHz).
OK, that does somewhat alleviate my concerns w.r.t. the performance regression. On my device, the transmission time seems to be fast enough for every day use in most cases. Hence, I'd probably be willing to accept a small increase in transmission time as long as it is not a major problem for most users.
The "additional codes" parameter is not always taken into account instantly. When starting the app, select this parameter, return to the universal remote control tab and send codes: Depending to the initial state, the patterns are neither added (number remains at 20), nor deleted (number remains at 287). It goes both ways almost unpredictably. Even if you go back in preferences and change the setting.
Restarting the application (exiting the app from the list of active apps) seems to apply the new setting.
Sometimes it works the first time.
Tested on 2 phones.
I do remember encountering this problem a while a go. I guess this bug must have been in the app for some time already, although nobody noticed it so far since we did not have that many additional patterns up until now. I'll open a separated issue for this, since it is not directly related to your PR.
So far, things are looking quite good. I'll probably merge the PR soon, the only major question remaining for me is whether or not we should include the Furrtek patterns, Again, if you did manage to find a device that works with the Furrtek patterns, please tell me, otherwise I'll probably just leave them out for now and we can still add them at some later point in time when we are sure that they actually work.
So far, things are looking quite good. I'll probably merge the PR soon, the only major question remaining for me is whether or not we should include the Furrtek patterns
I think you should leave it in, because why not
@nm17
So far, things are looking quite good. I'll probably merge the PR soon, the only major question remaining for me is whether or not we should include the Furrtek patterns
I think you should leave it in, because why not
Because, if it turns out that they don't work, we'll end up having a bunch of useless patterns in the app that increase the overall transmission time without actually doing anything. This would be a nuisance to all users which I very much intend to avoid.
Thank you for the review ;)
I might considering leaving out the Furrtek codes by default as I am still uncertain to what extent they actually work.
I will try to clearly identify these codes in the existing databases.
did you find any device that could be turned off with one of the Furrtek patterns?
Yes devices are turned off by these lists but I have little to identify precisely which for which pattern. Identification is tricky in this context :p I will come back to it with a more scientific approach.
And yes, the risk is to add some nuisance by sending useless patterns. BTW the sendbydefault option is quite useful in this case.