mib2-toolbox
mib2-toolbox copied to clipboard
jilleb
[REQUEST] Virtual Cockpit use CarPlay google map or apple map
I wonder if this idea can be implemented?
Virtual cockpit using CarPlay Output google maps or apple maps.
Or can install GOOLE EARTH in MIB2 high
Thanks for the suggestion.
Various people have been attempting to get carplay/AA into the virtual cockpit with little success, it's a real shame VAG don't support this. Efforts are also being made to prevent blocking built in maps on VC when AA nav is started.
The enabling of Google Earth in cars that don't have it though is generally a licencing issue which is outside the scope of this project unfortunately.
Until the version of Waze 4.66.0.1 there was a trick to make it work, which was to start a route (the virtual cockpit map was deactivated), stop the planned route (the map reappeared on the dashboard) and start a route again planned in Waze, without losing the car map on the dashboard. Version 4.66.0.1 can be downloaded from the web apkmirror.com
But since version 4.66.90.900 they modified Waze and said trick stopped working :(
There is an open thread on the Waze support forum about this:
https://waze.centercode.com/project/feedback/view.html?cap=c4c17e7b1c714fd4811d8431529d23aa&uf=e8ddabb7f2d94bef9eab137fa1b5a448&slsid=1080d884f753465c83cbce1789d9a405
This trick never worked with Google Maps, at least on my Leon. It also does not work with the new Tomtom AmiGO.
It would be great if with mib2-toolbox it could be possible to have both again, the map in the virtual cockpit, and an active navigation with android auto in the infotainment.
Yes there's a long running thread here already about that: https://github.com/jilleb/mib2-toolbox/issues/10
That trick was technically a bug in Waze, where it didn't properly flag to Android that navigation was started. I'm not surprised that fixed it.
I've found the flags in Java code in MIB already where this navigation_active status of tracked but they're in strange compiled Java code (jxe file) that's proving hard to override so far.
Thanks @andrewleech for the URL of thread # 10, which I hadn't read ...
The funny thing is that, at least in the new SEAT León Mk4, its MIB3 unit does allow to have active navigation in android auto, and simultaneously displaying the car's own navigator map on the dashboard ...
I don't know what is the reason given by the VAG group for not allowing it in the MIB2 units.
The Audi TT has CarPlay in the VC, so why stop at the maps? The display in the VC is horrible during all CarPlay functions. I'll be watching this one.
I talked to a seller on Alibaba.com, he said if I purchased VC through him, he could/would enable the CarPlay in VC and program the dual RS tach's that I don't have on my RSQ8 (2021's have dueling tachs, they are awesome).
It should be very nice!!! Have to exist some way how to enable Carplay in VC or in least original MAP in VC if Carplay is running
Those are like android based VC. You can even watch video inside your speedo. ....
On Tue, Jul 13, 2021 at 6:09 AM DAP56 @.***> wrote:
The Audi TT has CarPlay in the VC, so why stop at the maps? The display in the VC is horrible during all CarPlay functions. I'll be watching this one.
I talked to a seller on Alibaba.com, he said if I purchased VC through him, he could/would enable the CarPlay in VC and program the dual RS tach's that I don't have on my RSQ8 (2021's have dueling tachs, they are awesome).
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjilleb%2Fmib2-toolbox%2Fissues%2F159%23issuecomment-878630492&data=04%7C01%7C%7C6ce7b45dfe1f4647796308d94581b288%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637617245616944734%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=yHQxiK2tusXebVIToh3Q9dSaBfLSKBZFLy9gkAlsNTo%3D&reserved=0, or unsubscribe https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAPPZMUTRZN6Q23EJQFVQNVDTXNRY7ANCNFSM43CW7ULQ&data=04%7C01%7C%7C6ce7b45dfe1f4647796308d94581b288%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637617245616954729%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2FNIDYCubePVJXSrtTh9wn2eikRJk2ZXCoiTJNEvufrs%3D&reserved=0 .
Only original is interesting, i have with android based systems not good expirience :-(
All of the VC sellers on Aliexpress and Alibaba say that the VC's lock after 100km, so if its possible to enable CarPlay in VC (like TT and R8) then it has to be done on a new VC. I'm tempted to spend the $600 Euro and find out, but I realize that won't buy me happiness. That only comes from within.
Unfortunately I don't think it'll be so easy to just swap out a TT VC into a different car, as I believe the ability to show full maps / carplay / androidauto in them is based on a separate LVDS connection, eg it acts like a separate monitor, not just a MOST connection like other VC's. These cars don't have a screen in the dash, so they use the VC as the main / only screen.
Eg. TT VC (https://it.aliexpress.com/item/1005002826165506.html)
Note LVDS connector:
Compared to Passat VC (https://it.aliexpress.com/item/1005001369470372.html)
Thanks for the photos of the VC. In my RSQ8, there's night vision. I'm trying to bereft, but I think that means LVDS.
Thanks @andrewleech for the URL of thread # 10, which I hadn't read ...
The funny thing is that, at least in the new SEAT León Mk4, its MIB3 unit does allow to have active navigation in android auto, and simultaneously displaying the car's own navigator map on the dashboard ...
I don't know what is the reason given by the VAG group for not allowing it in the MIB2 units.
Interesting that your Skoda MIB3 functions that way, my Audi MIB3 does not.
Unfortunately I don't think it'll be so easy to just swap out a TT VC into a different car, as I believe the ability to show full maps / carplay / androidauto in them is based on a separate LVDS connection, eg it acts like a separate monitor, not just a MOST connection like other VC's. These cars don't have a screen in the dash, so they use the VC as the main / only screen.
Eg. TT VC (https://it.aliexpress.com/item/1005002826165506.html)
Compared to Passat VC (https://it.aliexpress.com/item/1005001369470372.html)
andrewleech consider, that all MLBevo VC (and even MQB for Audi A3 for example) have LVDS connector as well TT, because maps with 3D view are showed only by LVDS (generated in MIB), so maybe it would be even possible to clone Carplay/AA view and redirect to it. Also that can be the difference in the software, because map view stream is on different interface.
all MLBevo VC (and even MQB for Audi A3 for example) have LVDS connector as well TT, because maps with 3D view are showed only by LVDS (generated in MIB), so maybe it would be even possible to clone Carplay/AA view and redirect to it.
Sure, maybe. I don't have one of these cars so can't look into it myself.
What I have seen though is it's currently quite hard to access / clone the view as it's implemented as a h264 steam directly from the phone. This stream is hardware decoded by the Tegra chip direct to display. The command line tools to take screenshots work fine on normal screens, but only get a black frame when AA/CP is on screen - much like computers 10 years ago when movies needed to be hardware decoded direct to screen.
The raw h264 steam can be sent to a file of disk as well on screen, but I never found any other way to decode / frame grab from that on unit.
But somehow it is send from unit to VC to visualize the maps in 3D, and it is rendered in unit not VC, then transferred by LVDS/HSD. OFC it's just theoretical talk, newer dived in the unit code... As far I know in Audi TT it was done, but there VC was acting as a main screen for unit, because it was the only screen i the car.
Yeah, when it's LVDS, they acts just like a computer monitor and the hardware decoding sends it straight to that monitor.
If you mean the internal navigation engine running in 3d / Google Earth mode then that's generated internally and could be screenshot / copied easily enough.
It's the display of AA/CP that's a stream needing hardware (graphics) decoding on the MIB2 / 2.5 platform.
There's certainly some more hardware capabilities in newer platforms, but I haven't seen any of them personally: https://forums.macrumors.com/threads/review-2021-volkswagen-tiguan-adds-wireless-carplay-with-multi-screen-support-for-apple-maps-directions.2295243/
I know how LVDS looks like, but I just wonder if ti can be redirected/copied from display view to signal from unit to VC.
Yeah as I tried to describe earlier, I've tried to do exactly this - to screenshot / capture / clone / redirect the AA/CP video stream onto VC. I have been able to send PNG images to VC, in theory I could do this one after another to simulate video.
root@mmx:/eso/bin/apps> export LIBIMG_CFGFILE=/etc/imgprocessing.cfg
root@mmx:/eso/bin/apps> LD_LIBRARY_PATH=/eso/lib:/armle/lib:/root/lib-target:/armle/lib/dll /eso/bin/apps/loadandshowimage /fs/sdb0/triplej.png &
[1] 13144184
root@mmx:/eso/bin/apps> /eso/bin/apps/dmdt sc 4 -9
But I have not been able to get decoded video in any way. Possibly with a C program and intimate knowledge of the Tegra processor it might be possible to read frames back out of the graphics engine frame buffers, but I couldn't find any info about doing this.
This is actually very promising what about sending screenshot of what is visible on mib2 display to vc this way? To obtain screenshot I believe there are some functions already on mib2 which can be used
Yes that's what I was attempting, but unfortunately screenshots of AndroidAuto / Carplay come up blank / black as the tegra cpu hw acceleration is used to directly decode the h264 AA/CP streams to screen. I've been unable to find a way to grab / screenshot frames from this.
Yeah as I tried to describe earlier, I've tried to do exactly this - to screenshot / capture / clone / redirect the AA/CP video stream onto VC. I have been able to send PNG images to VC, in theory I could do this one after another to simulate video.
root@mmx:/eso/bin/apps> export LIBIMG_CFGFILE=/etc/imgprocessing.cfg root@mmx:/eso/bin/apps> LD_LIBRARY_PATH=/eso/lib:/armle/lib:/root/lib-target:/armle/lib/dll /eso/bin/apps/loadandshowimage /fs/sdb0/triplej.png & [1] 13144184 root@mmx:/eso/bin/apps> /eso/bin/apps/dmdt sc 4 -9
But I have not been able to get decoded video in any way. Possibly with a C program and intimate knowledge of the Tegra processor it might be possible to read frames back out of the graphics engine frame buffers, but I couldn't find any info about doing this.
@andrewleech , This is pretty awesome. Maybe we can make it in such a way that the Virtual Cockpit is called as a specific display manager context number. Android Auto gal.json has a config item which allows you to change the context number to something else, as well as selecint a different output device/channel. This can maybe help to get the live image working on the VC. I've played a lot with these contexts, and for instance I got it working in such a way that Android Auto would start when I would shift in reverse (instead of the main screen shifting to the rear view camera).
Maybe we can make it in such a way that the Virtual Cockpit is called as a specific display manager context number
By memory the VC is already a context, well the small map for it is available as a context that can be switched to on the main screen... but as far as I can tell the Tegra decoder is pointing straight to the display interface, whereas the VC isn't a standard display / framebuffer.
Hmm good point. I tried switching the AA context to VC and vice versa, but was never succesful there.
An other idea would maybe to have the VC handle the info from Android Auto, like the navigation directions, and media metainfo. The VC is able to handle it, I've seen people installing a Golf8/Leonmk4 MIB3 into the Golf7/Leonmk3, and suddenly got this data displayed on their virtual cockpits.
The AA settings have an option to dump the h264 steam to disk too, I had thought of trying to send that to a named pipe into a sw decoder, but not sure where to find a decoder I could realistically compile for arm/qnx
I did think of trying to render the nav/media meets 6 to send yeah, but couldn't find any such meta in the Java code handing AA at all :-(. The main AA interface is via a binary application, I don't know that it receives / shares this information at all?
I'll check. I would expect it's handled in a similar way as the "mobile navigation is active" data
There's a bunch of stuff related to all kinds of media and navigation metadata inside the gal bin:
[DSIAndroidAuto2Impl] %s : road='%s', turnSide=%s, event=%s, turnAngle=%i, turnNumber=%i, valid=%i
as well as here:
Which could maybe be used to get data from AA to the cluster (not the map stream or AA video stream, but just navi and media metadata instead of "Android Auto" on screen.
And maybe we could use the renderer area of the gal.json to force AA to be shown on the VC?
# DISPLAYABLE_EXTERNAL_SMARTPHONE = 59 (previously used DISPLAYABLE_EXTERNAL_DVD_VIDEO = 34)
"displayableID":59,
# preferred renderer (defaults to lowest available if preferred is not available)
# 0 - default
# 1 - nvss
# 2 - nvmedia
# 3 - qc
# 4 - cinemo
"type":0,
"nvss":{
# NvSSVideoOutputDevice_LVDS = 0 (default), NvSSVideoOutputDevice_HDMI = 1
"outputDevice":1,
# Setting provideTimestamps = false disables the NV decoder "late frame dropping" functionality
"provideTimestamps":true```
Interesting, those details all being in DSI functions should mean they're shared to other apps, my understanding is DSI is the system-wide messaging system. Perhaps there's a way to access them from some command line system, or even if the main Java so doesn't have handlers for then currently they could possibly be added!
I did have a play with those gal.json settings previously but saw no change at all... I don't know if I exhaustively tried everything but remember it not appearing to do much so gave up. Maybe searching through the gal bin in ghidra would shed some more light on these settings, perhaps find some other debug / export decoded frame settings...
These DSI functions are quite interesting.... I never found a way to interface with them, but maybe I didn't look too hard ;-) And then there's also the Exlap channel, which is capable of exchanging car data to and from the MIB. I was able to interface this externally (from a phone connected to AA), and it's possible to connect to this interface via the wlan connection with the MIB.
grep -R DSI /mnt/app/eso/bin/ does highlight a few apps worth closer investigation...