pyatv
pyatv copied to clipboard
postlund
Investigating Companion api Keyboard implementation
What to investigate?
Find out how to get keyboard updates and send keyboard text through the companion protocol.
So I've started debugging some stuff in the console with my iPhone connected, current progress is that I am able to subscribe to the right events and get actual changes by adding the following in the connect method:
await self.subscribe_event("_tiC") # fired when there's text input changes through on screen keyboard appletv
await self.subscribe_event("_tiStarted") # fired when keyboard is available
await self.subscribe_event("_tiStopped") # fired when keyboard is no longer available
await self._send_command("_tiStart", {}) # starts text input session
Now I'm trying to figure out the command to send actual text input. I've found that keyboard presses only send back a _tiC event in the console and I can't seem to figure out yet what the command should be to actually send text.
Some current events:
_tiStarted
{'_i': '_tiStarted', '_x': 568430212, '_c': {'_tiV': 1, '_tiD': b'bplist00\xd4\x01\x02\x03\x04\x05\x06\x07\x0eX$versionY$archiverT$topX$objects\x12\x00\x01\x86\xa0_\x10\x0fNSKeyedArchiver\xd3\x08\t\n\x0b\x0c\r^documentTraits[sessionUUID]documentState\x80\x06\x80\x0c\x80\x01\xad\x0f\x10\x15\x19\x1a!%/0159=U$null\xd2\x11\x12\x13\x14UdocStV$class\x80\x02\x80\x05\xd2\x16\x12\x17\x18_\x10\x12contextBeforeInput\x80\x03\x80\x04ZTesteraaaa\xd2\x1b\x1c\x1d\x1eZ$classnameX$classes_\x10\x0fTIDocumentState\xa2\x1f _\x10\x0fTIDocumentStateXNSObject\xd2\x1b\x1c"#_\x10\x10RTIDocumentState\xa2$ _\x10\x10RTIDocumentState\xd5&\'()\x12*++-.SbIdSappVlocAppXtiTraits\x80\x07\x80\x08\x80\x08\x80\t\x80\x0b_\x10\x12com.apple.TVSearchVSearch\xd22\x1234Uflags\x12T\x10\x1a\x86\x80\n\xd2\x1b\x1c67_\x10\x11TITextInputTraits\xa28 _\x10\x11TITextInputTraits\xd2\x1b\x1c:;_\x10\x11RTIDocumentTraits\xa2< _\x10\x11RTIDocumentTraitsO\x10\x106\xa7\xfa+\x03\xadL\x1d\xbd\x04\xab\xec\x96:6\x9d\x00\x08\x00\x11\x00\x1a\x00$\x00)\x002\x007\x00I\x00P\x00_\x00k\x00y\x00{\x00}\x00\x7f\x00\x8d\x00\x93\x00\x98\x00\x9e\x00\xa5\x00\xa7\x00\xa9\x00\xae\x00\xc3\x00\xc5\x00\xc7\x00\xd2\x00\xd7\x00\xe2\x00\xeb\x00\xfd\x01\x00\x01\x12\x01\x1b\x01 \x013\x016\x01I\x01T\x01X\x01\\\x01c\x01l\x01n\x01p\x01r\x01t\x01v\x01\x8b\x01\x92\x01\x97\x01\x9d\x01\xa2\x01\xa4\x01\xa9\x01\xbd\x01\xc0\x01\xd4\x01\xd9\x01\xed\x01\xf0\x02\x04\x00\x00\x00\x00\x00\x00\x02\x01\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x17'}, '_t': 1}
2022-04-29 11:46:38 DEBUG [pyatv.protocols.companion.api]: Got event _tiStarted from device: {'_tiV': 1, '_tiD': b'bplist00\xd4\x01\x02\x03\x04\x05\x06\x07\x0eX$versionY$archiverT$topX$objects\x12\x00\x01\x86\xa0_\x10\x0fNSKeyedArchiver\xd3\x08\t\n\x0b\x0c\r^documentTraits[sessionUUID]documentState\x80\x06\x80\x0c\x80\x01\xad\x0f\x10\x15\x19\x1a!%/0159=U$null\xd2\x11\x12\x13\x14UdocStV$class\x80\x02\x80\x05\xd2\x16\x12\x17\x18_\x10\x12contextBeforeInput\x80\x03\x80\x04ZTesteraaaa\xd2\x1b\x1c\x1d\x1eZ$classnameX$classes_\x10\x0fTIDocumentState\xa2\x1f _\x10\x0fTIDocumentStateXNSObject\xd2\x1b\x1c"#_\x10\x10RTIDocumentState\xa2$ _\x10\x10RTIDocumentState\xd5&\'()\x12*++-.SbIdSappVlocAppXtiTraits\x80\x07\x80\x08\x80\x08\x80\t\x80\x0b_\x10\x12com.apple.TVSearchVSearch\xd22\x1234Uflags\x12T\x10\x1a\x86\x80\n\xd2\x1b\x1c67_\x10\x11TITextInputTraits\xa28 _\x10\x11TITextInputTraits\xd2\x1b\x1c:;_\x10\x11RTIDocumentTraits\xa2< _\x10\x11RTIDocumentTraitsO\x10\x106\xa7\xfa+\x03\xadL\x1d\xbd\x04\xab\xec\x96:6\x9d\x00\x08\x00\x11\x00\x1a\x00$\x00)\x002\x007\x00I\x00P\x00_\x00k\x00y\x00{\x00}\x00\x7f\x00\x8d\x00\x93\x00\x98\x00\x9e\x00\xa5\x00\xa7\x00\xa9\x00\xae\x00\xc3\x00\xc5\x00\xc7\x00\xd2\x00\xd7\x00\xe2\x00\xeb\x00\xfd\x01\x00\x01\x12\x01\x1b\x01 \x013\x016\x01I\x01T\x01X\x01\\\x01c\x01l\x01n\x01p\x01r\x01t\x01v\x01\x8b\x01\x92\x01\x97\x01\x9d\x01\xa2\x01\xa4\x01\xa9\x01\xbd\x01\xc0\x01\xd4\x01\xd9\x01\xed\x01\xf0\x02\x04\x00\x00\x00\x00\x00\x00\x02\x01\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x17'}0
_tiStopped
{'_i': '_tiStopped', '_x': 568430233, '_c': {}, '_t': 1}
_tiC
{'_i': '_tiC', '_x': 568430219, '_c': {'_tiV': 1, '_tiD': b'bplist00\xd4\x01\x02\x03\x04\x05\x06\x07\x0eX$versionY$archiverT$topX$objects\x12\x00\x01\x86\xa0_\x10\x0fNSKeyedArchiver\xd3\x08\t\n\x0b\x0c\r^documentTraits[sessionUUID]documentState\x80\x06\x80\x0c\x80\x01\xad\x0f\x10\x15\x19\x1a!%/0159=U$null\xd2\x11\x12\x13\x14UdocStV$class\x80\x02\x80\x05\xd2\x16\x12\x17\x18_\x10\x12contextBeforeInput\x80\x03\x80\x04Qa\xd2\x1b\x1c\x1d\x1eZ$classnameX$classes_\x10\x0fTIDocumentState\xa2\x1f _\x10\x0fTIDocumentStateXNSObject\xd2\x1b\x1c"#_\x10\x10RTIDocumentState\xa2$ _\x10\x10RTIDocumentState\xd5&\'()\x12*++-.SbIdSappVlocAppXtiTraits\x80\x07\x80\x08\x80\x08\x80\t\x80\x0b_\x10\x12com.apple.TVSearchVSearch\xd22\x1234Uflags\x12T\x10\x1a\x86\x80\n\xd2\x1b\x1c67_\x10\x11TITextInputTraits\xa28 _\x10\x11TITextInputTraits\xd2\x1b\x1c:;_\x10\x11RTIDocumentTraits\xa2< _\x10\x11RTIDocumentTraitsO\x10\x106\xa7\xfa+\x03\xadL\x1d\xbd\x04\xab\xec\x96:6\x9d\x00\x08\x00\x11\x00\x1a\x00$\x00)\x002\x007\x00I\x00P\x00_\x00k\x00y\x00{\x00}\x00\x7f\x00\x8d\x00\x93\x00\x98\x00\x9e\x00\xa5\x00\xa7\x00\xa9\x00\xae\x00\xc3\x00\xc5\x00\xc7\x00\xc9\x00\xce\x00\xd9\x00\xe2\x00\xf4\x00\xf7\x01\t\x01\x12\x01\x17\x01*\x01-\x01@\x01K\x01O\x01S\x01Z\x01c\x01e\x01g\x01i\x01k\x01m\x01\x82\x01\x89\x01\x8e\x01\x94\x01\x99\x01\x9b\x01\xa0\x01\xb4\x01\xb7\x01\xcb\x01\xd0\x01\xe4\x01\xe7\x01\xfb\x00\x00\x00\x00\x00\x00\x02\x01\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x0e'}, '_t': 1}
2022-04-29 11:47:10 DEBUG [pyatv.protocols.companion.api]: Got event _tiC from device: {'_tiV': 1, '_tiD': b'bplist00\xd4\x01\x02\x03\x04\x05\x06\x07\x0eX$versionY$archiverT$topX$objects\x12\x00\x01\x86\xa0_\x10\x0fNSKeyedArchiver\xd3\x08\t\n\x0b\x0c\r^documentTraits[sessionUUID]documentState\x80\x06\x80\x0c\x80\x01\xad\x0f\x10\x15\x19\x1a!%/0159=U$null\xd2\x11\x12\x13\x14UdocStV$class\x80\x02\x80\x05\xd2\x16\x12\x17\x18_\x10\x12contextBeforeInput\x80\x03\x80\x04Qa\xd2\x1b\x1c\x1d\x1eZ$classnameX$classes_\x10\x0fTIDocumentState\xa2\x1f _\x10\x0fTIDocumentStateXNSObject\xd2\x1b\x1c"#_\x10\x10RTIDocumentState\xa2$ _\x10\x10RTIDocumentState\xd5&\'()\x12*++-.SbIdSappVlocAppXtiTraits\x80\x07\x80\x08\x80\x08\x80\t\x80\x0b_\x10\x12com.apple.TVSearchVSearch\xd22\x1234Uflags\x12T\x10\x1a\x86\x80\n\xd2\x1b\x1c67_\x10\x11TITextInputTraits\xa28 _\x10\x11TITextInputTraits\xd2\x1b\x1c:;_\x10\x11RTIDocumentTraits\xa2< _\x10\x11RTIDocumentTraitsO\x10\x106\xa7\xfa+\x03\xadL\x1d\xbd\x04\xab\xec\x96:6\x9d\x00\x08\x00\x11\x00\x1a\x00$\x00)\x002\x007\x00I\x00P\x00_\x00k\x00y\x00{\x00}\x00\x7f\x00\x8d\x00\x93\x00\x98\x00\x9e\x00\xa5\x00\xa7\x00\xa9\x00\xae\x00\xc3\x00\xc5\x00\xc7\x00\xc9\x00\xce\x00\xd9\x00\xe2\x00\xf4\x00\xf7\x01\t\x01\x12\x01\x17\x01*\x01-\x01@\x01K\x01O\x01S\x01Z\x01c\x01e\x01g\x01i\x01k\x01m\x01\x82\x01\x89\x01\x8e\x01\x94\x01\x99\x01\x9b\x01\xa0\x01\xb4\x01\xb7\x01\xcb\x01\xd0\x01\xe4\x01\xe7\x01\xfb\x00\x00\x00\x00\x00\x00\x02\x01\x00\x00\x00\x00\x00\x00\x00>\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x0e'}
Expected outcome
Have a working keyboard event and a keyboard input method on the companion api.
So I've been looking into more logs and it looks like the input commands are passed through XPC? I've looked around and saw it pop up 2 or 3 times around issues here but nothing really specific about implementations. Hoping to find more digging into this.
XPC: Client requests sending data payload <private> to identifier <private>
So I've been able to decode the _tiC event using the nska_deserialize library. I didn't seem to get it working with other libraries.
loaded = nska_deserialize.deserialize_plist_from_string(msg['_tiD'])
input_string = ''
if len(loaded) >= 3:
ds = loaded[2]['documentState']
if ds.__contains__('docSt'):
ds = ds['docSt']
if ds.__contains__('contextBeforeInput'):
input_string = ds['contextBeforeInput']
I have also been trying to inser text using some HID conversion which seemed to work partially, I can't seem to send the "a" key though, which is a serious problem. So back to the drawing board, trying to find out how the remote on the iphone is sending it's data.
I have experimented sending my own _tiC event but that didn't seem to do anything.
I've seen some _hidT events pass by with 5 arguments so that would be my next best bet, no idea as of yet what the arguments should be and how to trigger it.
Sorry about my absence regarding your work, it's really great that you are looking in to this! I know a lot of people would like to see keyboard support (it would also be possible to implement in MRP and DMAP for older devices). Unfortunately I don't have any free time to help out with this right now, I will if I find a slot. But great work so far, appreciate it! 👍 Feel free to make updates to the documentation and send a PR!
@SightAndSound-Roger Hey, are you still working on this? I would be interested to know how far you got? It sounded like you were getting close to solving this?
Keyboard support is the most important feature for me and I would love to offer my help anywhere possible (although I am not a python programmer in nature)
@Fredde87 unfortunately not, I've been searching for the way to send strings but to no avail. I've tried polyfilling with separate hid commanda but that didn't work for special chars and for some reason the a char... I've given up for now, after investigating the console on iOS and tvOS side I couldn't find a definitive solution. Still hoping for someone fresh to take a look!
Keyboard support is possible via MRP as well, probably easier to implement as I have already created the protocol messages involved (base is here). So it's technically possible to go that route as well.
I have tried the MRP (over Airplay) route but that doesn't seem to work with the keyboard. Perhaps I've been doing something wrong?
I haven't dug into keyboard details, but looking at the log when accessing the keyboard on the Apple TV (possibly also making some input) should shed some light on how that work. Maybe TextInputMessage is used to set text? Trying everything is probably the naive solution. The proper one would be if I finished up the support in atvproxy some day so that we can look at the traffic again like we could when MRP was a stand-alone protocol.
Having the proxy would be a bliss, I've took a shot at fixing that too but failed miserably. I've implemented the keyboard before with C# and the regular MRP protocol, the same messages did not go over Airplay :/
So I've looked it up, it required the GetKeyboardSessionMessage and then indeed the TextInputMessage, however during debugging I think I got some error about the amount of parameters in the message.
These are the logs that happen around text input (using the control center remote):
iOS
tvOS
I'ts not much but perhaps it rings a bell for someone else.
I'm just assuming that everything that used to work when MRP was stand-alone also works now. That might of course not be the case but it would seem strange to me if keyboard didn't work. But of course, it's just guesses on my end.
I believe the two keys in tiC are tiV and tiD, probably standing for Version and Data. The data is RTI data (I think RTI stands for ~Real~ Remote Text Input) and version refers to the version of that data. I have no idea what kind of data we are talking about here, what it looks like or what is contains. But that's what I have found so far.
tiD is some sort of NSKeyedArchive which I managed to deserialize with the nska_deserialize lib, however it appears that it's only feedback but looking at the logging it sort of looks like it goes both ways.
I've had a hard time actually serializing one of these archives to send back, I'm expecting it to need some sort of session id included which the _tiStart appears to return.
I agree that I think tiD is some sort of feedback only, because I see in my iOS logs that it is only received post me sending any keys.
Sorry if I am being a bit dumb here. But looking at your _tiStarted/_tiC samples above, they appear to be binary plists. If I decode them using bplist I get the following,
_tiStarted
{'$version': 100000, '$archiver': b'NSKeyedArchiver', '$top': {'documentTraits': b'', 'sessionUUID': b'', 'documentState': b''}, '$objects': [b'$null', {'docSt': b'', '$class': b''}, {'contextBeforeInput': b'', '$class': b''}, b'Testeraaaa', {'$classname': b'TIDocumentState', '$classes': [b'TIDocumentState', b'NSObject']}, {'$classname': b'RTIDocumentState', '$classes': [b'RTIDocumentState', b'NSObject']}, {'bId': b'', 'app': b'', 'locApp': b'', 'tiTraits': b'', '$class': b''}, b'com.apple.TVSearch', b'Search', {'flags': 1410341510, '$class': b''}, {'$classname': b'TITextInputTraits', '$classes': [b'TITextInputTraits', b'NSObject']}, {'$classname': b'RTIDocumentTraits', '$classes': [b'RTIDocumentTraits', b'NSObject']}, b'6\xa7\xfa+\x03\xadL\x1d\xbd\x04\xab\xec\x96:6\x9d']}
_tiC
{'$version': 100000, '$archiver': b'NSKeyedArchiver', '$top': {'documentTraits': b'', 'sessionUUID': b'', 'documentState': b''}, '$objects': [b'$null', {'docSt': b'', '$class': b''}, {'contextBeforeInput': b'', '$class': b''}, b'a', {'$classname': b'TIDocumentState', '$classes': [b'TIDocumentState', b'NSObject']}, {'$classname': b'RTIDocumentState', '$classes': [b'RTIDocumentState', b'NSObject']}, {'bId': b'', 'app': b'', 'locApp': b'', 'tiTraits': b'', '$class': b''}, b'com.apple.TVSearch', b'Search', {'flags': 1410341510, '$class': b''}, {'$classname': b'TITextInputTraits', '$classes': [b'TITextInputTraits', b'NSObject']}, {'$classname': b'RTIDocumentTraits', '$classes': [b'RTIDocumentTraits', b'NSObject']}, b'6\xa7\xfa+\x03\xadL\x1d\xbd\x04\xab\xec\x96:6\x9d']}
So is the unique identifier you might need to include not just the last byte string? So the "6\xa7\xfa+\x03\xadL\x1d\xbd\x04\xab\xec\x96:6\x9d" value that is received at the end of each bplist?
I think in the _tiStart it tells you what is already on the screen of the AppleTV. So in your example you had previously searched for "Testeraaaa" I am guessing? And then you press the letter "a" which is seen in the _tiC plist.
So what happens if you decode the bplist, take the last value that seems to be the only unique value and created your own bplist and send it back using _tiC?
There doesn't seem to be much unique between each bplist so you could try just take the _tiStart, decode it, use it as a template and then just sent it back as a _tiC but change third object under $objects (which is equal to "Testeraaaa" and "a" in these examples).
Or is this what you have tried already?
I have actually sort of tried that, however I have not been certain the plist format I was sending was actually encoded any good. I haven't ran any deeper into the plists, only looked for some libs that could encode and decode them but every one of them gave different results going either way.
I have tried copying and injecting stuff in the bytestring using (I believe) the sessionUUID in there but did not get any luck as of yet.
I do agree tiC might be the one though.
Would you be able to share some of the test code you have written to try sending _tiC, HID commands etc? I’m not a python programmer so that would save me some time getting a test bed up and running and then I can do a bit of trial and error after that to see if I get anywhere
HID commands are already in pyatv (see pyatv/protocols/companion/api.py def "hid_command")
Here's the last stuff that I hacked in. ( I see I was sending HID over mrp since it looked like it had more HID support) Also after a few quick tests, _tiC cannot be send as command, it is an event. I believe that isn't the one we're gonna be needing.
Here's a little test file + class I whipped up.
import asyncio
from pyatv import Protocol
from pyatv.interface import RemoteControl
from pyatv.protocols.companion import CompanionAPI
from pyatv.protocols.mrp import MrpProtocol
from pyatv.protocols.mrp import messages
from pyatv import connect as pyatv_connect, scan as pyatv_scan, pair as pyatv_pair
import logging
logging.basicConfig(
level=logging.DEBUG,
datefmt="%Y-%m-%d %H:%M:%S",
format="%(asctime)s %(levelname)s [%(name)s]: %(message)s",
)
class AppleTVTester:
def __init__(self, atv):
self.atv = atv
async def init_keyboard(self):
api = self.get_companion_api()
await api.subscribe_event("_tiC")
await api.subscribe_event("_tiStarted")
await api.subscribe_event("_tiStopped")
return await api._send_command("_tiStart", {"_tiC": 1})
def get_companion_api(self) -> CompanionAPI:
setup_data = self.atv._protocol_handlers.get(Protocol.Companion)
api: CompanionAPI = setup_data.interfaces[RemoteControl].api
return api
async def get_mrp_protocol(self) -> MrpProtocol:
setup_data = self.atv._protocol_handlers.get(Protocol.MRP)
remote_control = setup_data.interfaces[RemoteControl]
return remote_control.protocol
async def hid(self, page: int, command: int, down: bool):
logging.info(f"hid {page} {command} {down}")
protocol = await self.get_mrp_protocol()
await protocol.send(messages.send_hid_event(page, command, down))
async def hid_command(self, page: str, command: str):
await self.hid(int(page), int(command, 16), True)
await self.hid(int(page), int(command, 16), False)
async def hid_up(self, page: str, command: str):
return await self.hid(int(page), int(command, 16), False)
async def hid_down(self, page: str, command: str):
return await self.hid(int(page), int(command, 16), True)
async def set_text(self, text: str):
setup_data = self.atv._protocol_handlers.get(Protocol.Companion)
api: CompanionAPI = setup_data.interfaces[RemoteControl].api
logging.info("===================================================================== BEFORE SEND COMMAND =====================================================================================")
res = await api._send_command("_tiStart", {"_tiC": 1, "_tiV": 1, "_tiS": "TEST"})
logging.info(
"===================================================================== AFTER SEND COMMAND =====================================================================================")
logging.info(res)
return 'done'
await self.hid_command('7', '0x9c')
for element in range(0, len(text)):
char = text[element]
await self.hid(0x07, ord(char), True)
await self.hid(0x07, ord(char), False)
APPLETV_IP = ""
COMPANION_CREDENTIALS = ""
AIRPLAY_CREDENTIALS = ""
async def test():
loop = asyncio.get_running_loop()
try:
results = await pyatv_scan(loop, hosts=[APPLETV_IP])
except Exception as ex:
logging.error(f'Failed scanning for device {APPLETV_IP}; {ex}')
results = None
if not results:
logging.info(f"Device:{APPLETV_IP} not found!")
return None
else:
logging.info(f'Found:{APPLETV_IP}')
for service in results[0].services:
logging.info(f'With protocol: {APPLETV_IP} {service.protocol.name}')
atvConfig = results[0]
atvConfig.set_credentials(Protocol.Companion, COMPANION_CREDENTIALS)
atvConfig.set_credentials(Protocol.AirPlay, AIRPLAY_CREDENTIALS)
atv = await pyatv_connect(atvConfig, loop)
tester = AppleTVTester(atv)
keyboard_start_response = await tester.init_keyboard()
logging.info(keyboard_start_response)
await tester.set_text("TEST")
logging.info("===================================================================== DONE =====================================================================================")
logging.info("===================================================================== DONE =====================================================================================")
asyncio.run(test())
My best guess is that ti stands for "text input" and the C is likely for Command.
I've created a PR that implements this "text input" method https://github.com/postlund/pyatv/pull/1905.
You're a legend @michalmo , a quick test resulted in glorious success. I'll be testing this for a bit in the near future, hopefully adding the keyboard start and stop events too.