stf
stf copied to clipboard
openstf
iOS support
Awesome, inspiring stuff.
TLDR; is iOS support on your roadmap?
I've been tinkering around with a solution that works on iOS devices (or simulators). The approach streams snapshots over websockets to a canvas and forwards user-generated mouse/keyboard events to Appium which replays them against a device. For a stand alone solution, with a single device, it works.
next up on the roadmap:
- improve performance of image capture
- support multicasting of a device
- support streaming multiple concurrent devices (i.e. one per browser tab)
- polish the code (env variables and such)
- approval w/ stakeholders to release as open source
I understand its hard to comment on a solution you haven't seen. Before I get too much further down this proof-of-concept, I wanted to solicit feedback Are there any plans to support iOS in STF? Have you already been down this path and abandoned it?
proof of concept approach
image capture
To capture images, I've evaluated:
- using
libimobiledevicewhich uses devloper disk image to capture on the device to achieve 5-10 fps. Unfortunatly these are large files (3MB TIFFs), that are begging to be pre-processed into a PNG. - using QuickTime's screen mirroring; and screenshoting that (on the host) which allows ~25 fps, but won't scale beyond the first device.
- forking xrecord to implement a QT mirroing solution that output jpegs (and would allow multiple devices)
devices
Testing against a physical device is desirable however without jailbreaking the device there are some limitations (lockscreen, home button).
The iOS simulator was able to produce highest framerates, and the closest experience to what STF provides for Android, but there is a nagging feeling it's not going to scale or meet the requirements (e.g. managed in MDM, recgnoize an internal cert chain)
Shortcomings
- Appium is limited to the underlying device automation implementation. On iOS it is UIATarget
- UIATarget has an api to send the application to the background for a duration. Appium wants to launch an application on startup. On a physical device this means there's no interacting with Home Screen, system properties without jailbreaking the device and getting creative. For simulators, there may be a way to mimic, but would need to figure out how to support concurrent simulators.
- perceptable lag (~1-2 seconds) on keyboard events propagating back to the device. A short (100ms) buffer for keyboard events may help, but special keys like Delete make it interesting
- Duplication of inventory in Selenium Grid and STF. We've got a node module that watches USB for new device connections, starts an Appium server and registeres it with a Selenium grid, but we need to federate that back into STF for discovery
Hi,
Thanks for the awesome work. We would like to have iOS support, yes. However we currently do not have the resources to develop it by ourselves, but if someone provides a reasonably functional version I may be able to port the rest of the functionality. I would be very interested in seeing what you've got so far.
Btw you mentioned you were hoping to perhaps compress TIFF frames into PNG. I think PNG is a lost cause for screen capture purposes as the compression is very very slow compared to JPG. JPG (w/ libjpeg-turbo) is the fastest method we've found so far, but over time we're planning on moving away from JPG frames to transferring raw x264 frames instead. As long as the device supports hardware encoding, this method should yield the highest FPS.
I'm still working on this. The code's in much better shape, and the stakeholders are listening.
Thanks for the tip on TIFF to JPG. It shaved 100ms off the image processing.
Amazing.
For capturing images, I get that you are using com.apple.mobile.screenshotr which firsts captures a TIFF frame and then you have to convert it to JPEG; which will not get you the max frame-rate.
Since iOS 8 there is com.apple.cmio.iOSScreenCaptureAssistant which gives you an encoded video stream. If you are able to get the stream (you may have to do some reverse-engineering) and then somehow forward that to the browser, you'll probably get the highest FPS.
On the Browser side then you could try to do any necessary processing and show it by using Media Source Extensions or WebRTC.
If you need any help in getting the approval of stakeholders, just let them know that we can add the name of your company in the credits as a big contributor on http://openstf.io, if that is the case just let us know ;)
The effort stalled. Stakeholders haven't said no, but they've shifted priorities.
I'm able to sample stills from a video stream, but it's drinking from a firehose. I was refactoring the image acquisition code to send the results back over a WebSocket (rather than diskIO), when I bumped into a hard limit of one device at a time https://github.com/WPO-Foundation/xrecord/issues/1
Hi ,
Can you please share your solution? We want to test it out for iOS devices?
If anyone has a SD solution available please share the same. We are in dire need of getting this on iOS.
If you feel like you need an iOS implementation faster, consider posting a bounty on this issue (see badge below). If the bounty gets large enough, perhaps we or someone else will take a look at it.
Dear Sorccu, you mentioned to me that you would give me the exact quote for developing the iOS Support. Can you please give me as soon as possible?
Regards Pattabhi
@sorccu @kwv it might be worth checking this out https://github.com/phonegap/ios-deploy and https://github.com/phonegap/ios-sim for building and deploying iOS apps to device and simulators. I know STF does not currently support iOS but it would be nice to deploy apps via the web browser. The tool also supports lldb which is useful for out putting console logs.
Curious to the original proposed solution that got stalled, as well as any other potential solutions. How far can you scale device support (or even simulator support)?
From what I encountered, unless I'm mistaken, just toying with Appium and physical iOS devices, there seemed to be a single device limitation per instance of XCode on Mac OS. Therefore, to reliably scale to more iOS devices (to work in parallel), one has to deploy 2 virtual machines (the max licensed amount allowed by Apple, on the physical Mac host) running Mac OS with the same XCode and Appium setup, etc. In that setup, assign a specific iOS device per VM via USB port/device allocation in the VM software, and the remaining (last) device to the physical Mac. Each instance (physical Mac, VMs) have their own XCode and Appium to control the device. So there's a max device limit of 3 iOS devices per Mac machine (using the VMs). This doesn't scale as well compared to Android, as more Mac machines would be needed to scale higher.
@daluu
XCode supports testing multiple real devices using UIAutomator since v6.3.2
https://github.com/appium/appium/issues/273#issuecomment-110145744
this might be useful for video recording https://github.com/isonic1/flick
@Ashraf-Ali-aa This tools about iOS recording is using capture pictures and combine them to video file. maybe not fitted for stf.
@codeskyblue it works also on Android, should be able to be used without problems
Since stf got minicap, combine the image stream into video is a better way in my view.
@gunta I have a question, how to know ios orientation from command line.
facebook have implemented an extension of xcuitest that allows remote interactions with ios devices and appium has adopted the code.
https://github.com/facebook/WebDriverAgent https://github.com/appium/appium-xcuitest-driver
related links: https://github.com/facebook/FBSimulatorControl https://github.com/rsms/peertalk
I think that's probably the reason why no one has contributed a solution yet. It takes a huge amount of developer time and there are clearly financial benefits for end users. In other words it would be a viable product. Not many are willing to give that away for free.
On Tuesday, 21 June 2016, Naresh Jain [email protected] wrote:
any update on iOS support? If anyone has a iOS solution available for OpenSTF please share, that will help my company to reduce cost on devices.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openstf/stf/issues/64#issuecomment-227408518, or mute the thread https://github.com/notifications/unsubscribe/AAB-_bzGMKCJOp7_YKDQZARghu4O20xWks5qN8WwgaJpZM4FyyUk .
@daluu I never made it past the scaling limitation.
I started down the path of spinning up VMs for each tethered device, but I started to recognize the difference between being a trailblazer and stumbling around in the woods. Defining a repeatable approach was going to be challenging, let alone coming up with a solution that would fit back into STF.
@Ashraf-Ali-aa Capturing video and sending the stream back to a browser was a problem. @codeskyblue is right, it is easier to use images over a websocket to fake a stream. Munging the QT frames into a websocket was next, but that required stronger opinions on the architecture. (Does the app collecting the screen pipe directly to the browser or through a middle agent? If its direct, what supports discovery, or starts the app?) I wanted to leverage a lot of STF's architecture decisions for this, but the approach had already diverged.
I wonder how the commercial products work, if they use an app installed on the device which functions similar to minicap. Rather than pulling from the host machine (like the approach outlined above), push from the device itself (using private APIs?). In the near term FBSimulatorControl looks very interesting.
If nothing else, it just re-enforces that @sorccu and @gunta are doing amazing work.
Hello guys! Like @kwv i'm trying find a way to screencast the ios screen, i am currently try apply this code from @isonic1, this one https://github.com/isonic1/flick, to make a minicap for IOS. if you guys have some advise to me please tell me.
@NareshJain91 I am dividing the problem, first i want identify the device and set it visible in stf, next i want make the screencast from ios devices! that is my main work, for now, but we want keep going until get full functionality presented in the android. So far i am fighting to detect iphone and make screencast.Now with flick i am capable to detect the udid for all devices and take screenshots in all ios devices. I will try this path to make a solution.
@HelderVSousa Sure, if you need my assistance anywhere please let me know. Till now i am able to capture ScreenShots and video from iOS Devices, also i can get how many devices are connected and their UDID. But major problem i am stuck is how to give command from browser/System to iPhone, if it resolves i will be able to implement rest things, for that i have also raised the question to Developer apple forum, but most of have replied that apple do not allow it.
If you find something to give commend from mac to iOS, let me know. Thanks
They probably meant that Apple wouldn't allow you to publish an app that does that. However, you should be able to use private APIs to your heart's content if you never publish your app in the App Store.
@HelderVSousa you can use https://github.com/phonegap/ios-deploy to detect connected iOS devices
@Ashraf-Ali-aa thanks for the tip, for now my work is fully focused on detect and screen capture ios, my recent research and tests show me the flick can be the tool i am looking for, i will create minicapIOS based on approach in minicap. But in future i will check it to see if i can use to control the ios.