[iOS] NetStream.Play.Failed on video playback via VideoTexture

[itlancer]

trafficstars

Problem Description

NetStream.Play.Failed error on video playback via VideoTexture with iOS devices. So you cannot play video with VideoTexture for iOS devices. It critical for iOS applications that uses video playback with Stage3D.

Tested with latest AIR 33.1.1.856 with multiple different iOS devices with different OS versions with different MP4 H.264 videos. Same problem in all cases. The same issues exists if you try to create similar application using Starling.

There is no such issues with Windows, macOS and Android.

Related issue (not the same): https://github.com/airsdk/Adobe-Runtime-Support/issues/211

Steps to Reproduce

1. Launch code below with any iOS device. It creates Context3D.
2. Click anywhere on stage. VideoTexture and NetStream will be created and tried to play video.

Note that renderMode in application manifest should be direct.

Application example with sources, example of video and Scout log attached. ios_videotexture_video_playback_bug.zip

package {
	import flash.display.Sprite;
	import flash.events.MouseEvent;
	import flash.display.Stage3D;
	import flash.events.Event;
	import flash.events.NetStatusEvent;
	import flash.net.NetStream;
	import flash.display3D.Context3D;
	import flash.display3D.IndexBuffer3D;
	import flash.geom.Matrix3D;
	import flash.net.NetConnection;
	import flash.display3D.textures.VideoTexture;
	import com.adobe.utils.AGALMiniAssembler;
	import flash.display3D.Context3DProgramType;
	import flash.display3D.Context3DVertexBufferFormat;
	import flash.display3D.VertexBuffer3D;
	import flash.display3D.Program3D;
	
	public class IOSVideoTextureVideoPlaybackBug extends Sprite {
		private var stage3D:Stage3D;
		private var context3D:Context3D;
		private var indexbuffer:IndexBuffer3D;
		private var matrix:Matrix3D = new Matrix3D();
		private var nc:NetConnection;
		private var ns:NetStream;
		private var videoTexture:VideoTexture;
		
		public function IOSVideoTextureVideoPlaybackBug() {
			stage3D = stage.stage3Ds[0];
			stage3D.addEventListener(Event.CONTEXT3D_CREATE, contextCreated);
			stage3D.requestContext3D();
			
			stage.addEventListener(MouseEvent.CLICK, click);
		}
		
		private function click(e:MouseEvent):void {
			trace("click");
			var vertices:Vector.<Number> = Vector.<Number>([
			1, -1, 0, 1, 0,
			1, 1, 0, 1, 1,
			-1, 1, 0, 0, 1,
			-1,-1, 0, 0, 0
			]);
			
			var vertexbuffer:VertexBuffer3D = context3D.createVertexBuffer(4, 5);
			vertexbuffer.uploadFromVector(vertices, 0, 4);
			
			indexbuffer = context3D.createIndexBuffer(6);
			indexbuffer.uploadFromVector(Vector.<uint>([0, 1, 2, 2, 3, 0]), 0, 6);
			
			var vertexShaderAssembler:AGALMiniAssembler = new AGALMiniAssembler();
			vertexShaderAssembler.assemble(Context3DProgramType.VERTEX, "m44 op, va0, vc0\n" + "mov v0, va1");
			
			var fragmentShaderAssembler:AGALMiniAssembler = new AGALMiniAssembler();
			fragmentShaderAssembler.assemble( Context3DProgramType.FRAGMENT, "tex ft1, v0, fs0 <2d,linear, nomip>\n" + "mov oc, ft1");
			
			var program:Program3D = context3D.createProgram();
			program.upload( vertexShaderAssembler.agalcode, fragmentShaderAssembler.agalcode);
			
			context3D.setVertexBufferAt(0, vertexbuffer, 0, Context3DVertexBufferFormat.FLOAT_3);
			context3D.setVertexBufferAt(1, vertexbuffer, 3, Context3DVertexBufferFormat.FLOAT_2);
			
			videoTexture = context3D.createVideoTexture();
			context3D.setTextureAt(0, videoTexture);
			
			context3D.setProgram(program);
			matrix.appendScale(1, -1, 1);
			context3D.setProgramConstantsFromMatrix(Context3DProgramType.VERTEX, 0, matrix, true);
			
			nc = new NetConnection();
			nc.addEventListener(NetStatusEvent.NET_STATUS, netStatusHandler);
			nc.connect(null);
		}
		
		private function contextCreated(event:Event):void {
			trace("contextCreated");
			
			context3D = stage.stage3Ds[0].context3D;
			context3D.enableErrorChecking = true;
			context3D.configureBackBuffer(300, 240, 4, false, false, true);
			trace(context3D.driverInfo);
		}
		
		private function netStatusHandler(event:NetStatusEvent):void {
			trace(event.info.code);
			switch (event.info.code){
				case "NetConnection.Connect.Success":
					ns = new NetStream(nc);
					ns.addEventListener(NetStatusEvent.NET_STATUS, netStatusHandler);
					ns.client = {onMetaData:getMeta, onPlayStatus:onPlayStatus};
					ns.play("neon.mp4");
					videoTexture.attachNetStream(ns);
					
					videoTexture.addEventListener(Event.TEXTURE_READY, renderState);
					break;
				case "NetStream.Play.StreamNotFound":
					trace("Stream not found");
					break;
				default:
					break;
			}
		}

		private function getMeta(mdata:Object):void {
			trace("metadata");
		}

		private function onPlayStatus(infoObject:Object):void {
			trace("onPlayStatus", infoObject.code);
			ns.seek(0);
		}

		private function renderState(e:Event):void {
			videoTexture.removeEventListener(Event.TEXTURE_READY, renderState);
			trace("renderState");
			render();
			addEventListener(Event.ENTER_FRAME, enterFrame);
		}
	
		private function enterFrame(event:Event):void {
			render();
		}

		private function render():void {
			context3D.clear(1, 0, 0, 1);
			context3D.drawTriangles(indexbuffer);
			context3D.present();
		}
	}
}

Actual Result: Video not played. In Scout log you will see NetStream.Play.Failed error:

contextCreated
OpenGL
click
NetConnection.Connect.Success
NetStream.Play.Start
metadata
NetStream.Play.Failed
NetStream.Play.Stop

Expected Result: Video playback without errors.

Known Workarounds

Use NetStream::attachNetStream() first and then NetStream::play(). In sample change:

ns.play("neon.mp4");
videoTexture.attachNetStream(ns);

to

videoTexture.attachNetStream(ns);
ns.play("neon.mp4");

*Video also doesn't work with MP4 H.264 videos for iOS: https://github.com/airsdk/Adobe-Runtime-Support/issues/211

[itlancer]

Issue still exists with AIR 50.2.3.6.

[Jason0247]

@itlancer Sorry, I don't quite understand. What are the benefits of using VideoTexture to play videos? Perhaps this question is very naive. On iOS, I have been using stageVideo for MP4 video playback, and using VideoTexture looks very complex.

[itlancer]

@Jason0247, common pros of VideoTexture against StageVideo:

• You can use it for complex render using Stage3D/Starling with other GPU-accelerated graphics, texts etc.
• You can rotate video.
• You can "draw" video content to BitmapData (make screenshot).
• You can "clear" video content. Similar to Video::clear().
• You can display video on any type of "form". Plane, cylinder, ... Use video in different effects/animations. Even for particle effects.
• Display semi-transparent video. If you have some graphics content below video.
• No "zoom out" issue https://github.com/airsdk/Adobe-Runtime-Support/issues/1942

To simplify VideoTexture usage you can take a look to Starling https://doc.starling-framework.org/current/starling/textures/Texture.html#fromNetStream() or Feathers https://feathersui.com/learn/as3-starling/video-player/.

[ajwfrost]

Okay I can't believe how long it's taken us to actually find the problem here, apologies.. but it turns out, the code to handle video texture on iOS is very bespoke and a little hidden, the video decoder isn't created in the same way as on any other platform. And it turns out, the condition for initialising this special case is set up during the NetStream 'play' command if the runtime knows that it's outputting to a VideoTexture object.

In this particular test case though, this condition doesn't get set, and video initialisation fails...

					ns.play("neon.mp4");
					videoTexture.attachNetStream(ns);

The workaround is just to ensure that the video texture is set up on the NetStream object, prior to passing in the URL:

					videoTexture.attachNetStream(ns);
					ns.play("neon.mp4");

Please just check that this works for you too... apologies again for how long this took!

[itlancer]

@ajwfrost Thank you very much, it works! Workaround added to original issues. Anyway I think this issue should be fixed, may be with lower priority cause it could stuck a lot of developers and sometimes in complex multimedia systems could be hard to use NetStream::attachNetStream() first and then NetStream::play() always.