AMF3 connect failed and decode error

[lgjx123]

1. src/main/java/org/red5/client/net/rtmp/BaseRTMPClientHandler.java onCommand method

   log.trace("onCommand: {}, id: {}", command, command.getTransactionId());
   final IServiceCall call = command.getCall();
   final String methodName = call.getServiceMethodName();
   log.debug("Service name: {} args[0]: {}", methodName, (call.getArguments().length != 0 ? call.getArguments()[0] : ""));
   if ("_result".equals(methodName) || "_error".equals(methodName)) {
       final IPendingServiceCall pendingCall = conn.getPendingCall(command.getTransactionId());
       log.debug("Received result for pending call - {}", pendingCall);
       if (pendingCall != null) {
           if ("connect".equals(methodName)) {
               Integer encoding = (Integer) connectionParams.get("objectEncoding");
               if (encoding != null && encoding.intValue() == 3) {
                   log.debug("Setting encoding to AMF3");
                   conn.getState().setEncoding(IConnection.Encoding.AMF3);
               }
           }
       }
       handlePendingCallResult(conn, (Invoke) command);
       return;
   }
   

   if ("connect".equals(methodName)) is unreach code, it should be if ("connect".equals(pendingCall.getServiceMethodName()))

2. server use these code to send data to client （org.red5.server.net.rtmp.codec.RTMPProtocolEncoder encodeCommand(IoBuffer out, ICommand command)）

   Output output = new org.red5.io.amf.Output(out);
   final IServiceCall call = command.getCall();
   final boolean isPending = (call.getStatus() == Call.STATUS_PENDING);
   log.debug("Call: {} pending: {}", call, isPending);
   if (!isPending) {
       log.debug("Call has been executed, send result");
       Serializer.serialize(output, call.isSuccess() ? "_result" : "_error");
   } else {
       log.debug("This is a pending call, send request");
       final String action = (call.getServiceName() == null) ? call.getServiceMethodName() : call.getServiceName() + '.' + call.getServiceMethodName();
       Serializer.serialize(output, action); // seems right
   }
   if (command instanceof Invoke) {
       Serializer.serialize(output, Integer.valueOf(command.getTransactionId()));
       Serializer.serialize(output, command.getConnectionParams());
   }
   

   and decode data like this( org.red5.server.net.rtmp.codec.RTMPProtocolDecoder decodeInvoke(Encoding encoding, IoBuffer in) )

   if (action != null) {
       Invoke invoke = new Invoke();
       invoke.setTransactionId(Deserializer.<Number> deserialize(input, Number.class).intValue());
       // now go back to the actual encoding to decode parameters
       if (encoding == Encoding.AMF3) {
           input = new org.red5.io.amf3.Input(in);
           ((org.red5.io.amf3.Input) input).enforceAMF3();
       } else {
           input = new org.red5.io.amf.Input(in);
       }
       // get / set the parameters if there any
       Object[] params = in.hasRemaining() ? handleParameters(in, invoke, input) : new Object[0];
   

   red5-client will thrown decode error cause Serializer.serialize(output, command.getConnectionParams());(still AMF0 but skip) when use AMF3.

   Now I try to override decodeInvoke method in RTMPClientProtocolDecoder ` @Override public Invoke decodeInvoke(Encoding encoding, IoBuffer in) {

     // for response, the action string and invokeId is always encoded as AMF0 we use the first byte to decide which encoding to use
     in.mark();
     byte tmp = in.get();
     in.reset();
     Input input;
     if (encoding == Encoding.AMF3 && tmp == AMF.TYPE_AMF3_OBJECT) {
         input = new org.red5.io.amf3.Input(in);
         ((org.red5.io.amf3.Input) input).enforceAMF3();
     } else {
         input = new org.red5.io.amf.Input(in);
     }
     // get the action
     String action = Deserializer.deserialize(input, String.class);
     if (log.isTraceEnabled()) {
         log.trace("Action {}", action);
     }
     // throw a runtime exception if there is no action
     if (action != null) {
         Invoke invoke = new Invoke();
         invoke.setTransactionId(Deserializer.<Number> deserialize(input, Number.class).intValue());
         // get / set the parameters if there any
         Object[] params = in.hasRemaining() ? clientHandleParameters(encoding,in, invoke, input) : new Object[0];
         // determine service information
         final int dotIndex = action.lastIndexOf('.');
         String serviceName = (dotIndex == -1) ? null : action.substring(0, dotIndex);
         // pull off the prefixes since java doesnt allow this on a method name
         if (serviceName != null && (serviceName.startsWith("@") || serviceName.startsWith("|"))) {
             serviceName = serviceName.substring(1);
         }
         String serviceMethod = (dotIndex == -1) ? action : action.substring(dotIndex + 1, action.length());
         // pull off the prefixes since java doesn't allow this on a method name
         if (serviceMethod.startsWith("@") || serviceMethod.startsWith("|")) {
             serviceMethod = serviceMethod.substring(1);
         }
         PendingCall call = new PendingCall(serviceName, serviceMethod, params);
         invoke.setCall(call);
         return invoke;
     } else {
         // TODO replace this with something better as time permits
         throw new RuntimeException("Action was null");
     }
   

   } `

` /** * Sets incoming connection parameters and / or returns encoded parameters for use in a call. * * @param in * @param notify * @param input * @return parameters array */ private Object[] clientHandleParameters(Encoding encoding,IoBuffer in, Notify notify, Input input) {

    Object[] params = new Object[] {};
    List<Object> paramList = new ArrayList<Object>();
    final Object obj = Deserializer.deserialize(input, Object.class);
    if (obj instanceof Map) {
        // Before the actual parameters we sometimes (connect) get a map of parameters, this is usually null, but if set should be
        // passed to the connection object.
        @SuppressWarnings("unchecked")
        final Map<String, Object> connParams = (Map<String, Object>) obj;
        notify.setConnectionParams(connParams);
    } else if (obj != null) {
        paramList.add(obj);
    }

    // now go back to the actual encoding to decode parameters
    if (encoding == Encoding.AMF3) {
        input = new org.red5.io.amf3.Input(in);
        ((org.red5.io.amf3.Input) input).enforceAMF3();
    } else {
        input = new org.red5.io.amf.Input(in);
    }

    while (in.hasRemaining()) {
        paramList.add(Deserializer.deserialize(input, Object.class));
    }
    params = paramList.toArray();
    if (log.isDebugEnabled()) {
        log.debug("Num params: {}", paramList.size());
        for (int i = 0; i < params.length; i++) {
            log.debug(" > {}: {}", i, params[i]);
        }
    }
    return params;
}

` and it fixed the decode error.

[mondain]

I think some of the formatting is messed up in your post; its a little hard to follow.

[solomax]

Maybe this might be filed as PR with Test and the fix?

[mondain]

Test your code against 1.0.8-M8, it has AMF decoding fixes

[solomax]

client still unable to connect to RTMPS in tunneled mode :( going to check native later

On Tue, Aug 16, 2016 at 1:31 AM, Paul Gregoire [email protected] wrote:

Test your code against 1.0.8-M8, it has AMF decoding fixes

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Red5/red5-client/issues/35#issuecomment-239886903, or mute the thread https://github.com/notifications/unsubscribe-auth/ADsPf3O7XB9Xb9sdp5DhGwKejByFcDdmks5qgLCfgaJpZM4JaNUV .

WBR Maxim aka solomax