Add option to pass raw packet to custom microservice de/serializers

[Dallin343]

Feature Request

We would like the ability to pass the raw packet from an external service to a custom deserializer, which can process the packet and return the proper format for pattern matching.

Is your feature request related to a problem? Please describe.

When we integrated Nest microservices with our external services, we realized that there is currently no way to route events to the proper controller when using formats that don't conform to Nest's requirements. Specifically, when using RabbitMQ and Kafka, we use custom headers to delineate event types. Because of the domain of our work, we cannot change the format of our events, meaning we cannot include the "pattern" key required by Nest. If we could pass the packet to a custom deserializer, we could perform all necessary processing to return an object with the correct format, including the "pattern" key.

Describe the solution you'd like

We would like to add a configuration option alongside custom serializers and deserializers which, when enabled, would pass raw packets to the de/serializer instead of the content alone.

Teachability, Documentation, Adoption, Migration Strategy

This can be used by providing a custom de/serializer and setting a "useRawPacket" configuration option.

What is the motivation / use case for changing the behavior?

In order to be "platform-agnostic", Nest needs to support formats that may not fit its own. Allowing custom serializers and deserializers to process packets into a supported format will allow microservices to support any external service, regardless of message format.

[kamilmysliwiec]

Would you like to create a PR to address this issue? This has been discussed many times in the past and currently, the only solution is to create a custom transport strategy.

[Dallin343]

Absolutely, I'll clean up our implementation and create a PR when I have time.

[leonardovillela]

Hi @kamilmysliwiec, I think this is already supported by microservice deserializer option.

[jeanbmar]

Hi @kamilmysliwiec, I think this is already supported by microservice deserializer option.

No, the JSON socket attempts to parse the raw data to JSON before it arrives in the deserializer.

private handleMessage(data: string) {
    this.contentLength = null;
    this.buffer = '';

    let message: Record<string, unknown>;
    try {
      message = JSON.parse(data);
    } catch (e) {
      throw new InvalidJSONFormatException(e, data);
    }
    message = message || {};
    this.socket.emit(MESSAGE_EVENT, message);
  }

[jeanbmar]

@kamilmysliwiec I can code this PR quicky but I would like your opinion on my solution first.

The main difficulty with raw TCP packets is that NestJS users are responsible for their delimitation. We are not talking about the de/serialization level. We are talking about the socket level and how we delimit a raw message from net.Socket data event.

The solution I have in mind:

1. The existing JsonSocket is split in two classes:

• a generic TcpSocket abstract class which contains generic socket logic,
• a new JsonSocket class which inherits TcpSocket and implements the data / message events as in the existing class.

2. The new TcpSocket class is exposed so users can inherit it and implement their own delimitation of messages from socket data. i.e. they are responsible for managing the socket data event and for emitting the Nest message event.
3. A socket option is added for the Transport.TCP transport when createMicroservice is called. Default value is JsonSocket so we are 100% backward compatible. Users can specify their own derived child from TcpSocket instead, which will be used inside getSocketInstance.
4. I've mainly talked about packet decoding in server class, but packet encoding follows the same logic in client class.

What's your opinion about it? Thank you for your time.

[ologbonowiwi]

this still open?

[jg572h]

Is there an example somewhere of how to use this?

[jeanbmar]

@jg572h JsonSocket is an example of a raw tcp socket implementation. You can implement your own socket the same way, and pass it to ClientTCP and ServerTCP instances using the socketClass option.