Support base64-encoded nested messages

[girst]

This is not part of the spec, but youtube loves to pass base64 encoded messages in string fields of other messages. So when a string is encountered, attempt decoding and recursing into it.

Hope that's understandable, I'm not too familiar with protobuf's terminology. In essence, I'm doing some reversing of youtube's internal browse_ajax API, which is littered with this idiosyncrasy. So to decode such a message, I currently have to pass the protobuf-message to your tool, copy-paste the base64-encoded submessage, decode it, and pass it through again (often multiple times).

Here's how such a message is displayed currently:

and now with my patch applied:

both times invoked with echo '4qmFsgJSEhhVQ1Y1dkNpM2pQSmRVUlp3QU9PX0ZOZlEaNkVnWjJhV1JsYjNNWUF5QUFNQUU0QWVvREUwTm5RVk5EWjJsVmFXOWZRVzVoTWtJdFZ6USUzRA=='|base64 -d|./main.py

[mildsunrise]

hi, thanks for the patch! it's a pretty specific use-case (quoted, urlsafe base64 encoded, missing padding) so I'm not sure it's a good fit to have it by default. However, the config system already allows the user to specify custom handlers for fields, so you could go ahead and define one for base64-encoded nested messages. If you place this in protobuf_config.py:

types = {
  "root": {
    80226972: "msg1",
  },
  "msg1": {
    3: "base64_message",
  },
}

def parse_base64_message(x, type):
  from urllib.parse import unquote
  from subprocess import run
  import base64

  x = unquote(x.read().decode('ascii')) + '=='
  x = base64.urlsafe_b64decode(x)
  output = run('./main.py', input=x, capture_output=True).stdout
  return "base64 encoded message => " + output.decode().rstrip()

native_types = {
  "base64_message": (parse_base64_message, 2),
}

You'll get the desired output. But I understand you want this to happen in the chunk handler to avoid having to explicitely specify it for fields... this can't be easily done for now, I'll think of the best way to make it possible.

[girst]

On Sun, Dec 13, 2020 at 10:28:55AM -0800, Alba Mendez wrote:

hi, thanks for the patch! it's a pretty specific use-case (urlsafe base64 encoded, missing padding) so I'm not sure it's a good fit to

Note that the way I've implemented this, it works for any of the following combinations:

• urlencoded / not urlencoded
• urlsafe base64 (-_) / normal base64 (+/)
• with padding / without padding

In other words, I tried to keep it as non-special-case as possible.

You'll get the desired output. But I understand you want this to happen in the chunk handler to avoid having to explicitely specify it for fields... this can't be easily done for now, I'll think of the best way to make it possible.

Correct. In my use case (reverse engineering) i don't know the field numbers beforehand.

How about a command line flag (passed to the StandardParser class) that enables additional heuristics (maybe someone else has other needs that could be added to it)? I can't think of a good name, but something like --try-harder or something?