Silent data corruption when writing a message exceeding 2GB in size

[zabetak]

What version of protobuf and what language are you using? Version: v21.12 Language: Java

What operating system (Linux, Windows, ...) and version? Ubuntu 20.04.5 LTS

What runtime / compiler are you using (e.g., python version or gcc version)

What did you do?

1. Create a (huge) message greater than 2GB.
2. Write the message to a file.
3. Read back the message from the file.

https://github.com/zabetak/protobuf-large-message is simple project reproducing the problem with a reduced message model from the protobuf tutorial.

What did you expect to see I was expecting the message creation (Step 1) or write (Step 2) to fail with a meaningful message or the message to be read correctly from the file.

What did you see instead? I got the following exception while attempting to read the message from the file.

Exception in thread "main" com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large.  May be malicious.  Use CodedInputStream.setSizeLimit() to increase the size limit.
	at com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:162)
	at com.google.protobuf.CodedInputStream$StreamDecoder.refillBuffer(CodedInputStream.java:2781)
	at com.google.protobuf.CodedInputStream$StreamDecoder.readRawByte(CodedInputStream.java:2859)
	at com.google.protobuf.CodedInputStream$StreamDecoder.readRawVarint64SlowPath(CodedInputStream.java:2648)
	at com.google.protobuf.CodedInputStream$StreamDecoder.readRawVarint32(CodedInputStream.java:2542)
	at com.google.protobuf.CodedInputStream$StreamDecoder.readMessage(CodedInputStream.java:2405)
	at com.github.zabetak.protobuf.bugs.protos.AddressBook$Builder.mergeFrom(AddressBook.java:440)
	at com.github.zabetak.protobuf.bugs.protos.AddressBook$1.parsePartialFrom(AddressBook.java:742)
	at com.github.zabetak.protobuf.bugs.protos.AddressBook$1.parsePartialFrom(AddressBook.java:734)
	at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:86)
	at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:91)
	at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:48)
	at com.google.protobuf.GeneratedMessageV3.parseWithIOException(GeneratedMessageV3.java:364)
	at com.github.zabetak.protobuf.bugs.protos.AddressBook.parseFrom(AddressBook.java:219)
	at com.github.zabetak.protobuf.bugs.serde.WriteReadBook.readBooks(WriteReadBook.java:63)
	at com.github.zabetak.protobuf.bugs.serde.WriteReadBook.main(WriteReadBook.java:35)

The reason for the exception is clear and understood but now that the file is (permanently?) corrupted there is not much we can do.

Anything else we should know about your project / environment I understand that it is not a good practice to write huge messages but there should be some kind of guard against data corruption.

This problem came up in actual deployments of Apache Hive, Spark, etc, due to ORC metadata that are stored as protobuf messages.

Probably the ORC team will reconsider the message layout in future versions but it may be a good idea to find a way to prevent this from within the protobuf library.

Relevant tickets:

• https://issues.apache.org/jira/browse/HIVE-11268
• https://issues.apache.org/jira/browse/HIVE-11592
• https://issues.apache.org/jira/browse/HIVE-26987
• https://issues.apache.org/jira/browse/ORC-1361

[shaod2]

While I'm seeking for an answer, have you tried increase the limit as prompted?

[zabetak]

Thanks for looking into this @shaod2 ! It is not possible to increase the limit further cause it is already set to the maximum value Integer.MAX_VALUE ~2GB.

If there is no way to read back messages greater than 2GB then I would suggest to add checks to prevent this messages from being constructed/serialized in the first place.

[zabetak]

@shaod2 Did you have a chance to look into this? Do you confirm that it is a problem that should be addressed?

[github-actions[bot]]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please add a comment.

This issue is labeled inactive because the last activity was over 90 days ago.

[zabetak]

The issue is still relevant and IMHO quite important to avoid unexpected data loss.

[github-actions[bot]]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please add a comment.

This issue is labeled inactive because the last activity was over 90 days ago.

[zabetak]

This is still relevant and should remain active.

[github-actions[bot]]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please add a comment.

This issue is labeled inactive because the last activity was over 90 days ago.

[zabetak]

Still relevant!

[YoshuaNava]

Hi! Thank you for reporting this issue.

I'm also a user of Protobuf, and I wanted to ask if the size limit applies to

1. Any Protobuf that reaches more than 2GB,
2. Protobufs that store N amount of objects, where N surpasses Integer.MAX_VALUE,

This is unclear from the documentation.

Thank you in advance :pray:

Best regards, Yoshua Nava

[github-actions[bot]]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please add a comment.

This issue is labeled inactive because the last activity was over 90 days ago. This issue will be closed and archived after 14 additional days without activity.

[github-actions[bot]]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please reopen it.

This issue was closed and archived because there has been no new activity in the 14 days since the inactive label was added.

[ufengtao]

@YoshuaNava max size is 2GB, https://protobuf.dev/programming-guides/proto-limits/