Implement zlib compression

[joe-mann]

Description

Implemented the SQL compression protocol. This new feature is enabled by:

• Adding compress=true in DSN.
• cfg.Apply(Compress(True))

A new PR to revive, rebase, and complete https://github.com/go-sql-driver/mysql/pull/649.

Checklist

• [x] Code compiles correctly
• [x] Created tests which fail without the change (if possible)
• [x] All tests passing
• [x] Extended the README / documentation, if necessary
• [x] Added myself / the copyright holder to the AUTHORS file

Summary by CodeRabbit

• New Features
  • Introduced zlib compression support in the MySQL driver for Go, including a compress parameter for compression modes and a minCompressLength parameter to specify the minimum packet size for compression.
  • Added new authors B Lamarche and Joe Mann to the project.
• Bug Fixes
  • Added error handling for scenarios where compression is requested but not supported by the server.
• Documentation
  • Updated README.md to detail new compression features and parameters.
• Tests
  • Enhanced benchmark and unit tests to cover new compression functionality.
• Chores
  • Updated various internal components to support compression during MySQL communication, including packet reading/writing and connection handling.

[joe-mann]

@julienschmidt, @methane please review and let me know if any further changes are required. We would very much like to have this feature merged and available to us.

[coveralls]

coverage: 81.056% (-0.9%) from 81.989% when pulling ee70acf1bb32ae2c26611d59d206761d38bee7cd on joe-mann:compression into 8a327a3575a42f7222f6e51263326d5a0eaecab0 on go-sql-driver:master.

[shogo82148]

updates https://github.com/go-sql-driver/mysql/pull/649

[joe-mann]

@shogo82148, @julienschmidt, @methane, I've made some small changes and additions from the original PR. Please can you take a look?

[dolmen]

@joe-mann Please git rebase (or git pull --rebase). Do not merge master into your branch.

[methane]

@dolmen We don't require rebase.

• We use "Squash and merge" commit so no need for clean history in pull request branch.
• rebase breaks PR sometime. python/cpython is suffered from massive mention from broken PR that author tried to rebase.

[joe-mann]

@dolmen, README formatting is reverted.

[methane]

• disabled: no compression, default value
• preferred: use zlib compression if offered by the server
• required: use zlib compression, error if not offered by the server
• true/1: equivalent to preferred
• false/0: equivalent to disabled

I don't think "required" is needed at all, as MySQL Connector doesn't provide it. Please start with only true/false. disable/preferred/required would be added later if we learned they are needed.

[coderabbitai[bot]]

[!IMPORTANT]

Auto Review Skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

The update enhances the MySQL driver in Go with zlib compression support, enabling better control over data packet compression and decompression. These changes improve performance and flexibility in database interactions. Various files have been modified to incorporate new parameters, functions, and error handling related to compression.

Changes

Files	Change Summary
AUTHORS	Added B Lamarche and Joe Mann as new authors.
README.md	Introduced support for zlib compression in the MySQL driver, adding compress and minCompressLength parameters.
benchmark_test.go, compress_test.go, packets_test.go, connection_test.go, driver_test.go	Updated and added tests for compression functionality, including new parameters in initDB, handling benchmarking with compression, and ensuring correctness in compression/decompression processes. Adjusted test setups to align with new compression features.
compress.go, compress_test.go	Added functionality for compressing and decompressing data using zlib, including helper functions for managing compression processes.
connection.go, connector.go, packets.go	Updated to handle packet compression, including new fields and methods for managing compression states and sequence numbers. Adjusted packet reading and writing to accommodate compression.
dsn.go	Enhanced the Config struct to include compression settings, updated FormatDSN and parseDSNParams to handle compression-related parameters.
const.go, errors.go, infile.go	Introduced a new constant for minCompressLength, added ErrNoCompression error variable, and updated handleInFileRequest to sync sequence numbers.

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

• X
• Mastodon
• Reddit
• LinkedIn

Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai generate interesting stats about this repository and render them as a table.
  • @coderabbitai show all the console.log statements in this repository.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[dveeden]

Would this have an option to choose between zlib and zstd and maybe also to set the compression level?

$ mysql --help
...
  -C, --compress      Use compression in server/client protocol.
...
  --compression-algorithms=name 
                      Use compression algorithm in server/client protocol.
                      Valid values are any combination of
                      'zstd','zlib','uncompressed'.
  --zstd-compression-level=# 
                      Use this compression level in the client/server protocol,
                      in case --compression-algorithms=zstd. Valid range is
                      between 1 and 22, inclusive. Default is 3.
...

See also CLIENT_COMPRESS and CLIENT_ZSTD_COMPRESSION_ALGORITHM in the docs. Note that Protocol::HandshakeResponse41 has an optional field when CLIENT_ZSTD_COMPRESSION_ALGORITHM is set.

[methane]

Maybe. But not in 1.9. This PR is still inefficient. I will need more improvement.

[dveeden]

@methane Feel free to contact me if you need any help.

[methane]

I realized we need massive rewrite. So nitpick comments are not helpful to me.

[SongZhen0704]

@joe-mann @methane Hello, I want to use mysql compress in gorm, so may i known whether this pr can be merged?

Thank you

[methane]

This branch works for simple cases. But when I checked edge cases, I found some difficult issues. I think we need some rewrite.

At the moment, please use this branch at your own risk.

[SongZhen0704]

This branch works for simple cases. But when I checked edge cases, I found some difficult issues. I think we need some rewrite.

At the moment, please use this branch at your own risk.

Thanks for your reply