[YUNIKORN-2249] Add compression option to getQueueApplication API

[targetoee]

trafficstars

What is this PR for?

When the cluster has many pods, the API response size can be quite large (e.g. 50,000 pod creating a 30MB response in getQueueApplications). It causes long response time due to data transmission. So this PR add a compression option to getQueueApplications API. If user specifies the 'Accept-Encoding' header as gzip, then the scheduler will compress the data before sending back to user.

What type of PR is it?

• [ ] - Bug Fix
• [x] - Improvement
• [ ] - Feature
• [ ] - Documentation
• [ ] - Hot Fix
• [ ] - Refactoring

Todos

The compression option can be applied to other APIs facing similar response size issues.

What is the Jira issue?

YUNIKORN-2249

How should this be tested?

local build

Screenshots (if appropriate)

N/A

Questions:

N/A

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 90.05236% with 95 lines in your changes are missing coverage. Please review.

Project coverage is 79.24%. Comparing base (4816aa2) to head (40b9649). Report is 71 commits behind head on master.

:exclamation: Current head 40b9649 differs from pull request most recent head b811993. Consider uploading reports for the commit b811993 to get more accurate results

Files	Patch %	Lines
pkg/webservice/handlers.go	85.14%	25 Missing and 12 partials :warning:
pkg/scheduler/ugm/manager.go	78.68%	10 Missing and 3 partials :warning:
pkg/scheduler/objects/queue.go	43.75%	7 Missing and 2 partials :warning:
pkg/events/event_system.go	53.33%	7 Missing :warning:
pkg/log/rate_limited_logger.go	33.33%	6 Missing :warning:
pkg/scheduler/objects/application.go	93.33%	3 Missing and 1 partial :warning:
pkg/scheduler/partition.go	85.71%	4 Missing :warning:
pkg/scheduler/scheduler.go	66.66%	4 Missing :warning:
pkg/events/event_streaming.go	97.29%	1 Missing and 1 partial :warning:
pkg/scheduler/context.go	0.00%	2 Missing :warning:
... and 5 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #757      +/-   ##
==========================================
+ Coverage   77.72%   79.24%   +1.51%     
==========================================
  Files          82       82              
  Lines       13430    11418    -2012     
==========================================
- Hits        10439     9048    -1391     
+ Misses       2664     2040     -624     
- Partials      327      330       +3     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[targetoee]

Thanks @chia7712 @pbacsko @wilfred-s for the advice. I've adjust the function so that it can be reused now. The test case is added also. Please take a look.

[targetoee]

Some adjust based on the review. Additionally, the compression step will be skipped if the data size is smaller than MTU. This can prevent unnecessary compression. The check of header is corrected also (in previous commit, it use 'Content-Encoding' to determine whether the request is asking for compression, but it's not the typical usage of this header).

[chia7712]

@targetoee please take a look at https://issues.apache.org/jira/browse/YUNIKORN-2499 for the failed test

[pbacsko]

@targetoee is this still a draft?

[targetoee]

@targetoee is this still a draft?

Yes, there is still something which needs to be modified. I will change the tag after the modification.

[pbacsko]

@targetoee is this still a draft?

Yes, there is still something which needs to be modified. I will change the tag after the modification.

A draft means that you don't want the changes to be reviewed just yet because modifications are still being made. We're already reviewing this PR. I think this can be marked as "Ready for review".

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 92.85714% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 77.15%. Comparing base (5716f46) to head (3f806ba). Report is 31 commits behind head on master.

Files	Patch %	Lines
pkg/webservice/webservice.go	92.85%	1 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #757      +/-   ##
==========================================
- Coverage   79.40%   77.15%   -2.25%     
==========================================
  Files          82       97      +15     
  Lines       11317    12011     +694     
==========================================
+ Hits         8986     9267     +281     
- Misses       2009     2407     +398     
- Partials      322      337      +15     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.