8253413: [REDO] [REDO] G1 incorrectly limiting young gen size when using the reserve can result in repeated full gcs

[tschatzl]

Hi all,

can I have reviews for this change that fixes a bug in how young gen is calculating when G1 is using the region reserve already - in that case it does not bound the young gen by what is available any more, resulting in a guaranteed full gc (from the original change at JDK-8244603.

The reason for backing out and redoing this change already twice is not because of bugs by itself, but it triggered bugs which have been fixed in the meantime. For reference, the original review thread is here.

Here's the text, a bit edited because I am not intending to immediately follow up with the uncommit during young gc changes:

In particular, with this change the following issues two related issues are fixed:

• 8238858: G1 Mixed gc young gen sizing might cause the first mixed gc to immediately follow the prepare mixed gc
• 8244603: G1 incorrectly limiting young gen size when using the reserve can result in repeated full gcs

These have been grouped together because it is too hard to separate them out as the bugs required significant rewrite in the young gen sizing.

This results in G1 following GCTimeRatio much better than before, leading to less erratic heap expansion. That is, constant loads do not result in that much overcommit any more.

Some background:

At end of gc and when the remembered set sampling thread (re-)evaluates the young gen G1 calculates two values:

• the desired (unconstrained) size of the young gen; desired/unconstrained meaning that these values are not limited by actually existing free regions. This value is interesting for adaptive IHOP so that it (better) converges to a fixed value.
• the actual target size for the young gen, i.e. after taking constraints in available free regions into account, and whether we are currently already needing to use parts of the reserve or not.

Some problems that were fixed with the current code:

• there were some calculations to ensure that "at least one region will be allocated" every time g1 recalculates young gen but that really resulted in g1 increasing the young gen by at least one. You do not always want that, particularly since we regularly revise the young gen. What you want is a minimum desired eden size.
• the desired and target young gen size calculation was done in a single huge method. This change splits up the code a bit.
• the code that calculated the actual target young length has been very inconsistent in handling the reserve. I.e. the limit to the actually available free regions only applied in some cases, and notably not if you were already using the reserve, causing strange behavior where at least the calculated young gen target length was higher than available free regions. This could cause full gcs.
• I added some detailed trace-level logging for the ergonomic decisions which really helps when debugging issues, but might be too large/intrusive for the code - I am not sure whether to leave it in.

For reviewing I think the best entry point for this change is G1Policy::update_young_list_target_length(size_t) where the new code first calculates a desired young gen length and then target young length.

• currently, and like before, MMU desired length overrides the pause time desired length. I.e. if a GCPauseTimeIntervalMillis is set, the spacing between gcs is more important than actual pause times. The difference is that now that there is an explicit comment about this behavior there :)

• when eating into the reserve (last 10% of heap), we at most allow use of the sizer's min young length or half of the reserve regions (rounded up!), whichever is higher. This is an arbitrary decision, but since supposedly at that time we are already close to the next mixed gc due to adaptive ihop, so we can take more.

• note that all this is about calculating a target length, not the actual length. Actual length may be higher e.g. due to gclocker.

Testing: tier1-5, perf testing without differences

Thanks, Thomas

---

Progress

• [x] Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• [x] Change must not contain extraneous whitespace
• [x] Commit message must refer to an issue

Issue

• JDK-8253413: [REDO] [REDO] G1 incorrectly limiting young gen size when using the reserve can result in repeated full gcs

Reviewers

• Ivan Walulya (@walulyai - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk pull/9573/head:pull/9573
$ git checkout pull/9573

Update a local copy of the PR:
$ git checkout pull/9573
$ git pull https://git.openjdk.org/jdk pull/9573/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 9573

View PR using the GUI difftool:
$ git pr show -t 9573

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/9573.diff

[bridgekeeper[bot]]

:wave: Welcome back tschatzl! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk[bot]]

@tschatzl The following label will be automatically applied to this pull request:

• hotspot-gc

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[mlbridge[bot]]

Webrevs

• 02: Full - Incremental (9724c732)
• 01: Full - Incremental (337a6671)
• 00: Full (a7deed34)

[openjdk[bot]]

@tschatzl This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8253413: [REDO] [REDO] G1 incorrectly limiting young gen size when using the reserve can result in repeated full gcs

Reviewed-by: iwalulya, kbarrett

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 391 new commits pushed to the master branch:

• 3601e30df794db122d8d04fb3c04868ccbaa0baf: 8290909: MemoryPoolMBean/isUsageThresholdExceeded tests failed with "isUsageThresholdExceeded() returned false, and is still false, while threshold = MMMMMMM and used peak = NNNNNNN"
• 37c0a13647e74fd02823a3f621e986f96904b933: 8292350: Use static methods for hashCode/toString primitives
• 44532009fff11884aa6f16a997b771c41cb01d2f: 8292628: x86: Improve handling of constants in trigonometric stubs
• 07c9ba74fa3baebffcc15d3ee6ef941edf6be1a3: 8292686: runtime/cds/appcds/TestWithProfiler.java SIGSEGV in TableStatistics ctr
• 235151ead89f9102e3a57ba8f88807f180887866: 8292676: Remove two kerberos tests from problem list
• df5209e70fd92ec6bda4e7356a3ad121732f6c66: 8292683: Remove BadKeyUsageTest.java from Problem List
• 74d3330e106f2f920bf264356e4f25f8f6c11580: 8292682: Code change of JDK-8282730 not updated to reflect CSR update
• 57aac2ab6569c18a56e9a36f174bb0bf09955f83: 8290981: Enable CDS for zero builds
• 6a8a531359295849113aa14fd6cba21c886decf3: 8292564: x86: Remove redundant casts in Assembler usages
• 7244dd6fab0c516ed76af594593b8378512620c8: 8292267: Clean up synchronizer.hpp
• ... and 381 more: https://git.openjdk.org/jdk/compare/92067e200346c41c2f43763edc01c97c7da1a9e6...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

[tschatzl]

Thanks @walulyai @kimbarrett for your reviews.

/integrate

[openjdk[bot]]

Going to push as commit a3ec0bb03a5de805fc4b45477925aa18b875bc79. Since your change was applied there have been 395 commits pushed to the master branch:

• 27b0f7726b70127c0ed714cfc1041d3da71a9dc3: 8292318: Memory corruption in remove_dumptime_info
• 9a65524e2f98c1b4e253dcb637a708cec7b591bc: 8290300: Use standard String-joining tools where applicable
• f9004fe4438c30eb639e3c36e6531c306b836e36: 8292561: Make "ReplayCompiles" a diagnostic product switch
• 2fbb9362032df26582c389b7114cf0a215ed3afd: 8292691: Move CompilerConfig::is_xxx() inline functions out of compilerDefinitions.hpp
• 3601e30df794db122d8d04fb3c04868ccbaa0baf: 8290909: MemoryPoolMBean/isUsageThresholdExceeded tests failed with "isUsageThresholdExceeded() returned false, and is still false, while threshold = MMMMMMM and used peak = NNNNNNN"
• 37c0a13647e74fd02823a3f621e986f96904b933: 8292350: Use static methods for hashCode/toString primitives
• 44532009fff11884aa6f16a997b771c41cb01d2f: 8292628: x86: Improve handling of constants in trigonometric stubs
• 07c9ba74fa3baebffcc15d3ee6ef941edf6be1a3: 8292686: runtime/cds/appcds/TestWithProfiler.java SIGSEGV in TableStatistics ctr
• 235151ead89f9102e3a57ba8f88807f180887866: 8292676: Remove two kerberos tests from problem list
• df5209e70fd92ec6bda4e7356a3ad121732f6c66: 8292683: Remove BadKeyUsageTest.java from Problem List
• ... and 385 more: https://git.openjdk.org/jdk/compare/92067e200346c41c2f43763edc01c97c7da1a9e6...master

Your commit was automatically rebased without conflicts.

[openjdk[bot]]

@tschatzl Pushed as commit a3ec0bb03a5de805fc4b45477925aa18b875bc79.

:bulb: You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.