8319876: Reduce memory consumption of VM_ThreadDump::doit

[elifaslan1]

trafficstars

This pull request contains a backport of commit 8ec6b8de3bb3d7aeebdcb45d761b18cce3bab75e from the openjdk/jdk repository to JDK 17.

This backport significantly reduces RSS memory usage during ThreadMXBean.dumpAllThreads(boolean, boolean) execution.

This is not a clean backport due to differences between JDK 17 and upstream:

Threading Context: Changed ResourceMark rm(VMThread::vm_thread()) to ResourceMark rm; • Similar to JDK 21 backport

Minor conflict resolved: _threads = nullptr; → _threads = NULL; to match JDK 17 conventions

GHA tested additionally ran tier1 tests on my local mac and passed.

---

Progress

• [x] Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• [x] Change must not contain extraneous whitespace
• [x] Commit message must refer to an issue
• [ ] JDK-8319876 needs maintainer approval

Issue

• JDK-8319876: Reduce memory consumption of VM_ThreadDump::doit (Enhancement - P4)

Reviewers

• Paul Hohensee (@phohensee - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk17u-dev.git pull/4006/head:pull/4006
$ git checkout pull/4006

Update a local copy of the PR:
$ git checkout pull/4006
$ git pull https://git.openjdk.org/jdk17u-dev.git pull/4006/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 4006

View PR using the GUI difftool:
$ git pr show -t 4006

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk17u-dev/pull/4006.diff

Using Webrev

Link to Webrev Comment