Replacing SecurityManager with a tracking agent

[jlahoda]

This is a prototype of an attempt to replace the deprecated SecurityManager with a custom agent-based hooks.

What functionality is kept: -preventing inadvertent System.exit -tracking file I/O (probably the most complex part, might be missing some hooks) -checking for deprecated system property access -checking calls to setAccessible to open Unsafe (this didn't work on JDK 8+, but can be fixed) -checking/preventing calls to System.setSecurityManager. Possibly no longer that critical, as it is not possible to disable the hooks using this method. -tracking new Window creation (used by internal execution)

What functionality is not kept: -checkConnect for connecting over network. This didn't seem to be actually used, as the original code was:

        try {
            checkPermission(allPermission);
            return;
        } catch (SecurityException e) {
        }
//proceed with the actual check

But checkPermission seems to always(?) pass, so the check was never performed -checkLogger, which was disabled anyway -tweaks related to installation of a custom Swing clipboard - these appear to exists for JDK-4818143, which is marked as fixed, so presumably not needed anymore.

When looking at the patch, TrackingAgent is the agent class that does various transformations to install hooks, TrackingHooks are the callback, TopSecurityManager (o.n.bootstrap), FileChangedManager (masterfs), SecMan (core.execution) are the actual uses of the hooks.

[mbien]

this is pretty cool!

[jlahoda]

I've updated the patch to reflect most of the comments (tracking agent and hooks are now modules, SPI is separate from the callback, etc.) I've put a comment on conversation where I think it is difficult to implement the requested change.

[jlahoda]

I wonder if there are any further comments on this. It might be reasonable to try to avoid too much dependency on the SecurityManager.

[matthiasblaesing]

Given that we are early in the NB16 cycle, now would be a good time to merge this if ready.

[mbien]

all green. excellent!

[mbien]

JEP 451: Prepare to Disallow the Dynamic Loading of Agents will require the -XX:+EnableDynamicAgentLoading flag (sooner or later, will print warnings first).

It might be better to load the agent using JVM flags on startup (instead of dynamically), since there are probably good reasons JEPs like that exist.

[neilcsmith-net]

@mbien isn't it already doing this - https://github.com/apache/netbeans/pull/3386/files#diff-d4b605d5fdbc82d1afa51dc1d6c3ce44516cf46c5e2ac0c79a98118bdcefb896R195

Or are there other uses in there? I assume we'll need a Windows launcher update too? Wish that could be a script too!