Listeners are not cleaned up immediately when calling EventBus#unregister
ListenerList#forceRebuild simply sets a boolean to true, and relies on the next event post to do the actual cleanup:
https://github.com/MinecraftForge/EventBus/blob/master/src/main/java/net/minecraftforge/eventbus/ListenerList.java#L223-L231
This means that if unregister is called and the event is never fired again (such as for startup events) the event listener will not be released from memory.
Additionally, there's no way to do this manually even though getListeners is public, since it requires the bus ID which is not exposed.
Doing otherwise can result in a race condition sadly.
Could we at least expose a way to do it manually in situations where it will definitely be safe?
As a work around, you can use Java WeakReferences to avoid the listeners preventing large objects from being garbage collected. Check out the PR linked above for an example.