vscode-perforce
vscode-perforce copied to clipboard
stef-levesque
Bug: Too many files changing externally triggers too many p4 sync subprocesses
I'm using this extension on a project with ~30K ish files.
If I perform a large merge/resolve externally or anything that modifies thousands of files at once it will cause the extension to spawn several thousand subprocesses, mostly doing p4 sync, all at once sending my system into an unrecoverable death spiral. There needs to be some kind of throttle for this. Is this something I can disable?
The editOnFileSave/editOnFileCreate functionality is great, but needing to always remember to close VS Code before changing many files in the project or experience virtual memory death is painful.
Yeah having a configurable limit to restrict the number of calls makes a lot of sense. Disabling the various settings (edit on file save/create/delete/modify) should prevent this from occurring until a fix can be released.
Pretty slammed at work, but will see if I can find some time to take a look at this.
+1
I am experiencing the same exact problem at work. We have a huge perforce project and if I forget to close VS Code while doing an external sync, it causes large load spikes on the perforce servers and our admins reach out to me asking what I'm doing. This plugin is critical to how I work, but I have been asked to disable the plugin due to the slowness that it is causing on the p4 servers when this issue occurs.
FWIW, I only use this plugin for the editOnFileSave/editOnFileCreate behavior, as well as the annotations in the sidebars. I do not use it to browse change lists or sync files.
I appreciate any time that you can find for this issue. Thank you!
I've also been experiencing this at work. VSCode will sometimes spawn thousands of subprocesses when running various build commands across our large code base. Like @jcw5002 I primarily use this for checking out or adding new files.
+1 I also experience this issue at work. Any attention you can devote to this issue would be much appreciated. Thank you for making a great plugin that's very useful!
@stef-levesque I've been experiencing the same issues as the folks above. I took a stab at an approach where the perforce commands executed are limited via a library called bottleneck. I haven't had a chance to create any tests or do any manual testing to see if the configuration I have provided is sufficient. But, in the very least, it limits the maximum number currently running perforce commands to 50. It is a queue that drops the oldest command when that maximum has been reached.
https://github.com/stef-levesque/vscode-perforce/pull/147
Thanks @pjdanfor I'll make a pre-release build with your fix in, so people here can test the feature.
I tried to expose the bottleneck parameters the best I could in 7c5339f7f40c7c0154d82018789853c3935e9489
I also updated the pre-release build available here: https://github.com/stef-levesque/vscode-perforce/releases
You can download and install the .vsix file via the VSCode command palette:
Extensions: Install from VSIX...
Please let us know if this help mitigate the issues, and if you tried tweaking any of the bottleneck parameters:
I had this problem too, but I don't think it's always related to external changes - it just happens periodically. Maybe I'll try the prerelease build since those new settings look useful; however, they could use a little more explanation (what's the unit on min time - seconds, milliseconds, something else? what does the penalty actually do? how do the strategies work?).
@stef-levesque what is the status of this? Seems like the changes were merged into master but have not been pushed out in a new release - When are you planning on releasing the latest code?
Sorry, I haven't been a good maintainer as of lately.
I should push a new version by the end of the week.
bottleneck can be configured in by tweaking the values in perforce.bottleneck.
The extension version 3.2.0 should be available soon.
So it's great that you did something about this! However, it's less helpful than I'd hoped. At least I can reduce "max concurrent" which is pretty clearly documented, so hopefully that alone will be enough to improve the situation, but the sheer number of options you've added, coupled with the poor documentation, makes it next to impossible to choose good settings.
The descriptions for "high water" and "max concurrent" are sufficient, I would say. The description for "min time" makes sense, but doesn't say what unit the time is given in (based on the default I would guess seconds, but it should really be mentioned explicitly). The "penalty" description gives no indication of what the penalty does, but that might not be important given the final point.
There is zero indication of what the different strategies do, how they work, why you might want to pick one over another, etc. If you feel it's necessary to allow the user to choose a strategy, you need to improve the description to let them know what the difference is between the strategies. Personally, I think this is too many options; I'd prefer to see you pick a strategy and stick with it.
(It looks like the "edit on save" and "edit on modify" functionality also no longer works... maybe because the actions are queued but the queue is already really long and taking forever to get through? I have no idea how it works, but anyway, I might have to revert to the previous version if I can't get that working.)
@CelticMinstrel you are right, this feature is not meant for the lambda user.
I exposed the Bottleneck parameters for power users, to help debug and propose solutions for their particular cases.
With enough collaboration from the community, we might end up with settings generic enough for most use cases.
I didn't documented those parameters in detail since, as I mention, it is meant power user. And the official documentation for Bottleneck can be found on their repo.
I'll reopen the issue, since the slowdown is not fix for your use case.
A link to their repo would be handy to have, then (probably can find it by searching but it's faster if there's a link).
https://github.com/SGrondin/bottleneck/blob/master/README.md
See the Construction and Strategies sections.
It would also help to know which version is the P4 server, how many files you typically have in your change lists, how much is being shelved, do you auto detect add/edit, ...
The Perforce server version seems to be "2018.1/1687724", and I typically have few files in the default changelist (probably less than 20), and perhaps a few in other pending changelists. (I'm not counting shelved files there, of which I have quite a few; most of the pending changelists have only shelved files.) I have edit on modify and edit on save enabled, and auto-detection of adds; however, it seems to no longer work with the new bottleneck settings, so for the time being I'm just checking out everything. I've also noticed that the source control icon on the sidebar now permanently has the wait icon on it, but no p4 tasks are being spawned.