Slow artifact upload when saving to windows network share

[ksulli]

We are using GoCD 21.1.0 (12439-5a4959c7c4ede49165ec961b0219126cd5aa9e52) on a Windows server, running as a service under a domain account.

When our artifact store was on a local drive, artifact upload took about 2mins (for a 6GB artifact).

After changing the destination to use a network share, the time went up to 26 minutes, but just copying the file from the source to the destination using total commander still takes only one or two minutes (probably no problem with connectivity).

I tried looking for the reason this could be slow.

Using procmon from sysinternals, I noticed that the file is being written to the share in about 1kb chunks. I tried looking through the code, I found that ArtifactsService.saveFile() is using IOUtils.copyLarge() with the default buffer size, which should be 8KB https://commons.apache.org/proper/commons-io/apidocs/src-html/org/apache/commons/io/IOUtils.html

https://github.com/gocd/gocd/blob/4a1befaf5f2e6cb9604b2133041051747fda50ed/server/src/main/java/com/thoughtworks/go/server/service/ArtifactsService.java#L84

I thought about mapping the network drive to a drive letter, but MS very much doesn't recommend this.

Is there any way I could further diagnose or help fix this issue?

[chadlwilson]

In the older commons-io version we are using (2.6, due to bugs/regressions in all subsequent versions) the buffer seems to be 4kB rather than 8kB, but not sure that would make a huge difference (who knows though?) and not sure about the discrepancy with what sysinternals sees.

If you are in a position to test a Windows build in realish-world-conditions, I could try changing the buffer size and producing a test Windows installer with increased buffer size.

[ksulli]

Yes, I can do that and re-test. Edit: I ran the artifact upload again with procmon, and while the output buffering seems to vary, it might be actually the input buffering that is causing too busy communication. It looks like the 600MB zipped file is read in 512B chunks which then expand to whatever and are written to disk. Very ocassionally, the 512 zipped bytes expand to more than 4k and I can see a double-write.

So maybe its not going through the path I thought earlier, but here https://github.com/gocd/gocd/blob/4a1befaf5f2e6cb9604b2133041051747fda50ed/server/src/main/java/com/thoughtworks/go/server/service/ArtifactsService.java#L81

An interesting comparison is with the unzip(File zip, File destDir) method where the ZipInputStream is explicitly created from a BufferedInputStream https://github.com/gocd/gocd/blob/4a1befaf5f2e6cb9604b2133041051747fda50ed/base/src/main/java/com/thoughtworks/go/util/ZipUtil.java#L126

I haven't been able to trace how the ArtifactsService input stream is created, as far as I can tell it starts its life as a MultipartFile.getInputStream here, which may not be explicitly buffered? https://github.com/gocd/gocd/blob/4a1befaf5f2e6cb9604b2133041051747fda50ed/server/src/main/java/com/thoughtworks/go/server/controller/ArtifactsController.java#L212

[chadlwilson]

Thanks for this, this is great work.

Just to clarify - if I produce an experimental build do you have a way to test without upgrading a real production server? I meant 'similar network and storage speed conditions' rather than suggesting using an experimental build to upgrade a production server.

[ksulli]

Sorry for the delay, happy new year! It is really no problem for me to upgrade prod with an experimental build and test (esp. compared with the inconvenience of the 25 minute artifact upload). We use GoCD to automate builds and releases but we do not have a real continuous delivery environment set up, all our triggers are manual and we run the pipelines only a few times a week (also I can make a snapshot of the VM pre-upgrade). We do not have the capacity to have a "testing" GoCD server.

[chadlwilson]

There are experimental builds ( https://www.gocd.org/download > Experimental tab 23.1.0-15890+) with

• the buffer size increased 4kB to 8kB for the artifact copies
• input stream buffered when uploading a directory
• ability to configure buffer size via -Dartifact.copy.buffer.size=16384.

This build is likely to be close to what will released as 23.1.0 but definitely need to be careful if using in your main environment, especially as you are about 2 years behind in versions. Might be better to get to a released version of 22.3.0 and validate the problem still exists (but that your server is stable) rather than going straight to an experimental build. And do the norm of taking backups of your config and database so you can roll back if there are problems.

[ksulli]

We just got to install the build yesterday (23.1.0.-16068) after previously upgrading to 22.3, and in the first run after the upgrade we haven't seen much improvement, I will investigate further.

[chadlwilson]

Yeah, I had relatively low expectations that thie change in buffer size would fix the problem, but the changes at least are unlikely to have done any harm.

I guess it would help if you can do a similar analysis as earlier and see if it's at least writing chunks differently, and in bigger sizes.

I suppose the bigger question might be how, in Java-land, to replicate somrthing similar to what total commander (or windows itself) is doing when dealing with such large files.

[chadlwilson]

Did you try using the system property I added to increase further and see if it makes any difference in your case? Can try 16k, 32k etc.

Official 23.1.0 is out now btw, so can do so on a official build.

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 90 days. If you can still reproduce this error on the master branch using local development environment or on the latest GoCD Release, please reply with all of the information you have about it in order to keep the issue open. Thank you for all your contributions.

[chadlwilson]

Closing this now, as we seem stuck here and unable to determine what the problem is and whether it is something specific to GoCD and its buffers or just the nature of writing to Windows shares via Java. We can re-open this if further investigation can be done to track down the problem and trace it to GoCD.