io icon indicating copy to clipboard operation
io copied to clipboard

The JNA implementation seems unbarably slow on Windows (7)

Open spangaer opened this issue 6 years ago • 2 comments

When running things like sbt eclipse SBT munches through the local Ivy repository very aggressively. On Windows it used the JNA code for this

at sbt.internal.io.WinMilli$.getModifiedTimeNative(Milli.scala:271)

and it seems to be unbearably slow. Many of the file accesses take multiple seconds twice, once to figure out a symlink that needs to be resolved (repo is relocated via symlink) and once to access the actual file. (observed via procmon) Because these slow cases run in the hundreds (for certain projects; iterates multiple times too), it can take up to 15 minutes to munch through.

When enabling -Dsbt.io.jdktimestamps=true the same file access becomes sub millisecond. (I will re-iterate on Monday with a cold(er) system to verify).

This is a:

  • SBT 1.1.4
  • Intel i7 4th gen
  • Win 7
  • Spectre/Meltdown fixes enabled (given the milisecond story that might be of relevance)
  • Trend Micro AV and more anti-nasty protection

spangaer avatar Apr 13 '18 16:04 spangaer

Thanks for the report @spangaer

eed3si9n avatar Apr 14 '18 19:04 eed3si9n

Confirming improvement by system property flag.

  • post weekend run
  • post reboot run (force cold FS cache)
  • post eclipse start run

All have consistent improvement. It's not blazingly fast, because it generates 300k file related procmon entries for a single run. But only between 0 and 5 (across runs) are over 1 second instead of hundreds, the remainder is in the millisecond or sub-millisecond range and at least the CPU is doing something else then waiting for timestamp info.

spangaer avatar Apr 16 '18 07:04 spangaer