Reduce CGO dependency

[tomberek]

CGO tends to make Go project hard to reuse in interesting ways. Static compilation and cross compilation become hard. There is a PoC here https://github.com/tomberek/desync/tree/tom/no_cgo that rips out various features to allow a CGO_ENABLED=0 build to succeed. Very minimal testing was done.

Things done:

• Replace zstd with https://github.com/klauspost/compress/tree/master/zstd#zstd
• remove sftp related things
• remove checks for windows. (needs cgo? perhaps easier way?)
• remove fuse and mount-index
• remove progressbar
• modify untar.go (makeDevice is problematic)

Tested:

• builds without CGO
• static build (very limited functionality testsing)
• wasm cross compile (runs in browser, have not tested functionality)

[folbricht]

• It would be nice if we could switch to a native zstd implementation. https://github.com/klauspost/compress/tree/master/zstd#zstd appears to be in beta at the moment, but may become a viable alternative.

• Regarding the windows bit, that is there to find the home directory. Not sure why it needs cgo, but Go 1.13 introduces https://tip.golang.org/pkg/os/#UserConfigDir which may be an alternative.

• The other bits can likely be solved with build flags.

[Joe-Improbable]

@folbricht how do you feel about using the above library now that it's stable?

I can take a look at integrating it if you're happy with the library

[folbricht]

My only concern was the potential performance impact, but it's worth a try. Should be quite straightforward to try out. Perhaps with a build flag to still be able to use the C library where every bit of performance matters? Feel free to give it a try. Should just be a change to https://github.com/folbricht/desync/blob/master/compress.go to switch over. I'm interested seeing the difference in performance, but won't be able to get to it before the weekend.

[folbricht]

@Joe-Improbable https://github.com/folbricht/desync/pull/154 uses the native zstd library if the build tag klauspost is used, build it with go build -tags klauspost.

My initial tests are promising. Didn't notice any difference in overall performance. Please let me know if you see anything wrong with this. Going to make that the default once it's soaked a bit.

[Joe-Improbable]

Hey @folbricht, sorry I forgot to get back to you with my results.

Both tests were done on a 20 GB folder, which compresses to around 11 GB

OSX:

Library	Tarring	Untarring
ZSTD	real 4m25.330s user 7m32.600s sys 2m50.358s	real 1m18.001s user 2m2.762s sys 1m33.829s
Compress (ZSTD)	real 4m37.778s user 11m7.084s sys 2m57.361s	real 1m25.092s user 3m0.346s sys 1m34.232s

Windows:

Library	Tarring	Untarring
ZSTD	7.25 mins	2.14 mins
Compress (ZSTD)	9.28 mins	3.26 mins

So it looks likes there's no difference with the native library on OSX (though it uses more CPU time ), however it's slightly slower on Windows.

[folbricht]

The native Go implementation of zstd is now the default in https://github.com/folbricht/desync/pull/155.