evaluate "Docker for Mac" as a Dinghy replacement

[codekitchen]

trafficstars

You have to agree to an NDA to get into the private beta, but I've gleaned plenty of information from public sources:

https://blog.docker.com/2016/03/docker-for-mac-windows-beta/ https://news.ycombinator.com/item?id=11352594

I've been saying for 18 months that Docker should solve the OS X dev problem themselves, so I'm a big fan of this. From what I've read so far it seems pretty close to Dinghy in terms of features.

• Mounted volumes are shared from the host using an as-yet-unknown solution that appears to support fsevents. Hopefully its performance is at least close to NFS, if not better.
• Claims that it remaps MacOS X UIDs into Linux ones (no more permissions problems), which could mean a couple different things. Hopefully it is aimed at editing code on a mounted volume.
• DNS resolving, apparently with routing the docker network interfaces directly to the host rather than through any proxy. I'd considered doing this too but never got around to it, it's a nice improvement over Dinghy's current solution. The HTTP proxy would still be nice for making SSL termination easy but that could be layered on top.
  • Not sure how DNS names will be chosen. I've come to really like the docker-compose based names but even if it's just <containername>.docker.local that'd be ok.
• Uses xhyve, which is about 10% slower than virtualbox for typical dev tasks in my testing. However, I've already switched to the dinghy xhyve driver myself because it's so much more pleasant than virtualbox for this use case.

Overall it sounds really nice. The proof will be once we can actually try it out, but if it works as well as they claim I'll probably just deprecate Dinghy once Docker for Mac is publicly released. That's assuming Docker for Mac is a free product, I'd hope it would be but they haven't said, and it sounds like at least parts of it will not be open source.

[codekitchen]

Looks like that's already been clarified, Docker for Mac will be free and open source.

https://news.ycombinator.com/item?id=11353747

[ryansch]

Agreed. This looks to be very exciting. I'm waiting for my invite :-)

[rkazak]

Me too!

[mrmachine]

I've been testing the beta today. The way it maps permissions for mounted volumes is different, and a little weird. It seems to map all bind mounted volumes to whatever UID:GID you are currently running as inside the container. So when you gosu foo, the owner and group of all your bind mounted volumes will change from root to foo.

In theory, this sounds like it should resolve the existing issue where you have to create/modify a user inside the container to match the UID:GID of the bind mounted volumes (which can't be changed). But I was previously working around that by doing gosu "$(stat -c '%u' "${DIR}"):$(stat -c '%g' "${DIR}")" which no longer works as expected.

So at the very least, this different behaviour has broken parity with images that work under regular Docker Engine.

Here's what their docs say. Sounds like this feature might be short-lived:

Presently, all requesting processes are treated as owners and group members on all bind mounted file system resources. This behavior simultaneously solves uid/gid mapping headaches and creates new compatibility (and possibly security) issues. A finer-grained ownership model is in development.

[codekitchen]

I tested out the beta for a bit yesterday, but ran into I/O errors when trying to bundle install onto a host-mounted volume. I haven't had a chance to dig further yet. Interestingly they're almost exactly the same git clone I/O errors I got when trying to run the unfsd NFS daemon as a non-root user. git seems to be especially sensitive to filesystem bugs.

[uberjay]

Unfortunately the Docker for Mac beta is significantly slower than dinghy for my workflow. I use the vmware driver -- between the osxfs driver burning significant CPU and the overhead of xhyve, a build which normally takes a little over 4 minutes ends up taking over 20 minutes.

I currently use vagrant for most of my development environment management, but jumped on the docker for mac beta in hopes it would simplify my workflow. And it does! However, the performance is bad enough that it's unusable for me. I have no doubt that xhyve performance will improve over time... eventually, it may be a good alternative. Dinghy to the rescue! :)

[codekitchen]

Yeah, I believe them when they say the FS performance will improve over time, they should be able to beat NFS performance using their architecture, with some work.

[michaellopez]

I've been playing with Docker for Mac the past few days. What I'm missing is the DNS functionality that Dinghy provides. I'm hoping that they include a custom DNS but looking at this thread it doesn't look like it...anytime soon anyways. Is there anyway to break out the DNS part of Dinghy into its own component that can be reused by Dinghy and standalone by Docker for Mac users?

[codekitchen]

Yep, the HTTP proxy and DNS server have been split into a separate project, install instructions are at https://github.com/codekitchen/dinghy-http-proxy#os-x

[michaellopez]

@codekitchen That's great. Thank you. I will give it a spin....I still find Docker for Mac unstable so I'll stick to Dinghy for the time being. 🍰

[cweagans]

osxfs still can't compete with nfs or even the vbox guest additions. It's so slow. IMO, Docker for Mac is not a complete replacement yet.

[igorvpcleao]

Hey @codekitchen, Sorry for a silly question. Is it possible to run dinghy with docker-beta?

Thx!

[michaellopez]

@igorvpcleao From what I understand, the only thing you'd want from Dinghy would be its proxy & DNS which are available here https://github.com/codekitchen/dinghy-http-proxy. One major aspect of Docker for mac is to make osxfs performant and stable. So the NFS and fsevents from Dinghy should not really be needed in Docker for mac.

So if/once https://forums.docker.com/t/custom-dns-for-resolving-containers/8361 gets fixed/merged then I think that Dinghy might be up for deprecation all together.

But until Docker for mac is stable (which it wasn't when I tried it last month) I'm sticking to Dinghy. Now I just received e-mail from Docker that Docker for mac/windows is RC. Might be worth testing it out again.

[igorvpcleao]

Thanks @michaellopez!

Good to hear that. I don't know why, but I'm still experiencing problems with shared folders. Solving this DNS issue will make the transference among host and container faster? It's currently taking a long time to render my rails pages when I use shared folders.

[codekitchen]

Yeah that's a great summary, thanks @michaellopez

One twist is that we've got devs using Linux laptops and running Docker directly on the host, and it's nice that the Dinghy HTTP proxy works the same way for them. So we may continue on using it anyway, even if Docker for Mac introduces its own solution there.

[michaellopez]

@igorvpcleao No, proxy & DNS is only for resolving virtual hosts on the host to a container ip and port. It does nothing with shared folders or osxfs in Docker for mac.

[mrmachine]

Until osxfs is as fast as nfs I think there will still be a need for dinghy. It would be good to be able to use docker for Mac with nfs until that happens. Will it be possible?

[codekitchen]

I don't currently have any plans to support that, I haven't even looked into whether it's possible to tell docker for mac to disable its own built-in filesystem so that it could use NFS instead.

[rmtsrc]

Since Mac Docker now has a public beta I thought I'd try rebenchmarking. Here's my results, I guess I'll stick with docker-osx-dev for now.

Mac benchmark (without Docker)

dd if=/dev/zero of=/tmp/output bs=8k count=100k; rm -f /tmp/output
102400+0 records in
102400+0 records out
838860800 bytes transferred in 1.132447 secs (740750602 bytes/sec)

docker-osx-dev (revision ea0617d)

dd if=/dev/zero of=/usr/src/app/output bs=8k count=100k; rm -f /usr/src/app/output
app_1  | 102400+0 records in
app_1  | 102400+0 records out
app_1  | 838860800 bytes (839 MB) copied, 0.261691 s, 3.2 GB/s

Dinghy Docker over NFS (4.4.1)

dd if=/dev/zero of=/usr/src/app/output bs=8k count=100k; rm -f /usr/src/app/output
app_1  | 102400+0 records in
app_1  | 102400+0 records out
app_1  | 838860800 bytes (839 MB) copied, 26.5691 s, 31.6 MB/s

Mac Docker Public Beta (1.12.0-rc2-beta17 build: 9779)

dd if=/dev/zero of=/usr/src/app/output bs=8k count=100k; rm -f /usr/src/app/output
app_1  | 102400+0 records in
app_1  | 102400+0 records out
app_1  | 838860800 bytes (839 MB) copied, 34.3933 s, 24.4 MB/s

[mickaelperrin]

Thanks for the benchmark.

How would you explain the fact that dockerosx-dev is faster than native ?

If I clearly understood, you ran that commands inside your containers ? and /usr/src/app/output is mounted on the host ? So this is a benchmark only on writing performance ?

Could you provide the same benchmark but for reading performance ?

Thanks

[codekitchen]

As I understand it your docker-osx-dev benchmark is meaningless, because docker-osx-dev only does one-way syncing from the OS X host to the VM, not the other way around. So you're simply measuring the VM write performance. That's the primary reason we chose NFS for Dinghy, rather than rsync, because we have containers that we need to write to shares and persist them back to the OS X host.

[EugenMayer]

little update on this, the current docker-for-mac releases are not longer invite only, you do no longer need to enter a private beta key and therefore no NDA/Terms acceptance needed. This probably helps you.

I am very surprised to see that the rc2 does perform that well against NFS - most of the people in this thread, me included, are not using osxfs since its is absolutely to slow for everything: https://forums.docker.com/t/file-access-in-mounted-volumes-extremely-slow-cpu-bound/8076/109

I am here, because i am considering switching from docker for mac with rsync ( custom solution ) to dinghy. Eventhough rsync as the solution lets you run the containers in good speed, the sync is plain PITA in the end. On the paper, its working - but:

• fswatcher are worse then the shares on OSX
• fswacher ( ruby wrapper or the c implementation from brew ) needs about 80% of my CPU when watching my dev folder. This scales with then number of file, feels linear
• due to the way the events are propageted, there is no "bundling or grouping" this means, lets say you trigger rsync on a fswatcher event. No consider you switch a branch in SCM - you happen to have about 400 or more fswatcher events - each tries to sync all the data. This is VERY slow due to the connection overhead
• fswatcher regulary forgets to fire events - it is not relyable or convenient to use

So in the end i am about to give up with rsync ( being using it with vboxn, then fusion now docker for mac ) and trying to think about a NFS solution.

In general, anything except file sharing is pretty cool in docker for mac, so all i want is docker for mac with NFS most probably. But if the performance is as bad as up there ( close to osxfs ) i would rahter not use it.

Any real world examples?

[michaellopez]

@EugenMayer I'm not sure if you are looking for "testimonals" for Dinghy...But we are using Dinghy in our daily workflow at my company. I can't see how working with Docker on a mac can be done without it at the moment. We use modern Macbook Pros and so far no performance issue. Watchers are inside the container, Dinghy forwards file system events on OSX into the container. I can't say that we have very large trees being watched, but I am very meticulous about our builds, just watching what really needs watching. I often see tool.watch('**'), which of course will not perform well with a large tree. I'd be happy to answer any questions, or run something on our setup given that you provide something to run :)

[EugenMayer]

You run the watcher to compile/generate any kind of assets in the container right? If you lets say change ruby files, you would not need watchers for that, those are synced using NFS, right?

Even without any node_modules/bower/coffescript/scss files, only watching for js/css/code we end up haing arround 12k files. This does, for sure, kill fswatcher + rsync but does totaly defeat osxfs ( we are talking about a factor far above 100 slower then rsync/native)

Am i right that NFS read performance is considerable bette then its write performance? Most of the people in the linked thread, when using synthetic benchmarks, use write benchmarks. I consider those benchmarks useless. No data generated ( written ) in the VM should ever happen to be saved on the code-base. Those data either stays in the container volume, or if to be persisted, on volumes whatever.

osxfs is pretty slow in reading also, so thats were i would need compare it to NFS and then NFS to native ( native is what i have right now ). When running a PHP cli up, parsing about 600 PHP files to bootstrap the app, reading performance does matter a lot :)

In case, when we have them all here: Docker 4 Mac, dinghy - what about dlite. How does dinghy compare to dlite?

[michaellopez]

@EugenMayer Yes, we use watchers inside the container to do things like recompile on save. The files are synced with NFS, but NFS does NOT trigger file system events inside the container. That's why Dinghy uses https://github.com/codekitchen/fsevents_to_vm, it reacts to file system events on OSX and sends a packet to the VM that trigger file system events for the container.

I honestly haven't worked on projects where I need to watch 12k files. I can't understand how it comes to that, but I respect that fact that those kinds of repositories do exist and needs to be worked with.

I do not have any source on NFS performance at hand, but I have a vague memory of reading somewhere that write performance better than read, yes.

If only the "checkout another branch" scenario, or similar, is the only case where the watchers poses a problem (meaning, commands that run seldom) and if you are using Dinghy, then maybe you can kill the fsevents_to_vm process, do the checkout and restart it again with dinghy up (note: untested :)

I have no experience with dlite.

[EugenMayer]

I am not happy with this amount of files - but you cant change it. Though, its at least not only repo :)

dlite 1.x seems to do less, which fits my needs a bit more as its seems. It does not support switching the provider, which is nice on dinghy side, though i use xhyve anyway.

dinghy provides dnsmasq and used with xhyve that colides with my boxen-stack dnsmasq. No needs for proxies (at least i can disable that).

It seems like dlite 2 and dinghy seem to be very similar - but i guess i am hijacking this topic.

Coming back to the performance, both using NFS while dlite uses the OSX build in server. Not sure if this harms or is better, but that could be different in performance terms ( either way )

I guess i try both of them compared against docker4mac+rsync and report back on my "real world test"

[EugenMayer]

Made some tests with dinghy and docker 4 mac:

dinghy

NFS drush cc all  7.97s user 1.40s system 28% cpu 32.817 total

NO SHARE drush cc all 7.52s user 0.79s system 75% cpu 10.993 total

Docker for Mac

OSXFS drush cc all 8.42s user 4.06s system 24% cpu 51.265 total

NO SHARE drush cc all 7.02s user 0.72s system 77% cpu 10.005 total

---

So NFS vs OSXFS is about 1.5 times slower, but in seconds, thats quiet some bit: 32s vs 51s

Bot compared to no share with 10s is 3-5 times slower. But since adds on each command this is for us, unusable.

Find: The import of the DB dump was a lot faster on Docker 4 mac "no share", so the I/O performance in terms of write is a lot better

---

So overall, the rsync approach is a lot better in general. Syncing all my 12k files takes 3 seconds, differentials sync, if triggered "grouped" takes less then 0.5s - no matter what there is. Running the app i up to 5 times faster.

[cweagans]

As an aside for anyone that's not familiar with drush: the commands above are "bootstrapping" Drupal (loading and interpreting all the PHP files), connecting to the database, and then clearing or invalidating all of the caches. That could be any combination of cache tables in the db, redis, memcache, etc. That process is a pretty good benchmark for how Drupal will perform under average conditions.

[codekitchen]

Can we keep this issue discussion to the original purpose please? This side discussion is interesting but would be a lot better in a separate issue.

[EugenMayer]

Created https://github.com/EugenMayer/docker_sync as a result of all the tests, including the tests with dinghy. It has docker-for-mac support ( and docker-machine and all others ). Might be a way for dinghy also ( since dinghy is more, dns..proxy..) .. but it is rsync based, not NFS based, due to the performance issues with NFS.

When you plan to switch to docker-for-mac you might want to look at that gem, probably out-sourcing the sync part and keeping anything else like the http-proxy/dnsmasq in dinghy.

Update: I added unison support as a strategy, so this should be a proper replacement for the NFS based sync. Rsync is supported and the default, people can chose per sync-endpoint