[macOS] hdiutil failures when creating DMGs

[wkiefer]

Description

We use https://github.com/create-dmg/create-dmg as a small script wrapper around hdiutil to create and notarize DMG files from our macOS applications.

However, when switching to the macos-13 runner, we get the following error when calling hdiutil create

hdiutil: create: returning 49168
hdiutil: create failed - Resource busy

Platforms affected

• [ ] Azure DevOps
• [x] GitHub Actions - Standard Runners
• [ ] GitHub Actions - Larger Runners

Runner images affected

• [ ] Ubuntu 20.04
• [ ] Ubuntu 22.04
• [ ] macOS 11
• [ ] macOS 12
• [x] macOS 13
• [ ] Windows Server 2019
• [ ] Windows Server 2022

Image version and build link

Image: macos-13
Version: 20230426.3
Included Software: https://github.com/actions/runner-images/blob/macOS-13/20230426.3/images/macos/macos-13-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/macOS-13%2F20230426.3

Is it regression?

https://github.com/actions/runner-images/releases/tag/macOS-12%2F20230425.1

Expected behavior

hdiutil create should not return error 49168, resource busy

Actual behavior

hdiutil create consistently returns error 49168, resource busy

Repro steps

I will try to make a public repo sample repro — right now steps are to build with Xcode 14.3 on the macOS 13 image, then run

create-dmg \
  --app-drop-link 240 332 \
  --icon "Sample.app" 240 108 \
  --icon-size 128 \
  --volname "Sample" \
  --window-pos 100 100 \
  --window-size 480 582 \
  --codesign "Your Developer ID Application" \
  "/output/path/Sample.dmg" \
  "/input/folderWithApp/"

Or simply try to run any hdiutil create command in the context of a macOS 13 runner

[Alexey-Ayupov]

Hello @wkiefer, we will take a look, however, please remember macOS 13 is still in beta version.

[vpolikarpov-akvelon]

Hi @wkiefer. I didn't succeed reproducing the issue. Both hdiutil and create-dmg work flawlessly for me. Could you provide more detailed repro steps or share a link to the failing workflow?

[geoffthemedio]

I think we have hit the same issue. https://github.com/freeorion/freeorion/tree/MacOS_action_versions https://github.com/freeorion/freeorion/actions/runs/4902807369/jobs/8755515334?pr=4509#step:9:426

CPack Error: Error executing: /usr/bin/hdiutil create -ov -srcfolder "/Users/runner/work/freeorion/freeorion/build/_CPack_Packages/MacOSX/DragNDrop/FreeOrion_2023-05-06.ba56d96_Test_MacOSX_10.15" -volname "FreeOrion" -fs "HFS+" -format UDZO "/Users/runner/work/freeorion/freeorion/build/_CPack_Packages/MacOSX/DragNDrop/temp.dmg"
CPack Error: Error generating temporary disk image.
hdiutil: create failed - Resource busy

CPack Error: Problem compressing the directory
CPack Error: Error when generating package: FreeOrion
Command PhaseScriptExecution failed with a nonzero exit code

** BUILD FAILED **


The following build commands failed:
	PhaseScriptExecution CMake\ PostBuild\ Rules /Users/runner/work/freeorion/freeorion/build/build/FreeOrion.build/Release/package.build/Script-571234464C2A5D0D43F1AB42.sh (in target 'package' from project 'FreeOrion')

[vpolikarpov-akvelon]

Hey @geoffthemedio, @wkiefer. Looks like the issue is gone as an image version 20230509.4 have been rolled out. Could you try to run your workflow again?

[geoffthemedio]

I reran the action and it completed successfully. I'm not sure that nothing else changed and that it's doing the exact same thing, but at least I can reproduce the issue now.

[Neverous]

I think I'm hitting similar issues or at least issues with hdiutil: hdiutil: create failed - No child processes and the newest image didn't help really :cry:

Runner image:

  Image: macos-13
  Version: 20230509.4
  Included Software: https://github.com/actions/runner-images/blob/macOS-13/20230509.4/images/macos/macos-13-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/macOS-13%2F20230509.4

[vpolikarpov-akvelon]

Hi @Neverous. It looks similar indeed. But I suppose you might get this error because of the failure on the previous step:

error: /Applications/Xcode_14.2.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: no LC_RPATH load command with path: /Users/runner/work/efibooteditor/Qt/6.5.0/macos/lib found in: /Users/runner/work/efibooteditor/efibooteditor/build/dist/_CPack_Packages/Darwin/DragNDrop/EFIBootEditor-v0.0.0-3f25f4b38b30f8575f17628808319e6017e44bc7-macos-13-qt-6.5.0/dist/efibooteditor.app/Contents/MacOS/efibooteditor (for architecture x86_64), required for specified option "-delete_rpath /Users/runner/work/efibooteditor/Qt/6.5.0/macos/lib"

Could you ensure that it doesn't affect packaging?

[Neverous]

I get the same thing in macos 12: https://github.com/Neverous/efibooteditor/actions/runs/4908677336/jobs/8925056030#step:12:232 and 11: https://github.com/Neverous/efibooteditor/actions/runs/4908677336/jobs/8925056236#step:12:232 and they go through fine so I would guess that it doesn't, but Ill see If I can do anything about it maybe it will help.

[Neverous]

Yeah, the rpath error is just that both macdeployqt and CPack generator that I use to make the final package are trying to adjust rpaths, and both have no option to skip it from what I can tell... I think its harmless, at least for hdiutil.

Anyway, I got to very simple reproduction steps I think: https://github.com/Neverous/efibooteditor/actions/runs/4997753556/jobs/8952437874 (the issue seems to be somewhat random as just restarting the job sometimes succeeds and sometimes not :thinking: https://github.com/Neverous/efibooteditor/actions/runs/4997791581)

[vpolikarpov-akvelon]

Hey. I finally realized what causes this issue. Looks like it's XProtectBehaviorService that was introduced in macOS 13 Ventura. If you are not lucky enough it may lock your newly created dmg and hdiutil will fail. I can suggest implementing retry in you workflow to workaround this.

Another option may be shutting aforementioned service down (or just killing the process). You may use something like this:

echo killing...; sudo pkill -9 XProtect >/dev/null || true;
echo waiting...; while pgrep XProtect; do sleep 3; done;

I can't recommend doing that because it may cause issues in the future that we can't predict and avoid. Also it may have security related consequences. Use it on your own risk.

[Neverous]

Tried both, neither works for me, I must be hitting something slightly different then :thinking: (hdiutil: create failed - No child processes seems not to be so random like the Resource busy one, but so far I wasn't able to create simple reproduction steps than running the whole build)

retrying 20 times killing XProtect

[vpolikarpov-akvelon]

You a right, Maciej, this is something different. Killing XProtect only helps when you get "Resource busy". I still suspect that previous error may somehow cause failure of hdiutil. I understand that it didn't matter on macOS 11 & 12, but something else may have changed so now it's important. Could you try to get rid of "install_name_tool" error? Just to be sure that it's really nothing.

[Neverous]

Ah should've mentioned that with macdeployqt vs CPack comment, did that (disabled macdeployqt for tests, leaving only cpack dmg generation): https://github.com/Neverous/efibooteditor/commit/e7ec720c531000d1ea36a486c269463f798e23e9#diff-1e7de1ae2d059d21e1dd75d5812d5a34b0222cef273b7c3a2af62eb747f9d20aL410 , https://github.com/Neverous/efibooteditor/actions/runs/4997683758/jobs/8952293026#step:12:234

[vpolikarpov-akvelon]

Did you try enabling debug output for hdiutil? I suppose it may be done by setting CPACK_COMMAND_HDIUTIL option to /path/to/hdiutil -debug

[Neverous]

Don't see anything interesting unfortunately, just that diskimages-helper died :disappointed: logs

2023-05-25 17:08:25.255 hdiutil[11275:37357] [DIHelperProxy watchForHelperDeath] helper exited early
2023-05-25 17:08:25.255 hdiutil[11275:37357] helper died
2023-05-25 17:08:25.255 hdiutil[11275:37357] setHelperDoneWithResult: _helperDone = YES, _threadResultsError = 10

[vpolikarpov-akvelon]

Hey @Neverous. Looks like your issue may be resolved by running hdiutil using sudo.

I added this line to CMakeLists.txt and packaging works flawlessly:

set(CPACK_COMMAND_HDIUTIL "/usr/bin/sudo /usr/bin/hdiutil")

[Neverous]

Yes, thank you! It works indeed. Interesting, some permissions changed with macOS 13?, not sure why it would now require elevated access :thinking: but it packages successfully now with sudo :bow: .

[vpolikarpov-akvelon]

Glad to hear! Well, looks like problems have been solved so i'm closing this issue for now. Feel free to reopen it if problems return.

To sum up:

• If hdiutil create fails on macOS 13 with No child processes then try running it with sudo;
• If it fails on macOS 13 with Resource busy sometimes then try killing XProtectBehaviorService.

[grzegorzkrukowski]

This is still a case on latest macOS13 runners - it's happening 80% of the time and non of mentioned workaround here is helping

[PatTheMav]

I don't agree that this issue is actually resolved given that XProtect seems to (seemingly at random) lock down created disk images on CI runners. Killing the process was first "not recommended" but has seemingly been elevated to the accepted solution for that issue.

With macOS 14' release happening soon I would expect a lot of developers racing to have updates ready for it which might exacerbate this issue (i.e. if the volume of unseen application bundles and disk images might trigger XProtect more often).

[jpfeuffer]

We have the same issue and killing a security relevant system process is definitely not a valid solution!

[jpfeuffer]

https://github.com/OpenMS/OpenMS/actions/runs/5931058665/job/16082111148

[vpolikarpov-akvelon]

Hey @jpfeuffer. Unfortunately, it is the only solution we have. Mac OS doesn't support automation well. The process we are talking about wouldn't bother you probably if you build your software by hand, but when you configure automation, process becomes fast enough for you to start observing this and similar phenomena. To workaround this you may either kill process that malicious process, configure retries or pauses between steps.

[jpfeuffer]

The problem is that your solutions are not working solutions: /path/to/hdiutil -debug does not work because hdiutil expects the "action" first. So no debugging inside CPack which can only prepend arguments. sudo does not help. And neither does the killing of the process before running CTest/CPack.

See https://github.com/OpenMS/OpenMS/actions/runs/5938356857/job/16102624563

[jcelerier]

also seeing this in azure pipelines more and more: https://dev.azure.com/ossia/libossia/_build/results?buildId=3542&view=logs&j=7bab896a-24f8-544f-51eb-43745367a332&t=0a75dee1-c6f9-5e3c-8b1c-a50cc807cb5c

[tsteven4]

we have been having the same issue. It is intermittent but not rare. In our case hdiutil is being called from Qt's macdeployqt. A recent failures is https://github.com/GPSBabel/gpsbabel/actions/runs/7251472805/attempts/1

ERROR: Bundle creation error: "hdiutil: create failed - Resource busy\n"

[tsteven4]

our failures are on macos12.

[apparentsoft]

I'm getting this intermittently as well. This is for a rather large DMG, of more than 150MB

[fwcd]

Linking the upstream issue for completeness: https://gitlab.kitware.com/cmake/cmake/-/issues/25671

[echlebek]

I think this issue should be re-opened. Observed this happening in this CI run: https://github.com/SumoLogic/sumologic-otel-collector/actions/runs/9415385897/job/25936308280

It's good that there is a workaround, but I don't think it's a long-term resolution for this issue.