homebrew-core icon indicating copy to clipboard operation
homebrew-core copied to clipboard

swift 5.7

Open Bo98 opened this issue 3 years ago • 7 comments

Created by brew update-swift. 🚀


Created with brew bump-formula-pr.

resource blocks may require updates.

Bo98 avatar Sep 13 '22 04:09 Bo98

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. To keep this pull request open, add a help wanted or in progress label.

github-actions[bot] avatar Sep 15 '22 13:09 github-actions[bot]

@Bo98 it's failing on Linux because we need to pass -fPIC to libCyaml when it is being built:

FAILED: lib/libYams.so Sources/Yams/CMakeFiles/Yams.dir/Constructor.swift.o Sources/Yams/CMakeFiles/Yams.dir/Decoder.swift.o Sources/Yams/CMakeFiles/Yams.dir/Emitter.swift.o Sources/Yams/CMakeFiles/Yams.dir/Encoder.swift.o Sources/Yams/CMakeFiles/Yams.dir/Mark.swift.o Sources/Yams/CMakeFiles/Yams.dir/Node.Mapping.swift.o Sources/Yams/CMakeFiles/Yams.dir/Node.Scalar.swift.o Sources/Yams/CMakeFiles/Yams.dir/Node.Sequence.swift.o Sources/Yams/CMakeFiles/Yams.dir/Node.swift.o Sources/Yams/CMakeFiles/Yams.dir/Parser.swift.o Sources/Yams/CMakeFiles/Yams.dir/Representer.swift.o Sources/Yams/CMakeFiles/Yams.dir/Resolver.swift.o Sources/Yams/CMakeFiles/Yams.dir/String+Yams.swift.o Sources/Yams/CMakeFiles/Yams.dir/Tag.swift.o Sources/Yams/CMakeFiles/Yams.dir/YamlError.swift.o swift/Yams.swiftmodule 
2022-09-18T05:03:20.5644136Z : && /home/linuxbrew/.linuxbrew/Cellar/swift/5.7/libexec/bin/swiftc -output-file-map Sources/Yams/CMakeFiles/Yams.dir/Release/output-file-map.json -incremental -j 2 -emit-library -o lib/libYams.so -module-name Yams -module-link-name Yams -emit-module -emit-module-path swift/Yams.swiftmodule -emit-dependencies -DSWIFT_PACKAGE -DYams_EXPORTS -module-cache-path "/tmp/swift-20220917-16921-s77h9l/build/swiftpm-linux-x86_64/x86_64-unknown-linux-gnu/yams/module-cache" -O -enable-testing -Xcc -DYAML_DECLARE_STATIC -Xcc -I/tmp/swift-20220917-16921-s77h9l/yams/Sources/CYaml/include -Xcc -F/tmp/swift-20220917-16921-s77h9l/build/foundation-linux-x86_64 -I /tmp/swift-20220917-16921-s77h9l/build/foundation-linux-x86_64/swift /tmp/swift-20220917-16921-s77h9l/yams/Sources/Yams/Constructor.swift /tmp/swift-20220917-16921-s77h9l/yams/Sources/Yams/Decoder.swift /tmp/swift-20220917-16921-s77h9l/yams/Sources/Yams/Emitter.swift /tmp/swift-20220917-16921-s77h9l/yams/Sources/Yams/Encoder.swift /tmp/swift-20220917-16921-s77h9l/yams/Sources/Yams/Mark.swift /tmp/swift-20220917-16921-s77h9l/yams/Sources/Yams/Node.Mapping.swift /tmp/swift-20220917-16921-s77h9l/yams/Sources/Yams/Node.Scalar.swift /tmp/swift-20220917-16921-s77h9l/yams/Sources/Yams/Node.Sequence.swift /tmp/swift-20220917-16921-s77h9l/yams/Sources/Yams/Node.swift /tmp/swift-20220917-16921-s77h9l/yams/Sources/Yams/Parser.swift /tmp/swift-20220917-16921-s77h9l/yams/Sources/Yams/Representer.swift /tmp/swift-20220917-16921-s77h9l/yams/Sources/Yams/Resolver.swift /tmp/swift-20220917-16921-s77h9l/yams/Sources/Yams/String+Yams.swift /tmp/swift-20220917-16921-s77h9l/yams/Sources/Yams/Tag.swift /tmp/swift-20220917-16921-s77h9l/yams/Sources/Yams/YamlError.swift  -Xlinker -soname -Xlinker libYams.so  -L /tmp/swift-20220917-16921-s77h9l/build/swiftpm-linux-x86_64/x86_64-unknown-linux-gnu/yams/lib  -L /tmp/swift-20220917-16921-s77h9l/build/foundation-linux-x86_64/lib   -L /home/linuxbrew/.linuxbrew/lib   -L /usr/lib/gcc/x86_64-linux-gnu/11   -L /home/linuxbrew/.linuxbrew/Cellar/llvm/14.0.6_1/lib -Xlinker -rpath -Xlinker /tmp/swift-20220917-16921-s77h9l/build/foundation-linux-x86_64/lib:  lib/libCYaml.a  -ldispatch  /tmp/swift-20220917-16921-s77h9l/build/foundation-linux-x86_64/lib/libFoundation.so  -lswiftDispatch  -lgcc  -lgcc_s  -lc  -lgcc  -lgcc_s && :
2022-09-18T05:03:20.5644744Z ld.gold: error: lib/libCYaml.a(api.c.o): requires unsupported dynamic reloc 11; recompile with -fPIC
2022-09-18T05:03:20.5645122Z ld.gold: error: lib/libCYaml.a(emitter.c.o): requires unsupported dynamic reloc 11; recompile with -fPIC
2022-09-18T05:03:20.5645558Z ld.gold: error: lib/libCYaml.a(parser.c.o): requires unsupported dynamic reloc 11; recompile with -fPIC
2022-09-18T05:03:20.5645934Z ld.gold: error: lib/libCYaml.a(scanner.c.o): requires unsupported dynamic reloc 11; recompile with -fPIC
2022-09-18T05:03:20.5646284Z ld.gold: error: lib/libCYaml.a(writer.c.o): requires unsupported dynamic reloc 11; recompile with -fPIC
2022-09-18T05:03:20.5646622Z ld.gold: error: lib/libCYaml.a(reader.c.o): requires unsupported dynamic reloc 11; recompile with -fPIC

danielnachun avatar Sep 18 '22 09:09 danielnachun

Manually updated Yams to 5.0.1 to fix that issue. Upstream use 5.0.0 but enabling PIC is the only change in 5.0.1 so it should be good to use.

Bo98 avatar Sep 18 '22 17:09 Bo98

Had a peek inside the runner, since swift only has a handful of dependents so it shouldn't be taking this long.

It's been stuck testing vapor for about 5 hours.

carlocab avatar Sep 21 '22 08:09 carlocab

Linux dependents need revision bumps, I think. But that may be better done separately.

carlocab avatar Sep 21 '22 10:09 carlocab

Oh oops, swift-format should indeed have been updated alongside.

I don't think anything else should have broke however. Will need to take a look at that. Weirdly, everything else seems to fail when reading a file...

Bo98 avatar Sep 21 '22 13:09 Bo98

swift-format changes are in #111254. Feel free to pull them in here.

mrstegeman avatar Sep 21 '22 15:09 mrstegeman

Any progress on this? I'd love to see this merged :)

bartekpacia avatar Sep 26 '22 18:09 bartekpacia

You can help fix the build for vapor getting stuck in the test.

SMillerDev avatar Sep 26 '22 18:09 SMillerDev

You can help fix the build for vapor getting stuck in the test.

I'd love to, if I only had time :). I'm counting on good open source people to deal with this. I didn't mean to be rude or annoying, just wanted to express some interest in getting this merged. Thank you for work!

bartekpacia avatar Sep 26 '22 18:09 bartekpacia

I'm counting on good open source people to deal with this.

You can be one of good open source people who volunteer their time.

Here's some advice: no pull request ever got merged because random people posted comments expressing interested in getting it merged. Code only gets merged because people put in effort to make it so.

SMillerDev avatar Sep 27 '22 07:09 SMillerDev

I've had the week off - apologies. Will be looking into this tonight.

Bo98 avatar Sep 30 '22 15:09 Bo98

Combing over the PR's CI failure log, it looks like we are still faced with a variety of failures:

  1. brew test --retry --verbose create-api: this seems like it might be a problem with create-api itself.
==> /home/linuxbrew/.linuxbrew/Cellar/create-api/0.1.0/bin/create-api generate /home/linuxbrew/.linuxbrew/Cellar/create-api/0.1.0/share/create-api/test-spec.json --config-option module=TestPackage
  Error: ERROR! The spec is missing or invalid. Expected value to be parsable as Mapping but it was not.
  Generating code for test-spec.json...
  1. brew test --retry --verbose publish: unhelpful error
Error: publish: failed
  An exception occurred within a child process:
    BuildError: Failed executing: /home/linuxbrew/.linuxbrew/Cellar/publish/0.9.0/bin/publish new
  /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/formula.rb:2493:in `block in system'
  /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/formula.rb:2429:in `open'
  1. brew test --retry --verbose swift-format: It appears this was fixed in https://github.com/Homebrew/homebrew-core/pull/111254.
 ==> Testing swift-format
 ==> /home/linuxbrew/.linuxbrew/Cellar/swift-format/0.50600.1/bin/swift-format test.swift
  <unknown>: error: Unable to format test.swift: The loaded '_InternalSwiftSyntaxParser' library is from a toolchain that is not compatible with this version of SwiftSyntax
  
 ==> Testing swift-format (again)
 ==> /home/linuxbrew/.linuxbrew/Cellar/swift-format/0.50600.1/bin/swift-format test.swift
  <unknown>: error: Unable to format test.swift: The loaded '_InternalSwiftSyntaxParser' library is from a toolchain that is not compatible with this version of SwiftSyntax
 Error: swift-format: failed
  1. brew test --retry --verbose swiftdraw: Some kind of XML parsing error
==> /home/linuxbrew/.linuxbrew/Cellar/swiftdraw/0.13.1/bin/swiftdraw /tmp/swiftdraw-test-20220921-137440-1gup5ut/fish.svg --format sfsymbol
Failure: The operation could not be completed. (SwiftDraw.XMLParser.Error error 3.)
  1. brew test --retry --verbose vapor: No output at all. Timeout?
  2. brew test --retry --verbose xclogparser: Not able to parse a log file. Perhaps the latest version of Xcode made changes to their log format? Probably the package author needs to update their parser.
Error: The file in /tmp/xclogparser-test-20220921-144760-t0jco6/test.xcactivitylog is not a valid SLF log
[567](https://github.com/Homebrew/homebrew-core/actions/runs/3078070811/jobs/4993456204#step:11:578)

I'd like to help out with any of these, but I'd like to avoid duplicating tedious debugging. @SMillerDev and @Bo98, can you recommend a good starting task?

marcprux avatar Oct 04 '22 17:10 marcprux

Trying to reproduce the errors locally is a good start

SMillerDev avatar Oct 04 '22 18:10 SMillerDev

I suspect some/all of these just need revision bumps. Swift 5.7 itself is building fine, it's just some of its dependents that are failing their tests. Some of them may require changes to support Swift 5.7.

danielnachun avatar Oct 04 '22 22:10 danielnachun

I suspect some/all of these just need revision bumps.

What is the policy on this? Is the availability of Swift 5.7 to be held up for everyone due to these projects?

marcprux avatar Oct 04 '22 23:10 marcprux

What is the policy on this? Is the availability of Swift 5.7 to be held up for everyone due to these projects?

Usually when we do dependent testing we'd like to know why the tests are failing, as they may indicate those packages will need fixes for the newer version of Swift. That doesn't mean they have to be fixed for the Swift 5.7 to be merged - we just want to understand what may have changed and follow up with fixes soon after.

danielnachun avatar Oct 05 '22 00:10 danielnachun

Revision bumping will indeed fix them. I just had to look into what exactly the issue was and I had very little time this last couple weeks unfortunately.

Basically I think our luck ran out in terms of ABI stability - Swift doesn't guarantee it on Linux (only macOS) and I guess something changes somewhere in a non-obvious way in 5.7.

Basically I'm thinking of moving to versioned lib directories (which will break the linkage on version bumps) and/or linking the stdlib statically everywhere, which is something that will happen by default in the future: https://github.com/apple/swift-evolution/blob/main/proposals/0342-static-link-runtime-libraries-by-default-on-supported-platforms.md. Doing this will prevent this delay/confusion next time (5.8).

I'll push something tomorrow.

Bo98 avatar Oct 05 '22 05:10 Bo98

Seeing how we're trying to get rid of those in every other software I'd hate to have this introduced in swift now.

SMillerDev avatar Oct 05 '22 06:10 SMillerDev

Seeing how we're trying to get rid of those in every other software I'd hate to have this introduced in swift now.

Static linking or versioned library directories?

danielnachun avatar Oct 05 '22 06:10 danielnachun

Honestly, both.

SMillerDev avatar Oct 05 '22 06:10 SMillerDev

Basically I'm thinking of moving to versioned lib directories (which will break the linkage on version bumps) and/or linking the stdlib statically everywhere

Would this have the additional benefit of enabling brew to install swift formula without having to depend on the swift formula itself? If so, that would save a ton of CI minutes for workflows that rely on these tools.

marcprux avatar Oct 05 '22 12:10 marcprux

Seeing how we're trying to get rid of those in every other software I'd hate to have this introduced in swift now.

I do agree that it would be nice if things like Go had a small libc-like library. But only if it's ABI stable - I would hate it if we had to revision bump every Go dependent on every patch update and it's a similar story here.

If upstream don't provide ABI stability there's nothing we can really do about it, particularly given static stdlib linking will be a default soon.

Would this have the additional benefit of enabling brew to install swift formula without having to depend on the swift formula itself?

It would yes.

Bo98 avatar Oct 05 '22 13:10 Bo98

Static linking isn't great, but if upstream is going to make it the default, then we may as well go with it.

carlocab avatar Oct 05 '22 14:10 carlocab

I've just revision bumped for now. There's an upstream bug that needs fixing before we can use static stdlib linking so I'll revisit the situation in 5.7.1.

I've however added static libraries to the Swift formula with this release since we didn't have them before. --static-swift-stdlib will now work, unless you are using FoundationNetworking or FoundationXML (because of that bug).

Bo98 avatar Oct 07 '22 15:10 Bo98

@Bo98 Can you elaborate on the upstream bug that prevents static linking?

marcprux avatar Oct 07 '22 23:10 marcprux

https://github.com/apple/swift/pull/61372

Bo98 avatar Oct 07 '22 23:10 Bo98

apple/swift#61372

Bummer. Hopefully @MaxDesiatov can fix it.

marcprux avatar Oct 07 '22 23:10 marcprux

https://github.com/apple/swift/pull/61372 is now closed. See comments on that PR for more details.

MaxDesiatov avatar Oct 11 '22 22:10 MaxDesiatov

apple/swift#61372 is now closed. See comments on that issue for more details.

Thanks for the heads up. Good to know in case we need in the future to send patches to any packages affected. It'll become more of an issue when SE-0342 is implemented so I'll likely wait until then.

Bo98 avatar Oct 11 '22 22:10 Bo98