swift icon indicating copy to clipboard operation
swift copied to clipboard

Runtime crash: 'freed pointer was not the last allocation'

Open glbrntt opened this issue 1 year ago • 6 comments

Description

One of the tests in the attached repository reliably crashes at runtime. This seems to be related to DiscardingTaskGroup as switching to TaskGroup<Void> doesn't crash.

Reproduction

  • Extract grpc-swift.zip
  • Using Swift version 6.0-dev (LLVM 723ba90c22d3da1, Swift ae7e24dc658e067) (swiftlang/swift:nightly-6.0-jammy, sha256:4624a3c7404f479e55d69dc5e41f94e399cb0c11f291f81689d81e593b409610)
  • On the crash branch run swift test --filter ClientRPCExecutorTests.testRetriesCantBeExecutedForTooManyRequestMessages

Stack dump

freed pointer was not the last allocation
*** Signal 6: Backtracing from 0xffff95f6f200... done ***

*** Program crashed: Aborted at 0x0000000000002b82 ***

Thread 0 crashed:

 0                  0x0000ffff95f6f200 <unknown> in libc.so.6
 1 [ra]             0x0000ffff95f2a67c <unknown> in libc.so.6
 2 [ra]             0x0000ffff95f17130 <unknown> in libc.so.6
 3 [ra]             0x0000ffff96fa4c24 swift::swift_Concurrency_fatalErrorv(unsigned int, char const*, std::__va_list) + 43 in libswift_Concurrency.so
 4 [ra]             0x0000ffff96fa4c84 swift::swift_Concurrency_fatalError(unsigned int, char const*, ...) + 95 in libswift_Concurrency.so
 5 [ra]             0x0000ffff96fa6be8 swift_task_dealloc + 127 in libswift_Concurrency.so
 6 [ra]             0x0000aaaacfb1a840 ClientRPCExecutor.RetryExecutor.executeAttempt<A>(stream:metadata:retryStream:method:attempt:responseHandler:) + 215 in grpc-swiftPackageTests.xctest at /code/Sources/GRPCCore/Call/Client/Internal/ClientRPCExecutor+RetryExecutor.swift:209:18
 7 [async]          0x0000aaaacfb1a448 closure #1 in closure #3 in closure #1 in ClientRPCExecutor.RetryExecutor.execute<A>(request:method:options:responseHandler:) in grpc-swiftPackageTests.xctest at /code/Sources/GRPCCore/Call/Client/Internal/ClientRPCExecutor+RetryExecutor.swift:134
 8 [async] [thunk]  0x0000aaaacfb207ac partial apply for closure #1 in closure #3 in closure #1 in ClientRPCExecutor.RetryExecutor.execute<A>(request:method:options:responseHandler:) in grpc-swiftPackageTests.xctest at //<compiler-generated>
 9 [async] [thunk]  0x0000aaaacfb04dec thunk for @escaping @isolated(any) @callee_guaranteed @async () -> (@out A) in grpc-swiftPackageTests.xctest at //<compiler-generated>
10 [async] [thunk]  0x0000aaaacfb07324 partial apply for thunk for @escaping @isolated(any) @callee_guaranteed @async () -> (@out A) in grpc-swiftPackageTests.xctest at //<compiler-generated>
11 [async] [system] 0x0000ffff96fa63dc completeTaskWithClosure(swift::AsyncContext*, swift::SwiftError*) in libswift_Concurrency.so


Registers:

 x0 0x0000000000000000  0
 x1 0x0000000000002b87  11143
 x2 0x0000000000000006  6
 x3 0x0000ffff917fee00  01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ················
 x4 0x0000ffff97eeab58  00 b8 da 6a 16 71 b9 b7 00 00 00 00 44 01 06 00  ·¸Új·q¹·····D···
 x5 0x00000000fbad2887  4222429319
 x6 0x6f70206465657266  8029953751171560038
 x7 0x6177207265746e69  7023117819608067689
 x8 0x0000000000000083  131
 x9 0x61207473616c2065  6998721859838287973
x10 0x0000ffff95ef4860  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ················
x11 0x0000000000000000  0
x12 0x0000000000002180  8576
x13 0x0000000000000000  0
x14 0x0000ffff8c00fdb8  60 79 00 8c ff ff 00 00 01 00 00 00 00 00 00 00  `y··ÿÿ··········
x15 0x0000ffff8c00fdb0  b8 f7 56 97 ff ff 00 00 60 79 00 8c ff ff 00 00  ¸÷V·ÿÿ··`y··ÿÿ··
x16 0x0000000000000001  1
x17 0x0000ffff96fe0578  4c 70 f1 95 ff ff 00 00 48 3f f6 96 ff ff 00 00  Lpñ·ÿÿ··H?ö·ÿÿ··
x18 0x0000ffff8c010120  20 fc 00 8c ff ff 00 00 58 b0 b1 cf aa aa 00 00   ü··ÿÿ··X°±Ïªª··
x19 0x0000000000002b87  11143
x20 0x0000ffff917fee00  01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ················
x21 0x0000000000000006  6
x22 0x0000ffff8c00f610  10 f2 00 8c ff ff 00 00 68 a7 b1 cf aa aa 00 00  ·ò··ÿÿ··h§±Ïªª··
x23 0x0000ffff917ff608  87 2b 00 00 00 00 00 00 b0 82 90 d4 aa aa 00 00  ·+······°··Ôªª··
x24 0x0000ffff8c011c60  00 00 00 00 00 00 00 00 80 1a 01 8c ff ff 00 00  ············ÿÿ··
x25 0x0000000000005019  20505
x26 0x0000ffff917ff5f0  30 eb 00 8c ff ff 00 00 80 e3 7f 91 ff ff 00 00  0ë··ÿÿ···ã··ÿÿ··
x27 0x0000000000000000  0
x28 0x0000ffff917ff5c0  00 27 00 8c ff ff 00 00 00 00 00 00 00 00 00 00  ·'··ÿÿ··········
 fp 0x0000ffff917fdea0  70 df 7f 91 ff ff 00 00 7c a6 f2 95 ff ff 00 00  pß··ÿÿ··|¦ò·ÿÿ··
 lr 0x0000ffff95f6f1ec  e1 03 13 aa e2 03 15 aa 00 7c 40 93 68 10 80 d2  á··ªâ··ª·|@·h··Ò
 sp 0x0000ffff917fdea0  70 df 7f 91 ff ff 00 00 7c a6 f2 95 ff ff 00 00  pß··ÿÿ··|¦ò·ÿÿ··
 pc 0x0000ffff95f6f200  1f 04 40 31 e0 97 80 5a e0 ff ff 17 e0 03 16 aa  ··@1à··Zàÿÿ·à··ª


Images (49 omitted):

0x0000aaaacf3a0000–0x0000aaaad2a52f48 <no build ID>                            grpc-swiftPackageTests.xctest /code/.build/aarch64-unknown-linux-gnu/debug/grpc-swiftPackageTests.xctest
0x0000ffff95ef0000–0x0000ffff96077404 aa6e122fa39ae02d412afb49d75e33281fcd2805 libc.so.6                     /usr/lib/aarch64-linux-gnu/libc.so.6
0x0000ffff96f40000–0x0000ffff96fbf380 cf1639bfd9a10cae9968dcb7baf58403d1a7106a libswift_Concurrency.so       /usr/lib/swift/linux/libswift_Concurrency.so

Backtrace took 1.54s

Expected behavior

Test should complete

Environment

Swift version 6.0-dev (LLVM 723ba90c22d3da1, Swift ae7e24dc658e067) Target: aarch64-unknown-linux-gnu

Additional information

  • Doesn't reproduce on macOS
  • The repo also has a fix branch which switches from DiscardingTaskGroup to TaskGroup<Void>, the same test doesn't crash.

glbrntt avatar Jul 26 '24 08:07 glbrntt

Hmm, these happen when the task local allocator has allocated in some order a b c and frees b before c etc. This is rather unexpected, a discarding taskgroup itself doesn't really task allocate anything hmmm

I'll look into this

ktoso avatar Jul 30 '24 01:07 ktoso

@ktoso Any luck on this? We started to catch similar crash in working project when we do parallel await of two async let requests. But I can't reproduce it on small example project :(

distomin avatar Nov 04 '24 10:11 distomin

No news I'm afraid. Reproducers would certainly help... we've not caught it yet.

ktoso avatar Nov 04 '24 11:11 ktoso

We're hitting this in Vapor too with

swift-driver version: 1.120.2 Apple Swift version 6.1 (swiftlang-6.1.0.106.4 clang-1700.0.9.2)
Target: arm64-apple-macosx15.0

0xTim avatar Mar 02 '25 18:03 0xTim

Also hitting this using Vapor 1.113.2, but only synchronous work. Coredump backtrace produces:

#9  0x0000608a5547c503 in $s8NIOPosix8SelectorC9whenReady8strategy11onLoopBegin_yAA0B8StrategyO_yyXEyAA0B5EventVyxGKXEtKFAA15NIORegistrationV_Tg5 (strategy=..., loopStart=..., body=<error reading variable: Cannot access memory at address 0x0>, self=...) at .build/checkouts/swift-nio/Sources/NIOPosix/SelectorGeneric.swift:368

Using Swift 6.0.3 on Arch Linux (6.13.4-arch1-1) with Swift 5.10 Tools Version.

Very strange as I fixed it downgrading SwiftNIO to 2.79.0 (and Vapor to 1.107.0), but can no longer reproduce it using the latest Vapor and SwiftNIO.

RandomHashTags avatar Mar 06 '25 10:03 RandomHashTags

++ Reported from other forum

Task { 
    for workout in items {
        
        // Crash on Xcode 16.2 Swift 6.0.3
        async let result1 = workout.getStrokeCount()
        async let result2 = workout.getHeartRates() 
        async let result3 = workout.getSwimmingDistance()
        let (data1, data2, data3) = await (result1, result2, result3)
        
        // ... consuming given data
        
    }
}

func getStrokeCount() async -> WorkoutStrokeCountData? {
    await withCheckedContinuation { continuation in
        let healthStore = HKHealthStore()
        let strokeType = ...
        let predicate = ...
        
        let strokeQuery = HKSampleQuery(sampleType: strokeType, predicate: predicate, limit: HKObjectQueryNoLimit, sortDescriptors: [.init(keyPath: \HKSample.startDate, ascending: true)], resultsHandler: { (query, results, error) in
            
            if let error {
                Logger.e("\(error)")
                continuation.resume(returning: nil)
            }
            
            if let results = results {
                ...
                continuation.resume(returning: WorkoutStrokeCountData(rawData: results))
            } else {
                continuation.resume(returning: nil)
            }
            
        })
        
        healthStore.execute(strokeQuery)
    }
}

func getHeartRates() async -> [WorkoutExecutionRecord]? {
    await withCheckedContinuation { continuation in
        let healthStore = HKHealthStore()
        let heartRateType = ...
        let predicate = ...
        
        let heartRateQuery = HKSampleQuery(sampleType: heartRateType, predicate: predicate, limit: HKObjectQueryNoLimit, sortDescriptors: [.init(keyPath: \HKSample.startDate, ascending: true)], resultsHandler: { (query, results, error) in
            
            if let error {
                Logger.e("\(error)")
                continuation.resume(returning: nil)
            }
                            
            if let results = results {
                ...
                continuation.resume(returning: items)
            } else {
                continuation.resume(returning: nil)
            }                
        })
        
        healthStore.execute(heartRateQuery)
    }
}

Still occurring with Xcode 16.3 and macOS 15.4

Problem seems related to "async let"

andoma93 avatar Apr 08 '25 19:04 andoma93

Any updates on this issue?

mayumi-hokama-stb avatar Apr 16 '25 11:04 mayumi-hokama-stb

Hello @ktoso , any update with this issue? Thanks you

andoma93 avatar Apr 17 '25 11:04 andoma93

Is there any news os this issue? 😁

Thanks 🙏

Edit: Sorry, thought I pinned it to my account for easy finding 🤦‍♂️

kimdv avatar May 10 '25 07:05 kimdv

I prepared small one-screen app, where this error stably appears. To call crash run this app on Xcode 16.2 emulator iPhone 11 iOS 18.3.1. Example of stable crash with using let async freed pointer was not the last allocation Swift:

https://github.com/IrixVol/LetAsyncCrash

Please note that the error will only occur when the minimum target is less than or equal to 15

IrixVol avatar Jun 08 '25 21:06 IrixVol

The same bug: https://github.com/swiftlang/swift/issues/81771

IrixVol avatar Jun 09 '25 11:06 IrixVol

We don't use async let as is, but it still blows up given hard enough use of async APIs. Unfortunately downgrading to 2.79 is not an option due to more modern Swift codebase.

fingon avatar Jun 27 '25 08:06 fingon

Although I don't have a small reproducible chunk of code to contribute, I think this is potentially the same bug that is causing crashes in OpenAPIKit's test suite when run on Linux (never macOS). It was causing inconsistent failures in all Swift 5.10+ versions until a few days ago -- now it is still inconsistent in all other versions but it is a consistent failure for the nightly Docker image, so if it helps, you can run swift test --filter ExternalDereferencingDocumentTests.test_example against OpenAPIKit (target the feature/444/multipart-media branch if you want to be certain to reproduce the error) on an x86_64 linux machine with the Nightly Swift build as of today and you will hit this error (or one that manifests too similarly for me to know the difference).

Test Case 'ExternalDereferencingDocumentTests.test_example' started at 2025-11-24 16:18:46.595
freed pointer was not the last allocation
[...]
↳ Testing Library Version: 6.3-dev (98430f2d6cb8d81)
↳ Target Platform: x86_64-unknown-linux-gnu

I am happy to provide the full log output with stack trace if helpful, but I mostly thought it may be useful information that this is suddenly a consistent failure with the latest nightly builds.

[EDIT] Actually, even more interesting is that this is a consistent failure for the nightly docker image for the jammy tag but not for the focal tag!

mattpolzin avatar Nov 25 '25 01:11 mattpolzin

Apple folks: rdar://132772297

weissi avatar Nov 26 '25 15:11 weissi