Parca v0.22 panic in arrow/array

[gites]

After upgrading Parca to v0.22 (and Parca Agent to v0.32) I'm seeing it constantly crashing with this panic:

{"addr":":7070","caller":"server.go:90","level":"info","msg":"starting server","name":"parca","ts":"2024-08-28T13:22:01.970854348Z"}
panic: arrow/array: index out of range

goroutine 121997 [running]:
github.com/apache/arrow/go/v16/arrow/array.(*Binary).Value(...)
        github.com/apache/arrow/go/[email protected]/arrow/array/binary.go:56
github.com/parca-dev/parca/pkg/profile.serializedArrowLocationSize(0xc02b1cd7a0?, 0x4c?, 0x0?, 0x0, 0x0, {0xc02d379980?, 0x3?, 0x1?}, {0x0, 0x0, ...}, ...)
        github.com/parca-dev/parca/pkg/profile/encode.go:397 +0xe38
github.com/parca-dev/parca/pkg/profile.EncodeArrowLocation(0x7de3aebae4f3f38e, 0x1, 0x0, 0x0, 0x0, {0xc02d379980, 0x3, 0x3?}, {0x0, 0x0, ...}, ...)
        github.com/parca-dev/parca/pkg/profile/encode.go:296 +0x125
github.com/parca-dev/parca/pkg/normalizer.(*arrowToInternalConverter).AddLocationsRecordV1(0xc02b1cdf28, {0xc00a27d240?, 0x0?}, {0x5b6cf30, 0xc00fae8db0})
        github.com/parca-dev/parca/pkg/normalizer/arrow.go:587 +0x150b
github.com/parca-dev/parca/pkg/normalizer.(*arrowToInternalConverter).AddLocationsRecord(0xc02b1cdf28, {0x5b444a0, 0xc0375a42a0}, {0x5b6cf30, 0xc00fae8db0})
        github.com/parca-dev/parca/pkg/normalizer/arrow.go:130 +0x189
github.com/parca-dev/parca/pkg/profilestore.(*ProfileColumnStore).write(0xc0016d0420, {0x5b444a0, 0xc0375a42a0}, {0x5b5ddf8, 0xc03e5934e0})
        github.com/parca-dev/parca/pkg/profilestore/profilecolumnstore.go:270 +0x9ac
github.com/parca-dev/parca/pkg/profilestore.(*ProfileColumnStore).Write.func1()
        github.com/parca-dev/parca/pkg/profilestore/profilecolumnstore.go:180 +0x30
created by github.com/parca-dev/parca/pkg/profilestore.(*ProfileColumnStore).Write in goroutine 121996
        github.com/parca-dev/parca/pkg/profilestore/profilecolumnstore.go:179 +0x11f

Parca cmd options:

        - /parca
        - --config-path=/etc/parca/parca.yaml
        - --cors-allowed-origins=*
        - --enable-persistence
        - --storage-path=/data
        - --storage-enable-wal
        - --storage-index-on-disk
        - --debuginfo-cache-dir=/debug
        - --log-format=json
        - --log-level=info
        - --insecure
        - --profile-share-server=none
        - --debuginfod-upstream-servers=none

parca.yaml

object_storage:
  bucket:
    type: s3
    config:
      aws_sdk_auth: true
      bucket: <bucket-name>
      endpoint: s3.us-west-2.amazonaws.com
      region: us-west-2
      insecure: false
      signature_version2: false

scrape_configs: null

Happy to provide more debug info if needed.

[metalmatze]

Thanks for reporting. We'll look into that!

[denisgolius]

Thanks for reporting. We'll look into that!

Hey! Any updates on this?

[brancz]

https://github.com/parca-dev/parca/pull/5251 should fix it

[gites]

Let's test it in the wild 😀

[brancz]

Let us know how it goes!

[gites]

Running stable since the deployment. Thanks for the fix!

parca-0                                                    5/5     Running     0               19h