stream-chat-swiftui
stream-chat-swiftui copied to clipboard
Implement performance benchmarks
๐ Issue Links
Resolves https://github.com/GetStream/ios-issues-tracking/issues/659
๐ฏ Goal
Implement Performance Benchmarks
๐ Notion
- https://www.notion.so/stream-wiki/XCMetrics-5ef80c5ae9d44772966ab99f6c6d094a
๐ Summary
- https://github.com/GetStream/stream-swift-performance-benchmarks stores the benchmarks
- XCMetrics are executed on every PR (including release) by
first PR's approval,manually on any branchandnightly by cron - XCMetrics won't be executed if there are no changes to the
Sourcesfolder - Collected metrics:
:hitches_total_duration,:duration,:hitch_time_ratio,:frame_count,:frame_rate,:number_of_hitches - Metrics are collected from real devices on Firebase TestLab. Firebase offers us 5 runs per day (although, for some reason, sometimes it's even more than that), which should be enough most of the time (
1 cron, 4 PRs). If this isn't enough, we can set up a paid account that offers free 30 minutes per day (1run โ 4min) - Our custom XCMetrics threshold equals to Xcode default one -
10% - Performance tests are executed on
MessageList(with mocked static data: messages/images/videos/pdfs/giphies) andChannelList(without mocks) - As far as I understood, Apple requires developers to run performance tests on the same machine + same iPhone (read more), which is impossible on CI. That's why there is no configured baseline in this PR, and the XCTests accordingly will always pass. Worries aside, because I implemented a custom calculation & assertion on the fastlane level.
๐จ Showcase
target |
metric |
branch |
release/4.48.1 |
performance |
status |
|---|---|---|---|---|---|
| MessageList | Hitches total duration | 5.0 ms | 20 ms | 75.0% ๐ผ | ๐ข |
| Duration | 2.6 s | 5 s | 48.0% ๐ผ | ๐ข | |
| Hitch time ratio | 2.0 ms per s | 10 ms per s | 80.0% ๐ผ | ๐ข | |
| Frame count | 199 | 300 | 33.7% ๐ผ | ๐ข | |
| Frame rate | 78.4 fps | 100 fps | 21.6% ๐ผ | ๐ข | |
| Number of hitches | 0 | 1 | 100.0% ๐ผ | ๐ข | |
| ChannelList | Hitches total duration | 10.0 ms | 20 ms | 50.0% ๐ผ | ๐ข |
| Duration | 2.6 s | 5 s | 48.0% ๐ผ | ๐ข | |
| Hitch time ratio | 3.9 ms per s | 10 ms per s | 61.0% ๐ผ | ๐ข | |
| Frame rate | 74.7 fps | 100 fps | 25.3% ๐ผ | ๐ข | |
| Number of hitches | 0 | 1 | 100.0% ๐ผ | ๐ข |
๐งช Manual Testing Notes
bundle exec fastlane xcmetrics
โ๏ธ Contributor Checklist
- [x] I have signed the Stream CLA (required)
- [x] This change follows zero โ ๏ธ policy (required)
- [ ] This change should be manually QAed
- [ ] Changelog is updated with client-facing changes
- [ ] New code is covered by unit tests
- [ ] Comparison screenshots added for visual changes
- [ ] Affected documentation updated (docusaurus, tutorial, CMS)
๐ Meme

| 6 Errors | |
|---|---|
| :no_entry_sign: | Please use more than one word. 793e29eb9da3743d9109d4c0182d47c7a59825be |
| :no_entry_sign: | Please use more than one word. e54eb7db7c9af9e8bb5feae886e3f31a9f40440d |
| :no_entry_sign: | Please use more than one word. 091328b4b610dcc64bb25d1b7ab0b14f71e80a45 |
| :no_entry_sign: | Please start subject with capital letter. 4ad1505de4767411d5519ab49ab0ce8ca986a4b6 |
| :no_entry_sign: | Please use more than one word. 4ad1505de4767411d5519ab49ab0ce8ca986a4b6 |
| :no_entry_sign: | Please use more than one word. 6a4bf8a33eea6662b04de6fa013322775cea9e1b |
| 1 Warning | |
|---|---|
| :warning: | Big PR |
Generated by :no_entry_sign: Danger
StreamChat XCMetrics
target |
metric |
release/4.49.0 |
branch |
performance |
status |
|---|---|---|---|---|---|
| MessageList | Hitches total duration | 20 ms | 30.9 ms | -54.5% ๐ฝ | ๐ด |
| Duration | 5 s | 2.5 s | 50.0% ๐ผ | ๐ข | |
| Hitch time ratio | 10 ms per s | 12.4 ms per s | -24.0% ๐ฝ | ๐ด | |
| Frame rate | 100 fps | 79.3 fps | 20.7% ๐ผ | ๐ข | |
| Number of hitches | 1 | 2 | -100.0% ๐ฝ | ๐ด | |
| ChannelList | Hitches total duration | 20 ms | 38.4 ms | -92.0% ๐ฝ | ๐ด |
| Duration | 5 s | 2.5 s | 50.0% ๐ผ | ๐ข | |
| Hitch time ratio | 10 ms per s | 15.4 ms per s | -54.0% ๐ฝ | ๐ด | |
| Frame rate | 100 fps | 73.9 fps | 26.1% ๐ผ | ๐ข | |
| Number of hitches | 1 | 3 | -200.0% ๐ฝ | ๐ด |
StreamChat XCMetrics
target |
metric |
benchmark |
branch |
performance |
status |
|---|---|---|---|---|---|
| MessageList | Hitches total duration | 8.3 ms | 30.0 ms | -261.45% ๐ฝ | ๐ด |
| Duration | 2.6 s | 2.48 s | 4.62% ๐ผ | ๐ข | |
| Hitch time ratio | 3.3 ms per s | 12.11 ms per s | -266.97% ๐ฝ | ๐ด | |
| Frame rate | 79 fps | 79.29 fps | -0.37% ๐ฝ | ๐ก | |
| Number of hitches | 0.7 | 2.6 | -271.43% ๐ฝ | ๐ด | |
| ChannelList | Hitches total duration | 10.8 ms | 37.6 ms | -248.15% ๐ฝ | ๐ด |
| Duration | 2.6 s | 2.49 s | 4.23% ๐ผ | ๐ข | |
| Hitch time ratio | 4.2 ms per s | 15.03 ms per s | -257.86% ๐ฝ | ๐ด | |
| Frame rate | 75 fps | 73.49 fps | 2.01% ๐ผ | ๐ข | |
| Number of hitches | 0 | 3.6 | -360.0% ๐ฝ | ๐ด |
StreamChat XCMetrics
target |
metric |
benchmark |
branch |
performance |
status |
|---|---|---|---|---|---|
| MessageList | Hitches total duration | 8.3 ms | 24.2 ms | -191.57% ๐ฝ | ๐ด |
| Duration | 2.6 s | 2.5 s | 3.85% ๐ผ | ๐ข | |
| Hitch time ratio | 3.3 ms per s | 9.7 ms per s | -193.94% ๐ฝ | ๐ด | |
| Frame rate | 79 fps | 79.33 fps | -0.42% ๐ฝ | ๐ก | |
| Number of hitches | 0.7 | 2.4 | -242.86% ๐ฝ | ๐ด | |
| ChannelList | Hitches total duration | 10.8 ms | 35.9 ms | -232.41% ๐ฝ | ๐ด |
| Duration | 2.6 s | 2.47 s | 5.0% ๐ผ | ๐ข | |
| Hitch time ratio | 4.2 ms per s | 14.54 ms per s | -246.19% ๐ฝ | ๐ด | |
| Frame rate | 75 fps | 73.94 fps | 1.41% ๐ผ | ๐ข | |
| Number of hitches | 0 | 3.2 | -320.0% ๐ฝ | ๐ด |
Quality Gate passed
Issues
47 New issues
0 Accepted issues
Measures
0 Security Hotspots
71.2% Coverage on New Code
0.0% Duplication on New Code
StreamChat XCMetrics
target |
metric |
benchmark |
branch |
performance |
status |
|---|---|---|---|---|---|
| MessageList | Hitches total duration | 8.3 ms | 49.2 ms | -492.77% ๐ฝ | ๐ด |
| Duration | 2.6 s | 2.49 s | 4.23% ๐ผ | ๐ข | |
| Hitch time ratio | 3.3 ms per s | 19.75 ms per s | -498.48% ๐ฝ | ๐ด | |
| Frame rate | 79 fps | 78.57 fps | 0.54% ๐ผ | ๐ข | |
| Number of hitches | 0.8 | 4.0 | -400.0% ๐ฝ | ๐ด | |
| ChannelList | Hitches total duration | 10.8 ms | 40.9 ms | -278.7% ๐ฝ | ๐ด |
| Duration | 2.6 s | 2.49 s | 4.23% ๐ผ | ๐ข | |
| Hitch time ratio | 4.2 ms per s | 16.41 ms per s | -290.71% ๐ฝ | ๐ด | |
| Frame rate | 76 fps | 73.89 fps | 2.78% ๐ผ | ๐ข | |
| Number of hitches | 0.8 | 3.8 | -375.0% ๐ฝ | ๐ด |
StreamChat XCMetrics
target |
metric |
benchmark |
branch |
performance |
status |
|---|---|---|---|---|---|
| MessageList | Hitches total duration | 8.3 ms | 60.1 ms | -624.1% ๐ฝ | ๐ด |
| Duration | 2.6 s | 2.49 s | 4.23% ๐ผ | ๐ข | |
| Hitch time ratio | 3.3 ms per s | 24.12 ms per s | -630.91% ๐ฝ | ๐ด | |
| Frame rate | 79 fps | 77.79 fps | 1.53% ๐ผ | ๐ข | |
| Number of hitches | 0.8 | 4.2 | -425.0% ๐ฝ | ๐ด | |
| ChannelList | Hitches total duration | 10.8 ms | 41.7 ms | -286.11% ๐ฝ | ๐ด |
| Duration | 2.6 s | 2.48 s | 4.62% ๐ผ | ๐ข | |
| Hitch time ratio | 4.2 ms per s | 16.82 ms per s | -300.48% ๐ฝ | ๐ด | |
| Frame rate | 76 fps | 73.81 fps | 2.88% ๐ผ | ๐ข | |
| Number of hitches | 0.8 | 3.6 | -350.0% ๐ฝ | ๐ด |
StreamChat XCMetrics
target |
metric |
benchmark |
branch |
performance |
status |
|---|---|---|---|---|---|
| MessageList | Hitches total duration | 8.3 ms | 32.5 ms | -291.57% ๐ฝ | ๐ด |
| Duration | 2.6 s | 2.49 s | 4.23% ๐ผ | ๐ข | |
| Hitch time ratio | 3.3 ms per s | 13.06 ms per s | -295.76% ๐ฝ | ๐ด | |
| Frame rate | 79 fps | 78.88 fps | 0.15% ๐ผ | ๐ข | |
| Number of hitches | 0.8 | 2.8 | -250.0% ๐ฝ | ๐ด | |
| ChannelList | Hitches total duration | 10.8 ms | 44.2 ms | -309.26% ๐ฝ | ๐ด |
| Duration | 2.6 s | 2.49 s | 4.23% ๐ผ | ๐ข | |
| Hitch time ratio | 4.2 ms per s | 17.79 ms per s | -323.57% ๐ฝ | ๐ด | |
| Frame rate | 76 fps | 73.73 fps | 2.99% ๐ผ | ๐ข | |
| Number of hitches | 0.8 | 4.0 | -400.0% ๐ฝ | ๐ด |
Closing the PR for now. The branch is still active.