stream-chat-swiftui icon indicating copy to clipboard operation
stream-chat-swiftui copied to clipboard

Implement performance benchmarks

Open testableapple opened this issue 1 year ago โ€ข 8 comments

๐Ÿ”— Issue Links

Resolves https://github.com/GetStream/ios-issues-tracking/issues/659

๐ŸŽฏ Goal

Implement Performance Benchmarks

๐Ÿ”– Notion

  • https://www.notion.so/stream-wiki/XCMetrics-5ef80c5ae9d44772966ab99f6c6d094a

๐Ÿ“ Summary

  • https://github.com/GetStream/stream-swift-performance-benchmarks stores the benchmarks
  • XCMetrics are executed on every PR (including release) by first PR's approval, manually on any branch and nightly by cron
  • XCMetrics won't be executed if there are no changes to the Sources folder
  • Collected metrics: :hitches_total_duration, :duration, :hitch_time_ratio, :frame_count, :frame_rate, :number_of_hitches
  • Metrics are collected from real devices on Firebase TestLab. Firebase offers us 5 runs per day (although, for some reason, sometimes it's even more than that), which should be enough most of the time (1 cron, 4 PRs). If this isn't enough, we can set up a paid account that offers free 30 minutes per day (1run โ‰ˆ 4min)
  • Our custom XCMetrics threshold equals to Xcode default one - 10%
  • Performance tests are executed on MessageList (with mocked static data: messages/images/videos/pdfs/giphies) and ChannelList (without mocks)
  • As far as I understood, Apple requires developers to run performance tests on the same machine + same iPhone (read more), which is impossible on CI. That's why there is no configured baseline in this PR, and the XCTests accordingly will always pass. Worries aside, because I implemented a custom calculation & assertion on the fastlane level.

๐ŸŽจ Showcase

target metric branch release/4.48.1 performance status
MessageList Hitches total duration 5.0 ms 20 ms 75.0% ๐Ÿ”ผ ๐ŸŸข
Duration 2.6 s 5 s 48.0% ๐Ÿ”ผ ๐ŸŸข
Hitch time ratio 2.0 ms per s 10 ms per s 80.0% ๐Ÿ”ผ ๐ŸŸข
Frame count 199 300 33.7% ๐Ÿ”ผ ๐ŸŸข
Frame rate 78.4 fps 100 fps 21.6% ๐Ÿ”ผ ๐ŸŸข
Number of hitches 0 1 100.0% ๐Ÿ”ผ ๐ŸŸข
ChannelList Hitches total duration 10.0 ms 20 ms 50.0% ๐Ÿ”ผ ๐ŸŸข
Duration 2.6 s 5 s 48.0% ๐Ÿ”ผ ๐ŸŸข
Hitch time ratio 3.9 ms per s 10 ms per s 61.0% ๐Ÿ”ผ ๐ŸŸข
Frame rate 74.7 fps 100 fps 25.3% ๐Ÿ”ผ ๐ŸŸข
Number of hitches 0 1 100.0% ๐Ÿ”ผ ๐ŸŸข

๐Ÿงช Manual Testing Notes

  • bundle exec fastlane xcmetrics

โ˜‘๏ธ Contributor Checklist

  • [x] I have signed the Stream CLA (required)
  • [x] This change follows zero โš ๏ธ policy (required)
  • [ ] This change should be manually QAed
  • [ ] Changelog is updated with client-facing changes
  • [ ] New code is covered by unit tests
  • [ ] Comparison screenshots added for visual changes
  • [ ] Affected documentation updated (docusaurus, tutorial, CMS)

๐ŸŽ Meme

gif

testableapple avatar Mar 06 '24 14:03 testableapple

6 Errors
:no_entry_sign: Please use more than one word.
793e29eb9da3743d9109d4c0182d47c7a59825be
:no_entry_sign: Please use more than one word.
e54eb7db7c9af9e8bb5feae886e3f31a9f40440d
:no_entry_sign: Please use more than one word.
091328b4b610dcc64bb25d1b7ab0b14f71e80a45
:no_entry_sign: Please start subject with capital letter.
4ad1505de4767411d5519ab49ab0ce8ca986a4b6
:no_entry_sign: Please use more than one word.
4ad1505de4767411d5519ab49ab0ce8ca986a4b6
:no_entry_sign: Please use more than one word.
6a4bf8a33eea6662b04de6fa013322775cea9e1b
1 Warning
:warning: Big PR

Generated by :no_entry_sign: Danger

github-actions[bot] avatar Mar 06 '24 14:03 github-actions[bot]

StreamChat XCMetrics

target metric release/4.49.0 branch performance status
MessageList Hitches total duration 20 ms 30.9 ms -54.5% ๐Ÿ”ฝ ๐Ÿ”ด
Duration 5 s 2.5 s 50.0% ๐Ÿ”ผ ๐ŸŸข
Hitch time ratio 10 ms per s 12.4 ms per s -24.0% ๐Ÿ”ฝ ๐Ÿ”ด
Frame rate 100 fps 79.3 fps 20.7% ๐Ÿ”ผ ๐ŸŸข
Number of hitches 1 2 -100.0% ๐Ÿ”ฝ ๐Ÿ”ด
ChannelList Hitches total duration 20 ms 38.4 ms -92.0% ๐Ÿ”ฝ ๐Ÿ”ด
Duration 5 s 2.5 s 50.0% ๐Ÿ”ผ ๐ŸŸข
Hitch time ratio 10 ms per s 15.4 ms per s -54.0% ๐Ÿ”ฝ ๐Ÿ”ด
Frame rate 100 fps 73.9 fps 26.1% ๐Ÿ”ผ ๐ŸŸข
Number of hitches 1 3 -200.0% ๐Ÿ”ฝ ๐Ÿ”ด

Stream-SDK-Bot avatar Mar 07 '24 12:03 Stream-SDK-Bot

StreamChat XCMetrics

target metric benchmark branch performance status
MessageList Hitches total duration 8.3 ms 30.0 ms -261.45% ๐Ÿ”ฝ ๐Ÿ”ด
Duration 2.6 s 2.48 s 4.62% ๐Ÿ”ผ ๐ŸŸข
Hitch time ratio 3.3 ms per s 12.11 ms per s -266.97% ๐Ÿ”ฝ ๐Ÿ”ด
Frame rate 79 fps 79.29 fps -0.37% ๐Ÿ”ฝ ๐ŸŸก
Number of hitches 0.7 2.6 -271.43% ๐Ÿ”ฝ ๐Ÿ”ด
ChannelList Hitches total duration 10.8 ms 37.6 ms -248.15% ๐Ÿ”ฝ ๐Ÿ”ด
Duration 2.6 s 2.49 s 4.23% ๐Ÿ”ผ ๐ŸŸข
Hitch time ratio 4.2 ms per s 15.03 ms per s -257.86% ๐Ÿ”ฝ ๐Ÿ”ด
Frame rate 75 fps 73.49 fps 2.01% ๐Ÿ”ผ ๐ŸŸข
Number of hitches 0 3.6 -360.0% ๐Ÿ”ฝ ๐Ÿ”ด

Stream-SDK-Bot avatar Mar 08 '24 10:03 Stream-SDK-Bot

StreamChat XCMetrics

target metric benchmark branch performance status
MessageList Hitches total duration 8.3 ms 24.2 ms -191.57% ๐Ÿ”ฝ ๐Ÿ”ด
Duration 2.6 s 2.5 s 3.85% ๐Ÿ”ผ ๐ŸŸข
Hitch time ratio 3.3 ms per s 9.7 ms per s -193.94% ๐Ÿ”ฝ ๐Ÿ”ด
Frame rate 79 fps 79.33 fps -0.42% ๐Ÿ”ฝ ๐ŸŸก
Number of hitches 0.7 2.4 -242.86% ๐Ÿ”ฝ ๐Ÿ”ด
ChannelList Hitches total duration 10.8 ms 35.9 ms -232.41% ๐Ÿ”ฝ ๐Ÿ”ด
Duration 2.6 s 2.47 s 5.0% ๐Ÿ”ผ ๐ŸŸข
Hitch time ratio 4.2 ms per s 14.54 ms per s -246.19% ๐Ÿ”ฝ ๐Ÿ”ด
Frame rate 75 fps 73.94 fps 1.41% ๐Ÿ”ผ ๐ŸŸข
Number of hitches 0 3.2 -320.0% ๐Ÿ”ฝ ๐Ÿ”ด

Stream-SDK-Bot avatar Mar 08 '24 10:03 Stream-SDK-Bot

StreamChat XCMetrics

target metric benchmark branch performance status
MessageList Hitches total duration 8.3 ms 49.2 ms -492.77% ๐Ÿ”ฝ ๐Ÿ”ด
Duration 2.6 s 2.49 s 4.23% ๐Ÿ”ผ ๐ŸŸข
Hitch time ratio 3.3 ms per s 19.75 ms per s -498.48% ๐Ÿ”ฝ ๐Ÿ”ด
Frame rate 79 fps 78.57 fps 0.54% ๐Ÿ”ผ ๐ŸŸข
Number of hitches 0.8 4.0 -400.0% ๐Ÿ”ฝ ๐Ÿ”ด
ChannelList Hitches total duration 10.8 ms 40.9 ms -278.7% ๐Ÿ”ฝ ๐Ÿ”ด
Duration 2.6 s 2.49 s 4.23% ๐Ÿ”ผ ๐ŸŸข
Hitch time ratio 4.2 ms per s 16.41 ms per s -290.71% ๐Ÿ”ฝ ๐Ÿ”ด
Frame rate 76 fps 73.89 fps 2.78% ๐Ÿ”ผ ๐ŸŸข
Number of hitches 0.8 3.8 -375.0% ๐Ÿ”ฝ ๐Ÿ”ด

Stream-SDK-Bot avatar Mar 08 '24 14:03 Stream-SDK-Bot

StreamChat XCMetrics

target metric benchmark branch performance status
MessageList Hitches total duration 8.3 ms 60.1 ms -624.1% ๐Ÿ”ฝ ๐Ÿ”ด
Duration 2.6 s 2.49 s 4.23% ๐Ÿ”ผ ๐ŸŸข
Hitch time ratio 3.3 ms per s 24.12 ms per s -630.91% ๐Ÿ”ฝ ๐Ÿ”ด
Frame rate 79 fps 77.79 fps 1.53% ๐Ÿ”ผ ๐ŸŸข
Number of hitches 0.8 4.2 -425.0% ๐Ÿ”ฝ ๐Ÿ”ด
ChannelList Hitches total duration 10.8 ms 41.7 ms -286.11% ๐Ÿ”ฝ ๐Ÿ”ด
Duration 2.6 s 2.48 s 4.62% ๐Ÿ”ผ ๐ŸŸข
Hitch time ratio 4.2 ms per s 16.82 ms per s -300.48% ๐Ÿ”ฝ ๐Ÿ”ด
Frame rate 76 fps 73.81 fps 2.88% ๐Ÿ”ผ ๐ŸŸข
Number of hitches 0.8 3.6 -350.0% ๐Ÿ”ฝ ๐Ÿ”ด

Stream-SDK-Bot avatar Mar 08 '24 14:03 Stream-SDK-Bot

StreamChat XCMetrics

target metric benchmark branch performance status
MessageList Hitches total duration 8.3 ms 32.5 ms -291.57% ๐Ÿ”ฝ ๐Ÿ”ด
Duration 2.6 s 2.49 s 4.23% ๐Ÿ”ผ ๐ŸŸข
Hitch time ratio 3.3 ms per s 13.06 ms per s -295.76% ๐Ÿ”ฝ ๐Ÿ”ด
Frame rate 79 fps 78.88 fps 0.15% ๐Ÿ”ผ ๐ŸŸข
Number of hitches 0.8 2.8 -250.0% ๐Ÿ”ฝ ๐Ÿ”ด
ChannelList Hitches total duration 10.8 ms 44.2 ms -309.26% ๐Ÿ”ฝ ๐Ÿ”ด
Duration 2.6 s 2.49 s 4.23% ๐Ÿ”ผ ๐ŸŸข
Hitch time ratio 4.2 ms per s 17.79 ms per s -323.57% ๐Ÿ”ฝ ๐Ÿ”ด
Frame rate 76 fps 73.73 fps 2.99% ๐Ÿ”ผ ๐ŸŸข
Number of hitches 0.8 4.0 -400.0% ๐Ÿ”ฝ ๐Ÿ”ด

Stream-SDK-Bot avatar Mar 08 '24 15:03 Stream-SDK-Bot

Closing the PR for now. The branch is still active.

testableapple avatar Apr 15 '24 09:04 testableapple