shorebirdtech
fix: patch inflation failures (`pipe reader has been dropped`, `failed to fill whole buffer`)
We had two reports over Discord last night of patches failing on some devices.
app_id: 56869fe2-d053-4d24-9946-5bddb8f03e77 reports seeing this on iOS but not Android. https://discord.com/channels/1030243211995791380/1351103165772333147 https://admin.shorebird.dev/apps/56869fe2-d053-4d24-9946-5bddb8f03e77/releases/197194 Flutter 3.29.2
app_id: 1848aa9b-cda7-4c38-a557-0d2843ef8e6a reports seeing this on Android but not iOS: https://discord.com/channels/1030243211995791380/1351532251413741649 https://admin.shorebird.dev/apps/1848aa9b-cda7-4c38-a557-0d2843ef8e6a/releases/197456 Flutter 3.27.2
Both seem to have the same logs.
From Android:
03-18 17:47:40.861 3954 4168 E flutter : updater::updater: [shorebird] Decompression thread failed: pipe reader has been dropped
03-18 17:47:40.869 3954 4030 E flutter : updater::updater: [shorebird] Update failed: failed to fill whole buffer
03-18 17:47:40.869 3954 4030 I flutter : updater::updater: [shorebird] Update thread finished with status: Update had error
From iOS:
결함 02:24:15.734744+0900 Runner [shorebird] Decompression thread failed: pipe reader has been dropped
These failures are happening in (at least) Flutter 3.27.2, 3.29.2, and 3.32.6.
The code in question is here: https://github.com/shorebirdtech/updater/blob/main/library/src/updater.rs#L527
Which hasn't been changed in many months. Our Rust code also hasn't changed dependencies: https://github.com/shorebirdtech/updater/blob/main/Cargo.lock
Given that this is two separate versions, makes me wonder if something changed on the server side. Or if Google Storage could be sending invalid patch files which might just be failing in this odd way?
This code should be reading from/writing to disk: https://github.com/shorebirdtech/updater/blob/main/library/src/updater.rs#L508
@eseidel Is this fixed? I see this has been moved to Support, but not sure what that means.
I've not seen any reports since those initial two. It might have had something to do with backend troubles we were experiencing at the time, but since I've not been able to repro, I've not investigated further.
So the short answer is I assume it's resolved, but don't know.
@shaswatk
Not sure if this helps, its a workaround - if its still an issue.
ran into this on iOS a couple of weeks ago (Flutter 3.29.x patches worked fine for Android, but not for iOS — same "decompression thread failed" error - spotted on console app on mac).
To solve for iOS:
First, set a flutter version e.g. 3.27.1
dart pub global activate fvm
fvm install 3.27.1
fvm use 3.27.1
Then build the ipa:
fvm flutter build ipa
then run Shorebird release explicitly specifying:
shorebird release ios --flutter-version=3.27.1
hopefully I havent missed a step...
Love shorebird by the way!
@eseidel I am seeing the same behaviour with our latest release, pushed last week. Patches are going through on iOS, but not on Android. Can someone please look into this? We haven't been able to use Shorebird properly for over a month now due to this issue. It may be something to do with our account specifically, I don't know. But would really appreciate this being solved quickly. We were told this would be done by April, but looks like no one even looked at this.
04-28 23:42:22.427 3982 3982 I flutter : updater::cache::updater_state: [shorebird] No existing state file found: File /data/user/0/com.misfits.mobile/files/shorebird_updater/state.json does not exist, creating new state.
04-28 23:42:22.430 3982 3982 I flutter : [INFO:flutter/shell/common/shorebird/shorebird.cc(263)] Shorebird updater: no active patch.
04-28 23:42:22.431 3982 3982 I flutter : updater::updater: [shorebird] Reporting launch start.
04-28 23:42:22.431 3982 3982 I flutter : [INFO:flutter/shell/common/shorebird/shorebird.cc(286)] Starting Shorebird update
04-28 23:42:22.443 3982 4071 I flutter : updater::network: [shorebird] Sending patch check request: PatchCheckRequest { app_id: "1848aa9b-cda7-4c38-a557-0d2843ef8e6a", channel: "stable", release_version: "1.5.9+137", platform: "android", arch: "aarch64" }
04-28 23:42:22.509 3982 3982 I flutter : updater::updater: [shorebird] Reporting successful launch.
04-28 23:42:23.103 3982 4071 I flutter : updater::updater: [shorebird] Patch check response: PatchCheckResponse { patch_available: true, patch: Some(Patch { number: 1, hash: "29ef0687750dccad382936ed7baba731ae50809a360054e184294ad774e4e11b", download_url: "https://cdn.shorebird.cloud/api/v1/patches/1848aa9b-cda7-4c38-a557-0d2843ef8e6a/android/aarch64/193638/dlc.vmcode", hash_signature: None }), rolled_back_patch_numbers: Some([]) }
04-28 23:42:25.086 3982 4143 E flutter : updater::updater: [shorebird] Decompression thread failed: pipe reader has been dropped
04-28 23:42:25.087 3982 4071 E flutter : updater::updater: [shorebird] Update failed: failed to fill whole buffer
04-28 23:42:25.089 3982 4071 I flutter : updater::updater: [shorebird] Update thread finished with status: Update had error
@alanmacdonald1 Thanks for your input, but we already generate our builds using the flutter_version flag since our codebase gives errors with the latest version, so we end up using the version 3.27.2 for our builds.
Hi @eseidel, I hope you doing well. Unfortunately I'm now facing this issue with flutter version 3.29.0
For iOS here is the logs
default 10:56:54.808542+0500 Runner [shorebird] Reporting launch start.
default 10:56:54.808641+0500 Runner [INFO:flutter/shell/common/shorebird/shorebird.cc(270)] Starting Shorebird update
default 10:56:54.813107+0500 Runner [shorebird] Sending patch check request: PatchCheckRequest { app_id: "3ddab9e0-8e5a-4795-b52", channel: "stable", release_version: "1.2.1+415", platform: "ios", arch: "aarch64" }
default 10:56:54.828097+0500 Runner [shorebird] Reporting successful launch.
default 10:56:55.379053+0500 Runner [shorebird] Patch check response: PatchCheckResponse { patch_available: true, patch: Some(Patch { number: 2, hash: "hash_code", download_url: "https://cdn.shorebird.cloud/api/v1/patches/3ddab98e5a-4795-b52f/ios/aarch64/199979/dlc.vmcode", hash_signature: None }), rolled_back_patch_numbers: Some([1]) }
default 10:56:55.379164+0500 Runner [shorebird] Falling back from patch 1
default 10:56:55.379216+0500 Runner [shorebird] Downloading patch from: https://cdn.shorebird.cloud/api/v1/patches/3ddab9e0-8e5a-4795-b52f/ios/aarch64/199979/dlc.vmcode
default 10:56:56.214876+0500 Runner [shorebird] Writing patch to: "/private/var/mobile/Containers/Data/Application/E1066795-9986-4EAC-AB21/Library/Application Support/shorebird/shorebird_updater/downloads/2"
default 10:56:56.223876+0500 Runner [shorebird] Wrote patch to: "/private/var/mobile/Containers/Data/Application/E1066795-9986-4EAC-AB21/Library/Application Support/shorebird/shorebird_updater/downloads/2"
default 10:56:56.224747+0500 Runner [shorebird] Inflating patch from "/private/var/mobile/Containers/Data/Application/E1066795-9986-4EAC-AB21/Library/Application Support/shorebird/shorebird_updater/downloads/2"
fault 10:56:56.301891+0500 Runner [shorebird] Update failed: failed to fill whole buffer
default 10:56:56.301943+0500 Runner [shorebird] Update thread finished with status: Update had error
fault 10:56:56.302046+0500 Runner [shorebird] Decompression thread failed: pipe reader has been dropped
Hi @eseidel, I hope you doing well. Unfortunately I'm now facing this issue with flutter version 3.29.0
For iOS here is the logs
default 10:56:54.808542+0500 Runner [shorebird] Reporting launch start. default 10:56:54.808641+0500 Runner [INFO:flutter/shell/common/shorebird/shorebird.cc(270)] Starting Shorebird update default 10:56:54.813107+0500 Runner [shorebird] Sending patch check request: PatchCheckRequest { app_id: "3ddab9e0-8e5a-4795-b52", channel: "stable", release_version: "1.2.1+415", platform: "ios", arch: "aarch64" } default 10:56:54.828097+0500 Runner [shorebird] Reporting successful launch. default 10:56:55.379053+0500 Runner [shorebird] Patch check response: PatchCheckResponse { patch_available: true, patch: Some(Patch { number: 2, hash: "hash_code", download_url: "https://cdn.shorebird.cloud/api/v1/patches/3ddab98e5a-4795-b52f/ios/aarch64/199979/dlc.vmcode", hash_signature: None }), rolled_back_patch_numbers: Some([1]) } default 10:56:55.379164+0500 Runner [shorebird] Falling back from patch 1 default 10:56:55.379216+0500 Runner [shorebird] Downloading patch from: https://cdn.shorebird.cloud/api/v1/patches/3ddab9e0-8e5a-4795-b52f/ios/aarch64/199979/dlc.vmcode default 10:56:56.214876+0500 Runner [shorebird] Writing patch to: "/private/var/mobile/Containers/Data/Application/E1066795-9986-4EAC-AB21/Library/Application Support/shorebird/shorebird_updater/downloads/2" default 10:56:56.223876+0500 Runner [shorebird] Wrote patch to: "/private/var/mobile/Containers/Data/Application/E1066795-9986-4EAC-AB21/Library/Application Support/shorebird/shorebird_updater/downloads/2" default 10:56:56.224747+0500 Runner [shorebird] Inflating patch from "/private/var/mobile/Containers/Data/Application/E1066795-9986-4EAC-AB21/Library/Application Support/shorebird/shorebird_updater/downloads/2" fault 10:56:56.301891+0500 Runner [shorebird] Update failed: failed to fill whole buffer default 10:56:56.301943+0500 Runner [shorebird] Update thread finished with status: Update had error fault 10:56:56.302046+0500 Runner [shorebird] Decompression thread failed: pipe reader has been dropped
Im having same problem with 3.29.0 with IOS
Can someone please help me
Thanks for the update. We've still not been able to reproduce locally, so don't currently have any theories as to why this might have happened. Good to hear it's limited to older flutter versions.
dev...d.shorebird_code_push_example E updater::c_api: [shorebird] Error checking for update: Patch check request failed due to network error. Please check your internet connection.
Just got a report of this on discord for 3.32.7 https://discord.com/channels/1030243211995791380/1081696590147899575/1397648596068991156
Hey - please be aware that shorebird becomes unusable because of this issue and getting 0 patch installs on iOS. Appreciate all the work but thinking now to cut the subscription till this issue is fixed.
Facing same issue here - any solution or workaround for this?
To confirm @aghyad97 you're also seeing "pipe reader has been dropped" in your install messages, or something else? https://docs.shorebird.dev/faq/#patch-not-showing-up lists the most common causes if you aren't seeing "pipe reader has been dropped".
dev...d.shorebird_code_push_example E updater::c_api: [shorebird] Error checking for update: Patch check request failed due to network error. Please check your internet connection.
@JimmyZDD This is most commonly due to network conditions blocking connection to Shorebird's servers. Shorebird uses GCP and Cloudflare. https://docs.shorebird.dev/faq/#can-i-use-shorebird-in-my-country
Got another report via discord using Flutter 3.32.7: https://admin.shorebird.dev/apps/ab946920-13eb-4c82-a12d-3222646bb0f8/releases/291760 https://discord.com/channels/1030243211995791380/1401971164158361611
Log from discord:
Odd that the outer-mode error ("Update failed"): https://github.com/shorebirdtech/updater/blob/d47321b633176a2c2048a3a13a8658bebf96e1de/library/src/updater.rs#L677
is reported before the inner thread reports an error? https://github.com/shorebirdtech/updater/blob/d47321b633176a2c2048a3a13a8658bebf96e1de/library/src/updater.rs#L527
I would not have expected inflate to be triggered if the download had an error. And if the download succeeded, I would have exepcted an error from inflate before we saw the outer error about upload finishing?
We don't try to join the decompression thread, but maybe we need to?
No update. Working on hiring more engineers at the moment, we're spread a bit thin.