mozilla
building firebeat fails with "Generator response status code was 504"
Per @mwargers in bug 989041, attempting to generate an APK for Firebeat fails with the error "Generator response status code was 504":
07-31 14:40 > git clone [email protected]:loreV/firebeat.git Cloning into 'firebeat'... remote: Counting objects: 1037, done. remote: Compressing objects: 100% (2/2), done. remote: Total 1037 (delta 0), reused 0 (delta 0), pack-reused 1035 Receiving objects: 100% (1037/1037), 45.02 MiB | 1.16 MiB/s, done. Resolving deltas: 100% (215/215), done. Checking connectivity... done. 07-31 14:40 > mozilla-apk-cli firebeat firebeat.apk Building with https://controller-review.apk.firefox.com Generator response status code was 504
I note that a ZIP of the files in the repository takes up almost 2MB. Could this be an issue with the amount of data being transmitted?
Note that zipping the contents of the firebeat repository and uploading the package to https://marketplace.firefox.com/developers/validator succeeds with "Your app passed validation with no errors and 10 warnings."
https://marketplace.firefox.com/developers/upload/2c051ec57b4f4e77a4d1e4be405f256a
I am unable to reproduce this locally.
mozilla-apk-cli --endpoint http://localhost:8080 firebeat firebeat.apk
This command completes and I have a properly formatted apk, no 504.
When reproducing using your steps, it's interesting that it takes 1 minute and 46 seconds before it fails.
The 504 could be coming from an nginx gateway timeout issue, perhaps timeouts could be examined and put into this bug. I would set them high as some apks take a long time to build and transmit over the network.
Another piece of the puzzel is S3 which is used in storing and transmitting the app data.
I would look at the server logs for
- apk-controller.log
- apk-generator.log
- apk-signer.log
I don't think this apk is exceptionally large, did anything in the infrastructure change? Perhaps nginx rules were reset, the network topology has a lot more latency now (say moving part of the cluster to another hosting provider), etc.
Also: I've updated the README of apk-factory-service with couple minor updates https://github.com/mozilla/apk-factory-service/pull/87
I don't think this apk is exceptionally large, did anything in the infrastructure change? Perhaps nginx rules were reset, the network topology has a lot more latency now (say moving part of the cluster to another hosting provider), etc.
@andymckay, @dannycoates, @kumar303: Do y'all have any insight into what might be going on here?