GoogleChromeLabs
bubblewrap update throwing: ERROR Failed to download Web Manifest responded with status 403
Describe the bug
When I try updating my app using command bubblewrap update --appVersionName="3.0.10"
Its throwing below error:
cli ERROR Failed to download Web Manifest https://www.mustakbil.com/manifest.json.Responded with status 403
When I try opening https://www.mustakbil.com/manifest.json in browser its working fine.
To Reproduce
I am not sure if the issue is with my manifest.json or project config.
I haven't made any changes to the project, I just run the update command on the same project which was working fine.
I use Cloudflare, so is there any way to check if Cloudflare is blocking access to manifest.json when using bubblewrap cli
Expected behavior It should work without any issue.
Desktop (please complete the following information):
- OS: Windows 10
Running: curl -v https://www.mustakbil.com/manifest.json from the command-line gives me a 403 error too:
* Trying 104.26.0.23...
* TCP_NODELAY set
* Connected to www.mustakbil.com (104.26.0.23) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
* start date: Jul 16 00:00:00 2020 GMT
* expire date: Jul 16 12:00:00 2021 GMT
* subjectAltName: host "www.mustakbil.com" matched cert's "*.mustakbil.com"
* issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fe6f100f800)
> GET /manifest.json HTTP/2
> Host: www.mustakbil.com
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 403
< date: Wed, 09 Jun 2021 10:08:04 GMT
< content-type: text/html; charset=UTF-8
< cf-chl-bypass: 1
< permissions-policy: accelerometer=(),autoplay=(),camera=(),clipboard-read=(),clipboard-write=(),fullscreen=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=()
< cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< expires: Thu, 01 Jan 1970 00:00:01 GMT
< x-frame-options: SAMEORIGIN
< cf-request-id: 0a91d760ca000040cbb78c4000000001
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v2?s=q12Ayr%2BgjNxYW3wQYZ92oB%2FgVB75gu%2BCHUh5Xvk50xUkN97JiGcw6H50cg4jrnOHyR0lNQdqeDdTfnchlP2%2BtWBDa9RoMt4WPjwuGzkduAjze%2FA7GSS%2B7QwbLBD%2FKQ%3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"report_to":"cf-nel","max_age":604800}
< strict-transport-security: max-age=2592000; includeSubDomains; preload
< x-content-type-options: nosniff
< server: cloudflare
< cf-ray: 65c98e7ad9e440cb-LHR
< alt-svc: h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400, h3=":443"; ma=86400
<
<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]> <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]> <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>
<title>Please Wait... | Cloudflare</title>
<meta name="captcha-bypass" id="captcha-bypass" />
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" type="text/css" media="screen,projection" />
<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" type="text/css" media="screen,projection" /><![endif]-->
<style type="text/css">body{margin:0;padding:0}</style>
<!--[if gte IE 10]><!-->
<script>
if (!navigator.cookieEnabled) {
window.addEventListener('DOMContentLoaded', function () {
var cookieEl = document.getElementById('cookie-alert');
cookieEl.style.display = 'block';
})
}
</script>
<!--<![endif]-->
<script type="text/javascript">
//<![CDATA[
(function(){
window._cf_chl_opt={
cvId: "2",
cType: "managed",
cNounce: "91257",
cRay: "65c98e7ad9e440cb",
cHash: "5662d05b89c0d53",
cFPWv: "g",
cTTimeMs: "4000",
cLt: "n",
cRq: {
ru: "aHR0cHM6Ly93d3cubXVzdGFrYmlsLmNvbS9tYW5pZmVzdC5qc29u",
ra: "Y3VybC83LjY0LjE=",
rm: "R0VU",
d: "UbDRT7xv9dgzj66utAPgjqwY76906W0nRkGRf8febscfqFcOaNCzOSYBXSWfa0fqr21uklCs3gV3yHse417Z5wIc+D8SUPzLFVE+SZurfs5n9sZOr33qEqD6PEQxFwLGYeyMM6+t841D5LGs8gvj4xvkylY+R7eJ1pljFMI/YpTY0LlIQ9FGPv6qaur996eKYhxJTqBDKOAtSqIJ89W4fJnnynld9tC6Y5N7uKt5hU0bWQaCeRT9RVohdZ6NFj2t4MujO4hsxIRDaXqkc3MQN3d/eiH0LnBPT7ac/SjmfnWK9KEbO8L0S+IGWb0YlODZb4u2u9Q8ukcXsHJtwkkw5uxKM/UDHSOInEemZmPKMOk0FY6nAlEATUqx1uLwdx9yBiOVo0rVzkw4ku5BOSaj5B7NhA6gVuJk3K15LllxW4faK+fzdsJPGNwlGxX2koa6V+PUpqdf0I7nxKyvmJss7vSofUYp9Cis+CbTS0zO38RgK70LmpKM3zQvzjrq9v/ScuJI63cjy8Vath0aI3+tA2RCeC/h3nbao0TGMwv5MqXVq+Qwsy9weDMQ1cl+bf5D2Nhbjvkx41Ul3Nq+hLlcFN3U1U8cc7zngo99ArXWBDHetVHlDLz43t5dDoB77/udU34N7av3nhoMI6s+MQfQz4YqnvU8RdOwwjen27o3YtO1EjoWr6TpwKY31bt/5DVT5iMEAxZFn++0VdbGd7poUiosU+N/0bfrRfECZg9DciuJhFzK6sOIw/7B0HOt7zQ+ZkWHj/IZAViI/LbV5V1tPx+rGrRUVeJbe/jfW6ibGb2TE+5nXn8Lv/Yjo2tCM4fpgXxJRCdIf5u4qAV5cCMRPfVFuefGWc1UXWQb9ZWqDeNVBzu5VHfzt70d/XlJGzVKOH6mzOQiGakcMe2RYWCdIaK3RQoYYxBdS7cvVJgTuH867zGt2dgEtDE5Q5GtW5BQ",
t: "MTYyMzIzMzI4NC4zMDYwMDA=",
m: "ZKy/2wYfLNKyj1YWcHXdkk5SpU/kQ9xMt7nbKZ5lqZk=",
i1: "g50nWVHKaqyDPdX1BU3qNg==",
i2: "DDNkH5hRu7Zw0bD5WbFZlg==",
zh: "NKTV0MtcWwOs0S4kCXGaNs7PzvA7BUxrA9LFHKvzFoA=",
uh: "szTQexRnAk6IzW67QzqGQgGxkzC/vAX/8LMiqwBnOyk=",
hh: "9Z1zHeNxOF+aFAMJQ36E9eL82239/cyQBTLnOIXqpqo=",
}
};
}());
//]]>
</script>
<style type="text/css">
#cf-wrapper #spinner {width:69px; margin: auto;}
#cf-wrapper #cf-please-wait{text-align:center}
.attribution {margin-top: 32px;}
.bubbles { background-color: #f58220; width:20px; height: 20px; margin:2px; border-radius:100%; display:inline-block; }
#cf-wrapper #challenge-form { padding-top:25px; padding-bottom:25px; }
#cf-hcaptcha-container { text-align:center;}
#cf-hcaptcha-container iframe { display: inline-block;}
@keyframes fader { 0% {opacity: 0.2;} 50% {opacity: 1.0;} 100% {opacity: 0.2;} }
#cf-wrapper #cf-bubbles { width:69px; }
@-webkit-keyframes fader { 0% {opacity: 0.2;} 50% {opacity: 1.0;} 100% {opacity: 0.2;} }
#cf-bubbles > .bubbles { animation: fader 1.6s infinite;}
#cf-bubbles > .bubbles:nth-child(2) { animation-delay: .2s;}
#cf-bubbles > .bubbles:nth-child(3) { animation-delay: .4s;}
</style>
</head>
<body>
<div id="cf-wrapper">
<div class="cf-alert cf-alert-error cf-cookie-error" id="cookie-alert" data-translate="enable_cookies">Please enable cookies.</div>
<div id="cf-error-details" class="cf-error-details-wrapper">
<div class="cf-wrapper cf-header cf-error-overview">
<h1 data-translate="managed_challenge_headline">Please wait...</h1>
<h2 class="cf-subheadline"><span data-translate="managed_checking_msg">We are checking your browser...</span> www.mustakbil.com</h2>
</div>
<div class="cf-section cf-highlight cf-captcha-container">
<div class="cf-wrapper">
<div class="cf-columns two">
<div class="cf-column">
<div class="cf-highlight-inverse cf-form-stacked">
<form class="challenge-form managed-form" id="challenge-form" action="/manifest.json?__cf_chl_managed_tk__=564f217630b311595cc9b7f4cd54e513f1f49e99-1623233284-0-AYVexoRyzFCBcL0IctL_LnBHTtlhyPIz_vyvnWgZkq0v5HEY4iJ6K4DhSamcfGZMIBzkDWFft8BNVzogYM8p9w5ke02CdOKFE0ooUjxSA5mXRWBiDBb6vMCMBpME9RshjSZKH2qtsnpIfzVghth_iHxK1WKtrz-fUssL61fmanEiHz92W0hAZzQQJo1tKsBHb130VxoHSybKP1SABTOSgTVMHtrw2b6_duqfJg6uWtUlDyCLksjGlpx1_TPLOTXO67JH2vkUc_bHczw-ymcM47eZmFYRhLz067y8kc4-E9wIWF24GXJQnJZiSO38gZi0piCnDRPGhweh5HUcPSAOiNxvSkl7gTpT4PVig_YaQKtEI6KdpOqSNWiG-T2xpNU8dt9y02e2-PZNb_Fva0HYDaouqf2E-AzbPrUw-tNHL-PMNCpJYjBUEtUbPas1PM2rG3KqEGVj_SPcEvqJ8gSkDBsjw-eLXK1Mar5CoCDbtkF2GXRPV6QHoMHQQpASxDcsxreBM8DHcslxx63tAk1GKqANcOTP79bmQ8vgAqxtBzPl-sxOkE6TvrYfZAegKD_E3INrzCEGHRChky1L0LNPvCqn0NpmsmjmWYjrv2lHjLwPK_eetKwwsCzUXds4lUPyXn4o80ZgWsYlr_TS2cvw6uR9BfU9UcRLeg2n39-eLFBUyyoXS0rJDNa4ISVPb1d1Eg" method="POST" enctype="application/x-www-form-urlencoded">
<div id='cf-please-wait'>
<div id='spinner'>
<div id="cf-bubbles">
<div class="bubbles"></div>
<div class="bubbles"></div>
<div class="bubbles"></div>
</div>
</div>
<p data-translate="please_wait" id="cf-spinner-please-wait">Please stand by, while we are checking your browser...</p>
<p data-translate="redirecting" id="cf-spinner-redirecting" style="display:none">Redirecting...</p>
</div>
<input type="hidden" name="r" value="c4733712ff50ef19808d21c8c7b9329d252bd2c0-1623233284-0-AXTEeF5iH58q2yotpy8yOiCmxrkl6YjDp6xZrBxaJeNpCF5AQ8QK1YNkvPdG3IPnRVUIycO+4S9vpKilIfCexI4uU82sbSh1qHHTAXqys9+1djGgGs1Y2a8+UdpE2z/h2jrwyLi+QaUlgNIYc1YULLsc+Vy5WbWj7PnjCgaImMaOag31HKdul/LmZE4n89tivSmal1rZ94kQc6DjYtNuIKMjNfkJbzUnBQagbJDbd8AjNpU9NW1Cit8i97v4/uzoKCzYI+7gvQ94jC0XdtAXmAk9E/EACpAcOrA7EkC3uoBHfDm6+8MZW1jIwYyVBkTFTzIlxZ+7I+5S7/AX24QEBFZY/GuqXLdS5NSWb4slOQHbVwCC1K2nkgAjX2t8VIFNgisPWuQRan1JXvVkRDAxYsAx4gvCGbU/q9oa+3AooUIET9GGs5u4gdjuoOyeV0qzsNis1gYGGGjAhVSSIHt9cj5z6c+YJ5gfRL2oIHC7LcvEFovekZxDAUuZXoYvXfQLMD5BNuHUDBQGb6r33vO1MYvc/Tu/WPlTsVOua4SUZEtvkXKKtqZLdfYneBIOK/ooMLRMJhrdgTrZKcD4tOWq9U98CrgaAgejZbMXRDZUIygGDdQfBhLMvnTJjbAGT7Jm2SgOkcWwRFH6sWx2ewOTr0vS4BiQ60iMnnmOdB2IAKCZOKDcx7jR3AtiS+e3u4fZRi3K+4iPT7mIeKbqAUB4u88WeQ60bT5Nwc2DPFu1ua0raSXntu/sLVlVlE/PvgHYc7XJbI0XL6rKtHigA0XPcd/5A0xagHkay3RNEBJaWdoQVPPvlgV/QfjIuKquElEJOHP2tTaeo3BsRIPe0iRDKdjD/rn2Yfgb+n0lbEeYF++KO+8yTVxA70hRQehApCBIQV5sVnOEas+8UPjHj4eUb8/06dE3bQqFI8MuM6aAVI5JgbvdMlXmwagFKAKasegLo5PK2D9+jhD5WyPrB6rafu/gNtX1B4bM1SpsV7VRwV/OCL8db7MUn/mN6pEggbi0bIJKQpEOqMzK9JHVAAZgBu+PYZpsdTKucr1bx3uwtk42VZc4hDfsx7+m7W3LpQsW8Onqqc34pvIN3Y+/IDLbA77c6+CLEfBM4n3LzVNqarvgqO5LS8aIgqmXisu4O4FS/OcBqTe3atIEztCpTsGhSII5Vr5N5yHjcUDpcURlh2skHkmTR1kjSXNBLXuZ1fLfPHgJmez7sBOmzDuFT+N7mGbiWdjknYVeV9lbr/Mj0Amt92/8rvjReJq5vAH0OP/tuVIEydKIk+tNFp1WrvzbACF+63eKTxlGR2mvl7vSQDhLpnURMXCNGrVGV0X7UhjG8Q==">
<input type="hidden" name="cf_captcha_kind" value="h">
<input type="hidden" name="vc" value="cda734bc48d884db3cf7f22d934ccf4f">
<noscript id="cf-captcha-bookmark" class="cf-captcha-info">
<h1 data-translate="turn_on_js" style="color:#bd2426;">Please turn JavaScript on and reload the page.</h1>
</noscript>
<div id="no-cookie-warning" class="cookie-warning" data-translate="turn_on_cookies" style="display:none">
<p data-translate="turn_on_cookies" style="color:#bd2426;">Please enable Cookies and reload the page.</p>
</div>
<script type="text/javascript">
//<![CDATA[
var a = function() {try{return !!window.addEventListener} catch(e) {return !1} },
b = function(b, c) {a() ? document.addEventListener("DOMContentLoaded", b, c) : document.attachEvent("onreadystatechange", b)};
b(function(){
var cookiesEnabled=(navigator.cookieEnabled)? true : false;
if(!cookiesEnabled){
var q = document.getElementById('no-cookie-warning');q.style.display = 'block';
}
});
//]]>
</script>
<div id="trk_captcha_js" style="background-image:url('/cdn-cgi/images/trace/captcha/nojs/h/transparent.gif?ray=65c98e7ad9e440cb')"></div>
</form>
<script type="text/javascript">
//<![CDATA[
(function(){
var isIE = /(MSIE|Trident\/|Edge\/)/i.test(window.navigator.userAgent);
var trkjs = isIE ? new Image() : document.createElement('img');
trkjs.setAttribute("src", "/cdn-cgi/images/trace/managed/js/transparent.gif?ray=65c98e7ad9e440cb");
trkjs.id = "trk_managed_js";
trkjs.setAttribute("alt", "");
document.body.appendChild(trkjs);
var cpo=document.createElement('script');
cpo.type='text/javascript';
cpo.src="/cdn-cgi/challenge-platform/h/g/orchestrate/managed/v1?ray=65c98e7ad9e440cb";
document.getElementsByTagName('head')[0].appendChild(cpo);
}());
//]]>
</script>
</div>
</div>
<div class="cf-column">
<div class="cf-screenshot-container">
<span class="cf-no-screenshot"></span>
</div>
</div>
</div>
</div>
</div>
<div class="cf-section cf-wrapper">
<div class="cf-columns two">
<div class="cf-column">
<h2 data-translate="why_captcha_headline">Why do I have to complete a CAPTCHA?</h2>
<p data-translate="why_captcha_detail">Completing the CAPTCHA proves you are a human and gives you temporary access to the web property.</p>
</div>
<div class="cf-column">
<h2 data-translate="resolve_captcha_headline">What can I do to prevent this in the future?</h2>
<p data-translate="resolve_captcha_antivirus">If you are on a personal connection, like at home, you can run an anti-virus scan on your device to make sure it is not infected with malware.</p>
<p data-translate="resolve_captcha_network">If you are at an office or shared network, you can ask the network administrator to run a scan across the network looking for misconfigured or infected devices.</p>
</div>
</div>
</div>
<div class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300">
<p class="text-13">
<span class="cf-footer-item sm:block sm:mb-1">Cloudflare Ray ID: <strong class="font-semibold">65c98e7ad9e440cb</strong></span>
<span class="cf-footer-separator sm:hidden">•</span>
<span class="cf-footer-item sm:block sm:mb-1"><span>Your IP</span>: 77.96.185.83</span>
<span class="cf-footer-separator sm:hidden">•</span>
<span class="cf-footer-item sm:block sm:mb-1"><span>Performance & security by</span> <a rel="noopener noreferrer" href="https://www.cloudflare.com/5xx-error-landing" id="brand_link" target="_blank">Cloudflare</a></span>
</p>
</div><!-- /.error-footer -->
</div>
</div>
<script type="text/javascript">
window._cf_translation = {};
</script>
</body>
</html>
* Connection #0 to host www.mustakbil.com left intact
* Closing connection 0
Can you check if running the below changes anything:
bubblewrap update --appVersionName="3.0.10" --fetchEngine="node-fetch"
My guess here is that Clouldflare is getting more agressive when blocking traffic from robots. They have a page with instructions around managing good bots.
It seems the solution here is to edit your robots.txt and explicitly allow robots to read manifest.json. You probably want to add the URL for any images in the Web Manifest too.
❗ ❗ Be aware that modifying your robots.txt file could block other bots on the web (eg: Googlebot) and make then unable to crawl your pages. Be careful and read the docs before implementing it ❗ ❗
@JudahGabriel FYI
bubblewrap update --appVersionName="3.0.10" --fetchEngine="node-fetch" is also throwing same error.
I think robots.txt doesn't enforce any access rules. It just act as a guideline for good bots i,e, which contents should they crawl and index and which content they should avoid indexing.
So apparently the only solution is to whitelist ip which could work when a user has a dedicated IP address but when user has a dynamic IP they will have to update firewall rule each time their IP changes.
Is it possible to provide path to manifest.json on local machine?
I think robots.txt doesn't enforce any access rules. It just act as a guideline for good bots i,e, which contents should they crawl and index and which content they should avoid indexing.
Yep, that's right, but it seems Cloudflare may be using robots.txt to block bots in their bot management system. Another option is to disable the bot management?
Is it possible to provide path to manifest.json on local machine?
Not yet. #401 is the feature request for this.
It's probably also worth reaching out to Cloudflare, to understand why it's being blocked (from both Bubblewrap and curl)
Cloudflare may be using robots.txt to block bots in their bot management system
Their website mentions this "Robots.txt is a file on a web server outlining the rules for bots accessing properties on that server. However, the file itself does not enforce these rules. Essentially, anyone who programs a bot is supposed to follow an honor system and make sure that their bot checks a website's robots.txt file before accessing the website. Malicious bots, of course, typically do not follow this system – hence the need for bot management."
I actually tried whitelisting my ip through their firewall, but even that didnt work.
I will also contact Cloudflare to find more details about the issue.