RFC: Usage of experimentalForceLongPolling option.

[mikelehen]

Are you experiencing the error Could not reach Cloud Firestore backend. Backend didn't respond within 10 seconds despite your network seeming healthy? Please visit https://debug-my.firebaseapp.com/ and check the results at the end. If the tests "with default options" fail (or are very slow) but the "with forceLongPolling" ones succeed, then this indicates your traffic is likely going through a proxy that is buffering responses in a way that is not compatible with Firestore.

As a workaround, you can force long-polling as follows:

firebase.firestore().settings({ experimentalForceLongPolling: true });

Have you enabled experimentalForceLongPolling and experimentalAutoDetectLongPolling to solve a reproducible connection issue related to a specific environment?

We would like to know:

1. What environment causes the problem (app platform, antivirus software, network proxy, network conditions, etc.)?
2. What is the behavior without experimentalForceLongPolling or experimentalAutoDetectLongPolling?
3. Does experimentalAutoDetectLongPolling completely resolve the issue?
4. Does experimentalForceLongPolling completely resolve the issue?
5. Please visit https://debug-my.firebaseapp.com/, wait for "All tests done" (about 60 seconds), paste the results into a gist, and paste the link in your comment.

If there is an existing comment concerning the environment you are using, feel free to just add a 👍 to it.

[tgangso]

Hi,

I was going to enable this feature because it describes an issue that sounds similar to what many of my users are describing when using my app on corporate networks.

But from my testing this results in an absurd amount of network requests when querying large (5- 10000+ documents) collections (using "@angular/fire": "5.1.2" + "firebase": "5.10.0", with enablePersistence), it looks like it gets one channel request for every document in the collection, causing very long loading times waiting for the result. And I found it unusable in its current state and have deactivated it again. After deactivating I am back to normal amount of network requests (1 for the collection).

Hope it can be improved as many users report issues that sounds like this feature would fix.

[mikelehen]

@tgangso Thanks for the feedback! (and sorry it didn't work out for you). We'll discuss to see if we can do anything about the performance. In the meantime, were you able to determine if this solved any issues for your users? And if so, can you comment more on the corporate network environment that was causing issues? Thanks.

[tgangso]

@mikelehen I have not been able to reproduce the issue myself, but generally users on Equinor/Statoil corporate network experience the issue, also some users with adblocker (VPN based i think) have reported this.

I never deployed the setting to production because of the performance issues, so I am unable to confirm if it helps or not, but I have some users that can help testing when/if performance is improved.

[tgangso]

I did end up adding an option for the user to enable this manually, and have asked a few users to try it out, and it did indeed help with the issue, and they where able to use the application.

So I hope this option is not removed, as it will help even if performance is lower. Not all my users have huge amount of documents.

One user had issues on his Mac, same network as his phone (where it works). gist.

One user is on a "high security" corporate network, gist.

Behaviour without experimentalForceLongPolling, is no data is retrieved or being sent to firestore, but works when enabled.

Hope it helps.

Maybe it could be possible for firebase-js to detect when this is needed as a fallback?

[tgangso]

Added a gist for the mac scenario also.

Let me know if you want me to test anything else.

[mikelehen]

@tgangso Thanks very much for the data. We'll plan to keep the option around unless the problem stops happening or we devices a better solution (like automatic fallback).

[tgangso]

@mikelehen great, I can confirm a few others have tried the setting with success. Automatic fallback would be great if possible.

[mikelehen]

@tgangso FYI- There was a backend change that should cut down on the number of network requests when using this option. Would you be interested in trying it again and reporting on whether it behaves better now?

[tgangso]

@mikelehen i did some quick testing now, and it definately feels better, looks like there are more documents per request now. Will keep the setting on for a bit and test.

[rodrigoreis22]

We're looking into adding this as a fallback for our web application. Some small percentage of users get the message Could not reach firestore backend and we're using the latest sdk version.

Is it possible to enable this setting after the app is already initialized only in case of a failure reaching the firestore backend? Ideally I want to enable this only on a catch flow .

[mikelehen]

@rodrigoreis22 Not super easily. But you can call firebase.app().delete() and then call firebase.initializeApp(...) again, and then firebase.firestore() will return a brand new instance which you can call .settings({experimentalForceLongPolling: true}) on... but note that any existing writes / listeners / etc. will need to be reestablished.

[dubvi5]

Hi. I cannot connect to firestore without experimentalForceLongPolling.

1. Windows, on a corporate network with McAfee endpoint security.

2. On first load I get "Could not reach Cloud Firestore backend. Backend didn't respond within 10 seconds.", but it works if I reload the page.

3. experimentalForceLongPolling seems to completely resolve the issue.

4. https://gist.github.com/dubvi5/1731f17940531d94451aeded761d08ad

Thanks.

[dcc82]

Hello, we have a Firestore connection issue with one of our computers at work. The error message says "Firestore (6.3.0): Could not reach Cloud Firestore backend. Backend didn't respond within 10 seconds..."

This issue doesn't occur on the other computers within the same network. Enabling ExperimentalForceLongPolling seems to resolve the issue.

The problematic one is a Windows PC with "Symantec Endpoint Protection" installed. The others are all Mac computers without antivirus.

Here's the gist: https://gist.github.com/dcc82/270cc8eac2f0fe312aa4ac8befad7925

Thanks.

[bhar4t]

Hi

I enabled the feature and I tried feature enabled project with online, mid-range mobile and low-end mobile network options of chrome debugger.

I still getting error message as "Firestore (6.3.0): Could not reach Cloud Firestore backend. Backend didn't respond within 10 seconds..." in only on low-end mobile. except both is working fine in the both condition whether with feature enabled or not.

[mikelehen]

@bhar4t Thanks for the report. experimentalForceLongPolling is meant to work around a specific "broken" network configuration that's unrelated to your connection speed. So I wouldn't expect it to have an effect when testing in Chrome Debugger regardless of your emulated connection speed (low-end mobile, etc.).

Low-end mobile is likely hitting that log message just because it's taking longer than 10 seconds, but you should still get results if you wait long enough... it'll just be slower due to the low connection speed. experimentalForceLongPolling won't help.

[abrice]

Hi,

• Firebase Auth works in all environments (with and without long polling).
• Firebase Firestore works in localhost but fails completely in corporate environment with Win10/WinDefender/Bluecoat layers. The corporate environment is a very standard.

With experimentalForceLongPolling set to true, the corporate environment works beautifully with excellent performance and responsiveness. The gist is here: https://gist.github.com/abrice/de6984ffb7361cf4c071d574e49eb552.

Note, we only use the web (no mobile).

So, the issue then becomes, is 'experimental' going to become 'production'? It seems very risky to proceed with a solution that may or may not be withdrawn at any time.

The Firebase Auth/Firestore solution is truely impressive and ForceLongPolling works! Please move it to production status.

[mikelehen]

@abrice Thanks for the feedback!

The high-level answer is:

1. We aren't currently planning to make experimentalForceLongPolling the default behavior because it has some performance drawbacks and it should not be necessary for the vast majority of users (but we're using this github issue to collect / monitor feedback in that regard).
2. That said, we have no intention to remove experimentalForceLongPolling until / unless we have a better option for any users that are encountering the connection issues it currently solves.

For your specific case, can you elaborate on "Bluecoat layers"? I guess Blue Coat Systems was acquired by Symantec and perhaps this is the current version of their proxy software: https://www.symantec.com/products/proxy-sg-and-advanced-secure-gateway ?

It looks like it is known to cause problems do to SSL interception / proxying, e.g. with Windows Update and Office 365. So I'm not at all surprised it causes issues for Firestore as well, although interestingly this is the first report I've seen.

I don't suppose you or your IT department would be interested in engaging with Symantec on this issue and see if it can be resolved? I'd be happy to participate in the discussion and provide technical details, etc. If you're interested, feel free to reach out / CC me at [email protected]. Thanks!

[tgangso]

Any update on if it is possible to make it fall back automatically to forceLongPolling?

I agree there are performance drawbacks, and should not be the default option.

I get a few questions every week from users experiencing problems (mostly on corporate networks), and enabling this in the settings of my app always resolve it.

[abrice]

Hi @mikelehen ,

The corporate runs O365 and has dealt with the challenges of Windows Updates. The network side is complex as there are a few other parties involved that provision various levels of proxy servers, anti-virus, load balancing, etc as-a-service. There's quite a lot of complex traffic moving about so, unfortunately, I can't help any further on pursuing a more in-depth technical examination.

But it's excellent news about your approach to experimentalForceLongPolling. Thank you!

[mikelehen]

@tgangso Adding a fallback mechanism is still on our radar but not something we're actively pursuing. Unfortunately it would be pretty complicated to add since it's hard to differentiate this case from being offline or even just on a slow network, so it's not obvious when to employ the fallback, etc. Right now we're still gathering feedback on the scope of the problem. Thanks!

[DJ-Shady02]

Hi @mikelehen

I have been seeking help from Firebase Support, and after one and a half week we came up with changing the firestore settings to experimentalForceLongPolling:true. With this setting, I was able to reduce loading times (when reading through .get()) from 50-60 seconds to 137 milliseconds. Before enabling forceLongPolling I would get a error when trying to add anything due to timeout.

When using any website requesting data from Cloud Firestore, I am unable to use the site. On my own however, I enable the setting simply for me to be able to actually use it. I have been working on this from two different computers, and tried on to different networks as well. Furthermore, I have tried disabling anything from proxies to firewalls - even Windows Defender Firewall. Nothing helped except enabling forceLongPolling.

[mikelehen]

@DJ-Shady02 Thank you for the information! Usually this issue only manifests in the face of specific corporate proxy or antivirus software, etc., so the fact that you're seeing it on two different computers and two different networks, even with disabling proxies / firewalls is unusual. I would still guess that there's some common factor that's causing it to happen, but it's hard to know what it might be. If by chance you're able to collect any more information that helps isolate the root cause (by trying from yet more devices including phones or tablets) or more networks (coffee shop, etc.), let us know. Thanks!

[remyayad]

Hello Michael, I am facing an issue with my angular app not loading data from Firestore in a timely manner. The results to your test are displayed in the following gist : https://gist.github.com/remyayad/db36b4434d0f8628f65d236276e9c754

This happens only the network of the company I work for. Unfortunately the entire IT infrastructure is managed by a third-party company, therefore I cannot determine what ports are being blocked, if proxies are used, firewall security policies, etc. The anti-virus software is Symantec Endpoint Protection and it seems as though there is a web filter probably implemented via a proxy to filter web pages we go on. It's called zScaler. The angular app displays just fine, but the content that is stored in Firestore doesn't appear. If I let the computer there for an hour or two, I can see that the page I was on has the data from Firestore loaded (God knows how long it took for the data to appear). I enabled the experimentalForceLongPolling option for Firebase, but seeing the results of the debug utility, I certainly must be doing something wrong...

[mikelehen]

@remyayad Hrm. Your results aren't what I would expect to see with the "normal" proxy issue. It looks like the test page worked for you both with and without forceLongPolling. But it sounds like in your Angular app, Firestore isn't working (well) regardless of whether you enable experimentalForceLongPolling.

Can you open a separate github issue so we can investigate? It would be helpful if you could reproduce the issue in a fairly minimal app (remove Angular, etc. if possible) and then capture a Firestore log (add firebase.firestore.setLogLevel('debug') to your app) and a HAR file (load the page with chrome dev tools Network tab open, wait ~90 seconds, then right click and choose "Save all as HAR with content").

[remyayad]

@mikelehen Thanks for the quick reply! Will do and I'll keep you posted.

[DJ-Shady02]

@mikelehen Finally, I have an update!

I have bought myself a new computer. Used Firestore for a month without issues. I install Bullguard, bam! Same issue as before. Being assured the issue was created by Bullguard, I contacted their support. It appears their "Safe Surfing" (The function name is translated from my native language, so it may vary), is causing the issue. Disabling it will trade a warning notification on sketchy websites with fully functional firestore without using forceLongPolling - worth it!

While this is not directly helpful to you on the topic of the experimentalForceLongPolling option, it can help others avoid it.

[wilhuff]

@DJ-Shady02 Thanks for the report!

We're fairly certain that this problem isn't going away and are working on building logic into the SDK that will be able to automatically fall back on long polling.

[tgangso]

working on building logic into the SDK that will be able to automatically fall back on long polling.

Great news, thank you.

[prescottprue]

Thanks a bunch for making this setting - it seems to solve issues with Firestore loading in Cypress and has been proposed as a possible solution to a note about bad performance of Firestore in Cypress.

[wilhuff]

@Kretin1 This issue is not a general discussion forum--we're collecting feedback on specific circumstances in which experimentalForceLongPolling has helped or hurt.

It sounds like you may have some unique concerns so please file an issue describing exactly which components you're using and what you're trying to do.