nightscout-librelink-up
nightscout-librelink-up copied to clipboard
timoschlueter
You have been blocked?.. Then - Error(`failed to get last entry)
How to resolve it? While starting the script:
\n\n\n\n \n\nSorry, you have been blocked
\nYou are unable to access libreview.io
\nWhy have I been blocked?
\n\nThis website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.
\nWhat can I do to resolve this?
\n\nYou can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.
\n
I'm not using this product, but when analyzing what the error might be on my own script, I came across this issue.
I found adding the same headers that are present when you visit www.libreview.com got through cloudflare blocking the request.
To the devs here, adding the following headers appeased Cloudflare:
$headers = [
'product: llu.android',
'version: 4.7.0',
'Content-type: application/json',
'accept: application/json, text/plain, */*',
'accept-language: en-US,en;q=0.9',
'cache-control: no-cache',
'content-type: application/json;charset=UTF-8',
'newyu-lv-web-version: 3.16.19',
'origin: https://www.libreview.com',
'pragma: no-cache',
'referer: https://www.libreview.com/',
'sec-ch-ua: "Google Chrome";v="123", "Not:A-Brand";v="8", "Chromium";v="123"',
'sec-ch-ua-mobile: ?0',
'sec-ch-ua-platform: "macOS"',
'sec-fetch-dest: empty',
'sec-fetch-mode: cors',
'sec-fetch-site: cross-site',
'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36',
];
to the login endpoint
and then adding the same ones (with an Authorization header) to the graph endpoint.
I hope this helps everyone here keep getting their data.
I've tested this and for me at least, simply adding the additional headers did not work. What's interesting is that running the programme using npm start works perfectly, whereas running inside a docker container returns the above error, using the same user credentials in both cases.
I think the problem is to do with cookies not being captured correctly when running in docker as indicated by this line:
if (!navigator.cookieEnabled) {\n window.addEventListener('DOMContentLoaded', function () {\n var cookieEl = document.getElementById('cookie-alert');\n cookieEl.style.display = 'block';
I'll continue to look into this problem and hopefully come up with a resolution.
I am on vacation right now but I will take a look at this problem as soon as monday.
I don't think I am handling cookies at all in the current version. Implementing a cookie jar in the axios calls might solve this issue.
Thank you all for investigating and bringing this up!
@timoschlueter I think you're exactly right. I'm happy to help so I can try to update this and create a PR for you. I have some understanding of axios and cookiejar so i'll see what I can do.
That's awesome! Feel free to create a PR whenever you have something ready. I will be able to review and merge from my phone. :)
@timoschlueter I've spent a couple of hours trying to resolve this, but can't seem to get it working within a dockerized instance.
I've committed my changes here if this helps you at all. It is still not working, but maybe what I've done will save you some time.
To confirm, the programme works perfectly when I run it locally using npm start (or with the shell script that you suggested in the readme), but as soon as I try to run it in a docker container, Cloudflare blocks the request.
Cloudflare is returning a cookie named __cf_bm, and using cookiejar is capturing this cookie and sending it on any subsequent requests, both inside and out of the docker container. I've also tried to ping the login endpoint using a simple curl request, which returns the correct data. and I've also entered the docker container and ran the same curl request, which also returns the correct response.
There seems to be some very slight difference somewhere between the dockerized and non-dockerized requests, but I cannot find any variation. I've tried logging both the HTTP request and the reponse from LibreView both inside and outside of docker and cannot find any differences here.
Maybe I've missed something, but hopefully this information is helpful to you. I'll try to look at this again maybe tomorrow or Monday. Maybe a fresh set of eyes will notice something blindingly obvious.
EDIT: -
I just tried running this using npm start on a remote ubuntu server and I'm seeing the same Cloudflare error message over there with or without using CookieJar.
ANOTHER EDIT: - i noticed that I was using an older version of Node locally, and this was working. Downgrading the node version in the Dockerfile has fixed this issue. This may not be the most ideal fix (I imagine you'd want to use a newer version of Node where possible), however I think it will serve as a temporary solution if nothing else. I'll open up a PR and you can decide if you want to merge this change or not.
PR #130 opened.
Even with node v 16.16.0:
root@ovz3:~/l3/nightscout-librelink-up# node -v
v16.16.0
root@ovz3:~/l3/nightscout-librelink-up# bash start.sh
> [email protected] start
> npx ts-node --transpile-only src/index.ts
[info]: Starting cron schedule: */1 * * * *
[info]: renew token
[error]: "<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]> <html class=\"no-js ie7 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 8]> <html class=\"no-js ie8 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if gt IE 8]><!--> <html class=\"no-js\" lang=\"en-US\"> <!--<![endif]-->\n<head>\n<title>Attention Required! | Cloudflare</title>\n<meta charset=\"UTF-8\" />\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\" />\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge\" />\n<meta name=\"robots\" content=\"noindex, nofollow\" />\n<meta name=\"viewport\" content=\"width=device-width,initial-scale=1\" />\n<link rel=\"stylesheet\" id=\"cf_styles-css\" href=\"/cdn-cgi/styles/cf.errors.css\" />\n<!--[if lt IE 9]><link rel=\"stylesheet\" id='cf_styles-ie-css' href=\"/cdn-cgi/styles/cf.errors.ie.css\" /><![endif]-->\n<style>body{margin:0;padding:0}</style>\n\n\n<!--[if gte IE 10]><!-->\n<script>\n if (!navigator.cookieEnabled) {\n window.addEventListener('DOMContentLoaded', function () {\n var cookieEl = document.getElementById('cookie-alert');\n cookieEl.style.display = 'block';\n })\n }\n</script>\n<!--<![endif]-->\n\n\n</head>\n<body>\n <div id=\"cf-wrapper\">\n <div class=\"cf-alert cf-alert-error cf-cookie-error\" id=\"cookie-alert\" data-translate=\"enable_cookies\">Please enable cookies.</div>\n <div id=\"cf-error-details\" class=\"cf-error-details-wrapper\">\n <div class=\"cf-wrapper cf-header cf-error-overview\">\n <h1 data-translate=\"block_headline\">Sorry, you have been blocked</h1>\n <h2 class=\"cf-subheadline\"><span data-translate=\"unable_to_access\">You are unable to access</span> libreview.io</h2>\n </div><!-- /.header -->\n\n <div class=\"cf-section cf-highlight\">\n <div class=\"cf-wrapper\">\n <div class=\"cf-screenshot-container cf-screenshot-full\">\n \n <span class=\"cf-no-screenshot error\"></span>\n \n </div>\n </div>\n </div><!-- /.captcha-container -->\n\n <div class=\"cf-section cf-wrapper\">\n <div class=\"cf-columns two\">\n <div class=\"cf-column\">\n <h2 data-translate=\"blocked_why_headline\">Why have I been blocked?</h2>\n\n <p data-translate=\"blocked_why_detail\">This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.</p>\n </div>\n\n <div class=\"cf-column\">\n <h2 data-translate=\"blocked_resolve_headline\">What can I do to resolve this?</h2>\n\n <p data-translate=\"blocked_resolve_detail\">You can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.</p>\n </div>\n </div>\n </div><!-- /.section -->\n\n <div class=\"cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300\">\n <p class=\"text-13\">\n <span class=\"cf-footer-item sm:block sm:mb-1\">Cloudflare Ray ID: <strong class=\"font-semibold\">86d040*******</strong></span>\n <span class=\"cf-footer-separator sm:hidden\">•</span>\n <span id=\"cf-footer-item-ip\" class=\"cf-footer-item hidden sm:block sm:mb-1\">\n Your IP:\n <button type=\"button\" id=\"cf-footer-ip-reveal\" class=\"cf-footer-ip-reveal-btn\">Click to reveal</button>\n <span class=\"hidden\" id=\"cf-footer-ip\">81.177.****</span>\n <span class=\"cf-footer-separator sm:hidden\">•</span>\n </span>\n <span class=\"cf-footer-item sm:block sm:mb-1\"><span>Performance & security by</span> <a rel=\"noopener noreferrer\" href=\"https://www.cloudflare.com/5xx-error-landing\" id=\"brand_link\" target=\"_blank\">Cloudflare</a></span>\n \n </p>\n <script>(function(){function d(){var b=a.getElementById(\"cf-footer-item-ip\"),c=a.getElementById(\"cf-footer-ip-reveal\");b&&\"classList\"in b&&(b.classList.remove(\"hidden\"),c.addEventListener(\"click\",function(){c.classList.add(\"hidden\");a.getElementById(\"cf-footer-ip\").classList.remove(\"hidden\")}))}var a=document;document.addEventListener&&a.addEventListener(\"DOMContentLoaded\",d)})();</script>\n</div><!-- /.error-footer -->\n\n\n </div><!-- /#cf-error-details -->\n </div><!-- /#cf-wrapper -->\n\n <script>\n window._cf_translation = {};\n \n \n</script>\n\n</body>\n</html>\n"
[error]: Invalid authentication token. Please check your LibreLink Up credentials Cannot read properties of undefined (reading 'status')
[error]: LibreLink Up - No AuthTicket received. Please check your credentials.
@AndyLow91 Yes, I just git cloned the "cookies" branch :
git clone --branch=Cookies https://github.com/Nightscout-Pro/nightscout-librelink-up/
I was able to dig into this issue and found out that this is not an issue that is cookie related but rather coming from a rate limit that Abbott has put on the LibreLink endpoints. I sent a few requests from Postman immediately one after another and after a few requests, the endpoint returned HTTP 429 Too Many Requests. I am now waiting for the rate limit to cool down and try again.
The rate limit seems to be user-account bound. I am not able to connect to the API from my MacBook any more since I hit the reate limit, and also checked from my AWS Lightsail instance where I have my Nightscout running. I now also get the same error on my Lightsail instance.
The question now is: Why are you all running into the rate limit? I only triggered the rate limit by bursting requests to the LibreLink endpoint. Before that, my instance was working fine with a configured interval of 5 minutes.
@AndyLow91, @librestore24: What interval have you set up on your instance?
I am now waiting for the rate limit to cool down so that I can check further.
@timoschlueter I have mine set to 1 minute interval, and since I made the changes here, everything is running fine for me now.
I think they probably do have a rate limit set, and that you hit it when you sent multiple requests in a short burst, but the problem described in this issue is not to do with a 429 error, so maybe that's not an issue for regular users.
@AndyLow91: I merged your PR into the release/2.5.1 branch but i am not able to get it working on my machine. I get a 403 Forbidden error when i start the script. From Postmen, everything works fine.
From which region are you testing? I have the region set to "DE".
@timoschlueter Yes, when I was testing, I was also using DE region. I've just ran it again and it is working still for me. Although the user that i was testing this our for has since told me that there is a problem with LibreLinkUp and no data being shared. I think this is a problem with Abbott though.
I noticed on Abbott's LibreView login page that they are doing some updates on their service and posted that there may be some downtime (although they said it would be on 2nd April so I'm not sure if this is related.)
Running the script now, I am getting:
[info]: Starting cron schedule: */1 * * * *
[info]: renew token
[info]: Logged in to LibreLink Up
[info]: Found 1 LibreLink Up connection.
[info]: -> The following connection will be used: ***REDACTED***
[info]: No new measurements to upload
@AndyLow91 you are totally right. Your code works without any problems. The issue on my machine was that I was using Node 18. I missed your comment on the Node 16 downgrade.
I reproduced the problem by switching back and forth between Node 16 and 18 and I am 100% able to reproduce the fact that everything works fine with 16 but breaks with 18. This is very strange. I even tried moving from axios to request -> same result. Node 18 breaks this script.
I am still investigating since I dont feel comfortable relying on an old Node version for the future.
Just tested Node 21: Also broken. I have no idea what is going on here, to be honest.
That was quite a ride: The problem is that Cloudflare implemented a DDoS protection mechanism based on TLS fingerprinting. Cloudflare seems to detect that we are connecting from a Node 18+ client and denies access. The method of detection is pretty good described here: https://httptoolkit.com/blog/tls-fingerprinting-node-js/
To fight this kind of DDOS protection, I am now generating a custom cypher-list for the axios client so that the script looks unique from Cloudflares point of view. After adding the custom cypher-list to the HttpsAgent of axios, everything worked fine on Node 16, 18, 20 and even 21.
@AndyLow91: In order to use a custom HttpsAgent had to remove the axios axios-cookiejar-support dependency since this package does not allow for additional HttpAgents to be configured. I implemented a simple Cookie Jar HttpAgent which seems to also work without problems. We now have one HttpAgent for the cookoes (cookieAgent) and a HttpsAgent for the Cloudflare DDOS bypass (stealthHttpsAgent).
I will be testing for an hour or so on my machine as well as on my AWS LightSail instance. If everything works we should have version 2.5.1 with all the fixes released today :)
Interesting, that it works normally (without TLS fingerprinting?) in Xdrip4io5 - Librelink Up manager - https://github.com/JohanDegraeve/xdripswift/tree/master/xdrip/Managers/LibreLinkUp
But Xdrip4io5 requires the use of a heartbeat, for example, a G7 or L3 sensor - to fine work.
Cloudflare seems to have blocked some runtimes that are used for web scraping by adding their TLS cyphers to the block list. This includes NodeJS and Python. Most web scrapers are written in those "languages" so it makes kind of sense that they might want to block those scripts from accessing their APIs.
Xdrip4iOS is written in Swift, which i doubt Cloudflare would block, because that would include most iOS apps. The LibreLink Up App seems also still be working fine. You can even access the API through Postman without any problems or hacked cypher-lists. It seems to me that the TLS fingerprinting is targeted at web scrapers and people like us who are accessing APIs that are not officially available.
Everything seems to work fine now. I published version 2.5.1. Its available now.
Sorry if my comment is not related to your project and you can remove the comment if you don't think it is appropriate. We use Python to accessing their APIs and experienced similar issues. Thanks this thread to clarify the TLS issue. Python's requests library does not offer options to modify TLS parameters. We end up using a custom adapter to specify SSL and solve the issue.
@unowhat I certainly won't remove your comment! This is very important information for future developers or maintainers. This information must be available to everyone. Since you are using Python this seems to prove my theory about the TLS fingerprinting thats going on at Cloudflare. Glad to hear that you were able to fix the issue :)
I would like to know if a new account is needed? I specifically made a new VPS, downloaded the latest version of the script, but I get the same blocking error. I tried node 16,17,20,21 to no avail. Is it possible that the ban is tied to the account? Thank you.
The problem is still relevant… Apparently the script does not work for Node users (any version). Changing account and servers doesn't help
@JMdia what do you mean by "Node users"? The Docker image also runs Node. I tested it on locale NodeJS (16, 18 and 20) and also inside Docker.
What are the errors you are getting?
If necessary, I can provide ssh access from the vps and nightscout data. The guys specially made a separate machine for testing because many users have this problem
@JMdia there is a new branch (https://github.com/timoschlueter/nightscout-librelink-up/tree/feature/shuffle-ciphers) that contains a new way of shuffling the cipher for the TLS fingerprint bypass. Do you have any chance of testing this on your end?