samc621
"Not in cart, trying again"
I am running this on windows without docker, and I got the bot to work. it opens up the browser, selects the size and then gets stuck in an endless loop saying "not in cart, trying again". I am trying to test run on the nike website and in the browser it gives me an error saying
"We had an issue with your request. If you continue experiencing issues, try refreshing the page.
[ Code: 9F502A89 ]
Thanks in advance...
Hi @Diablozone, thanks for opening this issue. I have seen this error before, however, I've seen the product also add to cart (and it continue to checkout) in the presence of this error. Are you seeing the same?
no the product doesnt add to cart. should I keep trying again and again?
what happens is that the page keeps refreshing and this error keeps looping. eventually the chromium tab closes automatically
can you give some insight into why this error occurs or which part of the code its coming from so that I may be able to look into it further
@Diablozone seems I am seeing the same behaviour, now. There is a 429 error coming from the API when clicking any of the DOM elements for style, size, or ATC button. I am going to need to look into this further and will circle back on it ASAP.
Sure, Thanks a lot!
On Wed, Jun 9, 2021 at 7:04 PM Samuel Corso @.***> wrote:
@Diablozone https://github.com/Diablozone seems I am seeing the same behaviour, now. There is a 429 error coming from the API when clicking any of the DOM elements for style, size, or ATC button. I am going to need to look into this further and will circle back on it ASAP.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/samc621/SneakerBot/issues/28#issuecomment-857699359, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANBWNBVWG6CWYRIPEVNHS2DTR5UYBANCNFSM46I3ABPQ .
@Diablozone just updating this here, the error on the nike integration appears to be a CORS issue. You will see some of the requests failing in the Chrome devtools network tab. I've started looking into it but haven't gotten to the bottom of it, yet. It just started happening afaik.
Does this mean that of I use a firefox webdriver the issue might be solved?
On Thu, Jun 10, 2021, 21:07 Samuel Corso @.***> wrote:
@Diablozone https://github.com/Diablozone just updating this here, the error on the nike integration appears to be a CORS issue. You will see some of the requests failing in the Chrome devtools network tab. I've started looking into it but haven't gotten to the bottom of it, yet. It just started happening afaik.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/samc621/SneakerBot/issues/28#issuecomment-858727762, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANBWNBTECMEGDFGPLONUDNTTSDL45ANCNFSM46I3ABPQ .
Doesnt work. Also, I tried a couple of Shopify sites but they just don't open. The new chromium tab opens and closes automatically
On Tue, Jun 15, 2021, 17:20 Samuel Corso @.***> wrote:
@Diablozone https://github.com/Diablozone its definitely worth a try.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/samc621/SneakerBot/issues/28#issuecomment-861431395, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANBWNBWC7KIW4VRRFVJRYGTTS446RANCNFSM46I3ABPQ .
@Diablozone can you provide a sample Task object for me to test with? I was recently testing on Kith and some others and it was working fine.
@samc621 I am seeing this with your API example:
{
"site_id": 1,
"url": "https://www.nike.com/t/lebron-18-black-white-basketball-shoe-M6DgN2/CQ9283-100",
"style_index": 1,
"size": "8.5",
"shipping_speed_index": 0,
"billing_address_id": 1,
"shipping_address_id": 1,
"notification_email_address": "myemail"
}
Side question, what is style_index and is it just the style like Best 10-18 or Best 1-9 in the example url above?
Hi @labboy0276, yes I noted this issue a couple of weeks ago. I believe that it is a CORS error, and I haven't gotten around to addressing it yet.
@labboy0276 just copying this over here for reference: https://user-images.githubusercontent.com/38767335/125101214-fa64c380-e0a7-11eb-92dd-cbdb50f2e127.png
I am going to throw this here, but haven't tried it yet.
Have you thought about using a header like this @samc621
headers: {
'user-agent': userAgent,
'sec-fetch-dest': 'none',
'accept': '*/*',
'sec-fetch-site': 'cross-site',
'sec-fetch-mode': 'cors',
'accept-language': 'en-US'
}.
I have seen this in other bots for scrapping the interwebz.
@labboy0276 have you tested this with the Nike issue? Curious if it might help.
negative @samc621 just putting it there as I havent had time to test it out yet. If someone else can, that would be helpful.
@labboy0276 I added this:
await page.setExtraHTTPHeaders({
'user-agent': `${userAgent}`,
'sec-fetch-dest': 'none',
accept: '*/*',
'sec-fetch-site': 'cross-site',
'sec-fetch-mode': 'cors',
'accept-language': 'en-US'
});
But it doesn't seem to be doing the trick. I took a look in the console and it looks like a 429 error (blank response) on this API.
There is a new captcha on the footsites, I've never repro'd it in the browser but I'm curious if this might help reduce bot detection so that we don't hit it there. I'll give it a go in a bit.
So it looks like, testing with these headers on the footsites, I got blocked. It took me to a Terms of Service page. Testing without them, I hit the captcha I was expecting.
All in all, I don't think setting these headers is helping much. I will need another approach to both the Nike issue and the footsites captcha.
I did some research on the 429 status code error and from what I got out of it, a 429 is a rate limiter error. Meaning the user sent too many requests at once. Could this be fixed by adding some timeouts to puppeteer?
@Kohlsen yes, that is indeed the common meaning for a 429 response code. I have tried that, among numerous other things, but to no avail. I keep looking at it when I get a chance, and I think others are too. Let me know if you get anything to work!
I'm also experiencing this issue on nike.com. When replicating the "clickthrough/search" behaviour on my own browser it doesn't occur
@jhgeluk yes, I think it has something to do with the browser fingerprint and/or the speed of the navigations. I have some potential solutions for this, including:
- Randomizing all of the
window.navigatorproperties. - Using puppeteer to emit mousemovement events when clicking a DOM node.
- Using more delay in between certain actions.
I just haven't had time to implement this, but if someone is willing to give it a try, please feel free!
I feel like they mostly check the browser fingerprint, I've simulated mouse movements & keyboard movements and even added a randomized amount of links it will visit before entering nike to create a sort of "browser history".
@jhgeluk yes, the browsing history was a good idea for creating a cookie profile. Have you checked window.navigator in the console? These are many of the properties used by fingerprinting libraries, and puppeteer-extra does a good job of randomizing some of them, but we can do more on top of that. We can also look through Nike's FE code for the fingerprinting code which might be stopping us. I am pretty confident that it is indeed a fingerprinting issue because this happens even when I don't use any proxies (just a regular residential IP) and as you said, the mousemovements aren't helping.
Looks like puppeteer is also missing an event listener for Network.responseReceivedExtraInfo, this might trigger some libraries like FingerprintJS
@jhgeluk yes that makes a lot of sense, can you try to identify what libraries Nike might be using? That way we can reverse engineer to a solution. This is commonly how I solve issues like this.