NeweggBot
NeweggBot copied to clipboard
Ataraksia
Changed to wishlist based and updated to work with new UI
Now is wishlist based, meaning all combo items should now work. Also means you can use your newegg account asynchronously with the bot - purchase something for yourself that the bot isn't looking for and not interfere with the bot. Also means that the new UI of "This item isnt available" when attempting to add to your cart if out of stock does not need to be concerned about (easy to click out of via code, but will slow you down and potentially make you lose the item you're getting for being too slow).
Checkout UI has been recently changed with multiple steps. Made changes to work with the new UI. Edit: I believe this might be a CAN only thing, or at least depends on region.
Domain setting is now in config file, can at least be set to "ca" and "com", probably others as well. Removes the need of having two branches for this project.
Note: Some stuff was pulled from https://github.com/Codolophier/NeweggBot. Though, due to a lot of rework it was only a handful of things that were kept. New, untested version is on a different branch. More info on this message
Nice timing! I'm trying to get one of the Ryzens that Newegg is trickling out today.
I noticed that there is no package.json file while the ReadMe references using 'npm install' to install the necessary packages.
Also the default path for chrome in the config_template seems a bit unusual.
Is there a way to run this headless, so I don't have to look at chrome? Seems to pause (or crash) if I minimize Chrome.
A way to run it without a browser at all would be even better - why use puppeteer? Does this method make it harder to detect bots?
@bauerfyr That was a whoops moment. Fixed that package.json thing. Something I forgot to add from Codolophier's branch.
The default chrome install was basically a "This'll probably be close to the default install but also not exactly". Just due to not knowing whether it'll be in the x86 folder or not, or if someone wants to use firefox. Mostly kept the same as that's what Codolophier's default option was.
Running headless seems interesting, I'll see if I can do something about that. Basically just need to add an option to input the 6 digit 2FA via console. First time dealing with nodejs, or even javascript in general. So might take a small-ish amount of time for me to do, should have enough time within the next few days. Basically just learning to get console input which I'd assume is easy. Tho, dont want to jinx it.
I'd assume it pauses when minimizing is due to not fully reloading the page if it's minimized. Dont know why puppeteer was originally used on the main branch. I only used it as with stated above, first time dealing with javascript. My programming experiences arent really in web or bot stuff so I couldnt make anything from scratch unless researching about. I really like having some example to start with and mess about when learning something so this was an ideal solution I guess.
Don't know how long I'll continue this as going by newegg Twitter, they have some sort of "solution" for the low in demand stuff which is apparently going to be put in effect soon(tm). However idk how well their solution will be, and if it's meh I might switch over to working on the BirdBot by natewong1313. Is either a headless solution or non browser based. Was originally meant for Switch purchases on Wallmart/Bestbuy. Just add in Newegg (and for me, maybe other CA stuff as that's where I am) and now it's a multi store solution for low stock purchases.
Thank you for your contributions, still trying to get my hands on my 3080 haha,
Hey, @northmatt, I'm quite new to GitHub and don't know how else to contact you. Just a heads up, my windows pc keeps crashing after using this bot for about 30 minutes. I do not know how to get any of the log files. I was constantly checking task manager and everything seemed fine but then after a while the git bash terminal refuses to close, the Newegg page does not load, and shortly after my pc crashes. I'm quite new to coding so I'm currently trying to understand how the code works but I haven't found an exact line where the issue exists.
Edit: I don't know how but the problem went away after a day
@jsrao3 Nice to see that it has been fixed.
If you ever want to know what some steps for debugging would be like then it'd be: -Look at Event Viewer for Win crash events. -Make sure that the tab isnt minimized and probably test with Windows PowerShell. -Use chrome and look at the debug-log file created by chrome. Doesnt log anything to do with the bot but does log chrome issues.
Hey! Hope you are well @northmatt. Thanks for your contributions. I was wondering if you could help with an issue I ran into. I got this running in macOS and have got it to open the browser and everything. It runs fine until it gets to the CVV2 entry area.
NeweggBot-master % node NeweggBot.js [2021-01-26T03:20:05.635] [INFO] Newegg Shopping Bot - Newegg Shopping Bot Started [2021-01-26T03:20:05.639] [INFO] Newegg Shopping Bot - Please don't scalp, just get whatever you need for yourself [2021-01-26T03:20:16.965] [WARN] Newegg Shopping Bot - Manual authorization code required by Newegg. This should only happen once. [2021-01-26T03:20:31.572] [TRACE] Newegg Shopping Bot - Logged in [2021-01-26T03:20:36.545] [INFO] Newegg Shopping Bot - Item(s) added to cart, checking cart [2021-01-26T03:20:36.663] [INFO] Newegg Shopping Bot - Subtotal of cart is 7 [2021-01-26T03:20:36.663] [INFO] Newegg Shopping Bot - Cart checked, attempting to purchase
It gets to this point then just stops. It doesn't input the CVV or give any errors.
@momobadarni seems like the issue lies with different NE domains having different UIs (I worked in CA) or a new UI change. The "Place order" button for me was "Review your order" and the bot looked for those words specifically. So any differences would basically get it stuck. Quick change is on line 221 to change " 'Review your order' " to " 'Place order' " or " 'order' " (which basically just searches for any button with "place order"/"order" in it, depending on what you tell it to look for. I like being specific so it doesnt accidently press something else, but also means any UI changes from domain changes or updates can mess it up).
Im going to look into whether this is on CA as well and see if I need to make it work differently with multiple domains or just accept a newer UI. I hope to have this out soon with hopefully a headless option too, as I have mostly gotten that finished.
edit: Quick update, seems like the button changes based on domain changes or something. That or maybe browser (different browser, JS support for the site, aspect ratio, etc)?
@northmatt Interesting. So I went ahead and changed the code to account for the UI change in regard to the place order button. But whenever I run it, the bot never gets further than the "Cart checked, attempting to purchase" section. It never even says it is "waiting for CVV input element". That seems to be where it gets stuck. Side note: is there a way to change the program to use Firefox Nightly? I think that worked batter on macOS, I tried to change the path to Firefox via the config file but then it says that it could not launch the browser process.
Sorry to keep bothering you, I'm not very JavaScript savvy 🙂.
@momobadarni I checked on the US site too and it was the same "Review your order". Maybe because I am still on a CA IP, idk. So it's a bit hard for me to test this myself. Though, basically you want to inspect element the button and check what is in the text as it is cap/symbol sensitive. Though, sometimes things arent exactly what they seem for button text. I find that even though the web/visible button has all caps, in code only the first letter is caps. So the external text for the secure order button is "SECURE ORDER" but the internal text is "Secure order". Sometimes there are symbols, so before the wishlist based system I was working on the old URL add to cart system which had popup issues when adding out of stock items to the cart. The external button text was "REMOVE ITEM(s)" and the internal button text was "{nbst}Remove{nbst}item(s)" or something of sorts (cant really remember what was inside the braces).
Now, going by previous PRs Im going to be making some guesses, but cant confirm that this'll work. On line 221
replace await page.waitForXPath("//button[contains(., 'Review your order')]", { timeout: 500 }) with await page.waitForSelector("#btnCreditCard", { timeout: 500 })
I hope that's the button I expect it to be given my educated guessing. Though, educated guessing is only just nice guessing, no confirmations. This is usually the final "ordering" button which seems to have an id tag. I find it annoying that the rest of the buttons generally dont have a proper id tag, but eh. Though anyways, if this is the case then it seems like CA NE has an extra step to review all your info.
Now as for firefox stuff, I believe you have to also do the specific npm firefox stuff too for that. So that'd be npm install puppeteer -PUPPETEER_PRODUCT=firefox and maybe PUPPETEER-PRODUCT=firefox npm i puppeteer if you get browser revision errors on the install command. If you already did that and still have issues, idk what to do. Almost none of this is in my experience (MacOS issues, JS issues, browser issues, puppeteer issues, etc).
@northmatt
That worked! Thank you so much for the help. The button ID for the US website is btnCreditCard. Sorry I couldn't troubleshoot that myself. I greatly appreciate the help!
Well, bit late, but got the headless support stuff fixed. Also incase for those getting issues with being stuck on 2FA after an hour, have a fix for that. Plus an option for the issue above being included. The info is in the updated readme file.
@northmatt impressive work here.
I'm still having an issue with 2FA every time I run this even though I don't need to enter the 2FA on the chrome window I already have open. When I run the program it launches a new chrome window. In an effort to make this new window "trusted" I opened a new tab and tried to log into gmail but after entering my username I'm met with the following gmail error:
"Couldn’t sign you in This browser or app may not be secure. Learn more Try using a different browser. If you’re already using a supported browser, you can refresh your screen and try again to sign in."
Another possible clue is that adjusting the TFA_base_wait, TFA_wait_add, and TFA_wait_cap config values doesn't seem to make a difference even when multiplying them by 20x the default values in the templates. I tried this with Skip_TFA at True & False and Do_First_TFA at True & False.
Any idea why I'm repeatedly asked for 2FA?
@Ribwich NE seems to have been updated and I think the waiting feature might have been removed, or changed. Before learning about that feature I was going to add gmail support so that it could automatically find the 2FA email and input it itself. Though, didnt go that route as idk how much people would like a bot reading their emails (was going to make it search for NE emails at least), having the bot potentially mess up and do smth to their email account (deleting an email due to say a major UI update changing how the bot interacts with things), and lastly discovered the waiting feature that was easier to implement at the time.
Recently though I've been looking into how cookies and localstorage work so I can hopefully make the bot open a trusted session all the time. Feel like this'd be the more ideal solution if I can figure out how it all works. Im hoping localstorage just doesnt work by default with puppeteer (not even stored in RAM or anything) as if it does I dont understand why NE wouldn't have all the data to see that its current session is trusted after the first 2FA attempt (generates the cookies at login, then should keep them until bot shutdown). That or maybe it needs to launch with the cookies to work, idk. Plan to do a quick update soon to fix all the memory leaks in the main loop. Really kills perf after a few hrs if you're on a quick refresh time.
Also, usually you shouldn't need to make the new window trusted. I think it was an IP based thing, login to a trusted browser and your IP would be trusted for 24hrs or until 12am at where ever NE's servers are. Tho, dont think it matters now that this thing is removed.
edit: just tried a test now in an incognito tab. Tho instead of reloading by changing my URL I pressed the back button next to the email (basically the thing that says "bring my back to the email page so I can edit my email"). Accepted my email again and got to my password authentication. So maybe just need to change how it acts with the sign-in page.
@Ribwich Well email 2FA issue should be fixed now. Cookies and localStorage are stored so it should only ever ask for passwords. edit: or not. I spoke too soon. Might have to do with NE thinking the browser isnt secure. Probs some http certificate thing. That or maybe smth isnt saving correctly for sessions. Tho I doubt that as if the bot is forcefully stopped and ran again within 1hr of first starting it wont ask for a 2FA or password, just instant login.
@Ribwich okay, now it seems to be fixed. Tested for a session a day for a few days in a row each of which was a fair few hours long, didnt get asked for 2FA. Tho, it seems like headless sessions wont work anymore.
edit: after pulling make sure to redo npm install as new dependencies were added.
@northmatt hmmm, I'm still being asked for 2FA every ~1 hour. I did a fresh pull and install with only the config file copied from the earlier version. Any tips for something I may be doing wrong?
@Ribwich Interesting. What is the refresh_time/randomized_wait_ceiling (maybe getting kicked from too quick refreshes), site_domain (maybe different domain is harsher then where Im testing), and headless (just doubling checking that this is false)? As for other questions: are you using a VPN, is your chrome upto date, and does your NE account have the 2FA option enabled in account settings (when disabled you still get 2FA, just isnt enforced as harshly)?
Also as another test, open an incognito tab (outside of bot browser, as in your person one) and login to NE. When asked for the 2FA wait 1 or 2 mins and then click the back button with your email next to it for the "edit my email". Then reaccept your email and see if it asks you for the password instead.
@northmatt I was having the same 2FA issue as @Ribwich but your test with the incognito worked for me.
In addition to that, I'm still having an issue where the bot just keeps adding more of the same item to the cart but doesn't check out: https://photos.app.goo.gl/LV2WAWp4mNP93pEV9
@xbryanthomasx I'll see what I can do about that. Im pretty sure PR #35 has a fix for that (tho I do not know if it's still working as it doesnt seem to be maintained anymore). So it's mostly a matter of merging and updating the code. I believe it only starts causing issues on mine when the page loads too slowly which I never encountered for. Was "Subtotal of cart is $###" logged in the console? Need to know the last thing called to know where it started having a fit.
Good to know that the other way of waiting for 2FA bypass works though. I unfortunately dont know why 2FA gets asked for you two and not for me. I think it might have to do with US vs CA location having different security measures. Tho that's all I can guess.
Will try getting an update soon with the two fixes.
Hey @northmatt, I've been using your fork for some time and while it works, it seems as though wishlists update a lot slower then product pages. This means items that are hot in stock never get picked up before they're out of stock again, making it sort of pointless. Can you consider readding single item functionality back? I would put this somewhere else, but I can't open a issue page on your fork.
@Febau31 Interesting, I am adding it back and have gotten most of the stuff that I've been tasked to do done. However, it would be nice to have evidence of this claim as I dont want misinformation being spread. Hopefully a new version with the fixes will be up soon.
@northmatt Hey Matt, I understand that, however, empirical evidence is hard to obtain. While I can't offer definitive evidence, I've been using your fork for over three weeks and haven't gotten a drop from the GPUs I've been following. Before that I was using a auto-refresher and it would usually pop 1-3 a week with a item that immediately sold out and I was too slow to grab. I don't have access to another bot to compare, but have seen some hearsay around reddit confirming a similar things.
It's never even made it to checkout AFAIK and I added pretty much every model from the GPUs I've been following to my wishlist. If you added stats to the bot based on failed/attempted checkouts/finds you could compare it yourself by running it by item number vs wishlist. I would also be interested in whether or not my initial findings are untrue.
Just got the latest version setup and running. Thank you @northmatt for working on this!
@Febau31 that's fair. I havent had much of any luck on the wishlist based system in CA for the long time that I've used it.
Now, I have got a CA version working and upon checking it with US I came across a not so nice bug with NE. I went beck to the old version of adding items to the cart via URL and it seems like there are some items that'll basically lock your NE account. Went with some random OOS (Out Of Stock) item (cheapest 3080 at the time, which was some MSI one) for a test run. NE either has two kinds of carts or likes to visually split it up as such - an in stock cart and OOS cart. Adding this item seems to have added a ghost item to my OOS cart that NE really seems to hate. On browser my cart (in stock and OOS) will always show as empty no matter how many items are in it (the cart icon on top right will display the correct amount of in stock items in cart tho) and I cannot add any other OOS items to my OOS cart. In the app I can view and seemingly checkout any in stock items in my cart. You also have the option to clear the OOS cart, however when clearing it fails to do so. So currently the US side of my account is locked and I cant really do much bot testing for that.
I have tried looking into this issue by adding other OOS items to a non-signed in NE browser and it didnt seem to have issues. Which means that only specific items will lock your account. I dont know how common these items are. I have also looked into removing the item from my cart by looking into the scripts of NE, tho the scripts are minorly obfuscated (probably from compiler just making it more optimized for itself) and it comes to be very hard to get a good understanding of it. Spent fair few hours now trying to debug and understand the web side of NE without much luck. Now Im just going through NE support to see if they can remove the item for me. I ideally wanted to find a way to remove myself so that I can post the function and if anyone gets troubles it's an easy fix. Tho given there isnt an easy fix Im reluctant to post the code. If so there'll def need to be some warning thing for it.
@broehrig17 Ey, glad to hear that.
Most of this stuff I'd rather discuss across DMs to prevent poaching and counter-intelligence, however I'm pretty sure Newegg has a global refresh duration and if you refresh multiple items, it counts towards a lockout timer. The more items you refresh, the faster you will hit this threshold. Cart counts towards this as well. I reached a threshold where I could run a couple items and cart on a auto-refresher without ever getting locked out. Randomizing the refresh also seems to reduce the chances of the lockout being triggered, so it is logarithmic in nature to account for human randomness. The more linear the refresh, the faster you hit the threshold. Going over this threshold just brings up the temporary ban page.
Obviously refreshing different items through different proxies and having them added to cart the moment they're in stock on the cart/logged in account is a way to deal with multiple items. This is difficult to do unless you have a built in proxy list functionality for refreshing. The cart page would manage the add after being notified by the proxy page that it's in stock.
DMs be nice and all, but this was something I was stumped on and wanted to see if anyone could have any input on it.
So on mentioning proxies I decided to try my VPN and finally got the "remove OOS item from cart" popup. However, what I was experiencing Im almost certain isnt a lockout from global refresh. As when I tested this with the bot the first time, it failed on the first few tries. When trying non-logged in incognito tabs it got locked on the first try with the MSI GPU. However trying other items during the sessions it wouldnt lock up. I also surprisingly havent had any issues on the CAN side of my account despite refreshing on a 5 const/5 rand interval with many 3080s for some hours.
While it's nice that accounts can be unlocked without the need of NE support, it still isnt an ideal solution for the everyday person to accomplish.
Yeah, never ran into the issue with the cart lockout you're talking about when using a auto-refresher. Obviously it doesn't happen with the wishlist which is another reason it tells me it's not working for hot items.
If having a item in the cart first actually makes it so it doesn't trigger the lockout, you could have some dummy items in the cart that give removed before checkout to simulate a real cart.
Since your "browser_executable_path" is written for windows, why not use edge as it's force installed? "C:/Program Files (x86)/Microsoft/Edge/Application/msedge.exe" Also how about including a basic run script for real noobs/lazy. ` %CD% node neweggbot.js pause
` Lastly is const puppeteer = require('puppeteer-extra') const stealthPlugin = require('puppeteer-extra-plugin-stealth') const readline = require("readline") const log4js = require("log4js") const config = require('./config.json')
suppose to install / prompt for install of the packages? Because it doesn't for me. Also when installing the modules, I couldn't get the script to find the modules. I had to copy them to the directory of the script.
Also you can't add regular gpus to watchlist now, only bundles?
After a couple hours, the 2fa expired, and running the script again, it keeps thinking 2fa is passed... it tried 20 times immediately and now my newegg account is locked.
Firstly I'd like to state Im moving my pushes to the "untested" branch until US peeps can test it. Secondly I'd like to state this may or may not be my last support push for US NE. I'll most likely at least continue to push undocumented CAN updates, potentially documented ones.
edit: the "untested" branch can be viewed on my actually repository right here.
@Febau31 @Ribwich @xbryanthomasx if ya still interested got some of the stuff done finally. Bit long I know, srry. 2fa will probs still be asked, but I think should get bypassed now. Should work with itemlists too. However the issue with the mask popup isnt fixed.
@Chaython First point is fair, to this I'll give a fair response. At this point is laziness.
Second point is eh, Im pretty sure Windows likes to complain about scripts made from other computers saying the file is untrusted. You can usually hold shift and right click in the directory to openup powershell to said directory (or type "powershell" in the directory area), then press the up key and then enter. Runs last command which will probs just be "npm start".
Lastly the "npm install" should install said modules if the "package.json" is in the directory where you run it. This shouldn't cause issues when running it. I did just test it to make sure the package.json file was setup correctly and it worked. If you or anyone else has issues still then do say so and I'll see if I can see what's up.
Also the 2fa thing I think is fixed. I hope the lockout was only for a day. Srry about that.