Updates for Checkout flow (elements and references) and Over Price Limit Behavior as well as some minor changes to logging and project structure

[Codolophier]

Configuration

• Added new configuration option for "over_price_limit_behavior" to determine how to handle instances in which the cart total is over the configured "price_limit".

Checkout Flow

• Updated URL check for the cart from ShoppingCart to cart
• Updated the check_cart method to interrogate for zero balance
• Updated the check_cart method to incorporate over_price_limit_behavior when the cart total exceeds the configured price_limit value
• Replaced the reference for the Biz.GlobalShopping.ShoppingCart.checkOut script with a reference to the "Secure Checkout" button. The script doesn't seem to be there anymore (at least from what I could see).

Minor changes

• Replaced "report" method with log4js logger calls for segregation of message types (e.g., info, error, warn, etc.).
• Added package.json for dependencies and scripts.

[awesomepandapig]

This works great and fixes the issues I was having! One note though on line 162 it's supposed to read if(config.auto_submit == 'true') but it instead reads if(config.auto_submit. Because of this it orders even when testing :/

[Frosty8738]

I seem to still show an empty cart on checkout, unless that's normal behavior?

[Codolophier]

This works great and fixes the issues I was having! One note though on line 162 it's supposed to read if(config.auto_submit == 'true') but it instead reads if(config.auto_submit. Because of this it orders even when testing :/

Correct. In my config.json file, I specified the flag as a boolean as opposed to a string. I should have updated that in the README file.

[thatgiraffe]

Hi, big newbie here. First, thanks for the time spent by Ataraksia and Codolophier, you're helping a lot of people get a card solely for their own enjoyment. This pr helped me get past a few issues where I can now check out with a practice item that's in stock. However when trying with several items, all of which are out of stock, it automatically goes to the cart page and tries to checkout with nothing in cart. Reports in the script continue all the way to "cannot find the place order button" without the browser leaving the empty cart page. It doesn't seem to be circling back and pinging the item pages for the cart button. Any thoughts on how to get around this?

[Codolophier]

@thatgiraffe I have not tried ordering multiple items where they are mix of in-stock and sold out. @Ataraksia can comment more authoritatively on this; however, I don't think the bot in its current state can handle the mix. I say this because:

1. The check_cart method is simply checking for price limit as opposed to which items are in the cart (i.e., purchasing what can be purchased vs. continuing to attempt to add sold out items to the cart).
2. There is no tracking of what has been successfully purchased vs. what is still outstanding (both in-process or spanning invocations).

To handle the mix would take a bit of extra coding, but far from impossible.

[thatgiraffe]

Understood. Thanks for the reply. However, I'm referring to a list of items (say various 3080s) that are all out of stock. It doesn't seem to circle back and continuously ping stock when everything is found out of stock. Instead it seems to try to check out with an empty cart and then finishes with the last report "Please make sure that your Newegg account defaults....."

[Frosty8738]

Understood. Thanks for the reply. However, I'm referring to a list of items (say various 3080s) that are all out of stock. It doesn't seem to circle back and continuously ping stock when everything is found out of stock. Instead it seems to try to check out with an empty cart and then finishes with the last report "Please make sure that your Newegg account defaults....."

Like this? My cart seems to just endlessly refresh with nothing in it. Like a week or so ago I had the original bot working where it actually showed the sold out items in my cart but it doesn't do that anymore, maybe Newegg changed something on their end since then. But am I right to assume this isn't normal behavior or is it?

[arcreigh]

I tested out your branch but can't for the life of me get it to click on that damn 'Continue to payment' button so that the third step unlocks allowing for the bot to progress to the cvv / credit card selection steps. I am not completely familiar with Puppeteer but I would think it would work the same as the prior steps in lines 112-122 However this does not appear to be the case. Heh guess this is what I get for being a network engineer trying to buy a bloody 5900x so my poor i7 can rest in its i'll fated failed pump heat death.

	// Find the "Continue to Payment" button and click it (if it exists)
	try {
		const [button] = await page.$x("//button[contains(., 'Continue to payment')]")
		if (button) {
			logger.info("Continuing to payment")
			await button.click()
		}
	} catch (err) {
		logger.error("Cannot find the Continue to button")
		logger.error(err)
	}

For reference the above is what I attempted to throw on lines 123 - 133 in the hopes that it would work.

Of interest puppeteer finds the CVV field and even tries populating it with data from the config file. However since that step is not technically unlocked the purchase can not continue.

Update

$x("//button[contains(., 'Continue to payment')]")

Is indeed valid according to firefox.

However putting a logger.debug on the page comes back undefined I wonder if I need some kind of delay in here to allow the page to load before it attempts to find the element.

[arcreigh]

The readme should also be cleaned up just a bit regarding GPU's as this has the capability of purchasing any single item on newegg.

[Codolophier]

@Frosty8738

My cart seems to just endlessly refresh with nothing in it. Like a week or so ago I had the original bot working where it actually showed the sold out items in my cart but it doesn't do that anymore, maybe Newegg changed something on their end since then. But am I right to assume this isn't normal behavior or is it?

This could be a change from Newegg - although I am not certain. The bot keeps trying to add the item(s) to the cart using the URL https://secure.newegg.com/Shopping/AddtoCart.aspx?Submit=ADD&ItemList={item-number}. If you call this endpoint manually, you'll see that the same "your cart is empty" message will be displayed as opposed to "Sold out".

For example:

The URL https://secure.newegg.com/Shopping/AddtoCart.aspx?Submit=ADD&ItemList=N82E16819118147 will produce a cart with the item in it (Intel Core i9 CPU).

The URL https://secure.newegg.com/Shopping/AddtoCart.aspx?Submit=ADD&ItemList=N82E16819113663 will produce the "Oh.. seems like the cart is empty..." message (AMD 5950x CPU was the intended Item - currently Sold Out).

[Frosty8738]

@Codolophier

For example:

The URL https://secure.newegg.com/Shopping/AddtoCart.aspx?Submit=ADD&ItemList=N82E16819118147 will produce a cart with the item in it (Intel Core i9 CPU).

The URL https://secure.newegg.com/Shopping/AddtoCart.aspx?Submit=ADD&ItemList=N82E16819113663 will produce the "Oh.. seems like the cart is empty..." message (AMD 5950x CPU was the intended Item - currently Sold Out).

Oh interesting I didn't realize that, thank you. So if it's configured with multiple product numbers it will try to fetch any/all of them and will attempt to check out as long as it grabbed 1 of them?

[Codolophier]

t can't for the life of me get it to click on that damn 'Continue to payment' button so that the third step unlocks allowing for the bot to progress to the cvv / credit card selection steps.

Question: Do you have your Newegg account setup for your Shipping, Payment, and Billing defaults? From the sound of it, the flow you are experiencing is a bit different from the flow the bot is set up to handle.

NOTE: For the billing defaults, you have to select the address to be associated with your default payment method.

[Codolophier]

So if it's configured with multiple product numbers it will try to fetch any/all of them and will attempt to check out as long as it grabbed 1 of them

I've done some experimenting and it seems that with the call to https://secure.newegg.com/Shopping/AddtoCart.aspx?Submit=ADD&ItemList={item_number}, if you specify a comma-delimited list of items, as long as they are all able to be added to the cart, the add will succeed. Otherwise, the add fails.

For example:

https://secure.newegg.com/Shopping/AddtoCart.aspx?Submit=ADD&ItemList=N82E16819118147,N82E16819113567 will add two items to the cart - both of which are available.

Adding a third item to the cart, that is sold out will produce an empty cart:

https://secure.newegg.com/Shopping/AddtoCart.aspx?Submit=ADD&ItemList=N82E16819118147,N82E16819113567,N82E16819113663

[arcreigh]

t can't for the life of me get it to click on that damn 'Continue to payment' button so that the third step unlocks allowing for the bot to progress to the cvv / credit card selection steps.

Question: Do you have your Newegg account setup for your Shipping, Payment, and Billing defaults? From the sound of it, the flow you are experiencing is a bit different from the flow the bot is set up to handle.

NOTE: For the billing defaults, you have to select the address to be associated with your default payment method.

This seems to be a new purchasing flow. Directly after secure payment. Billing defaults are indeed set.

[Frosty8738]

IP banned

[arcreigh]

IP banned

Uh oh

[arcreigh]

Wait no that doesnt make sense, if it was an IP ban it'd impact all browsers / clients on one network. This only seems to impact the browser directly controlled by puppeteer.

[Frosty8738]

Wait no that doesnt make sense, if it was an IP ban it'd impact all browsers / clients on one network. This only seems to impact the browser directly controlled by puppeteer.

Well the bot was on Firefox and I got IP banned on Firefox AND Brave (which is chromium based).

[arcreigh]

What do you mean IP banned? Do you have a picture of the error message? An IP ban would affect every device on your network not just one PC. It could be your account or even some other device tracking method. There are many ways that can be accomplished.

[jabramsja]

Newegg started doing IP bans last night I think. I don't have a screenshot, but they want you to email the webmaster and beg for them to let you back in.

[Frosty8738]

@arcreigh

I can still log in from my phone.

[arcreigh]

Haha! That is a WAF! (Web Application Firewall) looks like they have VERY recently implemented some new security measures.

[Frosty8738]

Man I just want a 3080 already :/

[Codolophier]

IP banned

Were you polling every 5 seconds? One thing you may want to do is use a VPN service for the bot (if you can run on a Raspberry PI or other such machine). Part of the invocation script would start a new VPN connection every few hours or so to avoid your main IP becoming banned.

EDIT: Perhaps we should introduce some randomness into the refresh time. Maybe that would help avoid - or at least delay - the ban-hammer?

[Codolophier]

@arcreigh

This seems to be a new purchasing flow. Directly after secure payment. Billing defaults are indeed set.

I just did a test with an existing item and the flow was:

1. Shipping (default address specified, no selection needed)
2. Delivery (no selection needed)
3. Payment (no selection needed, just presentation of the CVV input)
4. Place Order button - click
5. Done - no further intervention needed

In my run, the only steps that required any intervention was the input of the CVV code and the click of the "Place Order" button.

[arcreigh]

Out of curiosity where are you located? I am in the US. I wonder if the flow is different geographically.

[arcreigh]

Step 2 Delivery is where I have to click on the continue to payment. Step 2 has some options such as faster shipping / require signature on delivery. But the key for me at least would be to click on the continue to payment button in order to see the CVV2 box.

[arcreigh]

Another interesting thing to note is a GET request during login that sometimes appears.

https://www.newegg.com/areyouahuman?referer=/areyouahuman?itn=true&referer=https%3A%2F%2Fsecure.newegg.com%2FNewMyAccount%2FAccountLogin.aspx%3Fnextpage%3Dhttps%253A%252F%252Fwww.newegg.com%252F&why=9

This doesn't pop up the usual page to check if you are a human no rather it just sits in the console and requires you to copy the link and goto it. After that then you can login.

It seems to 307 temp redirect the login button UNTIL this step is done.

[Codolophier]

Out of curiosity where are you located? I am in the US. I wonder if the flow is different geographically.

@arcreigh I am located in the US as well.

[Codolophier]

Step 2 Delivery is where I have to click on the continue to payment. Step 2 has some options such as faster shipping / require signature on delivery. But the key for me at least would be to click on the continue to payment button in order to see the CVV2 box.

Originally, I had the same type of interventions; but, once I set all the defaults they went away.