Check Stock Timing Improvements

[bravochar]

I'm working on some improvements to the stock checking flow in master and I'd like some feedback on my ideas before I get too far into it. I'm focusing on stores/amazon.py and mostly on check_stock().

By removing the initial "Wait for page to load..." section that looks for either the footer or the Amazon dogs, I've saved about a second and a half. The diff of the relevant section is:

+        perf_start = time.perf_counter()
         while True:
             # Sanity check to see if we have any offers
             try:
-                # Wait for the page to load before determining what's in it by looking for the footer
-                footer: List[WebElement] = WebDriverWait(
-                    self.driver, timeout=DEFAULT_MAX_TIMEOUT
-                ).until(
-                    lambda d: d.find_elements_by_xpath(
-                        "//div[@class='nav-footer-line'] | //div[@id='navFooter'] | //img[@alt='Dogs of Amazon']"
-                    )
-                )
-                if footer and footer[0].tag_name == "img":
-                    log.info(f"Saw dogs for {asin}.  Skipping...")
-                    return False
-
-                log.debug(f"After footer page title {self.driver.title}")
-                log.debug(f"             page url: {self.driver.current_url}")
-
                 offers = WebDriverWait(self.driver, timeout=DEFAULT_MAX_TIMEOUT).until(
                     lambda d: d.find_element_by_xpath(
                         "//div[@id='aod-container'] | "
                         "//div[@id='olpOfferList'] | "
                         "//div[@id='backInStock' or @id='outOfStock'] |"
                         "//span[@data-action='show-all-offers-display'] | "
-                        "//input[@name='submit.add-to-cart' and not(//span[@data-action='show-all-offers-display'])]"
+                        "//img[@alt='Dogs of Amazon']"
                     )
                 )
+                log.info(f"Wait for offers took {time.perf_counter()-perf_start} secs")
+                if offers and offers.tag_name == "img":
+                    log.info(f"Saw dogs for {asin}.  Skipping...")
+                    return False
+

The time shown by the perf counter was about 4.2 seconds before moving the dogs check to the offers Xpath versus about 2.7 after. Before putting together a pull request, what was the motivation for waiting for the full page load?

The only reason I can think of was to enable the line I removed from the offers Xpath regarding the ATC button without the all offers display, but that's even more interesting to me. Before I removed that Xpath, the check would error out after only one second:

2021-04-11 07:13:28|0.6.2|INFO|==================================================
2021-04-11 07:13:28|0.6.2|INFO|Waiting for home page.
2021-04-11 07:13:29|0.6.2|INFO|Already logged in
2021-04-11 07:13:31|0.6.2|INFO|Checking stock for items.
2021-04-11 07:13:32|0.6.2|INFO|Wait for offers took 1.0477783782407641 secs
2021-04-11 07:13:32|0.6.2|WARNING|NOT YET IMPLEMENTED: PDP represents only item worth considering.  No other sellers available. TODO: Parse pricing and Add To Cart from PDP if item qualifies.
2021-04-11 07:13:32|0.6.2|INFO|No offers found.  Moving on.
2021-04-11 07:13:36|0.6.2|INFO|Wait for offers took 1.0394511334598064 secs
2021-04-11 07:13:36|0.6.2|INFO|Item is currently unavailable.  Moving on...
2021-04-11 07:13:41|0.6.2|INFO|Wait for offers took 1.063650256022811 secs
2021-04-11 07:13:41|0.6.2|WARNING|NOT YET IMPLEMENTED: PDP represents only item worth considering.  No other sellers available. TODO: Parse pricing and Add To Cart from PDP if item qualifies.
2021-04-11 07:13:41|0.6.2|INFO|No offers found.  Moving on.
^C2021-04-11 07:13:43|0.6.2|INFO|Caught the interrupt signal.  Exiting.

Has anyone talked about skipping the offers page check and using the ATC button directly to save that time? I see that we're actually carting with the OfferID currently, so if that is saving more than a second it's probably a wash to try and support the PDP directly as a first option.

Finally, should I be working off of master or development to put together a pull request for this, if it's worth pursuing?

[bravochar]

I'm currently testing this, and it looks like it has the added benefit of completing the stock check in far less time when the item is out of stock. It's only taking about 1.25 seconds to verify that an item is out of stock, so this could help people checking multiple ASINs in one instance. Previously, I was seeing about 3.75 seconds.

[Wrapzii]

This looks good and is much faster... i don't know why he is waiting for the footer to load... my overall wait time inbetween refreshes is now under 4 seconds when it was 5 seconds with the 3 second delay..

[sircambridge]

I implemented your change!! it seems to work for me. Ive also removed the "time.sleep(1) # wait a second for page to load" inside navigate_pages. now that you mention it, I think theres lots of room to trim for speed, like looking for buttons to be clickable instead of waiting for page loads. right now a full checkout takes me 7. some seconds, which is almost always too slow for the merch im trying to get (nvidia card)

[louislgf]

was looking through the code for version 0.6.5 , seems like it doesn't do dog checks now?

not sure about footer yet

[Wrapzii]

does not check for footer in newest hotfix so thats better but speed of searching or finding add to cart did not increase...