serpapi
[eBay API] Valid category ID searches not returning complete results
A customer reported that some category ID searches on eBay are not returning full results. The issue is intermittent and some category IDs function fine. I've provided examples of both non-functioning and functioning categories in the links below.
The only part we scrape are related_items known as "You May Also Like" on eBay. Otherwise, we only return an error saying "Ebay hasn't returned any results for this query". The HTML preview shows the full results available:
Public links: Playground 1 | Playground 2 | Playground 3 Working Category
Internal links: Intercom | Search Inspect 1 | Search Inspect 2 | Search Inspect 3 Working Category
Thank you @alexbarron - just looking for some further discussion / clarification.
I've noticed one of the direct eBay URL examples for a specific category (https://www.ebay.com/b/Womens-Clothing/15724/bn_661783) has content, but not in terms of those currently supported (no organic_results): https://serpapi.com/ebay-search-api
ie.
Just curious if this is more of a request to add those elements to our eBay parser? (ex. "shop by category", "limited time deals", "deals direct from the brand" etc.). Unless I am understanding this incorrectly? Thanks!
Thanks for looking into this @btaunt and good question. The customer reported that we're not scraping the organic results, but this may just be a region issue rather than a parser issue. Sorry for not catching this sooner.
The user shared a il.ebay.com link for Women's Clothing on eBay Israel which include organic results. You may need to open this in an incognito window. I got the Israeli store initially following this link, but after clicking something, I think a cookie got set switching me to the US store which legitimately does not serve organic results for this category as seen in your screenshot.
The Israeli page looks like this:
The customer tried setting _salic=100 for Israel in our API, but US results still get served matching your screenshot. In this case, it's legitimate there are no organic results, but the issue is we're not serving results localized to Israel.
We're currently missing support for Israel for the ebay_domain parameter. I suspect adding Israel could solve the issue. List of supported eBay domains.
If I change the ebay_domain to ebay.co.uk, I get organic results for the Women's Clothing category:
Regarding those other page elements, those would be nice to add support for eventually, but I believe those are unrelated to the customer's request.
Thanks for all the details @alexbarron!
What are your thoughts on this @ocvit, since you worked closely on this originally. It seems (from my testing) that the il.ebay.com simply displays / forwards to existing ebay.com. I'm assuming that it isn't actually supported?
I added the Israel-specific domain for testing, and targeted the appropriate /b/category_id etc, but I think that these organic results would be incorrect?
It requires separate locale parameter similar to the one I've implemented for eBay Product API.
Basically, it is controlled by dp1 cookie. For eBay Search it looks like:
"bbl/#{country_code}6bb0b758^"
# for Israel
"bbl/IL6bb0b758^"
Just passing it alongside the .com domain does the trick.
The customer followed up on this today, and confirmed it also happens for the US region.
More inspects for investigation - these are returned empty, although there are some blocks we were scraping before:
I haven't been able to recreate any searches where no organic_results are returned but otherwise available.
@marm123 Those last two examples are actually not showing any organic results when accessed directly, either. It looks like they're to top-level categories, and only final (or maybe just small-ish) categories actually show organic results.
Example:
- Clothing... -> Women - no organic results
- Clothing... -> Women -> Women's Clothing - no organic results
- Clothing... -> Women -> Women's Clothing -> Activewear - organic results
Thanks @trusche! Yes, these examples were more about search not returning any results while there are still some blocks visible in the HTML (e.g., we previously scraped "You may also like" block, and it's visible in the HTML, but not scraped in the JSON).
Although the initial premise was different, I think it still falls under the "searches not returning complete results" category. Do you think this should be a separate issue?
I also think that if the initial post was purely about organic_results missing, the example shown on the screenshot might not be relevant because of the reason you shared (these categories don't have organic_results). @alexbarron, can you clarify if these were valid bugs or potentially just categories without organic_results (expected)?
EDIT. If the latter, we should probably not return the Error message if there is still data other than organic_results, like in the screenshot.
@marm123 I'd suggest that we keep this issue narrowed to the one where a search yields organic results but we don't parse them. That's the case with certain combinations of location and category ID (the Israel examples), with blank search term. A fix for that is in progress.
Let's create a new issue for category searches without organic results if that is something customers need?
This issue is fixed for the case of localised category searches, which now return organic results again, for example:
Thank you, @trusche. I see that you reopened the issue. Was that by mistake or intentional?
Intentional, not sure if we wanted to close it before customer notifications?
Thanks for clarifying, @trusche. Yes, we want to close the issue when it's fixed and follow up with customers afterward. We monitor the closed issues queue, so if we see that the issue is not closed, we assume that there is still work that needs to be done.
Thanks @trusche @marm123 . This closes the issue with missing organic results on category pages, but doesn't cover the other elements on the page. The high volume customer was interested in both.
I've created a new issue for the other case. https://github.com/serpapi/public-roadmap/issues/3109