General web scraper feature

[rkantos]

I am requesting a feature to make it possible for the user to configure a web scraper for scraping product names and why not also prices.

In general many grocery shops at least in Europe provide a way to find products by just going to their website.tld/EAN. Then often it is just the case of finding the correct selector/XPath with the product name and price but also other details of course. Naturally sometimes these selectors need to be updated, but I think that is a minor issue. Alternatively someone probably can come up with more thought out ways to extract the details programmatically without using the css selectors. Of course one way to have the selectors up-to-date would be to have it crowd-sourced. I like this idea since it highlights the community.

Basic layout I can imagine for the app:

• Settings section - Webshop Scrapers
  • Scraping website.tld (with tooltip explaining that the scraper only works with webshops that work with website.tld/EAN URLs)
  • Product name CSS-selector (with tooltip on how to find one easily with say Chrome)
  • Product price CSS-selector
  • User Agent configuration (some defaults with configurable custom UA)

[rkantos]

One idea for crowd-sourcing the CSS-selectors would be to make service similar to https://hubhop.mobiflight.com/presets/ https://github.com/MobiFlight/HubHop-Website

This database integrates directly in to the Mobiflight user application that then utilizes the variables to communicate with the "service" or Microsoft Flight Simulator.

[rkantos]

For reference:

As such it won't work with the iOS app

Having this supported by any of the great community contributed apps is just another different topic, not tracked here.

I personally use grocy daily on Android, without an app resp. the app is called "Firefox". As long as iOS has a webbrowser, grocy and Barcode Buddy will work also there.

Since Barcode Buddy provides all and would be kind of obsolete when implementing everything in grocy itself, it's just not on my list of the things to do next for grocy. I know, unpopular opinion, among so much others.

And fun fact: Grocy Android also already has an OFF integration since v2.0.0.

The Grocy-Android indeed has great functionality with it's MLKit barcode scanner and Open Food Fact query/scraping. I noticed this functionality a couple weeks back and decided to add support to Grocy for the grocery store that I use the most. You can get it from here and modify for your own shop of course: https://github.com/rkantos/grocy-android/commit/c0131e53bdd0bfc2b6672a597ac11778a9a0d1b1

You only need to edit the URL here: https://github.com/rkantos/grocy-android/blob/master/app/src/main/java/xyz/zedler/patrick/grocy/api/FoodieApi.java#L41

and the css-selector for product name here: https://github.com/rkantos/grocy-android/blob/master/app/src/main/java/xyz/zedler/patrick/grocy/helper/DownloadHelper.java#L2808

I didn't fully integrate it with Grocy-Android as how OFF (tooltips etc) is done as I didn't plan to PR it. Instead I made a request to Grocy-Android for a more generic feature as I see it could very easily be added to it (might even do it myself): https://github.com/patzly/grocy-android/issues/556

Originally posted by @rkantos in https://github.com/grocy/grocy/issues/158#issuecomment-1156381243

[dominiczedler]

We decided to close issues where we think that they will never be implemented. We have limited time for this project and thus we want to concentrate on the important things and enhancements which keep the app running.

You can always contribute with code which can be used in production environment.