exotic-amazon
exotic-amazon copied to clipboard
platonai
A complete solution to crawl amazon at scale completely and accurately.
 想问问这个问题怎么解决
请问单机每天能处理多少页面请求，需要做关键字下asin采集，单机内存32G，日采集能达到100万级别吗
https://www.woolworths.co.za/prod/Women/New-In/Knitwear/Woolworths-Cashmere-Jumper/A-507948144?isFromPLP=true https://superbalist.com/women/beauty/nutricosmetics/glow-from-within-single-serving-sachets-10g/1016223?clickRef=catalogue https://superbalist.com/women/beauty/nutricosmetics/glow-from-within-pure-hydrolyzed-collagen-250g/925744?clickRef=catalogue 不知道为什么，这些url在爬取时会报no url found，但是别的url不会
我再application文件种修改mongodb配置项，但是运行后日志中仍有尝试连接默认mongoDB 配置为 ``` gora.mongodb.override_hadoop_configuration=true gora.mongodb.servers=127.0.0.1:17017 gora.mongodb.login=root gora.mongodb.secret=xxxxxxxx spring.data.mongodb.uri=mongodb://root:[email protected]:17017/scent ``` 日志中关键信息： ``` 2023-11-11 20:40:45.774 INFO [main] o.m.d.cluster - Cluster created with settings {hosts=[127.0.0.1:17017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms'} 2023-11-11 20:40:45.940 INFO [.0.1:17017]...
Hi There, Thanks for opening source this great project. I tried the code with prod mode and kept meeting the following issues: WARN a.p.p.p.b.e.i.BrowserEmulatorImplBase - 36431. Page is ROBOT_CHECK(11.54 KiB)...
在爬取两个小时后报Driver pool is exhausted numRunning: 0, availableMemory: 452.54 MiB, memoryToReserve: 1.00 GiB, shortage: -599216128 B 这个是如何释放的？有手动释放的方法吗
你好，关于以上问题，我在范例里面看到可以通过设置系统参数 System.setProperty( DefaultWebSocketContainerFactory.WEBSOCKET_INCOMING_BUFFER_PROPERTY, Long.toString((long) DefaultWebSocketContainerFactory.MB * 24)); 来解决，但是我在代码里设置后，还是不能正常转化我需要爬取的网页。 1.请问我是否要像范例里一样，对page.onLoadEventFired进行重写？ 2.请问这个Web socket的默认超时时间是多久？可以重新设置吗?