exotic-amazon icon indicating copy to clipboard operation
exotic-amazon copied to clipboard

单一资源模式采集 amazon.com 出现 503 错误

Open swlcyx opened this issue 1 year ago • 2 comments

14:29:19.835 [r-worker-9] INFO a.p.p.c.component.LoadComponent.Task - 29745. 💔 ⚡ U for N got 1462 0 <- 0 in 1.292s, fc:1/1 Exception(1462) httpCode: 503 | 5ZwMFW35 | search_keywords | https://www.amazon.com/s?k=teething+toy+for+dogs&language=en_US&page=2 -expires PT24H -ignoreFailure -isResource -label search_keywords -parse -refresh -requireImages 50 -requireSize 3000000

用代理爬取会出现代理返回的过期时间是半个小时后,但是此代理爬取几分钟后会大概率出现被亚马逊识别后一直返回503页面,针对此情况会有60%左右的链接出现上面的info,想问下这部分503 的页面是被取消爬取了还是会进行重试,以及代理部分是否是等到代理过期才会换代理,还是代理多次返回503后就进行更换代理

swlcyx avatar Apr 11 '23 06:04 swlcyx