The program keeps throwing exceptions

[ziqizh]

Edit: I digged deeper and realized that the webpage https://www.ebay.com/signin/s periodically forbids GET method. For example, you will get 405 error if you use wget https://www.ebay.com/signin/s When the GET method is allowed, the program runs into the second exception, NoSuchElementException.

The program throws two kinds of exception when logging in. Any ideas?

I ran it by: docker run -d -p 3647:3647 -e 'EBAY_USERNAME=myaccount' -e 'EBAY_PASSWORD=mypassword' ruippeixotog/ebay-snipe-server:0.2.3-SNAPSHOT

2019-12-05 14:40:11,019 INFO  Slf4jLogger - Slf4jLogger started
2019-12-05 14:40:11,112 INFO  SnipeServer$ - Logging in into eBay
2019-12-05 14:40:11,112 DEBUG EbayLoginManager - Getting the sign in cookie for ebay.com
Exception in thread "main" org.jsoup.HttpStatusException: HTTP error fetching URL. Status=405, URL=https://www.ebay.com/signin/s
	at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:760)
	at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:706)
	at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:299)
	at net.ruippeixotog.scalascraper.browser.JsoupBrowser.executeRequest(JsoupBrowser.scala:73)
	at net.ruippeixotog.scalascraper.browser.JsoupBrowser.$anonfun$executePipeline$3(JsoupBrowser.scala:84)
	at scala.Function1.$anonfun$andThen$1(Function1.scala:52)
	at scala.Function1.$anonfun$andThen$1(Function1.scala:52)
	at net.ruippeixotog.scalascraper.browser.JsoupBrowser.post(JsoupBrowser.scala:39)
	at net.ruippeixotog.scalascraper.browser.JsoupBrowser.post(JsoupBrowser.scala:30)
	at net.ruippeixotog.ebaysniper.ebay.EbayLoginManager.forceLogin(EbayLoginManager.scala:30)
	at net.ruippeixotog.ebaysniper.ebay.EbayClient.login(EbayClient.scala:30)
	at net.ruippeixotog.ebaysniper.SnipeServer$.delayedEndpoint$net$ruippeixotog$ebaysniper$SnipeServer$1(SnipeServer.scala:38)
	at net.ruippeixotog.ebaysniper.SnipeServer$delayedInit$body.apply(SnipeServer.scala:20)
	at scala.Function0.apply$mcV$sp(Function0.scala:34)
	at scala.Function0.apply$mcV$sp$(Function0.scala:34)
	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
	at scala.App.$anonfun$main$1$adapted(App.scala:76)
	at scala.collection.immutable.List.foreach(List.scala:389)
	at scala.App.main(App.scala:76)
	at scala.App.main$(App.scala:74)
	at net.ruippeixotog.ebaysniper.SnipeServer$.main(SnipeServer.scala:20)
	at net.ruippeixotog.ebaysniper.SnipeServer.main(SnipeServer.scala)

2019-12-04 20:43:43,160 INFO  Slf4jLogger - Slf4jLogger started
2019-12-04 20:43:43,251 INFO  SnipeServer$ - Logging in into eBay
2019-12-04 20:43:43,251 DEBUG EbayLoginManager - Getting the sign in cookie for ebay.com
Exception in thread "main" java.util.NoSuchElementException
	at java.util.ArrayList$Itr.next(ArrayList.java:862)
	at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:40)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:448)
	at scala.collection.IterableLike.head(IterableLike.scala:106)
	at scala.collection.IterableLike.head$(IterableLike.scala:105)
	at net.ruippeixotog.scalascraper.model.LazyElementQuery.head(ElementQuery.scala:40)
	at net.ruippeixotog.scalascraper.scraper.ContentExtractors$.net$ruippeixotog$scalascraper$scraper$ContentExtractors$$$anonfun$attr$1(ContentExtractors.scala:77)
	at net.ruippeixotog.scalascraper.scraper.ContentExtractors$$anonfun$attr$2.extract(ContentExtractors.scala:77)
	at net.ruippeixotog.scalascraper.scraper.ContentExtractors$$anonfun$attr$2.extract(ContentExtractors.scala:77)
	at net.ruippeixotog.scalascraper.scraper.HtmlExtractor.apply(HtmlExtractor.scala:23)
	at net.ruippeixotog.scalascraper.scraper.HtmlExtractor.apply$(HtmlExtractor.scala:23)
	at net.ruippeixotog.scalascraper.scraper.ContentExtractors$$anonfun$attr$2.apply(ContentExtractors.scala:77)
	at net.ruippeixotog.scalascraper.scraper.ContentExtractors$.net$ruippeixotog$scalascraper$scraper$ContentExtractors$$$anonfun$formDataAndAction$1(ContentExtractors.scala:103)
	at net.ruippeixotog.scalascraper.scraper.ContentExtractors$$anonfun$8.extract(ContentExtractors.scala:102)
	at net.ruippeixotog.scalascraper.scraper.ContentExtractors$$anonfun$8.extract(ContentExtractors.scala:102)
	at net.ruippeixotog.scalascraper.scraper.HtmlExtractor.$anonfun$mapQuery$1(HtmlExtractor.scala:43)
	at net.ruippeixotog.scalascraper.scraper.HtmlExtractor$$anon$2.extract(HtmlExtractor.scala:60)
	at net.ruippeixotog.scalascraper.dsl.ScrapingOps$ElementsScrapingOps.$anonfun$extract$1(ScrapingOps.scala:15)
	at scalaz.Monad.$anonfun$map$2(Monad.scala:14)
	at scalaz.IdInstances$$anon$1.point(Id.scala:20)
	at scalaz.Monad.$anonfun$map$1(Monad.scala:14)
	at scalaz.IdInstances$$anon$1.bind(Id.scala:22)
	at scalaz.Monad.map(Monad.scala:14)
	at scalaz.Monad.map$(Monad.scala:14)
	at scalaz.IdInstances$$anon$1.map(Id.scala:19)
	at scalaz.syntax.FunctorOps.map(FunctorSyntax.scala:10)
	at net.ruippeixotog.scalascraper.dsl.ScrapingOps$ElementsScrapingOps.extract(ScrapingOps.scala:15)
	at net.ruippeixotog.scalascraper.dsl.ScrapingOps$ElementsScrapingOps.$greater$greater(ScrapingOps.scala:19)
	at net.ruippeixotog.ebaysniper.ebay.EbayLoginManager.forceLogin(EbayLoginManager.scala:27)
	at net.ruippeixotog.ebaysniper.ebay.EbayClient.login(EbayClient.scala:30)
	at net.ruippeixotog.ebaysniper.SnipeServer$.delayedEndpoint$net$ruippeixotog$ebaysniper$SnipeServer$1(SnipeServer.scala:38)
	at net.ruippeixotog.ebaysniper.SnipeServer$delayedInit$body.apply(SnipeServer.scala:20)
	at scala.Function0.apply$mcV$sp(Function0.scala:34)
	at scala.Function0.apply$mcV$sp$(Function0.scala:34)
	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
	at scala.App.$anonfun$main$1$adapted(App.scala:76)
	at scala.collection.immutable.List.foreach(List.scala:389)
	at scala.App.main(App.scala:76)
	at scala.App.main$(App.scala:74)
	at net.ruippeixotog.ebaysniper.SnipeServer$.main(SnipeServer.scala:20)
	at net.ruippeixotog.ebaysniper.SnipeServer.main(SnipeServer.scala)

[tryangle]

I get this as well. I don't know if it makes a difference but I am running: java -version openjdk version "1.8.0_222" OpenJDK Runtime Environment (build 1.8.0_222-8u222-b10-1ubuntu1~16.04.1-b10) OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)

[ruippeixotog]

It may be because eBay changed their sign in page. I'll take a look at this.

[ruippeixotog]

Unfortunately this is happening because they are now showing Captcha pages to prevent automated login:

<!doctype html>
<html>
 <head> 
  <meta NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"> 
  <meta http-equiv="cache-control" content="max-age=0"> 
  <meta http-equiv="cache-control" content="no-cache"> 
  <meta http-equiv="expires" content="0"> 
  <meta http-equiv="expires" content="Tue, 01 Jan 1980 1:00:00 GMT"> 
  <meta http-equiv="pragma" content="no-cache"> 
  <meta http-equiv="refresh" content="10; url=/distil_r_captcha.html?requestId=98a3a58d-e29a-4360-9916-b25a9b969158&amp;httpReferrer=%2Fsignin%2F"> 
  <script type="text/javascript">
	(function(window){
		try {
			if (typeof sessionStorage !== 'undefined'){
				sessionStorage.setItem('distil_referrer', document.referrer);
			}
		} catch (e){}
	})(window);
</script> 
  <script type="text/javascript" src="/nkfytkqtoxtljvzb.js" defer></script>
  <style type="text/css">#d__fFH{position:absolute;top:-5000px;left:-5000px}#d__fF{font-family:serif;font-size:200px;visibility:hidden}#ryutavyvevzbtareszzbwcaq{display:none!important}</style>
 </head> 
 <body> 
  <div id="distilIdentificationBlock">
   &nbsp;
  </div>   
 </body>
</html>

[geotheory]

Ah. Does this mean R.I.P.?

[ziqizh]

IMO we can solve this by enabling JavaScript. Not sure how to do it tho

[geotheory]

Possibly https://medium.com/@jsoverson/bypassing-captchas-with-headless-chrome-93f294518337

[ruippeixotog]

Enabling JavaScript is easy enough, it suffices to use an HtmlUnitBrowser instead of a JsoupBrowser in EbayClient.scala. But that is not enough, the server still gets FailingHttpStatusCodeException: 405 Method Not Allowed for https://www.ebay.com/signin/s.

The sole purpose of captchas are to prevent people to automate actions throughout the website. Even if someone found a way to do bypass them today I'm sure eBay will continue to close those holes. It's a cat-and-mouse game I don't have the time or willingness to pursue, so unless someone else wants to play with the project, it's R.I.P. 🙁

[geotheory]

Probably exposing my ignorance of the authentication process, but I wonder if a manual workaround might be to log in manually on a regular browser, copy the cookie across to your server machine and make available to that login request?

[ruippeixotog]

It all depends on how they are checking for automation. We can copy cookies as a way to bypass captchas on authentication, but there's nothing preventing eBay from adding captchas on the other requests and actions too (if they haven't already). Simple engines like jsoup and HtmlUnit are easily detectable; real browsers controlled by a web driver like Selenium may have a better chance, but it's still a considerable effort to implement support for them and there's no guarantee it will work.

[geotheory]

Bah I miss this

[anoduck]

So in the great words of Dr. McCoy, Chief Surgeon of the starship Enterprise, "It's dead, Jim."

[wicadmin]

What is everyone else using these days?

[anoduck]

Just by using duckduckgo to search for "ebay snipe" I came across this project. It was written within 9 months ago as of date of posting, and hopefully still might work.

https://sourceforge.net/projects/esniper/