anemone

anemone copied to clipboard

Avoid using URI.escape/unescape when Ruby 2.7 is running

2

Hi, Thanks for the useful gem! This PR suppresses the below errors. ``` /BUNDLE_ROOT/ruby/2.7.0/gems/anemone-0.7.2/lib/anemone/page.rb:157: warning: URI.unescape is obsolete /BUNDLE_ROOT/ruby/2.7.0/gems/anemone-0.7.2/lib/anemone/page.rb:157: warning: URI.escape is obsolete ```

ts-3156

With this new release we would have the ability to use Proxy with authentication

ghost

Support Phantomjs

1

Support Phantomjs

labolado

HTTP request header support

8

Hi, I needed to add some additional HTTP request headers and didn't see any support for that currently. Happy to change anything / add more specs to cover the change....

ashmckenzie

Support other charsets

1

I'm using Anemone. It's excelletnt!! But Garbled characters occur, if the page's charset is not UTF-8 or US-ASCII. So I want to support other charsets.

takuros

Give focus_crawl a chance to access page body before discarding it

3

For site-specific crawlers, it's fair enough to use `focus_crawl` like this: ``` anemone.focus_crawl do |page| if page.doc page.doc.search('.//a[@href]').map { |a| URI.parse(a[:href]) } else page.links end end ``` However when using...

lankz

Fix for URI::InvalidURIError: bad URI(is not URI?).

1

Fix for URI::InvalidURIError: bad URI(is not URI?). For example, URI('http://google.com/åäö') fails otherwise.

christianhellsten

Mongoid Storage

1

Also, add this model anemone_page.rb in your app ``` ruby class AnemonePage include Mongoid::Document field :url field :headers, type: Moped::BSON::Binary field :data, type: Moped::BSON::Binary field :body, type: Moped::BSON::Binary field :links,...

zakwanhaj

Added only_links_like Feature

1

Hi! Thanks a lot for anemone guys! I missed a feature that I have implemented, basically the reverse of `skip_links_like`, which I have called `only_links_like` to help the crawler to...

hartator

added gzip unzipping if the content-encoding for a fetched page is gzip

2

themusicman