scrapify

scrapify copied to clipboard

Support for Basic Authentication

2

Some pages require basic authentication. Add support for username, password before crawling pages ``` class Library html "http://mylibrary.com" username "foo" password "bar" end ```

sathish316

Support for crawling authenticated content. Use Mechanize instead of Nokogiri to login and retain session info before crawling content that requires authentication Example: ``` class Project html "http://jira.myproject.com" login "http://jira.myproject.com/login"...

sathish316

Crawl data across pages by extracting from an array of html urls ``` class Book include Scrapify::Base html ["http://www.bookstore.com/fiction/1", "http://www.bookstore.com/fiction/2", "http://www.bookstore.com/fiction/3"] end ```

sathish316

Support optional datatype for each attribute Example: ``` class IMDB attribute :title, type: String attribute :votes, type: Integer attribute :rating, type: Decimal attribute :released_data, type: DateTime end ```

sathish316

All paginated content have some way of moving to next page. Get extra selector to fetch next page and crawl till last page is reached Example: ``` class Book include...

sathish316

Support crawling of paginated content using placeholder for page in html url and range or array of pages Example: ``` class Book include Scrapify::Base html "http://www.bookstore.com/fiction/:page", page: 1..100 end ```

sathish316

Support for find all with conditions like: ``` IMDB.all(director: 'Christopher Nolan') ``` OR ``` IMDB.where(director: 'Christopher Nolan') ```

sathish316

Tolerance to malformed XML

1

Is Scrapify tolerant to errors in XML like unmached tags and so on? Cheers

franciscolourenco