Error when using cookiefile

[jakeman2048]

When attempting to use a cookiefile with linkchecker 9.3:

AttributeError: 'NoneType' object has no attribute 'from_file'
      cookies.from_file =  !AttributeError: 'NoneType' object has no attribute 'from_file'    line: for cookie in cookies.from_file(config["cookiefile"]):
      cookies.from_file =  !AttributeError: 'NoneType' object has no attribute 'from_file'System info:  File "/usr/local/lib/python2.7/dist-packages/linkcheck/director/aggregator.py", line 47, in new_request_session

Released on: 16.7.2014
Python 2.7.3 (default, Mar 13 2014, 11:03:55)
[GCC 4.7.2] on linux2
System info:
LinkChecker 9.3

Looks like it might have something to do with this (line 67 of linkcheck/director/aggregator.py):

self.cookies = None

[remko]

I get the same problem: specifying '--cookiefile somefile' makes link checker crash. Using LinkChecker 9.3 16.7.2014 on Mac OS X Yosemite.

Problem seems to be a clash of the cookies in new_request_session. I got it to compile by adding

from .. import cookies

at the top, and replacing new_request_session by:

def new_request_session(config, a_cookies):
    """Create a new request session."""
    session = requests.Session()
    if a_cookies:
        session.cookies = a_cookies

However, it doesn't work with just that, the merge_cookies doesn't expect a list (any more?) but a dict or another cookiejar, so that needs to be fixed too.

[kqr]

I also have this problem. Were there ever any solution?

[kvz]

Same problem here

[kvz]

Downgrading to 9.2 fixed it for me.

[kartagis]

@kvz how did you downgrade to 9.2? Also, could you send me the 9.2 version?

[kvz]

I saved LinkChecker==9.2 in my requirements.txt

[kartagis]

Thanks.

[kartagis]

@kvz I saved LinkChecker==9.2 in my requirements.txt too like you did, then did python setup.py sdist --manifest-only;python setup.py build;sudo setup.py install and linkchecker ended up being 9.4. How did you pull that off?

[candlerb]

Same problem with 9.3 (installed via "apt-get install linkchecker" on Ubuntu 18.04)

Attempted workaround:

apt-get remove --purge linkchecker
apt-get autoremove --purge
apt-get install python-pip
pip install linkchecker==9.2 requests==2.2.0

But it then fails with:

This program requires Python requests 2.2.0 or later.

This message comes from /usr/local/lib/python2.7/dist-packages/linkcheck/__init__.py

import requests
if requests.__version__ <= '2.2.0':
    raise SystemExit("This program requires Python requests 2.2.0 or later.")

So actually it requires 2.2.1 or later. And it fails with 2.18.4 (default Ubuntu 18.04 version) because:

>>> '2.18.4' < '2.2.0'
True

Workaround is to install 2.2.1 :-(

But it still doesn't work; cookies.from_file returns a nested list of cookies, whereas director.aggregator.new_request_session calls requests.cookies.merge_cookies which requires either a dict or CookieJar instance (and fails silently for other types of object)

Here is a quick frig to fix the worst of it:

--- cookies.py.orig	2020-11-25 11:49:14.665475155 +0000
+++ cookies.py	2020-11-25 12:07:23.251174002 +0000
@@ -36,12 +36,12 @@
             line = line.rstrip()
             if not line:
                 if lines:
-                    entries.append(from_headers("\r\n".join(lines)))
+                    entries.extend(from_headers("\r\n".join(lines)))
                 lines = []
             else:
                 lines.append(line)
         if lines:
-            entries.append(from_headers("\r\n".join(lines)))
+            entries.extend(from_headers("\r\n".join(lines)))
         return entries

--- director/aggregator.py.orig	2020-11-25 11:52:41.669800926 +0000
+++ director/aggregator.py	2020-11-25 11:58:09.426317182 +0000
@@ -39,7 +39,7 @@
     # XXX proxies
     if config["cookiefile"]:
         for cookie in cookies.from_file(config["cookiefile"]):
-            session.cookies = requests.cookies.merge_cookies(session.cookies, cookie)
+            session.cookies = requests.cookies.merge_cookies(session.cookies, {cookie.name: cookie.value})
     return session

But as far as I can see, this has never worked.