ipwb icon indicating copy to clipboard operation
ipwb copied to clipboard
verified publisher icon oduwsdl

Entering the sample URL on welcome page doesn't work for me

Open akavel opened this issue 8 years ago • 6 comments

either ;) [maybe related to #310, maybe not]

I tried typing the URL mentioned in readme (http://www.cs.odu.edu/~salam/) in the "input box" on the "welcome page":

issue-ipwb-sample-input-field

C:\dnload\ipfs-etc>\python27\python.exe ipwb/ipwb replay
This version of ipwb is outdated. Please run pip install --upgrade ipwb.
Executing first-run procedure on provided sample data.
Processing WARC records in salam-home.warc complete
 * Running on http://localhost:5000/ (Press CTRL+C to quit)
Getting CDXJ Lines with the URI-R http://www.cs.odu.edu/~salam/ from samples/indexes/sample-encrypted.cdxj
Getting CDXJ Lines with http://www.cs.odu.edu/~salam/ in ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
Could not edu,odu,cs)/~salam/ in CDXJ at ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
127.0.0.1 - - [08/Dec/2017 20:33:43] "GET /memento/*/http://www.cs.odu.edu/~salam/ HTTP/1.1" 200 -

akavel avatar Dec 08 '17 19:12 akavel

This has to do with how ipwb replay is canonicalizing the URI-R entered in the input box to a SURT prior to searching in the CDXJ. The search is for edu,odu,cs/~salam per

Could not find edu,odu,cs)/~salam/ in CDXJ at /usr/local/lib/python2.7/site-packages/ipwb/samples/indexes/sample-encrypted.cdxj

when the mementos in the sample files are for edu,odu,cs)/~salam (sans slash). The README has been corrected and currently does not refer to either, as it was being used incorrectly. Searching for http://cs.odu.edu/~salam should work now in the input box.

I will need to document an example of using this search box. @akavel Please do a fresh pull and let me know if you are unable to search for http://cs.odu.edu/~salam in the input box.

machawk1 avatar Dec 09 '17 01:12 machawk1

Getting redirected to below URL now:

http://localhost:5000/20160305192247/cs.odu.edu/~salam

and logs are as below:

C:\dnload\ipfs-etc>\python27\python.exe ipwb/ipwb replay
Executing first-run procedure on provided sample data.
Processing WARC records in salam-home.warc complete
IPWB replay started on http://localhost:5000
Retrieving URI-Ms from ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
Retrieving URI-Ms from ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
Getting CDXJ Lines with the URI-R http://cs.odu.edu/~salam from samples/indexes/sample-encrypted.cdxj
Getting CDXJ Lines with http://cs.odu.edu/~salam in ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
CDXJ Line: edu,odu,cs)/~salam 20160305192247 {"locator": "urn:ipfs/QmeVWGtnfuJ1QnpmtKKnyArVgEpq7v31kktEfh6c8mDiXE/QmZWKQRBNXNrVZ69LoGpMNJi5NU66gDhnGtQukWJepv7Kr", "encryption_method": "xor", "encryption_key": "radon", "mime_type": "text/html", "status_code": "200"}
Unknown exception occurred while fetching from ipfs.
'module' object has no attribute 'exec_info'

Didn't try to diff it, so not 100% sure, but at a glance looks like a dupe of #310 to me now.

akavel avatar Dec 09 '17 15:12 akavel

Ugh, typo with exec_info, fixed in #323 and now live in master. Please perform the above again and let me know what error you get.

machawk1 avatar Dec 09 '17 15:12 machawk1

Weirdly, when I enter the address with a raw tilde (http://www.cs.odu.edu/~salam/), it seems to still take me to the .../memento/*/... URL:

http://localhost:5000/memento/*/http://www.cs.odu.edu/~salam/

with log like below:

C:\dnload\ipfs-etc>\python27\python.exe ipwb/ipwb replay
Executing first-run procedure on provided sample data.
Processing WARC records in salam-home.warc complete
IPWB replay started on http://localhost:5000
Retrieving URI-Ms from ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
Retrieving URI-Ms from ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
Getting CDXJ Lines with the URI-R http://www.cs.odu.edu/~salam/ from samples/indexes/sample-encrypted.cdxj
Getting CDXJ Lines with http://www.cs.odu.edu/~salam/ in ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
Could not find edu,odu,cs)/~salam/ in CDXJ at ipwb/ipwb\samples\indexes\sample-encrypted.cdxj

But if I right-click the address you suggested in a comment and pick "Copy link address..." (or whatever it is in English), then Firefox copies it urlencoded (as http://cs.odu.edu/%7Esalam), which seems to take me to an URL resembling the one in #310:

http://localhost:5000/20160305192247/cs.odu.edu/~salam

with logs looking similar (as #310) too:

C:\dnload\ipfs-etc>\python27\python.exe ipwb/ipwb replay
Executing first-run procedure on provided sample data.
Processing WARC records in salam-home.warc complete
IPWB replay started on http://localhost:5000
Retrieving URI-Ms from ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
Retrieving URI-Ms from ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
Getting CDXJ Lines with the URI-R http://cs.odu.edu/~salam from samples/indexes/sample-encrypted.cdxj
Getting CDXJ Lines with http://cs.odu.edu/~salam in ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
CDXJ Line: edu,odu,cs)/~salam 20160305192247 {"locator": "urn:ipfs/QmeVWGtnfuJ1QnpmtKKnyArVgEpq7v31kktEfh6c8mDiXE/QmZWKQRBNXNrVZ69LoGpMNJi5NU66gDhnGtQukWJepv7Kr", "encryption_method": "xor", "encryption_key": "radon", "mime_type": "text/html", "status_code": "200"}
Unknown exception occurred while fetching from ipfs.
<type 'exceptions.AttributeError'>
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 56954)
Traceback (most recent call last):
  File "C:\python27\lib\SocketServer.py", line 290, in _handle_request_noblock
    self.process_request(request, client_address)
  File "C:\python27\lib\SocketServer.py", line 318, in process_request
    self.finish_request(request, client_address)
  File "C:\python27\lib\SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\python27\lib\SocketServer.py", line 652, in __init__
    self.handle()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 290, in handle
    rv = BaseHTTPRequestHandler.handle(self)
  File "C:\python27\lib\BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 325, in handle_one_request
    return self.run_wsgi()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 267, in run_wsgi
    execute(self.server.app)
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 255, in execute
    application_iter = app(environ, start_response)
  File "C:\python27\lib\site-packages\flask\app.py", line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File "C:\python27\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "C:\python27\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "C:\python27\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "ipwb/ipwb\replay.py", line 399, in showMementoAtDatetime
    return show_uri(urir, datetime)
  File "ipwb/ipwb\replay.py", line 534, in show_uri
    sys.exit()
SystemExit
----------------------------------------
CDXJ Line: edu,odu,cs)/~salam 20160305192247 {"locator": "urn:ipfs/QmeVWGtnfuJ1QnpmtKKnyArVgEpq7v31kktEfh6c8mDiXE/QmZWKQRBNXNrVZ69LoGpMNJi5NU66gDhnGtQukWJepv7Kr", "encryption_method": "xor", "encryption_key": "radon", "mime_type": "text/html", "status_code": "200"}
Unknown exception occurred while fetching from ipfs.
<type 'exceptions.AttributeError'>
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 56956)
Traceback (most recent call last):
  File "C:\python27\lib\SocketServer.py", line 290, in _handle_request_noblock
    self.process_request(request, client_address)
  File "C:\python27\lib\SocketServer.py", line 318, in process_request
    self.finish_request(request, client_address)
  File "C:\python27\lib\SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\python27\lib\SocketServer.py", line 652, in __init__
    self.handle()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 290, in handle
    rv = BaseHTTPRequestHandler.handle(self)
  File "C:\python27\lib\BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 325, in handle_one_request
    return self.run_wsgi()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 267, in run_wsgi
    execute(self.server.app)
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 255, in execute
    application_iter = app(environ, start_response)
  File "C:\python27\lib\site-packages\flask\app.py", line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File "C:\python27\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "C:\python27\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "C:\python27\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "ipwb/ipwb\replay.py", line 399, in showMementoAtDatetime
    return show_uri(urir, datetime)
  File "ipwb/ipwb\replay.py", line 534, in show_uri
    sys.exit()
SystemExit
----------------------------------------
CDXJ Line: edu,odu,cs)/~salam 20160305192247 {"locator": "urn:ipfs/QmeVWGtnfuJ1QnpmtKKnyArVgEpq7v31kktEfh6c8mDiXE/QmZWKQRBNXNrVZ69LoGpMNJi5NU66gDhnGtQukWJepv7Kr", "encryption_method": "xor", "encryption_key": "radon", "mime_type": "text/html", "status_code": "200"}
Unknown exception occurred while fetching from ipfs.
<type 'exceptions.AttributeError'>
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 56958)
Traceback (most recent call last):
  File "C:\python27\lib\SocketServer.py", line 290, in _handle_request_noblock
    self.process_request(request, client_address)
  File "C:\python27\lib\SocketServer.py", line 318, in process_request
    self.finish_request(request, client_address)
  File "C:\python27\lib\SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\python27\lib\SocketServer.py", line 652, in __init__
    self.handle()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 290, in handle
    rv = BaseHTTPRequestHandler.handle(self)
  File "C:\python27\lib\BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 325, in handle_one_request
    return self.run_wsgi()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 267, in run_wsgi
    execute(self.server.app)
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 255, in execute
    application_iter = app(environ, start_response)
  File "C:\python27\lib\site-packages\flask\app.py", line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File "C:\python27\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "C:\python27\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "C:\python27\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "ipwb/ipwb\replay.py", line 399, in showMementoAtDatetime
    return show_uri(urir, datetime)
  File "ipwb/ipwb\replay.py", line 534, in show_uri
    sys.exit()
SystemExit
----------------------------------------
CDXJ Line: edu,odu,cs)/~salam 20160305192247 {"locator": "urn:ipfs/QmeVWGtnfuJ1QnpmtKKnyArVgEpq7v31kktEfh6c8mDiXE/QmZWKQRBNXNrVZ69LoGpMNJi5NU66gDhnGtQukWJepv7Kr", "encryption_method": "xor", "encryption_key": "radon", "mime_type": "text/html", "status_code": "200"}
Unknown exception occurred while fetching from ipfs.
<type 'exceptions.AttributeError'>
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 56960)
Traceback (most recent call last):
  File "C:\python27\lib\SocketServer.py", line 290, in _handle_request_noblock
    self.process_request(request, client_address)
  File "C:\python27\lib\SocketServer.py", line 318, in process_request
    self.finish_request(request, client_address)
  File "C:\python27\lib\SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\python27\lib\SocketServer.py", line 652, in __init__
    self.handle()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 290, in handle
    rv = BaseHTTPRequestHandler.handle(self)
  File "C:\python27\lib\BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 325, in handle_one_request
    return self.run_wsgi()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 267, in run_wsgi
    execute(self.server.app)
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 255, in execute
    application_iter = app(environ, start_response)
  File "C:\python27\lib\site-packages\flask\app.py", line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File "C:\python27\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "C:\python27\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "C:\python27\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "ipwb/ipwb\replay.py", line 399, in showMementoAtDatetime
    return show_uri(urir, datetime)
  File "ipwb/ipwb\replay.py", line 534, in show_uri
    sys.exit()
SystemExit
----------------------------------------
CDXJ Line: edu,odu,cs)/~salam 20160305192247 {"locator": "urn:ipfs/QmeVWGtnfuJ1QnpmtKKnyArVgEpq7v31kktEfh6c8mDiXE/QmZWKQRBNXNrVZ69LoGpMNJi5NU66gDhnGtQukWJepv7Kr", "encryption_method": "xor", "encryption_key": "radon", "mime_type": "text/html", "status_code": "200"}
Unknown exception occurred while fetching from ipfs.
<type 'exceptions.AttributeError'>
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 56962)
Traceback (most recent call last):
  File "C:\python27\lib\SocketServer.py", line 290, in _handle_request_noblock
    self.process_request(request, client_address)
  File "C:\python27\lib\SocketServer.py", line 318, in process_request
    self.finish_request(request, client_address)
  File "C:\python27\lib\SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\python27\lib\SocketServer.py", line 652, in __init__
    self.handle()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 290, in handle
    rv = BaseHTTPRequestHandler.handle(self)
  File "C:\python27\lib\BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 325, in handle_one_request
    return self.run_wsgi()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 267, in run_wsgi
    execute(self.server.app)
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 255, in execute
    application_iter = app(environ, start_response)
  File "C:\python27\lib\site-packages\flask\app.py", line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File "C:\python27\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "C:\python27\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "C:\python27\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "ipwb/ipwb\replay.py", line 399, in showMementoAtDatetime
    return show_uri(urir, datetime)
  File "ipwb/ipwb\replay.py", line 534, in show_uri
    sys.exit()
SystemExit
----------------------------------------
CDXJ Line: edu,odu,cs)/~salam 20160305192247 {"locator": "urn:ipfs/QmeVWGtnfuJ1QnpmtKKnyArVgEpq7v31kktEfh6c8mDiXE/QmZWKQRBNXNrVZ69LoGpMNJi5NU66gDhnGtQukWJepv7Kr", "encryption_method": "xor", "encryption_key": "radon", "mime_type": "text/html", "status_code": "200"}
Unknown exception occurred while fetching from ipfs.
<type 'exceptions.AttributeError'>
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 56964)
Traceback (most recent call last):
  File "C:\python27\lib\SocketServer.py", line 290, in _handle_request_noblock
    self.process_request(request, client_address)
  File "C:\python27\lib\SocketServer.py", line 318, in process_request
    self.finish_request(request, client_address)
  File "C:\python27\lib\SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\python27\lib\SocketServer.py", line 652, in __init__
    self.handle()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 290, in handle
    rv = BaseHTTPRequestHandler.handle(self)
  File "C:\python27\lib\BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 325, in handle_one_request
    return self.run_wsgi()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 267, in run_wsgi
    execute(self.server.app)
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 255, in execute
    application_iter = app(environ, start_response)
  File "C:\python27\lib\site-packages\flask\app.py", line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File "C:\python27\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "C:\python27\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "C:\python27\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "ipwb/ipwb\replay.py", line 399, in showMementoAtDatetime
    return show_uri(urir, datetime)
  File "ipwb/ipwb\replay.py", line 534, in show_uri
    sys.exit()
SystemExit
[...]

And did I try doing Ctrl-Shift-R each time.

akavel avatar Dec 09 '17 16:12 akavel

I am having a hard time replicating this but will continue until I can do so. Here's one dead-end I just tried, documented here so I don't go down that path again.

I thought at one point the replay system was using the CDXJ file cached by reference in the ~/.ipfs/config, which would cause the lookup to not resolve. This was spurred by a request for the URI-R http://www.cs.odu.edu/~salam/ producing an error of "not found in XXX CDXJ", which yielded memento.us CDXJ entries when dereferenced on my local file system. I was positive that I was running just ipwb replay to invoke the replay system. I nuked my IPFS config, ran ipfs init again, started ipwb w/ 5mementos WARC, killed ipwb replay, start with ipwb replay and could not replicate the issue.

Another theory is that the IPFSAPI.cat function is timing out. This is present in the try block that is throwing the exception. That the exception is an AttributionError is weird but I will need to look at the IPFSAPI and the signal module docs to see if that is the reason. Perhaps when the entry is not found in IPFS, this exception is thrown. Why the payload is not found after supposedly just being added by firstRun() is an additional issue that will need to be investigated if this is the case.

machawk1 avatar Dec 12 '17 02:12 machawk1

Weirdly, when I enter the address with a raw tilde (http://www.cs.odu.edu/~salam/), it seems to still take me to the .../memento/*/... URL:

http://localhost:5000/memento/*/http://www.cs.odu.edu/~salam/

with empty result page and log like below:

C:\dnload\ipfs-etc>\python27\python.exe ipwb/ipwb replay
Executing first-run procedure on provided sample data.
Processing WARC records in salam-home.warc complete
IPWB replay started on http://localhost:5000
Retrieving URI-Ms from ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
Retrieving URI-Ms from ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
Getting CDXJ Lines with the URI-R http://www.cs.odu.edu/~salam/ from samples/indexes/sample-encrypted.cdxj
Getting CDXJ Lines with http://www.cs.odu.edu/~salam/ in ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
Could not find edu,odu,cs)/~salam/ in CDXJ at ipwb/ipwb\samples\indexes\sample-encrypted.cdxj

But if I right-click the address you suggested in a comment and pick "Copy link address..." (or whatever it is in English), then Firefox copies it urlencoded (as http://cs.odu.edu/%7Esalam), which then takes me to the same URL as in #310:

http://localhost:5000/20160305192247/cs.odu.edu/~salam

which works for me with the patch I described in #310.

akavel avatar Dec 14 '17 21:12 akavel