refactor code to separate API logic from core.py logic

[msramalho]

Description

TLDR: I (and surely others) cannot use unfurl as a python library but refactoring code.py would fix that.

The current core.py contains both the logic needed to do something like:

from unfurl.core import Unfurl

unfurl = Unfurl(remote_lookups=False)
unfurl.add_to_queue(data_type='url', key=None, value=url)
unfurl.parse_queue()
print(unfurl.generate_json())

as well as the logic needed to run the flask API.

That would normally not pose an issue except doing a fresh installation of the library does not seem to work for python 3.10 on linux (possibly other OSes and versions) since the API code uses flask-restx and the current version of that is using a try/catch import strategy (the reason is described in this SO question):

try:
    from flask.helpers import _endpoint_from_view_func
except ImportError:
    from flask.scaffold import _endpoint_from_view_func

the code inside try does not work on the setup I describe so the execution defaults to the catch block.

So, in order to run this library I would have to manually install Flask-Scaffold, but the problem is that doing so leads to a conflict dependency:

The conflict is caused by:
    dfir-unfurl 20230901 depends on flask>=2.2.0
    flask-scaffold 0.5.1 depends on Flask==0.10.1

The result is I cannot do from unfurl.core import Unfurl and thus cannot use the library :cry:

A solution I've tried that works is separating the flask logic into a new file that's not core.py so that doing the import does not lead us down the flask nightmare, that should be feasible since core.py does not need the flask logic, and only the flask logic needs core.py :eyes:

Would it make sense for you to separate these?

[msramalho]

Note: I could not choose the labels.

[obsidianforensics]

Hi there - thanks so much for the detailed issue; your write-up made it easy to understand the problem. I don't see an issue separating the Flask UI part out from core.py and into its own file, thanks for the suggestion! I'll try to work on this and get it changed for the next release.

[msramalho]

Thank you for considering the issue and having a look, I'd have liked to suggest the changes directly but don't have a lot of availability in the near future.

[obsidianforensics]

While working on #189, I encountered a similar issue with dependencies. I updated the versions of flask and Werkzeug to 2.3.0 and flask-restx to 1.2.0, and that seems to have resolved it.

I'll still work on separating out the web ui into its own file, but thought you'd like to know in the mean time.

[msramalho]

That's great news then! I'll keep an eye out for releases.

I figure I could try to do a monkey patch to temporarily solve it as suggested here but just using an empty function, since I don't expect it to be called.

[msramalho]

I see #189 has been merged but also that no new version has been released, do you know if there's a planned time for that?

[obsidianforensics]

Hi - I'm finishing up some other updates and plan on a release in the next few days - sorry this has taken so long!