Contextualise feedback

[brettkromkamp]

@machawk1 Hi Mat, not sure if you have actually taken a look at Contextualise yet. But, if you have, I would love to hear what you think of it. What works? What really doesn't? Thanks in advance.

[machawk1]

Hi @brettkromkamp, not just yet. I am still in the process of getting it working. I will provide feedback as soon as I can accomplish that then have a look around and provide feedback.

[brettkromkamp]

@machawk1 Still trying to get it to work? OMG! Is there anything I can help with? I presume it is PostgreSQL that is giving you the most problems, or?

[machawk1]

@brettkromkamp I don't want to chalk it up to PostgreSQL just yet. Using the Docker configuration, I should be seeing something at 127.0.0.1:5000 or localhost:5000 but don't. I was hoping that @ibnesayeed would provide some feedback on getting it working with his extensive Docker familiarity, but just got the suggestion to prematurely refactor/orchestrate using Docker Compose. :\

Should I be able to see some interface even if PostgreSQL is not yet correctly set up?

As I get a spare moment, I will work on it further, but those sort of moments come in small bursts. I am intent on getting it working, as I anticipate the project to be personally useful. :)

[brettkromkamp]

@machawk1 This is what I see when PostgreSQL is not up. So, if the database is not correctly set up, you should be seeing something (a stack trace, etc.) in the browser because Flask is running in development mode. So, if you are not seeing anything then I think it’s Docker-related.

[brettkromkamp]

@machawk1 No stress, though! I understand that this is something you can only work on now and then. I just don't want you hitting your head on a brick wall if I can help with something.

[machawk1]

@brettkromkamp I was expecting something akin to a stack trace as you mentioned. That would give me some indication that requests are hitting the app in the container. I think it is probably a simple fix but have yet to loop back to work on it more.

[ibnesayeed]

I was hoping that @ibnesayeed would provide some feedback on getting it working with his extensive Docker familiarity, but just got the suggestion to prematurely refactor/orchestrate using Docker Compose.

I was not pushing for a prematurely refactor, but I believed (based on my "extensive Docker familiarity") that a micro-service based set up here would be much easier than having a single monolithic Docker image, even for experimentation. Unfortunately, at that time the documentation of this repo was not up to the mark where I could understand the working without hunting through the code. The repo does not even contain a sample config file, but the documentation suddenly introduces one. I have noticed that only the dbname, username, and password fields of the DB are configurable in this repo, but to make it work with DB in a separate container it should also expose a way to configure dbhost (which defaults to localhost in the topic-db repo).

A monolithic Docker image:

• would need an init system (such as supervisord or a custom script) to ensure all the necessary services are up and running
• won't be able to leverage official images of various components (such as the official image of PostgeSQL) as a result the responsibility to install all necessary components, configure, and initialize will be of a single image
• would take longer to build and layers will not be utilized efficiently
• will cause difficulty in migrating database to a newer version

If the DB were to run in a separate container, this application needs to be simply set up to talk to that other container for storage and for that Docker will make the name of the service as as a resolvable host as long as both share the same Docker network (which is easy to setup in a compose file). That means if the name of the database service is db and it runs the process on port 5432, the application service can talk to db:5432 instead of localhost:5432.

[brettkromkamp]

@ibnesayeed Thanks for pointing out deficiencies in both Contextualise and TopicDB... much appreciated. I will fix.

[ibnesayeed]

Thanks for pointing out deficiencies in both Contextualise and TopicDB... much appreciated. I will fix.

No repository is perfect from the day one, they are improved over time with collaboration, constructive feedback, and contributions. I appreciate your work, though I am yet to explore the utility of this tool in my workflow and the maturity of application.

A side question, did you consider making topic-db database agnostic with the help of Object Relational Mapping libraries? Unless you are using features of Postgre that are essential for the application to function and are not present in other DBs, you might benefit from using a low-barrier DB (such as SQLite) for beginners, but allow utilization of more scalable DBs when necessary.

[brettkromkamp]

@ibnesayeed TopicDB used SQLite in its early phase of development. Nonetheless, it was always my intention to switch to PostgreSQL with it being the database that I am most familiar with. In theory after some minor refactoring it should be possible using some kind of connection-specific OO strategy pattern for TopicDB and, by extension, Contextualise to be database-agnostic. Saying that, I'm not sure if you have seen the list of, in their majority, user-facing issues that need to be addressed and I expect that my focus will be on these before anything else, really.

[ibnesayeed]

Since I do not know much about this project, I was hoping to see a demo/playground installation somewhere, but the README only has a screenshot available to feel the application. This reminds me of a cool feature of Play-with-Docker called, Try in PWD, which allows any application to be deployed by a single click and explored in an ephemeral environment as long as there is a Docker Compose file available somewhere publicly accessible. Some projects add a Try in PWD button in their README that include a compose file in the repository.

[brettkromkamp]

That does sound very interesting and makes a lot of sense. It would be great if Contextualise could make use of something like Try in PWD. Basically, what this all comes down to is that I just have to bite the bullet and learn Docker :)

[brettkromkamp]

@machawk1 @ibnesayeed I'll take a stab at getting support in place for multiple back-end persistence stores (i.e., databases) for TopicDB and by extension Contextualise. I'll keep you in the loop.

It would actually be quite cool if Contextualise could run on SQLite.

[brettkromkamp]

@ibnesayeed @machawk1 @nomatica @vladimirkosolapov Came across this application the other day (haven't tried it myself, yet), but it might be of interest: Roam.

[machawk1]

Thanks for the link, @brettkromkamp. I was not aware of Roam. I gave it a spin and it seems pretty intuitive -- I just wish it was self-hostable.

[nomatica]

@brettkromkamp I am familiar with Roam. Have played around with it. IMy understanding is that that contextualize will provide more automatic "semantic" like tagging. Is that correct?

[brettkromkamp]

@nomatica I apologize for the delay in replying. Well, Contextualise has features like associations that allow you to establish semantically very accurate relationships between topics. Tagging, for example, is built on top of associations. What I mean by that is when you tag a topic, what the application is doing "under the hood" is creating the appropriate association between the topic that is being tagged and the "tag topic" (a topic is created to represent the tag: this tag topic is what the topic that is being tagged is connected to). In addition, the association is appropriately configured: the type of association that is created when tagging is one "categorisation" and the respective roles between the topic that is being tagged and the tag topic is, "member" and "category", respectively (done in TopicDB by the TopicStore.set_tag-method).

Contextualise also has the concept of "scope" that allows the user to organise their documented knowledge (i.e., whatever body of knowledge is being managed) into different contexts. In practical terms that means that topics can be inter-connected differently depending on the active "scope" (or context). What's more, the occurrences (i.e., resources) attached to a topic are also scope-specific.

So, yes... Contextualise, which is built on top of the topic maps paradigm, has a powerful underlying semantic model.

[brettkromkamp]

@nomatica Truth be told I still have to implement the user interface for tagging. Nonetheless, it will be quick to implement as the topic maps engine already has support for tagging.

[komish]

@brettkromkamp I don't want to chalk it up to PostgreSQL just yet. Using the Docker configuration, I should be seeing something at 127.0.0.1:5000 or localhost:5000 but don't. I was hoping that @ibnesayeed would provide some feedback on getting it working with his extensive Docker familiarity, but just got the suggestion to prematurely refactor/orchestrate using Docker Compose. :\

Should I be able to see some interface even if PostgreSQL is not yet correctly set up?

As I get a spare moment, I will work on it further, but those sort of moments come in small bursts. I am intent on getting it working, as I anticipate the project to be personally

@brettkromkamp @machawk1 I'm not sure if you got this running, but In my testing this was due to the container needing to listen in all interfaces. I adjusted the final line in the Dockerfile to look like this.

CMD python -m flask run --host 0.0.0.0

Then I saw various issues in the application reading the settings.ini (missing keys). I realized that the README references a sample settings.ini but it's missing DATABASE.Host and DATABASE.Port among others. I just found it easier to copy the sample-settings.ini file and just change the database values to user the docker user.

Now I'm getting DB connection issues because Postgres isn't running, so I've stopped there for now.

Hope this helps.

[brettkromkamp]

@komish Thank you very much for contributing to Contextualise. I have updated both the README and the Dockerfile in accordance to your comments. Unfortunately, my knowledge of Docker is limited so I cannot take this specific issue much further. At some point @ibnesayeed suggested a "micro service" based Docker setup with each component (e.g., the database, web server, application) in their own container instead of a monolithic Docker image (which is the current implementation).

[brettkromkamp]

@komish At some point in the not too distant future I intend to make Contextualise publicly available so people can check it out to see if it covers their needs before having to set it up locally.

[machawk1]

Quick update, I pulled fresh from master (35fe4735df35a756d21a2d993cc1d622027d1bba) and ran through the three docker steps in the README. After the third step, I opened http://127.0.0.1:5000/ in my browser, was shown the trace UI (attached) and the application reported the following on the command line.

Command line output (click to expand)

 * Serving Flask app "contextualise" (lazy loading)
 * Environment: development
 * Debug mode: on
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 193-987-549
172.17.0.1 - - [27/Jan/2020 20:14:51] "GET / HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/var/lib/postgresql/.local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/var/lib/postgresql/.local/lib/python3.8/site-packages/flask/cli.py", line 83, in find_best_app
    app = call_factory(script_info, app_factory)
  File "/var/lib/postgresql/.local/lib/python3.8/site-packages/flask/cli.py", line 119, in call_factory
    return app_factory()
  File "/contextualise/contextualise/__init__.py", line 161, in create_app
    from contextualise import api
  File "/contextualise/api.py", line 19, in <module>
    KNOWLEDGE_GRAPH_API_KEY = config["API"]["KnowledgeGraph"]
  File "/usr/local/lib/python3.8/configparser.py", line 960, in __getitem__
    raise KeyError(key)
KeyError: 'API'
172.17.0.1 - - [27/Jan/2020 20:14:52] "GET /?__debugger__=yes&cmd=resource&f=style.css HTTP/1.1" 200 -
172.17.0.1 - - [27/Jan/2020 20:14:52] "GET /?__debugger__=yes&cmd=resource&f=jquery.js HTTP/1.1" 200 -
172.17.0.1 - - [27/Jan/2020 20:14:52] "GET /?__debugger__=yes&cmd=resource&f=debugger.js HTTP/1.1" 200 -
172.17.0.1 - - [27/Jan/2020 20:14:53] "GET /?__debugger__=yes&cmd=resource&f=console.png HTTP/1.1" 200 -
172.17.0.1 - - [27/Jan/2020 20:14:53] "GET /?__debugger__=yes&cmd=resource&f=ubuntu.ttf HTTP/1.1" 200 -

macOS 10.14.6, Docker version 19.03.5, build 633a0ea

[brettkromkamp]

@machawk1 I’ll fix (related to this: https://github.com/brettkromkamp/contextualise/issues/21).

[komish]

@machawk1 in the meantime while #21 is getting resolved, just add the these lines to your settings.ini.

https://github.com/brettkromkamp/contextualise/blob/master/settings-sample.ini#L14-L15

[brettkromkamp]

Some very relevant feedback with regards to Contextualise is available on HackerNews, here: https://news.ycombinator.com/item?id=22282583

[machawk1]

In adding the settings.ini with those from the README (in the Docker section), I am shown the trace like https://github.com/brettkromkamp/contextualise/issues/22#issuecomment-539150015 :

[DATABASE]
Username = postgres
Password = postgres
Database = postgres
Host = db
Port = 5432

[EMAIL]
Username = changeme
Password = changeme
Server = mail.changeme.com
Sender = Change Me

Should other values be put in-place here to make the db accessible?

[brettkromkamp]

@machawk1 No additional values, as far as I can see, are needed for the database. Is the database up? In all of these Docker-related files where is the database actually started? The docker-compose.yml file?

[brettkromkamp]

@machawk1 Just out of curiosity... have you actually ever got Contextualise up and running? It seems like the Docker setup is still not 100% right, so I presume you have never had the Contextualise Docker container working, or? What about the traditional (non-Docker) approach to installing the application? Have you tried that? If yes, did you get it to work?

[machawk1]

@brettkromkamp I thought I had but not so sure now. Any setup I did might have been on a previous system (when this GH issue started). With that in-mind, I am open to trying to get it installed on my current system sans Docker.

Going at the "Install the Development Version" section in the README naively. Is the expectation that the users will build psycopg2 from source? If not, pip3 install psycopg2 then installing contextualise via pip3 still reports errors and fails. This might be due to the non-inclusion of the libpq headers.

I think that section of the README could use some unpacking for further testing and development. Advice/suggestions you have on this are welcomed and something I can test from the (not too far from)naive perspective.

[brettkromkamp]

@machawk1 Tomorrow, I will install Contextualise on a completely fresh (Ubuntu) VM and will document every step I make. I’m not aware of Contextualise being anything else than a very standard Flask/PostgreSQL application but seeing how much people are seemingly struggling to install/configure the application makes me wonder. I’ll get back to you on this...