scava icon indicating copy to clipboard operation
scava copied to clipboard

Published 20 hours ago • verified publisher icon crossminer

Register project: Documentation Readers

Open creat89 opened this issue 6 years ago • 29 comments

Hello,

I have finished the last elements for the documentation readers and they can be now added to the platform UI.

More specifically there are two (three depending on the capacities of the UI) types of documentation:

  • Git-Based (org.eclipse.scava.repository.model.documentation.gitbased). The requirements are the same as a Git project (ony URL), as it is based on it.
  • Systematic (org.eclipse.scava.repository.model.documentation.systematic). There are two types of documentation systematic:
    • No login necessary. In this aspect there are only two mandatory fields: URL and EXECUTIONFREQUENCY
    • Login necessary. For a documentation behind a login, we need more fields that are mandatory: URL, EXECUTIONFREQUENCY, LOGINURL, USERNAME, USERNAMEFIELDNAME, PASSWORD and PASSWORDFIELDNAME.

As you can see for the documentation systematic, if no login is necessary only two fields must be filled, but if a login is necessary, then all the fields are mandatory. So, I don't know if it will be easier to create two types of Systematic one with login and one without login. That's up to you and the capacities of the UI.

Apart from that there is one more limitation, the gitbased documentation cannot be used along with a git project, because internally they use the same model and the documentation extends the reader of the project.

creat89 avatar May 30 '19 17:05 creat89

@creat89, the commit 2cbbe67 should add a support to the documentation readers.

ambpro avatar Jun 18 '19 09:06 ambpro

Hello @ambpro, for Documentation Git-Based I get the following issue when I create a project:

oss-app_1         | Adding new project
oss-app_1         | Jun 18, 2019 1:46:44 PM org.restlet.engine.log.LogFilter afterHandle
oss-app_1         | INFO: 2019-06-18    13:46:44        172.28.0.10     -       172.28.0.8      8182    POST    /projects/create        -       201     1146    248     64      http://oss-app:8182     Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0     http://192.168.42.216:5601/
oss-app_1         | Error: null
oss-app_1         | null
oss-app_1         | Jun 18, 2019 1:46:44 PM org.restlet.resource.ServerResource doCatch
oss-app_1         | WARNING: Exception or error caught in server resource
oss-app_1         | Internal Server Error (500) - The server encountered an unexpected condition which prevented it from fulfilling the request
oss-app_1         |     at org.restlet.resource.ServerResource.doHandle(ServerResource.java:540)
oss-app_1         |     at org.restlet.resource.ServerResource.get(ServerResource.java:743)
oss-app_1         |     at org.restlet.resource.ServerResource.doHandle(ServerResource.java:618)
oss-app_1         |     at org.restlet.resource.ServerResource.doNegotiatedHandle(ServerResource.java:679)
oss-app_1         |     at org.restlet.resource.ServerResource.doConditionalHandle(ServerResource.java:357)
oss-app_1         |     at org.restlet.resource.ServerResource.handle(ServerResource.java:1044)
oss-app_1         |     at org.restlet.resource.Finder.handle(Finder.java:236)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Router.doHandle(Router.java:422)
oss-app_1         |     at org.restlet.routing.Router.handle(Router.java:639)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.engine.application.StatusFilter.doHandle(StatusFilter.java:140)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.engine.CompositeHelper.handle(CompositeHelper.java:202)
oss-app_1         |     at org.restlet.engine.application.ApplicationHelper.handle(ApplicationHelper.java:75)
oss-app_1         |     at org.restlet.Application.handle(Application.java:385)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Router.doHandle(Router.java:422)
oss-app_1         |     at org.restlet.routing.Router.handle(Router.java:639)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Router.doHandle(Router.java:422)
oss-app_1         |     at org.restlet.routing.Router.handle(Router.java:639)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.engine.application.StatusFilter.doHandle(StatusFilter.java:140)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.engine.CompositeHelper.handle(CompositeHelper.java:202)
oss-app_1         |     at org.restlet.Component.handle(Component.java:408)
oss-app_1         |     at org.restlet.Server.handle(Server.java:507)
oss-app_1         |     at org.restlet.engine.connector.ServerHelper.handle(ServerHelper.java:63)
oss-app_1         |     at org.restlet.engine.adapter.HttpServerHelper.handle(HttpServerHelper.java:143)
oss-app_1         |     at org.restlet.ext.jetty.JettyServerHelper$WrappedServer.handle(JettyServerHelper.java:273)
oss-app_1         |     at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:313)
oss-app_1         |     at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
oss-app_1         |     at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
oss-app_1         |     at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:626)
oss-app_1         |     at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:546)
oss-app_1         |     at java.lang.Thread.run(Thread.java:748)
oss-app_1         | Caused by: java.lang.NullPointerException
oss-app_1         |     at org.eclipse.scava.platform.client.api.Util.escape(Util.java:52)
oss-app_1         |     at org.eclipse.scava.platform.client.api.Util.generateErrorMessage(Util.java:45)
oss-app_1         |     at org.eclipse.scava.platform.client.api.Util.generateErrorMessageRepresentation(Util.java:33)
oss-app_1         |     at org.eclipse.scava.platform.client.api.ProjectListResource.doRepresent(ProjectListResource.java:92)
oss-app_1         |     at org.eclipse.scava.platform.client.api.AbstractApiResource.represent(AbstractApiResource.java:55)
oss-app_1         |     at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
oss-app_1         |     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
oss-app_1         |     at java.lang.reflect.Method.invoke(Method.java:498)
oss-app_1         |     at org.restlet.resource.ServerResource.doHandle(ServerResource.java:524)
oss-app_1         |     ... 47 more

After, if I want to add or see the projects page I get the following issue:

oss-app_1         | Error: null
oss-app_1         | null
oss-app_1         | Jun 18, 2019 1:49:20 PM org.restlet.resource.ServerResource doCatch
oss-app_1         | WARNING: Exception or error caught in server resource
oss-app_1         | Internal Server Error (500) - The server encountered an unexpected condition which prevented it from fulfilling the request
oss-app_1         |     at org.restlet.resource.ServerResource.doHandle(ServerResource.java:540)
oss-app_1         |     at org.restlet.resource.ServerResource.get(ServerResource.java:743)
oss-app_1         |     at org.restlet.resource.ServerResource.doHandle(ServerResource.java:618)
oss-app_1         |     at org.restlet.resource.ServerResource.doNegotiatedHandle(ServerResource.java:679)
oss-app_1         |     at org.restlet.resource.ServerResource.doConditionalHandle(ServerResource.java:357)
oss-app_1         |     at org.restlet.resource.ServerResource.handle(ServerResource.java:1044)
oss-app_1         |     at org.restlet.resource.Finder.handle(Finder.java:236)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Router.doHandle(Router.java:422)
oss-app_1         |     at org.restlet.routing.Router.handle(Router.java:639)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.engine.application.StatusFilter.doHandle(StatusFilter.java:140)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.engine.CompositeHelper.handle(CompositeHelper.java:202)
oss-app_1         |     at org.restlet.engine.application.ApplicationHelper.handle(ApplicationHelper.java:75)
oss-app_1         |     at org.restlet.Application.handle(Application.java:385)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Router.doHandle(Router.java:422)
oss-app_1         |     at org.restlet.routing.Router.handle(Router.java:639)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Router.doHandle(Router.java:422)
oss-app_1         |     at org.restlet.routing.Router.handle(Router.java:639)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.engine.application.StatusFilter.doHandle(StatusFilter.java:140)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.engine.CompositeHelper.handle(CompositeHelper.java:202)
oss-app_1         |     at org.restlet.Component.handle(Component.java:408)
oss-app_1         |     at org.restlet.Server.handle(Server.java:507)
oss-app_1         |     at org.restlet.engine.connector.ServerHelper.handle(ServerHelper.java:63)
oss-app_1         |     at org.restlet.engine.adapter.HttpServerHelper.handle(HttpServerHelper.java:143)
oss-app_1         |     at org.restlet.ext.jetty.JettyServerHelper$WrappedServer.handle(JettyServerHelper.java:273)
oss-app_1         |     at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:313)
oss-app_1         |     at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
oss-app_1         |     at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
oss-app_1         |     at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:626)
oss-app_1         |     at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:546)
oss-app_1         |     at java.lang.Thread.run(Thread.java:748)
oss-app_1         | Caused by: java.lang.NullPointerException
oss-app_1         |     at org.eclipse.scava.platform.client.api.Util.escape(Util.java:52)
oss-app_1         |     at org.eclipse.scava.platform.client.api.Util.generateErrorMessage(Util.java:45)
oss-app_1         |     at org.eclipse.scava.platform.client.api.Util.generateErrorMessageRepresentation(Util.java:33)
oss-app_1         |     at org.eclipse.scava.platform.client.api.ProjectListResource.doRepresent(ProjectListResource.java:92)
oss-app_1         |     at org.eclipse.scava.platform.client.api.AbstractApiResource.represent(AbstractApiResource.java:55)
oss-app_1         |     at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
oss-app_1         |     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
oss-app_1         |     at java.lang.reflect.Method.invoke(Method.java:498)
oss-app_1         |     at org.restlet.resource.ServerResource.doHandle(ServerResource.java:524)
oss-app_1         |     ... 47 more
oss-app_1         |

EDIT: Interesting the project was added correctly to MongoDB

creat89 avatar Jun 18 '19 13:06 creat89

By looking on the stack trace, I guess that the above issue is already fixed by the commit https://github.com/crossminer/scava/commit/7e232944198e100368ab0de9037337fcd757f8db. let me know if the current update of dev branch works fine for you?

ambpro avatar Jun 18 '19 13:06 ambpro

Ok, I'll build again the platform once the build will be finished

creat89 avatar Jun 18 '19 14:06 creat89

Hello,

Checking the UI I have seen these issues for the documentation systematic with login:

image

where Password Field Name is considered as a password rather than text.

And, Martin and I, we're seeing that the data is not being passed to Mongo DB, either with or without login.

creat89 avatar Sep 16 '19 13:09 creat89

@creat89 I pushed a fix ( commit 7d7fa48) to handle the above issue. Could you check it again in your side?

ambpro avatar Sep 23 '19 09:09 ambpro

Hello @ambpro,

When I create a Documentation Systematic I get the following error:

oss-app_1         | Internal Server Error (500) - The server encountered an unexpected condition which prevented it from fulfilling the request
...
oss-app_1         | Caused by: java.lang.NullPointerException
oss-app_1         |     at org.eclipse.scava.platform.client.api.ProjectCreationResource.createProject(ProjectCreationResource.java:195)
oss-app_1         |     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
oss-app_1         |     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
oss-app_1         |     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
oss-app_1         |     at java.lang.reflect.Method.invoke(Method.java:498)
oss-app_1         |     at org.restlet.resource.ServerResource.doHandle(ServerResource.java:521)
oss-app_1         |     ... 45 more

Thus, nothing is stored in Mongo.

And still, I get the password dots in the password field, despite you have changed the file (I have checked and built the platform some minutes ago and used the incognito mode). So, I don't know what's going on.

creat89 avatar Sep 24 '19 16:09 creat89

@creat89

When I create a Documentation Systematic I get the following error: Thus, nothing is stored in Mongo.

The commit (7d28be0) should fix that. Now, I am able to register a project with a Documentation Systematic connector.

And still, I get the password dots in the password field, despite you have changed the file (I have checked and built the platform some minutes ago and used the incognito mode).

I am pretty sure that it's a browser cache problem! can you try Ctrl + F5 on your browser and let me know if it works?

ambpro avatar Sep 25 '19 08:09 ambpro

Hello @ambpro,

The connector works well, but the data is missing for the Systematic Documentation without login. I'm getting this: image Instead of something like this: image

And the Password field is still showing like password despite using Ctrl+F5 and incognito mode.

creat89 avatar Oct 02 '19 10:10 creat89

Hello @creat89

The connector works well, but the data is missing for the Systematic Documentation without login. I'm getting this

It's a bit strange because with the current metric-platform build, I can add a new documentation systematic through the Register project . with no-login measure-project-wiki

with login image

ambpro avatar Oct 18 '19 14:10 ambpro

@ambpro with the last build:

  • No login: If execution frequency field is filled, the data is stored in Mongo. However, we can still click on save without indicating the frequency. If this is the case the entry is not stored.
  • Login: If execution frequency field is filled, the data is stored in Mongo. We can still click on save without indicating the frequency. However, once we click on save, the button becomes gray and it is impossible to make it active until we delete the project. Filling the execution frequency field does not change anything. The password field name is still showing dots.

creat89 avatar Oct 18 '19 15:10 creat89

@creat89 , does the previous commit (c57a4a0) solve the problem ?

ambpro avatar Oct 21 '19 16:10 ambpro

Hello @ambpro,

In some cases yes, in some cases no. These are my observations: No login:

  • Currently, the project is never saved correctly, whether we fill the frequency field or not. In fact it is storing only the data shown in https://github.com/crossminer/scava/issues/252#issuecomment-537427108.

Login:

  • The project is saved correctly in MongoDB even if the frequency field is empty.
  • The password field name is still showing the dots.
  • The project can be saved without the password and username fields-related completed.

creat89 avatar Oct 22 '19 10:10 creat89

@ambpro, I don't know if this information will make easier to fix the issue, but if the username and password fields are empty, the reader consider that automatically as without login. This could make easier to fix the issue, as currently the creation of the project with login stores the data correctly even if these fields are empty.

creat89 avatar Oct 22 '19 10:10 creat89

@creat89

Systematic (org.eclipse.scava.repository.model.documentation.systematic). There are two types of documentation systematic:

  • No login necessary. In this aspect there are only two mandatory fields: URL and EXECUTIONFREQUENCY
  • Login necessary. For a documentation behind a login, we need more fields that are mandatory: URL, EXECUTIONFREQUENCY, LOGINURL, USERNAME, USERNAMEFIELDNAME, PASSWORD and PASSWORDFIELDNAME.

No login: If execution frequency field is filled, the data is stored in Mongo. However, we can still click on save without indicating the frequency. If this is the case the entry is not stored.

Login: If execution frequency field is filled, the data is stored in Mongo. We can still click on save without indicating the frequency.

I don't know if this information will make easier to fix the issue, but if the username and password fields are empty, the reader consider that automatically as without login.

AFAIU and based on the documentation reader spec, all the fields are mandatory depend on the reader type option (no login- with login). Is it still the case? Otherwise, could you clarify this point with more details?

ambpro avatar Oct 22 '19 12:10 ambpro

It's correct what you're saying, if you set a username and password you need to set the login URL and the username field and the password field. Otherwise the platform will throw an error indicating that you're missing some fields. However, what I'm trying to say is that you can create an object with the username and password fields empty and that would be equivalent to a no-login. Is that clear?

creat89 avatar Oct 22 '19 13:10 creat89

However, what I'm trying to say is that you can create an object with the username and password fields empty and that would be equivalent to a no-login. Is that clear?

Ok. I'll adjust the reader's saving process and I will let you know asap!

ambpro avatar Oct 22 '19 14:10 ambpro

@creat89 I pushed some fixes related to what we discussed earlier. Can you re-test it again?

ambpro avatar Nov 08 '19 16:11 ambpro

Hello @ambpro,

  • No login: No matter whether the execution frequency is set or not, the saved object in MongoDB is still wrong. Like in https://github.com/crossminer/scava/issues/252#issuecomment-537427108
  • Login: Everything is stored correctly whether the user specifies an execution frequency or not. So this is ok, except for the password field name, which it is still showing as password.

image

I don't know if I'm seeing truly the last version. I'm using the latest version of the 11th Nov 2019 and using Firefox on a private windows to prevent the cache. Here is a screen:

image

creat89 avatar Nov 12 '19 11:11 creat89

Hello @creat89 Related to cache issue, there you go a screenshot of the documentation-systematic template form from a fresh docker instance deployed this morning! documentation-systematic

I see that the screenshot you have attached is not 100% up-to-date as the mine. Do you build the docker images using: docker-compose build --no-cache as mentionned in here ?

ambpro avatar Nov 12 '19 11:11 ambpro

I'll deploy again the newest version, but my question would be, how can we ensure that the cache is removed? I'm using a private window, which shouldn't store the cache. I have just tried Internet Explorer and it is the same. I guess it is something related on the docker side...

creat89 avatar Nov 12 '19 11:11 creat89

I'll deploy again the newest version, but my question would be, how can we ensure that the cache is removed? I'm using a private window, which shouldn't store the cache. I have just tried Internet Explorer and it is the same. I guess it is something related on the docker side...

Yes! It seems that it's related to a docker images caching and it's not related to the browser you 're using!

The command docker-compose build --no-cache should deal with that (I hope that :) )

ambpro avatar Nov 12 '19 11:11 ambpro

I'm sure you'll not believe but still nothing :/

The weird thing is that I always use docker-compose build --no-cache and before that I have even done a system prune.

creat89 avatar Nov 12 '19 12:11 creat89

@aabherve could you have a check on your side about this issue ? Thanks

ambpro avatar Nov 12 '19 13:11 ambpro

Hi @creat89 , you are using which branch of scava-deployment? If you are using the dev brach you should view the download and compilation of the web-admin code when you do the build.

Try running docker-compose build --no-cache admin-webapp. Do you see the building of node.js code?

MarcioMateus avatar Nov 12 '19 13:11 MarcioMateus

I don't think. Because it seems that ther code for the web-admin is the old one. I'll update it and see what happens again. In all the cases, I only see this: image

creat89 avatar Nov 12 '19 13:11 creat89

Yes, everything seems to work fine and as expected. The missing lines in the docker were the problem. I'll close the issue and thanks @ambpro and @MarcioMateus

creat89 avatar Nov 12 '19 13:11 creat89

Hello,

I'm having this issue and I don't know what's the problem or what field is wrong:

image

creat89 avatar Nov 25 '19 14:11 creat89

So, sometimes when you click wrongly the login necessary, some of the fields are expected to be filled, but those are hidden.

creat89 avatar Nov 25 '19 14:11 creat89