I am working on an (almost) one-step dev setup powered by Docker Compose.

Run docker compose up -d in the repo root
Run ./docker_compose_setup.sh (and wait a long time) to do the initial setup

It is almost entirely working, but I am having trouble with:

~Solr search doesn't work. The config and indexing seem to work, but searches always fail with zero results. I note two seemingly related issues:~

~During indexing, there are numerous errors like~

Failed to add documents to Solr: Solr responded with an error (HTTP 400): [Reason: ERROR: [doc=port.port.29001] cannot set an index-time boost, unindexed or norms are omitted for field name: xdaliclock]
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/haystack/backends/solr_backend.py", line 100, in update
    self.conn.add(docs, commit=commit, boost=index.get_field_weights())
  File "/usr/local/lib/python3.6/dist-packages/pysolr.py", line 1050, in add
    solrapi=solrapi,
  File "/usr/local/lib/python3.6/dist-packages/pysolr.py", line 565, in _update
    "post", path, message, {"Content-type": "text/xml; charset=utf-8"}
  File "/usr/local/lib/python3.6/dist-packages/pysolr.py", line 463, in _send_request
    raise SolrError(error_message % (resp.status_code, solr_message))

~Nevertheless the data and indexes do appear to be created in the Docker volume.~

~If I expose the Solr web console and access the logging section, the log shows the following error repeatedly. The times seem to correspond to when I perform a search in the app.~

org.apache.solr.common.SolrException: can not sort on multivalued field: name_length
    at org.apache.solr.schema.SchemaField.checkSortability(SchemaField.java:164)
    at org.apache.solr.schema.TrieField.getSortField(TrieField.java:163)
    at org.apache.solr.schema.SchemaField.getSortField(SchemaField.java:150)
    at org.apache.solr.search.SortSpecParsing.parseSortSpecImpl(SortSpecParsing.java:202)
    at org.apache.solr.search.SortSpecParsing.parseSortSpec(SortSpecParsing.java:60)
    at org.apache.solr.search.QParser.getSortSpec(QParser.java:285)
    at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:188)
    at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:270)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
    at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:724)
    at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:530)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
    at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
    at org.eclipse.jetty.server.Server.handle(Server.java:534)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
    at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
    at java.lang.Thread.run(Thread.java:750)

Update: the Solr issues were both due to me configuring the wrong directory with build_solr_schema. After fixing that, I find that search does work correctly.

There doesn't seem to be any documentation about loading mpstats data, so I haven't managed to set that up yet.

May 04 '22 15:05 amake

Codecov Report

Merging #339 (514e018) into main (ed3d8ef) will not change coverage. The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #339   +/-   ##
=======================================
  Coverage   81.17%   81.17%           
=======================================
  Files         111      111           
  Lines        2762     2762           
  Branches      216      216           
=======================================
  Hits         2242     2242           
  Misses        474      474           
  Partials       46       46

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update ed3d8ef...514e018. Read the comment docs.

May 04 '22 15:05 codecov[bot]

Awesome!

mystats data gets filled when users submit their data (a weekly job submits data about installed ports from everyone who installed the mpstats port).

We probably need a script that will generate some fake entries for the sake of development & testing.

May 05 '22 12:05 mojca

Can you please also add some super basic documentation?

May 05 '22 13:05 mojca

I found some permissions-related issues and am working on a fix. I'll un-draft when ready.

May 05 '22 14:05 amake

Thanks, there is some sample data for mpstats here - we use it for performing tests. https://github.com/macports/macports-webapp/blob/main/app/tests/sample_data/submissions.json

But the data needs to modified, at least the port names.

May 05 '22 17:05 arjunsalyan

This is now ready. I had to adjust the data storage strategy in order to get things working seamlessly on first run. Previously I had accidentally hit upon a strategy that only works if you manually adjust things as you go.

Thanks, there is some sample data for mpstats here

Thanks. If people don't mind I'd like to skip mpstats for this PR.

May 06 '22 12:05 amake

Add Docker Compose-based setup

Codecov Report