Scholia unavailability

[fnielsen]

Describe the bug the Scholia Webapp running on Toolforge is unavailability much of the time and crashes.

To Reproduce Steps to reproduce the behavior:

1. Go to https://scholia.toolforge.org

Expected behavior Web pages should be served

** What to do** This may be related to a lot of automated request on the Scholia webapp with automated tools (AI?). There is a lot of request to static assets

• Use the static assets on the server provided be the Wikimedia
• Move to Async. See #2725
• Cut of the processing that takes too long time.
• Login - worst case

[fnielsen]

https://wikitech.wikimedia.org/wiki/Help%3AToolforge/Web

ln -s python/scholia/scholia/app/static/ static

[fnielsen]

app.py in /www/python/src/

from scholia.app import create_app
from flask import url_for as _flask_url_for
from urllib.parse import urlencode

# The Toolforge static URL base
TOOLFORGE_STATIC_URL = "https://tools-static.wmflabs.org/scholia"

app = create_app()
app.config['APPLICATION_ROOT'] = '/scholia/'

def _cdn_url_for(endpoint, **values):
    if endpoint == "static":
        filename = values.pop("filename", "")
        qs = f"?{urlencode(values)}" if values else ""
        # Files should live in ~/www/static and are served at:
        # https://tools-static.wmflabs.org/<tool>/<filename>
        return f"{TOOLFORGE_STATIC_URL}/{filename.lstrip('/')}{qs}"
    # Everything else uses normal url_for
    return _flask_url_for(endpoint, **values)

# Make templates use our function as url_for
app.add_template_global(_cdn_url_for, name="url_for")

if __name__ == '__main__':
    app.run()

[WolfgangFahl]

add an anubis denial of service protection frontend? https://github.com/TecharoHQ/anubis

[fnielsen]

add an anubis denial of service protection frontend? https://github.com/TecharoHQ/anubis

The major problem is that we are behind Wikimedia proxy. There is very little information. No user-agent, no IP number (beyond the proxied IP number).

[fnielsen]

I have moved the static files. I see there are still some static requests that are hitting the Flask app, so there is still a bit of work to do.

[fnielsen]

Entry on the Cloud Wikimedia mailing list https://lists.wikimedia.org/hyperkitty/list/[email protected]/thread/CBNZGTSMXREHINSAJYHZ2FMTHORNAF5T/

[WolfgangFahl]

https://scholia.wikidata.dbis.rwth-aachen.de/ is now available as a mirror. https://scholia.wikidata.dbis.rwth-aachen.de/backend shows the backend

What other mirrors are out there?

[WolfgangFahl]

we might want to fix https://github.com/WDscholia/scholia/issues/2746 especially the --port option. Then multiple servers can run on a single machine e.g. https://scholia-qlever.wikidata.dbis.rwth-aachen.de/ with https://scholia-qlever.wikidata.dbis.rwth-aachen.de/backend