INGInious icon indicating copy to clipboard operation
INGInious copied to clipboard

Published 20 hours ago • verified publisher icon UCL-INGI

Installation guide: recommended installation for 100s simultaneous submissions

Open erelsgl opened this issue 4 years ago • 17 comments

In the installation guide, it is not entirely clear what configuration is recommended when it is expected that 100-200 students submit their solution in the same time.

From my experience, the simple Python web server cannot handle so many submissions. Indeed, the documentation says "In production environments, you can use lighttpd in replacement of the built-in Python server", but not everyone understands what "production environment" means.

Also, in the beginning it is written "You may want to enable the LDAP/SAML2 plugin or use FCGI/UWSGI instead of the web.py default webserver" - not everyone understands that "FCGI" the same as "fast CGI" and that "UWSGI" the same as "wsgi".

I suggest to clarify the documentation as follows.

If you intend to use INGInious with many simultaneous submissions, you should use a professional web-server, such as Lighttpd or Apache. To use Lighttpd, install the fast CGI module:

  $ pip3 install --upgrade git+https://github.com/UCL-INGI/[email protected]#egg=INGInious[cgi]

To use Apache, install the UWCGI module:

  $ pip3 install --upgrade git+https://github.com/UCL-INGI/[email protected]#egg=INGInious[uwgsi]

Then, proceed with configuring Lighttpd or Apache as explained below

erelsgl avatar Feb 16 '21 06:02 erelsgl

Thank you for these suggestions. This part has remained untouched for a while now. I think it's also worth adding some hints on how to scale the components correctly depending on the expected load.

anthonygego avatar Feb 16 '21 08:02 anthonygego

Indeed! Our locally-developed auto-grader worked perfectly before the start of term, but in the first submission to the public server, 100 students tried to submit simultaneously and it crashed miserably... We had to move to Apache at once.

erelsgl avatar Feb 16 '21 19:02 erelsgl

QUESTION: what is "uwgsi" in the installation guide? Should it be "uwsgi"?

erelsgl avatar Feb 17 '21 20:02 erelsgl

ANOTHER QUESTION: After I install and configure Apache2 as explained in that page, how can I check my installation? I tried to open http://<my-ip-address>/static but got an error "Unable to connect".

erelsgl avatar Feb 18 '21 05:02 erelsgl

  • Yes, it is uwsgi You'll need it if you want to perform an nginx deployment. However, if your Linux distribution provides uwsgi, it's better and simpler to install it with their own packages. You do not need it with Apache, uwsgi is a specific service on top of WSGI.
  • Apache is quite verbose, I think you can find all the debug information into the logs (especially the error log, which would be located somewhere in /var/log/{httpd|apache2}). You can also ensure the LogLevel is high enough in the Apache configuration file.

anthonygego avatar Feb 18 '21 07:02 anthonygego

To avoid clashes with other listening processes, I moved the two modules to ports 8081 and 8082 respectively, and reloaded Apache2. My configuration looks like this:

<VirtualHost *:8081>
       LoadModule wsgi_module /usr/local/lib/python3.6/dist-packages/mod_wsgi/server/mod_wsgi-py36.cpython-36m-x86_64-linux-gnu.so
       WSGIScriptAlias / "/usr/local/bin/inginious-webapp"
       WSGIScriptReloading On

       Alias /static /usr/local/lib/python3.6/dist-packages/inginious/frontend/static

        <Directory "/usr/local/bin">
            <Files "inginious-webapp">
                Require all granted
            </Files>
        </Directory>

        <DirectoryMatch "/usr/local/lib/python3.6/dist-packages/inginious/frontend/static">
            Require all granted
        </DirectoryMatch>

       ServerAdmin [email protected]
       DocumentRoot /var/www/inginious
</VirtualHost>

<VirtualHost *:8082>
       LoadModule wsgi_module /usr/local/lib/python3.6/dist-packages/mod_wsgi/server/mod_wsgi-py36.cpython-36m-x86_64-linux-gnu.so
        WSGIScriptAlias / "/usr/local/bin/inginious-webdav"
        WSGIScriptReloading On

        <Directory "/usr/local/bin">
            <Files "inginious-webdav">
                Require all granted
            </Files>
        </Directory>
</VirtualHost>

Now, apache2 -M shows, among others,

wsgi_module (shared)

and apache2 -S shows, among others,

*:8081                 104.248.40.179 (/etc/apache2/sites-enabled/inginious.conf:1)
*:8082                 104.248.40.179 (/etc/apache2/sites-enabled/inginious.conf:22)

which shows that Apache2 does read the wsgi configuration from the files. However, I still cannot connect to http://localhost:8081 or http://localhost:8082. What is the problem?

erelsgl avatar Feb 18 '21 15:02 erelsgl

These config files look good at the first sight. Have a look to the error logs to see if the wsgi module or apache prints anything bad.

anthonygego avatar Feb 18 '21 16:02 anthonygego

Here is the file error.log just after reloading Apache2:

[Thu Feb 18 18:57:46.994117 2021] [socache_shmcb:info] [pid 12632] AH00830: Shared memory socache initialised
[Thu Feb 18 18:57:46.994139 2021] [ssl:info] [pid 12632] AH01887: Init: Initializing (virtual) servers for SSL
[Thu Feb 18 18:57:46.994146 2021] [ssl:info] [pid 12632] AH01876: mod_ssl/2.4.29 compiled against Server: Apache/2.4.29, Library: OpenSSL/1.1.1
[Thu Feb 18 18:57:46.994214 2021] [mpm_prefork:notice] [pid 12632] AH00163: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 mod_wsgi/4.5.17 Python/3.6 configured -- resuming normal operations
[Thu Feb 18 18:57:46.994219 2021] [mpm_prefork:info] [pid 12632] AH00164: Server built: 2020-08-12T21:33:25
[Thu Feb 18 18:57:46.994222 2021] [core:notice] [pid 12632] AH00094: Command line: '/usr/sbin/apache2'
[Thu Feb 18 18:57:47.997715 2021] [wsgi:info] [pid 16827] mod_wsgi (pid=16827): Initializing Python.
[Thu Feb 18 18:57:48.000974 2021] [wsgi:info] [pid 16831] mod_wsgi (pid=16831): Initializing Python.
[Thu Feb 18 18:57:48.004725 2021] [wsgi:info] [pid 16830] mod_wsgi (pid=16830): Initializing Python.
[Thu Feb 18 18:57:48.006544 2021] [wsgi:info] [pid 16829] mod_wsgi (pid=16829): Initializing Python.
[Thu Feb 18 18:57:48.011627 2021] [wsgi:info] [pid 16828] mod_wsgi (pid=16828): Initializing Python.
[Thu Feb 18 18:57:48.063143 2021] [wsgi:info] [pid 16831] mod_wsgi (pid=16831): Attach interpreter ''.
[Thu Feb 18 18:57:48.067184 2021] [wsgi:info] [pid 16830] mod_wsgi (pid=16830): Attach interpreter ''.
[Thu Feb 18 18:57:48.072145 2021] [wsgi:info] [pid 16831] mod_wsgi (pid=16831): Imported 'mod_wsgi'.
[Thu Feb 18 18:57:48.072434 2021] [wsgi:info] [pid 16827] mod_wsgi (pid=16827): Attach interpreter ''.
[Thu Feb 18 18:57:48.073417 2021] [wsgi:info] [pid 16830] mod_wsgi (pid=16830): Imported 'mod_wsgi'.
[Thu Feb 18 18:57:48.080393 2021] [wsgi:info] [pid 16829] mod_wsgi (pid=16829): Attach interpreter ''.
[Thu Feb 18 18:57:48.083918 2021] [wsgi:info] [pid 16829] mod_wsgi (pid=16829): Imported 'mod_wsgi'.
[Thu Feb 18 18:57:48.084460 2021] [wsgi:info] [pid 16827] mod_wsgi (pid=16827): Imported 'mod_wsgi'.
[Thu Feb 18 18:57:48.088675 2021] [wsgi:info] [pid 16828] mod_wsgi (pid=16828): Attach interpreter ''.
[Thu Feb 18 18:57:48.091951 2021] [wsgi:info] [pid 16828] mod_wsgi (pid=16828): Imported 'mod_wsgi'.

It contains only info lines.

erelsgl avatar Feb 19 '21 06:02 erelsgl

I'm almost sure some INGInious logs should be printed there too. Maybe the inginious-webapp has actually not returned, which would explain why you get unable to connect error messages and why there is no log. This is possible if it tries to run the integrated webserver.

Open your /usr/bin/inginious-webapp file and comment the blocks:

  • From line https://github.com/UCL-INGI/INGInious/blob/v0.6/inginious-webapp#L32 to line https://github.com/UCL-INGI/INGInious/blob/v0.6/inginious-webapp#L43
  • From line https://github.com/UCL-INGI/INGInious/blob/v0.6/inginious-webapp#L66

However, that shouldn't prevent you from accessing the static files, that are not served through WSGI, and Apache is supposed to log that, even if there are not found. You may want to have a look to the access_log to ensure this. What kind of HTTP error code are you displayed when you try to connect ?

anthonygego avatar Feb 19 '21 07:02 anthonygego

When I do curl http://localhost:8081 I get curl: (7) Failed to connect to localhost port 8081: Connection refused

similarly when I try port 8082. Nothing appears in access.log.

I thought it might be a firewall problem, but ufw status shows Status: inactive.

erelsgl avatar Feb 19 '21 13:02 erelsgl

You can run netstat -pnlt to see if apache actually binds the correct IP and ports (for listening on every interface it should display 0.0.0.0 or :: for IPv6). If it only binds the public IP, you won't be able to connect localhost.

anthonygego avatar Feb 19 '21 14:02 anthonygego

Indeed, it does not listen on these ports: netstat -pnlt|grep apache

tcp6       0      0 :::443                  :::*                    LISTEN      10382/apache2
tcp6       0      0 :::80                   :::*                    LISTEN      10382/apache2

But it does show the virtual hosts when I do apache2 -S:

VirtualHost configuration:
*:443                  bad-kan.com (/etc/apache2/sites-enabled/badkan2.com-le-ssl.conf:2)
*:80                   104.248.40.179 (/etc/apache2/sites-enabled/badkan2.com.conf:1)
*:8081                 104.248.40.179 (/etc/apache2/sites-enabled/inginious.conf:1)
*:8082                 104.248.40.179 (/etc/apache2/sites-enabled/inginious.conf:22)

erelsgl avatar Feb 19 '21 14:02 erelsgl

At least some progress :-) You may need to add some Listen directives for those ports in your configuration files. I recommend you to refer to the Apache documentation, I don't know if this would be enough

anthonygego avatar Feb 19 '21 14:02 anthonygego

Hi @erelsgl, How did you solve your issue for apache2 not listening on port 8080. I have the same issue. My users are unable to access webdav via URL generated by inginious. This is the configuraton of apache

apache2 -S
[Tue Jun 15 08:30:45.671752 2021] [so:warn] [pid 3368:tid 140203586923456] AH01574: module wsgi_module is already loaded, skipping
[Tue Jun 15 08:30:45.672188 2021] [so:warn] [pid 3368:tid 140203586923456] AH01574: module wsgi_module is already loaded, skipping
AH00548: NameVirtualHost has no effect and will be removed in the next release /etc/apache2/ports.conf:5
VirtualHost configuration:
*:443                  inginious.uac.bj (/etc/apache2/sites-enabled/inginious-le-ssl.conf:2)
*:80                   is a NameVirtualHost
         default server XXX.YYY.bj (/etc/apache2/sites-enabled/000-default.conf:1)
         port 80 namevhost XXX.YYY.bj (/etc/apache2/sites-enabled/000-default.conf:1)
         port 80 namevhost XXX.YYY.bj (/etc/apache2/sites-enabled/inginious.conf:1)
*:8080                XXX.YYY.bj (/etc/apache2/sites-enabled/inginious.conf:26)
ServerRoot: "/etc/apache2"

Also the result of netstat

sudo netstat -pnlt | grep apache
tcp6       0      0 :::80                   :::*                    LISTEN      344/apache2         
tcp6       0      0 :::443                  :::*                    LISTEN      344/apache2

Thanks. Emery.

akemery avatar Jun 15 '21 10:06 akemery

@akemery this is what worked for me: https://docs.inginious.org/en/v0.7/admin_doc/install_doc/installation.html#using-apache-on-ubuntu-18-04
But it was for the previous version. Currently I am working on other things..

erelsgl avatar Jun 15 '21 12:06 erelsgl

@erelsgl , I followed the guidelines in https://docs.inginious.org/en/v0.7/admin_doc/install_doc/installation.html#using-apache-on-ubuntu-18-04, I am able to connect to webdav using the generated password and my login.

image

But the connection failed. Displaying the server's errors logs, inginious tell that there is error in configuration.yaml at line 1 and column 1. I am using inginious version v.0.8.dev1+g0c9a010e. I don't know if the error is related to the version I am using.

Maybe @anthonygego or @Drumor are more able to help me on that.

This my configuration.yaml

allow_registration: true
allow_deletion: false
backend: local
backup_directory: /var/www/html/inginious_bkp
local-config:
    debug_host: xxxx.yy.bj
    tmp_dir: /var/www/html/agent_tmp
mongo_opt:
    database: INGInious
    host: localhost
plugins: []
superadmins:
- eka
tasks_directory: /var/www/html/inginious_courses
use_minified_js: true
webterm: https://xxx.yyyy.bj:4449
webdav_host: http://xxxx.yyy.bj:4449
log_level: INFO
session_parameters:
    timeout: 86400  # 24 * 60 * 60, # 24 hours in seconds
    ignore_change_ip: False # change this to True if you want user to keep their session if they change their IP
    secure: False # change this to True if you only use https
    secret_key: "zzzzzz70ee0319b54eyyyyyyyyy3f2c95442d1147335yyyyy"

This is the error logs parts.

[Tue Jun 15 17:16:48.848254 2021] [wsgi:error] [pid 15745:tid 140431740368640] [client 41.85.181.159:57044] mod_wsgi (pid=15745): Exception occurred processing WSGI script '/usr/local/bin/inginious-webdav'.
[Tue Jun 15 17:16:48.850835 2021] [wsgi:error] [pid 15745:tid 140431740368640] [client 41.85.181.159:57044] Traceback (most recent call last):
[Tue Jun 15 17:16:48.850926 2021] [wsgi:error] [pid 15745:tid 140431740368640] [client 41.85.181.159:57044]   File "/usr/local/bin/inginious-webdav", line 53, in <module>
[Tue Jun 15 17:16:48.850936 2021] [wsgi:error] [pid 15745:tid 140431740368640] [client 41.85.181.159:57044]     config = load_json_or_yaml(configfile)
[Tue Jun 15 17:16:48.850950 2021] [wsgi:error] [pid 15745:tid 140431740368640] [client 41.85.181.159:57044]   File "/usr/local/lib/python3.6/dist-packages/inginious/common/base.py", line 33, in load_json_or_yaml
[Tue Jun 15 17:16:48.850958 2021] [wsgi:error] [pid 15745:tid 140431740368640] [client 41.85.181.159:57044]     return inginious.common.custom_yaml.load(f)
[Tue Jun 15 17:16:48.850971 2021] [wsgi:error] [pid 15745:tid 140431740368640] [client 41.85.181.159:57044]   File "/usr/local/lib/python3.6/dist-packages/inginious/common/custom_yaml.py", line 37, in load
[Tue Jun 15 17:16:48.850978 2021] [wsgi:error] [pid 15745:tid 140431740368640] [client 41.85.181.159:57044]     return original_yaml.load(stream, OrderedLoader)
[Tue Jun 15 17:16:48.850991 2021] [wsgi:error] [pid 15745:tid 140431740368640] [client 41.85.181.159:57044]   File "/usr/local/lib/python3.6/dist-packages/yaml/__init__.py", line 114, in load
[Tue Jun 15 17:16:48.850999 2021] [wsgi:error] [pid 15745:tid 140431740368640] [client 41.85.181.159:57044]     return loader.get_single_data()
[Tue Jun 15 17:16:48.851010 2021] [wsgi:error] [pid 15745:tid 140431740368640] [client 41.85.181.159:57044]   File "/usr/local/lib/python3.6/dist-packages/yaml/constructor.py", line 51, in get_single_data
[Tue Jun 15 17:16:48.851018 2021] [wsgi:error] [pid 15745:tid 140431740368640] [client 41.85.181.159:57044]     return self.construct_document(node)
[Tue Jun 15 17:16:48.851030 2021] [wsgi:error] [pid 15745:tid 140431740368640] [client 41.85.181.159:57044]   File "/usr/local/lib/python3.6/dist-packages/yaml/constructor.py", line 55, in construct_document
[Tue Jun 15 17:16:48.851037 2021] [wsgi:error] [pid 15745:tid 140431740368640] [client 41.85.181.159:57044]     data = self.construct_object(node)
[Tue Jun 15 17:16:48.851079 2021] [wsgi:error] [pid 15745:tid 140431740368640] [client 41.85.181.159:57044]   File "/usr/local/lib/python3.6/dist-packages/yaml/constructor.py", line 100, in construct_object
[Tue Jun 15 17:16:48.851087 2021] [wsgi:error] [pid 15745:tid 140431740368640] [client 41.85.181.159:57044]     data = constructor(self, node)
[Tue Jun 15 17:16:48.851098 2021] [wsgi:error] [pid 15745:tid 140431740368640] [client 41.85.181.159:57044]   File "/usr/local/lib/python3.6/dist-packages/yaml/constructor.py", line 429, in construct_undefined
[Tue Jun 15 17:16:48.851104 2021] [wsgi:error] [pid 15745:tid 140431740368640] [client 41.85.181.159:57044]     node.start_mark)
[Tue Jun 15 17:16:48.851129 2021] [wsgi:error] [pid 15745:tid 140431740368640] [client 41.85.181.159:57044] yaml.constructor.ConstructorError: could not determine a constructor for the tag None
[Tue Jun 15 17:16:48.851137 2021] [wsgi:error] [pid 15745:tid 140431740368640] [client 41.85.181.159:57044]   in "/var/www/html/configuration.yaml", line 1, column 1

Thanks Emery.

akemery avatar Jun 15 '21 17:06 akemery

webterm: https://xxx.yyyy.bj:4449 webdav_host: http://xxxx.yyy.bj:4449

You should explicitely use quoting marks here because the colon is part of the YAML syntax

anthonygego avatar Jun 15 '21 18:06 anthonygego