Add own service for memcached

[EHJ-52n]

Description

Goal: "one service per container"

Background

• Docker documentation: "Run multiple services in a container"
• devops.stackexchange: "Why it is recommended to run only one process in a container?"

Changes

• add own service for memcached
• remove memcached from geonode image
• add MEMCACHED_OPTIONS variable to env files
• enable memcached by default

Checklist

Reviewing is a process done by project maintainers, mostly on a volunteer basis. We try to keep the overhead as small as possible and appreciate if you help us to do so by completing the following items. Feel free to ask in a comment if you have troubles with any of them.

For all pull requests:

• [x] Confirm you have read the contribution guidelines
• [ ] You have sent a Contribution Licence Agreement (CLA) as necessary (not required for small changes, e.g., fixing typos in the documentation)
• [x] Make sure the first PR targets the master branch, eventual backports will be managed later. This can be ignored if the PR is fixing an issue that only happens in a specific branch, but not in newer ones.

The following are required only for core and extension modules (they are welcomed, but not required, for contrib modules):

• [ ] There is a ticket in https://github.com/GeoNode/geonode/issues describing the issue/improvement/feature (a notable exemption is, changes not visible to end-users)
• [ ] The issue connected to the PR must have Labels and Milestone assigned
• [ ] PR for bug fixes and small new features are presented as a single commit
• [ ] Commit message must be in the form "[Fixes #<issue_number>] Title of the Issue"
• [ ] New unit tests have been added covering the changes, unless there is an explanation on why the tests are not necessary/implemented
• [ ] This PR passes all existing unit tests (test results will be reported by travis-ci after opening this PR)
• [ ] This PR passes the QA checks: flake8 geonode
• [ ] Commits changing the settings, UI, existing user workflows, or adding new functionality, need to include documentation updates
• [ ] Commits adding new texts do use gettext and have updated .po / .mo files (without location infos)

Submitting the PR does not require you to check all items, but by the time it gets merged, they should be either satisfied or inapplicable.

[cla-bot[bot]]

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have the users @EHJ-52n on file. In order for us to review and merge your code, please contact the project maintainers to get yourself added.

[gitguardian[bot]]

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them. Once a secret has been leaked into a git repository, you should consider it compromised, even if it was deleted immediately. Find here more information about risks.

---

<sup><sub>🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Our GitHub checks need improvements? Share your feedbacks!</sub></sup>

[codecov[bot]]

Codecov Report

Merging #8818 (2ed1236) into master (006332b) will increase coverage by 0.00%. The diff coverage is n/a.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #8818   +/-   ##
=======================================
  Coverage   61.95%   61.95%           
=======================================
  Files         867      867           
  Lines       51577    51577           
  Branches     6467     6467           
=======================================
+ Hits        31954    31956    +2     
+ Misses      18084    18081    -3     
- Partials     1539     1540    +1     

[gannebamm]

Welcome @EHJ-52n I like the idea of this PR a lot.

There is a ticket in https://github.com/GeoNode/geonode/issues describing the issue/improvement/feature (a notable exemption is, changes not visible to end-users)

I see this is a change that shall not be visible to end-users but nonetheless, I would like to have a proper improvement description as an issue.

@afabiani @giohappy Do you confirm my demand, or do you guys think this can be a PR without issue?

Same for https://github.com/GeoNode/geonode/pull/8817

[EHJ-52n]

...this option MEMCACHED_LOCATION is not currently driving the memcached lock...

@afabiani I am unsure what you are referring to by this?

I see this is a change that shall not be visible to end-users...

@gannebamm what do you want to say by this? Should the memcached configuration not be visible to "end-users"? Is an admin operating GeoNode an end-user here?

[afabiani]

@EHJ-52n please take a look here

as you can see the locking mechanism is created only taking into account two conditions:

1. sherlock is installed
2. it is able to create a lock

    import pylibmc
    import sherlock
    from sherlock import MCLock as Lock

    sherlock.configure(
        expire=settings.MEMCACHED_LOCK_EXPIRE,
        timeout=settings.MEMCACHED_LOCK_TIMEOUT)
    memcache_client = pylibmc.Client(
        [settings.MEMCACHED_LOCATION],
        binary=True)
    lock_type = "MEMCACHED"

no check is done against the MEMCACHED_ENABLED boolean

[gannebamm]

I see this is a change that shall not be visible to end-users...

@gannebamm what do you want to say by this? Should the memcached configuration not be visible to "end-users"? Is an admin operating GeoNode an end-user here?

I was referring to the checklist item

There is a ticket in https://github.com/GeoNode/geonode/issues describing the issue/improvement/feature (a notable exemption is, changes not visible to end-users)

And was thinking out loud, that we do not need an issue for that PR, since it is not something end-users will face.

[cla-bot[bot]]

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have the users @EHJ-52n on file. In order for us to review and merge your code, please contact the project maintainers to get yourself added.

[EHJ-52n]

@afabiani

Are you recommending to rephrase MEMCACHED_LOCK_EXPIRE to SHERLOCK_LOCK_EXPIRE or CACHE_LOCK_EXPIRE and MEMCACHED_LOCK_TIMEOUT, too?

In addition, surrounding the memcache_client initialization with a check for MEMCACHE_ENABLED?

try:
    import pylibmc
    import sherlock
    from sherlock import MCLock as Lock

    sherlock.configure(
        expire=settings.SHERLOCK_LOCK_EXPIRE,
        timeout=settings.SHERLOCK_LOCK_TIMEOUT)
    if settings.MEMCACHED_ENABLED:
        memcache_client = pylibmc.Client(
            [settings.MEMCACHED_LOCATION],
            binary=True)
        lock_type = "MEMCACHED"
    else:
        raise Exception
except Exception:
    from django.core.cache import cache
    from contextlib import contextmanager
    lock_type = "MEMCACHED-LOCAL-CONTEXT"
    memcache_client = None

I am not sure if the else case is required because I did not fully check the task module.

[giohappy]

@EHJ-52n can you please take a look at the conflicts?

[EHJ-52n]

@giohappy Fixed the conflicts. Are you okay with the changes?

[giohappy]

@EHJ-52n I have gone through the PR and comments.

IMHO the thing about MEMCACHE_ENABLED driving the Django cache but not the locking mechanism is outside this PR. I agree that mixing different mechanisms with variables sharing the same prefix can be confusing and error-prone, but it's something that should be addressed in another task.

Possible solutions (for another task):

• make MEMCACHE_ENABLED dive both (as in the last example from @EHJ-52n)
• rename MEMCACHE_ENABLED to something like DJANGO_MEMCACHE_ENABLED

[giohappy]

@EHJ-52n please align geonode-project once this is merged

[EHJ-52n]

@giohappy working on this - see GeoNode/geonode-project#479.