joomla-cms icon indicating copy to clipboard operation
joomla-cms copied to clipboard

[6.1] Variables from non natural environment

Open Fedik opened this issue 7 months ago • 47 comments

Pull Request for Issue #36898 .

Alternative to PR #45070 .

Summary of Changes

This is complete (I hope) implementation of environment variables in Joomla.

Key points:

  • By default the feature is disabled. Create .env in root folder to enable it (can be just an empty file).
  • I tried to keep the variable names in sync with Joomla Docker. If you find something is missing or a typo, please let me know.
  • The installation can be completed with environment variables (except language part). This work with CLI and WEB installer. User can provide all options via environment variables (including admin user info) or a few (like DB only, then installer will ask for missing options).
  • The configuration options provided by environment variables cannot be edited in backend, also via CLI.

Testing Instructions

Test installation, test with Web and in CLI installer

  1. Create .env with DB options:
JOOMLA_DB_TYPE="mysql"
JOOMLA_DB_HOST="Your DB host"
JOOMLA_DB_USER="Your DB user"
JOOMLA_DB_PASSWORD="Your DB password"
JOOMLA_DB_NAME="Your db name"
JOOMLA_DB_PREFIX="DB prefix"

And run installer. You should be asked for Site name, and User information. Then installation should be completed, as usual.

  1. Create .env with all installation parameters and user information:
JOOMLA_DB_TYPE="mysql"
JOOMLA_DB_HOST="Your DB host"
JOOMLA_DB_USER="Your DB user"
JOOMLA_DB_PASSWORD="Your DB password"
JOOMLA_DB_NAME="Your db name"
JOOMLA_DB_PREFIX="DB prefix"

JOOMLA_SITE_NAME="Test installation"
JOOMLA_PUBLIC_FOLDER=""

JOOMLA_ADMIN_USER="Your admin user"
JOOMLA_ADMIN_USERNAME="Your admin username"
JOOMLA_ADMIN_PASSWORD="Your admin user password"
JOOMLA_ADMIN_EMAIL="Your admin user email"

And run installer. You will not be asked for Site name, and User information. The installation should be completed, as usual.

Test the existing site

Create .env with DB options, and copy options from configuration.php. (skip this step if the site was installed with use of .env)

JOOMLA_DB_TYPE="mysql"
JOOMLA_DB_HOST="Your DB host"
JOOMLA_DB_USER="Your DB user"
JOOMLA_DB_PASSWORD="Your DB password"
JOOMLA_DB_NAME="Your db name"
JOOMLA_DB_PREFIX="DB prefix"

Then visit the site, all should work as before.

Link to documentations

Please select:

  • [ ] Documentation link for docs.joomla.org:
  • [ ] No documentation changes for docs.joomla.org needed
  • [ ] Pull Request link for manual.joomla.org: TBD
  • [ ] No documentation changes for manual.joomla.org needed

@Llewellynvdm please have a look if it will be good with Joomla Docker, thanks!

Fedik avatar May 24 '25 08:05 Fedik

dont we already have a PR for this #45070

brianteeman avatar May 24 '25 08:05 brianteeman

Correct, but that is incomplete implementation. This PR is alternative to that.

Fedik avatar May 24 '25 08:05 Fedik

Correct, but that is incomplete implementation. This PR is alternative to that.

Would be good to comment on that pr etc

brianteeman avatar May 24 '25 09:05 brianteeman

Correct, but that is incomplete implementation. This PR is alternative to that.

Would be good to comment on that pr etc

I've allowed myself to add a reference to the issue and a hint to the other PR at the top of the description.

richard67 avatar May 24 '25 17:05 richard67

Some suggestions:

  1. Use the $_SERVER instead of $_ENV

    Both can be disabled in php.ini, but the $_ENV is disabled by default: https://github.com/php/php-src/blob/201c691fab036b40f8b2ddcfd253fd21089ed799/php.ini-production#L652

    variables_order = "GPCS"
    

    That's why I use getenv() as a fallback:

    // getenv() is not thread-safe and it can cause segmentaion fault, so we should try $_SERVER first
    $envs = !empty($_SERVER) ? $_SERVER : getenv();
    
  2. Use the symfony/dotenv

    it's more advanced than the vlucas/phpdotenv and supports some good features like creating environments (dev, prod, test, staging and etc.) and dumping envs in production (.env.local.php).

    Also Joomla already uses some Symfony's components, so why not to use another one?

voronkovich avatar May 25 '25 00:05 voronkovich

Both can be disabled in php.ini, but the $_ENV is disabled by default

I tested on PHP 8.1 and 8.4, $_ENV is always present but unpopulated, with variables_order = "GPCS" and variables_order = "EGPCS".

The idea, that the feature is disable by default. And when User add .env then the Dotenv library will populate $_ENV, and we can use it.

Use the symfony/dotenv

To me Dotenv also looks good, and well supported library. And I would prefer something light. But if people will insist it could be changed to anything else, easily at any point of time, because it used only in bootstrap.

Fedik avatar May 25 '25 09:05 Fedik

$_ENV is disabled by default on most php installations. When disabled it will return an empty value

brianteeman avatar May 25 '25 09:05 brianteeman

@Fedik

The idea, that the feature is disable by default. And when User add .env then the Dotenv library will populate $_ENV, and we can use it.

Then the real environment variables won't work in cases where the $_ENV is disabled.

Sometimes it's very useful to run program with a changed environment. For example:

JOOMLA_PROXY_ENABLED=false php cli/joomla.php core:update

voronkovich avatar May 25 '25 19:05 voronkovich

But if people will insist it could be changed to anything else, easily at any point of time, because it used only in bootstrap.

It would be a BC break, because the symfony/dotenv works slightly different. It merges .env files, but the vlucas/phpdotenv doesn't change already loaded values.

Example

# .env
HELLO='env'
# .env.dev
HELLO='env.dev'

vlucas/phpdotenv:

<?php

require 'vendor/autoload.php';

Dotenv\Dotenv::createImmutable(__DIR__, ['.env', '.env.dev'])->safeLoad();

// Outputs: 'env'
echo $_ENV['HELLO'];

symfony/dotenv:

<?php

require 'vendor/autoload.php';

(new Symfony\Component\Dotenv\Dotenv())->bootEnv(__DIR__ . '/.env', 'dev');

// Outputs: 'env.dev'
echo $_ENV['HELLO'];

voronkovich avatar May 25 '25 21:05 voronkovich

I added code to check for empty ENV.

It would be a BC break, because the symfony/dotenv works slightly different. It merges .env files, but the vlucas/phpdotenv doesn't change already loaded values.

I switched shortCircuit to false, should be the same now

Fedik avatar May 31 '25 08:05 Fedik

I switched shortCircuit to false, should be the same now

Nothing is changed. It still works like in the example above. I've created a repository with an example: https://github.com/voronkovich/dotenv-example.

You can't make the vlucas/phpdotenv works the same way as the symfony/dotenv (believe me, I've already tried).

I suggest to simplify this PR and load only the .env file. Later, in other PRs, we can add support for either symfony/dotenv or env.dev.

voronkovich avatar Jun 01 '25 04:06 voronkovich

Can you move this code block upward outside the if statement? Without it real environment variables won't work if .env file is not present and $_ENV is disabled.

This is intentional. To enable envs on the site User should create an .env file (at least empty), or enable in php.ini (variables_order parameter). If you have VPS, then last option should not be a problem for you.

For most Users it does not need, so we do not need all that (for now) to be always enabled. However it can be discussed.

Fedik avatar Jun 01 '25 12:06 Fedik

This is intentional. To enable envs on the site User should create an .env file (at least empty), or enable in php.ini (variables_order parameter).

I've never seen an application that requires enabling an option to make environment variables work. Because it doesn't make sense.

In Symfony everything works out of the box. In Laravel everything works out of the box. In WordPress everything works out of the box. Even curl doesn't require you to do anything to make envs work. :)

Joomla will be the first one.

voronkovich avatar Jun 01 '25 21:06 voronkovich

what's the difference between .env and .env.dev?

First one is for production, second one for development. Or whatever User decide. It is kind of look up list for which files to look.

In the example:

# .env
JOOMLA_DB_NAME=potato

#.env.dev
JOOMLA_DB_NAME=potato_dev

Will be used potato_dev

Will add to gitignore, but not very important.

Fedik avatar Jun 04 '25 08:06 Fedik

@Fedik, Your example won't work, because you use Dotenv::createImmutable(). The suffix "immutable" means that the existing environment variables are never changed. You should rearrange the files like this:

Dotenv\Dotenv::createImmutable(JPATH_ROOT, ['.env.dev', '.env'], false)->safeLoad();

Alternatively you can use Dotenv::createMutable(), but it rewrites real envs which is very bad idea.

voronkovich avatar Jun 04 '25 09:06 voronkovich

@Fedik,

second one for development. Or whatever User decide.

The .dev has a precise meaning: "development". If you want to use it for overriding the .env then I would suggest you to rename it to .env.local (local overrides).

Some examples: Symfony, NextJS

voronkovich avatar Jun 04 '25 09:06 voronkovich

It is good as it is. Can be updated any time later.

Fedik avatar Jun 07 '25 09:06 Fedik

This significant part of our security concept in terms of file access is based upon the idea, that confidential values (as DB credentials) are stored in .php files and are therefore unaccessible via direct webserver calls. This PR and the idea of .env files within the webroot breaks that concept.

So, I see 3 options:

  1. rename the file to .env.php, add a die() statement and adjust the parser to ignore that statement
  2. only allow .env files in a setup, where the JROOT is not the webroot
  3. block access to the file using webserver config files

And to be honest none of the mentioned options is a great solution.

SniperSister avatar Aug 14 '25 08:08 SniperSister

By default the feature is disabled. Create .env in root folder to enable it (can be just an empty file).

If I read the code correctly, the feature can either be enabled by adding the file or by defining the JOOMLA_ENVIRONMENT variable in $_ENV - correct?

SniperSister avatar Aug 14 '25 08:08 SniperSister

Correct. If you have set your server to enable $_ENV, and add JOOMLA_* environment variables from there, it will be also enabled.

Fedik avatar Aug 14 '25 08:08 Fedik

Ok, thanks for the confirmation! Just wanted to make that I'm not overlooking something

SniperSister avatar Aug 14 '25 08:08 SniperSister

About web access to the env file, I think it is valid concern. Even though the feature meant to be for people who know thing or two about what they doing.

Symfony dotenv allows to load .env.php, but it also still allows .env, I not very wanted to switch to that, but need to look.

Fedik avatar Aug 14 '25 09:08 Fedik

I tested this locally under Laragon 6.0. Both tests (small .env and complete .env) worked fine with Web. But using CLI (with the incomplete .env), the process entered a loop when the password was asked. The error log shows (numerous entries): [15-Aug-2025 18:37:51 UTC] PHP Deprecated: rtrim(): Passing null to parameter #1 ($string) of type string is deprecated in D:\laragon\www\PR-Test\libraries\vendor\symfony\console\Helper\QuestionHelper.php on line 416

dautrich avatar Aug 15 '25 18:08 dautrich

@dautrich can you share the .env file you use for CLI? without private data, replace it to random strings. and how did you applied the PR changes for test?

Fedik avatar Aug 15 '25 19:08 Fedik

Here my .env:

.env.zip

dautrich avatar Aug 15 '25 19:08 dautrich

Thanks. I just tried CLI with elements from your env (but with my values), and all went well. Found little bug, but was different kind of validation. Not an error from QuestionHelper.php.

Not sure, maybe something related to Laragon? What happen if you enter empty password? Should get: Screenshot 2025-08-15_22-20-48

Fedik avatar Aug 15 '25 19:08 Fedik

After I entered my superuser account, the installation immediately went into a loop. I was not able to enter a password.

dautrich avatar Aug 15 '25 19:08 dautrich

I used the prebuilt package "Joomla_6.0.0-alpha4-dev+pr.45523-Development-Full_Package.zip"

dautrich avatar Aug 15 '25 19:08 dautrich

After I entered my superuser account, the installation immediately went into a loop. I was not able to enter a password.

@dautrich what happen when you leave empty value for supper user? Do you get validation error kind of "field required" or it also start looping?

I used the prebuilt

Please try new prebuild, I fixed another little error. (Still no idea what happened in your CLI )

Fedik avatar Aug 15 '25 19:08 Fedik

I tried with the new prebuilt package. Same issue. See Screenshot:

Bildschirminhalt erfassen-1

The error message means "Path not found".

dautrich avatar Aug 15 '25 20:08 dautrich