spring-boot icon indicating copy to clipboard operation
spring-boot copied to clipboard

Support .env file for configuration properties

Open linux-china opened this issue 4 years ago • 40 comments

Lots of developers use .env to process own configuration for script or app. Now I add spring.config.import in application.properties as following:

spring.config.import=optional:file:.env

and SpringBoot app throws the following exception:

Caused by: java.lang.IllegalStateException: File extension is not known to any PropertySourceLoader. If the location is meant to reference a directory, it must end in '/'
	at org.springframework.boot.context.config.StandardConfigDataLocationResolver.getReferencesForFile(StandardConfigDataLocationResolver.java:199)
	at org.springframework.boot.context.config.StandardConfigDataLocationResolver.getReferences(StandardConfigDataLocationResolver.java:121)

I think .env file could be considered as properties file for spring.config.import .

linux-china avatar Nov 22 '20 03:11 linux-china

@linux-china Thanks for getting in touch. Simply considering a .env file as a properties file wouldn't do what I think most users expect.

When Spring Boot reads key/value pairs from the system environment, additional processing is required to map the typical UPPERCASE_WITH_UNDERSCORES style of environment variable keys to the canonical form of Spring Boot property keys. This mapping is not done when configuration is read directly from a properties file. I suspect most users would use the environment variable style of keys in a .env file, so the fact that the key name mapping is not performed when importing the file as a properties file would result in the expected properties not being present in the Spring Environment.

In order for this to work as expected, a .env file would need to be recognized as a file containing key/value pairs similar to system environment variables, and processed accordingly. This is, of course, a bigger change than what you've suggested.

Can you elaborate on the types of properties that you would find convenient to set in a .env file instead of a local application.properties file?

scottfrederick avatar Nov 24 '20 22:11 scottfrederick

@scottfrederick Now I use jbang to write some scripts for Spring Boot App, scripts and app share some same configuration, such as database url etc, and https://github.com/cdimascio/dotenv-java is very friendly for jbang and kscript. I also have some script managed by just command runner, and .env is also supported by oh-my-zsh dotenv plugin, and .env variables will be loaded into env variables automatically for some command line tools. I mean .env is convenient for developers.

But now still ok for me, I can use ln -s .env env.properties and spring.config.import=file:env.properties also works well. Or ln -s .env application.properties for a local application.properties for Spring Boot.

linux-china avatar Nov 24 '20 22:11 linux-china

@linux-china Here's a more concrete example of what I think would be the problem with loading a .env file as a properties file:

In a properties file, you can set a property like spring.application.name=myapp. If you want to set this property as an environment variable, you can set SPRING_APPLICATION_NAME=myapp. Either property will resolve to spring.application.name in the Spring Environment (e.g. using @Value(${spring.application.name})).

If you have SPRING_APPLICATION_NAME=myapp in a properties file, this will not be resolved to spring.application.name because the property key mapping logic is not applied when reading properties files. Properties file are expected to use the canonical form.

Since .env files typically contain keys that conform to the environment variable naming conventions, I think users would naturally set properties like SPRING_APPLICATION_NAME=myapp in the .env file and then be surprised when the application name wasn't set as expected (because the file is being read as a properties file and the property key mapping isn't done). This will work if you set spring.application.name=myapp in the .env file, but you'd have to understand how Spring Boot works to know to use that form.

I can use ln -s .env env.properties and spring.config.import=file:env.properties

In your use case, are you using the canonical form in .env, or how are you setting the properties there?

scottfrederick avatar Nov 25 '20 18:11 scottfrederick

yes, I use normal key style form in .env file, and my .env file content as following:

nick=linux_china
management.server.port=9999

then I create link for .env with ln -s .env env.properties

my application.properties as following:

spring.application.name=spring-boot24-demo
spring.config.import=optional:file:env.properties

All work well, @Value("${nick}") private String nick; and management listen port is 9999

but one problem is the env variable, management.server.port could not be used as env variable name, and you have described this problem. Maybe I should do some modification for dotenv plugin, and convert management.server.port to MANAGEMENT_SERVER_PORT automatically.

linux-china avatar Nov 25 '20 19:11 linux-china

We think there is value in supporting .env files as an enhancement to Spring Boot. We've marked the issue pending-design-work to consider a few implementation options:

  • Automatically process files named .env in the current directory
  • Add a new ConfigDataLocationResolver and ConfigDataLoader to support loading files with something like spring.config.import=envfile:.env (note that this would read the contents of the file as a set of environment variables, not as a properties file as originally suggested)
  • Add support for a new extensionless-file hint type for files containing environment variable key-value pairs with something like spring.config.import=file:/etc/config/.env[.env]

scottfrederick avatar Nov 30 '20 17:11 scottfrederick

@scottfrederick I mad a mistake about .env file format, and key should be uppercase like following:

KEY=xxx
DATABASE_URL=xxxxx

Some frameworks use multi .env files like Spring Boot profile, for example Vue.js CLI.

.env               
.env.local         
.env.[mode]   

I'm not sure some developers use following style or not.

.env
local.env
xxxx.env

Maybe Spring Boot should support to load .env.dev or dev.env styles by profile name.

linux-china avatar Nov 30 '20 18:11 linux-china

I think maybe the better approach would be to parse the .env file upon application startup and provide the content as "virtual" environment variables, whose keys could be addressed in application.properties by regular placeholders. As such:

.env file content: DB_USER=mockUserName

application.properties content: spring.datasource.hikari.username=${DB_USER}

RogerVFbr avatar Feb 07 '21 13:02 RogerVFbr

If anyone's still interested in using .env as another properties file, for me it works by adding

# Development properties (also used for docker-compose)
spring.config.import=optional:file:.env[.properties]

into application.properties.

The [.properties] is a hint for spring which translates to "Use this file without an extension as a .properties file."

TheBestPessimist avatar Feb 22 '21 12:02 TheBestPessimist

If anyone's still interested in using .env as another properties file, for me it works by adding

# Development properties (also used for docker-compose)
spring.config.import=optional:file:.env[.properties]

into application.properties.

The [.properties] is a hint for spring which translates to "Use this file without an extension as a .properties file."

It works, but still some work for Spring Boot. Spring Boot should consider additional processing is required to map the typical UPPERCASE_WITH_UNDERSCORES style of environment variable keys in .env file.

linux-china avatar Feb 22 '21 20:02 linux-china

One consideration that always comes up is that environment variables will impact all profiles, and while this sort of makes sense, it's sometimes tricky when running tests and things like that.

I think the implementation should take into consideration a way to set some environment variables for a specific profile — something like:

#
#
the_boot.ENV_VAR = value # ...will only impact the_boot profile

ENV_VAR_ALL = world      # ...will apply at application level (like a usual OS environment variable declared and accessible anywhere)

x80486 avatar Mar 03 '21 20:03 x80486

Not trying to disrupt anyone else's work here, but there's a package for that: https://search.maven.org/artifact/me.paulschwarz/spring-dotenv/2.3.0/jar

spam-n-eggs avatar Apr 19 '21 15:04 spam-n-eggs

I think maybe the better approach would be to parse the .env file upon application startup and provide the content as "virtual" environment variables, whose keys could be addressed in application.properties by regular placeholders. As such:

.env file content: DB_USER=mockUserName

application.properties content: spring.datasource.hikari.username=${DB_USER}

How do i "parse the .env file upon application startup and provide the content as "virtual" environment variables" So that i can read the values inside .env and use them on my application.properties file, i want to store my password on the .env than directly on the application.properties file. Please give me pointers. thanks

maranza avatar Jul 13 '21 12:07 maranza

The co.uzzu.dotenv.gradle Gradle plugin does that (in that context). If you need to read the environment variable values in the source code, you would need to use a library: io.github.cdimascio:dotenv-java.

I hope that Spring Boot, at some point, unifies all of that "natively".

x80486 avatar Jul 13 '21 12:07 x80486

Thank you so much. However maven seems to not be able to find the co.uzzu.dotenv.gradle from the repos. I am getting the following error

co.uzzu.dotenv.gradle:co.uzzu.dotenv.gradle.gradle.plugin:pom:1.1.0 was not found in https://repo.maven.apache.org/maven2

when trying both:

```
	<dependency>
		<groupId>co.uzzu.dotenv.gradle</groupId>
		<artifactId>co.uzzu.dotenv.gradle.gradle.plugin</artifactId>
		<version>1.1.0</version>
		<type>pom</type>
	</dependency>

and 
	<dependency>
		<groupId>co.uzzu.dotenv</groupId>
		<artifactId>gradle</artifactId>
		<version>1.0.1</version>
	</dependency>

Which i got from:

`https://mvnrepository.com/artifact/co.uzzu.dotenv/gradle/1.1.0`



maranza avatar Jul 14 '21 15:07 maranza

@maranza Even with the right repository configuration, you won't be able to use a Gradle plugin in Maven.

wilkinsona avatar Jul 14 '21 16:07 wilkinsona

Hi @wilkinsona thank you for clarity. so what i am trying to achive is currently not possible. loading application.properties value from a .env file?

maranza avatar Jul 14 '21 16:07 maranza

It's possible, but you'd have to write some code of your own to parse the .env file and load it as configuration data. The io.github.cdimascio:dotenv-java library that @x80486 linked to above may help you to do that.

wilkinsona avatar Jul 14 '21 16:07 wilkinsona

If anyone's still interested in using .env as another properties file, for me it works by adding

# Development properties (also used for docker-compose)
spring.config.import=optional:file:.env[.properties]

into application.properties.

The [.properties] is a hint for spring which translates to "Use this file without an extension as a .properties file."

It would be great if this could work with spring.profiles.active to fetch the correct .env, like: spring.config.import=optional:classpath:.env-${spring.profiles.active}[.properties]

membersound avatar Jul 20 '21 14:07 membersound

@membersound since spring.profiles.active has a list of values, which .env should spring load? imo this option is extremely difficult to implement for arguably little benefit

TheBestPessimist avatar Jul 21 '21 05:07 TheBestPessimist

I forgot that multiple profiles can be added to that property (as I always only use one profile at once).

But well, in general the setup might be as follows:

application.properties
application-test.properties
application-production.properties

Now if each of the environments contains secrets that could be externalized into a .env file, it would result in something like:

.env.example    #listing all values that should be configured by the user. should not be loaded by spring, and is commited.
.env            #general secrets that are valid for all profiles
.env-test       #profile specific values
.env-production #profile specific values

How would you tell spring to load first the .env, and then the profile-specific env?

Of course every application-*.properties could set the spring.config.import property itself, but I'd prefer having a one-line statement in only one file that covers all situations. (because normally we have some kind of application-commons.properties that all of our projects inherit from. And this commons project should define the spring.config.import value for all .env files, so that implementing projects don't have to care about .env configuration, but can simply throw them into the /src/main/resources folder and it works).

membersound avatar Jul 21 '21 07:07 membersound

A .env file is usually meant to be used by you only when working in your desktop/laptop. On higher environments you just refer to the environment variables and act accordingly.

x80486 avatar Jul 21 '21 10:07 x80486

If so, where do you put secrets for test/production then (assuming a locally build app)? At least they cannot be written into the application-*.properties, as those are usually committed.

membersound avatar Jul 21 '21 11:07 membersound

No, you refer to those secret values by using an environment variable in your application-*.properties files:

# .env file
DATABASE_PASSWORD = "ThePassword"

SPRING_PROFILES_ACTIVE = "production"
# application.yaml
spring:
  datasource:
    password: "${DATABASE_PASSWORD}"

How do you fetch and provide those secrets is outside the scope of this issue. You can as well create a .env file and source it with those values before starting the application — while using the same variables names.

If you have multiple .env[-profile] files you will have the same problem as having those secrets in the usual .properties files: where are you going to store them securely? On the other hand, if you have the means to store and retrieve secrets securely for each environment, but you can't provide them as environment variables, you could inject them by following the using the usual .env.template file and replacing the values accordingly — resulting in just a .env file in the end.

x80486 avatar Jul 21 '21 11:07 x80486

I would like to see the .env file be used to setup development environment. These custom variables will be referred in my local development profile.

My typical usecase for .env would be to have it on the project home directory.

File: .env

MY_KEY=My Local Value
...

On my local application.properties or application.yml file, would use it like,

my.app.properties=${MY_KEY} is working
. . .

Or

my:
 app:
    properties: '${MY_KEY} is working'

And expect my code using @Value("${my.app.properties}") to be able to resolve it before doing DI.

reflexdemon avatar Jul 31 '21 01:07 reflexdemon

I would like to see the .env file be used to setup development environment. These custom variables will be referred in my local development profile.

That would indeed be the ideal solution.

membersound avatar Jul 31 '21 08:07 membersound

Any plans to implement suggested by @reflexdemon solution soon?

petromir avatar Sep 11 '21 19:09 petromir

@petromir not really I’m afraid. We’re currently investing most of time on other issues. I suspect it might take a while for us to get to this one.

philwebb avatar Sep 12 '21 02:09 philwebb

How to use env files with application.yml config:

spring:
  config:
    import: optional:file:app.env[.properties]

  datasource:
    url: ${DATABASE_URL}
    username: ${DATABASE_USER}
    password: ${DATABASE_PASSWORD}

JayDi85 avatar Oct 13 '21 06:10 JayDi85

How to use env files with application.yml config:

spring:
  config:
    import: optional:file:app.env[.properties]

  datasource:
    url: ${DATABASE_URL}
    username: ${DATABASE_USER}
    password: ${DATABASE_PASSWORD}

works fine!

bdmstyle avatar Dec 26 '21 03:12 bdmstyle

I can confirm the spring.config.import: optional: approach works fine both for local development and automated deployments, as follows:

With application.properties:

spring.config.import=optional:classpath:.env[.properties]
spring.datasource.password=${DATABASE_PASSWORD}

Now you can create an .env file only on your local development machine with:

DATABASE_PASSWORD=secret

As well, you can use your CI/CD deployment jobs to provide the secret from secret-store, and pass it to the spring application as environment variable, eg with docker-compose:

services:
  app:
    environment:
      - DATABASE_PASSWORD=$DATABASE_PASSWORD

Unfortunately, it was not possible o add the spring.config.import=optional:classpath:.env[.properties] definition to a company-commons.properties file in a company-commons.jar library that is used by the implementation applications. The spring.config.import setting must be repeated for any application explicit, and somehow cannot be inherited.

membersound avatar May 17 '22 11:05 membersound