Enhance PropertySources to be profile aware

[twicksell]

Boot's feature of loading application-${profile}.properties/yml is excellent, but it would be great to see this extended into other PropertySources, such as those provided by EnvironmentPostProcessor or @PropertySource. This essentially would be offering profile awareness to library implementations who provide properties to the application. Similar options are available in https://github.com/Netflix/archaius/tree/2.x to manage profile based config loading from application or library.

Seeking opinions on:

1. Adding an extension of ResourcePropertySource along the lines of ProfileAwareResourcePropertySource which could be used in places like EnvironmentPostProcessor
2. Adding an annotation along the lines of @ProfileAwarePropertySource to achieve the same.
3. Adding a ProfileAwareEnvironmentPostProcessor to essentially apply the profile loading behavior to all existing PropertySources.

Example:
@PropertySource("myprops") with profile foo should result in two PropertySources added to the Environment, myprops.properties and myprops-foo.properties. myprops-foo.properties should be added directly before myprops.properties such that the more specific configurations override the less specific ones. This should be extended to include multiple profiles, with the ordering profiles being applied reflecting their precedence.

This would give us options for enabling this feature explicitly per PropertySource or globally. Any thoughts?

[snicoll]

We discourage you from using @PropertySource in Spring Boot so I am personally not keen on any change in that area.

If I understand you properly, you're looking for an API that does what we do for standard application.properties with a way to trigger that somewhere. Can you explain why the current infrastructure is not enough? If I got this right, it looks like you want to read application and myprops where myprops search location(s) may be different. Did I get this right?

[twicksell]

Yes, I want to be able to re-use the logic for what we call cascaded property loading (loading application.properties, application-profile1.properties, application-profile2.properties, etc.) for more than just application properties. This is primarily for library development, where a library includes within its jar some properties files. We want the library to also be able to have its properties loaded based on the currently active profiles.

For a more concrete example, imagine you are in a cloud/microservice arrangement where you own a service (ServiceA), and you want to communicate with some other service (ServiceB). Also imagine that each service has multiple copies of their deployment for different levels of testing, and that each of those deployments is given a profile. For the sake of example, lets say these deployments are each given profiles of staging and production.

If we are using something like Eureka+Ribbon, there are a number of configuration properties ServiceA would need to have to communicate with ServiceB. To reduce copy/paste, ServiceB might publish those configurations in a client jar, with an AutoConfiguration which loads those properties. The ServiceB library will want to provide different properties based on the environment/profile the code is running in. The jar will contain serviceb-staging.properties, and serviceb-production.properties. The jar would also contain an EnvironmentPostProcessor to load those properties, using the proposed ProfileAwareResourcePropertySource.

ServiceA would import this library, and when running with the staging profile, would have application-staging.properties and serviceb-staging.properties. The end result being when ServiceA runs with the staging profile, it consistently loads configs designated for the staging profile at both the application and library level.

[joshiste]

ServiceB might publish those configurations in a client jar

Environment dependent configurations in a library? I consider this an anti-pattern and feels really odd to me. Maybe you should take a look into spring-cloud-config...

[spencergibb]

😆 considering we modeled config server after one of their services 😉

[twicksell]

Yep, we absolutely have a remote config server, although we consider using it for more than temporary overrides to be an anti-pattern as putting all config there creates a central point of failure. If your app can't run correctly without the config-server being available, it'll be a very real problem when the config backend inevitably goes down.

So we very much still package default configurations in libraries. And we scope those to deployment considerations like Test/Prod/Staging, AWSAccount, AWS Region, etc. As we've adapted this to the Spring Boot world, we've found Profiles are a pretty good implementation for this concept. But the fact that only application properties are profile aware, while no other PropertySource has the same behavior doesn't quite match our expectations.

[snicoll]

There are a few other initiatives that might provider a lower-level API that ultimately can help you implementing this use case: #12412 to ease the loading of a YAML file and #3783 to provide a richer API.

[chrisgleissner]

We would also find this useful. We have a Spring Boot app that uses Spring Batch. This in turn uses a few nested contexts that load Spring XML files. (Nested contexts are a feature of Spring Batch to get around bean name clashes.) This means we have a main Spring Boot context (using the default property resolution) as well as nested contexts (which need to rely on @propertysource).

[philwebb]

I think we should close this one for now and see how the new ConfigData work pans out. Users can now easily have profile specific imports which might negate the need for this. E.g:

spring.config.import=mycustomimport:something
spring.config.activate.on-profile=production

We can always reopen this one if the above is still too cumbersome.

[l0co]

From my perspective this would still be useful feature to consider. For example we have some stack of base libraries across all technologies we use in the project. For example, we have a base library for mariadb, based on spring data. Besides it provides some dependencies configuration and plumbing code to be used in all services using mariadb in our project, we also want it to provide some basic configuration preset, which should then be used by all services, without repeating it in application.yml of each service. For example:

spring:
  datasource:
    driver-class-name: org.mariadb.jdbc.Driver
    username: ${maria.user}
    password: ${maria.pass}
    url: jdbc:mariadb://${maria.host}:${maria.port}/${maria.db}
  jpa:
    properties:
      hibernate:
        schema_update:
          unique_constraint_strategy: RECREATE_QUIETLY
        dialect: org.hibernate.dialect.MariaDB10Dialect

So, basically, we don't want to repeat all this stuff in each service, we just want it to be provided by the lib, and the service config is limited only to this:

maria:
  user: ...
  pass: ...
  db: ...

This can be done using @PropertySources and a custom bean, but unfortunately it neither supports yaml out of the box, nor custom profiles. And we need custom profiles configs for a number of reasons. For example we might want to have some profile which enables detailed SQL logs, or we might to want to be able turn on or off second level cache with some profile. Currently we aren't able to do that, because @PropertySources is very limited.

In my opinion spring.config.import is not a right solution here because we have at least to repeat this import line in all services using our lib, also for each profile we want to support, and considering we have around 15 of such libs, it would make a big copy&paste code to be included in each service application.yml.

There's somewhere in Spring a properly working code, supporting yamls, profiles, new config file processing etc. Why not to use the same code under @PropertySource implementation and remove current custom implementation?

BTW here is my working workaround to achieve the same results in current spring implementation.

[philwebb]

It's unfortunately going to be quite hard to change @PropertySource to work like this by default because it's part of Spring Framework and really should not have any knowledge of Spring Boot concerns. I think the best that we could do is to offer a PropertySourceFactory implementation in Spring Boot that would provide an opinionated way of loading the files.

Even then, it's going to be a little tricky because (as you've discovered) it's difficult to get the environment into the loader. We might be able to make a framework change so that PropertySourceFactory implementations could also be EnvironmentAware. The code we'd need to somehow share would be from StandardConfigDataLocationResolver.

Flagging for team discussion to see if there are any other ideas that would help us extend @PropertySource.

[philwebb]

@l0co One other abstraction that you might be able to use is EnvironmentPostProcessor. If you order it after ConfigDataEnvironmentPostProcessor then it should give you the Environment with the profiles applied. You can add additional PropertySources as required.

[l0co]

I can understand that this feature is a part of the framework and is not possible to be introduced within Spring Boot only.

In this scenario I can't see what can be done more about the subject. Maybe just some explicite examples of how to achieve it, maybe as the answer to the mentioned stackoverflow thread where people are directed first, looking for the solution to this problem in google. It can be either my way, using EnvironmentAware or EnvironmentPostProcessor. Whatever working, probably the easiest way would be the best one.

An maybe, looking for example on my solution, if this kind of MyPropertySourceFactory I've presented there, became a part of Spring Boot with clear usecase description in the documentation, it would probably be even easier for users to use it.