Importing config files across modules

[edudar]

I have two modules in the same project, one by another:

A
|-- application.yaml
|-- application-dev.yaml

B
|-- base-application.yaml
|-- base-application-dev.yaml

Been using custom EnvironmentPostProcessor to load base- config into A but with SB 2.4 I wanted to give config.import a try. But there are a few issues arisen.

(1)
In A/application.yaml I've added the following:

spring:
  config:
    import: classpath:/base-application.yaml

In A/application-dev.yaml it's respectively:

spring:
  config:
    import: classpath:/base-application-dev.yaml

Trying to start A fails with Config data location 'classpath:/base-application-dev.yaml' does not exist. That's rather confusing considering that it does exist indeed and that classpath:/base-application.yaml is loaded just fine from the same location. To verify that, I made a typo in the file name and it failed for 'classpath:/bas-application.yaml' as expected.

Renaming base-application-dev.yaml to something profile-unrelated like some-file.yaml still fails with resource does not exist.

(2)
If I don't change A/application-dev.yaml at all, I do see that SB takes B/base-application-dev.yaml into account when loading classpath:/base-application.yaml and properly loads it. The application starts just fine with all properties. So clearly it does exist in (1). But now the ordering of property sources is not correct. Here's what it is from environment:

...
application-dev.yaml
base-application-dev.yaml
base-application.yaml
application.yaml
...

base-dev overrides dev but application overrides base-application.

[wilkinsona]

@edudar Can you please provide a minimal sample that reproduces the behaviour that you have described?

[edudar]

Published to edudar/sb26210

Right now, classpath:/base-dev.yml is commented out in application-dev.yml. App starts and prints out:

INFO 42529 --- [           main] org.example.sb26210.app.App              : Config resource 'class path resource [application-dev.yml]' via location 'optional:classpath:/'
INFO 42529 --- [           main] org.example.sb26210.app.App              : Config resource 'class path resource [base-dev.yml]' via location 'classpath:/base.yml'
INFO 42529 --- [           main] org.example.sb26210.app.App              : Config resource 'class path resource [base.yml]' via location 'classpath:/base.yml'
INFO 42529 --- [           main] org.example.sb26210.app.App              : Config resource 'class path resource [application.yml]' via location 'optional:classpath:/'

If you uncomment classpath:/base-dev.yml, App would fail with

[main] DEBUG org.springframework.boot.diagnostics.LoggingFailureAnalysisReporter - Application failed to start due to an exception
org.springframework.boot.context.config.ConfigDataLocationNotFoundException: Config data location 'classpath:/base-dev.yml' cannot be found

[scottfrederick]

Thanks very much for the sample.

The multi-module aspect doesn't appear to be important here - I can also reproduce the issue with all four properties files in the same module. The key is to have the imports for base.yml and base-dev.yml while the dev profile is active. The import: classpath:/base-dev.yml isn't necessary, as is shown in your sample output above the profile-specific file is getting included because of the profile activation without the import. But the fact that you're getting an error, and that the error message is misleading, is a problem.

[edudar]

@scottfrederick shouldn't base.yml and application.yml also switch places in property sources ordering? So that the order is app-dev -> base-dev -> app -> base? This way when one imports an external config, it'd be possible to override some values in place.

[scottfrederick]

As stated in the documentation, profile-specific properties files override non-profile-specific properties files and properties from imported files override properties from the file that contains the import.

Without the import: classpath:/base-dev.yml, this appears to be working correctly in your sample, since the properties are resolved to these values when the dev profile is active:

2021-04-16 13:23:03.003  INFO 98563 --- [           main] org.example.sb26210.app.App              : a = app-dev
2021-04-16 13:23:03.005  INFO 98563 --- [           main] org.example.sb26210.app.App              : b = app
2021-04-16 13:23:03.005  INFO 98563 --- [           main] org.example.sb26210.app.App              : c = base-dev
2021-04-16 13:23:03.005  INFO 98563 --- [           main] org.example.sb26210.app.App              : d = base

With the import: classpath:/base-dev.yml in application-dev.yml, the code is a bit confused about whether it should consider base-dev.yml because it is the profile-specific form of base.yml that has already been imported or because it is explicitly imported.

[edudar]

@scottfrederick

properties from imported files override properties from the file that contains the import

I think there are two different views here.

(2.1) I do understand where the doc is coming from. If I have an external config like a configtree or consul or similar deployed in production/staging/uat/younameit, I do want values from there to override my bundled properties from a jar. Essentially, jar provides some predefined properties that can be overridden in production.

(2.2) But the example in this ticket is pretty much the opposite. That's why I named files base and app. If we move away from abstract a-b-c-d, let's imagine that base module contains a configuration for HTTP client and base.yml contains all sorts of timeouts and whatnot, headers, user agents, etc. Now, I'm thinking that as general guidance, I'd set read timeout to 1000ms: base.yml -> timeouts.read: 1000ms. At the same time in dev it can be longer, right, so base-dev.yml -> timeouts.read: 5000ms. Following this, I'm developing a specific application that won't tolerate such extensive timeouts. So my idea would be to import base.yml to get all properties but I override read timeout, so app.yml -> timeouts.read: 100ms. And similar to the base, when in dev, timeouts may be extended, so app-dev.yml -> timeouts.read: 500ms.

When I run my app in dev, I will get 500ms timeouts as it's set in config, because app-dev.yml takes precedence over all other files right now, including base-dev.yml. BUT. If I run my app without a profile, I'll get 1000ms timeout, because it's coming from base.yml and it overrides my 100ms setting for the app.yml. My point is that I'd probably expect the timeout to be 100ms, not 1000ms. However, if we stick to what (2.1) says, then running in dev should probably give me 5000ms timeout coming from base-dev.yml instead of 500ms from app-dev.yml.

[scottfrederick]

As you said initially, there are a few issues here.

One is a bug that's causing a confusing error message unrelated to modules and precedence of property loading. I've now split that out into #26147 so we can address that bug.

The rest of this discussion has to do with a specific problem you are trying to solve and whether your approach will work given the design of Spring Boot's property loading. Let's continue that discussion here.

[scottfrederick]

Breaking down the the comment above:

If I run my app without a profile, I'll get 1000ms timeout, because it's coming from base.yml and it overrides my 100ms setting for the app.yml.

This is the expected behavior, as stated in the documentation (the values from the imported file will take precedence over the file that triggered its import).

When I run my app in dev, I will get 500ms timeouts as it's set in config, because app-dev.yml takes precedence over all other files right now, including base-dev.yml. However, if we stick to what (2.1) says, then running in dev should probably give me 5000ms timeout coming from base-dev.yml instead of 500ms from app-dev.yml.

This is a side-effect of the fact that a file like base-dev.yml isn't being treated properly and can't be imported from app-dev.yml. #26147 should take care of that.

My point is that I'd probably expect the timeout to be 100ms (the value from app), not 1000ms (the value from base).

This is the part that is contrary to the explicit design of the import feature. If you need the order of precedence to work the way you are describing (where values from app can override values from base), then import is probably not the right mechanism to use. Aside from the fact that the precedence isn't in the order you need for this design, all apps using base would be required to explicitly import configuration from base to get the defaults, even if they didn't need to override the defaults. It would probably be better for base to always apply its own configuration defaults without requiring app to be explicit about it. You mentioned that you're currently using EnvironmentPostProcessor to load base- config, and that design is likely a better fit for what you're trying to do.

[scottfrederick]

Also see #24688 for a similar use case.

[edudar]

What order do we load the resources and how do we make sure a developer can override a contributed value

Yeah, that's an ultimate question in the late discussion here. I'll close this one then as there are two tickets that cover it now.