Too excessive component scanning for repository implementation fragments causing slow bootstrap [DATACMNS-1172]

[spring-projects-issues]

Réda Housni Alaoui opened DATACMNS-1172 and commented

We are using Spring Data JPA 1.10.11.RELEASE, therefore using Spring Data Commons 1.12.11.RELEASE. Integration test boot is particularly slow on our biggest application. After some digging, we found out that Spring Data ignition via CustomRepositoryImplementationDetector.detectCustomImplementation() takes 45 seconds.

I didn't find any issue opened on Spring Framework or Spring Data about this. I didn't notice a massive change of this part of the source code in the incoming Spring Data 2.0. This discourages our collaborators from writing integration test

---

Affects: 1.12.11 (Hopper SR11), 2.0 RC3 (Kay), 1.13.7 (Ingalls SR7)

Attachments:

• Screenshot from 2017-09-23 17-29-56.png (14.98 kB)
• Screenshot from 2017-09-23 17-30-28.png (15.46 kB)

Referenced from: pull request #248

Backported to: 1.13.8 (Ingalls SR8), 1.12.12 (Hopper SR12)

2 votes, 5 watchers

[spring-projects-issues]

Oliver Drotbohm commented

Do you have more information you can add as otherwise it's really hard to act on what you describe. Looks like the actual bottleneck is a class in Spring Framework. I'd be interested in how many repository interfaces your project contains, and whether they're located in packages with many sub-packages

[spring-projects-issues]

Réda Housni Alaoui commented

Hello Oliver,

The app has:

• 261 repositories
• a depth of 5 below the root package provided to Spring
• 365 jars in the final WAR

The scanned root package is present in a dozen of jars

[spring-projects-issues]

Réda Housni Alaoui commented

My laptop is a 2017 Razer Blade with nvme disk, 16 GB of RAM and Core i7 Quad Core 6700HQ running Ubuntu 17.
We have the issue on all dev machines

[spring-projects-issues]

Anthony Donnefort commented

According to the documentation:

If you use namespace configuration, the repository infrastructure tries to autodetect custom implementations by scanning for classes below the package we found a repository in.

This is not the case. CustomRepositoryImplementationDetector looks for the implementation anywhere in the base package.
We have a large codebase, looking for a single implementation that way takes hundreds of milliseconds. Multiply that by our 260 repositories, and you get startup times close to a minute.

Limiting the package search as documented would probably fix the problem

[spring-projects-issues]

Mark Paluch commented

I pushed a branch with a proposal that aligns scanning to how it's documented, we scan only for custom implementations in the package where the actual repository interface/fragment interface resides.

That's a change in comparison to how scanning works today. We need to extend our @Enable…Repositories annotations to allow configuration of the behavior:

@EnableJpaRepositories(limitImplementationBasePackages = false)
@Configuration
class AppConfiguration {
  // …
}

[spring-projects-issues]

Mark Paluch commented

We deployed snapshot versions for Hopper and Ingalls that limit implementation scanning to the repository package. Care to give the change a spin?

Hopper:

<dependency>
  <groupId>org.springframework.data</groupId>
  <artifactId>spring-data-commons</artifactId>
  <version>1.12.12.DATACMNS-1172-SNAPSHOT</version>
</dependency>

Ingalls:

<dependency>
  <groupId>org.springframework.data</groupId>
  <artifactId>spring-data-commons</artifactId>
  <version>1.13.8.DATACMNS-1172-SNAPSHOT</version>
</dependency>

You can obtain the artifacts from our snapshot repository:

<repositories>
  <repository>
    <id>spring-libs-snapshot</id>
    <url>https://repo.spring.io/libs-snapshot</url>
  </repository>
</repositories>

[spring-projects-issues]

Anthony Donnefort commented

On two different usecases init time for spring data with this snapshot went from 30s to 5s, and 1m30 to 20s.
That's awesome !

[spring-projects-issues]

Oliver Drotbohm commented

That's now merged into Ingalls and Hopper bugfix branches as well as master for Kay. I didn't merge the addition of a config option top enable exhaustive scanning again, to see whether users were really relying on that