Skip to content

Commit bf2950c

Browse files
mhalbritterphilwebb
mhalbritter
and
philwebb
committed
Add support for structured logging
Update Logback and Log4j2 integrations to support structured logging. Support for the ECS and Logstash JSON formats is provided out-of-the-box and the `StructuredLogFormatter` interface may be used to if further custom formats need to be supported. Closes gh-5479 Co-authored-by: Phillip Webb <[email protected]>
1 parent 89f3052 commit bf2950c

File tree

47 files changed

+2699
-26
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

47 files changed

+2699
-26
lines changed

‎spring-boot-project/spring-boot-docs/src/docs/antora/modules/how-to/pages/logging.adoc

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,9 @@ The following files are provided under `org/springframework/boot/logging/logback
5151

5252
* `defaults.xml` - Provides conversion rules, pattern properties and common logger configurations.
5353
* `console-appender.xml` - Adds a `ConsoleAppender` using the `CONSOLE_LOG_PATTERN`.
54+
* `structured-console-appender.xml` - Adds a `ConsoleAppender` using structured logging in the `CONSOLE_LOG_STRUCTURED_FORMAT`.
5455
* `file-appender.xml` - Adds a `RollingFileAppender` using the `FILE_LOG_PATTERN` and `ROLLING_FILE_NAME_PATTERN` with appropriate settings.
56+
* `structured-file-appender.xml` - Adds a `RollingFileAppender` using the `ROLLING_FILE_NAME_PATTERN` with structured logging in the `FILE_LOG_STRUCTURED_FORMAT`.
5557

5658
In addition, a legacy `base.xml` file is provided for compatibility with earlier versions of Spring Boot.
5759

‎spring-boot-project/spring-boot-docs/src/docs/antora/modules/reference/pages/features/logging.adoc

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -366,6 +366,14 @@ The properties that are transferred are described in the following table:
366366
| `LOG_LEVEL_PATTERN`
367367
| The format to use when rendering the log level (default `%5p`).
368368

369+
| configprop:logging.structured.format.console[]
370+
| `CONSOLE_LOG_STRUCTURED_FORMAT`
371+
| The structured logging format to use for console logging.
372+
373+
| configprop:logging.structured.format.file[]
374+
| `FILE_LOG_STRUCTURED_FORMAT`
375+
| The structured logging format to use for file logging.
376+
369377
| `PID`
370378
| `PID`
371379
| The current process ID (discovered if possible and when not already defined as an OS environment variable).
@@ -425,6 +433,94 @@ Handling authenticated request
425433

426434

427435

436+
[[features.logging.structured]]
437+
== Structured Logging
438+
439+
Structured logging is a technique where the log output is written in a well-defined, often machine-readable format.
440+
Spring Boot supports structured logging and has support for the following formats out of the box:
441+
442+
* xref:#features.logging.structured.ecs[Elastic Common Schema (ECS)]
443+
* xref:#features.logging.structured.logstash[Logstash]
444+
445+
To enable structured logging, set the property configprop:logging.structured.format.console[] (for console output) or configprop:logging.structured.format.file[] (for file output) to the id of the format you want to use.
446+
447+
448+
449+
[[features.logging.structured.ecs]]
450+
=== Elastic Common Schema
451+
https://www.elastic.co/guide/en/ecs/8.11/ecs-reference.html[Elastic Common Schema] is a JSON based logging format.
452+
453+
To enable the Elastic Common Schema log format, set the appropriate `format` property to `ecs`:
454+
455+
[configprops,yaml]
456+
----
457+
logging:
458+
structured:
459+
format:
460+
console: ecs
461+
file: ecs
462+
----
463+
464+
A log line looks like this:
465+
466+
[source,json]
467+
----
468+
{"@timestamp":"2024-01-01T10:15:00.067462556Z","log.level":"INFO","process.pid":39599,"process.thread.name":"main","service.name":"simple","log.logger":"org.example.Application","message":"No active profile set, falling back to 1 default profile: \"default\"","ecs.version":"8.11"}
469+
----
470+
471+
This format also adds every key value pair contained in the MDC to the JSON object.
472+
You can also use the https://www.slf4j.org/manual.html#fluent[SLF4J fluent logging API] to add key value pairs to the logged JSON object with the https://www.slf4j.org/apidocs/org/slf4j/spi/LoggingEventBuilder.html#addKeyValue(java.lang.String,java.lang.Object)[addKeyValue] method.
473+
474+
475+
476+
[[features.logging.structured.logstash]]
477+
=== Logstash JSON format
478+
479+
The https://github.com/logfellow/logstash-logback-encoder?tab=readme-ov-file#standard-fields[Logstash JSON format] is a JSON based logging format.
480+
481+
To enable the Logstash JSON log format, set the appropriate `format` property to `logstash`:
482+
483+
[configprops,yaml]
484+
----
485+
logging:
486+
structured:
487+
format:
488+
console: logstash
489+
file: logstash
490+
----
491+
492+
A log line looks like this:
493+
494+
[source,json]
495+
----
496+
{"@timestamp":"2024-01-01T10:15:00.111037681+02:00","@version":"1","message":"No active profile set, falling back to 1 default profile: \"default\"","logger_name":"org.example.Application","thread_name":"main","level":"INFO","level_value":20000}
497+
----
498+
499+
This format also adds every key value pair contained in the MDC to the JSON object.
500+
You can also use the https://www.slf4j.org/manual.html#fluent[SLF4J fluent logging API] to add key value pairs to the logged JSON object with the https://www.slf4j.org/apidocs/org/slf4j/spi/LoggingEventBuilder.html#addKeyValue(java.lang.String,java.lang.Object)[addKeyValue] method.
501+
502+
If you add https://www.slf4j.org/api/org/slf4j/Marker.html[markers], these will show up in a `tags` string array in the JSON.
503+
504+
505+
506+
[[features.logging.structured.custom-format]]
507+
=== Custom Structured Logging formats
508+
509+
The structured logging support in Spring Boot is extensible, allowing you to define your own custom format.
510+
To do this, implement the `StructuredLoggingFormatter` interface. The generic type argument has to be `ILoggingEvent` when using Logback and `LogEvent` when using Log4j2 (that means your implementation is tied to a specific logging system).
511+
Your implementation is then called with the log event and returns the `String` to be logged, as seen in this example:
512+
513+
include-code::MyCustomFormat[]
514+
515+
As you can see in the example, you can return any format, it doesn't have to be JSON.
516+
517+
To enable your custom format, set the property configprop:logging.structured.format.console[] or configprop:logging.structured.format.file[] to the fully qualified class name of your implementation.
518+
519+
Your implementation can use some constructor parameters, which are injected automatically.
520+
Please see the JavaDoc of xref:api:java/org/springframework/boot/logging/structured/StructuredLogFormatter.html[`StructuredLogFormatter`] for more details.
521+
522+
523+
428524
[[features.logging.logback-extensions]]
429525
== Logback Extensions
430526

‎spring-boot-project/spring-boot-docs/src/main/java/org/springframework/boot/docs/features/logging/structured/customformat/MyCustomFormat.java

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
/*
2+
* Copyright 2012-2024 the original author or authors.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* https://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
package org.springframework.boot.docs.features.logging.structured.customformat;
18+
19+
import ch.qos.logback.classic.spi.ILoggingEvent;
20+
21+
import org.springframework.boot.logging.structured.StructuredLogFormatter;
22+
23+
class MyCustomFormat implements StructuredLogFormatter<ILoggingEvent> {
24+
25+
@Override
26+
public String format(ILoggingEvent event) {
27+
return "time=" + event.getInstant() + " level=" + event.getLevel() + " message=" + event.getMessage() + "\n";
28+
}
29+
30+
}

‎spring-boot-project/spring-boot/src/main/java/org/springframework/boot/logging/LoggingSystemProperties.java

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -127,6 +127,8 @@ protected void apply(LogFile logFile, PropertyResolver resolver) {
127127
setSystemProperty(LoggingSystemProperty.EXCEPTION_CONVERSION_WORD, resolver);
128128
setSystemProperty(LoggingSystemProperty.CONSOLE_PATTERN, resolver);
129129
setSystemProperty(LoggingSystemProperty.FILE_PATTERN, resolver);
130+
setSystemProperty(LoggingSystemProperty.CONSOLE_STRUCTURED_FORMAT, resolver);
131+
setSystemProperty(LoggingSystemProperty.FILE_STRUCTURED_FORMAT, resolver);
130132
setSystemProperty(LoggingSystemProperty.LEVEL_PATTERN, resolver);
131133
setSystemProperty(LoggingSystemProperty.DATEFORMAT_PATTERN, resolver);
132134
setSystemProperty(LoggingSystemProperty.CORRELATION_PATTERN, resolver);

‎spring-boot-project/spring-boot/src/main/java/org/springframework/boot/logging/LoggingSystemProperty.java

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,18 @@ public enum LoggingSystemProperty {
8585
*/
8686
FILE_PATTERN("FILE_LOG_PATTERN", "logging.pattern.file"),
8787

88+
/**
89+
* Logging system property for the console structured logging format.
90+
* @since 3.4.0
91+
*/
92+
CONSOLE_STRUCTURED_FORMAT("CONSOLE_LOG_STRUCTURED_FORMAT", "logging.structured.format.console"),
93+
94+
/**
95+
* Logging system property for the file structured logging format.
96+
* @since 3.4.0
97+
*/
98+
FILE_STRUCTURED_FORMAT("FILE_LOG_STRUCTURED_FORMAT", "logging.structured.format.file"),
99+
88100
/**
89101
* Logging system property for the log level pattern.
90102
*/

‎spring-boot-project/spring-boot/src/main/java/org/springframework/boot/logging/log4j2/ElasticCommonSchemaStructuredLogFormatter.java

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
/*
2+
* Copyright 2012-2024 the original author or authors.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* https://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
package org.springframework.boot.logging.log4j2;
18+
19+
import org.apache.logging.log4j.Level;
20+
import org.apache.logging.log4j.core.LogEvent;
21+
import org.apache.logging.log4j.core.impl.ThrowableProxy;
22+
import org.apache.logging.log4j.core.time.Instant;
23+
import org.apache.logging.log4j.message.Message;
24+
import org.apache.logging.log4j.util.ReadOnlyStringMap;
25+
26+
import org.springframework.boot.json.JsonWriter;
27+
import org.springframework.boot.logging.structured.ApplicationMetadata;
28+
import org.springframework.boot.logging.structured.CommonStructuredLogFormat;
29+
import org.springframework.boot.logging.structured.StructuredLogFormatter;
30+
import org.springframework.util.ObjectUtils;
31+
32+
/**
33+
* Log4j2 {@link StructuredLogFormatter} for
34+
* {@link CommonStructuredLogFormat#ELASTIC_COMMON_SCHEMA}.
35+
*
36+
* @author Moritz Halbritter
37+
* @author Phillip Webb
38+
*/
39+
class ElasticCommonSchemaStructuredLogFormatter implements StructuredLogFormatter<LogEvent> {
40+
41+
private final JsonWriter<LogEvent> writer;
42+
43+
ElasticCommonSchemaStructuredLogFormatter(ApplicationMetadata metadata) {
44+
this.writer = JsonWriter.<LogEvent>of((members) -> logEventJson(metadata, members)).withNewLineAtEnd();
45+
}
46+
47+
private void logEventJson(ApplicationMetadata metadata, JsonWriter.Members<LogEvent> members) {
48+
members.add("@timestamp", LogEvent::getInstant).as(this::asTimestamp);
49+
members.add("log.level", LogEvent::getLevel).as(Level::name);
50+
members.add("process.pid", metadata::pid).whenNotNull();
51+
members.add("process.thread.name", LogEvent::getThreadName);
52+
members.add("service.name", metadata::name).whenHasLength();
53+
members.add("service.version", metadata::version).whenHasLength();
54+
members.add("service.environment", metadata::environment).whenHasLength();
55+
members.add("service.node.name", metadata::nodeName).whenHasLength();
56+
members.add("log.logger", LogEvent::getLoggerName);
57+
members.add("message", LogEvent::getMessage).as(Message::getFormattedMessage);
58+
members.add(LogEvent::getContextData)
59+
.whenNot(ReadOnlyStringMap::isEmpty)
60+
.usingPairs((contextData, pairs) -> contextData.forEach(pairs::accept));
61+
members.add(LogEvent::getThrownProxy).whenNotNull().usingMembers((thrownProxyMembers) -> {
62+
thrownProxyMembers.add("error.type", ThrowableProxy::getThrowable)
63+
.whenNotNull()
64+
.as(ObjectUtils::nullSafeClassName);
65+
thrownProxyMembers.add("error.message", ThrowableProxy::getMessage);
66+
thrownProxyMembers.add("error.stack_trace", ThrowableProxy::getExtendedStackTraceAsString);
67+
});
68+
members.add("ecs.version", "8.11");
69+
}
70+
71+
private java.time.Instant asTimestamp(Instant instant) {
72+
return java.time.Instant.ofEpochMilli(instant.getEpochMillisecond()).plusNanos(instant.getNanoOfMillisecond());
73+
}
74+
75+
@Override
76+
public String format(LogEvent event) {
77+
return this.writer.writeToString(event);
78+
}
79+
80+
}

‎spring-boot-project/spring-boot/src/main/java/org/springframework/boot/logging/log4j2/LogstashStructuredLogFormatter.java

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
/*
2+
* Copyright 2012-2024 the original author or authors.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* https://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
package org.springframework.boot.logging.log4j2;
18+
19+
import java.time.OffsetDateTime;
20+
import java.time.ZoneId;
21+
import java.time.format.DateTimeFormatter;
22+
import java.util.Set;
23+
import java.util.TreeSet;
24+
25+
import org.apache.logging.log4j.Level;
26+
import org.apache.logging.log4j.Marker;
27+
import org.apache.logging.log4j.core.LogEvent;
28+
import org.apache.logging.log4j.core.impl.ThrowableProxy;
29+
import org.apache.logging.log4j.core.time.Instant;
30+
import org.apache.logging.log4j.message.Message;
31+
import org.apache.logging.log4j.util.ReadOnlyStringMap;
32+
33+
import org.springframework.boot.json.JsonWriter;
34+
import org.springframework.boot.logging.structured.CommonStructuredLogFormat;
35+
import org.springframework.boot.logging.structured.StructuredLogFormatter;
36+
import org.springframework.util.CollectionUtils;
37+
38+
/**
39+
* Log4j2 {@link StructuredLogFormatter} for {@link CommonStructuredLogFormat#LOGSTASH}.
40+
*
41+
* @author Moritz Halbritter
42+
* @author Phillip Webb
43+
*/
44+
class LogstashStructuredLogFormatter implements StructuredLogFormatter<LogEvent> {
45+
46+
private JsonWriter<LogEvent> writer;
47+
48+
LogstashStructuredLogFormatter() {
49+
this.writer = JsonWriter.<LogEvent>of(this::logEventJson).withNewLineAtEnd();
50+
}
51+
52+
private void logEventJson(JsonWriter.Members<LogEvent> members) {
53+
members.add("@timestamp", LogEvent::getInstant).as(this::asTimestamp);
54+
members.add("@version", "1");
55+
members.add("message", LogEvent::getMessage).as(Message::getFormattedMessage);
56+
members.add("logger_name", LogEvent::getLoggerName);
57+
members.add("thread_name", LogEvent::getThreadName);
58+
members.add("level", LogEvent::getLevel).as(Level::name);
59+
members.add("level_value", LogEvent::getLevel).as(Level::intLevel);
60+
members.add(LogEvent::getContextData)
61+
.whenNot(ReadOnlyStringMap::isEmpty)
62+
.usingPairs((contextData, pairs) -> contextData.forEach(pairs::accept));
63+
members.add("tags", LogEvent::getMarker).whenNotNull().as(this::getMarkers).whenNot(CollectionUtils::isEmpty);
64+
members.add("stack_trace", LogEvent::getThrownProxy)
65+
.whenNotNull()
66+
.as(ThrowableProxy::getExtendedStackTraceAsString);
67+
}
68+
69+
private String asTimestamp(Instant instant) {
70+
java.time.Instant javaInstant = java.time.Instant.ofEpochMilli(instant.getEpochMillisecond())
71+
.plusNanos(instant.getNanoOfMillisecond());
72+
OffsetDateTime offsetDateTime = OffsetDateTime.ofInstant(javaInstant, ZoneId.systemDefault());
73+
return DateTimeFormatter.ISO_OFFSET_DATE_TIME.format(offsetDateTime);
74+
}
75+
76+
private Set<String> getMarkers(Marker marker) {
77+
Set<String> result = new TreeSet<>();
78+
addMarkers(result, marker);
79+
return result;
80+
}
81+
82+
private void addMarkers(Set<String> result, Marker marker) {
83+
result.add(marker.getName());
84+
if (marker.hasParents()) {
85+
for (Marker parent : marker.getParents()) {
86+
addMarkers(result, parent);
87+
}
88+
}
89+
}
90+
91+
@Override
92+
public String format(LogEvent event) {
93+
return this.writer.writeToString(event);
94+
}
95+
96+
}

0 commit comments

Comments
 (0)