Skip to content

Commit dfbe15e

Browse files
jeromevdljeromevdl
committed
documentation + migration guide
1 parent d3683b2 commit dfbe15e

File tree

3 files changed

+388
-57
lines changed

3 files changed

+388
-57
lines changed

‎docs/utilities/large_messages.md

+322
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,322 @@
1+
---
2+
title: Large Messages
3+
description: Utility
4+
---
5+
6+
The large message utility handles SQS and SNS messages which have had their payloads
7+
offloaded to S3 if they are larger than the maximum allowed size (256 KB).
8+
9+
!!! Notice
10+
The large message utility (available in the `powertools-batch` module with v1.16.1 or lower) is now deprecated
11+
and replaced by the `powertools-large-messages` described in this page.
12+
You can still get the documentation [here](sqs_large_message_handling.md)
13+
and the migration guide [here](#migration-from-the-sqs-large-message-utility).
14+
15+
## Features
16+
17+
- Automatically retrieve the content of S3 objects when SQS or SNS messages have been offloaded to S3.
18+
- Automatically delete the S3 Objects after processing is OK.
19+
- Compatible with the batch module (with SQS).
20+
21+
## Background
22+
23+
SQS and SNS message payload is limited to 256KB. If you wish to send larger message payload, you can leverage the
24+
[amazon-sqs-java-extended-client-lib](https://github.com/awslabs/amazon-sqs-java-extended-client-lib)
25+
or [amazon-sns-java-extended-client-lib](https://github.com/awslabs/amazon-sns-java-extended-client-lib) which
26+
offload the message to Amazon S3. See documentation
27+
([SQS](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-s3-messages.html)
28+
/ [SNS](https://docs.aws.amazon.com/sns/latest/dg/large-message-payloads.html))
29+
30+
When offloaded to S3, the message contains a specific message attribute and the payload only contains a pointer to the
31+
S3 object (bucket and object key).
32+
33+
This utility automatically retrieves messages which have been offloaded to S3 using the
34+
extended client libraries. Once the message payloads have been processed successful, the
35+
utility also delete the message payloads from S3.
36+
37+
This utility is compatible with
38+
versions *[1.1.0+](https://github.com/awslabs/amazon-sqs-java-extended-client-lib/releases/tag/1.1.0)*
39+
of amazon-sqs-java-extended-client-lib
40+
and *[1.0.0+](https://github.com/awslabs/amazon-sns-java-extended-client-lib/releases/tag/1.0.0)*
41+
of amazon-sns-java-extended-client-lib.
42+
43+
## Install
44+
45+
Depending on your version of Java (either Java 1.8 or 11+), the configuration slightly changes.
46+
47+
=== "Maven Java 11+"
48+
```xml hl_lines="3-7 16 18 24-27"
49+
<dependencies>
50+
...
51+
<dependency>
52+
<groupId>software.amazon.lambda</groupId>
53+
<artifactId>powertools-large-messages</artifactId>
54+
<version>{{ powertools.version }}</version>
55+
</dependency>
56+
...
57+
</dependencies>
58+
...
59+
<!-- configure the aspectj-maven-plugin to compile-time weave (CTW) the aws-lambda-powertools-java aspects into your project -->
60+
<build>
61+
<plugins>
62+
...
63+
<plugin>
64+
<groupId>dev.aspectj</groupId>
65+
<artifactId>aspectj-maven-plugin</artifactId>
66+
<version>1.13.1</version>
67+
<configuration>
68+
<source>11</source> <!-- or higher -->
69+
<target>11</target> <!-- or higher -->
70+
<complianceLevel>11</complianceLevel> <!-- or higher -->
71+
<aspectLibraries>
72+
<aspectLibrary>
73+
<groupId>software.amazon.lambda</groupId>
74+
<artifactId>powertools-large-messages</artifactId>
75+
</aspectLibrary>
76+
</aspectLibraries>
77+
</configuration>
78+
<executions>
79+
<execution>
80+
<goals>
81+
<goal>compile</goal>
82+
</goals>
83+
</execution>
84+
</executions>
85+
</plugin>
86+
...
87+
</plugins>
88+
</build>
89+
```
90+
91+
=== "Maven Java 1.8"
92+
93+
```xml hl_lines="3-7 16 18 24-27"
94+
<dependencies>
95+
...
96+
<dependency>
97+
<groupId>software.amazon.lambda</groupId>
98+
<artifactId>powertools-large-messages</artifactId>
99+
<version>{{ powertools.version }}</version>
100+
</dependency>
101+
...
102+
</dependencies>
103+
...
104+
<!-- configure the aspectj-maven-plugin to compile-time weave (CTW) the aws-lambda-powertools-java aspects into your project -->
105+
<build>
106+
<plugins>
107+
...
108+
<plugin>
109+
<groupId>org.codehaus.mojo</groupId>
110+
<artifactId>aspectj-maven-plugin</artifactId>
111+
<version>1.14.0</version>
112+
<configuration>
113+
<source>1.8</source>
114+
<target>1.8</target>
115+
<complianceLevel>1.8</complianceLevel>
116+
<aspectLibraries>
117+
<aspectLibrary>
118+
<groupId>software.amazon.lambda</groupId>
119+
<artifactId>powertools-large-messages</artifactId>
120+
</aspectLibrary>
121+
</aspectLibraries>
122+
</configuration>
123+
<executions>
124+
<execution>
125+
<goals>
126+
<goal>compile</goal>
127+
</goals>
128+
</execution>
129+
</executions>
130+
</plugin>
131+
...
132+
</plugins>
133+
</build>
134+
```
135+
136+
=== "Gradle Java 11+"
137+
138+
```groovy hl_lines="3 11"
139+
plugins {
140+
id 'java'
141+
id 'io.freefair.aspectj.post-compile-weaving' version '8.1.0'
142+
}
143+
144+
repositories {
145+
mavenCentral()
146+
}
147+
148+
dependencies {
149+
aspect 'software.amazon.lambda:powertools-large-messages:{{ powertools.version }}'
150+
}
151+
152+
sourceCompatibility = 11 // or higher
153+
targetCompatibility = 11 // or higher
154+
```
155+
156+
=== "Gradle Java 1.8"
157+
158+
```groovy hl_lines="3 11"
159+
plugins {
160+
id 'java'
161+
id 'io.freefair.aspectj.post-compile-weaving' version '6.6.3'
162+
}
163+
164+
repositories {
165+
mavenCentral()
166+
}
167+
168+
dependencies {
169+
aspect 'software.amazon.lambda:powertools-large-messages:{{ powertools.version }}'
170+
}
171+
172+
sourceCompatibility = 1.8
173+
targetCompatibility = 1.8
174+
```
175+
176+
## Permissions
177+
178+
As the utility interacts with Amazon S3, the lambda function must have the following permissions
179+
on the S3 bucket used for the large messages offloading:
180+
181+
- `s3:GetObject`
182+
- `s3:DeleteObject`
183+
184+
## Annotation
185+
186+
The annotation `@LargeMessage` can be used on any method where the *first* parameter is one of:
187+
188+
- `SQSEvent.SQSMessage`
189+
- `SNSEvent.SNSRecord`
190+
191+
=== "SQS Example"
192+
193+
```java hl_lines="8 13 15"
194+
import software.amazon.lambda.powertools.largemessages.LargeMessage;
195+
196+
public class SqsMessageHandler implements RequestHandler<SQSEvent, SQSBatchResponse> {
197+
198+
@Override
199+
public SQSBatchResponse handleRequest(SQSEvent event, Context context) {
200+
for (SQSMessage message: event.getRecords()) {
201+
processRawMessage(message, context);
202+
}
203+
return SQSBatchResponse.builder().build();
204+
}
205+
206+
@LargeMessage
207+
private void processRawMessage(SQSEvent.SQSMessage sqsMessage, Context context) {
208+
// sqsMessage.getBody() will contain the content of the S3 object
209+
}
210+
}
211+
```
212+
213+
=== "SNS Example"
214+
215+
```java hl_lines="7 11 13"
216+
import software.amazon.lambda.powertools.largemessages.LargeMessage;
217+
218+
public class SnsRecordHandler implements RequestHandler<SNSEvent, String> {
219+
220+
@Override
221+
public String handleRequest(SNSEvent event, Context context) {
222+
processSNSRecord(event.records.get(0)); // there are always only one message
223+
return "Hello World";
224+
}
225+
226+
@LargeMessage
227+
private void processSNSRecord(SNSEvent.SNSRecord snsRecord) {
228+
// snsRecord.getSNS().getMessage() will contain the content of the S3 object
229+
}
230+
}
231+
```
232+
233+
When the Lambda function is invoked with a SQS or SNS event, the utility first
234+
checks if the content was offloaded to S3. In the case of a large message, there is a message attribute
235+
specifying the size of the offloaded message and the message contains a pointer to the S3 object.
236+
237+
If this is the case, the utility will retrieve the object from S3 using the `getObject(bucket, key)` API,
238+
and place the content of the object in the message payload. You can then directly use the content of the message.
239+
If there was an error during the S3 download, the function will fail with a `LargeMessageProcessingException`.
240+
241+
After your code is invoked and returns without error, the object is deleted from S3
242+
using the `deleteObject(bucket, key)` API. You can disable the deletion of S3 objects with the following configuration:
243+
244+
=== "Don't delete S3 Objects"
245+
```java
246+
@LargeMessage(deleteS3Object = false)
247+
private void processRawMessage(SQSEvent.SQSMessage sqsMessage) {
248+
// do something with the message
249+
}
250+
```
251+
252+
!!! tip
253+
This utility works perfectly together with the batch module (`powertools-batch`), especially for SQS:
254+
255+
```java hl_lines="2 5-7 12 15 16" title="Combining batch and large message modules"
256+
public class SqsBatchHandler implements RequestHandler<SQSEvent, SQSBatchResponse> {
257+
private final BatchMessageHandler<SQSEvent, SQSBatchResponse> handler;
258+
259+
public SqsBatchHandler() {
260+
handler = new BatchMessageHandlerBuilder()
261+
.withSqsBatchHandler()
262+
.buildWithRawMessageHandler(this::processMessage);
263+
}
264+
265+
@Override
266+
public SQSBatchResponse handleRequest(SQSEvent sqsEvent, Context context) {
267+
return handler.processBatch(sqsEvent, context);
268+
}
269+
270+
@LargeMessage
271+
private void processMessage(SQSEvent.SQSMessage sqsMessage) {
272+
// do something with the message
273+
}
274+
}
275+
```
276+
277+
## Customizing S3 client configuration
278+
279+
To interact with S3, the utility creates a default S3 Client :
280+
281+
=== "Default S3 Client"
282+
```java
283+
S3Client client = S3Client.builder()
284+
.httpClient(UrlConnectionHttpClient.builder().build())
285+
.region(Region.of(System.getenv(AWS_REGION_ENV)))
286+
.build();
287+
```
288+
289+
If you need to customize this `S3Client`, you can leverage the `LargeMessageConfig` singleton:
290+
291+
=== "Custom S3 Client"
292+
```java hl_lines="6"
293+
import software.amazon.lambda.powertools.largemessages.LargeMessage;
294+
295+
public class SnsRecordHandler implements RequestHandler<SNSEvent, String> {
296+
297+
public SnsRecordHandler() {
298+
LargeMessageConfig.init().withS3Client(/* put your custom S3Client here */);
299+
}
300+
301+
@Override
302+
public String handleRequest(SNSEvent event, Context context) {
303+
processSNSRecord(event.records.get(0));
304+
return "Hello World";
305+
}
306+
307+
@LargeMessage
308+
private void processSNSRecord(SNSEvent.SNSRecord snsRecord) {
309+
// snsRecord.getSNS().getMessage() will contain the content of the S3 object
310+
}
311+
}
312+
```
313+
314+
## Migration from the SQS Large Message utility
315+
316+
- Replace the dependency in maven / gradle: `powertools-sqs` ==> `powertools-large-messages`
317+
- Replace the annotation: `@SqsLargeMessage` ==> `@LargeMessage` (the new module handles both SQS and SNS)
318+
- Move the annotation away from the Lambda `handleRequest` method and put it on a method with `SQSEvent.SQSMessage`
319+
or `SNSEvent.SNSRecord` as first parameter.
320+
- The annotation now handles a single message, contrary to the previous version that was handling the complete batch.
321+
It gives more control, especially when dealing with partial failures with SQS (see the batch module).
322+
- The new module only provides an annotation, the `SqsUtils` is not available anymore in this new version.

0 commit comments

Comments
 (0)