You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/content/utilities/batch.mdx
+61-27
Original file line number
Diff line number
Diff line change
@@ -15,12 +15,17 @@ The SQS batch processing utility provides a way to handle partial failures when
15
15
16
16
**Background**
17
17
18
-
When using SQS as a Lambda event source mapping, functions can be triggered with a batch of messages from SQS. If the Lambda function fails when processing the batch,
19
-
all messages in the batch will be returned to the queue. With this utility, messages within a batch are handled individually - only messages that were not successfully processed
20
-
are returned to the queue. More details on how Lambda works with SQS can be found in the [AWS documentation](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html).
18
+
When using SQS as a Lambda event source mapping, Lambda functions are triggered with a batch of messages from SQS.
19
+
20
+
If your function fails to process any message from the batch, the entire batch returns to your SQS queue, and your Lambda function is triggered with the same batch one more time.
21
+
22
+
With this utility, messages within a batch are handled individually - only messages that were not successfully processed
23
+
are returned to the queue.
21
24
22
25
<Notetype="warning">
23
26
While this utility lowers the chance of processing messages more than once, it is not guaranteed. We recommend implementing processing logic in an idempotent manner wherever possible.
27
+
<br/><br/>
28
+
More details on how Lambda works with SQS can be found in the <ahref="https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html">AWS documentation</a>
24
29
</Note><br/>
25
30
26
31
@@ -30,14 +35,37 @@ This utility requires additional permissions to work as expected. Lambda functio
30
35
31
36
## Processing messages from SQS
32
37
33
-
There are 2 ways to use this utility for processing SQS messages:
38
+
You can use either **[sqs_batch_processor](#sqs_batch_processor-decorator)** decorator, or **[PartialSQSProcessor](#partialsqsprocessor-context-manager)** as a context manager.
39
+
40
+
They have nearly the same behaviour when it comes to processing messages from the batch:
41
+
42
+
***Entire batch has been successfully processed**, where your Lambda handler returned successfully, we will let SQS delete the batch to optimize your cost
43
+
***Entire Batch has been partially processed successfully**, where exceptions were raised within your `record handler`, we will:
44
+
-**1)** Delete successfully processed messages from the queue by directly calling `sqs:DeleteMessageBatch`
45
+
-**2)** Raise `SQSBatchProcessingError` to ensure failed messages return to your SQS queue
46
+
47
+
The only difference is that **PartialSQSProcessor** will give you access to processed messages if you need.
48
+
49
+
## Record Handler
50
+
51
+
Both decorator and context managers require an explicit function to process the batch of messages - namely `record_handler` parameter.
34
52
35
-
**With a decorator:**
53
+
This function is responsible for processing each individual message from the batch, and to raise an exception if unable to process any of the messages sent.
36
54
37
-
Using the `sqs_batch_processor` decorator with your lambda handler function, you provide a `record_handler` which is responsible for processing individual messages. It should raise an exception if
38
-
it is unable to process the record. All records in the batch will be passed to this handler for processing, even if exceptions are thrown. After all messages are processed, any successfully processed
39
-
ones will be deleted from the queue. If there were any messages the `record_handler` couldn't process, `SQSBatchProcessingError` will be raised. You will not have accessed to the _processed_ messages
40
-
within the lambda handler - all processing logic should be performed by the `record_handler` function.
55
+
**Any non-exception/successful return from your record handler function** will instruct both decorator and context manager to queue up each individual message for deletion.
56
+
57
+
### sqs_batch_processor decorator
58
+
59
+
When using the this decorator, you need provide a function via `record_handler` param that will process individual messages from the batch - It should raise an exception if it is unable to process the record.
60
+
61
+
All records in the batch will be passed to this handler for processing, even if exceptions are thrown - Here's the behaviour after completing the batch:
62
+
63
+
***Any successfully processed messages**, we will delete them from the queue via `sqs:DeleteMessageBatch`
64
+
***Any unprocessed messages detected**, we will raise `SQSBatchProcessingError` to ensure failed messages return to your SQS queue
65
+
66
+
<Notetype="warning">
67
+
You will not have accessed to the <strong>processed messages</strong> within the Lambda Handler - all processing logic will and should be performed by the <code>record_handler</code> function.
68
+
</Note><br/>
41
69
42
70
```python:title=app.py
43
71
from aws_lambda_powertools.utilities.batch import sqs_batch_processor
If you require access to the result of processed messages, you can use this context manager.
57
87
58
-
If you require access to the result of processed messages, you can use the context manager. The result from calling `process()` on the context manager will be a list of
59
-
all the return values from your `record_handler` function.
88
+
The result from calling `process()` on the context manager will be a list of all the return values from your `record_handler` function.
60
89
61
90
```python:title=app.py
62
91
from aws_lambda_powertools.utilities.batch import PartialSQSProcessor
If you want to disable the defualt behavior where `SQSBatchProcessingError` is raised if there are any errors, you can pass the `suppress_exception` argument.
162
+
If you want to disable the default behavior where `SQSBatchProcessingError` is raised if there are any errors, you can pass the `suppress_exception` boolean argument.
134
163
135
-
<Notetype="warning">
136
-
If your Lambda function executes successfully and returns a response, all messages in the batch will be deleted from the queue.
@@ -154,7 +183,9 @@ with processor(records, record_handler):
154
183
155
184
You can create your own partial batch processor by inheriting the `BasePartialProcessor` class, and implementing `_prepare()`, `_clean()` and `_process_record()`.
156
185
157
-
All processing logic is handled by `_process_record()` whilst `_prepare()` and `clean()` take care of doing a setup/teardown of the processor, being called at start/end of processor's execution, respectively.
186
+
***`_process_record()`** - Handles all processing logic for each individual message of a batch, including calling the `record_handler` (self.handler)
187
+
***`_prepare()`** - Called once as part of the processor initialization
188
+
***`clean()`** - Teardown logic called once after `_process_record` completes
158
189
159
190
You can then use this class as a context manager, or pass it to `batch_processor` to use as a decorator on your Lambda handler function.
160
191
@@ -165,19 +196,18 @@ from random import randint
165
196
166
197
from aws_lambda_powertools.utilities.batch import BasePartialProcessor, batch_processor
0 commit comments