Skip to content

Commit 285054d

Browse files
committed
docs: address readability feedbacks
1 parent 592b5a8 commit 285054d

File tree

1 file changed

+61
-27
lines changed

1 file changed

+61
-27
lines changed

docs/content/utilities/batch.mdx

+61-27
Original file line numberDiff line numberDiff line change
@@ -15,12 +15,17 @@ The SQS batch processing utility provides a way to handle partial failures when
1515

1616
**Background**
1717

18-
When using SQS as a Lambda event source mapping, functions can be triggered with a batch of messages from SQS. If the Lambda function fails when processing the batch,
19-
all messages in the batch will be returned to the queue. With this utility, messages within a batch are handled individually - only messages that were not successfully processed
20-
are returned to the queue. More details on how Lambda works with SQS can be found in the [AWS documentation](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html).
18+
When using SQS as a Lambda event source mapping, Lambda functions are triggered with a batch of messages from SQS.
19+
20+
If your function fails to process any message from the batch, the entire batch returns to your SQS queue, and your Lambda function is triggered with the same batch one more time.
21+
22+
With this utility, messages within a batch are handled individually - only messages that were not successfully processed
23+
are returned to the queue.
2124

2225
<Note type="warning">
2326
While this utility lowers the chance of processing messages more than once, it is not guaranteed. We recommend implementing processing logic in an idempotent manner wherever possible.
27+
<br/><br/>
28+
More details on how Lambda works with SQS can be found in the <a href="https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html">AWS documentation</a>
2429
</Note><br/>
2530

2631

@@ -30,14 +35,37 @@ This utility requires additional permissions to work as expected. Lambda functio
3035

3136
## Processing messages from SQS
3237

33-
There are 2 ways to use this utility for processing SQS messages:
38+
You can use either **[sqs_batch_processor](#sqs_batch_processor-decorator)** decorator, or **[PartialSQSProcessor](#partialsqsprocessor-context-manager)** as a context manager.
39+
40+
They have nearly the same behaviour when it comes to processing messages from the batch:
41+
42+
* **Entire batch has been successfully processed**, where your Lambda handler returned successfully, we will let SQS delete the batch to optimize your cost
43+
* **Entire Batch has been partially processed successfully**, where exceptions were raised within your `record handler`, we will:
44+
- **1)** Delete successfully processed messages from the queue by directly calling `sqs:DeleteMessageBatch`
45+
- **2)** Raise `SQSBatchProcessingError` to ensure failed messages return to your SQS queue
46+
47+
The only difference is that **PartialSQSProcessor** will give you access to processed messages if you need.
48+
49+
## Record Handler
50+
51+
Both decorator and context managers require an explicit function to process the batch of messages - namely `record_handler` parameter.
3452

35-
**With a decorator:**
53+
This function is responsible for processing each individual message from the batch, and to raise an exception if unable to process any of the messages sent.
3654

37-
Using the `sqs_batch_processor` decorator with your lambda handler function, you provide a `record_handler` which is responsible for processing individual messages. It should raise an exception if
38-
it is unable to process the record. All records in the batch will be passed to this handler for processing, even if exceptions are thrown. After all messages are processed, any successfully processed
39-
ones will be deleted from the queue. If there were any messages the `record_handler` couldn't process, `SQSBatchProcessingError` will be raised. You will not have accessed to the _processed_ messages
40-
within the lambda handler - all processing logic should be performed by the `record_handler` function.
55+
**Any non-exception/successful return from your record handler function** will instruct both decorator and context manager to queue up each individual message for deletion.
56+
57+
### sqs_batch_processor decorator
58+
59+
When using the this decorator, you need provide a function via `record_handler` param that will process individual messages from the batch - It should raise an exception if it is unable to process the record.
60+
61+
All records in the batch will be passed to this handler for processing, even if exceptions are thrown - Here's the behaviour after completing the batch:
62+
63+
* **Any successfully processed messages**, we will delete them from the queue via `sqs:DeleteMessageBatch`
64+
* **Any unprocessed messages detected**, we will raise `SQSBatchProcessingError` to ensure failed messages return to your SQS queue
65+
66+
<Note type="warning">
67+
You will not have accessed to the <strong>processed messages</strong> within the Lambda Handler - all processing logic will and should be performed by the <code>record_handler</code> function.
68+
</Note><br/>
4169

4270
```python:title=app.py
4371
from aws_lambda_powertools.utilities.batch import sqs_batch_processor
@@ -53,10 +81,11 @@ def lambda_handler(event, context):
5381
return {"statusCode": 200}
5482
```
5583

56-
**With a context manager:**
84+
### PartialSQSProcessor context manager
85+
86+
If you require access to the result of processed messages, you can use this context manager.
5787

58-
If you require access to the result of processed messages, you can use the context manager. The result from calling `process()` on the context manager will be a list of
59-
all the return values from your `record_handler` function.
88+
The result from calling `process()` on the context manager will be a list of all the return values from your `record_handler` function.
6089

6190
```python:title=app.py
6291
from aws_lambda_powertools.utilities.batch import PartialSQSProcessor
@@ -73,8 +102,8 @@ def lambda_handler(event, context):
73102

74103
processor = PartialSQSProcessor()
75104

76-
with processor(records, record_handler):
77-
result = processor.process() # Returns a list of all results from record_handler
105+
with processor(records, record_handler) as proc:
106+
result = proc.process() # Returns a list of all results from record_handler
78107

79108
return result
80109
```
@@ -130,19 +159,19 @@ def lambda_handler(event, context):
130159

131160
## Suppressing exceptions
132161

133-
If you want to disable the defualt behavior where `SQSBatchProcessingError` is raised if there are any errors, you can pass the `suppress_exception` argument.
162+
If you want to disable the default behavior where `SQSBatchProcessingError` is raised if there are any errors, you can pass the `suppress_exception` boolean argument.
134163

135-
<Note type="warning">
136-
If your Lambda function executes successfully and returns a response, all messages in the batch will be deleted from the queue.
137-
</Note><br/>
164+
**Within the decorator**
138165

139166
```python:title=app.py
140167
...
141168
@sqs_batch_processor(record_handler=record_handler, config=config, suppress_exception=True) # highlight-line
142169
def lambda_handler(event, context):
143170
return {"statusCode": 200}
144171
```
145-
or
172+
173+
**Within the context manager**
174+
146175
```python:title=app.py
147176
processor = PartialSQSProcessor(config=config, suppress_exception=True) # highlight-line
148177

@@ -154,7 +183,9 @@ with processor(records, record_handler):
154183

155184
You can create your own partial batch processor by inheriting the `BasePartialProcessor` class, and implementing `_prepare()`, `_clean()` and `_process_record()`.
156185

157-
All processing logic is handled by `_process_record()` whilst `_prepare()` and `clean()` take care of doing a setup/teardown of the processor, being called at start/end of processor's execution, respectively.
186+
* **`_process_record()`** - Handles all processing logic for each individual message of a batch, including calling the `record_handler` (self.handler)
187+
* **`_prepare()`** - Called once as part of the processor initialization
188+
* **`clean()`** - Teardown logic called once after `_process_record` completes
158189

159190
You can then use this class as a context manager, or pass it to `batch_processor` to use as a decorator on your Lambda handler function.
160191

@@ -165,19 +196,18 @@ from random import randint
165196

166197
from aws_lambda_powertools.utilities.batch import BasePartialProcessor, batch_processor
167198
import boto3
199+
import os
168200

169-
def record_handler(record):
170-
return randint(0, 100)
171-
201+
table_name = os.getenv("TABLE_NAME", "table_not_found")
172202

173203
class MyPartialProcessor(BasePartialProcessor):
174204
"""
175-
Process a record and stores successful results at a DDB Table
205+
Process a record and stores successful results at a Amazon DynamoDB Table
176206
177207
Parameters
178208
----------
179209
table_name: str
180-
Table name to write results
210+
DynamoDB table name to write results to
181211
"""
182212

183213
def __init__(self, table_name: str):
@@ -203,9 +233,10 @@ class MyPartialProcessor(BasePartialProcessor):
203233
def _process_record(self, record):
204234
# It handles how your record is processed
205235
# Here we're keeping the status of each run
236+
# where self.handler is the record_handler function passed as an argument
206237
# E.g.:
207238
try:
208-
result = self.handler(record)
239+
result = self.handler(record) # record_handler passed to decorator/context manager
209240
return self.success_handler(record, result)
210241
except Exception as exc:
211242
return self.failure_handler(record, exc)
@@ -217,7 +248,10 @@ class MyPartialProcessor(BasePartialProcessor):
217248
return entry
218249

219250

220-
@batch_processor(record_handler=record_handler, processor=MyPartialProcessor("dummy-table"))
251+
def record_handler(record):
252+
return randint(0, 100)
253+
254+
@batch_processor(record_handler=record_handler, processor=MyPartialProcessor(table_name))
221255
def lambda_handler(event, context):
222256
return {"statusCode": 200}
223257
```

0 commit comments

Comments
 (0)