Skip to content

Commit 24acda9

Browse files
author
Tom McCarthy
committed
docs: add detail to batch processing
1 parent 64c7093 commit 24acda9

File tree

1 file changed

+74
-2
lines changed

1 file changed

+74
-2
lines changed

docs/content/utilities/batch.mdx

+74-2
Original file line numberDiff line numberDiff line change
@@ -35,8 +35,9 @@ There are 2 ways to use this utility for processing SQS messages:
3535
**With a decorator:**
3636

3737
Using the `sqs_batch_processor` decorator with your lambda handler function, you provide a `record_handler` which is responsible for processing individual messages. It should raise an exception if
38-
it is unable to process the record - this will lead to the message returning to the queue. If the function does not return an exception, the message will be deleted from the queue. When using the decorator, you
39-
will not have accessed to the processed messages within the lambda handler - all processing should be handled from the `record_handler` function.
38+
it is unable to process the record. All records in the batch will be passed to this handler for processing, even if exceptions are thrown. After all messages are processed, any successfully processed
39+
ones will be deleted from the queue. If there were any messages the `record_handler` couldn't process, `SQSBatchProcessingError` will be raised. You will not have accessed to the _processed_ messages
40+
within the lambda handler - all processing logic should be performed by the `record_handler` function.
4041

4142
```python:title=app.py
4243
from aws_lambda_powertools.utilities.batch import sqs_batch_processor
@@ -78,6 +79,77 @@ def lambda_handler(event, context):
7879
return result
7980
```
8081

82+
## Passing custom boto3 config
83+
84+
If you need to pass custom configuration such as region to the SDK, you can pass your own [botocore config object](https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html) to
85+
the `sqs_batch_processor` decorator:
86+
87+
```python:title=app.py
88+
from aws_lambda_powertools.utilities.batch import sqs_batch_processor
89+
from botocore.config import Config
90+
91+
config = Config(region_name="us-east-1") # highlight-line
92+
93+
def record_handler(record):
94+
# This will be called for each individual message from a batch
95+
# It should raise an exception if the message was not processed successfully
96+
return_value = do_something_with(record["body"])
97+
return return_value
98+
99+
@sqs_batch_processor(record_handler=record_handler, config=config) # highlight-line
100+
def lambda_handler(event, context):
101+
return {"statusCode": 200}
102+
```
103+
104+
Or to the `PartialSQSProcessor` class:
105+
```python:title=app.py
106+
from aws_lambda_powertools.utilities.batch import PartialSQSProcessor
107+
108+
from botocore.config import Config
109+
110+
config = Config(region_name="us-east-1") # highlight-line
111+
112+
def record_handler(record):
113+
# This will be called for each individual message from a batch
114+
# It should raise an exception if the message was not processed successfully
115+
return_value = do_something_with(record["body"])
116+
return return_value
117+
118+
119+
def lambda_handler(event, context):
120+
records = event["Records"]
121+
122+
processor = PartialSQSProcessor(config=config) # highlight-line
123+
124+
with processor(records, record_handler):
125+
result = processor.process()
126+
127+
return result
128+
```
129+
130+
131+
## Suppressing exceptions
132+
133+
If you want to disable the defualt behavior where `SQSBatchProcessingError` is raised if there are any errors, you can pass the `suppress_exception` argument.
134+
135+
<Note type="warning">
136+
If your Lambda function executes successfully and returns a response, all messages in the batch will be deleted from the queue.
137+
</Note><br/>
138+
139+
```python:title=app.py
140+
...
141+
@sqs_batch_processor(record_handler=record_handler, config=config, suppress_exception=True) # highlight-line
142+
def lambda_handler(event, context):
143+
return {"statusCode": 200}
144+
```
145+
or
146+
```python:title=app.py
147+
processor = PartialSQSProcessor(config=config, suppress_exception=True) # highlight-line
148+
149+
with processor(records, record_handler):
150+
result = processor.process()
151+
```
152+
81153
## Create your own partial processor
82154

83155
You can create your own partial batch processor by inheriting the `BasePartialProcessor` class, and implementing `_prepare()`, `_clean()` and `_process_record()`.

0 commit comments

Comments
 (0)