fix: batch processing util #155
Conversation
more SQS specific focus Update for sqs_batch_processor interface
Codecov Report
@@ Coverage Diff @@
## develop #155 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 28 28
Lines 806 827 +21
Branches 76 77 +1
=========================================
+ Hits 806 827 +21
Continue to review full report at Codecov.
|
| records = event["Records"] | ||
|
|
||
| with processor(records, record_handler): | ||
| processor.process() |
There was a problem hiding this comment.
Can you add a logger.debug here to provide a message that you're processing another record. We don't need to log the record for security reasons, just a message suffices.
There was a problem hiding this comment.
This is outside of the loop for processing single records, but I've added debug logging elsewhere which should suffice.
docs/content/utilities/batch.mdx
Outdated
|
|
||
| **Background** | ||
|
|
||
| When using SQS as a Lambda event source mapping, functions can be triggered with a batch of messages from SQS. If the Lambda function fails when processing the batch, |
There was a problem hiding this comment.
Let's add some line breaks to make it easier to read. The docs theme kinda forces you to do that or else it cramps the lines
docs/content/utilities/batch.mdx
Outdated
|
|
||
| ### PartialSQSProcessor | ||
| **With a decorator:** |
There was a problem hiding this comment.
Use a sub-heading #### as opposed to bold text - It allows customers and us to easily reference it in conversations
docs/content/utilities/batch.mdx
Outdated
|
|
||
| SQS integration with Lambda is one of the most well established ones and pretty useful when building asynchronous applications. One common approach to maximize the performance of this integration is to enable the batch processing feature, resulting in higher throughput with less invocations. | ||
| Using the `sqs_batch_processor` decorator with your lambda handler function, you provide a `record_handler` which is responsible for processing individual messages. It should raise an exception if |
There was a problem hiding this comment.
Let's add line breaks here to make it easier to read in our docs theme
docs/content/utilities/batch.mdx
Outdated
|
|
||
| A *naive* approach to solving this problem is to delete successful records from the queue before redriving's phase. The `PartialSQSProcessor` class offers this solution both as context manager and middleware, removing all successful messages from the queue case one or more failures occurred during lambda's execution. Two examples are provided below, displaying the behavior of this class. | ||
| **With a context manager:** |
There was a problem hiding this comment.
Use a sub-heading #### as opposed to bold text - It allows customers and us to easily reference it in conversations
Issue #, if available:
Description of changes:
Simplified the documentation, primarily by focusing more on the SQS specific implementation. Added a sqs_batch_processor interface to simplify the common use case. Fixed an issue where no exception was being raised for failed messages, which would have caused failed messages to be deleted from the queue.
Checklist
Breaking change checklist
RFC issue #:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.