-
Notifications
You must be signed in to change notification settings - Fork 433
feat: simple JSON Schema validator utility #153
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
heitorlessa
merged 20 commits into
aws-powertools:develop
from
heitorlessa:feat/validator-utility
Sep 18, 2020
Merged
Changes from 12 commits
Commits
Show all changes
20 commits
Select commit
Hold shift + click to select a range
14116f5
feat: add initial draft simple validator
heitorlessa 080c9fd
feat: add jmespath as optional dependency
heitorlessa 6bda0f5
feat: add custom jmespath functions support
heitorlessa c498b77
improv: add logging; linting
heitorlessa 1d21135
feat: add built-in envelopes
heitorlessa acc3b83
improv: make jmespath an official project dep
heitorlessa d7687c9
feat: add powertools_base64 custom fn
heitorlessa 349e88d
feat: add cloudwatch_logs based on Bryan's feedback
heitorlessa 49eb589
Merge branch 'develop' into feat/validator-utility
heitorlessa 3f45c1d
feat: add initial validator tests
heitorlessa 4dc8c47
improv: add validate functional tests
heitorlessa cce2ab4
improv: test built-in envelopes
heitorlessa c766c10
improv: test validator decorator
heitorlessa cd6c34f
improv: fix kinesis envelope test
heitorlessa 6877dee
improv: add docstrings
heitorlessa b8e45ea
chore: fix docstring; import order
heitorlessa 0926176
chore: update changelog
heitorlessa 1bd4445
docs: document validator utility
heitorlessa 4f1b4d7
docs: improve wording on jmespath fns
heitorlessa f7f3c99
improv: test to cover custom jmespath opts
heitorlessa File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
""" | ||
Simple validator to enforce incoming/outgoing event conforms with JSON Schema | ||
""" | ||
|
||
from .exceptions import InvalidEnvelopeExpressionError, InvalidSchemaFormatError, SchemaValidationError | ||
from .validator import validate, validator | ||
|
||
__all__ = [ | ||
"validate", | ||
"validator", | ||
"InvalidSchemaFormatError", | ||
"SchemaValidationError", | ||
"InvalidEnvelopeExpressionError", | ||
] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,33 @@ | ||
import logging | ||
from typing import Dict | ||
|
||
import fastjsonschema | ||
import jmespath | ||
from jmespath.exceptions import LexerError | ||
|
||
from .exceptions import InvalidEnvelopeExpressionError, InvalidSchemaFormatError, SchemaValidationError | ||
from .jmespath_functions import PowertoolsFunctions | ||
|
||
logger = logging.getLogger(__name__) | ||
|
||
|
||
def validate_data_against_schema(data: Dict, schema: Dict): | ||
try: | ||
fastjsonschema.validate(definition=schema, data=data) | ||
except fastjsonschema.JsonSchemaException as e: | ||
message = f"Failed schema validation. Error: {e.message}, Path: {e.path}, Data: {e.value}" # noqa: B306, E501 | ||
raise SchemaValidationError(message) | ||
except (TypeError, AttributeError) as e: | ||
raise InvalidSchemaFormatError(f"Schema received: {schema}. Error: {e}") | ||
|
||
|
||
def unwrap_event_from_envelope(data: Dict, envelope: str, jmespath_options: Dict): | ||
if not jmespath_options: | ||
jmespath_options = {"custom_functions": PowertoolsFunctions()} | ||
|
||
try: | ||
logger.debug(f"Envelope detected: {envelope}. JMESPath options: {jmespath_options}") | ||
return jmespath.search(envelope, data, options=jmespath.Options(**jmespath_options)) | ||
except (LexerError, TypeError, UnicodeError) as e: | ||
message = f"Failed to unwrap event from envelope using expression. Error: {e} Exp: {envelope}, Data: {data}" # noqa: B306, E501 | ||
raise InvalidEnvelopeExpressionError(message) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
"""Built-in envelopes""" | ||
|
||
API_GATEWAY_REST = "powertools_json(body)" | ||
API_GATEWAY_HTTP = API_GATEWAY_REST | ||
SQS = "Records[*].powertools_json(body)" | ||
SNS = "Records[0].Sns.Message | powertools_json(@)" | ||
EVENTBRIDGE = "detail" | ||
CLOUDWATCH_EVENTS_SCHEDULED = EVENTBRIDGE | ||
KINESIS_DATA_STREAM = "Records[*].kinesis.powertools_base64(data) | powertools_json(@)" | ||
CLOUDWATCH_LOGS = "awslogs.powertools_base64_gzip(data) | powertools_json(@).logEvents[*]" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
class SchemaValidationError(Exception): | ||
"""When serialization fail schema validation""" | ||
|
||
pass | ||
|
||
|
||
class InvalidSchemaFormatError(Exception): | ||
"""When JSON Schema is in invalid format""" | ||
|
||
pass | ||
|
||
|
||
class InvalidEnvelopeExpressionError(Exception): | ||
"""When JMESPath fails to parse expression""" |
22 changes: 22 additions & 0 deletions
22
aws_lambda_powertools/utilities/validation/jmespath_functions.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
import base64 | ||
import gzip | ||
import json | ||
|
||
import jmespath | ||
|
||
|
||
class PowertoolsFunctions(jmespath.functions.Functions): | ||
@jmespath.functions.signature({"types": ["string"]}) | ||
def _func_powertools_json(self, value): | ||
return json.loads(value) | ||
|
||
@jmespath.functions.signature({"types": ["string"]}) | ||
def _func_powertools_base64(self, value): | ||
return base64.b64decode(value).decode() | ||
|
||
@jmespath.functions.signature({"types": ["string"]}) | ||
def _func_powertools_base64_gzip(self, value): | ||
encoded = base64.b64decode(value) | ||
uncompressed = gzip.decompress(encoded) | ||
|
||
return uncompressed.decode() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,40 @@ | ||
import logging | ||
from typing import Any, Callable, Dict, Union | ||
|
||
from ...middleware_factory import lambda_handler_decorator | ||
from .base import unwrap_event_from_envelope, validate_data_against_schema | ||
|
||
logger = logging.getLogger(__name__) | ||
|
||
|
||
@lambda_handler_decorator | ||
def validator( | ||
handler: Callable, | ||
event: Union[Dict, str], | ||
context: Any, | ||
inbound_schema: Dict = None, | ||
outbound_schema: Dict = None, | ||
envelope: str = None, | ||
jmespath_options: Dict = None, | ||
): | ||
if envelope: | ||
event = unwrap_event_from_envelope(data=event, envelope=envelope, jmespath_options=jmespath_options) | ||
|
||
if inbound_schema: | ||
logger.debug("Validating inbound event") | ||
validate_data_against_schema(data=event, schema=inbound_schema) | ||
|
||
response = handler(event, context) | ||
|
||
if outbound_schema: | ||
logger.debug("Validating outbound event") | ||
validate_data_against_schema(data=event, schema=outbound_schema) | ||
|
||
return response | ||
|
||
|
||
def validate(event: Dict, schema: Dict = None, envelope: str = None, jmespath_options: Dict = None): | ||
if envelope: | ||
event = unwrap_event_from_envelope(data=event, envelope=envelope, jmespath_options=jmespath_options) | ||
|
||
validate_data_against_schema(data=event, schema=schema) |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.