diff --git a/Makefile b/Makefile index 584ed361ed9..d52d0b46cbb 100644 --- a/Makefile +++ b/Makefile @@ -6,6 +6,7 @@ target: dev: pip install --upgrade pip pre-commit poetry + poetry config --local virtualenvs.in-project true @$(MAKE) dev-version-plugin poetry install --extras "all datamasking-aws-sdk" pre-commit install diff --git a/docs/contributing/conventions.md b/docs/contributing/conventions.md new file mode 100644 index 00000000000..23db25b9ad7 --- /dev/null +++ b/docs/contributing/conventions.md @@ -0,0 +1,34 @@ +--- +title: Conventions +description: General conventions and practices that are applicable throughout to Powertools for AWS Lambda (Python) +--- + + + +## General terminology and practices + +These are common conventions we keep on building as the project gains new contributors and grows in complexity. + +As we gather more concrete examples, this page will have one section for each category to demonstrate a before and after. + +| Category | Convention | +| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Docstring** | We use [Numpy convention](https://numpydoc.readthedocs.io/en/latest/format.html){target="_blank"} with markdown to help generate more readable API references. For public APIs, we always include at least one **Example** to ease everyone's experience when using an IDE. | +| **Style guide** | We use black and [Ruff](https://beta.ruff.rs/docs/) to enforce beyond good practices [PEP8](https://pep8.org/). We use type annotations and enforce static type checking at CI (mypy). | +| **Core utilities** | Core utilities always accept `service` as a constructor parameter, can work in isolation, and are also available in other languages implementation. | +| **Utilities** | Utilities are not as strict as core and focus on community needs: development productivity, industry leading practices, etc. Both core and general utilities follow our [Tenets](https://docs.powertools.aws.dev/lambda/python/#tenets). | +| **Exceptions** | Specific exceptions live within utilities themselves and use `Error` suffix e.g. `MetricUnitError`. | +| **Git commits** | We follow [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/). We do not enforce conventional commits on contributors to lower the entry bar. Instead, we enforce a conventional PR title so our label automation and changelog are generated correctly. | +| **API documentation** | API reference docs are generated from docstrings which should have Examples section to allow developers to have what they need within their own IDE. Documentation website covers the wider usage, tips, and strive to be concise. | +| **Documentation** | We treat it like a product. We sub-divide content aimed at getting started (80% of customers) vs advanced usage (20%). We also ensure customers know how to unit test their code when using our features. | + +## Testing definition + +We group tests in different categories + +| Test | When to write | Notes | Speed | +| ----------------- | ----------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | +| Unit tests | Verify the smallest possible unit works. | Networking access is prohibited. Prefer Functional tests given our complexity. | Lightning fast (nsec to ms) | +| Functional tests | Guarantee functionality works as expected. It's a subset of integration test covering multiple units. | No external dependency. Prefer Fake implementations (in-memory) over Mocks and Stubs. | Fast (ms to few seconds at worst) | +| End-to-end tests | Gain confidence that a Lambda function with our code operates as expected. | It simulates how customers configure, deploy, and run their Lambda function - Event Source configuration, IAM permissions, etc. | Slow (minutes) | +| Performance tests | Ensure critical operations won't increase latency and costs to customers. | CI arbitrary hardware can make it flaky. We'll resume writing perf test after we revamp our functional tests with internal utilities. | Fast to moderate (a few seconds to a few minutes) | diff --git a/docs/contributing/documentation/rfcs.md b/docs/contributing/documentation/rfcs.md new file mode 100644 index 00000000000..6d344daceea --- /dev/null +++ b/docs/contributing/documentation/rfcs.md @@ -0,0 +1,8 @@ +--- +title: Writing Request For Comment (RFC) +description: Contributing RFCs to Powertools for AWS Lambda (Python) +--- + +## TBW + +Something great will come. diff --git a/docs/contributing/getting_started.md b/docs/contributing/getting_started.md new file mode 100644 index 00000000000..3cdcc6b1ddc --- /dev/null +++ b/docs/contributing/getting_started.md @@ -0,0 +1,104 @@ +--- +title: Your first contribution +description: All you need to know for your first contribution to Powertools for AWS Lambda (Python) +--- + + + +Thank you for your interest in contributing to our project - we couldn't be more excited! + +
+```mermaid +graph LR + Learn["Learn about contributions"] --> Find["Find areas to work / get mentoring"] --> Work["Prepare pull request"] --> Closing["Take learnings with you"] +``` +End-to-end process +
+ +## Types of contributions + +We consider any contribution that help this project improve everyone's experience to be valid, as long as you agree with our [tenets](../index.md#tenets){target="_blank"}, [licensing](../../LICENSE){target="_blank"}, and [Code of Conduct](#code-of-conduct). + +Whether you're new contributor or a pro, we compiled a list of the common contributions to help you choose your first: + +!!! info "Please check [existing open](https://github.com/aws-powertools/powertools-lambda-python/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc){target='_blank'}, or [recently closed](https://github.com/aws-powertools/powertools-lambda-python/issues?q=is%3Aissue+sort%3Aupdated-desc+is%3Aclosed){target='_blank'} issues before creating a new one." + Each type link goes to their respective template, or Discord invite. + +| Type | Description | +| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| [Documentation](https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=documentation%2Ctriage&projects=&template=documentation_improvements.yml&title=Docs%3A+TITLE){target="_blank" rel="nofollow"} | Ideas to make user guide or API guide clearer. It generally go from typos, diagrams, tutorials, the lack of documentation, etc. | +| [Feature request](https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=feature-request%2Ctriage&projects=&template=feature_request.yml&title=Feature+request%3A+TITLE){target="_blank" rel="nofollow"} | New functionalities or enhancements that could help you, your team, existing and future customers. Check out our [process to understand how we prioritize it](../roadmap.md#process){target="_blank"}. | +| [Design proposals](https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=RFC%2Ctriage&projects=&template=rfc.yml&title=RFC%3A+TITLE){target="_blank" rel="nofollow"} | Request for Comments (RFC) including user experience (UX) based on a feature request to gather the community feedback, and demonstrate the art of the possible. | +| [Bug report](https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=bug%2Ctriage&projects=&template=bug_report.yml&title=Bug%3A+TITLE){target="_blank" rel="nofollow"} | A runtime error that is reproducible whether you have an idea how to solve it or not. | +| [Advocacy](https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=community-content&projects=&template=share_your_work.yml&title=%5BI+Made+This%5D%3A+%3CTITLE%3E){target="_blank" rel="nofollow"} | Share what you did with Powertools for AWS Lambda. Blog posts, workshops, presentation, sample applications, podcasts, etc. | +| [Public reference](https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=customer-reference&projects=&template=support_powertools.yml&title=%5BSupport+Powertools+for+AWS+Lambda+%28Python%29%5D%3A+%3Cyour+organization+name%3E){target="_blank" rel="nofollow"} | Become a public reference to share how you're using Powertools for AWS Lambda at your organization. | +| [Discussions](https://discord.gg/B8zZKbbyET){target="_blank" rel="nofollow"} | Kick off a discussion on Discord, introduce yourself, and help respond to existing questions from the community. | +| [Static typing](https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=typing%2Ctriage&projects=&template=static_typing.yml&title=Static+typing%3A+TITLE){target="_blank" rel="nofollow"} | Improvements to increase or correct static typing coverage to ease maintenance, autocompletion, etc. | +| [Technical debt](https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=tech-debt%2Ctriage&projects=&template=tech_debt.yml&title=Tech+debt%3A+TITLE){target="_blank" rel="nofollow"} | Suggest areas to address technical debt that could make maintenance easier or provide customer value faster. Generally used by maintainers and contributors. | +| [Governance](https://github.com/aws-powertools/powertools-lambda-python/issues/new?assignees=&labels=internal%2Ctriage&projects=&template=maintenance.yml&title=Maintenance%3A+TITLE){target="_blank" rel="nofollow"} | Ideas to improve to our governance processes, automation, and anything internal. Typically used by maintainers and regular contributors. | + +## Finding contributions to work on + +[Besides suggesting ideas](#types-of-contributions) you think it'll improve everyone's experience, these are the most common places to find work: + +| Area | Description | +| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [Help wanted issues](https://github.com/aws-powertools/powertools-lambda-python/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3A%22help+wanted%22+){target="_blank" rel="nofollow"} | These are triaged areas that we'd appreciate any level of contribution - from opinions to actual implementation. | +| [Missing customer feedback issues](https://github.com/aws-powertools/powertools-lambda-python/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3Aneed-customer-feedback){target="_blank" rel="nofollow"} | These are items we'd like to hear from more customers before making any decision. Sharing your thoughts, use case, or asking additional questions are great help. | +| [Pending design proposals](https://github.com/aws-powertools/powertools-lambda-python/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3ARFC){target="_blank" rel="nofollow"} | These are feature requests that initially look good but need a RFC to enrich the discussion by validating user-experience, tradeoffs, and highlight use cases. | +| [Backlog items](https://github.com/orgs/aws-powertools/projects/3/views/3?query=is%3Aopen+sort%3Aupdated-desc){target="_blank" rel="nofollow"} | We use GitHub projects to surface what we're working on, needs triage, etc. This view shows items we already triaged but don't have the bandwidth to tackle them just yet. | +| [Documentation](https://docs.powertools.aws.dev/lambda/python/latest/){target="_blank"} | Documentation can always be improved. Look for areas that a better example, or a diagram, or more context helps everyone - keep in mind a diverse audience and English as a second language folks. | +| [Participate in discussions](https://discord.gg/B8zZKbbyET){target="_blank" rel="nofollow"} | There's always a discussion that could benefit others in the form of documentation, blog post, etc. | +| [Roadmap](../roadmap.md){target="_blank"} | Some roadmap items need a RFC to discuss design options, or gather customers use case before we can prioritize it. | +| Build a sample application | Using Powertools for AWS Lambda in different contexts will give you insights on what could be made easier, which documentation could be enriched, and more. | + +!!! question "Still couldn't find anything that match your skill set?" + Please reach out on [Discord](https://discord.gg/B8zZKbbyET){target="_blank" rel="nofollow"}, specially if you'd like to get mentoring for a task you'd like to take but you don't feel ready yet :) + + Contributions are meant to be bi-directional. There's always something we can learn from each other. + +## Sending a pull request + +!!! note "First time creating a Pull Request? Keep [this document handy.](https://help.github.com/articles/creating-a-pull-request/){target='blank' rel='nofollow'}" + +Before sending us a pull request, please ensure that: + +* [ ] You are working against the latest source on the **develop** branch. +* [ ] You check existing [open, and recently merged](https://github.com/aws-powertools/powertools-lambda-python/pulls?q=is%3Apr+is%3Aopen%2Cmerged+sort%3Aupdated-desc){target="_blank" rel="nofollow"} pull requests to make sure someone else hasn't addressed the problem already. +* [ ] You open an [issue](https://github.com/aws-powertools/powertools-lambda-python/issues/new/choose){target="_blank" rel="nofollow"} before you begin any implementation. We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful. +* [ ] Create a new branch named after the change you are contributing _e.g._ `feat/logger-debug-sampling` + +**Ready?** + +These are the steps to send a pull request: + +1. Run all formatting, linting, tests, documentation and baseline checks: `make pr` +2. Commit to your fork using clear commit messages. Don't worry about typos or format, we squash all commits during merge. +3. Send us a pull request with a [conventional semantic title](https://github.com/aws-powertools/powertools-lambda-python/pull/67). +4. Fill in the areas pre-defined in the pull request body to help expedite reviewing your work. +5. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. + +## Code of Conduct + +!!! info "This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct){target='_blank'}" + +For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact + with any additional questions or comments. + +## Security issue notifications + +If you discover a potential security issue in this project, we kindly ask you to notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. + +## Troubleshooting + +### API reference documentation + +When you are working on the codebase and you use the local API reference documentation to preview your changes, you might see the following message: `Module aws_lambda_powertools not found`. + +This happens when: + +* You did not install the local dev environment yet + * You can install dev deps with `make dev` command +* The code in the repository is raising an exception while the `pdoc` is scanning the codebase + * Unfortunately, this exception is not shown to you, but if you run, `poetry run pdoc --pdf aws_lambda_powertools`, the exception is shown and you can prevent the exception from being raised + * Once resolved the documentation should load correctly again diff --git a/docs/contributing/setup.md b/docs/contributing/setup.md new file mode 100644 index 00000000000..50533fad4b6 --- /dev/null +++ b/docs/contributing/setup.md @@ -0,0 +1,67 @@ +--- +title: Development environment +description: Setting up your development environment for contribution +--- + + + +[![Join our Discord](https://dcbadge.vercel.app/api/server/B8zZKbbyET)](https://discord.gg/B8zZKbbyET){target="_blank" rel="nofollow"} + +This page describes how to setup your development environment (Cloud or locally) to contribute to Powertools for AWS Lambda. + +
+```mermaid +graph LR + Dev["Development environment"] --> Quality["Run quality checks locally"] --> PR["Prepare pull request"] --> Collaborate +``` +End-to-end process +
+ +## Requirements + +!!! question "First time contributing to an open-source project ever?" + Read this [introduction on how to fork and clone a project on GitHub](https://docs.github.com/en/get-started/quickstart/contributing-to-projects){target="_blank" rel="nofollow"}. + +Unless you're using the pre-configured Cloud environment, you'll need the following installed: + +* [GitHub account](https://github.com/join){target="_blank" rel="nofollow"}. You'll need to be able to fork, clone, and contribute via pull request. +* [Python 3.8+](https://www.python.org/downloads/){target="_blank" rel="nofollow"}. Pick any version supported in [AWS Lambda runtime](https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html). +* [Docker](https://docs.docker.com/engine/install/){target="_blank" rel="nofollow"}. We use it to run documentation linters and non-Python tooling. +* [Fork the repository](https://github.com/aws-powertools/powertools-lambda-python/fork). You'll work against your fork of this repository. + +??? note "Additional requirements if running end-to-end tests" + + * [AWS CDK CLI](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_prerequisites){target="_blank"} + * [AWS Account bootstrapped with CDK](https://docs.aws.amazon.com/cdk/v2/guide/bootstrapping.html){target="_blank"} + * [AWS CLI installed and configured](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) + +## Cloud environment + +> **NOTE**. Be mindful of [Gitpod pricing structure](https://www.gitpod.io/pricing){target="_blank" rel="nofollow"} for long-running contributions. When in doubt, use the local environment below. + +To use a pre-configured environment, replace `YOUR_USERNAME` with your GitHub username or organization. + +```bash +https://gitpod.io/#https://github.com/YOUR_USERNAME/powertools-lambda-python #(1)! +``` + +1. For example, my username is `heitorlessa`.

Therefore, my final URL should be `https://gitpod.io/#https://github.com/heitorlessa/powertools-lambda-python` + +Once provisioned, it'll install all development dependencies and tools you'll need to contribute. + +## Local environment + +> Assuming you've got all [requirements](#requirements). + +You can use `make dev` to create a local virtual environment and install all dependencies locally. + +!!! note "Curious about what `make dev` does under the hood?" + We use `Make` to [automate common tasks](https://github.com/aws-powertools/powertools-lambda-python/blob/1ebe3275a5c53aed5a8eb76318e7d0af2367edfa/Makefile#L7){target="_blank" rel="nofollow"} locally and in Continuous Integration environments. + +## Local documentation + +You might find useful to run both the documentation website and the API reference locally while contributing: + +* **Docs website**: `make docs-local` + * If you prefer using Docker: `make docs-local-docker` +* **API reference**: `make docs-api-local` diff --git a/docs/roadmap.md b/docs/roadmap.md index c2de6829fb4..e42fae21c97 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -36,8 +36,9 @@ Data Masking will be a new utility to mask/unmask sensitive data using encryptio **Major updates** - [x] [RFC to agree on design and MVP](https://github.com/aws-powertools/powertools-lambda-python/issues/1858) -- [ ] [POC with AWS KMS as the default provider](https://github.com/aws-powertools/powertools-lambda-python/pull/2197) -- [ ] Documentation to guide customers how to bring their own provider (e.g., `ItsDangerous`) +- [x] [POC with AWS KMS as the default provider](https://github.com/aws-powertools/powertools-lambda-python/pull/2197) +- [ ] User-guide documentation and include when not to use it (e.g., when to use SNS data policy, CloudWatch Logs data policy) +- [ ] Decide whether to use Encryption SDK to bring their own provider or a simply a contract (e.g., `ItsDangerous`) ### Revamp Event Handler @@ -49,8 +50,8 @@ Based on customers feedback, we want to provide middleware authoring support for - [x] [Agree on experience for middleware support](https://github.com/aws-powertools/powertools-lambda-python/issues/953#issuecomment-1450223155) - [x] [RFC to outline initial thoughts on OpenAPI integration](https://github.com/aws-powertools/powertools-lambda-python/issues/2421) -- [ ] MVP for REST middleware -- [ ] MVP for OpenAPI and SwaggerUI +- [x] [MVP for REST middleware](./core/event_handler/api_gateway.md#middleware) +- [ ] [MVP for OpenAPI and SwaggerUI](https://github.com/aws-powertools/powertools-lambda-python/pull/3109) - [ ] [MVP for AppSync Batch invoke and partial failure support](https://github.com/aws-powertools/powertools-lambda-python/pull/1998) ### Lambda Layer in release notes @@ -112,10 +113,10 @@ We want to investigate security and scaling requirements for these special regio **Major updates** -- [ ] Gather agencies and customers name to prioritize it -- [ ] Investigate security requirements for special regions -- [ ] Update CDK Layer construct to include regions +- [x] Gather agencies and customers name to prioritize it +- [x] Investigate security requirements for special regions - [ ] Create additional infrastructure for special regions +- [ ] Update CDK Layer construct to include regions ### V3 diff --git a/mkdocs.yml b/mkdocs.yml index 1b9f4545239..0a844fd392f 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -40,6 +40,24 @@ nav: - Automation: automation.md - Roadmap: roadmap.md - Maintainers: maintainers.md + - Contributing: + # - contributing/index.md + - Development environment: contributing/setup.md + - Your first contribution: contributing/getting_started.md + - Conventions: contributing/conventions.md + # - Testing: + # - Unit tests: contributing/testing/unit_tests.md + # - Functional tests: contributing/testing/functional_tests.md + # - End-to-end tests: contributing/testing/e2e_tests.md + # - Documentation: + # - User guide: contributing/documentation/user_guide.md + # - API guide: contributing/documentation/api_guide.md + # - Tutorials: contributing/documentation/tutorials.md + # - Request For Comments (RFC): contributing/documentation/rfc.md + # - Tracks: + # - Overview: contributing/tracks/overview.md + # - Casual to regular contributor: contributing/tracks/casual_regular_contributor.md + # - Customer to advocate: contributing/tracks/customer_advocate.md theme: name: material