Skip to content

sample queries for API server correctness #1128

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
melange396 opened this issue Apr 7, 2023 · 1 comment
Open

sample queries for API server correctness #1128

melange396 opened this issue Apr 7, 2023 · 1 comment
Assignees

Comments

@melange396
Copy link
Collaborator

create a list of sample http queries to verify API server correctness. these queries should touch every endpoint, and do so through all available routes. the queries should also use many different parameters and options applicable for each endpoint, ideally covering them with an exhaustive set of such combinations. the queries can be generated by hand or extracted (and maybe modified) from production web server logs. the queries should use relatively older dates as their parameters so that we can be sure the returned results dont change, and remain "known good".

there are a few different ways to evaluate the responses from these queries, with increasing complexity of implementation (but also increasing value of correctness assurance):

  1. make sure each response returns a successful http code of 200 (and not 500 or similar).
  2. make sure each response returns a known/correct number of result items.
  3. make sure each response returns the exact known/correct result data.
    (getting the known/correct result data or counts is as simple as running the queries on an already working system.)

similar to what was done in #1086, this should be connected to a production-like db backend so that we have "real" data to pull from. also like that referenced PR, this could be done with locust, but that may limit us to just verifying http return codes and not allow deeper inspection like counting rows returned or matching content of responses.

this issue is closely related to #1068, but this doesnt necessarily need to be a proper part of our python suite of integration tests.

this would have saved us the trouble we saw with rollout of 0.4.8, which we hastily fixed in #1121

@dmytrotsko dmytrotsko self-assigned this Sep 15, 2023
@dmytrotsko
Copy link
Contributor

dmytrotsko commented Sep 15, 2023

This is already done. Just need to setup & deploy somewhere.
https://github.com/cmu-delphi/api_health_check/tree/main#api_health_check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants