Skip to content

Add support for src/common directory #1104

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Mar 20, 2023
Merged

Add support for src/common directory #1104

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b96b028
Add support for src/common directory
rzats Mar 6, 2023
2a5d2af
delete extra logger
rzats Mar 6, 2023
a53bdae
add to dates too
rzats Mar 6, 2023
2b2da06
update review comments & covidcastrow
rzats Mar 15, 2023
4df5aec
two more imports
rzats Mar 15, 2023
097e653
Review fixes
rzats Mar 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/ci.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,11 +57,12 @@ jobs:
docker build -t delphi_web_epidata -f ./devops/Dockerfile .
cd ../../../

# MODULE_NAME is used to set the location of the main.py file, see https://github.com/tiangolo/meinheld-gunicorn-docker#module_name
- name: Start services
run: |
docker network create --driver bridge delphi-net
docker run --rm -d -p 13306:3306 --network delphi-net --name delphi_database_epidata --cap-add=sys_nice delphi_database_epidata
docker run --rm -d -p 10080:80 --env "SQLALCHEMY_DATABASE_URI=mysql+mysqldb://user:pass@delphi_database_epidata:3306/epidata" --env "FLASK_SECRET=abc" --env "FLASK_PREFIX=/epidata" --network delphi-net --name delphi_web_epidata delphi_web_epidata
docker run --rm -d -p 10080:80 --env "MODULE_NAME=delphi.epidata.server.main" --env "SQLALCHEMY_DATABASE_URI=mysql+mysqldb://user:pass@delphi_database_epidata:3306/epidata" --env "FLASK_SECRET=abc" --env "FLASK_PREFIX=/epidata" --network delphi-net --name delphi_web_epidata delphi_web_epidata
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: where is MODULE_NAME used?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its used in the base image that gets built upon in devops/Dockerfile, "tiangolo/meinheld-gunicorn". see the documentation for that image, which i also mentioned in the associated issue (#1037) for this pr.

comments noting the purpose of MODULE_NAME along with that documentation link above should probably be included here, in the Makefile, and in devops/Dockerfile.

MODULE_NAME will also need to be populated in automation or ansible or whatever it is that runs the API server code, which we will probably see when we try to test running this on our staging and/or qa setups. (cc: @korlaxxalrok )

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, looks like we'd need to pass an additional environment variable.

docker ps

- run: |
Expand Down
18 changes: 9 additions & 9 deletions deploy.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,15 @@
"add-header-comment": true
},

"// common",
{
"type": "move",
"src": "src/common/",
"dst": "[[package]]/common/",
"match": "^.*\\.(py)$",
"add-header-comment": true
},

"// server",
{
"type": "move",
Expand Down Expand Up @@ -47,15 +56,6 @@
"add-header-comment": true
},

"// acquisition - common",
{
"type": "move",
"src": "src/acquisition/common/",
"dst": "[[package]]/acquisition/common/",
"match": "^.*\\.(py)$",
"add-header-comment": true
},

"// acquisition - fluview",
{
"type": "move",
Expand Down
2 changes: 2 additions & 0 deletions dev/local/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,8 +91,10 @@ web:
cd -

@# Run the web server
@# MODULE_NAME is used to set the location of the main.py file, see https://github.com/tiangolo/meinheld-gunicorn-docker#module_name
@docker run --rm -p 127.0.0.1:10080:80 \
$(M1) \
--env "MODULE_NAME=delphi.epidata.server.main" \
--env "SQLALCHEMY_DATABASE_URI=$(sqlalchemy_uri)" \
--env "FLASK_SECRET=abc" --env "FLASK_PREFIX=/epidata" --env "LOG_DEBUG" \
--network delphi-net --name delphi_web_epidata \
Expand Down
10 changes: 6 additions & 4 deletions devops/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,16 @@ LABEL org.opencontainers.image.source=https://github.com/cmu-delphi/delphi-epida

COPY ./devops/gunicorn_conf.py /app
COPY ./devops/start_wrapper.sh /
COPY ./src/server/ /app/app/
COPY --from=builder ./src/build/lib/ /app/app/lib/
RUN mkdir -p /app/delphi/epidata
COPY ./src/server /app/delphi/epidata/server
COPY ./src/common /app/delphi/epidata/common
COPY --from=builder ./src/build/lib/ /app/delphi/epidata/lib/

COPY requirements.api.txt /app/requirements_also.txt

RUN ln -s -f /usr/share/zoneinfo/America/New_York /etc/localtime \
&& rm -rf /app/app/__pycache__ /app/app/*.php \
&& chmod -R o+r /app/app \
&& rm -rf /app/delphi/epidata/__pycache__ \
&& chmod -R o+r /app/delphi/epidata \
&& chmod 755 /start_wrapper.sh \
&& pip install --no-cache-dir -r /tmp/requirements.txt -r requirements_also.txt
# the file /tmp/requirements.txt is created in the parent docker definition. (see:
Expand Down
106 changes: 0 additions & 106 deletions src/acquisition/common/logger.py
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion src/acquisition/covid_hosp/common/database.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

# first party
import delphi.operations.secrets as secrets
from delphi.epidata.acquisition.common.logger import get_structured_logger
from delphi.epidata.common.logger import get_structured_logger

Columndef = namedtuple("Columndef", "csv_name sql_name dtype")

Expand Down
2 changes: 1 addition & 1 deletion src/acquisition/covidcast/covidcast_meta_cache_updater.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

# first party
from delphi.epidata.acquisition.covidcast.database import Database
from delphi.epidata.acquisition.common.logger import get_structured_logger
from delphi.epidata.common.logger import get_structured_logger
from delphi.epidata.client.delphi_epidata import Epidata

def get_argument_parser():
Expand Down
4 changes: 2 additions & 2 deletions src/acquisition/covidcast/csv_importer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@
# first party
from delphi_utils import Nans
from delphi.utils.epiweek import delta_epiweeks
from delphi.epidata.acquisition.covidcast.covidcast_row import CovidcastRow
from delphi.epidata.acquisition.common.logger import get_structured_logger
from delphi.epidata.common.covidcast_row import CovidcastRow
from delphi.epidata.common.logger import get_structured_logger

DataFrameRow = NamedTuple('DFRow', [
('geo_id', str),
Expand Down
2 changes: 1 addition & 1 deletion src/acquisition/covidcast/csv_to_database.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
from delphi.epidata.acquisition.covidcast.csv_importer import CsvImporter, PathDetails
from delphi.epidata.acquisition.covidcast.database import Database, DBLoadStateException
from delphi.epidata.acquisition.covidcast.file_archiver import FileArchiver
from delphi.epidata.acquisition.common.logger import get_structured_logger
from delphi.epidata.common.logger import get_structured_logger


def get_argument_parser():
Expand Down
4 changes: 2 additions & 2 deletions src/acquisition/covidcast/database.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@

# first party
import delphi.operations.secrets as secrets
from delphi.epidata.acquisition.common.logger import get_structured_logger
from delphi.epidata.acquisition.covidcast.covidcast_row import CovidcastRow
from delphi.epidata.common.logger import get_structured_logger
from delphi.epidata.common.covidcast_row import CovidcastRow


class DBLoadStateException(Exception):
Expand Down
2 changes: 1 addition & 1 deletion src/acquisition/covidcast/delete_batch.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

# first party
from delphi.epidata.acquisition.covidcast.database import Database
from delphi.epidata.acquisition.common.logger import get_structured_logger
from delphi.epidata.common.logger import get_structured_logger


def get_argument_parser():
Expand Down
2 changes: 1 addition & 1 deletion src/acquisition/covidcast/file_archiver.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
import shutil

# first party
from delphi.epidata.acquisition.common.logger import get_structured_logger
from delphi.epidata.common.logger import get_structured_logger

class FileArchiver:
"""Archives files by moving and compressing."""
Expand Down
2 changes: 1 addition & 1 deletion src/acquisition/covidcast/signal_dash_data_generator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
# first party
import covidcast
import delphi.operations.secrets as secrets
from delphi.epidata.acquisition.common.logger import get_structured_logger
from delphi.epidata.common.logger import get_structured_logger


LOOKBACK_DAYS_FOR_COVERAGE = 56
Expand Down
2 changes: 1 addition & 1 deletion src/acquisition/covidcast/test_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
import pandas as pd

from delphi_utils import Nans
from delphi.epidata.acquisition.covidcast.covidcast_row import CovidcastRow
from delphi.epidata.common.covidcast_row import CovidcastRow
from delphi.epidata.acquisition.covidcast.database import Database
from delphi.epidata.server.utils.dates import day_to_time_value, time_value_to_day
import delphi.operations.secrets as secrets
Expand Down
0 src/acquisition/covidcast/covidcast_row.py → src/common/covidcast_row.py
View file
Open in desktop
File renamed without changes.
0 src/server/utils/logger.py → src/common/logger.py
View file
Open in desktop
File renamed without changes.
2 changes: 1 addition & 1 deletion src/server/_common.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
from sqlalchemy.engine import Connection, Engine
from werkzeug.local import LocalProxy

from .utils.logger import get_structured_logger
from delphi.epidata.common.logger import get_structured_logger
from ._config import SECRET, SQLALCHEMY_DATABASE_URI, SQLALCHEMY_ENGINE_OPTIONS
from ._exceptions import DatabaseErrorException, EpiDataException

Expand Down
2 changes: 1 addition & 1 deletion src/server/_printer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

from ._config import MAX_RESULTS, MAX_COMPATIBILITY_RESULTS
from ._common import is_compatibility_mode
from .utils.logger import get_structured_logger
from delphi.epidata.common.logger import get_structured_logger


def print_non_standard(format: str, data):
Expand Down
2 changes: 1 addition & 1 deletion src/server/endpoints/covidcast_meta.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
from .._params import extract_strings
from .._printer import create_printer
from .._query import filter_fields
from ..utils.logger import get_structured_logger
from delphi.epidata.common.logger import get_structured_logger

bp = Blueprint("covidcast_meta", __name__)

Expand Down
2 changes: 1 addition & 1 deletion src/server/utils/dates.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
from epiweeks import Week, Year
from typing_extensions import TypeAlias

from .logger import get_structured_logger
from delphi.epidata.common.logger import get_structured_logger

# Alias for a sequence of date ranges (int, int) or date integers
IntRange: TypeAlias = Union[Tuple[int, int], int]
Expand Down
4 changes: 2 additions & 2 deletions tests/acquisition/covidcast/test_covidcast_row.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
from pandas.testing import assert_frame_equal

from delphi_utils.nancodes import Nans
from delphi.epidata.acquisition.covidcast.covidcast_row import CovidcastRow, set_df_dtypes
from delphi.epidata.common.covidcast_row import CovidcastRow, set_df_dtypes
from delphi.epidata.acquisition.covidcast.test_utils import (
CovidcastTestRow,
covidcast_rows_as_api_compatibility_row_df,
Expand All @@ -14,7 +14,7 @@
)

# py3tester coverage target (equivalent to `import *`)
__test_target__ = 'delphi.epidata.acquisition.covidcast.covidcast_row'
__test_target__ = 'delphi.epidata.common.covidcast_row'


class TestCovidcastRows(unittest.TestCase):
Expand Down