Updated DB used by the sample app from mysql to postgres (#32)

Updated the DB used by the sample app from mysql to postgres. Tested
locally using docker and on EKS by running the one touch script. Was
able to connect to DB and do the basic CRUD operations through
vehicle-service APIs.


By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.

Author: AsakerMohd

sample-applications/vehicle-dealership-sample-app/ImageServiceApp/ImageServiceApp/settings.py

@@ -15,6 +15,10 @@
 import os
 from pathlib import Path
 
+from dotenv import load_dotenv
+
+load_dotenv()
+
 # Build paths inside the project like this: BASE_DIR / 'subdir'.
 BASE_DIR = Path(__file__).resolve().parent.parent
 
@@ -78,8 +82,12 @@
 
 DATABASES = {
     "default": {
-        "ENGINE": "django.db.backends.sqlite3",
-        "NAME": BASE_DIR / "db.sqlite3",
+        "ENGINE": "django.db.backends.postgresql",
+        "USER": os.environ.get("POSTGRES_USER"),
+        "NAME": os.environ.get("POSTGRES_DATABASE"),
+        "PASSWORD": os.environ.get("POSTGRES_PASSWORD"),
+        "HOST": os.environ.get("DB_SERVICE_HOST"),
+        "PORT": os.environ.get("DB_SERVICE_PORT"),
     }
 }
 

sample-applications/vehicle-dealership-sample-app/ImageServiceApp/requirements.txt

@@ -1,4 +1,5 @@
 Django~=5.0
 requests~=2.31.0
 boto3~=1.34.14
-python-dotenv~=1.0.0
+python-dotenv~=1.0.0
+psycopg2~=2.9.9

sample-applications/vehicle-dealership-sample-app/README.md

@@ -23,7 +23,7 @@ This directory contains code for a microservices based sample app that is used t
 
 ### EKS
 To get started with AWS EKS, you can run the one touch script as below.
-`bash script.sh <account_id> <cluster_name> <region> <mysql_password> <s3_bucket_name>`
+`bash script.sh <account_id> <cluster_name> <region> <postgres_password> <s3_bucket_name>`
 
 This will create the docker images, upload them to ECR and then create pods on EKS with those images. 
 
@@ -52,7 +52,7 @@ To deploy to EC2, you will have to go through the following steps.
       - AmazonS3FullAccess 
       - AmazonSQSFullAccess
     - Name one vehicle-service and the other image-service
-5. Go to RDS and create a MySQL DB with the following configurations:
+5. Go to RDS and create a Postgres DB with the following configurations:
     - Use the Dev/Test template
     - Update the Master username to `root` and create a password of your choosing. Write it down since you will need it later. 
     - In the Connectivity settings, choose the VPC and security group created in step 3. 
@@ -61,18 +61,15 @@ To deploy to EC2, you will have to go through the following steps.
 ```
 sudo dnf install python3.11
 sudo dnf install python3.11-pip
-sudo dnf install mariadb105
-sudo dnf install -y mariadb105-devel gcc python3.11-devel
+sudo dnf install postgresql15
 
-mysql -h <RDS_DB_Endpoint> -P 3306 -u root -p<password_from_step_5>
+createdb vehicle_inventory -h <rds_url> -U root
+createuser djangouser -h <rds_url> -U root
 
-CREATE DATABASE vehicle_inventory;
+psql -h <rds_url> -d vehicle_inventory -U root
 
-CREATE USER 'djangouser'@'%' IDENTIFIED BY '<password_of_your_choosing>';
-
-GRANT ALL PRIVILEGES ON vehicle_inventory.* TO 'djangouser'@'%' WITH GRANT OPTION;
-
-FLUSH PRIVILEGES;
+alter user djangouser with encrypted password '<password_of_your_choosing>';
+grant all privileges on database vehicle_inventory to djangouser;
 
 aws s3 sync s3://<s3_bucket_that_has_python_code> .
 
@@ -81,18 +78,19 @@ cd to the vehicle microservice directory and run:
 python3.11 -m pip install -r requirements.txt
 
 Create a .env file with the following: 
-MYSQL_ROOT_PASSWORD=<password_from_RDS_setup>
-MYSQL_DATABASE=vehicle_inventory
-MYSQL_USER=djangouser
-MYSQL_PASSWORD=<password_from_this_step>
+POSTGRES_ROOT_PASSWORD=<password_from_RDS_setup>
+POSTGRES_DATABASE=vehicle_inventory
+POSTGRES_USER=djangouser
+POSTGRES_PASSWORD=<password_from_this_step>
 DB_SERVICE_HOST=<RDS_DB_endpoint>
 DB_SERVICE_PORT=3306
 IMAGE_BACKEND_SERVICE_HOST=<image-service_ec2_public_IP>
 IMAGE_BACKEND_SERVICE_PORT=8000
 
 python3.11 manage.py migrate --noinput && python3.11 manage.py runserver 0.0.0.0:8001
 ```
-7. Connect to the `image-service` EC2 instance and run the following:
+7. Go to the EC2 console and select the `image-service`, Go Actions -> Networking -> Connect RDS database.
+8. Connect to the `image-service` EC2 instance and run the following:
 ```
 sudo dnf install python3.11
 sudo dnf install python3.11-pip
@@ -113,18 +111,18 @@ Now you should be able to access the APIs below through the EC2 addr:port of eac
 
 ### Locally with Docker
 To get started, make sure you either have you AWS creds in `$HOME/.aws` or the following: `AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN` are exported.
-1. Run `bash local_script.sh <mysql_pass> <s3_bucket_name>`. 
+1. Run `bash local_script.sh <postgres_pass> <s3_bucket_name>`. 
 This will create docker containers, move the requirement env variables there and start them up. 
 
 They should be accessible through `0.0.0.0:8000` for the image service and `0.0.0.0:8001` for the vehicle service. 
 
 ## APIs
 
 The following are the APIs and what they do:
-1. GET /vehicle-inventory/: returns all the vehicles entries for mysql db
-2. PUT /vehicle-inventory/: puts vehicle into db. For example: `curl -X POST http://0.0.0.0:8001/vehicle-inventory/ -d '{"make": "BMW","model": "M340","year": 2022,"image_name": "newCar.jpg"}'`
+1. GET /vehicle-inventory/: returns all the vehicles entries for postgres db
+2. POST /vehicle-inventory/: puts vehicle into db. For example: `curl -X POST http://0.0.0.0:8001/vehicle-inventory/ -d '{"make": "BMW","model": "M340","year": 2022,"image_name": "newCar.jpg"}'`
 3. GET /vehicle-inventory/<int>: returns vehicle entry with id = <int>
 4. GET /vehicle-inventory/<int>/image: returns image file information from S3 for the specific vehicle by calling the image microservice
 5. GET /images/name/<image_name>: returns image information for <image_name> from S3 if present. 
-6. PUT /images/name/<image_name>: creates an empty file in S3. This is an async endpoint since it will put image name in an SQS queue and not wait for the file to be created in S3. Instead, a long running thread will poll SQS and then create the image file later. 
+6. POST /images/name/<image_name>: creates an empty file in S3. This is an async endpoint since it will put image name in an SQS queue and not wait for the file to be created in S3. Instead, a long running thread will poll SQS and then create the image file later. 
 7. GET /image/remote-image: makes a remote http call to google.com. 

sample-applications/vehicle-dealership-sample-app/VehicleInventoryApp/MainService/migrations/__init__.py

@@ -81,10 +81,10 @@
 
 DATABASES = {
     "default": {
-        "ENGINE": "django.db.backends.mysql",
-        "USER": os.environ.get("MYSQL_USER"),
-        "NAME": os.environ.get("MYSQL_DATABASE"),
-        "PASSWORD": os.environ.get("MYSQL_PASSWORD"),
+        "ENGINE": "django.db.backends.postgresql",
+        "USER": os.environ.get("POSTGRES_USER"),
+        "NAME": os.environ.get("POSTGRES_DATABASE"),
+        "PASSWORD": os.environ.get("POSTGRES_PASSWORD"),
         "HOST": os.environ.get("DB_SERVICE_HOST"),
         "PORT": os.environ.get("DB_SERVICE_PORT"),
     }

sample-applications/vehicle-dealership-sample-app/VehicleInventoryApp/VehicleInventoryApp/settings.py

@@ -1,4 +1,4 @@
 Django~=5.0
 requests~=2.31.0
-mysqlclient~=2.2.1
+psycopg2~=2.9.9
 python-dotenv~=1.0.0

sample-applications/vehicle-dealership-sample-app/VehicleInventoryApp/requirements.txt

@@ -3,24 +3,24 @@
 version: '3'
 services:
   db:
-    image: mysql:8.0
+    image: postgres:14.0
     container_name: vehicle_inventory_db
     restart: always
     volumes:
-      - data:/var/lib/mysql
+      - data:/var/lib/postgres
     environment:
-      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
-      MYSQL_DATABASE: ${MYSQL_DATABASE}
-      MYSQL_USER: ${MYSQL_USER}
-      MYSQL_PASSWORD: ${MYSQL_PASSWORD}
+      POSTGRES_DB: ${POSTGRES_DATABASE}
+      POSTGRES_USER: ${POSTGRES_USER}
+      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
+      PGUSER: ${POSTGRES_USER}
     ports:
-      - "3306:3306"
+      - "5432:5432"
     healthcheck:
-      test: ["CMD", "mysql", "-h", "localhost", "-u", "root", "-p${MYSQL_PASSWORD}", "-e", "SELECT 1"]
+      test: ["CMD", "pg_isready", "-d", "vehicle_inventory", "-U", "djangouser"]
       timeout: 20s
       retries: 10
     expose:
-      - 3306
+      - 5432
 
   vehicle-inventory-backend:
     image: pythonsampleapp/vehicle-inventory-service
@@ -44,13 +44,17 @@ services:
       context: .
       dockerfile: ImageServiceApp/Dockerfile
     container_name: image-service-backend
-    command: sh -c "python3 manage.py migrate --noinput && python3 manage.py collectstatic --noinput && python3 manage.py runserver 0.0.0.0:8000"
+    command: sh -c "python3 manage.py runserver 0.0.0.0:8000"
     restart: always
     volumes:
       - ./ImageServiceApp/.:/image-service-app
       - $HOME/.aws/credentials:/home/app/.aws/credentials:ro
     ports:
       - "8000:8000"
+    depends_on:
+       db:
+          condition: service_healthy
+          
 volumes:
   data:
 

sample-applications/vehicle-dealership-sample-app/docker-compose.yaml

@@ -35,13 +35,11 @@ spec:
           image: ${REPOSITORY_PREFIX}/pythonsampleapp/vehicle-inventory-service:latest
           name: vehicle-inventory-backend
           env:
-            - name: MYSQL_DATABASE
+            - name: POSTGRES_DATABASE
               value: vehicle_inventory
-            - name: MYSQL_PASSWORD
-              value: ${MYSQL_PASSWORD}
-            - name: MYSQL_ROOT_PASSWORD
-              value: ${MYSQL_PASSWORD}
-            - name: MYSQL_USER
+            - name: POSTGRES_PASSWORD
+              value: ${POSTGRES_PASSWORD}
+            - name: POSTGRES_USER
               value: djangouser
           imagePullPolicy: Always
           ports:

sample-applications/vehicle-dealership-sample-app/eks/backend-deployment.yaml

@@ -29,36 +29,31 @@ spec:
     spec:
       containers:
         - env:
-            - name: MYSQL_DATABASE
+            - name: POSTGRES_DB
               value: vehicle_inventory
-            - name: MYSQL_PASSWORD
-              value: ${MYSQL_PASSWORD}
-            - name: MYSQL_ROOT_PASSWORD
-              value: ${MYSQL_PASSWORD}
-            - name: MYSQL_USER
+            - name: POSTGRES_PASSWORD
+              value: ${POSTGRES_PASSWORD}
+            - name: POSTGRES_USER
               value: djangouser
-          image: mysql:8.0
+          image: postgres:14.0
           livenessProbe:
             exec:
               command:
-                - mysql
-                - -h
-                - localhost
-                - -u
-                - root
-                - -p${MYSQL_PASSWORD}
-                - -e
-                - SELECT 1
+                - pg_isready
+                - -d
+                - vehicle_inventory
+                - -U
+                - djangouser
             failureThreshold: 10
             timeoutSeconds: 20
           name: vehicle-inventory-db
           ports:
-            - containerPort: 3306
-              hostPort: 3306
+            - containerPort: 5432
+              hostPort: 5432
               protocol: TCP
           resources: {}
           volumeMounts:
-            - mountPath: /var/lib/mysql
+            - mountPath: /var/lib/postgres
               name: data
       restartPolicy: Always
       volumes:

sample-applications/vehicle-dealership-sample-app/eks/db-deployment.yaml

@@ -12,7 +12,7 @@ metadata:
   name: db
 spec:
   ports:
-    - port: 3306
+    - port: 5432
   selector:
     io.kompose.service: db
 status:

sample-applications/vehicle-dealership-sample-app/eks/db-service.yaml

@@ -31,7 +31,7 @@ spec:
         - args:
             - sh
             - -c
-            - python3 manage.py migrate --noinput; python3 manage.py collectstatic --noinput; python3 manage.py runserver 0.0.0.0:8000
+            - python3 manage.py runserver 0.0.0.0:8000
           image: ${REPOSITORY_PREFIX}/pythonsampleapp/image-service:latest
           name: image-service-backend
           imagePullPolicy: Always
@@ -43,5 +43,11 @@ spec:
           env:
             - name: S3_BUCKET
               value: ${S3_BUCKET}
+            - name: POSTGRES_DATABASE
+              value: vehicle_inventory
+            - name: POSTGRES_PASSWORD
+              value: ${POSTGRES_PASSWORD}
+            - name: POSTGRES_USER
+              value: djangouser
       restartPolicy: Always
 status: {}

sample-applications/vehicle-dealership-sample-app/eks/image-backend-deployment.yaml

@@ -23,23 +23,26 @@ rm VehicleInventoryApp/.env
 rm ImageServiceApp/.env
 rm .env
 
-echo "MYSQL_ROOT_PASSWORD=${password}" >> VehicleInventoryApp/.env
-echo "MYSQL_DATABASE=vehicle_inventory" >> VehicleInventoryApp/.env
-echo "MYSQL_USER=djangouser" >> VehicleInventoryApp/.env
-echo "MYSQL_PASSWORD=${password}" >> VehicleInventoryApp/.env
+echo "POSTGRES_DATABASE=vehicle_inventory" >> VehicleInventoryApp/.env
+echo "POSTGRES_USER=djangouser" >> VehicleInventoryApp/.env
+echo "POSTGRES_PASSWORD=${password}" >> VehicleInventoryApp/.env
 echo "DB_SERVICE_HOST=db" >> VehicleInventoryApp/.env
-echo "DB_SERVICE_PORT=3306" >> VehicleInventoryApp/.env
+echo "DB_SERVICE_PORT=5432" >> VehicleInventoryApp/.env
 echo "IMAGE_BACKEND_SERVICE_HOST=image-service-backend" >> VehicleInventoryApp/.env
 echo "IMAGE_BACKEND_SERVICE_PORT=8000" >> VehicleInventoryApp/.env
 
+echo "POSTGRES_DATABASE=vehicle_inventory" >> ImageServiceApp/.env
+echo "POSTGRES_USER=djangouser" >> ImageServiceApp/.env
+echo "POSTGRES_PASSWORD=${password}" >> ImageServiceApp/.env
+echo "DB_SERVICE_HOST=db" >> ImageServiceApp/.env
+echo "DB_SERVICE_PORT=5432" >> ImageServiceApp/.env
 echo "AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}" >> ImageServiceApp/.env
 echo "AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}" >> ImageServiceApp/.env
 echo "AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN}" >> ImageServiceApp/.env
 echo "S3_BUCKET=${s3_bucket}" >> ImageServiceApp/.env
 
-echo "MYSQL_ROOT_PASSWORD=${password}" >> .env
-echo "MYSQL_DATABASE=vehicle_inventory" >> .env
-echo "MYSQL_USER=djangouser" >> .env
-echo "MYSQL_PASSWORD=${password}" >> .env
+echo "POSTGRES_DATABASE=vehicle_inventory" >> .env
+echo "POSTGRES_USER=djangouser" >> .env
+echo "POSTGRES_PASSWORD=${password}" >> .env
 
 docker-compose up --build

sample-applications/vehicle-dealership-sample-app/local_script.sh

@@ -14,10 +14,9 @@ rm ImageServiceApp/.env
 touch ImageServiceApp/.env
 
 export REPOSITORY_PREFIX=${account}.dkr.ecr.$region.amazonaws.com
-export MYSQL_ROOT_PASSWORD=${password}
-export MYSQL_DATABASE=vehicle_inventory
-export MYSQL_USER=djangouser
-export MYSQL_PASSWORD=${password}
+export POSTGRES_DATABASE=vehicle_inventory
+export POSTGRES_USER=djangouser
+export POSTGRES_PASSWORD=${password}
 export S3_BUCKET=${s3_bucket}
 
 docker-compose build

sample-applications/vehicle-dealership-sample-app/script.sh

@@ -14,11 +14,11 @@ then
 else 
     for config in $(ls ./eks/*.yaml)
     do
-        sed  -e 's#\${REPOSITORY_PREFIX}'"#${REPOSITORY_PREFIX}#g" -e 's#\${MYSQL_PASSWORD}'"#${MYSQL_PASSWORD}#g" -e 's#\${S3_BUCKET}'"#${S3_BUCKET}#g" ${config} | kubectl ${OPERATION} -f -
+        sed  -e 's#\${REPOSITORY_PREFIX}'"#${REPOSITORY_PREFIX}#g" -e 's#\${POSTGRES_PASSWORD}'"#${POSTGRES_PASSWORD}#g" -e 's#\${S3_BUCKET}'"#${S3_BUCKET}#g" ${config} | kubectl ${OPERATION} -f -
     done
 
     for config in $(ls ./eks/k8s-nginx-ingress/*.yaml)
     do
-        sed  -e 's#\${REPOSITORY_PREFIX}'"#${REPOSITORY_PREFIX}#g" -e 's#\${MYSQL_PASSWORD}'"#${MYSQL_PASSWORD}#g" -e 's#\${S3_BUCKET}'"#${S3_BUCKET}#g" ${config} | kubectl ${OPERATION} -f -
+        sed  -e 's#\${REPOSITORY_PREFIX}'"#${REPOSITORY_PREFIX}#g" -e 's#\${POSTGRES_PASSWORD}'"#${POSTGRES_PASSWORD}#g" -e 's#\${S3_BUCKET}'"#${S3_BUCKET}#g" ${config} | kubectl ${OPERATION} -f -
     done
 fi