View on GitHub

Tista Covid-19 Tracker

Simple application for tracking key pandemic data at the national, state, and county levels.

Tista COVID-19 Tracker

The repository contains all source code for the Tista COVID-19 Tracker Web Application.

Data sources

The list of the data sources what we are using for the application contains:

Other resources

Environment

Postgresql Database

Data collector service is using Postgresql database to store all information.

Data exporter service is using Postresql database to transform, analyze, and export Covid-19 data.

AWS S3 Buckets

data.tistatech.com

Data layer for Covid-19 Tracker application.

covid19.tistatech.com

Web application hosting.

Covid-19 Tracker Application Web servers

All Web server for Covid-19 Tracker Web Application are hosted on AWS S3 buckets.

Quick Start

The steps how to setup Tista COVID-19 Tracker Web Application:

  1. Get source code from the Github: https://github.com/TISTATechnologies/cv19.
  2. Create Postgresql database.
  3. Create AWS S3 bucket for data and setup static site hosting on it (ex: cv19data)
  4. Create AWS S3 bucket for Web Application and setup static site hosting on it (ex: cv19web).
  5. Create configuration file ~/cv19.dev.conf, use a .env.sample file as an example
  6. Initialize a new database for cv19 project: CV19_ENV=dev ./data/covid-database/migrate-db.sh init
  7. Collect the data: CV19_ENV=dev ./scripts/start-pull-data-services.sh
  8. To see historical and trends data we need to collect more data on each days: 
    cd <project dir>
for i in $(seq 90 -1 1); do
 dt=$(date +"%Y-%m-%d" -d "-${i} days")
 yes | ./scripts/start-pull-data-services.sh ${dt} || exit 1
done
  9. Export collected data to the json files: CV19_ENV=dev ./scripts/start-export-data-services.sh
  10. Upload all json files into the AWS S3 bucket with data: CV19_ENV=dev ./upload-covid-data-to-s3-bucket.sh
  11. Now you should have latest Covid-19 data on your AWS S3 bucket: http://cv19data.s3-website.us-east-1.amazonaws.com/covid/v1/daily/latest/us/dc/11001.json
  12. Build a web application and deploy it to the S3 bucket: 
    export CV19_ENV=dev
cd webapp
npm install
npm run build
npm run deploy
  13. Now you can open the Tista COVID-19 Tracker site: http://cv19web.s3-website.us-east-1.amazonaws.com

Development

We are using company github TISTATechnologies to store source code.

Get source code:

git clone git@github.com:TISTATechnologies/cv19.git

Requirements

  1. Nodejs 12+
  2. Postgresql Client 10+
  3. Python 3.8+
  4. awscli

Architecture

Covid-19 Tracker application architecture

Code structure

Configuration

Before build and deploy the application, you need to specify configuration using system environment variables or specify it inside a .env file at the root project’s level.

Please look into the .env.sample for details.

From other side you can create ~/cv19.xxx.conf configuration file inside your home directory and specify this configuration file with a system environment variable CV19_ENV=xxx.

Note: .env file has more priority than ~/cv19.xxx.conf file.

Release

  1. Freeze all future changes (new functionality or fixes) should be inside the develop branch.
  2. The develop branch should be at the top of the master branch 
    git checkout master
git pull
git checkout develop
git rebase master
git push origin develop
  3. Merge all changes from the develop branch into the master branch. Don’t make a loops with auto merge. 
    git co master
git merge develop
  4. Create new tag with the new version 
    git tag <new version>           # example: git tag 1.5.0
git push origin <new version>   # example: git push origin 1.5.0
  5. Push master branch to the remote 
    git push origin master
  6. Now you are ready to deploy a new release version to the Production.

NOTE: You must tag the new version release source code in the master branch before the build because the WebApp build script is using git tag to generate the _version.json file.

Build/deploy Web Application manually

See Configuration section before manual deploy.

cd <project directory>/webapp
npm install && \
npm run build && \
yes | npm run deploy

This script will install all required packages for WebApp, build React application and deploy it to the S3 bucket.

You can specify a configuration file ~/cv19.dev.conf and call this script:

export CV19_ENV=dev
cd <project directory>/webapp
npm install && \
npm run build && \
yes | npm run deploy

Create database

./data/covid-database/migrate-db.sh init

CI/CD

Github Actions are using to Build/Deploy/Test and to Collect/Test data.

Collect data

Export data

All exported data will be stored inside the ./build/covid/ directory.

Use a command ./scripts/upload-covid-data-to-s3-bucket.sh to deploy all data files from ./build/covid to the S3 Bucket.

NOTE: Don’t forget to remove all data inside the ./build/ directory which you are not planning to upload into the S3 Bucket.

Testing

Notes

Useful/Helper scripts

Collect all data for 90 days

cd <project dir>
for i in $(seq 90 -1 1); do
    dt=$(date +"%Y-%m-%d" -d "-${i} days")
    echo ""
    echo ""
    echo ""
    echo ""
    echo "==================================================================================="
    echo "|  PROCESSING   ${dt}                                                 |"
    echo "==================================================================================="
    yes | ./scripts/start-pull-data-services.sh ${dt} || exit 1
done

Export all data for 90 days

for i in $(seq 90 -1 1); do
    dt=$(date +"%Y-%m-%d" -d "-${i} days")
    yes | ./scripts/start-export-data-services.sh ${dt} || exit 1
done

Collect latest data and upload it to the S3 Bucket

This script can be executed by scheduler every day at 6:00 am to collect and export all new data.

rm  -rf ./build
yes | ./scripts/start-pull-data-services.sh \
&& yes | ./scripts/start-export-data-services.sh \
&& yes | ./scripts/upload-covid-data-to-s3-bucket.sh

Test collected and exported data

This script can be executed after the previous script to validate collected data.

cd tests/e2e
npm install
npm run generate
npm run test:database
npm run test:api

Data sources: Covid-19

License

The source code inside this repository are licensed under the Apache 2.0 License and the data content is licensed under the Creative Commons Attribution-ShareAlike 4.0 License.