Deployment

Local / Development deploy

deploy using runserver (DEBUG=True)

On your new droplet log in a root and create a new file env-setup with these variables. The env-setup file will be deleted once the install is complete. Update the values as required, e.g. fill in the correct password for MYSQL_USER_PASSWORD.

$ nano $HOME/env-setup

export APP_HOST=gaborone.uat.ambition.clinicedc.org
export APP_NAME=ambition
export APP_USER=ambition
export APP_FOLDER=app
export DJANGO_EXPORT_FOLDER=/home/$APP_USER/export
export DJANGO_ETC_FOLDER=/home/$APP_USER/.etc/$APP_NAME
export DJANGO_KEY_FOLDER=$DJANGO_ETC_FOLDER/crypto_fields
export DJANGO_LOG_FOLDER=/home/$APP_USER/log
export DJANGO_STATIC_FOLDER=/home/$APP_USER/static
export MYSQL_DATABASE=ambition
export MYSQL_USER=edc
export MYSQL_USER_PASSWORD=password  # need a password
export REPO=https://github.com/ambition-trial/ambition.git $APP_FOLDER
export VENV=ambition

Source the file env_setup.

$ source $HOME/env-setup
$ echo $APP_FOLDER

#output
app

As root, create the non-root account and setup keys for key-based authentication:

# as root, create a non-root account and set up keys
adduser $APP_USER
# add to sudo
usermod -aG sudo $APP_USER
# copy keys to the new account
mkdir /home/$APP_USER/.ssh
cp .ssh/authorized_keys /home/$APP_USER/.ssh
chown $APP_USER:$APP_USER -R /home/$APP_USER/.ssh
chmod 700 /home/$APP_USER/.ssh
chmod 600 /home/$APP_USER/.ssh/authorized_keys

Login as non-root account ambition and install dependencies.

sudo apt-get update
sudo apt-get -y upgrade
sudo apt-get -y install mysql-server-5.7 # if needed
sudo apt-get -y install mysql-client-5.7 libmysqlclient-dev libcups2-dev ipython3 python3-pip python3-dev python3-venv python3-cups python3-venv redis-server nginx curl wbritish

As of Feb 2019 ambition and edc modules are tested on python versions 3.7 and above. Ubuntu 18.04 LTS uses python 3.6.7.

To upgrade to python3.7:

sudo apt-get update
sudo apt-get install build-essential libpq-dev libssl-dev openssl libffi-dev zlib1g-dev python3.7 python3.7-venv python3-pip python3.7-dev

Prepare mysql

secure MySQL installation:

sudo mysql_secure_installation

load timezones into MySQL:

mysql_tzinfo_to_sql /usr/share/zoneinfo | sudo mysql mysql

create a MySQL database for the app:

echo "CREATE DATABASE $MYSQL_DATABASE CHARACTER SET utf8;" | sudo mysql
echo "CREATE DATABASE $MYSQL_DATABASE CHARACTER SET utf8;" | mysql -u root -p

Create a MySQL account, other than root, to be used on MySQL DB server:

echo "CREATE USER '$MYSQL_USER'@'localhost' IDENTIFIED BY '$MYSQL_USER_PASSWORD';FLUSH PRIVILEGES;" | sudo mysql
echo "CREATE USER '$MYSQL_USER'@'localhost' IDENTIFIED BY '$MYSQL_USER_PASSWORD';FLUSH PRIVILEGES;" | mysql -u root -p
echo "GRANT ALL PRIVILEGES ON *.* TO '$MYSQL_USER'@'localhost' WITH GRANT OPTION;" | sudo mysql
echo "GRANT ALL PRIVILEGES ON *.* TO '$MYSQL_USER'@'localhost' WITH GRANT OPTION;" | mysql -u root -p

Note: if on docker localhost will not work so use the docker IP or ‘%’

confirm new account can login to new DB:

echo "mysql -u $MYSQL_USER -p $MYSQL_DATABASE" | mysql

Create separate MySQL accounts to log in from each remote django host

i.e. for each $REMOTE_HOST:

set +o history  # temporarily disable bash history (so as not to save pw)
export MYSQL_DATABASE=db_name
export REMOTE_MYSQL_USER=edc-<host-desc>
export REMOTE_MYSQL_USER_PASSWORD=different_p@ssw0rd
export REMOTE_HOST=some_host

echo "CREATE USER '$REMOTE_MYSQL_USER'@'$REMOTE_HOST' IDENTIFIED BY '$REMOTE_MYSQL_USER_PASSWORD';FLUSH PRIVILEGES;" | mysql
echo "GRANT ALL PRIVILEGES ON \`$MYSQL_DATABASE\`.* TO '$REMOTE_MYSQL_USER'@'$REMOTE_HOST';" | mysql

unset REMOTE_MYSQL_USER_PASSWORD  # delete pw variable
set -o history  # re-enable bash history

Show users:

mysql -Bse 'select user, host from user;'

Confirm each REMOTE_MYSQL_USER can log in from their respective host:

echo "mysql -u $REMOTE_MYSQL_USER -p $MYSQL_DATABASE" | mysql

if you delete any user don’t forget to FLUSH PRIVILEGES;

Prepare the app and the virtualenv

login as non-root account ambition

create and source the virtualenv ambition:

python3.7 -m venv ~/.venvs/$VENV

activate the environment:

source ~/.venvs/$VENV/bin/activate

Important: Confirm you are in your new virtualenv before continuing.

Now git pull the app:

cd ~/ && git clone $REPO $APP_FOLDER

Install requirements into the virtualenv:

cd ~/app \
&& pip install --no-cache-dir -r requirements/stable.txt \
&& pip install --no-cache-dir -e .

This seems to not get installed from edc-base, so run it separately:

pip install --no-cache-dir django[argon2]

Now install the .env file. The .env file is not part of the REPO. Open another terminal and copy the apps .env file to the app root assumed coming from your machine, for example:

echo "scp ~/source/ambition/.envs/.local $APP_USER@$APP_HOST:~/app/.env"
scp ~/source/ambition/.envs/.local <app_user>@<app_host>:~/app/.env

There is also a sample .env file in the repo. To use that:

cp $HOME/$APP_FOLDER/env.sample $HOME/$APP_FOLDER/.env

Edit the .env file as needed:

nano ~/app/.env

Set permissions:

chmod 600 ~/app/.env

IMPORTANT: inspect the .env variables and edit as required

NOTE: DATABASE_URL password needs to be escaped if it contains special characters.:

import urllib

urllib.parse.quote('my_password$@')

python -c 'from urllib import parse; print(parse.quote("my_password$@"))'

# output 'my_password%24%40'

See https://github.com/joke2k/django-environ/blob/develop/README.rst#tips

Create the export and static folders:

# check the values
echo "DJANGO_ETC_FOLDER=$DJANGO_ETC_FOLDER" \
&& echo "DJANGO_EXPORT_FOLDER=$DJANGO_EXPORT_FOLDER" \
&& echo "DJANGO_KEY_FOLDER=$DJANGO_KEY_FOLDER" \
&& echo "DJANGO_LOG_FOLDER=$DJANGO_LOG_FOLDER" \
&& echo "DJANGO_STATIC_FOLDER=$DJANGO_STATIC_FOLDER"
# create the folders
mkdir -p $DJANGO_ETC_FOLDER \
&& mkdir -p $DJANGO_EXPORT_FOLDER \
&& mkdir -p $DJANGO_KEY_FOLDER \
&& mkdir -p $DJANGO_LOG_FOLDER \
&& mkdir -p $DJANGO_STATIC_FOLDER

Copy encryption keys into DJANGO_KEY_FOLDER . These are also not included in the REPO and are assumed to come from you. (or if just testing set DJANGO_AUTO_CREATE_KEYS=True):

echo "scp user* ambition@$APP_HOST:$DJANGO_KEY_FOLDER/"

Note: If you are setting up a test environment and you set DJANGO_AUTO_CREATE_KEYS=False in .env to create test keys, you need to set it to False.

Check:

cd ~/app \
&& python manage.py check

Output:

django.db.utils.ProgrammingError: (1146, "Table 'ambition.edc_lab_boxtype' doesn't exist")

Which is expected since you have not migrated the DB yet.

Note: Check the database server IP and the firewall config on both your server and the DB server if you get::

ERROR! (2003, "Can't connect to MySQL server on '10.131.71.175' (110)")

See also the README on database setup.

Now migrate (takes a while …):

python manage.py migrate

Collect static, note that if AWS_ENABLED, will test the connection:

python manage.py collectstatic

# note: you should purge CDN cache if required.

Import the holiday file, check the .env to make sure this is correct:

python manage.py import_holidays

Import randomization list file, check the .env to make sure this is correct!! Note, you need to manually copy a randomization list to DJANGO_ETC_FOLDER where the file name is the same as DJANGO_RANDOMIZATION_LIST in .env.:

python manage.py import_randomization_list

Now if you run check again there should not be any errors.:

python manage.py check

Output:

"System check identified no issues (0 silenced)."

Create a super user:

python manage.py createsuperuser

Now try runserver. be sure PORT 8000 is open on your server. If you get “Invalid HTTP_HOST header: …” check the .env file DJANGO_ALLOWED_HOSTS and add your DOMAIN or IP.:

python manage.py runserver 0.0.0.0:8000

IMPORTANT: If you plan to continue with the next section, don’t enter any data.

Production / Live deploy

using NGINX/GUNICORN (DEBUG=False)

Deploy onto an Ubuntu 18.04 server

(continued from above)

Since DEBUG=True above, some variables from the .env file were ignored.

Now set DEBUG=False in the .env file

With DEBUG=False, the app now looks for the encryption keys in DJANGO_KEY_FOLDER.

create DJANGO_KEY_FOLDER:

mkdir -p $DJANGO_KEY_FOLDER

Copy your production keys to the DJANGO_KEY_FOLDER. Echo command to use to copy keys to this host:

echo "scp user* $APP_USER@$APP_HOST:$DJANGO_KEY_FOLDER"

Open another terminal and use the above SCP command to copy encryption keys to this folder from your encryption key folder

for example:

scp user* ambition@206.189.16.89:~/.etc/ambition/crypto_fields

If you run runserver now, the pages will be rendered without static files as expected.:

python manage.py runserver 0.0.0.0:8000

Notice that the app created django_crypto_fields file:

ls -la $DJANGO_ETC_FOLDER

Set up gunicorn

See separate document README_deploy.rst.

Set up NGINX

See separate document README_deploy.rst.

UAT / Testing deploy

using NGINX/GUNICORN (DEBUG=True)

Deploy onto an Ubuntu 18.04 server

(continued from Local / Development deploy)

Log in as user uat.:

ssh uat@....

As user uat create a new virtualenv in the same way as above:

python3.7 -m venv ~/.venvs/ambition

source ~/.venvs/ambition/bin/activate

cd ~/app \
&& pip install --no-cache-dir -U -r requirements/stable.txt \
&& pip install -e .

Use the .env` variables to configure a system as a UAT server. Copy the .env from the LIVE server and edit:

sed -i -e s/DJANGO_LIVE_SYSTEM=True/DJANGO_LIVE_SYSTEM=False/g .env
sed -i -e s/ambition_production/ambition_uat/g .env
sed -i -e 's/\/home\/ambition/\/home\/uat/g' .env
sed -i -e s/DJANGO_RANDOMIZATION_LIST_FILE=randomization_list.csv/DJANGO_RANDOMIZATION_LIST_FILE=test_randomization_list.csv/g .env
sed -i -e 's/AWS_LOCATION=ambition\/static/AWS_LOCATION=ambition_uat\/static/g' .env
sed -i -e 's/\.ambition\.clinicedc\.org/\.uat\.ambition\.clinicedc\.org/g' .env

Diff:

diff -y /home/uat/app/.env /home/ambition/app/.env

Check DATABASE_URL points to ambition_uat:

cat .env | grep DATABASE_URL

Copy keys from LIVE:

cp /home/ambition/.etc/ambition/crypto_fields/user* /home/uat/.etc/ambition/crypto_fields/

Check:

python manage.py check

If you followed all the steps to setup the LIVE system, then restarting gunicorn and nginx should be all that is necessary:

sudo systemctl daemon-reload \
&& sudo systemctl restart gunicorn-uat

Reference

Deploy onto an Ubuntu 18.04 server

Misc