This post is part of a series of posts where I share my experiences developing, testing, and debugging the new implementation of my website, including this blog and my home page.

When I developed my new website, I used OrbStack to run a virtual developer version of the site on my development computer (the host). I recommend this setup since what you run in development can be very close to the site's final execution environment. I use terminal commands in the text below to do the OrbStack operations. I could have used the OrbStack application, but I find using terminal commands more efficient and easier to explain.

Create and prepare the OrbStack virtual development computer

I use OrbStack to run the latest stable version of Debian Linux. The virtual development Debian computer in OrbStack can simply be created by the following OrbStack management command in the terminal prompt on the host:

orbctl create debian test-debian

With this command, a Debian Linux machine named test-debian is created and started. In the OrbStack application OrbStack.app on my Mac, the newly created Debian Linux machine will then be shown like this:

If you later need to stop and start the Debian virtual machine, you can either use the OrbStack application pictured above or the following commands in your terminal prompt:

orbctl stop test-debian
orbctl start test-debian

I want the software on the virtual developer OrbStack machine to be as close as possible to how it will be on the live Linode machine I use for my site. Consequently, I ensure I can access the test machine using SSH instead of using the nice functionality OrbStack provides to access and manipulate the virtual machine directly. But to be able to do this, I have to install the SSH-server and prepare the machine using some OrbStack magic. With OrbStack and the orbctl command, we can do commands on the virtual machine but still have access to the files on the host machine. When doing the following command, the debian-pre-prepare.sh file is in the directory on the host machine where the command is performed from, but the sudo bash command is executed on the OrbStack virtual machine:

orbctl -m test-debian \
    sudo UNAME="webuser" UNPW="secret-password" \
    bash debian-pre-prepare.sh

Replace webuser and secret-password in the command above with the username of the user running the web platform and the password you want to give to this user. The script debian-pre-prepare.sh is executed as the superuser (root) on the virtual machine. An example of this script could be the following:

useradd -m $UNAME
echo "$UNAME:$UNPW" | chpasswd
adduser $UNAME sudo
chsh --shell /bin/bash $UNAME
apt-get -y install openssh-server

The script adds a new user (webuser) to the virtual machine and sets a password for the user. The user is then added to the superuser group, and the user's default shell is changed to Bash. Finally, the SSH server is installed. When we have done this, we can continue without the OrbStack magic and depend only on SSH to access the virtual machine from the host. On my Mac, the virtual machine can now be accessed as test-debian.orb.local, and I can log in on it with SHH using this command:

ssh webuser@test-debian.orb.local

I expect the user (developer) already created an RSA SSH-key pair. If not, read the ssh-keygen command documentation. The first thing I do is to add the public SSH key to the authorized keys of the web user and the superuser on the virtual machine. Then, I do not have to type the password whenever I use SSH to access the virtual machine. I copy the public SSH key of the developer (me) to the virtual machine with scp and then append this public key to the authorized_keys file of the web-user and the superuser (root):

scp ~/.ssh/id_rsa.pub webuser@test-debian.orb.local:.
ssh webuser@test-debian.orb.local \
    mkdir -m 700 .ssh
ssh webuser@test-debian.orb.local \
    cat id_rsa.pub >> .ssh/authorized_keys
ssh webuser@test-debian.orb.local \
    sudo -i mkdir -m 700 .ssh
ssh webuser@test-debian.orb.local \
    sudo -i cat /home/webuser/id_rsa.pub >> .ssh/authorized_keys

You should now be able to use SSH to log in to and perform commands on the virtual machine without typing in the password. Test if that works with these two commands (performs the command pwd as the web user and the superuser):

ssh webuser@test-debian.orb.local pwd
ssh root@test-debian.orb.local pwd

You should also be able to copy files to the virtual machine with scp without typing in the password.

Setup of the virtual OrbStack computer

The actual setup of the web server will be similar to the setup of the site's live server. More details on the setup of the live web server can be found in this blog post: A Debain/NGNIX/Gunicorn setup for web.py.

Instead of repeating the setup described for the live web server, we will in this post focus on creating a developer version of the site using the data from the live site (if it already exists). You can then test and develop a new or improved site implementation in the virtual OrbStack computer before it is uploaded to the live site. We find the following differences between the developer version and the live version of the site:

• The locally accessible domain of the virtual OrbStack computer will be orb.local. For example, if you configure a website to be accessed at the host www, you can use the URL “http://www.orb.local/” to access the website on the virtual OrbStack computer.
• We will not enable HTTPS on the virtual OrbStack computer. Therefore, the test websites will only be accessible through HTTP (and not HTTPS).
• The virtual OrbStack computer has no side effects on the actual data of the live web server, and the whole virtual computer can be deleted without any real data loss.
• As much as possible, the content of the developer version is fetched from the live version of the site. This includes copying the database content and other assets (e.g. images) from the live to the developer site.

The next two things to do on the virtual OrbStack Debian computer are the Initial configuration step and the Install the software needed step. I recommend that you create a script for each of these steps that you copy to the virtual OrbStack computer (with scp) and then perform the script as root on the virtual OrbStack computer:

scp initial-conf.sh root@test-debian.orb.local:.
ssh root@test-debian.orb.local bash initial-conf.sh

This will copy the script initial-conf.sh to the home directory of root on the virtual OrbStack computer and then execute the script using bash. An example of the initial-conf.sh script could be this:

echo "LANG=en_US.UTF-8" >> /etc/environment
echo "LC_ALL=en_US.UTF-8" >> /etc/environment
ln -fs /usr/share/zoneinfo/Europe/Oslo /etc/localtime
dpkg-reconfigure -f noninteractive tzdata

Install the web applications on the virtual OrbStack computer

After these two steps are performed, we can do the next step: Prepare the virtual Python environment for the web applications. Again, I recommend creating a script that performs these operations, copying the script to the virtual OrbStack computer, and then performing the script as the web user on the virtual OrbStack computer:

scp mk-pyenv.sh webuser@test-debian.orb.local:.
ssh webuser@test-debian.orb.local bash mk-pyenv.sh

This will copy the script mk-pyenv.sh to the home directory of webuser on the virtual OrbStack computer and then execute the script using bash. An example of the mk-pyenv.sh script could be this:

mkdir -p web/templates
cd web
python3 -m venv myweb-py
source myweb-py/bin/activate
pip3 install --upgrade pip
pip3 install web.py mariadb gunicorn
deactivate

Depending on your selected web platform and the Python modules you need, the actual Python modules installed with pip might differ. In this example, web.py, MariaDB and Gunicorn is used.

How to configure the web server is described in detail here: Configure the web server. On the virtual OrbStack computer, you should replace your-domain.something in this description with orb.local. On the host, we have a copy of the two files home.service and home from the «Configure the web server» description, and we create the following script conf-web-server.sh based on this description:

mkdir -p /var/www/static
mkdir /var/www/static/css
mkdir /var/www/static/js
mkdir /var/www/static/icons
mkdir /var/www/images
mkdir /var/www/dist
chmod a+w /var/www/images
cp home.service /etc/systemd/system/
cp home /etc/nginx/sites-available/
ln -s /etc/nginx/sites-available/home /etc/nginx/sites-enabled/home
systemctl restart nginx
systemctl daemon-reload
systemctl restart home

Then we can do this on the host to configure the web server on the virtual OrbStack computer:

scp home.service root@test-debian.orb.local:.
scp home root@test-debian.orb.local:.
scp conf-web-server.sh root@test-debian.orb.local:.
ssh root@test-debian.orb.local bash conf-web-server.sh

The final step is to install the code of the web applications on virtual OrbStack computer. You can follow the steps described here: Install the web application files. Also, here, replace your-domain.something with orb.local. You should now be able to access the web application (home page) with this URL:

http://www.orb.local/

Use the content from the live site

With a site's content, we mean the text, pictures, files, and similar that are not part of the implementation of the site but are the actual content meant for its visitors. For my example, all content is either in the database, images stored in the “/var/www/images” folder, or files in the “/var/www/dist” folder. This is also what I want to save when I do a backup of the live site's content.

To dump the content of the database, we will use the mysqldump tool. On a live site server where webdb is the database, dbuser is the username for the database user, and dbpasswd is the password of the database user, you can do the following command (you should of course replace the name of the database, the username and the password with the correct values for your site):

mysqldump -u dbuser -p'dbpasswd' webdb > webdb-backup.sql

To get the dumped content of this database installed on a database on the virtual OrbStack computer, we first need to copy the webdb-backup.sql file to the virtual OrbStack computer. Then, we can perform the steps in this init-webdb.sh script file:

service mariadb start
now=`date +"%Y%m%d"`
mysqldump -u dbuser -p'dbpasswd' webdb > "$now-webdb-backup.sql"
mysql -e "drop database if exists webdb;"
mysql -e "create database webdb;"
mysql -e "grant all on webdb.* to 'dbuser' identified by 'dbpasswd';"
mysql -e "grant all on webdb.* to 'dbuser'@'localhost' identified by 'dbpasswd';"
mysql -u dbuser -p'dbpasswd' -D webdb < webdb-backup.sql

The first line ensures that the database MariaDB is running. In the second line, we create a text string with today's date. In the third line, we dumb a backup of the existing database webdb to a file with today's date in the name. This might fail if the database is not created yet. I do it to ensure we do not lose any data, but since this database is used for testing only, we do not expect to have any important data that can be lost in this database. We can therefore skip this step. In lines 4 and 5, we delete and recreate the database webdb. In lines 6 and 7 we give the user dbuser access to the database with the password dbpasswd. In the last line we load the backup from the live site into the database on the virtual OrbStack computer.

To summarise, we copy the database backup and the script (with line 2 and 3 removed from the script) to the virtual OrbStack computer and then execute the script:

scp webdb-backup.sql root@test-debian.orb.local:.
scp init-webdb.sh root@test-debian.orb.local:.
ssh root@test-debian.orb.local bash init-webdb.sh

The webdb database on the virtual OrbStack computer will now be a complete copy of the webdb database on the live site. The remaining content, the images, and the files in the dist folder can be copied to the virtual OrbStack computer directly with scp. For example, if the live site is www.your-domain.something, we can do the following two commands as root on the virtual OrbStack computer:

scp root@www.your-domain.something:/var/www/images/*.* /var/www/images/
scp -r root@www.your-domain.something:/var/www/dist/* /var/www/dist/

An image illustration.png (if it exists) can now be accessed directly from the virtual OrbStack computer with the following URL:

http://www.orb.local/images/illustration.png

The dist folder can be browsed and traversed using this URL:

http://www.orb.local/dist/