Getting started with building apps using node and docker

Let me first start with my motivation to create this. I thought of exploring a few technologies and what better to build a simple sample. So, I will talk through an example of building a simple URL Shortener Service.

Though, I had some knowledge of working with node, express and mysql, I was totally new to things such as redis and docker. So, with limited prior knowledge, I started last weekend and spent mostly about an hour or so every day to get it to the state that it’s in right now.

The intent here is an introduction of these technologies. Hence I will not be going into the details but just touching the surface and creating a simple app.
So, let’s get started.

Here is what we will be covering—

1. Creating an express app with REST APIs and a minimalist UI.
2. Setting up database — MySql (through docker)
3. Introducing a cache — Redis (through docker)
4. Connecting the app to the database/ redis
5. Building a docker image for the app
6. Getting it all together using docker-compose

Source code in Github: https://github.com/sameerbhatt/url-shortener-app

Installing Node, Express
Installing node and express is pretty easy, just follow these respective links

1. Node — download and install from https://nodejs.org/
2. Express — https://expressjs.com/en/starter/installing.html

Either use —
npm install express --save
Or use the express generator -
npm install -g express-generator

Generating an App and setting up UI
I will use express generator to generate the app -

npx express-generator --no-view url-shortener-app
Note: Not generating any views to simplify things for now. We will anyways have a minimalistic UI

Test it out now by running the following and opening http://localhost:3000/ in the browser —

npm install
DEBUG=url-shortener-app:* npm start

You can use Visual Studio Code as the IDE, pretty good it is.
And very easy to run and debug the app to see what’s going on —

Select “Start Debugging”

Select “Node.js” environment

Creating UI

For UI, we will build a very basic one, using —

1. Bootstrap — https://getbootstrap.com/
2. axios for calling our APIs — https://axios-http.com/

Please refer to index.html for the details. I know, we can go into a lot of details/handling here and create awesome looking pages. However, that’s not the intent now 🙂

Sample UI

For calling the APIs using axios, here is a snippet —

axios.post('http://localhost:3000/api/link', {
url: inputURL // body
})
.then(function (response) {
    alert(response.data.shortenedURL);
})
.catch(function (error) {
    alert(error.data);
});

Let’s get the database up now

For database, we will use MySql and instead of setting up/installing on our machine, we will use it through a docker image.
Refer — https://hub.docker.com/_/mysql

// pull the mysql 8 image
docker pull mysql:8
// run - create a container out of the mysql image and run it
docker run -p 3306:3306 - name mysql-url-shortener -v ~/personal/mysql-data:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=root123 -d mysql:8

There are few very important options above, let’s go over them —

-p bind machine port to container port — your server will be exposed on port 3306
-v host_dir:container_dir — Host volumes to mount directories.
The -v ~/personal/mysql-data:/var/lib/mysql part of the command mounts the ~/personal/mysql-data directory from the underlying host system as /var/lib/mysql inside the container, where MySQL by default will write its data files. If you don’t use this option, all changes to your mysql databases will be lost, when the container is deleted.
-e MYSQL_ROOT_PASSWORD=root123 — this is setting the root password for mysql.

Creating tables in the database

With MySql up, let’s connect to it and explore/ create some tables.
You can use phpMyAdmin as well and connect to MySql but the below demonstrates much more.

// list all the running containers
docker ps
// access container's shell. replace CONTAINER_ID with the MySQL container_id from above
docker exec -it CONTAINER_ID bash
// connect to mysql
mysql - user=root - password
// create sample database and use it and create a Link table create database mydb;
use mydb;
create table if not exists Link(code varchar(8) primary key, url varchar(512) not null, INDEX(url));
// insert some dummy values to test
INSERT INTO Link VALUES('abcd123', 'https://www.abcd.com');

Connect to MySql from the node app

Goto the app directory and run the following to install mysql client on node —

npm install mysql2

Note: See why we use mysql2 and not mysql (took me quite a while to figure out) — https://stackoverflow.com/questions/50093144/mysql-8-0-client-does-not-support-authentication-protocol-requested-by-server/56509065#56509065

The connection code is pretty self explanatory here. You can refer to — https://expressjs.com/en/guide/database-integration.html#mysql
Here is a basic way to connect (refer db.js and index.js) —

var mysql = require('mysql2')
var pool = mysql.createPool({
    host: '127.0.0.1',
    user: 'root',
    password: 'root123',
    database: 'mydb'
});
pool.getConnection(function(err, connection){
    connection.query(SQL_STATEMENT, function (err, rows, fields) {
        ......
    }
});

Plugin-in the cache
For a cache, we will use Redis. Start Redis docker container —

docker run -p 6379:6379 - name redis-url-shortener -d redis:alpine

Install Redis client on node -

npm install - save redis

Things are pretty straightforward now.

• We just create a redis client (refer index.js)

const redis = require('redis');
const client = redis.createClient({
  host: '127.0.0.1',
  port: 6379
});

• Use get and set methods for managing the code:url mapping

client.get(code, function(err, response) {.....});
client.set(code, url);

Refer here for more details — https://docs.redislabs.com/latest/rs/references/client_references/client_nodejs/

Creating APIs

We would primarily need two APIs —

1. Take an input URL and generate/return a short link. Method: POST.
   Check the URL in the database, if present, return the short link.
   Else, generate a random 7 character string, insert into database, update cache and return.
2. Take a short link and redirect to the full URL. Method: GET
   Check in cache or database, if present 302 redirect.
   Else, return an 404 Not Found.
   Refer — https://expressjs.com/en/api.html#res.redirect

content-type for input/output: used json for now as it’s pretty much standard and can be extended if we need to add more fields.
Refer index.js for details.

Test out the App

So, the MySql and Redis container are running now. Test it out now by running the following and opening http://localhost:3000/ in the browser —

npm install
DEBUG=url-shortener-app:* npm start

Building a Docker image

Okay, so, now we have the entire system working. However, for a dev to setup on a machine, there are too many steps. Let’s try to create a docker image from your node app.
Note: You can also setup Jenkins and configure it to create a docker image automatically on commits to your code. We will try to do it manually here to get a better idea. This image can then be stored in any repository.
Create a Dockerfile in your app folder with the following content —

FROM node:14-alpine
# Create app directory
WORKDIR /usr/src/app
# Install app dependencies
# A wildcard is used to ensure both package.json AND package-lock.json are copied
COPY package*.json ./
RUN npm install
# Bundle app source
COPY . .
EXPOSE 3000
CMD ["npm", "start"]

The above should be quite self-explanatory and the comments would help. Though, more details can be found here — https://nodejs.org/en/docs/guides/nodejs-docker-webapp/

You can also use Visual Studio Code to generate it through a wizard — https://code.visualstudio.com/docs/containers/quickstart-node

Before building the image, make sure to change the MySql and Redis endpoint in your node app to MySql and Redis container name. That is, change the db.js and index.js files —

db.js
// from a docker container, use name of mysql container
var pool = mysql.createPool({
    host: 'mysql-url-shortener',
    user: 'root',
    password: 'root123',
    database: 'mydb'
});
index.js
// from a docker container, use name of redis container
const client = redis.createClient({
    host: 'redis-url-shortener',
    port: 6379
});

Let’s build the image and run it now —

// build image
docker build . -t url-shortener-app:1.0
// create a network
docker network create my-network
// run - create a container with the network created above
docker run -p 3000:3000 — network my-network -d url-shortener-app:1.0
// connect redis and mysql to the same network
docker network connect my-network REDIS_CONTAINER_ID
docker network connect my-network MYSQL_CONTAINER_ID

Creating a network and making all containers connect to it is very important. Remember, from node app we were connecting to MySql and Redis on localhost. However, with the node app running in a container, localhost points to the local container’s localhost. We have to make sure to have all containers connect to the same network and that ways, refer to MySql and Redis by their respective container names.

Getting everything together
Now that we have our mysql, redis and node app server images ready, to bring them together, we create a docker-compose.yml file and add the run configurations for each of our images as follows —
(yaml file alert: watch your indents/spaces)

version: '3.4'
services:
  urlshortenerapp:
    image: url-shortener-app:1.0
    ports:
      - 3000:3000
  mysql-url-shortener:
    image: mysql:8
    ports:
      - 3306:3306
    environment:
      - MYSQL_ROOT_PASSWORD=root123
    volumes:
      - ~/personal/mysql-data:/var/lib/mysql
    redis-url-shortener:
      image: redis:alpine
      ports:
        - 6379:6379

You will see that the above file has the same parameters/values as we defined when we were running using the docker run command.
Also, note that we aren’t creating a network explicitly here because
docker-compose would create a common network when bringing up the containers.
That’s it, you can bring all containers up or down using the following —

// get all containers up
docker-compose up

Test it out now by opening http://localhost:3000/ in the browser.

// bring down all containers
docker-compose down

Here are some useful docker commands —

// show all running containers
docker ps 
// show a history of containers - whether running or not
docker ps -a
// stop/start a container
docker stop CONTAINER_ID
docker start CONTAINER_ID
// show all local docker images
docker images
// show logs for a container, add -f to stream the logs
docker logs CONTAINER_ID
// open terminal/shell of the container. it = iterative terminal
docker exec -it CONTAINER_ID /bin/bash OR /bin/sh
// show/ create a network
docker network ls
docker network create name
// start all containers in the yaml
docker-compose -f FILE.yaml up
// remove/delete a container/ image
docker rm CONTAINER_ID
docker rmi IMAGE_ID

Thanks for getting through till here. I hope, it’s useful and you learned something new! :-)