In this post, I’ll walk you through how to fully self-host the n8n automation platform — from buying a domain and renting a VPS, to deploying it with Docker. Whether you're building workflows, scraping data, or experimenting with AI agents, n8n is flexible, extendable, and best of all — yours to control.
This guide is long, but contains all the necessary details.

We’re all feeling the AI-everywhere moment — from indie hackers to enterprises, everyone is “AI-washing” their workflows, products, and even job titles.
There’s this ongoing fear that AI will soon replace programmers, but let’s be honest: the truth, as always, lies somewhere in the middle.

Will AI assist us? Absolutely.

Will it replace developers entirely? Not anytime soon.

That’s where tools like n8n come in. While it’s not entirely (although a big portion is😊) branded as an "AI tool", because of its ability to integrate with LLMs and even build autonomous AI agents, it's riding the wave and helping automate tasks that previously required code or manual effort.

What You’ll Learn

By the end of this guide, you’ll know how to:

1. Set up a secure VPS (Virtual Private Server)
2. Purchase and configure a custom domain
3. Deploy and run n8n with Docker
4. Build your own private, extensible automation system — no subscription required

Here are the sections for quick reference:

• 🗄️Setup a secure VPS
  • 🔐SSH Keys — Securing Access to Your VPS
• 🌐Purchase and configure a custom domain
• 🤖Install n8n
  • 🧾A Note on Licensing — n8n Isn’t Fully Open Source

That said — if you're comfortable with a bit of Linux, Docker, and DNS configuration, or you're curious and ready to learn — this guide is for you.

For using a VPS I recommend also purchasing a VPS from any of cloud providers. My choice (again unaffiliated) is Hetzner.

Click a new project on their cloud console dashboard and name project to your liking (ex. N8N Automation).

And after that click “+Create server”

Chose the location closest to your location and in Image section click on “Apps” and choose “Docker CE”. This will install Ubuntu 24.04 together with Docker.

Next, for the Type of the VPS we will choose “Shared vCPU” and based on our app selection Hetzner recommends CPX21 which is a machine with 3 VCPUS 4GB of RAM and 80GB of SSD storage, that will cost you €7.05 /month.

If you check prerequisites on n8n documentation, it states that n8n is not CPU intensive,so this should be more than enough, though the disclaimer is that it varies depending on number of users, workflows and executions.

So since this is the case, and the fact you can rescale (scale up - bump up CPU and RAM) on Hetzner’s VPS, we will actually choose CX22 with 2VCPUS 4GB of RAM and 40GB SSD storage, that will cost you only €3.29/month.

Next, in the Networking section you can leave default selection of IPv4 and IPv6. (Note: having public IP costs €0.5/month)

This is an important step — your VPS will have a public IP, which means it's directly exposed to the internet. Even if you're just setting up a hobby project, you should always secure access properly. The best practice is to use SSH key authentication instead of passwords.

We'll start by generating a new SSH key:

ssh-keygen -t ed25519 -C "n8n-selfhost"

You’ll be prompted to:

1. Choose where to save the key (press Enter to use the default: ~/.ssh/id_ed25519)
2. Set a passphrase (optional but recommended)

Example output of ssh-keygen :

bojana@7OfNine:~$ ssh-keygen -t ed25519 -C "n8n-selfhost"
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/bojana/.ssh/id_ed25519):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/bojana/.ssh/id_ed25519
Your public key has been saved in /home/bojana/.ssh/id_ed25519.pub
The key fingerprint is:
SHA256:UujX91QnAsb1rrVAAMSS07KbgUxa3GI39zGIYB2OoRQ n8n-selfhost
The key's randomart image is:
+--[ED25519 256]--+
|  Eo++.Bo++..    |
| . o*+@.=.+o .   |
|  .*.+oB.. oo o o|
|  . o.o. ... o o.|
|      o+S . o +  |
|      oo   . * . |
|            . o  |
|                 |
|                 |
+----[SHA256]-----+

Now do:

bojana@7OfNine:~$ cat ~/.ssh/id_ed25519.pub
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIC4w30a+sB56njYFmueuHIs2WMxhqaFTruSjb9coSayT n8n-selfhost

And copy the output of the public key to the Hetzner’s form in SSH keys section:

And click “Add SSH key”

Next, we have option to attach volumes (additional storage), create firewalls (we will use ufw on the VPS itself for this) and/or use backups. Depending of seriousness of this project, you can choose to or not this option, it’s €0.66/month and it will create a copies of your server’s disk every day.

Finally, we can give this server a name, choose whatever will be helpful to you and hit “Create & Buy now” and wait until VPS is provisioned.

Once the VPS is provisioned, copy the Public IP assigned to it and do:

#:~$ ssh root@<VPS-Public-IP>

If everything went well you should be prompted with similar message as bellow, and asked to provide passphrase for a key if you created one after which you should be logged into your Ubuntu 24.04 instance!

The authenticity of host '<VPS-Public-IP>' can't be established.
ECDSA key fingerprint is SHA256:EHJB0s61tt4WcsuGPkhTRIn6CjuLz0lGOItY3mkT/dA.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '<VPS-Public-IP>' (ECDSA) to the list of known hosts.
Enter passphrase for key '/home/bojana/.ssh/id_ed25519':
Welcome to Ubuntu 24.04.1 LTS (GNU/Linux 6.8.0-52-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/pro

 System information as of Fri Jun 13 12:16:15 PM UTC 2025

  System load:  0.05              Processes:             129
  Usage of /:   4.3% of 37.23GB   Users logged in:       0
  Memory usage: 6%                IPv4 address for eth0: 91.99.136.92
  Swap usage:   0%                IPv6 address for eth0: 2a01:4f8:1c1a:3f29::1


Expanded Security Maintenance for Applications is not enabled.

0 updates can be applied immediately.

Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status


The list of available updates is more than a week old.
To check for new updates run: sudo apt update

root@n8n-automation:~#

Let’s verify docker and docker compose are there:

root@n8n-automation:~# docker -v
Docker version 27.5.1, build 9f9e405
root@n8n-automation:~# docker compose version
Docker Compose version v2.32.4

Before proceeding with n8n installation let’s take a few steps related to mentioned security and ssh access. We already connected using generated ssh key which is great, although connecting/sshing with root user is not recommended.

Let’s create non-root user with sudo privileges:

root@n8n-automation:~# adduser bojana

And add user to the sudo group:

root@n8n-automation:~# usermod -aG sudo bojana

Now, try logging in with newly created user:

bojana@7OfNine:~$ ssh bojana@<VPS-Public-IP>

We also want to allow login as the new user (with only the key):

rsync --archive --chown=username:username ~/.ssh /home/username

Now, to prevent root login and enforce key-only auth:

nano /etc/ssh/sshd_config

And ensure these lines are set:

PermitRootLogin prohibit-password
PubkeyAuthentication yes

Then restart the ssh service:

systemctl restart ssh

If you already own a domain, you don't have to purchase a new one. We can create a subdomain and use that when configuring n8n. If you don't own one, you can purchase one from domain registers such as:

• Porkbun
• Namecheap
• Hostinger

Or any other. I use porkbun for most of my projects, and haven't had any issues so far. I recommend either .dev or .cloud or even .xyz as they will cost you anywhere between 2$ and 8$ for a year. (Note: they usually renew and at bit higher prices so pay attention)

When you purchased a domain, and created account, say on Porkbun, you will want to go to their “domain management” dashboard and find your domain.

After that you select "DNS” which will take you to the page where you are able to manage DNS records for the domain. We simply need to insert an A record for our subdomain.

Click add and that's it. Depending on TTL it your new subdomain should propagate rather quickly.

To verify new subdomain resolves to your VPS's IP do:

dig +short n8n.mydmomain.com

While n8n offers self-hosting and provides source code access, it’s not licensed under a permissive open-source license like MIT or Apache 2.0.
Instead, it uses a Sustainable Use License (SUL) — a custom license designed to allow personal and internal use, while preventing commercial exploitation of the product itself.

This is important to understand before building a product or service around n8n.

Here’s a quick breakdown:

✅What is allowed under the SUL:

• Self-hosting n8n for personal use or inside your company/team
• Using it to automate tasks, processes, or workflows internally
• Extending or customizing n8n for your own non-commercial needs
• Embedding n8n in an internal tool used only within your organization

❌ What is not allowed under the SUL:

• Selling n8n as a service (e.g., "n8n Cloud clone" or "automation-as-a-service")
• Embedding n8n in a commercial SaaS offering
• Rebranding and reselling the platform or charging others to access your hosted version
• Offering hosted n8n to external users/customers, even if it's “free but monetized indirectly”

With this said, let's start with the first step in the guide.

Finally, we are at step where we want to setup the actual n8n platform.
In the previous step we verified that the docker and docker-compose are installed. We can optionally grant access to non-root user

sudo usermod -aG docker ${USER}
# Register the `docker` group memebership with current session without changing your primary group
exec sg docker newgrp

Now, you should be able to user docker commands without sudo:

bojana@n8n-automation:~$ docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

Now, we want to setup some env variables required for n8n installation.
Let's create a dir with the required .env file and (later) docker-compose file:

bojana@n8n-automation:~$ mkdir n8n-compose
bojana@n8n-automation:~$ cd n8n-compose/
bojana@n8n-automation:~/n8n-compose$

Now, within this folder create an .env file and paste the contents below (with changes lines with your domain/email etc.)

bojana@n8n-automation:~/n8n-compose$ nano .env

# DOMAIN_NAME and SUBDOMAIN together determine where n8n will be reachable from
# The top level domain to serve from
DOMAIN_NAME=example.com

# The subdomain to serve from
SUBDOMAIN=n8n

# The above example serve n8n at: https://n8n.example.com

# Optional timezone to set which gets used by Cron and other scheduling nodes
# New York is the default value if not set
GENERIC_TIMEZONE=Europe/Berlin

# The email address to use for the TLS/SSL certificate creation
SSL_EMAIL=user@example.com

Next, n8n documentation suggest creating directory local-files for sharing files between n8n instance and the host system. These files should be created with docker compose, but this ensures proper permissions and ownership are set.

bojana@n8n-automation:~/n8n-compose$ mkdir local-files
bojana@n8n-automation:~/n8n-compose$ ls
local-files

Finally, we create a following compose.yaml file:

services:
  traefik:
    image: "traefik"
    restart: always
    command:
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.mytlschallenge.acme.tlschallenge=true"
      - "--certificatesresolvers.mytlschallenge.acme.email=${SSL_EMAIL}"
      - "--certificatesresolvers.mytlschallenge.acme.storage=/letsencrypt/acme.json"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - traefik_data:/letsencrypt
      - /var/run/docker.sock:/var/run/docker.sock:ro

  n8n:
    image: docker.n8n.io/n8nio/n8n
    restart: always
    ports:
      - "127.0.0.1:5678:5678"
    labels:
      - traefik.enable=true
      - traefik.http.routers.n8n.rule=Host(`${SUBDOMAIN}.${DOMAIN_NAME}`)
      - traefik.http.routers.n8n.tls=true
      - traefik.http.routers.n8n.entrypoints=web,websecure
      - traefik.http.routers.n8n.tls.certresolver=mytlschallenge
      - traefik.http.middlewares.n8n.headers.SSLRedirect=true
      - traefik.http.middlewares.n8n.headers.STSSeconds=315360000
      - traefik.http.middlewares.n8n.headers.browserXSSFilter=true
      - traefik.http.middlewares.n8n.headers.contentTypeNosniff=true
      - traefik.http.middlewares.n8n.headers.forceSTSHeader=true
      - traefik.http.middlewares.n8n.headers.SSLHost=${DOMAIN_NAME}
      - traefik.http.middlewares.n8n.headers.STSIncludeSubdomains=true
      - traefik.http.middlewares.n8n.headers.STSPreload=true
      - traefik.http.routers.n8n.middlewares=n8n@docker
    environment:
      - N8N_HOST=${SUBDOMAIN}.${DOMAIN_NAME}
      - N8N_PORT=5678
      - N8N_PROTOCOL=https
      - NODE_ENV=production
      - WEBHOOK_URL=https://${SUBDOMAIN}.${DOMAIN_NAME}/
      - GENERIC_TIMEZONE=${GENERIC_TIMEZONE}
    volumes:
      - n8n_data:/home/node/.n8n
      - ./local-files:/files

volumes:
  n8n_data:
  traefik_data:

As stated in the docs, this creates two containers:

• n8n (actual n8n app)
• traefik (application proxy to manage TLS/SSL certificates and manage routing)

It also creates two docker volumes (n8n_data, traefik_data) and mounts the local-files directory we created earlier.

Finally, we run the docker-compose to bootstrap everything:

bojana@n8n-automation:~/n8n-compose$ sudo docker compose up -d

After this, you should see 2 running above mentioned containers:

bojana@n8n-automation:~/n8n-compose$ docker ps
CONTAINER ID   IMAGE                     COMMAND                  CREATED          STATUS          PORTS                                                                      NAMES
77f28a4f1e8f   docker.n8n.io/n8nio/n8n   "tini -- /docker-ent…"   14 seconds ago   Up 13 seconds   127.0.0.1:5678->5678/tcp                                                   n8n-compose-n8n-1
762beb6046f5   traefik                   "/entrypoint.sh --ap…"   14 seconds ago   Up 13 seconds   0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   n8n-compose-traefik-1

If everything went well and you see running containers check your configured subdomain (n8n.mydomain.com)

You should see n8n setup screen like below:

Fill up the form and hit Next. And fill out the required n8n's "customization form"

Next you can skip or register with email for some additional features on Community Edition such as folders, debug in editor, custom execution data etc.
If you register, you should receive an activation code on your email address:

After this you should receive your license with activation code on registered email, just click Activate License Key in the email and that's it.

And this is it, you should be able to create and edit workflows, create AI agents etc.

📋Wrapping Up

If you've made it this far and everything is up and running — congratulations! You've successfully deployed and secured your own self-hosted n8n instance.

If something didn’t work along the way, I encourage you to double-check the steps — it’s easy to miss a small detail, especially when security configurations are involved. I’ve aimed to make this guide thorough, not overwhelming, because skipping over things like proper SSH setup or server hardening can leave you vulnerable, even for a hobby project.

☁️“But I could just use n8n Cloud...”

Absolutely — and that’s a valid choice.

If you’re looking for a plug-and-play experience, the n8n Cloud Starter plan is a solid option. It saves time and abstracts away the complexity.

But here's the trade-off:

• With self-hosting, you own your data — no third party in the loop
• You can tinker and extend freely, without worrying about platform limitations
• You learn valuable skills — from infrastructure to automation

In the long run, a few euros a month for a VPS and domain is a small price to pay for freedom, control, and learning.

Thanks for following along — I hope this guide helped you build something real, useful, and yours.

Cheers, and happy automating!🎉

When diving into cloud services, I’ve found that theory alone can only take you so far. While it’s important to build a solid foundation of concepts, things often remain abstract until you apply them to real-world scenarios. Azure offers several load balancing solutions—many of which sound similar and serve overlapping purposes. Without seeing them in action or understanding the specific use cases, it's easy to get stuck in a loop of confusion, unsure of when to choose one over the other.

Before diving into the various Azure services, it’s worth clarifying what load balancing actually means. At its core, load balancing is about distributing incoming network traffic across multiple resources to ensure reliability, performance, and scalability.

Imagine you're at a busy coffee shop on a Monday morning. There's a single barista trying to take orders, make drinks, and handle payments — the line moves slowly, and customers start getting frustrated. Now imagine there are four baristas, each handling specific tasks or splitting the workload evenly. Suddenly, everything flows faster, orders are processed smoothly, and customers leave happy.

That’s the essence of load balancing: efficiently distributing work across multiple servers (or people) to improve speed, reliability, and user experience.

It's important to note that load balancer can be either hardware or software-based. In. context of cloud services it is mostly the software-based one.

Which server should handle each request load balancers are determining based on number of different algorithms. These fall into two categories:

• static
• dynamic

Static load balancing algorithms do not take into account current state of the work when distributing workload. It will not determine which servers are not utilised enough and which are overused. Typical static load balancing algorithms include round robin, least connections, IP hash, fixed partitioning etc. ****

Static load balancing is used in scenarios where traffic patterns are predictable and backend servers are homogenous and have similar capacity.

Dynamic load balancing is also a process of distributing traffic or workload across multiple servers but at runtime, and based on real-time information about server health, resource utilisation or connections count. ****

When a server becomes overloaded or fails, traffic can be automatically/dynamically rerouted to healthy instances.

So, unlike static load balancing, dynamic methods react to changing conditions in the system. (CPU usage, memory, network throughput..)

Dynamic load balancing is typically integrated with health probes so that traffic is not sent to degraded or unavailable nodes.

Due to continuous monitoring it may introduce latency and it's typically more complex than static load balancing.

In cloud environments like Azure, load balancing plays a critical role in ensuring that your applications stay responsive and available — especially under heavy traffic. But here’s where it gets tricky: Azure offers multiple load balancing services that, at first glance, seem to do the same thing. With names like Azure Load Balancer, Application Gateway, Front Door, and Traffic Manager, it’s easy to get them mixed up.

Azure Load Balancer operates at OSI Layer 4 (TCP/UDP) and distributes inbound traffic across multiple backend instances (like VMs) within a region. It provides high throughput and is ideal for network-level load balancing of VMs or container instances. In this lab, we’ll create a public Load Balancer to distribute HTTP traffic between two web server VMs.

1. First we will create a resource group for our little setup. (I will use ‘eastus’ for $LOCATION here). You can define following variables before start since we will be referencing these through the setup.

# Variables (replace values as needed)
RESOURCE_GROUP="az-load-balancer-lab"
LOCATION="eastus"
LB_NAME="MainLoadBalancer"
PUBIP_NAME="LBPublicIP"
BACKEND_POOL_NAME="MyBackendPool"

az group create --location $LOCATION --resource-group $RESOURCE_GROUP

2. Then we create a virtual network and a corresponding subnet.

az network vnet create -g $RESOURCE_GROUP -n MyVnet --subnet-name MySubnet

3. Now, you can use existing VMs or create a new ones. For the purpose of the exercise we will create 2 new Linux VMs (Ubuntu 24.04), without public IPs (because Load Balancer will provide one). We will generate ssh keys for each and install nginx. Finally we well create simple entry in /var/www/html/index.html echoing hostname, so we can observe which VM is responding to client request once we finish setting up everything.

So to create VM1 issue command:

az vm create -g $RESOURCE_GROUP -n VM1 --image Ubuntu2404 --vnet-name MyVNet --subnet MySubnet --public-ip-address "" --generate-ssh-keys --size Standard_B1s

Note: size argument is optional, if not supplied it will default to Standard_DS1_v2. I think Standard_B1s will suffice for our exercise. You can get list of all size with following command:

az vm list-sizes --location eastus --output table

We can proceed and create VM2 with same configuration:

az vm create -g $RESOURCE_GROUP -n VM2 --image Ubuntu2404 --vnet-name MyVNet --subnet MySubnet --public-ip-address "" --generate-ssh-keys --size Standard_B1s

Now we will crate a sample entry files in /var/www/html/index.html on each machine so we can see which machine is responding to the call.
But since these VMs do not have a public IPs, we will use command:

az vm run-command invoke

to remotely execute a script or command on Azure VM without needing to manually connect to it via SSH or RDP. This uses Azure VM Agent, which is a small service automatically isntalled on most Azure images to run these commands inside VM.

For VM1:

az vm run-command invoke \
  --resource-group $RESOURCE_GROUP \
  --name VM1 \
  --command-id RunShellScript \
  --scripts '
sudo apt update && sudo apt install -y nginx;
echo "Hello from - $(hostname)" | sudo tee /var/www/html/index.html
'

The command will install nginx and create the desired index.html with hostname of machine.
Do the same for VM2:

az vm run-command invoke \
--resource-group az-load-balancer-lab \
--name VM2 \
--command-id RunShellScript \
--scripts '
sudo apt update && sudo apt install -y nginx;
echo "Hello from - $(hostname)" | sudo tee /var/www/html/index.html

4. Now, we want to open port 80 on both VMs Network Security Group so that load balancer can reach the web service.

az vm open-port -g $RESOURCE_GROUP -n VM1 --port 80 --priority 1001
az vm open-port -g $RESOURCE_GROUP -n VM2 --port 80 --priority 1001

5. And create a public IP for the load balancer

az network public-ip create -g $RESOURCE_GROUP -n $PUBIP_NAME --sku Standard

6. In this step we will create actual load balancer with one frontend and one backend pool.

az network lb create -g $RESOURCE_GROUP -n $LB_NAME --sku Standard \
    --frontend-ip-name FrontEndConfig --public-ip-address $PUBIP_NAME \
    --backend-pool-name $BACKEND_POOL_NAME

7. Next, we want to create a health probe on port 80 (TCP).

A health probe is used to determine the health status of the instances in the back-end pool. This health probe determines if an instance is healthy and can receive traffic.

az network lb probe create --resource-group  $RESOURCE_GROUP --lb-name $LB_NAME --name MyProbe --protocol http --port 80 --path /

8. Create a load balancing rule for HTTP (port 80)

az network lb rule create -g $RESOURCE_GROUP --lb-name $LB_NAME -n HTTPRule \
    --protocol tcp --frontend-port 80 --backend-port 80 \
    --frontend-ip-name FrontEndConfig --backend-pool-name $BACKEND_POOL_NAME \
    --probe-name HealthProbe

9. Add VM NICs to the Load Balancer’s backend pool

NIC1=$(az vm show -g $RESOURCE_GROUP -n VM1 --query "networkProfile.networkInterfaces[0].id" -o tsv)
NIC2=$(az vm show -g $RESOURCE_GROUP -n VM2 --query "networkProfile.networkInterfaces[0].id" -o tsv)
az network nic ip-config address-pool add -g $RESOURCE_GROUP --lb-name $LB_NAME \
    --address-pool $BACKEND_POOL_NAME --nic-name $(basename $NIC1) --ip-config-name $(az network nic show -g $RESOURCE_GROUP --name $(basename $NIC1) --query "ipConfigurations[0].name" -o tsv)
az network nic ip-config address-pool add -g $RESOURCE_GROUP --lb-name $LB_NAME \
    --address-pool $BACKEND_POOL_NAME --nic-name $(basename $NIC2) --ip-config-name $(az network nic show -g $RESOURCE_GROUP --name $(basename $NIC2) --query "ipConfigurations[0].name" -o tsv)

10. And finally get the public IP of the Load Balancer

az network public-ip show -g $RESOURCE_GROUP -n $PUBIP_NAME --query "ipAddress" -o tsv

After running the above, navigate to the displayed IP address in a browser. You should hit one of two VMs via the Load Balancer’s IP.

And that's it, you've successfully configured Level 4 Load Balancer.

Since it operates and Layer 4 (Transport Layer) Azure Load Balancer is designed to handle high-throughput scenarios efficiently - ultra-low latency and high performance (think gaming servers, financial services, or real-time applications)

You should not use Azure Load Balancer if your web app only receives a small amount of traffic and existing infrastructure already deals with the load just fine.

In one one of the following labs we will explore Azure Application Gateway, offering various Layer 7 capabilities.

It is! Allow me a second to lament that my last entry on this blog was written on January 26th last year. Well, what can you do?

Why is writing generally a good idea, even if it's just a blog post with a limited audience? I wondered that myself.

Ah, yes, the "AI takeover" everyone is buzzing about. Why bother writing when you can ask ChatGPT to do it for you? In the tweet above, Paul Graham writes how people are using AI (or ChatGPT, to be precise) more and more to write all sorts of stuff for them because writing is hard. He then continues :

I warn you now, this is going to have unfortunate consequences, just as switching to living in suburbia and driving everywhere did. When you lose the ability to write, you also lose some of your ability to think.

.... I'm not warning about the switch to AI in the hope of averting it, but to warn the few people who care enough to save themselves, or their kids. Learn to use AI. It's a powerful technology, and you should know how to use it. But also learn how to write.

These few lines got me thinking about writing and what and how I can do to write more and write better.

To be clear, I am not talking about writing in general "literature form." I am talking about writing as a technical person, software engineer, engineer, or architect. After all, the goal of these writings is not to entertain or make you wonder (although it can sometimes?) but more to be precise and unambiguous, to communicate things well.

I have never considered myself good at writing (present-day included).
When I think about it, through most of high school and college, where these kinds of skills could be exercised in the form of essays, and so on, I was never good at it. At least the ones for Literature classes. You might guess why I've chosen the STEM path.

I think it's even harder to write in your non-native language. English is the lingua franca of today's day and age, especially in tech. So, being a non-native English speaker is one more barrier you must overcome.

Why writing is essential for software engineers?

Ok, so writing is not easy. But why should you still try to be better at it?

"Writing is thinking"

The quote above, which I searched for and found out was uttered by the VP of the Word team, holds the same theme as Paul Graham's tweet above - writing enables you to think, to carve out thoughts from your brain, for them to take form so that others can see them and understand them. They say that if you can think clearly, you can write clearly. It's just a matter of willingness to try and learn the skill.

As a software engineer, yes, you write code, but maybe more importantly - you communicate with others on your team and organization. While face-to-face communication is crucial, sometimes it's impossible to do it (especially with today's prevailing async communication and remote teams), and you need something more durable and with higher reach. Composing a feature/system documentation, setup guide, requirements review, or just a simple code review, the ability to write well is essential. If you can compose precise and concise writing, you can communicate well. It boils down to that - communication is a much-needed skill in the tool-belt of every engineer.

How do I write better?

That's a good question. As with any skill, you need to practice. A lot.

Besides that obvious advice, there are some universal rules for better writing, regardless of purpose and form. An excellent book called On Writing Well: The Classic Guide to Writing Nonfiction details basic rules related to writing and a bunch of examples that the author collected. The guide is nicely written, and although the main subject is a "meta" subject, it isn't dull or dry.

While the books cover a lot of principles, methods, and forms of writing, I will highlight here a few that I found interesting because I think they are easily applicable to the "software engineer" or writing a blog like this one.

Reduce clutter

When I think about clutter, the first thing that comes to my mind is local politicians. They often overuse certain words and phrases and talk too much without saying anything. This, of course, should be avoided. The book's author mentioned above compares clutter to weeds, constantly integrating into speech and writing. Go over your written sentences. Is anything creating clutter? Could you remove it?
This is also important when writing documentation for a product or a feature. If it's concise and easily understandable, it's considered good if you can quickly find the answer to your question.

Rewriting

This is the essence of writing well; it's intuitive and natural. It's a process. Not even professional writers get the sentence the "right way" for the first time. They rewrite, fiddle with it, and sentences evolve with every rewrite.

Writing is like a good watch—it should run smoothly and have no extra parts

Sequential storytelling

This is specifically important for "technical writing." Why? Because often you are explaining some concept or process to the audience. It forced you to make sure how it works so you can guide your audience to the same sequence of ideas and deductions that helped you to make the process clear.

It's the principle of leading readers who know nothing, step by step, to a grasp of subjects they didn't think they had an aptitude for or were afraid they were too dumb to understand.

The audience

In short, you are writing for yourself. Don't get intimidated by some large audience that will read your piece and criticize you.
That is not to say your writing shouldn't have structure and you shouldn't care,
on the contrary. There is no excuse if your reader is lost in the middle of the writing.
The first is a matter of style, and expressing who you are, the second is a matter of craft.

Never say anything in writing that you wouldn't comfortably say in conversation.

Another thing I often hear from others, or even myself, "No one will be reading this; no one cares about this." But that doesn't matter. Even if no one will, you will. For example, I often go back to some older blog posts because I've written some details that come in handy. Or they are simply a reminder about the interests I had and the learning I did.

Writing is not a contest.

In high school and college, I always feared that someone else would write better than I would if I wrote about something. This is nonsense. Writing is not a contest, and everyone should go at their own pace; the only one you are competing with is yourself.

This book has many other useful pieces of advice, and I recommend you take a look.
After all, one must read a lot to be better at writing.

Make a habit of reading what is being written today and what has been written by earlier masters. Writing is learned by imitation. If anyone asked me how I learned to write, I'd say I learned by reading the men and women who were doing the kind of writing I wanted to do and trying to figure out how they did it.

In the last article, I said that the DNS zone is defined for each domain and that it’s a file that consists of Resource Records. When the DNS resolver forwards the domain name to the DNS server, it gets the resource records that match that domain from it.

DNS zone can include a single domain name, one domain, many subdomains, or many domain names. Usually, the DNS zone is equivalent to 'domain’, but it doesn’t have to be.

• Resource records
  • SOA
  • A record
  • AAAA record
  • MX record
  • CNAME record
  • NS record
  • SRV record
  • PTR record
  • TXT record
• DNS security issues (DNSSEC)
• DNS tools

Resource Records

Resource Record is comprised of 5 fields:

• Name - it can be a domain name or host address
• Time to live (TTL) - how long the record will “live” in those caches I mentioned in the last article. It shows how stable the record is. The bigger this number is, the longer the record will be kept in the cache. For ex. SOA record which we will examine shortly this value is 86400 which is 24h (TTL values are in seconds).
• Class - for data that is important for the Internet, this is always IN, for other, non-Internet data other codes can be used, but very rarely.
• Type - type of record. You can see the most important types in the table below.
• Value - the value that is assigned to the RR (ex. address, name, etc.)

SOA (Start of Authority)

This is the first record in each DNS zone.
SOA record contains the name of the primary DNS server, email of the DNS administrator (optional, and nowadays mostly skipped due to spam issues), unique serial number, and various other indicators and timers.

$ORIGIN vps-playground.cloud.
$TTL 86400
; SOA Records
@		IN	SOA	hydrogen.ns.hetzner.com. dns.hetzner.com. 2023011205 86400 10800 3600000 3600

Serial number (version, in the example above 2023011205) for DNS zone file - this “number” is in format “yyyymmddnn” and it has to be incremented during every change so that secondary DNS knows and it can retrieve it. This was a common mistake when handling DNS zone files, not having the serial number incremented, and thus not propagating changes to secondary DNS servers. However, by using any of the VPS provider’s dashboards for updating DNS zone files, this value is incremented automatically, after you make any changes to the DNS zone file.

This process of sending a DNS record from a primary DNS server to a secondary is called a zone transfer. In the Part 1 I said that DNS data is sent over port UDP port 53, this zone transfer is an exception, and it’s sent over TCP (because we need to ensure the transfer was completed correctly, hence the more reliable protocol).

Refresh timer - after how many seconds secondary DNS will check for changes on the primary
Retry - if the check didn’t succeed, after how many seconds the retry will happen
Expire - How long DNS zone(s) from primary will be kept
Minimum - sets the time to live for negative answers that are cached*

*From the example above you can see $TTL directive defines times for all records in the zone file. This is the default value for positive answers (i.e., actual records). The minimum parameter in the SOA record sets the time to live for negative answers that are cached (cache misses if you will).

NS records

NS (name server) records identify the authoritative servers for a zone.
These records are usually defined, just after the SOA record. The format is:

zone [ttl] [IN] NS hostname

; NS Records, "@" - represents root
@	600	IN	NS	helium.ns.hetzner.de.
@	600	IN	NS	hydrogen.ns.hetzner.com.
@	600	IN	NS	oxygen.ns.hetzner.com.

A record (IPv4 address)

This is the key record of the DNS database, because they provide ma mapping from hostname to IP address. The format is:

hostname [ttl] [IN] A ipaddr

www	300	IN	A	142.132.165.72

AAAA record (IPv6 address)

IPv6 equivalent of A records. IPv6 records in your DNS zone don’t mean you have to answer DNS queries over IPv6. The format is:

hostname [ttl] [IN] AAAA ipaddr

f.root-servers.net. IN AAAA 2001:500:2f::f

MX

MX records are used by mail systems to route emails more efficiently. MX records specify the host's name prepared to accept the email for the specified domain.

The format of an MX record is: name [ttl] [IN] MX preference host ...

mail.vps-playground.cloud  86400 IN MX 10 mail

The preference or priority parameter is indicating which host has a priority for receiving/handling emails. In case of message send failure, the server will default to host with next priority preference (lower number means bigger priority).

When a user sends an email, the MTA (Mail Transfer Agent), sends a DNS query to identify the mail servers for the email recipients. SMTP connection will be established with the domain with the highest priority (mailserver1 in the example above).

CNAME record

This record allows aliases to be created. These “nicknames” are used to either shorten a long hostname or associate a function with a host. The real name is often called “canonical name” (hence “CNAME”). CNAME must always point to a domain, never to an IP address. Imagine I have this record in my DNS zone:

bojana.dev 120 IN CNAME blog.bojana.dev 

When a DNS software encounters CNAME record, it stops the query for this “nickname” and re-quires for the real name. For the example above, it means the querying will finish once A record for bojana.dev is found.

Pointing a CNAME to another CNAME, is possible (up to seven times, the eighth target must be a real hostname), but it’s considered inefficient because it requires multiple DNS lookups before the domain can be loaded which ultimately slower experience for the user. MX and NS records cannot point to a CNAME record (they have to point to A or AAAA record).

SRV records

This record specifies the location of the service (a host and a port) within a domain. Services such as VoIP, instant messaging, etc. This DNS record is unique because most other DNS records specify a server or an IP address, whereas the SRV record includes port at that IP address as well.

The format is: service.proto.name [ttl] [IN] SRV pri weight port target

The service is a service defined in the IANA assigned numbers database, proto is either TCP or UDP, name is a domain name to which the SRV record refers, pri is priority, weight is used for load balancing among several servers, port is a port on which service runs, and the target is the hostname of the server that provides the service.

Example :

; main server on port 80, backup on new box, port 8000
_http._tcp SRV 0 0 80 www-server.example.com.
	   SRV 10 0 8000 new-fast-box.example.com.

TXT records

This record adds arbitrary text to the host’s DNS records. Originally, it was intended as a place for human-readable notes. Now, you can also put some machine-readable data into TXT records.

You can create many TXT records for one domain.

Example:

TXT records don’t have any particular format, so they are sometimes used to add information for other purposes without requiring changes to the DNS system itself.

Most DNS records will put a limit on number of TXT records which can be created and how big TXT records can be.

SPF, DKIM, and DMARC records

SPF (Sender Policy Framework), DKIM (DomainKeys Identified Mail), and DMARC (Domain-based Message Authentication, Reporting, and Conformance) are all actually TXT records, and are all standards in an attempt to fight “unsolicited commercial email” aka spam. Going into details about these would require new blogpost.

DNSSEC

Initially, DNS protocol did not considered security. DNS name servers or resolvers could manipulate the content of any DNS record, thus causing the client to receive incorrect information.

RFC 3833 highlight some security threats to DNS, and how DNSSEC addresses these threats.

DNS Security Extensions (DNSSEC) is a security protocol created to mitigate this problem. The protection against attacks is done by digitally signing the data to help ensure its validity. This signing must happen at every level in the DNS lookup process.

DNSSEC implements a hierarchical digital signing policy across all layers of DNS. For example, in the case of a ‘bojana.dev’ lookup, a root DNS Server would sign a key for the .DEV nameserver, and the .DEV nameserver would then sign a key for bojana.dev’s authoritative nameserver.

DNSSEC is also designed to be backward compatible. What that means is, while improved security is of course important and preferred, it has to ensure that traditional DNS lookups still resolve correctly (without added security).

In the part 1 I mentioned the tree structure behind DNS, and root DNS servers at the beginning of this structure. DNSSEC creates a parent-child chain of trust that travels all the way up to the root zone. This chain of trust cannot be compromised at any layer of DNS.

The root zone itself needs to be validated (proven to be free of tampering or fraud). An interesting detail is that this validation of the root zone is actually done by human intervention. In something that’s called Root Zone Signing Ceremony. Selected individuals from around the world are gathered to sign the root DNSKEY in a public and audited way. This process is done every few months.

As mentioned above, DNS servers are vulnerable to broad spectrum of attacks, such as spoofing, DoS (Denial of Service), or interception of private personal information.

Besides DNSSEC, there are additional measures that operator of DNS zone can perform in order to secure their servers. Over-provisioning infrastructure - in simple terms allowing your nameservers to handle multiplies more traffic than expected, in order to overcome DDOS attacks.

Other strategies include using DNS firewall, DNS over TLS and DNS over HTTPS.
Again, DNS security is a broad topic, and would require at least a blogpost of it's own, if not many blogposts ;)

DNS tools

Of course, whether you are administering DNS zones or not, there will come time where you’ll need some kind of way to verify your DNS records are correct and up-to-date. There are numerous tools for analysis, detection etc. of DNS queries. One of them is nslookup. This is, of course, a command line tool. It comes preinstalled on both Windows and Unix systems.

nslookup bojana.dev
Server:  UnKnown
Address:  192.168.100.1

Non-authoritative answer:
Name:    bojana.dev
Address:  157.90.124.251

However, when you want to know some IP behind the domain name, it’s often easier to issue ping command, because it will perform nslookup automatically.

ping bojana.dev

Pinging bojana.dev [157.90.124.251] with 32 bytes of data:
Reply from 157.90.124.251: bytes=32 time=41ms TTL=49
Reply from 157.90.124.251: bytes=32 time=39ms TTL=49

Ping statistics for 157.90.124.251:
    Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 39ms, Maximum = 41ms, Average = 40ms

dig

(domain information groper) is also a tool for querying DNS servers.
A typical invocation of dig looks like this:
dig @server name type
Usage is as follows:

dig [server] [name] [type]

[server] The hostname or IP address the query is directed to

[name] DNS (Domain Name Server) of the server to query

[type] The type of DNS record to retrieve. By default (or if left blank), dig uses the A record type
In part 1, I also mentioned that there are 13 root DNS servers, and typically web browsers and operating systems already know about those addresses. If you just enter dig command without parameters, it will return you exactly that list of 13 root DNS servers:

$ dig
; <<>> DiG 9.18.8 <<>>
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58489
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 27

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;.                              IN      NS

;; ANSWER SECTION:
.                       516137  IN      NS      a.root-servers.net.
.                       516137  IN      NS      b.root-servers.net.
.                       516137  IN      NS      c.root-servers.net.
.                       516137  IN      NS      d.root-servers.net.
.                       516137  IN      NS      e.root-servers.net.
.                       516137  IN      NS      f.root-servers.net.
.                       516137  IN      NS      g.root-servers.net.
.                       516137  IN      NS      h.root-servers.net.
.                       516137  IN      NS      i.root-servers.net.
.                       516137  IN      NS      j.root-servers.net.
.                       516137  IN      NS      k.root-servers.net.
.                       516137  IN      NS      l.root-servers.net.
.                       516137  IN      NS      m.root-servers.net.

;; ADDITIONAL SECTION:
a.root-servers.net.     516137  IN      A       198.41.0.4
a.root-servers.net.     516137  IN      AAAA    2001:503:ba3e::2:30
b.root-servers.net.     516137  IN      A       199.9.14.201
b.root-servers.net.     516137  IN      AAAA    2001:500:200::b
c.root-servers.net.     516137  IN      A       192.33.4.12
c.root-servers.net.     516137  IN      AAAA    2001:500:2::c
d.root-servers.net.     516137  IN      A       199.7.91.13
d.root-servers.net.     516137  IN      AAAA    2001:500:2d::d
e.root-servers.net.     516137  IN      A       192.203.230.10
e.root-servers.net.     516137  IN      AAAA    2001:500:a8::e
f.root-servers.net.     516137  IN      A       192.5.5.241
f.root-servers.net.     516137  IN      AAAA    2001:500:2f::f
g.root-servers.net.     516137  IN      A       192.112.36.4
g.root-servers.net.     516137  IN      AAAA    2001:500:12::d0d
h.root-servers.net.     516137  IN      A       198.97.190.53
h.root-servers.net.     516137  IN      AAAA    2001:500:1::53
i.root-servers.net.     516137  IN      A       192.36.148.17
i.root-servers.net.     516137  IN      AAAA    2001:7fe::53
j.root-servers.net.     516137  IN      A       192.58.128.30
j.root-servers.net.     516137  IN      AAAA    2001:503:c27::2:30
k.root-servers.net.     516137  IN      A       193.0.14.129
k.root-servers.net.     516137  IN      AAAA    2001:7fd::1
l.root-servers.net.     516137  IN      A       199.7.83.42
l.root-servers.net.     516137  IN      AAAA    2001:500:9f::42
m.root-servers.net.     516137  IN      A       202.12.27.33
m.root-servers.net.     516137  IN      AAAA    2001:dc3::35

;; Query time: 850 msec
;; SERVER: 172.18.176.1#53(172.18.176.1) (UDP)
;; WHEN: Thu Jan 26 09:59:09 CET 2023
;; MSG SIZE  rcvd: 811

dig bojana.dev SOA

; <<>> DiG 9.18.8 <<>> bojana.dev SOA
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 47443
;; flags: qr rd ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;bojana.dev.                    IN      SOA

;; ANSWER SECTION:
bojana.dev.             0       IN      SOA     curitiba.ns.porkbun.com. dns.cloudflare.com. 2297766056 10000 2400 604800 3600

;; Query time: 0 msec
;; SERVER: 172.18.176.1#53(172.18.176.1) (UDP)
;; WHEN: Thu Jan 26 09:52:05 CET 2023
;; MSG SIZE  rcvd: 115

drill

Also a tool to get numerous information out of DNS. It is specifically designed to be used with DNSSEC. The name is sort of a pun on dig. If no arguments are given class defaults to 'IN' and type to 'A'.

bojana@nextcloud:~$ drill -T bojana.dev
.       518400  IN      NS      i.root-servers.net.
.       518400  IN      NS      b.root-servers.net.
.       518400  IN      NS      c.root-servers.net.
.       518400  IN      NS      f.root-servers.net.
.       518400  IN      NS      h.root-servers.net.
.       518400  IN      NS      d.root-servers.net.
.       518400  IN      NS      g.root-servers.net.
.       518400  IN      NS      e.root-servers.net.
.       518400  IN      NS      k.root-servers.net.
.       518400  IN      NS      j.root-servers.net.
.       518400  IN      NS      a.root-servers.net.
.       518400  IN      NS      l.root-servers.net.
.       518400  IN      NS      m.root-servers.net.
dev.    172800  IN      NS      ns-tld5.charlestonroadregistry.com.
dev.    172800  IN      NS      ns-tld3.charlestonroadregistry.com.
dev.    172800  IN      NS      ns-tld2.charlestonroadregistry.com.
dev.    172800  IN      NS      ns-tld1.charlestonroadregistry.com.
dev.    172800  IN      NS      ns-tld4.charlestonroadregistry.com.
bojana.dev.     10800   IN      NS      fortaleza.ns.porkbun.com.
bojana.dev.     10800   IN      NS      salvador.ns.porkbun.com.
bojana.dev.     10800   IN      NS      curitiba.ns.porkbun.com.
bojana.dev.     10800   IN      NS      maceio.ns.porkbun.com.
bojana.dev.     600     IN      A       157.90.124.251

This is also a verification of the story about how DNS names are resolved.

DNS might seem as a fairly simple system, however, there are many details, "nuts and bolts" included in order for it to work properly. I mentioned a few pitfalls of DNS and how new updates to the protocol are trying to address them.

Hope you enjoyed the reading, and are know a bit more knowledgable and appreciative of DNS. ;)
For the end, here's one haiku about DNS. :)

Using a password manager is a good security practice, of which I will not be talking much nor convince you to use one. Since you are reading this article, chances are you are already using one or want to try it out.

However, with the recent LastPass hack question arises whether it is safe to trust a 3rd party cloud provider to safely store all your valuable passwords. Using a password manager is a “no-brainer” for most of you us security-aware folks, but if you are also concerned about where and how they are stored as well, then this “self-hosted” option might be for you.

A "Silver bullet" solution doesn't exist. Nothing will make you 100% secure, however this way at least you will have  complete control.

Bitwarden is, as they claim on their website,  an integrated open-source password management solution for individuals, teams, and business organizations.

While they have plans for both personal use and business, they also have the option to self-host Bitwarden.

Self-host option of Bitwarden can be hosted on Linux, MacOS and Windows. We will be focusing on Linux hosting.

They do offer something they call “Unified Self-Host deployment” which is essentially all of the services needed for Bitwarden to run in one unified Docker container, whereas the “Standard Deployment Offer” uses 12 individual Docker containers working together. Furthermore, in the unified deployment you can choose between database providers such as Microsoft SQL Server, MySQL, and PostgreSQL, and  Standard Deployment uses Microsoft SQL Server. Or if you like to look at fancy diagrams, something like the below:

The “Unified self-host” option is still in beta, so we’re gonna go with “Standard Deployment Offer”.

• Prerequisites
• Domain setup
• Docker and Docker compose
• Bitwarden user and directory
• Install Bitwarden
• Set up 2FA

Prerequisites

Before continuing with the setup, there are some prerequisites that we need to fulfill.

• Domain - we need a domain for our Bitwarden instance
• Linux host - we need a Linux machine. It could be any Linux host, but for the sake of simplicity I will provision an Ubuntu (22.04) VPS on Hetzner Cloud, but you should be good with any provider and distribution.
• As per Bitwarden documentation, there are certain requirements in terms of system specification (see the minimum and recommended resource below)
• Time and patience 😊

I will use recommended system resources, so a Linux VPS with 2 cores and 4GB of RAM. (Note: on mentioned provider, this cost is about 5.35€/month)

As stated above, we will also need a domain/subdomain. If you already own a domain, you can create a subdomain for bitwarden.

Domain setup

I already have a domain purchased, so the first thing I will do is set up a DNS zone for my domain. By default, DNS zone is already set up wherever you purchased your domain, but I found it easier to be managed on the VPS provider side. Now, depending on your provider this might look different in different dashboards/systems, but it boils down to creating a file that will hold DNS records for your domain.

The next step is to configure nameservers. You can see I purchased my domain at porkbun.com, so I would need (as suggested on the screenshot above) to remove their nameservers and replace them with my VPS provider’s name servers (Hetzner) on the porkbun side, so in their dashboard for domain setup.

As I said, depending on your VPS provider and domain registrar this might look different. So I will go on porkbun dashboard for my domain, and there is a section “Authoritative Nameservers”, which I should edit and add Hetzner’s nameservers (as suggested in their message).

These take some time to propagate, usually about 24h.

Now, we should add A record to map our domain to the public IP address of our VPS. The “@” sign represents root which for my domain would be just:
vps-playground.cloud. The other record with the name “www” is optional but usually added.

With these configurations in place, let’s go and change the hostname on our VPS. But before that, I will create a non-root user.

root@ubuntu-4gb-nbg1-1:~# adduser bojana
Adding user `bojana' ...
Adding new group `bojana' (1000) ...
Adding new user `bojana' (1000) with group `bojana' ...
Creating home directory `/home/bojana' ...
Copying files from `/etc/skel' ...
New password:
Retype new password:
passwd: password updated successfully
Changing the user information for bojana
Enter the new value, or press ENTER for the default
        Full Name []: Bojana Dejanovic
        Room Number []:
        Work Phone []:
        Home Phone []:
        Other []:
Is the information correct? [Y/n] Y

And add the user to sudo group:

root@ubuntu-4gb-nbg1-1:~# usermod -aG sudo bojana
root@ubuntu-4gb-nbg1-1:~# su - bojana
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.
bojana@ubuntu-4gb-nbg1-1:~$

Now, we can see that our hostname is some generic one, assigned by VPS provider when we provisioned the machine.

bojana@ubuntu-4gb-nbg1-1:~$ hostname
ubuntu-4gb-nbg1-1
bojana@ubuntu-4gb-nbg1-1:~$ hostnamectl
 Static hostname: ubuntu-4gb-nbg1-1
       Icon name: computer-vm
         Chassis: vm
      Machine ID: ff55896032564222b6ccd48ae7410afc
         Boot ID: 18b27cdedfe54250992d1180efc17cdb
  Virtualization: kvm
Operating System: Ubuntu 22.04.1 LTS
          Kernel: Linux 5.15.0-57-generic
    Architecture: x86-64
 Hardware Vendor: Hetzner
  Hardware Model: vServer

We will use hostnamectl command to set a new hostname which of course will be the same as our domain name.

bojana@ubuntu-4gb-nbg1-1:~$ sudo hostnamectl set-hostname vps-playground.cloud

Additionally, we will modify entry to /etc/hosts file :

bojana@ubuntu-4gb-nbg1-1:~$ sudoedit /etc/hosts
127.0.1.1 vps-playground.cloud vps-playground
127.0.0.1 localhost

bojana@ubuntu-4gb-nbg1-1:~$ hostnamectl
 Static hostname: vps-playground.cloud
       Icon name: computer-vm
         Chassis: vm
      Machine ID: ff55896032564222b6ccd48ae7410afc
         Boot ID: 18b27cdedfe54250992d1180efc17cdb
  Virtualization: kvm
Operating System: Ubuntu 22.04.1 LTS
          Kernel: Linux 5.15.0-57-generic
    Architecture: x86-64
 Hardware Vendor: Hetzner
  Hardware Model: vServer

You will notice that your prompt hasn’t changed, and still displays the old hostname. Try logging out and then back in and the new hostname should be visible in the prompt.

bojana@vps-playground:~$

Now, to verify our domain is correctly set up, try issuing this command from your local machine:

[bojana@K7OfNine ~]# drill vps-playground.cloud
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 54204
;; flags: qr rd ; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; vps-playground.cloud.        IN      A

;; ANSWER SECTION:
vps-playground.cloud.   0       IN      A       142.132.165.72

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 124 msec
;; SERVER: 172.20.160.1
;; WHEN: Thu Jan 12 13:01:35 2023
;; MSG SIZE  rcvd: 74

We can see our record in answer section.

After the domain is setup correctly, we have to ensure that the ports 80 and 443 on VPS are open.

Depending on your distribution, how to do this may vary. I am using Ubuntu which comes with ufw preinstalled, which is a more user-friendly interface over iptables. To allow ports 80 and 443 we add two rules:

bojana@vps-playground:~$ sudo ufw allow 80/tcp
Rule added
Rule added (v6)
bojana@vps-playground:~$ sudo ufw allow 443/tcp
Rule added
Rule added (v6)

And to verify the rules are added:

bojana@vps-playground:~$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW       Anywhere
80/tcp                     ALLOW       Anywhere
443/tcp                    ALLOW       Anywhere
22/tcp (v6)                ALLOW       Anywhere (v6)
80/tcp (v6)                ALLOW       Anywhere (v6)
443/tcp (v6)               ALLOW       Anywhere (v6)

Docker and Docker compose

As I mentioned in the intro, we will deploy Bitwarden as an array of Docker containers. A precondition for this is to have Docker Engine installed. To install Docker Engine on your VPS it’s best to follow official documentation and install it using the repository. Here, in the Server section, you can find instructions for your distribution of choice.

Once complete, verify that you have both docker and docker compose installed:

bojana@vps-playground:~$ sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
2db29710123e: Pull complete
Digest: sha256:aa0cc8055b82dc2509bed2e19b275c8f463506616377219d9642221ab53cf9fe
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.
.......
bojana@vps-playground:~$ docker compose version
Docker Compose version v2.14.1

Bitwarden user and directory

Now that we have docker installed, Bitwarden documentation recommends creating a dedicated Bitwarden service account.

To create bitwarden user:

bojana@vps-playground:~$ sudo adduser bitwarden

Add bitwarden user to the docker group:

sudo usermod -aG docker bitwarden

Create bitwarden directory:

sudo mkdir /opt/bitwarden

Set permissions for the /opt/bitwarden directory and set bitwarden user as the owner

sudo chmod -R 700 /opt/bitwarden
sudo chown -R bitwarden:bitwarden /opt/bitwarden

Install Bitwarden

Before retrieving the installation script and running it, first ensure you visit https://bitwarden.com/host/ to request hosting installation Id and Key. After filling out firm with your email, you will retrieve the  installation id and key. Keep it somewhere, because we will need it during installation.

Before proceeding with installation script, let’s switch to user we created in the previous step and position to /opt/bitwarden directory.

bojana@vps-playground:~$ su - bitwarden
Password:
bitwarden@vps-playground:~$ cd /opt/bitwarden/

Now, we need to download the installation script for Bitwarden:

bitwarden@vps-playground:~$ curl -Lso bitwarden.sh <https://go.btwrdn.co/bw-sh> && chmod 700 bitwarden.sh

And run the script:

./bitwarden.sh install

First, we will be prompted to enter our domain name:

bitwarden@vps-playground:/opt/bitwarden$ ./bitwarden.sh install
 _     _ _                         _
| |__ (_) |___      ____ _ _ __ __| | ___ _ __
| '_ \\| | __\\ \\ /\\ / / _` | '__/ _` |/ _ \\ '_ \\
| |_) | | |_ \\ V  V / (_| | | | (_| |  __/ | | |
|_.__/|_|\\__| \\_/\\_/ \\__,_|_|  \\__,_|\\___|_| |_|

Open source password management solutions
Copyright 2015-2023, 8bit Solutions LLC
<https://bitwarden.com>, <https://github.com/bitwarden>

===================================================

bitwarden.sh version 2023.1.0
Docker version 20.10.22, build 3a2c30b
Docker Compose version v2.14.1

(!) Enter the domain name for your Bitwarden instance (ex. bitwarden.example.com): vps-playground.com

Next, we will enter y to use Let’s Encrypt to generate a free SSL cert:

(!) Do you want to use Let's Encrypt to generate a free SSL certificate? (y/n): y

Next, enter an email where Let’s Encrypt will notify you about SSL expirations, enter a  valid email so you can receive the email and be aware:

(!) Enter your email address (Let's Encrypt will send you certificate expiration reminders):

At this point, you should  see output from docker pulling the certbot image and retrieving the certificate:

Using default tag: latest
latest: Pulling from certbot/certbot
ca7dd9ec2225: Pull complete
9e124a36b9ab: Pull complete
42cba90def7f: Pull complete
036c0ab6a768: Pull complete
de6312618cf7: Pull complete
f2b159e8e18d: Pull complete
5c3094a661e9: Pull complete
ae4269d8cd1f: Pull complete
fc17a613a054: Pull complete
7560c853872e: Pull complete
ab060fadf2d2: Pull complete
b81696353590: Pull complete
144b4c29fbe6: Pull complete
Digest: sha256:f632e55104da84ba5b54bc858a67ac6c9b4f68790ea823515867c993153c3fb4
Status: Downloaded newer image for certbot/certbot:latest
docker.io/certbot/certbot:latest
Saving debug log to /etc/letsencrypt/logs/letsencrypt.log
Account registered.
Requesting a certificate for vps-playground.cloud

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/vps-playground.cloud/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/vps-playground.cloud/privkey.pem
This certificate expires on 2023-04-17.
These files will be updated when the certificate renews.

(!) Enter the database name for your Bitwarden instance (ex. vault):

This is pretty self-explanatory, enter the database name for Bitwarden instance.

At this point image called bitwarden/setup is being downloaded and after it’s finished it will prompt you to enter the installation id and installation key we retrieved from bitwarden’s website:

(!) Enter your installation id (get at <https://bitwarden.com/host>):
(!) Enter your installation key:

After this appropriate keys will be generated and installation should be complete.

Now we should start bitwarden with:

./bitwarden.sh start

This will pull all the necessary Docker images and start the containers.

If everything went well you should see the following message:

Bitwarden is up and running!
===================================================

You can verify all containers running by issuing docker ps command.

Now, try visiting your domain, and you should be able to see Bitwarden login/signup screen:

First, we need to create an account so click the link. Enter the required fields, of course choose a strong Master password, and click create account.

Now, you can log in to your self-hosted Bitwarden instance:

Set up 2FA

For the next step I would recommend is to enable 2FA.

Go to Account settings within your Bitwarden account, and click on the Security tab and then “Two-step login” on the “Providers” list click Manage for Authenticator app.

At this point, you should have installed either Google Authenticator or Authy app. I have Authy.

It will once more prompt you for Bitwarden’s master password, and after you should be able to see the QR code and scan it using Authy/Google authenticator app. A new app should be added to your list. And that’s it! You have successfully configured 2FA for your self-hosted Bitwarden instance.

As the warning in a screenshot on the Two-step login configuration page suggests you should backup recovery codes, in case something happens to the device you are using for 2FA.

A few notes about some additional features of Bitwarden and the installation itself. In this guide, I did not cover how to set up Bitwarden to be able to send various emails. SMTP configuration for Bitwarden can be set up in ./bwdata/env/global.override.env file by providing values for SMTP host, port, and such.

Bear in mind that the installation script uses ./bwdata/config.yml to generate the necessary assets for installation. After every change to this yml file you should run:

./bitwarden.sh rebuild

For instance, on one Bitwarden instance, I had to change the default ports (80 and 443) to something else because I already had some other  app using the ports on that host. So I had to modify the config and rebuild the images.

One additional note is that besides using  web access, you can also use Bitwarden desktop app. Just make sure that on Settings you enter the proper server URL for your instance.

After quite a bit of work, I must admit :) you have an operational, self-hosted, Bitwarden instance.

Hooray! 🎉

DNS - Domain Name System, is maybe the easiest protocol to grasp. At least, to grasp what its fundamental purpose is, and that is to translate or map human readable - symbolic names (such as bojana.dev) to IP addresses (such as 157.90.124.251).

It is also possible to map IP address to some symbolic name, although not as often used, but it is possible, and can come in  handy (for instance in when issuing traceroute command).

While DNS's fundamental purpose is pretty easy to grasp for everyone, it's internal workings are maybe not interesting for everyone, but since you are here reading about it, let's dive in. :)

• A bit of history
• DNS structure
• How DNS is organized
• How it works
• Name resolving

A bit of history

Once upon time, in a dawn of Internet, there was a file called hosts.txt which contained list of all computer names and their IP addresses. To be "up-to-date",  all computers had to download this file during the night, from a location where it was hosted. For a network of couple hundred computers, this wasn’t a big deal.

But, as network grow, and thousands of computers were added to the network, it was clear that this way of organization is not sustainable. First of all, the file would get too big. And second of all, and more important, if the names weren’t managed in one central place, name collisions would be often, which would be unimaginable for the big international network, due to potential load and delay.

For purpose of solving above mentioned problems, DNS was created. In the core od DNS system is hierarchy of domain names and the system of distributed databases that implements that structure.

As a relict of the past, hosts.txt still exist on all operating systems, not just as a backup for when we don’t have DNS available, but it enables to override contents of DNS. This can come in handy for local development purposes, for instance.

127.0.0.1 localhost
172.16.4.221 develop.mydomain.com

DNS Structure

DNS is a hierarchy structure, that is global (whole Internet).

It begins with root domain, with a label of an empty string (“ “). Every node in this tree is a symbolic name.

The path from the root over individual nodes generates a full name of domain, so called - Fully Qualified Domain Name (FQDN). Usually the dot at the end is omitted.

Fully qualified domain name consist of multiple segments:

• each segment consist of max. 63 characters
• maximum length of FQDN is 255 characters
• ASCII characters: letters, numbers, “_”, “-”
• Case-insensitive

etf.unibl.org

• org - is Top Level Domain (TLD)
• unibl - labels, subdomains
• etf - label, can be the name of a device and a subdomain

Examples:

• el.etf.unibl.org
• blog.bojana.dev
• www.youtube.com

TLD - subdomains of root domain

The root domain contains all top-level domains of the Internet. The core group of generic top-level domains consists of the com, net, org, biz and info domains. The number of generic Top Level Domains (gTLD) as of March 2018 exceeds 1200 domains.

ccTLD - Country Code TLD

These top level domains belong to individual countries. They were assigned by IANA  by two-letter ISO 3166 code of the country.

Some examples:

• rs - Republic of Serbia
• sr - Surinam
• eu - European Union
• ba - Bosnia and Herzegovina

While gTLDs have to obey international regulations, regulations related to ccTLD are regulated by each country domain regulation body. For example, for .rs domain it's "Serbian National Internet Domain Registry"  (RNIDS) , for .ba it's "Univerzitetski tele-informatički centar - UTIC", for .hr it's CARNet - Croatian Academic and Research Network, for .me it's Government of Montenegro :) and so on.

As you can see some TLD are more "desirable" than others. In general, anyone can buy or better say "lease" a domain for a period of at least one year (that's a period for renewal - meaning you will have to pay a fee for "owning" a domain each year).

Fun fact: .tv is ccTLD for Tuvalu, an island state in Polynesia. For it's abbreviation for "television" the .tv is quite popular, so much that 8.4% of the revenue of the Government of Tuvalu came from .tv royalties.
Similair story goes for .fm a ccTLD for Federated States Of Micronesia, also an island state, in Oceania. I think you can guess why this domain is popular.

How DNS is organized

Logical structure is physically organized in a distributed way. Whole tree is divided into ZONES.

DNS zone

Part of the tree (one or more nodes). It contains information of included domains, usually one zone is one node (domain). Administratively speaking, one zone belongs to one organization (one company, or university, or state, etc.). In a nutshell it is one text file defined on one server - DNS or NS server (name server).

Topology of domains is technically completely independent of physical network topology.

Computers/servers from one domain can belong to different, physically disjointed and separated networks. And on the other hand, computers/servers from one physical network can belong to different domains.

How it works

Primary DNS is defined for some node in the tree above (some domain), and the administrator on that primary DNS Server can modify data, add domains, change names, records, etc.

As it is not good for single server to be responsible for domains, we have a redundant copies on one or more Secondary DNS Servers. It is also good that they are separated (distant) from the primary DNS servers, so that they can take over if primary DNS fails for whatever reason.

Secondary DNS server(s), periodically retrieves copy of DNS zone form Primary DNS server.

Both primary and secondary DNS servers are called authoritative DNS servers, because they have whole DNS zones for particular domains. Both primary and secondary DNS servers have equal role in name resolving.

Name resolving

DNS servers are resolving the queries from the clients.

Name resolving is a process of finding IP addresses for the requested domain (name).
DNS resolver is a piece of software that resides both on a client and a server side.
On the client side, DNS resolver is sending a query to locally configured DNS server (over UDP port 53), if there is no data in the local cache.

On the server side, DNS resolver will check the local cache or zones database, and if there is no data for a particular name, it will send the query  to other DNS servers (primary and secondary), authoritative servers, for corresponding domain.

DNS server, that local one for the client, when resolving the name, it’s doing that in a recursive manner. What that means? It means it will either return or resolve the name completely or it will report an error. How the server will resolve the name, client has no interest in, client just wants the name to be resolved.

DNS server will iteratively ask other servers when trying to resolve the name. Other DNS servers will return partially resolved name or “the best possible answer”, and it will refer to other DNS servers in the hierarchy that can resolve the name.

For ex. let’s say the client is requesting following domain : el.etf.unibl.org

As we said, client doesn’t have this in it’s local cache so it will ask locally configured DNS server. That locally configured DNS server also doesn’t have the name neither in cache or zones database.

What happens next?

Well, we said that all domains start from one root domain (the empty string “ ” is a root of tree structure from above).

These servers are 13 well known root DNS servers, on well known IPs, and web browser and operating systems already know which addresses are those.

1. Let’s say it will ask this root DNS server  - 198.41.0.4 where is el.etf.unibl.org ?
2. 198.41.0.4 will then “say” - “Ok, I have no idea what el.etf.unibl.org is, but I know what .org. Because .org is a TLD which is defined on root server. I know authoritative servers for .org domain. Here are the addresses for them.
3. Then on of those authoritative servers will know where is unibl.org and will again provide the list where etf.unibl.org is
4. And eventually fully qualified domain name will be resolved. (el.etf.unibl.org)

In the next article we will see how are DNS zones defined, and examine each Resource Record (RR) that zone is comprised of.

References:
[1] Andrew S. Tanenbaum - "Computer Networks"
[2] IANA.org
[3] Top level domain, Wikipedia.org
[4] Country code top level domain, Wikipedia.org
Special thanks to Zlatko Dejanović for peer reviewing the article.

In the world of programming you noticed that term “async programming” is very frequent, and most often then not someone will ask you (or tell you) to do that particular task in “asynchronous way”. But what that even means?

Before mentioning any concrete programming language (like C# or JavaScript), let’s try to understand what asynchronous programming is in a broader way.

To come closer to understanding, let’s see what async programming isn’t.

Parallel programming

In parallel programming you are performing (or rather the code you write) multiple calculations at the same point in time. This is usually some CPU-bound stuff you want to do “in parallel”. For instance, let’s say you have an image and you want to apply some filter to it or do some kind of transformation to it. You could go around and say “I want to the transformation for first half of the image (pixel this to pixel that) on CPU1, and the other part of the picture I want to be processed on CPU2”. And right there we have an important distinction between parallel and async programming, for parallel programming you need to have multiple processors (or multiple CPU cores), and for async programming you don’t need to.

In the diagram above we have an illustration of parallel programming. We have two threads (running on each of the CPUs), thread t1 can perform above mention image transformation (first part of the image), and t2 as well for the other part, and this is done “in parallel”. At some point our code needs to handle the “assembling of the parts”, but let’s not dwell too much on that here.

CPU blocking

Let’s now imagine that we have a single core CPU, and that we have a regular desktop app.

We are implementing some kind of UI application. If we start certain program, after the initialization the app is waiting for some kind of user input, so it is idle. At some point in time, user for ex. moves the mouse, app becomes active, app handles mouse movement or keyboard input, and  goes idle again.

At some point in time, application decides to go and grab something from the disk, or over the network, or fetch something from the database. That call to the database, or disk, or network, takes some time to complete. If we do not use async programming our thread will actively wait blocking the CPU. As if you were staring at the watch and waiting for something to complete, doing nothing.
So our thread is doing nothing until the database call is complete.
Since our app also have the user interface (which is also a thread), application will not react to mouse moves for instance. Application will be frozen for the entire time. This is not a good idea as you can see.

How we can fix this? Well, we can run a little bit of code which issues a db call.
Now, we are using async processing, letting the database do it's work in the background. Something else is busy, not the CPU.

Now, when the time is elapsed, we will be informed, actively. So, in one point in time database will tell us that it is finished, and that will trigger another piece of asynchronous code, where we can do the processing the results of database query.
The important thing is that in the meantime, our CPU, our UI thread, is free.

Web server

Now let’s imagine that we are building a C# app that has a web server built-in (Kestrel) and that we are running a single process. Now what about threads? How many threads will our web server create? Well, it turns out, at least it’s a case for a C#, these web servers have a pool of threads and the number of these threads in the thread pool are equal to number of CPU cores. Why just two you might ask? Why not 100? Well, it’s because turns out threads take a lot of memory, so they are not so cheap to create, and furthermore, it takes some time to create them.

This thread pool that we mentioned, have a limit (configurable), a number of threads a machine can handle, because thread pool cannot and should not grow indefinitely. It can outnumber the number of available cpu cores, but typically you try to keep the number as low as possible, in order to keep your web server efficient, otherwise you will be consuming a lot of memory.

Let’s take a look at the diagram below. Image we have a cpu with two cores and our above imagined web server. We have two threads in the thread pool. Each HTTP request is assigned an idle thread in the thread pool.

Now our t1 thread is executing on cpu2, it needs some time to process the request, and let’s now imagine it also needs to say query the db for some additional information. If we don’t do async programming our thread will be blocked for quite some time and the end we are sending the result (response) back to the client.

The problem is that we don’t have just one client, we have multiple clients.

Imagine for example that we have a client on the phone, that issues another HTTP request. We now have t2 available in the thread pool .

And again some time to process the request, request some additional data from the db etc.

Now we have a problem. What if we have 3rd and 4th and so on, HTTP request, or 100s of HTTP requests. What the system will do in the background, it will try to create additional threads (t3, t4, t5…) and will try to schedule those threads on the CPUs, but we have only limited number of CPUs (2 in our example), so these threads are waiting and waiting, they are idle. They are just blocking the CPU.

Async programming to the rescue

Let’s now see how asynchronous programming is solving these issues. We have the same scenario as above. HTTP GET comes in, and thread t1 in processing the request, but when the database call is issued (to retrieve additional info), we hand back the thread t1 to the thread pool.

So now, when our second user with the phone is issuing the request, it can go again to thread t1 and use t1. Again performing a little bit of processing of the incoming web request and accessing the database. Then the result of the first db call is processed and returned to the second client.

And if you take a look now, we only need a single thread and we can process multiple HTTP requests at the same time. We are not doing parallel programming here, we are doing asynchronous programming. The important takeaway is that you are always using asynchronous programming when you are doing some kind of non-CPU bound work (accessing the database, file system, network, etc.)
If you program like this, your web server will really be efficient and it will scale really well. They can handle much more requests, than if you do locked programming.

In web server development, you don't need to do parallel programming on your own, it's done by the web server, as far as for ex. .NET Core  goes (Kestrel).
NodeJS is however different, as JavaScript is single threaded.
This is the reason why in the cloud so many web server have single thread, because if you need more threads, you spin up additional servers.

Docker containers, for instance, are often single threaded. From the point of view of a single server, there is no multi threading, but from overall view there is multi threading, because you have multiple servers.

Note: I tried to summarize in a written form this excellent lecture about async programming by Rainer Stropek

If you work as a developer or in any other IT related field, you've probably come across the term "reverse proxy". From my experience people are more often than not confused by the term. So what is a reverse proxy?

Maybe it's better to start of by explaining what is a forward proxy (often called proxy server or web proxy).

If there were no forward proxy, client would reach out directly to Server A (or B or C), and Server A would respond. However since forward proxy is in place, client will reach out to forward proxy (on a diagram above), forward proxy will reach out to Server A. When Server A responds, it will contact forward proxy and forward proxy will forward response to the client.

The question is why use this middleman when clients want to connect to the Internet. There are several reasons:

• avoid browsing restriction - ex. some content is restricted based on country from which you are accessing it, forward proxy can get around this and let user connect to a proxy (ex. from some allowed country) rather than connecting directly to the visiting site.
• block access to content - since it’s sitting in front of clients, forward proxy can also be used to restrict access to certain sites. (for ex. in schools or libraries)
• to protect their online identity

OK, but what about reverse proxy?

In contrast to forward proxy which sits in front of the clients that are trying to connect, reverse proxy (also a server) sits in front of one or more origin servers, intercepting requests from the clients. The reverse proxy server will then send requests to and receive responses from the origin server.

If there were no reverse proxy in place, requests from the client would go directly to Server A (or B or C). With reverse proxy in place, all requests will go to reverse proxy, and reverse proxy will forward the requests to appropriate origin server - A. Reverse proxy will receive responses from A and forward to the client.

Again question arises of benefit of this schema? There are several benefits of using a reverse proxy:

• load balancing - with reverse proxy in place you can distribute the traffic across a group of servers to maximize speed and utilize capacity. Also if one of the servers in the group goes offline, the load balancer redirects traffic to remaining online servers.
• web acceleration - reverse proxy can also cache content as well as perform SSL encryption, taking the load off your web server, thus improving their performance.
• security - since no request go directly to your backends, reverse proxy also protects their identities and makes harder for attacker to leverage a targeted attack against them (such as DDOS attack).

nginx - famous reverse proxy server

Nginx was created in 2004 by Russian developer Igor Sysoev.
Frustrated with Apache, and wanting to build a repleacement capable of handling 10.000 concurrent connections, with focus on performance, high concurrency and low memory usage.

Today, nginx serves the majority of world's top 1000 websites. This popularity, besides it's excellent performance, can be credited to it's relative ease of configuration and due to the fact it's relatively easy to start with.

Although often mentioned in the context of web servers, nginx, as it's core is a reverse proxy server, and it is because of this design that it performs so well.

We will use nginx to illustrate the reverse proxy mechanism.

Hands on

To "follow along" you can use any linux host, be it a cloud hosted VPS or a local VM with some version of linux, or WSL.
I am using an Ubuntu 20.04 on a WSL.

You can install nginx with the package manager :

apt-get install nginx

bojana@7OfNine:~$ nginx -v
nginx version: nginx/1.18.0 (Ubuntu)

To keep things simple, lets create a simple nginx configuration file in the home directory:

bojana@7OfNine:~$ touch nginx/nginx.conf

And let's add these lines to the config file:

events { }

http {

  server {
    listen 8888;

    location / {
     return 200 " Hello from NGINX\\n";
    }
  }
}

To understand the configuration file, let's discuss a few of a nginx configuration terms. Context - sections withing the configuration (namely events, http, server in our file) where directives can be set for that given context. You can think of a context as a scope. Directives - specific configuration option (ex. listen 8888), that gets set in the configuration file, and consists of a name and a value.

To start nginx with given configuration issue a command:

bojana@7OfNine:~$ nginx -c /home/bojana/nginx/nginx.conf

The -c argument is indicating that we will pass a configuration file, it's also worth noting that the path to the config file should be absolute. A good practice before reloading the nginx configuration is to verify if there is no syntax errors in the file itself (such as misplaced bracket or some misspelled directive or similar), kind of a dry run of an nginx configuration, to verify everything is correct (at least as far as syntax goes). To perform the test issue:

bojana@7OfNine:~/nginx/node$ nginx -t -c /home/bojana/nginx/nginx.conf
nginx: the configuration file /home/bojana/nginx/nginx.conf syntax is ok
nginx: configuration file /home/bojana/nginx/nginx.conf test is successful

We can see that everything is fine with the configuration file.
Let's test if the nginx is actually running by curl-ing the location:

bojana@7OfNine:~$ curl http://localhost:8888
 Hello from NGINX

To demonstrate nginx working as a reverse proxy, let's now create simple php server. Prerequisite to create a php server is that you have phpXx-cli package installed with Xx being the appropriate version. On my machine I installed php7.4-cli:

bojana@7OfNine:~$ sudo apt install php7.4-cli

After it's been installed, create a simple php server specifying host and a port like this:

bojana@7OfNine:~$ php -S localhost:9999
[Mon Jun 27 14:50:07 2022] PHP 7.4.3 Development Server (<http://localhost:9999>) started

PHP's builtin server defaults to serving the directory it's run from, so in our case that nginx directory.

We get a 404, because we didn't specify a path, nor we have index file in that root directory.

If we create the index.php file in that directory (for ex. with phpinfo function) and restart the php server from above. We will get the response.

<?php phpinfo(); ?>

bojana@7OfNine:~/nginx$ php -S localhost:9999
[Mon Jun 27 14:58:07 2022] PHP 7.4.3 Development Server (<http://localhost:9999>) started
[Mon Jun 27 14:58:10 2022] 127.0.0.1:48410 Accepted
[Mon Jun 27 14:58:10 2022] 127.0.0.1:48410 [200]: GET /

For the sake of simplicity let's just serve a simple text file:

bojana@7OfNine:~$ touch nginx/php_hello.txt
bojana@7OfNine:~$ echo "Hello from PHP" > nginx/php_hello.txt
bojana@7OfNine:~$ php -S localhost:9999 nginx/php_hello.txt
[Mon Jun 27 15:06:39 2022] PHP 7.4.3 Development Server (http://localhost:9999) started
....

bojana@7OfNine:~$ curl http://localhost:9999
Hello from PHP

Let's now say we want to access this php server via our nginx server. (the one on port 8888), for example by accessing  http://localhost:8888/php.

events { }

http {

  server {
    listen 8888;

    location / {
     return 200 " Hello from NGINX\\n";
    }

    location /php {
     proxy_pass "http://localhost:9999/";
    }

  }
}

We added new location (/php), and with directive "proxy_pass" we forwarded request to the php server.

Now let's reload the nginx configuration:

bojana@7OfNine:~$ sudo nginx -s reload

And let's test our reverse proxy by requesting that /php location :

bojana@7OfNine:~$ curl http://localhost:8888/php
Hello from PHP

We get a response from a PHP server, but accessed and reverse proxied via our nginx server.

There is no restrictions to this proxied server being on the same system either. For instance if I add this location:

location /blog {
      proxy_pass "https://bojana.dev/";
 }

bojana@7OfNine:~$ sudo nginx -s reload
bojana@7OfNine:~$ curl http://localhost:8888/blog
<!DOCTYPE html>
<html lang="en">
<head>

    <title>Bits and pieces</title>

It is clear that you can use any kind of backend to sit behind the nginx.

Here is the example with a sample node/express app as an illustration.

 location / {
     proxy_pass http://localhost:3000; # sample node app
    }

bojana@7OfNine:~/nginx/node$ node app.js
Example app listening on port 3000

bojana@7OfNine:~$ curl http://localhost:8888
Hello World!

const express = require('express')
const app = express()
const port = 3000

app.get('/', (req, res) => {
  res.send('Hello World!')
})

app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

I also have a sample asp.net core app running locally on Kestrel server. If I add this entry to nginx.conf:

location / {
         proxy_pass         http://localhost:5000;
}

bojana@7OfNine:~/aspnet-test$ /usr/bin/dotnet /home/bojana/aspnet-test/bin/Release/net6.0/aspnet-test.dll
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: <http://localhost:5000>
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: <https://localhost:5001>
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /home/bojana/aspnet-test/

I can access it on localhost:8888

So we see that it doesn't really matter what sits behind your nginx proxy server, php backend, node backend, dotnet backend... You can utilize it easily as a reverse proxy.

During the installation, Docker creates three different networking options.
You can list them with docker network ls:

bojana@linux:~$ docker network ls
NETWORK ID NAME DRIVER SCOPE
0c872e6d6453 bridge bridge local
10826dd62a8b host host local
cab99af2344e none null local

By default, bridge mode is selected, and containers reside on a private namespaced network within the host.
In the previous post, where we explored basic docker commands, we use docker run -p, to map a port from the host. This makes Docker create iptables rules that route traffic from the the host to the container.

“None” entry in the above list indicates that no configuration should be performed by Docker whatsoever. It is intended for custom networking requirements.
If we want explicitly to select container’s network, we pass –net to docker run.

A bridge is a Linux kernel feature that connects two network segments.
When you installed Docker, it quietly created a bridge called docker0 on the host.
You can verify that by issuing command ip addr show. Here is output on my machine:

bojana@linux:~$ ip addr show
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 96:00:00:b1:5e:b3 brd ff:ff:ff:ff:ff:ff
inet 188.34.194.63/32 scope global dynamic eth0
valid_lft 74220sec preferred_lft 74220sec
inet6 2a01:4f8:1c1c:a675::1/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::9400:ff:feb1:5eb3/64 scope link
valid_lft forever preferred_lft forever
3: docker0: mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:b6:9c:25:22 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:b6ff:fe9c:2522/64 scope link
valid_lft forever preferred_lft forever
5: veth51c3665@if4: mtu 1500 qdisc noqueue master docker0 state UP group default
link/ether 96:4d:57:fa:c0:99 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::944d:57ff:fefa:c099/64 scope link
valid_lft forever preferred_lft forever

docker0 is the virtual Ethernet bridge that uses 172.17.0.1/16 range.
veth51c3665 is the host side of the virtual interface pair that connect the container to the bridged network.

As we said in the previous article, by default when you launch a container, there are no published ports, so container would not be visible from outside the docker host, still we can access it from the docker host itself.

We will use nginxdemo/hello image in this example, because it contains the simple web server that prints out the IP address of the container, so a view form the inside, along with other things.

bojana@linux:~$ docker run -d nginxdemos/hello

Let’s connect to this running container and see what IP address it got assigned.
To access a shell of a running container we use:

bojana@linux:~$ docker exec -it jovial_wu /bin/ash

Note: jovial_wu is a random container name assigned when creating it, because we didn’t specify any, check yours with docker ps command

ip addr show on the container command line reveals:

/ # ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
8: eth0@if9: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether 02:42:ac:11:00:03 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.3/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever

We see that the address is from the above mentioned docker0 range.
If we paste this address to the browser, we should get nginx demo page:

It shows us what it sees from the inside of container, such as IP on the bridge plus the servername which is a linux hostname of the container (container id)

How can containers communicate with each other on the bridge?

Let’s create a second instance of the container:
docker run -d nginxdemos/hello

And connect to it:
docker exec -it inspiring_mcnulty2 /bin/ash

If we check the IP we see again that it’s from the bridge range:

/ # ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
11: eth0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether 02:42:ac:11:00:04 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.4/16 brd 172.17.255.255 scope glo

We can also ping the previously created container:

/ # ping 172.17.0.3
PING 172.17.0.3 (172.17.0.3): 56 data bytes
64 bytes from 172.17.0.3: seq=0 ttl=64 time=0.204 ms
64 bytes from 172.17.0.3: seq=1 ttl=64 time=0.189 ms
64 bytes from 172.17.0.3: seq=2 ttl=64 time=0.181 ms
64 bytes from 172.17.0.3: seq=3 ttl=64 time=0.192 ms
^C
--- 172.17.0.3 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.181/0.191/0.204 ms

This means we do have IP connectivity between two containers on the same bridge.
If we issue an ip route command, we can see there is a default route via the Docker host’s IP
on the bridge, which acts as a gateway.
We can also see that we have internet access from within the container, and the name resolution works:

/ # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=111 time=18.209 ms
64 bytes from 8.8.8.8: seq=1 ttl=111 time=18.524 ms
64 bytes from 8.8.8.8: seq=2 ttl=111 time=18.116 ms
64 bytes from 8.8.8.8: seq=3 ttl=111 time=18.414 ms
^C
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 18.116/18.315/18.524 ms
/ # ping google.com
PING google.com (142.250.185.110): 56 data bytes
64 bytes from 142.250.185.110: seq=0 ttl=112 time=28.885 ms
64 bytes from 142.250.185.110: seq=1 ttl=112 time=28.785 ms
64 bytes from 142.250.185.110: seq=2 ttl=112 time=29.383 ms
64 bytes from 142.250.185.110: seq=3 ttl=112 time=28.670 ms
^C
--- google.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 28.670/28.930/29.383 ms

In the default bridge configuration, all containers can communicate with one another because they are all on the same virtual network. However, you can create additional network namespaces to isolate containers from one another.

We said that the most common way of making a docker container visible from the outside is port mapping.

Let’s create another container this time specifying port mapping:

bojana@linux:~$ docker run -d nginxdemo/hello -p 81:80

One thing we should point out though is the fact that the name resolution between docker containers doesn’t work.
If we would attach to the above created container and tried to ping the first or the second container we created by it’s hostname, we wouldn’t be able to do so:

/ # hostname
0888f0c68423
/ # ping 167bcc074170
ping: bad address '167bcc074170'

User defined bridges

In principle, host name resolution is possible between the containers, just not on the default bridge.
Which means we have to create our own.
Use the docker network create command to create a user-defined bridge network:
bojana@linux:~$ docker network create --driver=bridge --subnet=172.172.0.0/24 --ip-range=172.172.0.128/25 --gateway=172.172.0.1 my-br0

You can go ahead and stop/remove all the containers we created so far so we have a clean slate.
Now, let’s create a new container that uses our newly created bridge:
bojana@linux:~$ docker run -d --name my-nginx --network my-br0 --hostname my-nginx nginxdemos/hello

And another instance with different name and hostname:
bojana@linux:~$ docker run -d --name my-nginx2 --network my-br0 --hostname my-nginx2 nginxdemos/hello

Now, if we use the docker inspect to check the network settings of newly created container:

bojana@linux:~$ docker inspect -f '{{ json .NetworkSettings.Networks }}' my-nginx
{"my-br0":{"IPAMConfig":null,"Links":null,"Aliases":["b91bde6aa527","my-nginx"],"NetworkID":"f9e797b2ab8826342ea8343b8bebe8e1459eea114aa0b015f56b01f90fe39ff8","EndpointID":"a1af70992c90967ebca917eeebaa67b4e7bc23bdfb2f352f283faaee223d4fc0","Gateway":"172.172.0.1","IPAddress":"172.172.0.128","IPPrefixLen":24,"IPv6Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"MacAddress":"02:42:ac:ac:00:80","DriverOpts":null}}

We see that our network settings were applied and containers have assigned, or if you will, joined our defined network. Now, if we try to ping one container from the other by it’s hostname, it will work:

/ # ping my-nginx2
PING my-nginx2 (172.172.0.129): 56 data bytes
64 bytes from 172.172.0.129: seq=0 ttl=64 time=0.166 ms
64 bytes from 172.172.0.129: seq=1 ttl=64 time=0.149 ms
64 bytes from 172.172.0.129: seq=2 ttl=64 time=0.211 ms
64 bytes from 172.172.0.129: seq=3 ttl=64 time=0.134 ms
^C
--- my-nginx2 ping statistics ---
And also if we ping by hostname another container:
/ # ping my-nginx
PING my-nginx (172.172.0.128): 56 data bytes
64 bytes from 172.172.0.128: seq=0 ttl=64 time=0.342 ms
64 bytes from 172.172.0.128: seq=1 ttl=64 time=0.134 ms
64 bytes from 172.172.0.128: seq=2 ttl=64 time=0.191 ms
64 bytes from 172.172.0.128: seq=3 ttl=64 time=0.128 ms
^C
--- my-nginx ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.128/0.198/0.342 ms

The conclusion is that Docker does provide name resolution between containers over a self-defined bridge.

Credits: A lot of content is borrowed from this excellent video by OneMarcFifty on youtube. Go and check the channel, it has some really interesting content, presented in a really nice and clean way. For this post, I tried to use just docker commands without having to use portainer and add some additional notes.

• Installing Docker
• Docker – open source container engine
• Basic architecture
• Basic Docker commands

Installing Docker (Linux, Windows, Mac)

I will not go into much details about how to install docker environment on your desired OS. There are a lot of quality articles out there that will help you with that. Official docker documentation seems to be a good option, but it’s up to you. Here are official guides for installing docker engine (desktop) on Ubuntu, Windows and MacOS.

Docker – open source container engine

In the last article we talked about how containerization has been a big trend in the IT world for the past few years, and it continues to be as we speak. Although it’s kind of a synonym for containers these days, it’s worth noting that Docker isn’t the only container engine or solution for managing containers out there. Container engines such as LXC, RKT, Railcar, CRI-O and others also exist. We also mentioned OCI – Open Container Initiative, a consortium created for purpose of creating open industry standards around container formats and runtimes.

Basic architecture

Docker uses a client-server architecture.
docker is an executable, a client, that talks to dockerd, a daemon process, that implements container and images operations. These two can run on the same system or the client can connect to a remote docker daemon.The client communicates with docker daemon over UNIX sockets, via REST API or over a network interface. Another example of docker client is docker compose.

As said above, docker client is the one who interfaces with docker daemon, by sending commands over the command line.

bojana@linux:~$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.6.1-docker)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 10
  Running: 2
  Paused: 0
  Stopped: 8
 Images: 5
 Server Version: 20.10.8
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
...........

An image is a template for the container. Typically, images are based on some Linux distribution with some additional customizations (these are called parent images, in you Dockerfile it refers to the content of FROM directive). This is because Linux distributions define a complete operating environment. Image doesn’t have to be based on some Linux distribution. For example, there is “scratch” image, an explicitly empty image, intended to be a basis for other images.

You may build an image which is based on the debian image, but installs .NET Core SDK and your application, as well as the configuration details needed to make your application run.

Basic Docker commands

To create your own image, you create a file name Dockerfile, with simple directives on how to create the image and run it.

Typically images are pulled from some private or public docker registry (such as Docker Hub).
To pull the image from the registry we use command docker pull:

bojana@linux:~$ docker pull debian
Using default tag: latest
latest: Pulling from library/debian
955615a668ce: Pull complete 
Digest: sha256:08db48d59c0a91afb802ebafc921be3154e200c452e4d0b19634b426b03e0e25
Status: Downloaded newer image for debian:latest
docker.io/library/debian:latest

The hex strings are the layers of the union file system.
Each instruction in a Dockerfile creates a layer in the image. When you change the Dockerfile and rebuild the image, only those layers which have changed are rebuilt. This is part of what makes Docker so lightweight and fast.

In the example above we used docker pull command to retrieve a docker image for Debian Linux distribution. Because we didn’t requested specific version or a tag it downloaded “latest” tag by default. On the docker hub web page you can find supported tags and respective Dockerfile links for each image of interest. https://hub.docker.com/_/debian

To inspect all locally available images, we use command docker images

bojana@linux:~$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
postgres latest 293e4ed402ba 3 months ago 315MB
jenkins/jenkins lts de181f8c70e8 4 months ago 569MB
portainer/portainer latest 580c0e4e98b0 5 months ago 79.1MB
hello-world latest d1165f221234 6 months ago 13.3kB

When we have an image, it very easy to create a running container out of it.

bojana@linux:~$ docker run -p 80:80 --detach --name web nginx:latest
Unable to find image 'nginx:latest' locally
latest: Pulling from library/nginx
a330b6cecb98: Pull complete 
5ef80e6f29b5: Pull complete 
f699b0db74e3: Pull complete 
0f701a34c55e: Pull complete 
3229dce7b89c: Pull complete 
ddb78cb2d047: Pull complete 
Digest: sha256:a05b0cdd4fc1be3b224ba9662ebdf98fe44c09c0c9215b45f84344c12867002e
Status: Downloaded newer image for nginx:latest
aab3ce850b9a2a6a8f2469b8773362133f95839406ce276a6d04d2bfc2b7f67f

We didn’t have the “nginx” image locally, so Docker had to pull it from the registry.
When we run this command, docker will install nginx:latest from the nginx repository hosted on dockerhub and run the software. After that one line of seemingly random characters will be displayed.
This is the identifier of the container we just created.
It seems nothing else happened, because we used –detach option and started the program in the background. Detached – means detached from the terminal. Typically, server software runs in detached mode because we don’t want to depend on terminal session.

We use docker ps to list all running containers:

bojana@linux:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8b101295f67e nginx:latest "/docker-entrypoint.…" 3 minutes ago Up 3 minutes 80/tcp

By default, when you create or run a container using docker create or docker run, it does not publish any of its ports to the outside world. Using the flag -p or –publish in the docker run command, will create a firewall rule which maps a container port to a port on the Docker host to the outside world.

With NGINX running in the container and port 80 mapped from the host, we can make HTTP requests to the container with curl. NGINX serves a generic HTML landing page by default.

bojana@linux$ curl localhost
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

If we want to see STDOUT from the container we use docker logs command, in this case nginx access log.

bojana@linux:~$ docker logs web
172.17.0.1 - - [06/Sep/2021:13:34:47 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.68.0" "-"

If we add -f flag we can examine a real-time stream of container’s output. To debug or troubleshoot the running container we could also start an interactive shell

bojana@linux:~$ docker exec -it nginx bash
root@aab3ce850b9a:/# apt-get update && apt-get -y install procps
root@aab3ce850b9a:/# ps ax
    PID TTY      STAT   TIME COMMAND
      1 ?        Ss     0:00 nginx: master process nginx -g daemon off;
     32 ?        S      0:00 nginx: worker process
     33 ?        S      0:00 nginx: worker process
     34 ?        S      0:00 nginx: worker process
     35 ?        S      0:00 nginx: worker process
     43 pts/0    Ss     0:00 bash
    372 pts/0    R+     0:00 ps ax

The process list reveals the nginx master daemon, an nginx worker, and our bash shell. When we exit the shell created with docker exec, the container continues to run.

We can start and stop the container:

bojana@linux:~$ docker stop nginx
nginx
bojana@linux:~$ docker ps
CONTAINER ID   IMAGE       COMMAND        CREATED      STATUS       PORTS   
bojana@linux:~$ docker start nginx
nginx
bojana@linux:~$ docker ps
CONTAINER ID   IMAGE       COMMAND      CREATED        STATUS       PORTS                                                                                  NAMES
aab3ce850b9a   nginx:latest "/docker-entrypoint.…"   52 minutes ago   Up 5 seconds            0.0.0.0:80->80/tcp, :::80->80/tcp  

When containers are stopped/exited they remain in the system in the dormant state. We can see all the containers on the local system (running/exited) by supplying -a flag to the docker ps command:

bojana@linux:~$ docker ps -a
CONTAINER ID   IMAGE                    COMMAND                  CREATED             STATUS                         PORTS                                                                                  NAMES
aab3ce850b9a   nginx:latest             "/docker-entrypoint.…"   54 minutes ago      Up 2 minutes                   0.0.0.0:80->80/tcp, :::80->80/tcp                                                      web
65787d8734ab   debian                   "/bin/echo 'Hello Wo…"   About an hour ago   Exited (0) About an hour ago                                                                                          strange_napier
33d2c89cdc09   nginxdemos/hello         "/docker-entrypoint.…"   7 days ago          Exited (0) 7 days ago    

It’s not particularly bad to keep them lying around, but it’s considered bad practice (due to possible name collisions with new containers). We stop and remove the container with:

bojana@linux:~$ docker stop nginx && docker rm nginx
nginx
nginx

And if we want to remove the image:

bojana@linux:~$ docker rmi nginx
Untagged: nginx:latest
Untagged: nginx@sha256:a05b0cdd4fc1be3b224ba9662ebdf98fe44c09c0c9215b45f84344c12867002e
Deleted: sha256:822b7ec2aaf2122b8f80f9c7f45ca62ea3379bf33af4e042b67aafbf6eac1941
Deleted: sha256:47dec9bde9e483e6265a6f52ab1e726724927e2e9d2ac358fdf58fbfcd6cf418
Deleted: sha256:7920a27f48f198550d59f64681b99441bbc3d2ce4778a855ce1ef9bafc96ae69
Deleted: sha256:a3c5a94eb1ea071c73dcea1969e0b71beea445d3b9d0735eaf6715d8e351434c
Deleted: sha256:e73eb58ed241e67a7a2c8589dde85eb72811eac1eb4cf3b586e40d2b9cc9d0c1
Deleted: sha256:b5d976dc9b0fa380affe1f6a17df18f02ab7debec2d35a0407fb863338591ed7
Deleted: sha256:d000633a56813933cb0ac5ee3246cf7a4c0205db6290018a169d7cb096581046

These are some basic docker commands to get you up and running when it comes to dealing with images and containers. There are a lot of other docker commands related to networking, volumes, building images etc. that we will be tackling in some of the upcoming blog posts.

In the last article, I talked about what virtualization is and how important the concept and technology is in the whole cloud computing paradigm.

Now I want to talk about containerization – a different approach to isolation that does not use a hypervisor, but instead it relies on a specific kernel features that isolate processes from the rest of the system.

So in short containerization is a form of operating system virtualization, where we have applications running in isolated user spaces called containers.

Each process “container” or “jail” has a private root file system and process namespace.
While sharing the kernel and other services of the underlying OS, they cannot access files or resources outside of their container.
In essence, containers are fully packaged and portable computing environments.

We said that this type of virtualization does not require hypervisor, and because there is no need for the virtualization of the hardware, resource overhead for this type of virtualization is low.

This means container start up time is pretty quick, or if you will start time is pretty short. Creation of a container has the overhead of creating a Linux process, which can be of the order of the miliseconds, while creating a VM can take seconds.

The containerized application can be run on various types of infrastructure—on bare metal, within VMs, and in the cloud—without needing to refactor it for each environment

How then containers relate to VMs?

Both are portable, isolated, execution environments, and both look and act as a full operating systems.
Unlike VM, which has an OS kernel, drivers to interact with hardware etc. a container is merely a mimic of an operating system. Container itself is abstracted away from the host OS, with only limited access to underlying resources – we can say it is a lightweight VM.

The containers-on-VMs architecture is standard for containerized applications that need to run on public cloud instances.

How containerization actually works?

We said that containerization relies on specific kernel features, but which features are those?
Containerization as we know it evolved from cgroups, a feature for isolating and controlling resource usage (e.g., how much CPU and RAM and how many threads a given process can access) within the Linux kernel.
cgroups were originally developed by Paul Menage and Rohit Seth of Google, and their first features were merged into Linux 2.6.24. Cgroups became Linux containers (LXC), with more advanced features for namespace isolation of components, such as routing tables and file systems.

Namespaces are a kernel mechanism for limiting the visibility that a group of processes has of the rest of a system. For example you can limit visibility to certain process trees, network interfaces, user IDs or filesystem mounts. namespaces were originally developed by Eric Biederman, and the final major namespace was merged into Linux 3.8.

Since kernel version 5.6, there are 8 kinds of namespaces. Namespace functionality is the same across all kinds: each process is associated with a namespace and can only see or use the resources associated with that namespace, and descendant namespaces where applicable. This way each process (or process group thereof) can have a unique view on the resources.

bojana@linux:~$ sudo lsns -p 270
        NS TYPE   NPROCS PID USER COMMAND
4026531835 cgroup    130   1 root /sbin/init
4026531836 pid       126   1 root /sbin/init
4026531837 user      130   1 root /sbin/init
4026531838 uts       123   1 root /sbin/init
4026531839 ipc       126   1 root /sbin/init
4026531840 mnt       114   1 root /sbin/init
4026531992 net       126   1 root /sbin/init

In Linux, lsns lists information about all the currently accessible namespaces

• cgroup
• mnt (mount points, filesystems)
• pid (processes)
• net (network stack)
• ipc (System V IPC)
• uts (hostname)
• user (UIDs)
• time namespace

Modern containers evolved from these 2 kernel features, and LXC served as a basis for Docker launched in 2013. In it’s early years it was based on using LXC, but later developed its own lib instead.

Docker

For most people nowadays when you say “container” Docker is the first association, but we see that the containerization technology is not that new (cgroups dating back to 2008). However, Docker expansion can be contributed to the set of tools it introduced taking advantage of already existing containerization technology.

Because of the rapid evolvement of docker tools, and new versions being incompatible with existing deployments, to counter this Docker Inc. became one of the founder member of Open Container Initiative, a consortium whose mission is to guide the growth of container technology in a healthily competitive direction that fosters standards and collaboration.

We will be talking about docker architecture and docker as a container engine in one of the following articles.

…or what enables the cloud?

• Hypervisor
• Full virtualization
• Paravirtualization
• Hardware assisted virtualization
• Type 1 vs Type 2

The term “virtualization” is an overloaded term that can describe many things, this is due to the way how technology evolved (competing vendors worked independently without benefit of standards).

But what virtualization really is and where it came from?

In the simple words, it’s the technology that makes it possible to run multiple operating systems (concurrently) on the same physical hardware. Virtualization software parcels out CPU, memory, and I/O resources, dynamically allocating their use among several “guest” operating systems and resolving resource conflicts.

The technology itself can be tracked back to 1960s, with the development of hypervisors¹ (supervisor of the supervisor), however virtualization didn’t took of up until 1990s when most enterprises had physical servers and “single vendor” IT stack, meaning legacy apps weren’t able to run on different vendor’s  hardware.

Virtualization was natural solution to 2 problems:

• Companies could partition their servers – reducing equipment and labor costs associated with equipment maintenance and energy consumption
• Run legacy apps on multiple operating system types and versions

One of the reason for renewed interest in virtualization for modern systems was ever-growing size of server farms (datacentres).

VMware was one of the first providers that successfully virtualized x86 architecture. Server farms eventually led to the rise of on-demand, Internet-connected virtual servers, the infrastructure we now know as cloud computing.

Hypervisor

As stated in the beginning of the article, virtualization is an overloaded term, used in many contexts, and there are many types of virtualization and many concepts, phrases and acronyms. We will try to tackle a few of them in this article.

A hypervisor (also known as a virtual machine monitor) is a software layer that sits between virtual machines (VMs) and the underlying hardware on which they run. This software is responsible for sharing the resources (such as memory and processing) among the guest operating systems, which are running independently, and doesn’t have to be of a same kind.

Full virtualization

Full Virtualization was introduced by IBM in the year 1966. Hypervisors, in the beginning, fully emulated underlying hardware having virtual replacements for all the basic resources such as : hard disks, network devices, interrupts, motherboard hardware, etc. In this mode guests are running without modifications, nothing is changed in the binary of the guest operating system itself , but because of the constant translation by the hypervisor between virtual and actual hardware, it incurs a performance penalty.

Hypervisor utilize what is called “trap and emulate” strategy. Essentially, each guest operating system “thinks” it is running on “bare metal” hardware and therefore it does exactly what if would have done on bare-metal processor, meaning it would try to execute certain privileged instructions thinking it has right privilege (but it doesn’t  have since it’s run as user-level process on top of a hypervisor). When this happens, it will result in a trap into the hypervisor and hypervisor will then *emulate* the intended functionality of the particular OS.

Paravirtualization

We said in past paragraph when explaining the full virtualization,  how guest operating system are run *unmodified* on top of the hypervisor.

Paravirtualization approach modifies guest OSes to include optimizations and avoid problematic instructions (ex. guest OS is able to see the real hardware resources). Basically, guest operating systems can detect their virtual state and actively cooperate with hypervisor to access hardware. This improves performance. The downside is that guest operating systems need substantial updates to run this way, and the way that guest operating systems need to be modified in great depends of the specific hypervisor in use. Xen introduced this type of virtualization.

Hardware assisted virtulization

This approach enables full virtualization using the help from the hardware, primarily from the host processors.

In this setup CPU has virtualization capabilities built into it. For instance CPU is able to “pretend” that is  2 or 3 or 4 independent separate computer systems to the OS running on it.

The benefits of the hardware assisted virtualization as oppose to paravirtualization is that the mentioned changes in the guest operating system are not needed, instead hypervisors are using extensions in the CPU itself to run some (or all) instructions directly on the hardware without a software emulation.

Hardware-assisted virtualization was added to x86 processors (Intel VT-x or AMD-V) in 2005 and 2006.

Nowadays most of the CPUs have this abilities.

Type 1 vs. Type 2 hypervisor

Many references draws distinction between two main types of hypervisors, Type 1 and Type 2.

The former one (“Type 1” or often referred as “bare metal”) is the one that runs directly on the hardware of the host, it doesn’t need a supporting  operating system, in fact it acts as a lightweight operating system. The physical machine were type 1 hypervisor is running serves for virtualization purposes only.

Because there is no overhead of the operating system, type 1 hypervisors are considered highly secure, and also very performant and stable. Usually they are used in enterprise environments.

Typical vendors for type 1 hypervisors are: VMWare vSphere with ESX/ESXi, KVM (Kernel-Based Virtual Machine), Microsoft Hyper-V, Oracle VM, Citrix Hypervisor (Xen Server), etc.

In contrast to Type 1, Type 2 hypervisors are user-space applications, running inside of an operating system. They are also called “hosted hypervisors”. They are also managing calls for CPU, memory, disk, network etc. But they do it through the operating system of the host. They are convenient as they are installed on OS as any other applications.

The downside of this type of hypervisors are that if resources not carefully allocated they can overhaul the system, causing the crash. This something that bare-metal hypervisors are doing dynamically, depending on the needs of particular VM. However, hosted hypervisors are really nice for testing and research projects.

Typical vendors for type 2 hypervisors are: Oracle VM Virtual Box, Vmware Workstation, Microsoft Hyper-V, Oracle VM, Parallels Desktop, etc.

Coming up next…

In one of the next articles there will be word about containers and containerization as a major trend and a companion of virtualization.

^{1 –} The term hypervisor is a variant of supervisor, a traditional term for the kernel of an operating system: the hypervisor is the supervisor of the supervisors, with hyper- used as a stronger variant of super-. The term dates to circa 1970; in the earlier CP/CMS (1967) system, the term Control Program was used instead

Before starting of, I must say there is a lot of ways to do this, depending on the situation and particular context you are in, but here I will be talking about how to deploy database which you have ready in an sql project within solution in Visual Studio. So the journey here is from sql project in the Visual Studio to the actual deployed database running in Azure.

Kind of a typical DevOps task you might say.

In my particular case, I had a couple of C# projects (targeting .NET Core) in a solution also, which are making up an API that I also wanted to include in my pipeline, as well as the above mentioned sql project.

So my idea of how it would look like is something like this:

Restore/Build cs projects –> Run tests –> Build and deploy db project

Very simple. Now, having an API in .NET core I wanted to use a Microsoft-hosted Linux agent. Good thing about these agents is that each time you run a pipeline, new VM is spun up and after pipeline finished executing it’s discarded. You don’t have to worry about configuring it etc. you just choose it and run it.

However, as I found out  currently Linux agents cannot build/deploy sql projects. So in order to complete my goal, I had to use two pipelines. First pipeline (Linux Agent) builds the .NET core projects in the solution, runs tests,  etc.

The other pipeline, triggered upon successful finish of the first one is running on a Windows agent, it uses MSBuild as a first step to build an sqlproj.

Then copies a dacpac file, which is a product of the database build and describes the database schema so it can be be updated/deployed. After which Azure SQL Dacpac task is run to actually deploy the db to an Azure SQL instance configured from that file created in the first step, and copied in the second.

Before jumping to Azure portal and configuring pipelines, let’s first make sure we have all the preconditions set.

Adjust sql database project settings

First of all, in order to deploy our db to Azure, we have to make sure to choose right target platform for our db.

To do so, go to the properties of your sql project in Visual Studio, and in “Project Settings” tab choose “Microsoft Azure SQL Database”

I mentioned above that sql project, when successfully built produces dacpac file.

What is DAC or dacpac in the first place?

According to Microsoft documentation:

A data-tier application (DAC) is a logical database management entity that defines all SQL Server objects – such as tables, views, and instance objects – associated with a user’s database. It is a self-contained unit of SQL Server database deployment that enables data-tier developers and DBAs to package SQL Server objects into a portable artifact called a DAC package, or .dacpac file.

A DACPAC is a Windows file with a .dacpac extension. The file supports an open format consisting of multiple XML sections representing details of the DACPAC origin, the objects in the database, and other characteristics. An advanced user can unpack the file using the DacUnpack.exe utility that ships with the product to inspect each section more closely

You can found more information about DAC here.

In short this file holds your database schema definition and when used with appropriate tools you can recreate your database from this file on another SQL instance. This file can be generated in multiple ways, you can extract in within SSMS or within Visual Studio, but since we want to include this file in the CI/CD pipeline, we will generate this file when sql project is built.

To do so, go again to Properties of db project, then Build tab, and enter your database name or whatever you like, to the “Build output file name” filed as below:

This name will be used when creating a file with dacpac extension during build. Make a note of it as we will use that in the pipeline.

When we change this properties, you can see in the sqlproj file that these two properties were added:

 <DSP>Microsoft.Data.Tools.Schema.Sql.SqlAzureV12DatabaseSchemaProvider</DSP>

 <SqlTargetName>MyDatabase</SqlTargetName>

Creating build pipeline in Azure DevOps

Now we can go on and create a pipeline for building sql project and deploying a database. I mentioned having two pipelines above, since I wanted to have it all connected, but I will just include steps for the second one, as for building a .NET core app/api there is already a predefined template in Azure.

As you can see in the screenshot above, the first step is to call MSBuild to build up our sql project.

The important field is “Project” where we put : **/*.sqlproj so it only looks for sql project and builds those.

Everything else can be left as default, such as MSBuild version and architecture.

The next step, copying files, is to take that .dacpac file generated after MSBuild has completed build for sql project, and copy it to Build.ArtifactStagingDirectory which is a predefined azure variable and it is typical used to publish build artifacts, such as our dacpac file here. You can find more about this and other variables here.

The final step is actual database deployment using Azure SQL DacpacTask.

This step requires you already have Azure SQL server provisioned and according database created.

I am using here SQL Server Authentication but you can use ConnectionString, Active Directory etc.

It is obvious that for Deploy type we should use SQL DACPAC file and in the DACPAC file we should enter the path to our dacpac. Now, since we named our dacpac file MyDatabase, by using path **/MyDatabase.dacpac we will instruct Azure, or rather Azure agent to search through all sub folders and find our file.

And that’s it!
You should be now have fully operational pipeline, that builds your database project and deploys it to Azure.

If you are by any means involved in any part of the software development process, chances are you have heard or used (or both) git, and for sure – github.

Github is great, you can create free account in no time, and be ready for pushing changes down your repos. There is just one catch – these repositories you are creating on the github are public.

Which is fine, for most uses cases, especially managing and maintaining open-source projects.

A lot of big companies have their repos publicly available on github. Companies like Google, Amazon and Microsoft, who recently acquired entire service and is recognized now as a biggest contributor on the whole github platform.

Github have an option for private repositories of course, but it is a paid service, and depending on size of the team and included features, prices vary.

7$/month is not something super pricey, especially if you are using git as a irreplaceable every day tool, whether you are a lone developer or working in a team. And you don’t want to mess around with configuring and maintaining a service, you want something that works right “out of the box”.

Private repositories are now available on the free plan on GitHub.

With that said, it is far more interesting (at least for me), if you would install and configure self-hosted git service yourself.

Why? Simply, because you can. 🙂

All you need is a raspberry pi, and a dozen minutes to spend on reading this how to. 😉 So let’s dive in.

Installing Raspbian Lite on RaspberryPi

If you are in a possession of any model of raspberry pi, and it is sitting in the drawer doing nothing (like it was a case with mine), you can put it to good and practical use.

I bought my piece of raspberry pi almost two years ago, it is a RaspberyPi 2 model B+. But any other variant will do, as the things we are going to install and configure will be working fine on any.

I have equipped mine with a 32GB SD card but a 16GB will suffice as well.

For to the image to be flashed to the SD card I’ve chosen Raspbian Lite, it’s smaller in size, saving space on our sd card, and we don’t need GUI for our purposes, since the most of the configuration we will be performing remotely through the CLI.

Raspbian is officialy supported OS by the Raspberry Pi Foundation, so you can easilly download image or .zip and flash it to the SD card with tool like Etcher as recommended on the docs page of the project.

Installing and configuring Gogs

Gogs is a cross-platform self-hosted git service written in Go.

Before we download it we need to setup a few things which are preruquisite for Gogs, as listed in their documentation those are:

1. MySQL database (MSSQL and PostgreSQL are also supported, but I’ve chosen MySQL)
2. Git (bash) version >= 1.7.1 for both server and client sides
3. functioning SSH server

Before performing any installs, be sure your system is up-to-date:

sudo apt-get update && sudo apt-get upgrade 

1) After this, we can install and configure MySQL server:

sudo apt-get install mysql-server 

If you were not prompted to enter a password of a root user type:

sudo mysql_secure_installation

You can answer the question as it suits your needs, as long as you have root access to the MySQL server.
In case you want some other user (other than root), to be used for accessing gogs database, you have to grant the permission to the created database or entire permissions.
After accessing MySQL command with:

sudo mysql -u root -p

and entering root’s password, perform:

GRANT ALL PRIVILEGES ON *.* TO 'raspberryuser'@'localhost' IDENTIFIED BY 'password'; 

Now, while we are at the MySQL prompt, we can create a gogs database with appropriate collation:

CREATE DATABASE IF NOT EXISTS gogs COLLATE utf8_general_ci ; 

2) Now, make sure you have installed git on your pi, by simple running:

sudo apt-get install git

3) As a last prerequisite gogs documentations mentions having functional SSH server. Now, when you run a gogs service it will run it’s own SSH server on default port 22. To avoid collision with the system SSH server, the easiest solution is to change port of the system ssh.
You can do that by editing following file:

nano /etc/ssh/sshd_config

Uncomment the line :

#Port 22

and change the port number to something else (ex. 2244).
You will need to restart the ssh service:

service ssh restart

Additionally, allow gogs to bind as privileged port, perform:

sudo setcap CAP_NET_BIND_SERVICE=+eip /path/to/gogs

Finally, now we can download gogs, simply perform :

wget https://dl.gogs.io/0.11.53/gogs_0.11.53_raspi2_armv6.zip 

in the command line. This should download the binary in your current folder.

Extract the contents of the file, and then:

cd extracted_folder 

Execute:

./gogs web  

It should launch the install page of the gogs service, which you can access externally from the web browser, by entering:

http://ip-of-your-raspberrypi:3000 

In my case that was:

http://192.168.0.14:3000 

And you should be prompted with the installation page, that looks like this:

Fill out the form to match your user and database settings, and the rest of the configuration involving application port, url and log path, as shown below

and hit ‘Install Gogs’. If everything went well, you will probably be redirected to the user login page. However, “locahost” will be used for hostname, so replace it with your pi’s IP address, so you can create account on your new installation of gogs.

Now you can click “Sign up now” to create your new account.

Now you can login with your newly created account, and start creating repos!

Now, we don’t want to launch gogs with ./gogs web everytime we lost ssh connection with our pi, it would be good to run gogs as daemon, so it’s runnig in the background and it’s always on.

Copy an init.d script from a extracted gogs folder:

 sudo cp /home/malina/gogs/scripts/init/debian/gogs /etc/init.d/gogs

and modify WORKING_DIR and USER

# PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC="Gogs"
NAME=gogs
SERVICEVERBOSE=yes
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME
WORKINGDIR=/home/malina/gogs
DAEMON=$WORKINGDIR/$NAME
DAEMON_ARGS="web"
USER=malina

Now we should make it run automatically on boot time with:

sudo chmod ug+x /etc/init.d/gogs

And to make sure it starts after the database server:

sudo update-rc.d gogs defaults 98

We can start gogs as any service with:

sudo service gogs start

If it for some reason service failed to start, perform reboot and then try again.

Additionally, you can configure port forwarding on your home router, so you can access your private github even when you are not at home.

And that’s it, now you have your own private github!

Go push some code! 😉