This weekend I was working on setting up a 3-tier deployment of an application I've been pretty fond of recently: Ghost. It's a publishing and blogging platform that has some really awesome features. It's fun to mess with in its own right, but if you wanted to deploy Ghost in a production setting you'd want to deploy more than just Ghost in order to do it effectively.

I drafted up a docker-compose.yml file and NGINX configuration to explore this idea and dive into what a production-ready Ghost deployment might work like. Here's a link to the finished product for those who want to dive right in:

Spencer Smolen / 3-Tier Ghost Site · GitLab

GitLab.com

GitLab

3-Tiered Application Architecture Explained

This is a working example of a 3-tier Ghost deployment. There are a ton of great articles and resources on 3-tier architecture so I won’t go into too much detail here as to the ins and outs of the framework. Below is a graph explaining how this application, and others that follow the framework, are organized.

The general idea is to separate the application into 3 parts: a frontend, a backend, and a middle tier that can speak to both the front end and the backend and prevents them from having to talk directly to each other.

Generally, there are a few reasons you might want to do this, but all of the advantages draw from one main fact: by doing this you achieve greater isolation of functional components that comprise your application. Among the benefits are:

• Increased security
• Easier maintenance
• Protection against single points of failure

In a proper 3-tier application, only one of the three tiers is accessible to the end user. This is the part of the application you interface with as you use it and it’s typically accessible via the Internet. This is the presentation layer.

Behind that is the real heart of the application, called the application layer. This is the workhorse of the application and it is usually an API of sorts whose job is just to compute and hand off the resulting work to the presentation layer.

Finally, there’s the data layer. This is the store of data that any application typically needs to run over time. This is where long-term results are stored, and it is referenced by the application layer while it does its heavy lifting.

An Implementation using Ghost

This repository comes in the form of a Docker Compose project. It is composed of 3 Docker containers, each corresponding to one of the tiers in the framework. Ghost, for those unfamiliar, is a blogging platform of sorts for everyone from at-home bloggers to massive publications and enterprise customers. The 3 containers contained in this project are:

NGINX

A reverse proxy whose job is just to interface with the outside world and be a middleman between Ghost and whoever is trying to access its content. Think of it like a currier who delivers pages on demand and goes back and asks the application for pages as people request them.

Ghost

This is the heart of the application. This is the part that generates web content, allows people to make blog posts, organizes all the content in a meaningful way so it can be searched, prevents things from crashing, etc.

MySQL

This is the database where Ghost stores all of its content that’s needed for long-term use. This is a database much like any other, nothing too fancy.

Using the Repository to Deploy Ghost

Below is the docker-compose.yml file that does the heavy lifting of setting up these 3 servers for us:

---
version: '3'
services:
  db:
    image: mysql:8
    expose: [3306, 33060]
    environment:
      MYSQL_DATABASE: ghost
      MYSQL_USER: ghost
      MYSQL_PASSWORD: "$SQL_USER_PASSWORD"
      MYSQL_ROOT_PASSWORD: "$SQL_ROOT_PASSWORD"
    networks: [backend_net]
    volumes: [mysql_data:/var/lib/mysql]
    healthcheck:
      test: ['CMD-SHELL', 'mysqladmin ping', '-h 127.0.0.1', ' --password="$$(cat /run/secrets/db_password)"', '--silent']
      interval: 3s
      retries: 5
      start_period: 30s
    secrets: [ db_password ]
  app:
    image: ghost:5
    depends_on:
      db:
        condition: service_healthy
    environment:
      database__client: mysql
      database__connection__host: db
      database__connection__user: ghost
      database__connection__database: ghost
      database__connection__password: "$SQL_USER_PASSWORD"
      url: http://$NGINX_HOST:$NGINX_PORT
    networks: [backend_net, frontend_net]
  web:
    image: nginx:stable
    ports: [$NGINX_PORT:443]
    volumes: [./nginx/:/etc/nginx/templates]
    networks: [frontend_net]
    depends_on: [app]
    environment:
      NGINX_HOST: $NGINX_HOST
      NGINX_PORT: $NGINX_PORT
    secrets: [ssl_cert,ssl_key,dhparam]
networks:
  frontend_net:
    driver: bridge
  backend_net:
    driver: bridge
volumes:
  mysql_data:
secrets:
  ssl_cert:
    file: ./secrets/nginx.crt
  ssl_key:
    file: ./secrets/nginx.key
  db_password:
    file: ./secrets/db_password

And you’ll have created both a private key and a certificate (you’ll need both files) as well as installed them on your machine so you don’t get that annoying self-signed certificate warning.

Once you’ve done that create a folder called ./secrets wherever you downloaded this repository and place them in there after naming the cert and the key nginx.crt and nginx.key, respectively.

After that, you’ll be ready to use the Makefile to prep the installation. Just type this in the folder with the Makefile  and type:

make compose-file
make passwords

After that, you can go ahead and run:

docker-compose up -d

in the folder where the docker-compose.yml file is generated and it should start booting up the images! At this point, you should be able to navigate in a browser to the domain name you made the certificate out to and see the fresh installation of Ghost. In my example, I was messing around with a domain ziinc.dev.

Although this project is comprised of containers it's designed in a way that allows you to have persistent data beyond the life of any one of the containers. This is achieved by using a Docker volume to hold the database's /var/lib/mysql directory. This is where the data is saved in a MySQL application. The lines below achieve this:

volumes:
  mysql_data:

By structuring the application this way, we can swap any container out with a new one should it go down or get destroyed so long as the Docker volume is managed correctly.

Finally, if you're feeling adventurous, you can easily take this Docker Compose file and use kompose to convert it into a set of Kubernetes entities for a more advanced deployment on Kubernetes. Just download it from here and run the following:

kompose convert -f docker-compose.yaml
kubectl apply -f .

Enjoy!

Recently I've been building a lot of packages for eLX and I've realized packages can be relatively simple to put together and the ability to archive and distribute your code is extremely powerful. Contrary to what you might think, you don't have to be a coder to be able to reap the benefits of building & packaging software. What was fascinating to me was that like other tools hijacked from the Linux development community, e.g. GNU make, there's massive value in these tools for system administrators as well.

First off –  what is a package, anyways? We're talking about .rpm and .deb files. Those things we drink up from tools like dnf and apt in order to download software. These files are basically archives similar to .rar or .zip files: they're just compressed collections of other files, i.e. they're really nothing fancy.

In the world of distributions downstream from Fedora, we use .rpm files to package software, while projects downstream from Debian use .deb files. All .rpm files start as a .spec file, which defines how to turn some raw source code into a built binary application and package it into an .rpm. Just to get an idea of what I mean, here's an actual spec to build lnav, a command line log navigator written in C.

Name:          lnav
Version:       0.11.1
Release:       1%{?dist}
Summary:       Curses-based tool for viewing and analyzing log files
License:       BSD
 
URL:           http://lnav.org
Source0:       https://github.com/tstack/lnav/releases/download/v%{version}/%{name}-%{version}.tar.bz2
 
BuildRequires: bzip2-devel
BuildRequires: gcc-c++
BuildRequires: libarchive-devel
BuildRequires: libcurl-devel
BuildRequires: make
BuildRequires: ncurses-devel
BuildRequires: openssh
BuildRequires: openssl-devel
BuildRequires: pcre2-devel
BuildRequires: readline-devel
BuildRequires: sqlite-devel
BuildRequires: zlib-devel
 
%description
%{name} is an enhanced log file viewer.
 
 
%prep
%setup -q
 
%build
%configure --disable-static --disable-silent-rules
%make_build
%install
%make_install
 
%files
%doc AUTHORS NEWS.md README.md
%license LICENSE
%{_bindir}/%{name}
%{_mandir}/man1/%{name}.1*
 

Believe it or not, this file has everything needed to be fed into a program called rpmbuild and produce a complete .rpm. Now getting into the nitty gritty of the spec file is beyond the scope of this article but let's suppose you already have one. Wouldn't it be nice to automate the compiling of your code and building of your packages with this spec file you? The technology here of compiling and packaging software is nowhere near new, but how can we marry the old world of packaging .rpm files with the new world of CI/CD?

The World of CI/CD Pipelines

In the past few years, CI/CD has become huge. Continuous Integration & Continuous Delivery refers to the ability of people who work with code to automate many of the tasks that used to be done by hand in such a way that has taken on its own style. CI/CD revolves around the idea of setting in motion a cascading sequence of events once a coder finishes and commits some piece of code to the code base.

One of the younger CI/CD pipeline tools is GitHub Actions. Recently I built a pipeline that would automatically package up the code I had just published into both an .rpm and .deb and publish it as a release asset in GitHub when tagged with a new version number. This is huge! Once the coding is committed, the entire process of building and publishing it in a consumable package form is automated.

The best news is that after front-loading the work by building the pipeline it requires almost zero maintenance. So if you're interested in sharing code, scripts, configurations, or keeping snapshots of them to archive yourself, e.g. your dotfiles, you're about the have the tools.

The Build Pipeline

The pipeline walks through 4 what are called "jobs" in most CI/CD frameworks. Each job completes a task that you define in an individual Docker container that is destroyed when the job has been completed. Here is a brief description of each job:

1. First, it's going to download a copy of the most recent commit as raw source code. The pipeline only runs when there is a new version tagged on one of the commits, e.g. v1.4. This typically occurs when new features have been added or some sizable amount of work has been done and it's time for a new release. We're gonna download those plain text files and archive them in a file named something like softwarename-1.4.tar.gz.
2. Next, we're going to take our .spec file which,  remember, is like the recipe for how to build the software from source. We have the source code we downloaded in the last step, now we've got this .spec file that has our instructions on how to build it. We're gonna run the recipe on the source code we downloaded in softwarename-1.4.tar.gz using a tool called rpmbuild and it's going to make our package.
3. Since half the Linux world speaks .rpm and the other half speaks .deb , we're going to use an awesome tool called alien to convert our .rpm to a .deb with minimal effort.
4. Finally, the pipeline is going to take both of these packages which it has cached in the process of stepping through the pipeline and (1) create a new release in GitHub and (2) publish both our Linux packages of our freshly baked software onto GitHub so people can freely download it.

Alright!

Walking Through the Jobs

GitHub Actions, like many other modern automation tools, is a YAML-based tool. However, rather than having a rigid API like some others, the entire system is itself somewhat "package" based. People on GitHub post modules that you can use referred to as "Actions". Using one of these published Actions usually just amounts to a few lines of YAML to carry out some task. To use a published "Action" you use the keyword uses in your workflow. For example:

 jobs:
   job1:
     steps:
      - name: Checkout repository
        uses: actions/checkout@v2

You'll see them throughout the pipeline.

Job #1:

  build_tarball:
    name: Build source archive
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      - name: Replace version in RPM spec
        run: sed -Ei 's/(^Version:[[:space:]]*).*/\1${{github.ref_name}}/' ${{ vars.PKG_NAME }}.spec

      - name: Create source archive
        run: tar -cvf ${{ vars.PKG_NAME }}-${{ github.ref_name }}.tar.gz *

      - name: Upload source archive as artifact
        uses: actions/upload-artifact@v3
        with:
          name: ${{ vars.PKG_NAME }}-${{ github.ref_name }}.tar.gz
          path: ${{ vars.PKG_NAME }}-${{ github.ref_name }}.tar.gz

First, we use the actions/checkout@v2 "action" to pull down our code into the current execution environment, which for whatever it's worth, is a container executing the steps of your pipeline.

Next, we go ahead and perform a quick search and replace using sed to update the version number in our spec file that triggered the pipeline in the first place. Then, we're going to archive the code by creating our tar file.

Finally, we're going to go ahead and upload that code as what's called an artifact. This isn't the final upload to our GitHub Release page. Rather, it's a way of putting it aside while the rest of the pipeline runs. Because each job runs in a new container, we need to make use of this artifact cache often in order to pass files from one job to the next.

Job #2:

Here's where the action is:

  build_rpm:
    name: Build .rpm package
    needs: build_tarball
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      - name: Replace version in RPM spec so correct source is downloaded when building RPM
        run: sed -Ei 's/(^Version:[[:space:]]*).*/\1${{github.ref_name}}/' ${{ vars.PKG_NAME }}.spec

      - name: Run rpmbuild on RPM spec to produce package
        id: rpm
        uses: naveenrajm7/rpmbuild@master
        with:
          spec_file: ${{ vars.PKG_NAME }}.spec

      - name: Upload .rpm package as artifact
        uses: actions/upload-artifact@v3
        with:
          name: ${{ vars.PKG_NAME }}-${{ github.ref_name }}-1.${{ env.DIST }}.${{ env.ARCH }}.rpm
          path: rpmbuild/RPMS/${{ env.ARCH }}/*.rpm

Because this new job starts in a fresh container we're going to start our second job off just like we did the first: by checking out our project with the actions/checkout@v2 action and updating the version in the .spec file.

Then we run the naveenrajm7/rpmbuild@master action which basically runs two rpmbuild commands that produce the rpm behind the scenes. Finally, we upload the .rpm as an artifact just like we did with the raw source code in the first job so we can have access so it in future jobs.

Job #3:

  build_deb:
    name: Build .deb package
    needs: build_rpm
    runs-on: ubuntu-latest
    steps:
      - name: Download .rpm artifact
        uses: actions/download-artifact@v3
        id: download
        with:
          name: ${{ vars.PKG_NAME }}-${{ github.ref_name }}-1.${{ env.DIST }}.${{ env.ARCH }}.rpm

      - name: Convert .rpm to .deb
        run: |
          sudo apt install -y alien
          sudo alien -k --verbose --to-deb *.rpm

      - name: Upload .deb package as artifact
        uses: actions/upload-artifact@v3
        with:
          name: ${{ vars.PKG_NAME }}-${{ github.ref_name }}-1.${{ env.DIST }}.${{ env.ARCH }}.deb
          path: ${{ vars.PKG_NAME }}*.deb

You're probably starting to see the pattern here. This time we're going to, instead of downloading our code, download one of the artifacts we uploaded in the earlier steps. We're going to download the .rpm from the last step and convert it to a .deb for Debian-based systems. To do that we'll run the alien command  and upload the resultant .deb as an artifact with the others.

Job #4:

Finally, in the 4th job, we're going to create our release! This is the GitHub event that these files are going to be uploaded with:

  release:
    name: Create release with all assets
    needs: [build_tarball, build_rpm, build_deb]
    runs-on: ubuntu-latest
    steps:
      - name: Download cached rpm, deb, and tar.gz artifacts
        uses: actions/download-artifact@v3

      - name: Release
        uses: softprops/action-gh-release@v1
        with:
          files: |
            ${{ vars.PKG_NAME }}-${{ github.ref_name }}.tar.gz/*.tar.gz
            ${{ vars.PKG_NAME }}-${{ github.ref_name }}-1.${{ env.DIST }}.${{ env.ARCH }}.rpm/**/*.rpm
            ${{ vars.PKG_NAME }}-${{ github.ref_name }}-1.${{ env.DIST }}.${{ env.ARCH }}.deb/**/*.deb

In this job, we're going to fetch all those "artifacts" we kept throwing up into the cache at the end of each job. Without any other arguments, the following line will download all items saved in the cache:

      - name: Download cached rpm, deb, and tar.gz artifacts
        uses: actions/download-artifact@v3

After that we've actually pulled down is:

1. an archive of the raw code in the format softwarename-1.4.tar.gz
2. an .rpm of the build, e.g. softwarename-1.4.rpm
3. a .deb of the build, e.g. softwarename-1.4.deb

Last we have quite a useful action here called softprops/action-gh-release@v1 that allows us to create a release and attach all our assets to it in the same step.  In this step, we upload our artifacts, and voilà! Our code has been shared. It will now be visible on the "Releases" page of our repo:

Keep in mind this process is totally automated. Once the code has been committed the pipeline starts running. You can keep tabs on it by navigating to the "Actions" tab at the top of your repository. There's a pretty comprehensive log for you to go through. It will look something like this:

Alright here's the finished copy of the pipeline below!

name: Build Linux Packages
on:
  push:
    tags:
      - "*.*.*"
env:
  DIST: el7
  ARCH: noarch

jobs:
  build_tarball:
    name: Build source archive
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      - name: Replace version in RPM spec so correct source is downloaded when building RPM
        run: sed -Ei 's/(^Version:[[:space:]]*).*/\1${{github.ref_name}}/' ${{ vars.PKG_NAME }}.spec

      - name: Create source archive
        run: tar -cvf ${{ vars.PKG_NAME }}-${{ github.ref_name }}.tar.gz *

      - name: Upload source archive as artifact
        uses: actions/upload-artifact@v3
        with:
          name: ${{ vars.PKG_NAME }}-${{ github.ref_name }}.tar.gz
          path: ${{ vars.PKG_NAME }}-${{ github.ref_name }}.tar.gz

  build_rpm:
    name: Build .rpm package
    needs: build_tarball
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      - name: Replace version in RPM spec so correct source is downloaded when building RPM
        run: sed -Ei 's/(^Version:[[:space:]]*).*/\1${{github.ref_name}}/' ${{ vars.PKG_NAME }}.spec

      - name: Run rpmbuild on RPM spec to produce package
        id: rpm
        uses: naveenrajm7/rpmbuild@master
        with:
          spec_file: ${{ vars.PKG_NAME }}.spec

      - name: Upload .rpm package as artifact
        uses: actions/upload-artifact@v3
        with:
          name: ${{ vars.PKG_NAME }}-${{ github.ref_name }}-1.${{ env.DIST }}.${{ env.ARCH }}.rpm
          path: rpmbuild/RPMS/${{ env.ARCH }}/*.rpm

  build_deb:
    name: Build .deb package
    needs: build_rpm
    runs-on: ubuntu-latest
    steps:
      - name: Download .rpm artifact
        uses: actions/download-artifact@v3
        id: download
        with:
          name: ${{ vars.PKG_NAME }}-${{ github.ref_name }}-1.${{ env.DIST }}.${{ env.ARCH }}.rpm

      - name: Convert .rpm to .deb
        run: |
          sudo apt install -y alien
          sudo alien -k --verbose --to-deb *.rpm

      - name: Upload .deb package as artifact
        uses: actions/upload-artifact@v3
        with:
          name: ${{ vars.PKG_NAME }}-${{ github.ref_name }}-1.${{ env.DIST }}.${{ env.ARCH }}.deb
          path: ${{ vars.PKG_NAME }}*.deb

  release:
    name: Create release with all assets
    needs: [build_tarball, build_rpm, build_deb]
    runs-on: ubuntu-latest
    steps:
      - name: Download cached rpm, deb, and tar.gz artifacts
        uses: actions/download-artifact@v3

      - name: Release
        uses: softprops/action-gh-release@v1
        with:
          files: |
            ${{ vars.PKG_NAME }}-${{ github.ref_name }}.tar.gz/*.tar.gz
            ${{ vars.PKG_NAME }}-${{ github.ref_name }}-1.${{ env.DIST }}.${{ env.ARCH }}.rpm/**/*.rpm
            ${{ vars.PKG_NAME }}-${{ github.ref_name }}-1.${{ env.DIST }}.${{ env.ARCH }}.deb/**/*.deb

To add it to your project just copy and paste the contents of the file goes in the .github/workflows folder in your repository. You can name the file anything you want it will be executed regardless. For reference you can see how it's used in my project provii here:

GitHub - kriipke/provii: provisioning tool to install pre-compiled binaries of your favorite command-line tools

provisioning tool to install pre-compiled binaries of your favorite command-line tools - GitHub - kriipke/provii: provisioning tool to install pre-compiled binaries of your favorite command-line tools

GitHubkriipke

Enjoy!

For those unfamiliar, $PS4 is a variable that can be set so that when you run your scripts using set -x, a shell option that allows you to print each line of your script as it's run, you can add some custom text to the beginning of each line. The set -x option works like this:

If I have the following shell script named test.sh:

#!/bin/bash 

set -x 

TEST_VERBIAGE="You should see this printed to your terminal"
printf %s "$TEST_VERBIAGE"

When I run it in the terminal it will look like this:

❯ ./test.sh  
+ TEST_VERBIAGE='You should see this printed to your terminal'
+ printf %s 'You should see this printed to your terminal'
You should see this printed to your terminal%

Back to $PS4. The power of $PS4 is not to be understated, but there isn't much in the bash man page that would lead you to believe there's much going on with it. Below is what you'll find in the man page about it:

The value of this parameter is expanded as with PS1 and the value is printed before each command bash displays during an execution trace.  The first character of the expanded value of PS4 is replicated multiple times, as necessary, to indicate multiple levels of indirection.  The default is ``+ ''.

That isn't very helpful, but it does explain that the default is +. If you look above at the example when I ran test.sh in the terminal you'll notice a + at the beginning of each line that was an executed command (the other line is the output that you would normally see in your terminal, so it's not included). This is where the output of $PS4 is placed.

Crafting a PS4 Variable

You can include anything you want in your $PS4 but it generally makes sense to include things that explain the context of what's being executed so that when a line fails, you can have some extra information to help you figure out why. You could include the value of certain variables in your script, the time, or anything you want.

A simple example would be to use $PS4 to print out the time in nanoseconds before each line so you can tell how long each command takes to execute when looking back over the trace. This would be done by putting the following line at the top of your script:

PS4='$(date +%N)'

One thing that should be noted when crafting your $PS4 is that you need to put the contents in single quotes: PS4='$(date +%N)'. That's because the value of $PS4 is executed at the beginning of each line, over and over, ad nauseam. If you put the contents in double quotes, like PS4="$(date +%N)", then anything that's going to be executed will only be executed once at the beginning of your script when you define the variable and each line will be prefixed by the same content, which isn't helpful.

For example, if we used the PS4="$(date)" example above we would get this:

❯ ./test.sh  
++ date +%N
+ PS4='152563673 '
152563673 TEST_VERBIAGE='You should see this printed to your terminal'
152563673 printf %s 'You should see this printed to your terminal'
You should see this printed to your terminal%  

When what we really wanted was this, which shows the difference in time in nanoseconds:

❯ ./test.sh  
+ PS4='$(date +%N) '
885045572 TEST_VERBIAGE='You should see this printed to your terminal'
890522778 printf %s 'You should see this printed to your terminal'
You should see this printed to your terminal%  

So to get that output our script is going to look like this:

#!/bin/bash 

set -x 

PS4='$(date +%N) '

TEST_VERBIAGE="You should see this printed to your terminal"
printf %s "$TEST_VERBIAGE"

A Solid Default $PS4

Now you may be using $PS4 to examine different things in your shell script but in general, I'm going to walk you through the default $PS4 that I use for my scripts. Without further ado here it is:

PS4='$(tput setaf 4)$(printf "%-12s\\t%.3fs\\t@line\\t%-10s" $(date +%T) $(echo $(date "+%s.%3N")-'$(date "+%s.%3N")' | bc ) $LINENO)$(tput sgr 0)'

The output looks like this:

The first column is the time in the format hours:minutes:seconds. The second column is the number of seconds since the script started running, e.g. how far into the script timewise. The third and fourth columns indicate the line in the script you're on when executing the command on that line in the trace.

This last one is massively useful. When your script hits a snag and you get an error, usually the first thing you want to know in a bash script was what line in my script executed the last command.

You're more than welcome to copy and paste this and use it as is. With what you know about $PS4 and how it's used you should be good to go. It may look funny when you copy it into your editor because there is actually a part of this $PS4 definition that is not in single quotes as I advised at the beginning of this article. That's actually on purpose. What it does is it captures the time in nanoseconds at the time the variable is defined to use as a reference when calculating the seconds since the script started! A neat little trick.

Note that the tput commands at the beginning and the end are what create the colored aspect of the script. This is super useful when you've got thousands of lines of this mixed without output from the commands as well in order to orient yourself while navigating the trace. However, to use tput you'll need ncurses installed.

This is a non-issue in 99% of situations but if you're in a Docker container using a barebones image you may not have it, so just take those parts out or install ncurses if you hit a snag.

Enjoy!

1. Have a deployment server set up to automate the installation of a new system
2. Create a golden image that is either (a) to physical media, such as a hard drive, or (b) stored as a virtual machine to be cloned or imported by your hypervisor

In this post, we're going to cover the first option.

There are a few ways to do this. One is using an out-of-the-box solution like Red Hat Satellite. There are some advantages to doing this. The first advantage is that solutions like these market themselves as turnkey solutions, meaning they have everything you need ready to go.

However, if your goal is to learn, you will definitely be robbing yourself of a lot if you choose to go this route. Additionally, you will be locked into using the software the way it was built. If there's something you want to do using a turnkey solution, but can't, then too bad you're kinda stuck. That being said, if you build a solution on your own you'll be able to tweak the system as your needs change.

Design Overview

When designing any system, the first step is determining your requirements. For this project, the design constraints are thus:

• To be able to pick from multiple distributions of Linux.
• To be able to pick from multiple releases of any given Linux distribution.
• To be able to deploy any of the above using either (a) a fully automated install or (b) a manual walk-through of the installation.
• When using an automated install, be able to pick from any number of pre-configured installation sets, e.g. pick from either a "base" install or one pre-configured as an IDM server, etc.
• To be able to deploy any of the above to either a bare-metal server or a virtual machine.

For the sake of simplicity and ease of use, I'm going to show you how to set all of this up with TrueNAS. TrueNAS Scale is a NAS-oriented variant of Debian that implements all the components we're going to need to do this and gives you an optional GUI to use to manage each component. We're going to be writing all the configuration files from scratch so there will be no sacrifice of functionality or future expansion.

It's worth noting that you can totally use the architecture outlined in this article without using TrueNAS at all, you'll just need to deploy each of the components by hand. With that in mind, let's see how we're going to architect this solution.

Components of the Deployment Server

I'm going to quickly outline the components we're going to need to make this deployment server possible. After that, we're going to walk through the boot process and explain how each component is used in the installation process.

1. TFTP server – TFTP (Trivial File Transfer Protocol) is a simple file transfer protocol that we're going to use to make PXE booting possible. PXE (pre-boot execution) booting is basically a way to bootstrap the boot process using parameters fed over the network at boot. When the machine boots up it will, if configured to, attempt a PXE boot. If it does, and things are configured correctly it will load up the files it needs to boot up into memory and run basically a small hard drive in memory and run on that. You can do a lot of things with PXE booting but in this scenario, we'll be using it to load up the installation disk only.
2. DHCP server - This is, unlike the rest of these protocols, not a file transfer protocol. Instead, this is something you should already have configured on your network that allows you to connect newly booted machines to your network. It is probably installed and runs on your router, but you may have one configured yourself. Either way, we're not going to be setting up a DHCP server in this tutorial, just modifying the configuration of your existing one for the sake of simplicity.
3. FTP server – FTP (File Transfer Protocol) is a protocol that's been around for a long time and is a nice choice for the miscellaneous files we're going to need to use during the automated part of the installation. This is where we'll host our kickstart files (I'll go back to this in a bit) as well as any other files we may need, e.g. configuration files, SSH keys we want to install on the servers, anything really.
4. NFS server – NFS (Network File Server) is the protocol we're going to use to host all the OS packages we need to install during the boot process. This is more or less a clone of the contents of the installation DVD, however, we can tailor it to our liking by adding some custom OS packages.

Alright! Those are the components we need. I know that sounds like a lot but to be honest, TrueNAS is the perfect tool for this job if you're looking to get this going in a single afternoon. I'm kind of a hardcore do-it-yourselfer and I assure you that you really can sacrifice nothing by using TrueNAS.

Now let's get into the way all these components fit in. There are two main installation scenarios we need to think about. I'll go over them briefly.

Automated Install Overview

Below are the steps the installation process will go through during an automated install, highlighting the way each component described in the last section fits into the picture:

1. Your machine initializes a PXE boot, which prompts it to receive its network parameters via DHCP. Upon doing so, the machine will be fed two things: (1) the IP address of the TFTP server hosting the PXE boot files and the (2) the path on the TFTP server of the file to boot from. If (a) your machine is configured to boot in PXE mode (b) your DHCP server is configured correctly this should lead you to step 2. If not, check your settings as described below.
2. Upon receiving the IP address of the TFTP server and the path of the boot file, the machine will then be presented with some options outlined in our PXE configuration file, hosted at /pxelinux.cfg/default on our TFTP server. At this point, the available deployment options described in /pxelinux.cfg/default will be read across your screen for you to select. Below is a basic example:

3. You will then manually select which option you want and hit enter. Depending on which option you select the correct kickstart file will be chosen and pulled down from our FTP server. This is made possible by a line like append ... inst.ks=ftp://10.0.0.2/ks/el9/ks.cfg in our  /pxelinux.cfg/default. Note that we can configure this to automatically select an option for us so we don't have to have any user interaction. However, keep in mind the implications of this. It's nice to have this little bit of human interaction that way you don't end up wiping a machine that gets connected to your network just but simply plugging in the ethernet jack and turning it on!

4. The kickstart file, e.g. kickstart.cfg, will have a line in it that points the installer to the location of the NFS server hosting all the OS repositories with all of the packages we need for the installation as defined in the kickstart file. This is made possible by a line like nfs --server=10.2.0.6 --dir=/mnt/store/deployment/mirror/el9/rhel-9.1 in our kickstart.cfg file.

Once that is complete you should have a fully automated install going on as pictured below and you can check back in in a few minutes for your fresh OS ready to go!

The non-automated version of this is the exact same except that in step 3, you will select an option which will, instead of initiating an automated install and showing the image above, present you with the familiar screen shown below allowing you to click through the installer as you normally would.

Configuring the Server

There are basically 3 steps to getting this up and running

1. Get your services up and running
2. Get your configuration files in place
3. Get your files that need to be hosted in-place

Alright, here we go!

Configuring the Services

I'm going to assume you already have a fresh install of TrueNAS configured as there are plenty of articles on how to do this. To be honest it's pretty simple just flash the ISO to a flash drive and install it disk you can boot from. There are no options during the installation it's pretty to the point.

Once you've got a TrueNAS installation and storage pool configured you should see something like this when you go to Datasets:

Go ahead and create a new Dataset within your storage pool by selecting "Add Dataset" off to the right and name it "Deployment".

Within that Dataset create 3 more:

1. ftp
2. mirror
3. pxelinux

Until you've got something that looks like this:

Next, go over to "Shares" on the left-hand navigation and click "Add" off to the right of "UNIX (NFS) Shares".

Go ahead and find the mirror dataset we created under the deployment dataset and click "Save".

When you do it will ask you if you want to enable the service. Press Enable Service.

Next, go down to System Settings > Services on the left-hand navigation.

You should see something like the following:

Click the little pencil icon to the right of "TFTP" to configure the TFTP service.

Go ahead and find the pxelinux dataset we created under the deployment and select the correct IP under Host and click Save.

Select the on/off switch under the Running column and also check the checkbox under Start Automatically on the row for TFTP.

Finally, go up to FTP and click the pencil to edit. At the bottom expand the Advanced Options and (1) check Allow Anonymous Login and (2) select the ftp dataset we created under deployment.

Afterward, navigate back to  System Settings > Services and check the on/off switch under the Running column and the checkbox on Start Automatically so you have something that looks like this:

Perfect! You now have all the services you need running you're just not hosting any of the files yet. We'll cover this in the next section.

If you chose to do all this on your own by hand then now you can tune back in for configuring the server.

Deploying Your PXELinux Files

The PXE Configuration File: /pxelinux.cfg/default

This is where you're going to give the general deployment options that you are going to select when booting up a new install. You're going to want to outline all possible deployment scenarios here because this is the only human interaction involved in the whole process if you choose an automated install. If the configuration you want isn't listed here you're going to have to select a guided installation and walk through it by hand. This isn't the end of the world but it's a much nicer experience with a fully automated install I've found. After all, you've set up a deployment server, you should be enjoying the fruits of your labor, right? :)

timeout 600
display boot.msg
default 4
prompt  1

label 1
  menu label ^Install RHEL 9.1 via Kickstart: Bare Metal
  kernel images/RHEL-9.1/vmlinuz
  append initrd=images/RHEL-9.1/initrd.img ip=dhcp inst.ks=ftp://10.2.0.8/ks/el9/ks-baremetal.cfg

label 2
  menu label ^Install RHEL 9.1 via Kickstart: Virtual Machine
  kernel images/RHEL-9.1/vmlinuz
  append initrd=images/RHEL-9.1/initrd.img ip=dhcp inst.ks=ftp://10.2.0.8/ks/el9/ks-vm.cfg

label 3
  menu label ^Install RHEL 9.0 via local mirror
  kernel images/RHEL-9.0/vmlinuz
  append initrd=images/RHEL-9.0/initrd.img ip=dhcp inst.repo=nfs:10.2.0.6:/mnt/store/deployment/mirror/el9/rhel-9.0/

label 4
  menu label Boot from ^local drive
  localboot 0x80


menu end

A few things to note here. All the paths listed here include boot.msg and images/.. are relative to the PXE boot file given in your DHCP parameters, which was probably pxelinux.0 (we'll get around to all this in the configuration section below). Getting all these files in the right place can be tricky, so just as a time saver, I've created a GitHub repository with a Makefile to automate this for you.

GitHub - kriipke/linux-deployment-server

Contribute to kriipke/linux-deployment-server development by creating an account on GitHub.

GitHubkriipke

If you open TrueNAS and open the "System Settings" menu on the left-hand navigation you will see an option in the menu for "Shell". Click on that and navigate to /tmp and clone the repository with git clone https://github.com/kriipke/linux-deployment-server.

I would highly recommend reading over the README.adoc to get a good understanding of what's going on here. I would also recommend opening up the Makefile and reading over the shell commands executed at each step to make sure you're not robbed of learning anything.

If you decided to read it over it should be pretty apparent how to use it as it's fairly documented. If you don't want to bother yourself with that, after you clone the repository just copy the contents of the folder /tmp/linux-deployment-server/pxelinux into the root of your TFTP server. This should be able to to be done with a command like:

cp -r /tmp/linux-deployment-server/pxelinux/* \
	/mnt/$YOUR_POOL_NAME/deployment/pxelinux/

On my system, since my pool's name is store, I'd have something like the following

/mnt/store/deployment/pxelinux
/mnt/store/deployment/pxelinux/ldlinux.c32
/mnt/store/deployment/pxelinux/images
/mnt/store/deployment/pxelinux/images/RHEL-9.1
/mnt/store/deployment/pxelinux/images/RHEL-9.1/.gitignore
/mnt/store/deployment/pxelinux/images/RHEL-9.0
/mnt/store/deployment/pxelinux/images/RHEL-9.0/.gitignore
/mnt/store/deployment/pxelinux/pxelinux.cfg
/mnt/store/deployment/pxelinux/pxelinux.cfg/default
/mnt/store/deployment/pxelinux/libutil.c32
/mnt/store/deployment/pxelinux/pxelinux.0
/mnt/store/deployment/pxelinux/menu.c32
/mnt/store/deployment/pxelinux/libcom32.c32

At this point, the only files you're missing are the vmlinuz and the initrd.img for each release. You can find these files on the installation DVD of each release at /images/pxeboot/vmlinuz and /images/pxeboot/initrd.img.

Copying these files over from each release is quite laborious and will get annoying really quickly so, if you choose, you can use the Makefile to make this a lot faster by:

1. downloading the DVD installation disk for each release and placing it the /tmp/linux-deployment-server/iso directory we created when you cloned the repository from GitHub.
2. Navigate to /tmp/linux-deployment-server and edit the first few lines of the Makefile with the right variables for the ISO & release you need the boot files from.
3. Typing make bootimgs in the same folder as the Makefile and it will place them in the correct directory. Make sure you change the TFTP_ROOT := ./pxelinux  line to the correct path in your TrueNAS store (it should be something like /mnt/$YOUR_STORE_NAME/deployment/pxelinux/images).

Once you do this you should have the full file listing, checking it against this one below:

/mnt/store/deployment/pxelinux
/mnt/store/deployment/pxelinux/ldlinux.c32
/mnt/store/deployment/pxelinux/images
/mnt/store/deployment/pxelinux/images/RHEL-9.1
/mnt/store/deployment/pxelinux/images/RHEL-9.1/vmlinuz
/mnt/store/deployment/pxelinux/images/RHEL-9.1/initrd.img
/mnt/store/deployment/pxelinux/images/RHEL-9.1/.gitignore
/mnt/store/deployment/pxelinux/images/RHEL-9.0
/mnt/store/deployment/pxelinux/images/RHEL-9.0/.gitignore
/mnt/store/deployment/pxelinux/images/RHEL-9.0/initrd.img
/mnt/store/deployment/pxelinux/images/RHEL-9.0/vmlinuz
/mnt/store/deployment/pxelinux/pxelinux.cfg
/mnt/store/deployment/pxelinux/pxelinux.cfg/default
/mnt/store/deployment/pxelinux/libutil.c32
/mnt/store/deployment/pxelinux/pxelinux.0
/mnt/store/deployment/pxelinux/menu.c32
/mnt/store/deployment/pxelinux/libcom32.c32

Deploying Your Kickstart Files

The kickstart files are extremely powerful. You can find a nice reference for the most recent version of Enterprise Linux (of all variations including Rocky, CentOS, etc.) here. For Fedora, the most up-to-date reference is here. To deploy these files copy all the files from /tmp/linux-deployment-server/ftp/ to /mnt/$YOUR_STORE_NAME/ftp/.

lang en_US
keyboard --xlayouts='us'
timezone America/New_York --utc
rootpw --plaintext changeme
user --groups=wheel --name=ansible --password=changeme --uid=1000 --gecos="For making unattended changes with Ansible." --gid=1000
reboot
text

nfs --server=10.2.0.6 --dir=/mnt/store/deployment/mirror/el9/rhel-9.1

bootloader --append="rhgb quiet crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M"
zerombr
clearpart --all --initlabel
autopart
network --bootproto=dhcp
skipx
firstboot --disable
selinux --enforcing
firewall --enabled --ssh

%packages
@^minimal-environment
qemu-guest-agent
openssh-server
-kexec-tools
-dracut-config-rescue
-plymouth*
-iwl*firmware
%end

The above example is the kickstart file I have included for EL9 variants for virtual machines. Really the only thing that makes it special for virtual machines is that it removes a lot of firmware that won't be used and installs the qemu-guest-agent package so that it'll play nicely with KVM hypervisors when booted up as a guest image.

Make sure to change the line with NFS in it your local NFS server IP. This will be the IP of your TrueNAS server. In all fairness, you could just as easily replace this line for one that starts with url which allows you to specify an internet-based repository instead of one hosted locally via NFS. The option is yours. See here for more information.

I could give you a handful of kickstarts but in all honesty, the whole point of the kickstart files is customization. My Kickstart files (or anyone else's) wouldn't do you any good because your needs are unique your kickstart files should reflect that.

It's important to understand that the heart of this deployment workflow is the kickstart files though. Having multiple, special-purpose kickstart files associated with different boot options in your /pxelinux.cfg/default is where the real magic happens.

The options are really limitless. CentOS has a pretty illustrative documentation section on post-install scripts that can be run from the kickstart files:

Kickstart script file format reference :: CentOS Docs Site

I won't tell you what your kickstarts should look like but I will give you some ideas to get you going. You could configure any of the following in your kickstarts:

• create a domain joined host
• add your favorite repositories (e.g. Docker, EPEL, etc.)
• create an Ansible user for immediate provisioning
• use a kickstart to deploy identical nodes in a Linux computing cluster
• set a security baseline for all newly deployed hosts on your network
• configure logging and remote monitoring
• deploy an agent such osquery, Chef, Puppet, etc.

Hosting a Local OS Repository via NFS

The idea behind this is simple. You're basically going to mount the installation DVD and copy the entire contents into a folder in /mnt/$YOUR_STORE_NAME/deployment/mirror. In order to help with this task, I've included a target in the Makefile for this.

1. Download the installation DVD for the release you want to mirror and place it the /tmp/linux-deployment-server/iso directory we created when you cloned the repository from GitHub.
2. Navigate to /tmp/linux-deployment-server and edit the first few lines of the Makefile with the right variables for the ISO & release you need the boot files from.
3. Typing make osrepo in the same folder as the Makefile and it will place them in the correct directory. Make sure you change the REPO_DIR=./mirror line to the correct path in your TrueNAS store (it should be something like /mnt/$YOUR_STORE_NAME/deployment/pxelinux/images).

The man sections

Understanding the man pages lies in grasping the purpose and utilization of the various sections of the man page corpus (which is just a fancy word for collection).

There are 8 standard man page sections that originate from the original Unix developers Dennis Ritchie and Brian Kernighan (see Part I of the series for more).

These sections are briefly described if you type the following command:

man 7 man-pages

If you do you'll see something like the following:

1 User commands (Programs)
       Commands that can be executed by the user from within a shell.

2 System calls
       Functions which wrap operations performed by the kernel.

3 Library calls
       All library functions excluding the system call wrappers
       (Most of the libc functions).

4 Special files (devices)
       Files found in /dev which allow to access to devices through
       the kernel.

5 File formats and configuration files
       Describes various human-readable file formats and
       configuration files.

6 Games
       Games and funny little programs available on the system.

7 Overview, conventions, and miscellaneous
       Overviews or descriptions of various topics, conventions and
       protocols, character set standards, the standard filesystem
       layout, and miscellaneous other things.

8 System management commands
       Commands like mount(8), many of which only root can execute.

Now you may have commented that the command you typed to view this was a little peculiar. Instead of the typical man command, e.g. man vi, we added a number in between between the word man and what we want a man page for. If you were adventerous and tried to run man-pages in your shell you will have noticed that man-pages is not even a command.

What's going on here is that we've queried section 7 of the man pages with this command and section 7 is labelled "Overview, conventions, and miscellaneous." Thus, this command just returned a man page that was just generally giving information on the "man pages."

Hopefully you're having an "ah-ha" moment right now and realizing that there is a lot more to the man pages that finding out what arguments a command takes. Indeed, if you look at the sections you'll find there are all kinds of goodies tucked away in various sections of the man pages.

One of the core concepts discussed in Part I of this article series is that Unix-based operating systems attempt, traditionally, to come with all the documentation for each component that ship with it. That being said, to locate all the man pages available for a given section, just browse the folders your man pages are in! If you type:

$ ls -1d /usr/share/man/man*
/usr/share/man/man1
/usr/share/man/man4
/usr/share/man/man5
/usr/share/man/man6
/usr/share/man/man7
/usr/share/man/man8
/usr/share/man/man9

If you wish to view the man pages available for section 7 just type

ls /usr/share/man/man7

There are other places, however, that man pages may be stored. Check the shell variable $MANPATH for a comprehensive list for your system (and user). For a more thorough explanation of how $MANPATH is generated see:

man 5 manpath

You'll notice that this is in section 5 of the man pages. This is because it is a reference to the configuration file /etc/manpath.config. This is an extremely useful section. Linux relies heavily on configuration files for proper operation and you are sure to spend a lot of time getting to know them as you use Linux or become acquainted with new tools.

I would highly recommend running the following command to check out what man pages there are for existing configuration files on your system if you haven't ever done this:

ls /usr/share/man/man5

If you don't see any man pages listed in following the above instructions, then don't worry. This mystery will be solved in the next section.

The man-db & man-pages Packages

Now I’ll go out on a limb and assume none of you are running Unix proper at home (or anywhere) but instead one of it’s direct descendants like a distribution of the much loved GNU/Linux.

While the details of how GNU/Linux emerged from the world of Unix is beyond the scope of this article, all you need to know for the sake of this article is that the package man-db makes this mostly possible, and is more or less a port of the original Unix manual pager utilities package for GNU/Linux.

If the man-db package is the paging utilities that let you scroll up and down and search in a man page, then man-pages is the packages that contains the actual man pages that get fed into the pager: the raw, un-rendered man pages.

To install these packages on your Linux distribution should they be missing, use the following:

sudo apt install man-db man-pages

sudo dnf install man-db man-pages 

The man command

I'm sure you are all fairly familiar with the man command if you're reading an article about learning Linux so I won't spend long here.

I would like to point out an extremely useful switch available on the command for those of you that are unfamiliar: -k. This switch provides the same functionality as the antiquated apropos command, which is basically a "search" feature of the man pages.

The -k switch is going to be your go-to tool for situations where you know, or expect, there to be man pages on a topic but you're just not sure what they are.

For example, if you're setting up NFS for the first time on a server or on a client and you just want to see what's available to you as far as man pages on the topic, you would type man -k nfs.  You might do this because you don't know what the configuration files are, or you don't know what commands you need to run on the client, or on the server, etc.

When I run man -k nfs on my system I get the following results:

blkmapd (8)          - pNFS block layout mapping daemon
confstr (3)          - get configuration dependent string variables
filesystems (5)      - Linux filesystem types: ext, ext2, ext3, ext4, hpfs, iso9660, J...
fs (5)               - Linux filesystem types: ext, ext2, ext3, ext4, hpfs, iso9660, J...
idmapd (8)           - NFSv4 ID <-> Name Mapper
idmapd.conf (5)      - configuration file for libnfsidmap
ipa-client-automount (1) - Configure automount and NFS for IPA
mount.nfs (8)        - mount a Network File System
mountstats (8)       - Displays various NFS client per-mount statistics
nfs (5)              - fstab format and options for the nfs file systems
nfs.conf (5)         - general configuration for NFS daemons and tools
nfs.systemd (7)      - managing NFS services through systemd.
nfs4_uid_to_name (3) - ID mapping routines used for NFSv4
nfsconf (8)          - Query various NFS configuration settings
nfsidmap (5)         - The NFS idmapper upcall program
nfsiostat (8)        - Emulate iostat for NFS mount points using /proc/self/mountstats
nfsmount.conf (5)    - Configuration file for NFS mounts
nfsservctl (2)       - syscall interface to kernel nfs daemon
nfsstat (8)          - list NFS statistics
rpc.idmapd (8)       - NFSv4 ID <-> Name Mapper
rpc.sm-notify (8)    - send reboot notifications to NFS peers
rpcdebug (8)         - set and clear NFS and RPC kernel debug flags
showmount (8)        - show mount information for an NFS server
sm-notify (8)        - send reboot notifications to NFS peers
umount.nfs (8)       - unmount a Network File System

You'll notice there is a wealth of information available to you on the topic. You're made aware of everything from filetypes to use with the mount command, to configuration files, to the general man 5 filesystems page.

I cannot overemphasize how much you will learn about Linux by exploring the man pages with man -k.

The Wild West of Documentation: /usr/share

There is often overflow documentation available in /usr/share or /usr/share/doc for complicated programs and frameworks you may have installed on your system. I say "or" because technically /usr/share/doc, which sounds like it would be where the documentation goes, is mentioned briefly in section 4.11.3 of the Filesystem Hierarchy Standard, there's rarely anything too juicy in there. Instead, you'll often find the real documentation – should it exist – in a folder named after the program in question within /usr/share.

For example, gnupg usually comes extremely well-documented, including information in /usr/share/gnupg.  What you'll find in this folder usually comes in two varieties:

1. Example configuration files
2. Manuals and how-to guides in the form of .txt, .html, or sometimes .pdf documents.

While there is no real convention here as how this folder should be structured or what should be contained in here, it's anyone's guess as to what you'll find there for any given program. However conventionless the content may be, the notion of applications that ship with full documentation is something we have the original Unix creators for. That in mind, usually what you'll find here is documentation too extensive for man pages and for that reason you'll usually find it for programs that are quite complex or multi-faceted.

Please note that /share/ directories exist all over the place. Don't forget to check /usr/local/share, $HOME/.local/share and /opt/*/share. All of these are potential locations for documentation and sample configuration files.

To be continued...

In the next article in this series we'll talk about how our use of the command line and the man pages can (and may already have) reveal much deeper information about the inner workings of Linux.

What do I mean by this? One may become familiar with things like static vs dynamically compiled binaries, the ELF binary structure, the difference between user-space and kernel-space. This type of knowledge is what I would consider knowledge of Linux itself, and we'll refer to it as "conceptual knowledge" of Linux.

On the other hand, there is a completely different kind of fluency in Linux that refers to the ability, for example, to clean and explore tabular data like a CSV in Linux or to fluidly “slice & dice” log files via the command line to summarize runtime errors in Linux. This knowledge isn't about a component of Linux but rather refers to a type of pragmatic "know-how" one posseses. We'll call this type of knowledge "procedural knowledge" of Linux.

These two types of knowledge people may have in various capacities based on how they use Linux. For example, a data scientist may have a ton of procedural knowledge of Linux from time spent doing data science at the command-line but lack a core understanding of how Linux itself works. On the other hand, someone with a Computer Science degree that took some classes on operating systems may have a deep understanding of how Linux works internally but have minimal command-line fluency or operational competancy.

In order to say that you "know Linux" in the general sense, however, you're going to want a healthy dose of both. You need a robust ability to get things done in an efficient manner in Linux (preferably at the command line), and you're going to need a decent understanding of what's going on undernearth the hood as you perform these tasks.

Now there's really no one place to acquire this competancy. A classroom may teach you the conceptual knowledge you need but typically procedural knowledge is acquired through blood sweat and tears at a terminal. Likewise, some people may have put in the hours at the command line but are still feeling like they are missing some core knowledge of Linux internals and how things work under the hood.

The purpose of this article is to meet you where you are, wherever you are in your journey in learning Linux, and give you the tools to round out your knowledge of Linux. The good news is that the path I'm going to suggest is the same for everyone, and you can build off of whatever knowledge you have in a meaningful way to do this.

What is this single remedy to learning Linux? It's not a mysterious answer, but rather the man pages. Before you say "that's silly I already know about the man pages!" I implore to you consider the two ideas. How the man pages are a part of a much larger and extremely ambitious documentation project in Linux is not exactly obvious. Additionally – and more importantly, why the man pages are the crux of learning Linux is even less obvious.

Throughout Parts I & II of this article series I'm going to try and illustrate this second point: why the man pages are the crux of learning Linux. In doing so we're going to cover a lot of of the first point. Namely, to outline the general attempt Linux has made at being the first operating-system whose entire manual comes included with the operating system itself and use this to your advantage. Starting to get curious? Keep reading.

Unix Programmer’s Manual I: An attempt at documentation

Dennis Richie & Ken Thompson’s work was published in tandem with the Unix operating system itself, the two sharing a birthday of November 3, 1971.

The original Unix Programmer’s manual included a small amount of system call and executable manuals and was far from comprehensive manual of the Unix I OS. Additionally, the entire thing was distributed as a printed document and had to be ordered and mailed out!

Also worth noting was that at this time, there was no man command due to the lack of a need for a way to view the manual pages on the computer. This will soon change.

Unix Programmer’s Manual II: A fully documented operating system is born

By the time the second Unix Programmer’s Manual had come out the entire manual was now available on networked repositories.

With the digital publication of the man pages and the rendering capabilities developed in themancommand which to more or less rendered roff files that had, in the past, been sent to physical printers instead to a tty for interactive navigation using a pager, Unix v2 was now the the first OS that had ever attempted to be fully self-documenting and provide that documentation in some fashion with the OS itself. Unix v2 came as a complete package.

What began as workplace pain point that Richie & Thompson had begrudgingly started prior to the release of the original Unix had been imbued with a enough passion and discipline by the time the second release had come out it had basically matured into the direct descendent of the man pages we look at today.

In addition to the new network-hosted repositories with Unix II was a new command: man. The man command formatted the text for the screen and displayed them using what is called a pager, which is the program that allows you to scroll up and down the man pages or search through them using a search phrase while displayed on the screen. The default pager on most modern Linux distributions is less but most is a common, more feature rich alternative.

MAN(1)                     Manual pager utils                     MAN(1)
NAME         

       man - an interface to the system reference manuals

SYNOPSIS         

       man [man options] [[section] page ...] ...
       man -k [apropos options] regexp ...
       man -K [man options] [section] term ...
       man -f [whatis options] page ...
       man -l [man options] file ...
       man -w|-W [man options] page ...

DESCRIPTION         

       man is the system's manual pager.  Each page argument given to
       man is normally the name of a program, utility or function.  The
       manual page associated with each of these arguments is then found
       and displayed.  A section, if provided, will direct man to look
       only in that section of the manual.  The default action is to
       search in all of the available sections following a pre-defined
       order (see DEFAULTS), and to show only the first page found, even
       if page exists in several sections.

       The table below shows the section numbers of the manual followed
       by the types of pages they contain.

       1   Executable programs or shell commands
       2   System calls (functions provided by the kernel)
       3   Library calls (functions within program libraries)
       4   Special files (usually found in /dev)
       5   File formats and conventions, e.g. /etc/passwd
       6   Games
       7   Miscellaneous (including macro packages and conventions),
           e.g. man(7), groff(7)
       8   System administration commands (usually only for root)
       9   Kernel routines [Non standard]

       A manual page consists of several sections.

       Conventional section names include NAME, SYNOPSIS, CONFIGURATION,
       DESCRIPTION, OPTIONS, EXIT STATUS, RETURN VALUE, ERRORS,
       ENVIRONMENT, FILES, VERSIONS, CONFORMING TO, NOTES, BUGS,
       EXAMPLE, AUTHORS, and SEE ALSO.

This “man page” is an example of the manual page for the command man. I have bolded some sections to which I would like to direct your attention.

Here we see the 9 man sections enumerated. It also is quick to name some commonly used section names such as DESCRIPTION, OPTIONS, SEE ALSO, etc. We also see some of those section names in action in the man manpage itself: NAME, SYNOPSIS, and DESCRIPTION.

With the release of the Unix Programmer’s Manual v2 the follow traditions had been established:

1. It is the responsibility of the operating system distributor to package and distribute the correct documentation with its corresponding software.
2. The manual page corpus will be divided up into sections that cover the breadth of working within the operating system including configuration files, shell commands, and system calls. If the man pages were a book, these would be the chapters.
3. Manual pages themselves are further divided up into familiar section names, most of which were already established this by this point including NAME, DESCRIPTION, SEE ALSO, etc.

While all of these are important concepts to appreciate as someone learning Linux, the most important of these "traditions" one might call them, is the first: that the operating system will come with a full set of a documentation. This is going to be the an essential fact to keep in the back of your mind always. Whenever in doubt about something in while operating in Linux, know that the answer lies in the palm of your hands.

Finding help documentation in Linux

There are 3 places you will typically see this tradition of the self-documented system evidenced when attempting to find information in Linux or other Unix-based operating systems.

1. If a command line program, you will be able to append -h or --help to the command and get an abbeviated help document explaining how to use the command.
2. Using the various sections or "chapters" of the man page corupus, which collectively include all man pages on a given system.
3. The directories /usr/share and /usr/share/doc for programs that came with your Linux distribution along with /usr/local/share and /usr/local/share/doc for programs that you've installed.

We'll be covering (2) and (3) extensively in this article series. The first you should already be familiar with and if not there's not much more to be said in the context of help documentation except that it exists, so now you know! One additional tip, about (1) - for some commands --help will give an extended version of whatever -h spits out.

Another note, I would search for help in the order listed above. For commands, first try the --help documentation, should that not have the answer, check the man page, and if that does not have the answer to try the appropriate share directory.

There is one notable exception to this rule and that is regarding documentation for commands that come built-in to your shell. These commands are not installed, nor are they provided by your Linux distribution, but rather come compiled into your shell. Commands that fall in this category include those like cd, ls, dirs, and a decent amount of other commands. You can access documentation for these commands by using the help command: just type help [command] just like you would for a man page. For a full list of these commands just type help without any arguments.

When you’re first starting off you may use Ansible to SSH into your machine’s as root using username and password. This is about as simple of an Ansible access situation as you’ll run into. Why?

1. All commonly deployed Linux distributions will provide you with a root user by default.
2. In most situations, whether it be forced upon you or you decided it best, the root user will have a password defined.

This is fine if you’re just messing around with Ansible for the first time, but remember the goal of Ansible is to have a control node, a machine that you will use to push configuration changes to all your other machines. So you will be concentrating a massive amount of power (to do good or bad) into a small space. You may just have 1 Ansible user, e.g. named ansible, and 1 control host, e.g. your administrative machine.

‌If Ansible is set up properly, anyone that has access to this control node will be able to do large amounts of harm to your network because this user is completely designed to be able to go into any machine and make any changes.

The larger your umbrella of devices you manage with Ansible grows, the higher the stakes get with how well you’re protecting this special, often unrestricted access the Ansible control node gets. That being said, the more complex your Ansible environment gets the more you need to think about best practices and how to ensure you’re at least giving some attention to how exposed you are and in what ways; even if just in a lab environment.

Now it’s beyond the scope of this article to explain the details of why, but I’m going to assume you understand the security concerns associated with configuring every machine in your lab to allow root access over SSH using a password. Not only is root access over SSH not advised, but neither is using passwords to authenticate users over SSH. So what’s a better alternative?

Well, we need to dedicate a single, non-root user to make all of our changes via Ansible. This way, we configure auditing and logging on that user in such a way that we simply cannot on the root user. This will give us the ability to go back and see what was done in the event that a bad actor was to gain access to your Ansible control node. Additionally, we may tailor the security requirements of that account exactly as needed for the kind of access and methods of access we expect from our ansible user.

Secondly, passwords are among the weakest forms of authentication. This, like using a non-root user for administrative tasks is beyond the scope of this article, however, suffice it to say SSH key pairs employ asymmetric cryptography that allows the generation and use of credentials that do not have to be sent back and forth while communicating which reduces the possibility of the credential being intercepted, among other things.

So we know we need:

1. A user to SSH into our machines as that is not the root user
2. A way to authenticate using SSH that relies on asymmetric key pairs and not a password.

For the rest of this article, we’ll be talking about how to configure each one of our hosts that we want to configure using Ansible, either now or in the future, to have this user

1. Previously created.
2. Granted sudo access.
3. Only accepting remote SSH authentication via an SSH key pair.

If you want to manage your environment securely using Ansible, but do not have federated access across your home lab that will allow you to create and manage a central Ansible user in your Active Directory environment. This is for those of you who have a medium-sized home lab and want to manage it with Ansible.

If you don’t have the federated access that will allow a single Ansible user you can log in to all your Windows, Linux, etc. boxes with. The simple alternative is to create a separate ansible user with a more or less identical setup across all your machines.

So let's say you have 20 machines in your home lab (virtual, bare metal, or containers). The easiest way to manage them with Ansible is to have all the same Ansible SSH parameters:

1. hosts accept the same SSH key pair
2. hosts have the same SSH port
3. hosts will have the same SSH username

Since creating 20 ansible users across 20 machines would be a slight pain I’ve devised a script below you can tweak and run on any fresh box you spin up that you want to manage with Ansible:

#!/bin/sh

# Create an ansible user that requires no password and 
# only accepts SSH authentication. Modify the SSH public
# keys and other variables below to create your ansible
# user as needed.

ANSIBLE_USERNAME="${ANSIBLE_USERNAME:-ansible}"
ANSIBLE_USERID="${ANSIBLE_USERID:-2000}"
ANSIBLE_HOMEDIR_PARENT="${ANSIBLE_HOMEDIR_PARENT:-/home}"
ANSIBLE_SSHKEY=$(cat <<EOF
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAkfocK6qGdfGZLECDB/E5WuOWajWpkoP12JnBrloezb
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFdp/F7yYvsewnZQGbJAGmNcFNbm3qOFCOvrprXDlP24
EOF
)

# 1. Make ansible user

useradd \
  --uid "$ANSIBLE_USERID" \
  --base-dir="$ANSIBLE_HOMEDIR_PARENT"
  --create-home \  
  --user-group \
  --system \
  "$ANSIBLE_USERNAME"

# 2. Configure ansible user to have unrestricted, passwordless sudo access

printf '%%%s\tALL=(ALL)\tNOPASSWD:\sALL' "$ANSIBLE_USERNAME" > /etc/sudoers.d/$ANSIBLE_USERNAME

# 3. Configure ansible user's SSH keys to allow incoming connections

mkdir --mode=0700 ~$ANSIBLE_USERNAME/.ssh
chown $ANSIBLE_USERNAME:$ANSIBLE_USERNAME -R ~$ANSIBLE_USERNAME/.ssh
printf "%s\n" $ANSIBLE_SSHKEY > ~$ANSIBLE_USERNAME/.ssh/authorized_keys
chmod 0500 ~$ANSIBLE_USERNAME/.ssh/authorized_keys
chown $ANSIBLE_USERNAME:$ANSIBLE_USERNAME -R ~$ANSIBLE_USERNAME/.ssh/authorized_keys

Shifting Left: Simplifying the Process Even Further

Now that the script is written up and it’s been written in such a way that we can tweak the variables at the top to control, for example, the name of the ansible user and the user's UID, we can use it in a handful of creative ways other than running it by hand on every fresh machine you spin up.

If we already know we want the ansible user on every machine we have. We can create a base image that already includes this ansible user. That way, every time we need a new machine spun up the ansible user will already be created!

Now, there are a few ways to do this. Just to spark your imagination I’ll describe two ways, one more involved than the other.

Embedding the Script in a Kickstart File

Below is a very simple ks.cfg file that can be used to fully, or partially automate the installation of Enterprise Linux distros such as Red Hat or Rocky Linux by including all the variables that you are asked for during the walkthrough-style installation of the OS normally. Instead of being asked for your timezone, firewall preferences, root password, etc. You can just point the fresh machine at (a) this file and (b) the DVD or some other repository of all the packages Linux will download during the initial install. Once the installer has the data and the instructions it can install Linux exactly how you requested it.

The advantage of having a kickstart file set up this way with our little Ansible user created in the post-installation script is that this method can be used for bare-metal installs and virtual machines in more or less the same way. In other words, once you get this little script the way you like it it’s both portable and powerful for generating like images.

lang en_US
keyboard --xlayouts='us'
timezone America/New_York --isUtc
rootpw $2b$10$G0m7VBOzrmZKF5t7NVE.EuFdSmcXKX/WZyMZ..M.UgUfly3A.NMLy --iscrypted
reboot
text
cdrom
bootloader --append="rhgb quiet crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M"
zerombr
clearpart --all --initlabel
auth --passalgo=sha512
skipx
firstboot --disable
selinux --enforcing
firewall --enabled
%post

# THE SCRIPT THAT WE WROTE ABOVE CAN BE PLACED HERE IN
# BETWEEN THE "%post" MARKERS TO CREATE THE ANSIBLE USER
# IMMEDIATELY AFTER THE INITIAL OS IS INSTALLED BEFORE FIRST BOOT

%end

Running the Script Manually & Imaging the Machine

Alternatively, if the above method sounds too complicated for getting you up and running in your home lab with Ansible you can always do the following:

1. Install a fresh distribution of Linux in whatever way you’re most familiar with. If you’re struggling just hop over to https://www.osboxes.org) and download your favorite distro for your favorite hypervisor, VirtualBox is free ;)
2. Download the script above.
3. After tweaking the variables including your SSH public keys and, if you’d like, changing the ansible username, make the script executable and run it using the following command: chmod a+x script.sh && ./script.sh
4. Depending on your hypervisor, these instructions will vary. But you will want to export the virtual machine to OVF format so that it’s just a single file.

Once you’ve exported the VM you should then be able to use that to create your other new VMs either by copying or importing it. As I said above these instructions will vary based on your hypervisor but these links may be illuminating:

Virtualbox

1.14. Importing and Exporting Virtual Machines

VMWare Workstation

https://docs.vmware.com/en/VMware-Workstation-Pro/15.0/com.vmware.ws.using.doc/GUID-62F39498-1492-4774-A38D-1EDD3DA3C046.html

Hyper-V

Export and import virtual machines

Shows you how to export and import virtual machines using Hyper-V Manager or Windows PowerShell.

Microsoft LearnBenjaminArmstrong