Tag: configuration management

Salt State – Intro to SaltStack Configuration

Summary

This article picks up from Configuration Management – Introduction to SaltStack and dives into Salt State. It assumes you have an installed and working SaltStack. I call this intro to SaltStack configuration because this is the bulk of salt. Setting up the salt states and configuration. Understanding the configuration files, where they go and the format is the most important part of Salt.

The way I learn is a guided tour with a purpose and we will be doing just that. Our goal is first to create a salt state that installs apache.

Prepping Salt – Configuration

We need to modify “/etc/salt/master” and uncomment the following

#file_roots:
#  base:
#    - /srv/salt
#

This is where the salt states will be stored. Then we want to actually create that directory.

mkdir /srv/salt

vi /etc/srv/salt/webserver.sls

The contents of webserver.sls are as follows

httpd:
  pkg:
    - installed

This is fairly simple. We indicate a state “apache”, and define that package should be installed. We can apply it specifically as follows.

Applying Our First Salt State

# salt saltmaster1.woohoosvcs.com state.apply webserver
[WARNING ] /usr/lib/python3.6/site-packages/salt/transport/zeromq.py:42: VisibleDeprecationWarning: zmq.eventloop.minitornado is deprecated in pyzmq 14.0 and will be removed.
    Install tornado itself to use zmq with the tornado IOLoop.
    
  import zmq.eventloop.ioloop

saltmaster1.woohoosvcs.com:
----------
          ID: httpd
    Function: pkg.installed
      Result: True
     Comment: The following packages were installed/updated: httpd
     Started: 16:31:35.154411
    Duration: 21874.957 ms
     Changes:   
              ----------
              apr:
                  ----------
                  new:
                      1.6.3-9.el8
                  old:
              apr-util:
                  ----------
                  new:
                      1.6.1-6.el8
                  old:
              apr-util-bdb:
                  ----------
                  new:
                      1.6.1-6.el8
                  old:
              apr-util-openssl:
                  ----------
                  new:
                      1.6.1-6.el8
                  old:
              centos-logos-httpd:
                  ----------
                  new:
                      80.5-2.el8
                  old:
              httpd:
                  ----------
                  new:
                      2.4.37-12.module_el8.0.0+185+5908b0db
                  old:
              httpd-filesystem:
                  ----------
                  new:
                      2.4.37-12.module_el8.0.0+185+5908b0db
                  old:
              httpd-tools:
                  ----------
                  new:
                      2.4.37-12.module_el8.0.0+185+5908b0db
                  old:
              mailcap:
                  ----------
                  new:
                      2.1.48-3.el8
                  old:
              mod_http2:
                  ----------
                  new:
                      1.11.3-3.module_el8.0.0+185+5908b0db
                  old:

Summary for saltmaster1.woohoosvcs.com
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  21.875 s

You’ll note 1) the annoying warning which I will be truncating from further messages but 2) that it installed httpd (apache). You can see it also installed quite a few other dependencies that apache required.

Let’s validate quickly

# systemctl status httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:httpd.service(8)

# ls -la /etc/httpd/
total 12
drwxr-xr-x.  5 root root  105 Nov 10 16:31 .
drwxr-xr-x. 80 root root 8192 Nov 10 16:31 ..
drwxr-xr-x.  2 root root   37 Nov 10 16:31 conf
drwxr-xr-x.  2 root root   82 Nov 10 16:31 conf.d
drwxr-xr-x.  2 root root  226 Nov 10 16:31 conf.modules.d
lrwxrwxrwx.  1 root root   19 Oct  7 16:42 logs -> ../../var/log/httpd
lrwxrwxrwx.  1 root root   29 Oct  7 16:42 modules -> ../../usr/lib64/httpd/modules
lrwxrwxrwx.  1 root root   10 Oct  7 16:42 run -> /run/httpd
lrwxrwxrwx.  1 root root   19 Oct  7 16:42 state -> ../../var/lib/httpd

Looks legit to me! We just installed our first salt state. You can “yum remove” httpd and apply again and it will install. The real power in configuration management is that it knows the desired state and will repeatedly get you there. It is not just a one and done. This is the main difference between provisioning platforms and configuration management.

Installing More Dependencies

WordPress also needs “php-gd” so let’s modify the salt state to add it and then reapply.

httpd:
  pkg:
    - installed

php-gd:
  pkg:
    - installed

Here you can see it did not try to reinstall apache but did install php-gd.

# salt saltmaster1.woohoosvcs.com state.apply webserver

saltmaster1.woohoosvcs.com:
----------
          ID: httpd
    Function: pkg.installed
      Result: True
     Comment: All specified packages are already installed
     Started: 16:41:36.742596
    Duration: 633.359 ms
     Changes:   
----------
          ID: php-gd
    Function: pkg.installed
      Result: True
     Comment: The following packages were installed/updated: php-gd
     Started: 16:41:37.376211
    Duration: 20622.985 ms
     Changes:   
              ----------
              dejavu-fonts-common:
                  ----------
                  new:
                      2.35-6.el8
                  old:
              dejavu-sans-fonts:
                  ----------
                  new:
                      2.35-6.el8
                  old:
              fontconfig:
                  ----------
                  new:
                      2.13.1-3.el8
                  old:
              fontpackages-filesystem:
                  ----------
                  new:
                      1.44-22.el8
                  old:
              gd:
                  ----------
                  new:
                      2.2.5-6.el8
                  old:
              jbigkit-libs:
                  ----------
                  new:
                      2.1-14.el8
                  old:
              libX11:
                  ----------
                  new:
                      1.6.7-1.el8
                  old:
              libX11-common:
                  ----------
                  new:
                      1.6.7-1.el8
                  old:
              libXau:
                  ----------
                  new:
                      1.0.8-13.el8
                  old:
              libXpm:
                  ----------
                  new:
                      3.5.12-7.el8
                  old:
              libjpeg-turbo:
                  ----------
                  new:
                      1.5.3-7.el8
                  old:
              libtiff:
                  ----------
                  new:
                      4.0.9-13.el8
                  old:
              libwebp:
                  ----------
                  new:
                      1.0.0-1.el8
                  old:
              libxcb:
                  ----------
                  new:
                      1.13-5.el8
                  old:
              php-common:
                  ----------
                  new:
                      7.2.11-1.module_el8.0.0+56+d1ca79aa
                  old:
              php-gd:
                  ----------
                  new:
                      7.2.11-1.module_el8.0.0+56+d1ca79aa
                  old:

Summary for saltmaster1.woohoosvcs.com
------------
Succeeded: 2 (changed=1)
Failed:    0
------------
Total states run:     2
Total run time:  21.256 s

The output of state.apply is rather long so we likely will not post too many more. With that said, I wanted to give you a few examples of the output and what it looks like.

Downloading Files

Next we need to download the WordPress files. The latest version is always available via https://wordpress.org/latest.tar.gz. Salt has a Salt State for managed file to download but it requires us to know the hash of the file to ensure it is correct. Since “latest” would change from time to time, we do not know what that is. We have two options. The first is to store a specific version on the salt server and provide that. The second is to use curl to download the file.

download_wordpress:
  cmd.run:
    - name: curl -L https://wordpress.org/latest.tar.gz -o /tmp/wp-latest.tar.gz
    - creates: /tmp/wp-latest.tar.gz

In order for salt to not download the file every time, we need to tell it what the command we are running will store. We tell it, it creates the “/tmp/wp-latest.tar.gz” so it should only download if that file does not exist.

Downloading is not all we need to do though, we also need to extract it.

Extracting

extract_wordpress:
  archive.extracted:
    - name: /tmp/www/
    - source: /tmp/wp-latest.tar.gz
    - user: apache
    - group: apache

The archive.extracted module allows us to specify a few important parameters that are helpful. Applying the state we can see its there!

[root@saltmaster1 salt]# ls -la /tmp/www/
total 4
drwxr-xr-x.  3 apache root     23 Nov 10 16:57 .
drwxrwxrwt. 11 root   root    243 Nov 10 16:59 ..
drwxr-xr-x.  5 apache apache 4096 Oct 14 15:37 wordpress

We actually want it in the “/var/www/html” root so the webserver.sls was modified to the following.

extract_wordpress:
  archive.extracted:
    - name: /var/www/html
    - source: /tmp/wp-latest.tar.gz
    - user: apache
    - group: apache
    - options: "--strip-components=1"
    - enforce_toplevel: False

The issue is the tar has the “wordpress” directory as the root and we want to strip that off. We need the options to pass to tar to strip it. We also need the enforce_toplevel to false as Salt expects a singular top level folder. I found this neat trick via https://github.com/saltstack/salt/issues/54012

# Before
# ls -la /var/www/html/wp-config*
-rw-r--r--. 1 apache apache 2898 Jan  7  2019 /var/www/html/wp-config-sample.php
# salt saltmaster1.woohoosvcs.com state.apply webserver

# After
# ls -la /var/www/html/wp-config*
-rw-r-----. 1 apache apache 2750 Nov 10 18:03 /var/www/html/wp-config.php
-rw-r--r--. 1 apache apache 2898 Jan  7  2019 /var/www/html/wp-config-sample.php

Sourcing the Config

We now have a stock WordPress install but we need to configure it to connect to the database.

For that I took a production wp-config.php and placed it in “/srv/salt/wordpress/wp-config.php” on the salt master. I then used the following salt state to push it out

/var/www/html/wp-config.php:
  file.managed:
  - source: salt://wordpress/wp-config.php
  - user: apache
  - group: apache
  - mode: 640

Set Running Salt State

What could would Apache do if it weren’t running. We do need a salt state to enable and run it!

start_webserver:
  service.running:
  - name: httpd
  - enable: True
# systemctl status httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-11-10 18:27:24 CST; 1min 5s ago

Final Words

Through this we configured a webserver.sls salt state. We used it to install apache and a basic php module necessary as well as push out a configuration file. As you can likely tell from these instructions, it is an iterative approach to configuring the salt state for your need.

This first iteration of the webserver.sls is far from complete or best practice. It is meant as a beginner’s guide to walking through the thought process. Below is the full webserver.sls file for reference

httpd:
  pkg:
    - installed

php-gd:
  pkg:
    - installed

download_wordpress:
  cmd.run:
    - name: curl -L https://wordpress.org/latest.tar.gz -o /tmp/wp-latest.tar.gz
    - creates: /tmp/wp-latest.tar.gz

extract_wordpress:
  archive.extracted:
    - name: /var/www/html
    - source: /tmp/wp-latest.tar.gz
    - user: apache
    - group: apache
    - options: "--strip-components=1"
    - enforce_toplevel: False

/var/www/html/wp-config.php:
  file.managed:
  - source: salt://wordpress/wp-config.php
  - user: apache
  - group: apache
  - mode: 640

start_webserver:
  service.running:
  - name: httpd
  - enable: True

Configuration Management – Intro to SaltStack

Summary

SaltStack or Salt for short is an open source configuration management platform. It was first released in the early 2010’s as a potential replacement for Chef and Puppet. In this guide we will walk through some high level details of Salt and a basic install. If you already have Salt installed, please skip ahead to the next article when it is published.

Next – Salt State – Intro to SaltStack Configuration

What is Configuration Management?

A configuration management tool allows you to remotely configure and dictate configurations of machines. Through this multi-part series we will work through that with the use case of https://blog.woohoosvcs.com/. At some point this site may need multiple front ends. It has not been decided if that method will be Kubernetes, Google App Engine or VMs. If the VM route is chosen it will make sense to have an easy template to use.

What Configuration Management is not

Configuration management typically does not involve the original provisioning of the server. There are typically other tools for that such as Terraform.

Salt Architecture

Salt has three main components to achieve configuration management. Those are the salt master, minion and client. Salt can be configured highly available with multi master but it is not necessary to start out there. For the sake of this document and per Salt’s best practices we can add that later if necessary. – https://docs.saltstack.com/en/latest/topics/development/architecture.html

Salt Client

The salt client is a command line client that accepts commands to be issued to the salt master. It is typically on the salt master. You can use it to trigger expected states.

Salt Master

The Salt Master is the broker of all configuration management and the brains. Requests/commands received from the client make their way to the master which then get pushed to the minion.

Salt Minion

The minion is typically loaded onto each machine you wish to perform configuration management on. In our case, it will be the new front ends we spin up as we need them.

Firewall Ports

The Salt Master needs ports TCP/4505-4506 opened. The minions check in and connect to the master on those ports. No ports are needed for the minions as they do not listen on ports.

More on this can be found on their firewall page – https://docs.saltstack.com/en/latest/topics/tutorials/firewall.html

Where to install

Typically you want the master to be well connected since the minions will be connecting to it. Even if you are primarily on prem, it is not a bad idea to put a salt master in the cloud.

Installing Salt

Prerequisites

OS: CentOS 8
VM: 1 core, 1 GB RAM, 12 GB HDD
Install: Minimal

Installation

For the installation we will be closely following Salt’s documentation on installing for RHEL 8 – https://repo.saltstack.com/#rhel

For the sake of this lab we will have the client, master and minion all on the same server but it will allow us to build out the topology.

Now to the install!

# I always like to start out with the latest up to date OS
sudo yum update

# Install the salt repo for RHEL/CentOS
sudo yum install https://repo.saltstack.com/py3/redhat/salt-py3-repo-latest.el8.noarch.rpm

# Install minion and master
sudo yum install salt-master salt-minion

# Reboot for OS updates to take effect
reboot

Post Install Configuration

We need to setup a few things first. The Salt guide is great at pointing these out – https://docs.saltstack.com/en/latest/ref/configuration/index.html

On the minion we need to edit “/etc/salt/minion”. The following changes need to be made. If/when you roll this out into production you can use DNS hostnames.

#master: salt
master: 192.168.116.186

We will also open up the firewall ports on the master

firewall-cmd --permanent --zone=public --add-port=4505-4506/tcp
firewall-cmd --reload

Start up Configuration Management Tools

Now we are ready to start up and see how it runs!

sudo systemctl enable salt-master salt-minion
sudo systemctl start salt-master salt-minion

Wait about 5 minutes. It takes a little bit to initialize. Once it has you can run “sudo salt-key -L”. When a minion connects to the master, the master does not allow it to connect automatically. It has to be permitted/admitted. salt-key can be used to list minions and allow them.

$ sudo salt-key -L
Accepted Keys:
Denied Keys:
Unaccepted Keys:
saltmaster1.woohoosvcs.com
Rejected Keys:

$ sudo salt-key -A
The following keys are going to be accepted:
Unaccepted Keys:
saltmaster1.woohoosvcs.com
Proceed? [n/Y] Y
Key for minion saltmaster1.woohoosvcs.com accepted.
[dwcjr@saltmaster1 ~]$ sudo salt-key -L
Accepted Keys:
saltmaster1.woohoosvcs.com
Denied Keys:
Unaccepted Keys:
Rejected Keys:

We used salt-key -A to accept all unaccepted keys.

Testing

$ sudo salt saltmaster1 test.version
[WARNING ] /usr/lib/python3.6/site-packages/salt/transport/zeromq.py:42: VisibleDeprecationWarning: zmq.eventloop.minitornado is deprecated in pyzmq 14.0 and will be removed.
    Install tornado itself to use zmq with the tornado IOLoop.
    
  import zmq.eventloop.ioloop

No minions matched the target. No command was sent, no jid was assigned.
ERROR: No return received
[root@saltmaster1 ~]# salt '*' test.version
[WARNING ] /usr/lib/python3.6/site-packages/salt/transport/zeromq.py:42: VisibleDeprecationWarning: zmq.eventloop.minitornado is deprecated in pyzmq 14.0 and will be removed.
    Install tornado itself to use zmq with the tornado IOLoop.
    
  import zmq.eventloop.ioloop

saltmaster1.woohoosvcs.com:
    2019.2.2

Well that is an ugly error code. It seems to have been introduced in 2019.2.1 but not properly fixed in 2019.2.2. My guess is the next release will fix this but it seems harmless. – https://github.com/saltstack/salt/issues/54759. We do, however, get the response so this is a success.

Final Words

At this point we have a salt-master and salt-minion setup, albeit on the same host. We have accepted the minion on the master and they are communicating. The next article will start to tackle setting up Salt states and other parts of the salt configuration.

Next – Salt State – Intro to SaltStack Configuration