Setting up the SaltStack Masterful Architecture on GCE

26 Nov 2016

During the thanksgiving weekend, I have been experimenting with SaltStack on Google Compute Engine. It takes some imagination to setup the SaltStack masterful architecture.

After reading and trying the tutorial, the first set of questions that came to my mind is:

  • How would the master know the minion's key?
  • And how would the minion know the master's finger?
  • Who gets to know whom first?
  • And how do they verify the identity of the other?

Now there's another article about preseeding the minion key on the master. That helps a little bit. But the questions still remain.

After thinking for a while, there's no way to solve the master-minion chicken-and-egg problem. A divine deity must intervene and introduce the minion to the master (and vice versa).

The divine deity is us. Our local machine can ssh into both the master and the recently created minion and, thus, can introduce them to each other.

With Fabric, we can orchestrate the introduction between the master and the recently created minion. Here's how it works:

  1. Generate the private-public key pair for the minion.
  2. Install the private key and the public key into /etc/salt/pki/minion/minion.pem and /etc/salt/pki/minion/, respectively, on the minion.
  3. Install the minion's public key into /etc/salt/pki/master/minions/[minion-id] on the master.
  4. Get the master finger and the master's IP address and generate the minion's config file.
  5. Put the minion's config file into /etc/salt/minion on the minion.
  6. Restart the minion service.

And that's it. The master accepts the minion, and the minion accepts the master.

The full source code that works on GCE with both Ubuntu VM and windows VM is here: Please read and read the source code on how it works.

Why don't we use salt-cloud instead?

As I understand, salt-cloud achieves almost the same thing. The main difference is that the master acts as the divine deity; it can SSH into all minions.

That won't work in my case because my team's infrastructure is fairly complicated. For example, sshing into certain machines require me to touch my USB security key and type my password.

Also, salt-cloud doesn't support creating the master.