# Dotfiles with Ansible

# TL:DR

How to quickly configure a fresh system. A quick introduction to Ansible and automation of personal setup for macOS and Arch Linux. Installation of apps, config files, and secrets.

# The problem

Setting up a fresh system might be a little cumbersome. You already have your perfect setup. Everything is installed, everything is configured.

Whether you got a new computer, are reinstalling due to an upgrade, or are just setting up a virtual machine, it is time-consuming to set up everything just as you like.

It is not only time-consuming, but it is also easy to forget some things or just forget how to configure something.

How can we solve that problem?

## Solutions

There are a few ways to handle this situation.

### Dotfiles

The simplest way is to copy config files and store them somewhere, for example, in a git repository. A lot of people store their `dotfiles` in a repository. We can make some README or other notes to remember how to set it up as we like.

This was also my first approach to this. But wouldn't it be better to automate it?

### Bash

I've created a lot of bash scripts to set up everything as I wanted, and it was working just fine. But it was not so good at maintenance.

So, looking for something better, I gave Ansible a try.

### Ansible

This is simple to use, easy to maintain, and gives a lot of possibilities to automate virtually anything. So what it is and how to use it?

# Introduction to ansible

## What is it?

Ansible is software that enables automation and orchestration. It can automate virtually any task.

> Ansible is an open source, command-line IT automation software application written in Python. It can configure systems, deploy software, and orchestrate advanced workflows to support application deployment, system updates, and more. ~ [ansible.com](https://www.ansible.com/how-ansible-works/)

It is designed around the following principles:

* agent-less architecture - low maintenance;
    
* simplicity - simple `YAML` syntax, using `SSH` to connect to the machines;
    
* scalability and flexibility - easily and quickly scale through modular design;
    
* idempotence and predictability - when the system is in the desired state, it will not change even if the playbook is run multiple times;
    

## How to install

Ansible can be installed using Python and `pip`. Or, for some systems like Arch Linux and macOS, we may use package managers.

For Arch Linux using `yay`:

```bash
yay -S ansible
```

For MacOS using `brew`:

```bash
brew install ansible
```

For more detailed information about installation look here: [documentation](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html).

## Basics

### Inventory

Ansible needs to know which machines it will manage. To achieve that, all we need is a simple inventory file in `ini` or `yaml` format.

```ini
[webservers]
192.168.1.11
192.168.1.12

[databases]
192.168.1.21
192.168.1.22
```

or

```yaml
webservers:
  hosts:
    app: 192.168.1.11
    app2: 192.168.1.12
databases:
  hosts:
    app: 192.168.1.21
    app2: 192.168.1.22
```

Inventory defines the managed nodes and groups of them. In the above example, we have two groups - `webservers` and `databases`. Each group has two IP addresses. By default, Ansible also creates two other groups - `all` and `ungrouped`. The first one contains all hosts, and the second one includes all hosts not grouped under anything other than `all`.

There are some more things that we can set in the inventory, like the user that we are using:

```yaml
webservers:
  hosts:
    app:
      ansible_host: 192.168.1.11
      ansible_user: app_user
```

There is more, and we can read about this in the official [documentation](https://docs.ansible.com/ansible/latest/inventory_guide/index.html).

### Task

A task is a single automation, like installing a package or creating a directory.

#### Create task

A task can be created in a separate `yaml` file or directly in a playbook.

```yaml
- name: Hello world task
  ansible.builtin.debug:
    msg: "Hello world!"
```

### Modules

Modules or plugins are units that are run from the command line or tasks. Ansible executes each module, usually on a managed node (host), and collects the return value from it. Modules can be collected into collections. We can use built-in modules, modules from the community, or create our own.

Modules can be called in tasks, as in the above task example.

To use the command line:

```bash
ansible home -m debug -a "msg=Hello!" -i inventory.yml
```

home is a group of hosts from the inventory that we specified with `-i inventory.yml`

This should result in:

```bash
pc | SUCCESS => {
    "msg": "Hello!"
}
```

pc is a host from the inventory.

### Collection

Collections are a distribution format for Ansible content that can include playbooks, roles, modules, and plugins. They are useful when using something that is not built-in, created by someone else.

To install collections, we can use `ansible-galaxy` and a requirements file.

Let's say that we need modules to install packages from `aur` and `brew`. To do this, we need `community.general` and `kewlfft.aur`.

With a `requirements.yml` file:

```yaml
collections:
  - community.general
  - kewlfft.aur
```

We can use galaxy:

```bash
ansible-galaxy install -r requirements.yml
```

It will install all collections from the file, and they will be available for use.

### Role

A structured way to organize playbooks into reusable components. It contains tasks, variables, files, templates, and more.

Roles go into the `roles` directory and have a specified structure. Below we have an example from [documentation](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_reuse_roles.html):

```bash
roles/
    common/               # this hierarchy represents a "role"
        tasks/            #
            main.yml      #  <-- tasks file can include smaller files if warranted
        handlers/         #
            main.yml      #  <-- handlers file
        templates/        #  <-- files for use with the template resource
            ntp.conf.j2   #  <------- templates end in .j2
        files/            #
            bar.txt       #  <-- files for use with the copy resource
            foo.sh        #  <-- script files for use with the script resource
        vars/             #
            main.yml      #  <-- variables associated with this role
        defaults/         #
            main.yml      #  <-- default lower priority variables for this role
        meta/             #
            main.yml      #  <-- role dependencies
        library/          # roles can also include custom modules
        module_utils/     # roles can also include custom module_utils
        lookup_plugins/   # or other types of plugins, like lookup in this case
```

By default, when using a role, Ansible will look for a `main.yml` (or `main.yaml`, or `main`) file.

However, it is still possible to have multiple files there and organize everything according to needs.

### Play

This is a section within a playbook that defines what to run and on which host group.

```yaml
- name: Simple play
  hosts: webservers
  roles:
    - common
  tasks:
    - name: Hello
      ansible.builtin.debug:
        msg: Hello!
```

This is one play that will run the role `common` and one task.

### Playbook

A playbook is a blueprint for automation. It is a simple configuration for what to run on nodes specified in the inventory.

Each playbook is built from plays that are run from top to bottom.

### Facts

Data related to the remote system Ansible manages is available as facts. To access those variables, use `ansible_facts` or `ansible_{name}`.

## How to use

Let's build a simple example that can be run locally.

### Inventory

To achieve this, we need to set up the inventory correctly.

Create `inventory.yml`:

```yaml
home:
  hosts:
    pc:
      ansible_host: localhost
      ansible_user: isur
      ansible_connection: local
```

I have created a group `home` with the `pc` host that has some settings:

* `ansible_host` - host - we want to use it locally, so `localhost`
    
* `ansible_user`\- this is the user that will be used on a machine - set it for your user
    
* `ansible_connection: local` - we do not want to use an SSH connection to our own system
    

### Running playbook

To run a playbook, use the command `ansible-playbook` with the path to the playbook as an argument. To select the inventory, use `-i`:

```bash
ansible-playbook play.yml -i inventory.yml
```

Before running this command, first create a playbook.

### Simple playbook

Create `playbook.yml`:

```yaml
- name: Simple play
  hosts: home
  tasks:
    - name: Hello user
      ansible.builtin.debug:
        msg: "Hello {{ ansible_user }}!"
```

Ansible has a built-in debug module that lets you print out messages. We can use variables with templating like above. `ansible_user` is a fact with information about the username.

Below you can see the result of that task.

```bash
TASK [Hello user] *******************************************************************
ok: [pc] => {
    "msg": "Hello isur!"
}
```

That might be all we need for a simple playbook. But when we have more tasks it might be better to move them to files and import them later.

```yaml
- name: Simple task imported
  hosts: home
  tasks:
    - ansible.builtin.import_tasks: ./tasks/hello.yml
```

That way, we can group some tasks or even import tasks into other tasks.

But there is an even better way to organize automation.

### With roles

Instead of grouping tasks into random directories, we can use `roles`.

Let's move the hello task into a role, and now the file structure should look like this:

```bash
./
    inventory.yml
    play.yml
    roles/
        hello/
            tasks/
                main.yml
```

And instead of importing tasks, we can use roles:

```yaml
- name: Play with roles
  hosts: home
  roles:
    - hello
```

`main.yml` in the hello role will look the same as any other task, but it will be easier to maintain and extend. Now we can add something more to it, so we will have some initial information when running the playbook.

```yaml
- name: Hello user
  ansible.builtin.debug:
    msg: "Hello {{ ansible_user }}!"

- name: Hello system
  ansible.builtin.debug:
    msg: "System: {{ ansible_os_family }}"
```

Now, after running this, we can see which user and what system is used:

```bash
TASK [hello : Hello user] ******************************************************
ok: [pc] => {
    "msg": "Hello isur!"
}

TASK [hello : Hello system] ****************************************************
ok: [pc] => {
    "msg": "System: Archlinux"
}
```

Those facts `ansible_user` and `ansible_os_family` might be useful when we want to do something depending on the operating system or username. We might also use environment variables with `ansible_env`.

# Personal system setup

## Goal

The goal is to prepare automation to handle two systems - Arch Linux and macOS. On both of them, I want to install apps, create directories, copy files, create symlinks for configs, and decrypt secrets (ssh).

## Plan

To keep everything organized, I will use roles for different groups of tasks. The only requirement is to have Ansible and `yay` or `brew` installed, depending on the system.

## Collections

Let's start with the required collections. As I am using Arch Linux and yay, I will use `kewlfft.aur`, and for macOS and brew, `community.general`. So, `requirements.yml` will have this:

```yaml
collections:
  - community.general
  - kewlfft.aur
```

## Inventory

For the localhost machine, the `inventory.yml` file will be simple. Remember to use the correct `user` and include `ansible_connection: local`, so Ansible will not use SSH.

```yaml
home:
  hosts:
    pc:
      ansible_host: localhost
      ansible_user: isur
      ansible_connection: local
```

## Playbook

The playbook will just load roles. For better organization, I am splitting all tasks into a few roles:

* general - some general stuff that I will always install
    
* tiling - setting up tiling managers and the system for that usage
    
* dev - everything I need for software development work
    
* gaming - some gaming stuff
    

So the `playbook.yml` file will look like this:

```yaml
- name: System Setup
  hosts: home
  roles:
    - general
    - tiling 
    - dev
    - gaming
```

## Roles

Each role will be prepared for both systems, Arch Linux and macOS. The Ansible fact `ansible_os_family` for Arch is `Archlinux` and for macOS is `Darwin`. With `when` and this fact, we can detect which task to run on which system.

I will not cover my entire setup here, but I will give some examples of how to do some tasks.

### Installation role

Let's install `discord` as an example. First, we need to create a `discord` role in the `roles` directory and a `main.yml` task.

`roles/discord/tasks/main.yml`

And in this task, we need to remember both systems.

```yaml
- name: Darwin | Install Discord from brew
  community.general.homebrew_cask:
    name: discord
    state: present
    accept_external_apps: true
  when: ansible_os_family == 'Darwin'

- name: Arch | Install Discord from aur
  kewlfft.aur.aur:
    name: discord
    use: yay
    state: present
  when: ansible_os_family == 'Archlinux'
```

And that's it. For Arch, it will install `discord` using `yay`, and for Mac, it will use `brew`. If the app is already installed on Mac in a different way, `accept_external_apps` will not raise an error in that case.

If tasks are more complex, we can split up the file into different files per system.

```yaml
- name: Darwin | Install Discord
  ansible.builtin.include_tasks: "./darwin.yml"
  when: ansible_os_family == 'Darwin'

- name: Arch | Include Linux specific tasks
  ansible.builtin.include_tasks: "./arch.yml"
  when: ansible_os_family == 'Archlinux'
```

And put the tasks defined earlier into `arch.yml` and `darwin.yml`.

This will load all tasks from the file specified in `ansible.builtin.include_tasks`.

The file tree right now would look like this:

```bash
.
├── collections.yml
├── inventory.yml
├── play.yml
└── roles
    └── discord
        └── tasks
            ├── arch.yml
            ├── darwin.yml
            └── main.yml
```

This is a simple installation without any additional steps. It might be a little too much work to create a `role` for each app if we just want to install a few things without anything extra.

We can do this in one role, even in one task, using loops. For example, let's create a role `communication` which will install apps for communication: `discord`, `slack`, and `thunderbird`.

The system selection task will look exactly the same as above.

```yaml
- name: Arch | Install communication apps
  kewlfft.aur.aur:
    name: "{{ app }}"
    use: yay
    state: latest
  loop:
    - discord
    - slack-desktop
    - thunderbird
  loop_control:
    loop_var: app
```

Now `name` instead of the name of the app is the item from the loop. We can define how the variable is named in `loop_control` and `loop_var`. Without defining the variable name, it will be `item`. This loop will run for each item specified under `loop`.

We can also move those items into variables.

To do this, we need to create a directory `vars` in the role and `main.yml`.

```yaml
arch_apps:
  - discord
  - slack-desktop
  - thunderbird

darwin_apps:
  - discord
  - slack
  - thunderbird
```

Now instead of passing a list of items, we pass a variable:

```yaml
- name: Arch | Install communication apps
  kewlfft.aur.aur:
    name: "{{ app }}"
    use: yay
    state: latest
  loop: "{{ arch_apps }}"
  loop_control:
    loop_var: app
```

This way, instead of creating multiple roles for a simple installation process, we can just do it like this.

```bash
.
├── collections.yml
├── inventory.yml
├── play.yml
└── roles
    ├── communication
    │   ├── tasks
    │   │   ├── arch.yml
    │   │   ├── darwin.yml
    │   │   └── main.yml
    │   └── vars
    │       └── main.yml
    └── discord
        └── tasks
            ├── arch.yml
            ├── darwin.yml
            └── main.yml
```

There are different ways to organize this stuff; everything is up to you.

### Symlink/copy files

What if I want to configure my apps and I have some `dotfiles`?

Let's configure the tiling manager. I am using `i3` on Arch and `aerospace` on MacOS. We can create different roles for them, or just a role `tiling` and store everything there.

Inside the role directory, we will now also need a `files` directory where we store our config files.

Select the system as before and split arch and darwin into `arch.yml` and `darwin.yml`.

The file tree will now look like:

```bash
.
├── collections.yml
├── inventory.yml
├── play.yml
└── roles
    ├── communication
    │   ├── tasks
    │   │   ├── arch.yml
    │   │   └── main.yml
    │   └── vars
    │       └── main.yml
    ├── discord
    │   └── tasks
    │       ├── arch.yml
    │       ├── darwin.yml
    │       └── main.yml
    └── tiling
        ├── files
        │   ├── aerospace.toml
        │   └── i3
        │       └── config.conf
        └── tasks
            ├── arch.yml
            ├── darwin.yml
            └── main.yml
```

Tasks for installing tiling stuff look exactly like before with the communication role. But this time, we need some config files:

```yaml
- name: Arch | Install tiling
  kewlfft.aur.aur:
    name: "{{ item }}"
    use: yay
    state: present
  loop:
    - i3-wm
    - rofi
    - feh

- name: Arch | Config tiling
  ansible.builtin.file:
    src: "{{ role_path }}/files/{{ item.src }}"
    dest: "{{ item.dest }}"
    state: link
  loop:
    - { src: "i3", dest: "{{ ansible_env.HOME }}/.config/i3" }
    - { src: "rofi", dest: "{{ ansible_env.HOME }}/.config/rofi" }
```

Using the `ansible.builtin.file` module, we can define where files should be. `src` points to the source for the file. The `role_path` variable points to the role path. `dest` is the destination, and `state: link` makes it a symbolic link.

This way, both `~/.config/i3` and `~/.config/rofi` are symbolic links to the config in our Ansible files.

If we want to copy files instead of making symbolic links, we can use the `ansible.builtin.copy` module with `src` and `dest`. We can also define permissions by using `mode` with permission numbers.

### Decrypt files

What if I tell you that we can store our secrets in a public repository?

For this example, I will set up SSH keys.

Let's create a role `ssh` and put our SSH key in the `files` directory.

The role directory will look like this:

```bash
    ├── ssh
    │   ├── files
    │   │   ├── id_rsa
    │   │   └── id_rsa.pub
    │   └── tasks
    │       └── main.yml
```

First, we need to encrypt the files. Go to the directory with the files and:

```bash
ansible-vault encrypt id_rsa id_rsa.pub
```

Set the password and confirm. Done, now the files are encrypted. Or, if you save your password to a file (e.g., in `~/.vault_pass`):

```bash
ansible-vault encrypt id_rsa id_rsa.pub --vault-password-file=$HOME/.vault_pass
```

THIS PASSWORD IS SUPER SECRET, NEVER PUT IT INTO THE REPOSITORY.

`encrypt` will encrypt the file. You can also use `edit`, `view`, and `decrypt` commands.

Now the files are safely encrypted. We can use them in our task to copy to the correct place in our system.

Let's go to `roles/ssh/tasks/main.yml`.

```yaml
- name: Ensure that ssh directory exists
  ansible.builtin.file:
    path: "{{ ansible_env.HOME }}/.ssh"
    state: directory
    mode: '700'

- name: Find SSH keys and config files
  ansible.builtin.find:
    paths: "{{ role_path }}/files"
    file_type: file
  register: found_files

- name: SSH Config
  ansible.builtin.copy:
    src: "{{ item.path }}"
    dest: "{{ ansible_env.HOME }}/.ssh/{{ item.path | basename }}"
    decrypt: true
    mode: '600'
  loop: "{{ found_files.files }}"
```

This will make sure that the `~/.ssh` directory exists. Find all the files we want to copy and copy them to the correct place. What is important here:

* `mode` needs correct permissions so our key will work correctly;
    
* `decrypt` - will copy decrypted files;
    

But how do we pass the password to the task?

It can be done with a password file.

## Run

If we have everything ready we can run this playbook.

First, install all required collections:

```bash
ansible-galaxy install -r requirements.yml
```

Now, to run the playbook:

```bash
ansible-playbook playbook.yml -i inventory.yml --vault-password-file=$HOME/.vault_pass
```

So now we have everything we wanted: installing apps, copying and symlinking files, creating directories, and decrypting secrets. Everything is set up in a way that we can use on multiple systems. If we use symlinks for configs and have some common configs between all systems, updating them in the repository will help us keep them on all machines.

If you would like to see my full setup: [https://github.com/Isur/dotfiles](https://github.com/Isur/dotfiles)