Establishing a VPN Connection

To begin, I set up a VPN connection from my OpenWRT router to the cloud provider using WireGuard. I created two VPCs in the cloud provider – one public and one private – to mimic the “WAN-LAN” scenario of at-home routers.

This setup provides isolation similar to a home network, where the resources on the private network can only be access by other resources on the same network, but they are also able to communicate with the outside world.

The intention is to have the private network as an extension to my “home” (at any given time).

Deploying a Cloud Router

I deployed a virtual machine that will act as a router spanning both networks. This needs to be across both networks as I need an endpoint to connect to (which requires an internet-exposed network) while still being able to access private resources.

I chose VyOS as the cloud router’s operating system because it is configuration-driven, allowing for an Infrastructure-as-Code (IaC) approach for easy re-deployment on any cloud provider.

Utilizing Object Storage for Plex Media Server

I adopted object storage to take advantage of the “unlimited” data offered by the cloud provider, and configured s3fs to mount the object storage on a specific node. With this, Plex can access data directly from the object storage bucket without many configuration changes or plugins to Plex.

The VPN connection allows me to access the Plex server securely as if it were local on both my PS5 and laptop. This setup ensures that the Plex interface remains non-accessible to the public and bypasses the bandwidth limit when proxying via the official Plex servers.

Securely Pushing Metrics from In-House Devices

By using the VPN connection, I can push metrics from my in-house devices directly, such as weather sensors without exposing my Prometheus instance to the public internet.

The VPN’s security layer wraps around all traffic, eliminating the need for implementing a CA chain for Prometheus when using platforms such as AWS IoT or Grafana Cloud (where devices are expected to communicate with a public HTTPS endpoint)

Automating At-Home Devices with HomeAssistant

I use HomeAssistant within the cloud provider to automate my at-home devices without worrying about downtime or maintaining a device inside my home. HomeAssistant is scriptable, easily re-deployable, and can bridge a wide range of IoT devices under a single platform, such as HomeKit and Hue.

I can now utilize my old infrastructure without worrying about maintaining hardware, and plan to deploy many services to the private cloud. Keep an eye out for a deeper breakdown on how I deployed and configured each element of my private cloud

I need to build images based off AMD64 architecture while working from an ARM64 machine. While this is possible directly using the qemu-system-x86_64 binary, it tends to be extremely slow due to the overhead of converting for the ARM architecture.

Workbench

• Ubuntu 22.04 LTS with libvirt installed
• MacBook Pro M2 with the Packer build files

Configuring the Libvirt Plugin

Connecting to the libvirt host

When using the libvirt plugin, I need to provide a Libvirt URI.

source "libvirt" "image" {
    libvirt_uri = "qemu+ssh://${var.user}@${var.host}/session?keyfile=${var.keyfile}&no_verify=1"
}

• qemu+ssh:// denotes that I’ll be using the QEMU / KVM Backend and connecting via SSH. The connection method denotes the rest of the arguments of the string
• ${var.user}@${var.host} is in the SSH syntax, this is the username and hostname of the machine that is running libvirt
• /session is to isolate the running builds from those on the system level. /system would work just as well.
• keyfile=${var.keyfile} is used to automatically authenticate to the remote machine without the need of a password. This is useful in the future when I automatically trigger the packer build from a Git repository
• no_verify=1 is added so that I can throw the build at any machine and have it “just work”. This is usually guided against due to spoofing attacks.

Communicating with the libvirt guest

communicator {
    communicator                 = "ssh"
    ssh_username                 = var.username
    ssh_bastion_host             = var.host
    ssh_bastion_username         = var.user
    ssh_bastion_private_key_file = var.private_key
  }

• The difference between ssh_* and ssh_bastion_* is that the first refers to the target virtual machine being built, and the latter refers to the “middle-man” machine.
  • I require this as I don’t plan to expose the VM to a network outside of the machine hosting it.
  • Since I won’t have access from my local workstation, I need to communicate with the virtual machine via the machine that is hosting it.
  • By adding ssh_bastion_* arguments, I’m telling packer that in-order to communicate with the VM, it needs to access the bastion machine first then execute all SSH commands through it.

Configuring the libvirt daemon

• Libvirt Volume Upload Documentation

My Observations

I came across a “Permission Denied” error when attempting to upload an existing image (in my case, the KVM Ubuntu Server Image). This was due to AppArmor not being provided a trust rule upon creation of the domain. This error is first visible in the following form directly from Packer:

==> libvirt.example: DomainCreate.RPC: internal error: process exited while connecting to monitor: 2024-05-24T16:41:42.574660Z qemu-system-x86_64: -blockdev {"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage","backing":null}: Could not open '/var/lib/libvirt/images/packer-cp8c6ap1ijp2kss08iv0-ua-artifact': Permission denied

At first, I assumed that there was an obvious permissions problem, and at first glace there was in-fact that. When looking at this file upon creation, it had root permissions where only the root user can read/write.

# ls -lah /var/lib/libvirt/images
-rw------- 1 root root  925M May 24 16:41 packer-cp8c6ap1ijp2kss08iv0-ua-artifact

This makes sense since libvirtd is running under the root user, which is the default configuration from the Ubuntu repository. I didn’t see any configuration option to manipulate what the permissions should be after an upload with libvirt either. This was an assumed problem since all QEMU instances are running under a non-root user, libvirt-qemu

# ps -aux | grep libvirtd
# ps -aux | grep qemu

root      145945  0.4  0.1 1778340 28760 ?       Ssl  16:43   0:10 /usr/sbin/libvirtd
libvirt+    3312  2.2 11.1 4473856 1817572 ?     Sl   May12 405:19 /usr/bin/qemu-system-x86_64

My second observation was that all images created directly within libvirt (e.g: with virt-manager) had what looked like “correct” permissions, those that matched the user that QEMU would eventually run under;

# ls -lah /var/lib/libvirt/images
-rw-r--r-- 1 libvirt-qemu kvm   11G May 24 17:11 haos_ova-11.1.qcow2

Since no-one else had reported this particular issue when using the libvirt plugin, I had gone down the route of PEBKAC.

Allowing packer-uploaded images as backing store

Thanks to this discussion on Stack Overflow, I found that AppArmor had been blocking the request to the specific file in question.

# dmesg -w
[1081541.249157] audit: type=1400 audit(1716568577.970:119): apparmor="DENIED" operation="open" profile="libvirt-25106acc-cfd8-40f7-a7c6-f5c1c63bc16c" name="/var/lib/libvirt/images/packer-cp8c6ap1ijp2kss08iv0-ua-artifact" pid=43927 comm="qemu-system-x86" requested_mask="w" denied_mask="w" fsuid=64055 ouid=64055

Here, I can see that AppArmor is doing three things;

• Denying an open request to the QEMU Image
  • apparmor="DENIED"
  • operation="open"
• Denying writing to the QEMU Image
  • denied_mask="w"
• Using a profile that is specific to the domain being launched
  • profile="libvirt-25106acc-cfd8-40f7-a7c6-f5c1c63bc16c"
  • This is achieved because libvirt will automatically push AppArmor rules upon creation of a domain. This also means that libvirt will be using some form of template file or specification to create rules.

This means that I need to find the template file that libvirt is using to design the rules, and allow for writing to packer-uploaded QEMU Images.

# /etc/apparmor.d/libvirt/TEMPLATE.qemu
# This profile is for the domain whose UUID matches this file.
# 

#include <tunables/global>

profile LIBVIRT_TEMPLATE flags=(attach_disconnected) {
  #include <abstractions/libvirt-qemu>
  /var/lib/libvirt/images/packer-** rwk,
}

As mentioned in the Stack Overflow post, simply adding /var/lib/libvirt/images/packer-** rwk, to the template file is enough to get past this issue.

End Result

By bringing everything together, I get a successful QCOW2 image visible in my default storage pool. I’m using the Ansible provisioner within the build block so that I can keep the execution steps separate from the Packer build script, and re-usable across different cloud providers.

Managing software installations for an animation studio can be a time-consuming process, especially if relying on the provided installation methods, and even more so when using a distribution that isn’t officially supported by the software.

I needed a streamlined workflow that allowed me to download DCCs from their creators, package them into *.deb files and deploy them to a local server for eventual installation on workstations. This will allow me to properly version control the versions of software from a system level, and to rapidly build DCC-backed docker containers without needing to run the lengthy install processes. I achieved this in the form of an Ansible role.

The Role

      packages:
      - name: houdini
        version: 20.0.625
        release: 1
        category: graphics
        architecture: amd64
        author: "WhoAmI? <someone@goes.here>"
        description: Houdini 20
        sources:
          - name: base
            s3:
              bucket: MyLocalPrivateS3Bucket
              object: "houdini/houdini-20.0.625-linux_x86_64_gcc11.2.tar.gz"
            path: /houdini.tar.gz
            archive: yes
        scripts:
          pre_build:
            - "{{ lookup('first_found', 'package_houdini.yaml')}}"
        repo:
          method: s3
          bucket: MyLocalPrivateS3Bucket
          object_prefix: "houdini"

---
- include_tasks: build.yml
  loop: "{{ packages }}"
  loop_control:
    loop_var: package

I want to execute an entire block in iteration, once per package to build. I want to use a block so that I can make use of the rescue: clause for cleanup in the event of a packaging failure, and to make use of the always: clause to release the package to a local repository.

In order to achieve this, I need to put the block in a separate task and iterate on the include_tasks task as the block task doesn’t accept a loop: argument. I’ve also named the iteration variable to package as I will be running iterations within.

- block:
  - name: create build directory
    file:
      path: /tmp/build
      state: directory

Getting the installer archive

From a HTTP source

- name: download archive
    ansible.builtin.get_url:
      url: "{{ source.url }}"
      dest: "{{ source.path }}"
      mode: '0440'
    loop: "{{ package.sources }}"
    loop_control:
      loop_var: source
    when: "source.url is defined"

If I download the package to an on-prem server (to perform a virus scan), I may choose to deliver the files in the form of a HTTP Server.

From S3

  - name: download from s3 bucket
    amazon.aws.s3_object:
      bucket: "{{ source.s3.bucket }}"
      object: "{{ source.s3.object }}"
      dest: "{{ source.path }}"
      mode: get
    loop: "{{ package.sources }}"
    loop_control:
      loop_var: source
    when: "source.s3 is defined"

If I’m working with a cloud-based studio, I may opt in to store the archive on S3 for cheaper expenses, especially if I tend to get multiple errors and need to debug the playbook (as bringing data into AWS is a cost)

From the Ansible controller

  - name: copy archives
    copy:
      src: "{{ source.path }}"
      dest: "{{ source.path }}"
    loop: "{{ package.sources }}"
    loop_control:
      loop_var: source
    when:
      - "source.archive is not defined"
      - "source.url is not defined"

If it’s my first time and I’m on-premises, I may choose to deliver the package straight from my machine to a build machine

From network storage attached to the remote machine

Or if I decide to store the archive on the network storage that’s attached to a build machine, I can choose to pull the file from the network directly without the need to copy.

Getting to the Application

  - name: extract archives
    unarchive:
      src: "{{ source.path }}"
      dest: /tmp/build.src
      remote_src: yes
    loop: "{{ package.sources }}"
    loop_control:
      loop_var: source
    when: "source.archive is defined"

Should I need to extract an archive, I provide an archive: yes attribute onto the package in iteration. The yes isn’t relevant, but the fact that the attribute exists is enough for Ansible to trigger this task based on the when: clause.

  - name: copy archives
    copy:
      src: "{{ source.path }}"
      dest: /tmp/build.src
    loop: "{{ package.sources }}"
    loop_control:
      loop_var: source
    when: "source.archive is not defined"

If we have the other case of not needing to extract anything, we can just copy the source over to the building directory.

  - name: prepare package
    include_tasks: "{{ item }}"
    loop: "{{ package.scripts.pre_build }}"
    vars:
      build_prefix: /tmp/build

Finally, we need to layout the contents in order for dpkg to create the package correctly. The packaging process requires a directory that will act as the “target root directory”.

If I have the file /tmp/build.src/test and build the directory /tmp/build.src, I would get a build.src.deb file that will create a /test file once installed.

Following this logic, I need to install the application I want into /tmp/build.src as if it were the root filesystem.

Since every application is different, I implemented the concept of pre_build scripts, in which this role itself will handle the pulling and releasing of packages, the preparation and destroy of the build directories and the templates and the execution for the packaging system. What it doesn’t handle is how to get the contents of the application itself.

package.scripts.pre_build points to a list of tasks to run to prepare package for packaging. This should not be specific to any distribution. As an example, for SideFX® Houdini, my pre_build is the following:

- name: find houdini installers
  ansible.builtin.find:
    paths: /tmp/build.src
    patterns: 'houdini.install'
    recurse: yes
  register: found_houdini_installers

- name: install houdini to directory
  shell: "{{ item.path }} \
    --make-dir \
    --auto-install \
    --no-root-check \
    /tmp/build/opt/hfsXX.X"
  loop: "{{ found_houdini_installers.files }}"

Where I use the find module to search for any Houdini installers (since it may be nested, I use the recurse: yes variable), and then run the Houdini installer as if I were at the machine.

There are some additional flags that I had removed from the snippet to make the installation automatic (--acceptEULA) and selective (--no-install-license-server). The latter I plan to package into their own individual packages (e.g: houdini-sesinetd, maya2024-houdini-engine etc.) but haven’t got round to it.

  - include_tasks: debian.yml
    when: ansible_os_family == 'Debian'

Finally, since I’m packaging for Debian, I place the procedures for building a Debian package into a debian.yml file. I plan to extend the packaging role to include Arch Linux and Fedora-based in the future.

- name: create debian directory
  file:
    path: /tmp/build/DEBIAN
    state: directory

- name: render debian control template
  template:
    src: debian.control.j2
    dest: /tmp/build/DEBIAN/control

At the “root” level, we require a DEBIAN directory, with a control file inside. This is the minimum to get a deb package. I’m making use of a Jinja2 template for the control file as it’s cleaner to manage down the road.

DEBIAN/control

Package: {{ package.name }}
Version: {{ package.version }}-{{ package.release }}
Section: {{ package.category }}
Priority: optional
Architecture: {{ package.architecture }}
Maintainer: {{ package.author }}
Description: {{ package.description }}

Building the package

- name: execute package build
  shell: "dpkg --build /tmp/build /tmp/{{ package.name }}-{{ package.version }}-{{ package.release }}-{{ package.architecture }}.deb"

- set_fact:
    local_package_location: "/tmp/{{ package.name }}-{{ package.version }}-{{ package.release }}-{{ package.architecture }}.deb"

I set the name of the result *.deb to follow a strict naming convention to avoid file conflicts. Lastly, once the package is built, I use set_fact to return the path to the *.deb file from the sub-task so that the build.yml file can deploy it to where it needs to go, be it S3 or a local repository. I do this as I may be building more than a debian package in the future.

Why?

I’m diving into Terraform as part of my adventure into the DevOps world, which I’ve adopted an interest in the past few months.

• I use 2 workstations with DigitalOcean
  • MacBook; for when I’m out and about
  • ArchLinux; for when I’m at home

Generating the API Tokens

Under API, located within the dashboards’ menu (on the left-hand side), I’m presented with the option to Generate New Token.

Followed by an interface to define;

• Name
  • I typically name this token as zai.dev or personal, as this token will be shared across my devices. While this approach isn’t the most secure (Ideally, I should have one token per machine), I’m going for the matter of convenience of having one token for my user profile.
• Expiry date
  • Since I’m sharing the token across workstations (including my laptop, which may be prone to theft), I set the expiration to the lowest possible value of 30 days.
• Write permissions
  • Since I’ll be using Terraform, and it’s main purpose is to ‘sculpt’ infrastructure, I require the token that it’ll use to connect to DigitalOcean to have write permissions.

Authenticating DigitalOcean Spaces

As the Terraform Provider allows the creation of Spaces, DigitalOceans’ equivalent to AWS’ S3-bucket, I should also create tokens for it. By navigating to the “Spaces Keys” tab under the APIs option, I can repeat the same steps as above

Installing the Tokens

Continuing from the setup of environment variables in my Synchronizing environment variables across Workstations post, I need to add 3 environment variables for connecting to DigitalOcean.

• DIGITALOCEAN_TOKEN
  • This is the value that is given to you after hitting “Generate Token” on the Tokens tab
• SPACES_ACCESS_KEY_ID
  • This is the value that is given to you after hitting “Generate Token” on the Spaces Tokens tab
• SPACES_SECRET_ACCESS_KEY
  • This is the one-time value that is given to you alongside the SPACES_ACCESS_KEY_ID value

Whilst I’m at it, I’m going to add the following environment variables so that I can use any S3-compliant tools to communicate with my object storage, such as the s3 copy command to push build artifacts

• AWS_ACCESS_KEY_ID=${SPACES_ACCESS_KEY_ID}
• AWS_SECRET_ACCESS_KEY=${SPACES_SECRET_ACCESS_KEY}

To keep things tidy, I created a separate environment file for digital ocean, under ~/.config/zai/env/digitalocean.sh

export DIGITALOCEAN_TOKEN="<DO_TOKEN>"
export SPACES_ACCESS_KEY_ID="<SPACES_KEY>"
export SPACES_SECRET_ACCESS_KEY="<SPACES_SECRET>"
export AWS_ACCESS_KEY_ID=${SPACES_ACCESS_KEY_ID}
export AWS_SECRET_ACCESS_KEY=${SPACES_SECRET_ACCESS_KEY}

What’s my situation?

• I use at least two workstations
  • MacBook Pro; for use when out and about
  • ArchLinux Desktop; for use when at home
  • Ubuntu Server; for hosting permanent services

What does this mean?

As I’m working across two devices, I need to make sure that the equivalent configuration is available across both devices and immediately. I use SyncThing as the technology to keep my personal configuration such as environment variables synchronized across all devices. I don’t use Git as there is an extra step of manually pulling down the configuration each time, in addition to as not having access to my local git repository at all times.

Mac & Linux are UNIX-based/like platforms, so I can keep my configuration files uniform. I use Bash scripts to define the environment variables needed for any APIs that I use.

How did I achieve it?

Directory structure & files needed

I use ~/.config/zai as my configuration directory and set SyncThing to watch it, and then set it on the other workstations to point to the same path. A file named rc.sh lives inside here centralize anything I want upon loading the terminal.

Installing SyncThing

Installing on Linux

Luckily for most Linux distributions, SyncThing is already provided in the pre-installed repositories.

pacman -S syncthing # Arch Linux
apt install syncthing # Debian / Ubuntu

# Enable & Start Syncthing
systemctl enable --now syncthing@<username>

Installing on macOS

On Mac it’s slightly more trivial, but the instructions are provided within the Downloadable ZIP File for macOS.

Sourcing the rc.sh from the shell

The following snippet needs to be placed in a shell initialization script which may differ depending on platform. The source command tells Bash to reference (and execute) the file that follows it

source ~/.config/zai/rc.sh

macOS

macOS will execute the ~/.bash_profile script upon opening a new Bash shell. I switch between zsh and bash from time to time, so either I manually execute /usr/bin/bash to take me to the Bash environment, or I’d just change the default shell under the Terminal properties.

Linux

Most linux platforms will execute ~/.bashrc upon opening a new shell, assuming that Bash is the default shell.

rc.sh

I keep this file simple, which is to loop through all the files inside the env/ subdirectory for bash files and execute them. This allows me to not have a single file with numerous lines.

for file in ~/.config/zai/env/*.sh; do
    source $file;
done

What’s next?

I’m diving into the world of DevOps, and will need to configure my local systems to;

• Hold the API Credentials for the cloud service(s) of my choice
• Hold the API Credentials for an S3 bucket location of my choice