Packaging proprietary software for Debian using Ansible

• See Role on GitHub

Managing software installations for an animation studio can be a time-consuming process, especially if relying on the provided installation methods, and even more so when using a distribution that isn’t officially supported by the software.

I needed a streamlined workflow that allowed me to download DCCs from their creators, package them into *.deb files and deploy them to a local server for eventual installation on workstations. This will allow me to properly version control the versions of software from a system level, and to rapidly build DCC-backed docker containers without needing to run the lengthy install processes. I achieved this in the form of an Ansible role.

The Role

      packages:
      - name: houdini
        version: 20.0.625
        release: 1
        category: graphics
        architecture: amd64
        author: "WhoAmI? <someone@goes.here>"
        description: Houdini 20
        sources:
          - name: base
            s3:
              bucket: MyLocalPrivateS3Bucket
              object: "houdini/houdini-20.0.625-linux_x86_64_gcc11.2.tar.gz"
            path: /houdini.tar.gz
            archive: yes
        scripts:
          pre_build:
            - "{{ lookup('first_found', 'package_houdini.yaml')}}"
        repo:
          method: s3
          bucket: MyLocalPrivateS3Bucket
          object_prefix: "houdini"

---
- include_tasks: build.yml
  loop: "{{ packages }}"
  loop_control:
    loop_var: package

I want to execute an entire block in iteration, once per package to build. I want to use a block so that I can make use of the rescue: clause for cleanup in the event of a packaging failure, and to make use of the always: clause to release the package to a local repository.

In order to achieve this, I need to put the block in a separate task and iterate on the include_tasks task as the block task doesn’t accept a loop: argument. I’ve also named the iteration variable to package as I will be running iterations within.

- block:
  - name: create build directory
    file:
      path: /tmp/build
      state: directory

Getting the installer archive

From a HTTP source

- name: download archive
    ansible.builtin.get_url:
      url: "{{ source.url }}"
      dest: "{{ source.path }}"
      mode: '0440'
    loop: "{{ package.sources }}"
    loop_control:
      loop_var: source
    when: "source.url is defined"

If I download the package to an on-prem server (to perform a virus scan), I may choose to deliver the files in the form of a HTTP Server.

From S3

  - name: download from s3 bucket
    amazon.aws.s3_object:
      bucket: "{{ source.s3.bucket }}"
      object: "{{ source.s3.object }}"
      dest: "{{ source.path }}"
      mode: get
    loop: "{{ package.sources }}"
    loop_control:
      loop_var: source
    when: "source.s3 is defined"

If I’m working with a cloud-based studio, I may opt in to store the archive on S3 for cheaper expenses, especially if I tend to get multiple errors and need to debug the playbook (as bringing data into AWS is a cost)

From the Ansible controller

  - name: copy archives
    copy:
      src: "{{ source.path }}"
      dest: "{{ source.path }}"
    loop: "{{ package.sources }}"
    loop_control:
      loop_var: source
    when:
      - "source.archive is not defined"
      - "source.url is not defined"

If it’s my first time and I’m on-premises, I may choose to deliver the package straight from my machine to a build machine

From network storage attached to the remote machine

Or if I decide to store the archive on the network storage that’s attached to a build machine, I can choose to pull the file from the network directly without the need to copy.

Getting to the Application

  - name: extract archives
    unarchive:
      src: "{{ source.path }}"
      dest: /tmp/build.src
      remote_src: yes
    loop: "{{ package.sources }}"
    loop_control:
      loop_var: source
    when: "source.archive is defined"

Should I need to extract an archive, I provide an archive: yes attribute onto the package in iteration. The yes isn’t relevant, but the fact that the attribute exists is enough for Ansible to trigger this task based on the when: clause.

  - name: copy archives
    copy:
      src: "{{ source.path }}"
      dest: /tmp/build.src
    loop: "{{ package.sources }}"
    loop_control:
      loop_var: source
    when: "source.archive is not defined"

If we have the other case of not needing to extract anything, we can just copy the source over to the building directory.

  - name: prepare package
    include_tasks: "{{ item }}"
    loop: "{{ package.scripts.pre_build }}"
    vars:
      build_prefix: /tmp/build

Finally, we need to layout the contents in order for dpkg to create the package correctly. The packaging process requires a directory that will act as the “target root directory”.

If I have the file /tmp/build.src/test and build the directory /tmp/build.src, I would get a build.src.deb file that will create a /test file once installed.

Following this logic, I need to install the application I want into /tmp/build.src as if it were the root filesystem.

Since every application is different, I implemented the concept of pre_build scripts, in which this role itself will handle the pulling and releasing of packages, the preparation and destroy of the build directories and the templates and the execution for the packaging system. What it doesn’t handle is how to get the contents of the application itself.

package.scripts.pre_build points to a list of tasks to run to prepare package for packaging. This should not be specific to any distribution. As an example, for SideFX® Houdini, my pre_build is the following:

- name: find houdini installers
  ansible.builtin.find:
    paths: /tmp/build.src
    patterns: 'houdini.install'
    recurse: yes
  register: found_houdini_installers

- name: install houdini to directory
  shell: "{{ item.path }} \
    --make-dir \
    --auto-install \
    --no-root-check \
    /tmp/build/opt/hfsXX.X"
  loop: "{{ found_houdini_installers.files }}"

Where I use the find module to search for any Houdini installers (since it may be nested, I use the recurse: yes variable), and then run the Houdini installer as if I were at the machine.

There are some additional flags that I had removed from the snippet to make the installation automatic (--acceptEULA) and selective (--no-install-license-server). The latter I plan to package into their own individual packages (e.g: houdini-sesinetd, maya2024-houdini-engine etc.) but haven’t got round to it.

  - include_tasks: debian.yml
    when: ansible_os_family == 'Debian'

Finally, since I’m packaging for Debian, I place the procedures for building a Debian package into a debian.yml file. I plan to extend the packaging role to include Arch Linux and Fedora-based in the future.

- name: create debian directory
  file:
    path: /tmp/build/DEBIAN
    state: directory

- name: render debian control template
  template:
    src: debian.control.j2
    dest: /tmp/build/DEBIAN/control

At the “root” level, we require a DEBIAN directory, with a control file inside. This is the minimum to get a deb package. I’m making use of a Jinja2 template for the control file as it’s cleaner to manage down the road.

DEBIAN/control

Package: {{ package.name }}
Version: {{ package.version }}-{{ package.release }}
Section: {{ package.category }}
Priority: optional
Architecture: {{ package.architecture }}
Maintainer: {{ package.author }}
Description: {{ package.description }}

Building the package

- name: execute package build
  shell: "dpkg --build /tmp/build /tmp/{{ package.name }}-{{ package.version }}-{{ package.release }}-{{ package.architecture }}.deb"

- set_fact:
    local_package_location: "/tmp/{{ package.name }}-{{ package.version }}-{{ package.release }}-{{ package.architecture }}.deb"

I set the name of the result *.deb to follow a strict naming convention to avoid file conflicts. Lastly, once the package is built, I use set_fact to return the path to the *.deb file from the sub-task so that the build.yml file can deploy it to where it needs to go, be it S3 or a local repository. I do this as I may be building more than a debian package in the future.