Hello USD - Part 15: Can a GitHub Action cache a OpenUSD Build?

July 22, 2023

This article is part of a series.

Continuing my validation scheme…

Is my code making the .usda files I think it should be? (XCTests) view
Does my code compile on Linux? view
- GitHub Action
- VM
[] Are my .usda files correct according to usdchecker?
- [] can I automate checking via swift
- [] could I even test that code on Linux with a GA <== WE ARE HERE

I’m skipping a step because I don’t want everything I’ve just learned about GitHub actions to dribble out my ear before I tackled an even harder problem with them.

TL;DR: https://github.com/carlynorama/GHActionsForOpenUSD/

Screenshot of a GitHub repo’s Action cache control panel with several caches including one for a python build.

Approach

As it turns out the folks managing the fabulous resource usg-assets are thinking about usd validation on GitHub themselves.

They have wisely chosen the path of linking their checker to the pipy usd-core python resource so they can avoid managing yet another build artifact.

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
          python-version: '3.7'

    - name: Install usd-core PyPI package
      run: pip install usd-core --user

I am not that wise.

I also have a different problem.

My Swift code won’t be running python, so a python library won’t help me.

I need a github runner with OpenUSD installed that can also run Swift 5.9 (eventually, fingers crossed). But I don’t want to run the 1hr+ build (on their hardware) every time. What a waste.

I could follow the route of guc, where the author builds his own binaries in a different repo and then goes and gets them when he needs them. Super valid.

I’m pretty clear for myself that I want more repo-by-repo flexibility, so I looked into GitHub’s caching action which, turns out, does exactly what I wanted it to do!

Note: I picked a manual-deletion caching strategy. The key is tied to the runner, and not the hash of any other resource (e.g. key: ${{ runner.os }}-${{ hashFiles('**/lockfiles') }}). Once my repo builds the cache, it won’t invalidate until I delete it by hand.

After many many false starts (GitHub Actions being WAY fiddlier than the total control I’m used to having) I got what I needed. I split the Clone & Build step into it’s own separate composite action so I wouldn’t have to if against the cache key every step of it.

Cached No-Python OpenUSD Build

└── .github
    └── workflows
    │   └── main.yml
    └── actions
        └── clone_and_build_no_python
            └── action.yml

##
## GitHub Actions File
## https://github.com/carlynorama/GHActionsForOpenUSD/
## Carlyn Maw 2023 Jul 21
##
## .github/workflows/main.yml


name: Build Linux

# When and on what to run
on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]
      
jobs:
  build:
    name: Build Linux
    runs-on: ubuntu-latest # Base machine
    steps:
      ## Get the files from the repo. 
      ## _Necessary_ because I call an external action in the repo.
      - name: Checkout This Repo
        uses: actions/checkout@v3
      ## updating apt-get, just good hygiene.
      - name: Update apt-get 
        run: sudo apt-get update
      ## ubuntu-latest didn't have OpenGL 
      - name: Set up packages (Linux)
        run: |
          sudo apt-get install -y libgl1-mesa-dev libglu1-mesa-dev          
      ## if the cache exists, recover it.
      - name: Restore OpenUSD (Repo and Build)
        id: cache-openusdnp-restore
        uses: actions/cache/restore@v3
        with:
          path: | # This is what I'm caching
            OpenUSD_REPO
            OpenUSD_BUILD_NOPYTHON
          key: ${{ runner.os }}-openusdnp
      ## If don't find the cache go clone and build.
      - name: Build Fresh No Python
        if: steps.cache-openusdnp-restore.outputs.cache-hit != 'true'
        uses: ./.github/actions/clone_and_build_no_python
      ## If do find the cache tell the environment variables.
      - name: Use Cached Build
        if: steps.cache-openusdnp-restore.outputs.cache-hit == 'true'
        run: |
          echo "USD_REPO_DIR=$PWD/OpenUSD_REPO" >> $GITHUB_ENV
          echo "USD_BUILD_DIR=$PWD/OpenUSD_BUILD_NOPYTHON" >> $GITHUB_ENV          
      ## Used for trouble shooting
      - name: Tell me about the Environnement
        run: |
          echo "-----"
          pwd  
          ls
          which python3
          python3 --version
          which python
          python --version
          echo $GITHUB_WORKSPACE
          echo $PATH
          echo $GITHUB_ENV
          echo .env
          echo "-----"          
      - name: Save OpenUSD Cache (Repo & Build)
        id: cache-openusdnp-save
        uses: actions/cache/save@v3
        with:
          path: |
            OpenUSD_REPO
            OpenUSD_BUILD_NOPYTHON            
          key: ${{ steps.cache-openusdnp-restore.outputs.cache-primary-key }}   
      ## Make sure it runs with the full path name
      - name: Confirms Runs
        run: |
            $USD_BUILD_DIR/bin/usdcat -h            
      ## Add to $GITHUB_PATH and $GITHUB_ENV/PYTHONPATH 
      ## Otherwise won't persist between steps. 
      ## Note: I did not add USD's location to the PATH of THIS step's shell. 
      ## Just the ones going forward.
      - name: Add OpenUSD Build to PATH and PYTHONPATH
        run: |
          export PYTHONPATH=$PYTHONPATH:$USD_BUILD_DIR/lib/python 
          echo "$USD_BUILD_DIR/bin" >> $GITHUB_PATH
          echo "PYTHONPATH=$PYTHONPATH" >> $GITHUB_ENV          
      ## Did PATH updates work.
      - name: Confirm in Path
        run: |
          usdcat -h

##
## GitHub Actions File
## https://github.com/carlynorama/GHActionsForOpenUSD/
## Carlyn Maw 2023 Jul 21
##
## .github/actions/clone_and_build_no_python/action.yml

name: Build Fresh No Python

runs:
  using: 'composite'
  steps:
    # Uses actions/checkout@v3 with a specified repo
    - name: Clone Pixar USD
      uses: actions/checkout@v3
      with:
        repository: PixarAnimationStudios/OpenUSD
        path: OpenUSD_REPO
    # Pass location of repo back out into calling script's GITHUB_ENV
    # Not perfect practice. It's a side effect rather than an output.
    - name: Add Repo Location to GITHUB_ENV
      run: |
        echo "USD_REPO_DIR=$PWD/OpenUSD_REPO" >> $GITHUB_ENV        
      shell: bash
    # Build and add the location to calling script's GITHUB_ENV
    - name: Build USD (no python)
      run: |
        python $USD_REPO_DIR/build_scripts/build_usd.py --no-python OpenUSD_BUILD_NOPYTHON
        echo "USD_BUILD_DIR=$PWD/OpenUSD_BUILD_NOPYTHON" >> $GITHUB_ENV        
      shell: bash
    # Did the build work? In this code I check that twice, 
    # both inside the subaction  and back out in the main. 
    - name: Confirms Runs
      run: |
          $USD_BUILD_DIR/bin/usdcat -h          
      shell: bash

Misc Lessons

EVERY STEP IS ITS OWN SHELL SESSION EVERY STEP IS ITS OWN SHELL SESSION EVERY STEP IS ITS OWN SHELL SESSION. Use $GITHUB_ENV (and TODO: inputs and outputs)
One can use shell scripts with GHA’s. Do more of that.
The error messages when an action fails can be misleading. For example, “Can’t find the file” doesn’t mean that it can’t actually find the file, it may be that the file is poorly formatted or the code lacks permission to use it.
VSCode’s GitHub Action Extension provides invaluable help.
When a GitHub provided action lets the programmer specify a path it’s a relative path under $GITHUB_WORKSPACE, which typically maps to the root folder of the checked out repo. This appears to require a literal string, i.e. cannot take a variable. (Jul 2023)
In a sub-action every shell based step needs to be told which shell to use.

Using calling sub-actions

A little primer on how to call an action in an external file depending on its location in or out of the calling repo.

on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]
      
jobs:
  build:
    name: Build Linux
    runs-on: ubuntu-latest
    steps:
      - name: Checkout This Repo
        uses: actions/checkout@v3
      - name: An actions folder in same repo
        uses: ./actions/hello_action
      - name: In .github/actions in same repo
        uses: ./.github/actions/hello_action
      - name: External repo
        uses: carlynorama/GHActionsForOpenUSD/actions/hello_action@main

Building With Python

If one instead needs the default build, which includes usdview, python packages need to be installed. That’s not my focus, but I provide the following example. This example cache’s pip but the time difference between cached and uncached python for this project seems to be < 1 second. Worth the learning, not worth the doing.

##
## GitHub Actions File
## https://github.com/carlynorama/GHActionsForOpenUSD/
##
## .github/workflows/cached_default_cachedPython.yml
## Carlyn Maw 2023 Jul 21
##


name: Run on Default Flavor (Cached Python)

on:
  push:
    branches: [ "default_build_test" ]
  pull_request:
    branches: [ "default_build_test" ]
      
jobs:
  build:
    name: Try Default (Cached Python)
    runs-on: ubuntu-latest
    steps:
      - name: Checkout This Repo
        uses: actions/checkout@v3
      - name: apt-get setup
        uses: carlynorama/GHActionsForOpenUSD/actions/apt_get_update_install@main
      ## Python setup script seems to have difficulty with foreign repo call
      ## Note that this one refers to an action folder in the repo.
      - name: Setup Python
        uses: actions/setup-python@v4
        with:
          # 3.10 is default
          python-version: '3.10'  
          cache: 'pip' # caching pip dependencies
          # Caching, by default, pins itself to changes in requirements.txt
          # I'm testing what overriding that means
          cache-dependency-path: actions/python_cached_setup/requirements.txt
      - name: Upgrade
        shell: bash
        run: |
          python -m pip install --upgrade pip          
      - name: Install
        shell: bash
        run: |
          pip install install -r actions/python_cached_setup/requirements.txt          
      - name: Restore OpenUSD (Repo and Build)
        id: cache-openusddf-restore
        uses: actions/cache/restore@v3
        with:
          path: |
            OpenUSD_REPO
            OpenUSD_BUILD_DEFAULT            
          key: ${{ runner.os }}-openusddf
      - name: Build Fresh
        if: steps.cache-openusddf-restore.outputs.cache-hit != 'true'
        uses: carlynorama/GHActionsForOpenUSD/actions/clone_and_build_default@main
      - name: Use Cached Build
        if: steps.cache-openusddf-restore.outputs.cache-hit == 'true'
        run: |
          echo "USD_REPO_DIR=$PWD/OpenUSD_REPO" >> $GITHUB_ENV
          echo "USD_BUILD_DIR=$PWD/OpenUSD_BUILD_DEFAULT" >> $GITHUB_ENV          
      - name: Save OpenUSD Cache (Repo & Build)
        id: cache-openusddf-save
        uses: actions/cache/save@v3
        with:
          path: |
            OpenUSD_REPO
            OpenUSD_BUILD_DEFAULT            
          key: ${{ steps.cache-openusddf-restore.outputs.cache-primary-key }}   
      - name: Confirms Runs
        run: |
            $USD_BUILD_DIR/bin/usdcat -h            
      - name: Add OpenUSD Build to PATH and PYTHONPATH
        run: |
          export PYTHONPATH=$PYTHONPATH:$USD_BUILD_DIR/lib/python 
          echo "$USD_BUILD_DIR/bin" >> $GITHUB_PATH
          echo "PYTHONPATH=$PYTHONPATH" >> $GITHUB_ENV          
      - name: Confirm in Path
        run: |
          usdcat -h

References

GitHub Docs

Runner Image Info

Example Actions, Dockerfiles, Workflows, StackOverflow answers

Handy Snippet

          echo "-----"
          pwd  
          ls
          which python3
          python3 --version
          which python
          python --version
          echo $GITHUB_WORKSPACE
          echo $PATH
          echo $GITHUB_PATH
          echo $GITHUB_ENV
          echo $PYTHONPATH
          echo "-----"
          $USD_BUILD_DIR/bin/usdcat -h
          usdcat -h