Hello USD - Part 4: Python setup

UPDATE - Most of what’s in this post ended up being educational, but a bad path. Actually working instructions instead of the travelogue: https://github.com/carlynorama/USDHelloWorld/blob/main/SETUP.md

A post in 5 Acts.

Knowledge Prereq

This post assumes significant understanding about the command line, PATH, PYTHONPATH, editing shell profiles and the vagaries of managing python environments.

Echoing yesterday all my USD builds live in a local user folder together with the pattern ~/opb/$SOME_BUILD_NAME none are currently added to the global path as this post will explore. opb stands for “off path bin” and it’s personal style, not anything official.

Software & Hardware

Act 1: Same set up as yesterday.

Acts 2 & 3 successfully executed on:

Act 4&5:

Same as Act 1, Act 5 is essentially Act 1 run on a similarly configured M1

Act 1: Doh… Well that didn’t work.

What happens if you try to do one of the Pixar python based tutorials after running the default build script from the Pixar repo on a system using Python 3.11?

Doing the default build is similar to yesterday’s install of just the C++ but there are some prerequisites. usdview requires PyOpenGL and PySide6. If you don’t want usdview, leave out installing them and add the --no-usdview to the build (shown in Act 3)

# if you care about isolating your system python and your playtime python and haven't already
# brew install python3 
# rehash
# which pip3 #to confirm using desired versions
pip3 install PyOpenGL
# pip3 install PySide6 # <- Bug in USD 23.05 PySide6 >= 6.5.0 
pip3 install PySide6==6.4.3 # <- UPDATED see epilogue
cd $DIRECTORY_WITH_CLONED_REPO
python3 USD/build_scripts/build_usd.py ~/opb/USDefault

After the build completes:

cd $DIRECTORY_WITH_FILES_TO_CONVERT #See yesterday's post
~/opb/USDefault/bin/usdcat -o threeBodyExample_fromArchive.usda ThreeBodyUSDZ.usdz

All works just fine.

Moving on to trying to try Hello World again

export PATH=$PATH:$HOME/opb/USDefault/bin;
export PYTHONPATH=$PYTHONPATH:$HOME/opb/USDefault/lib/python
python3

In REPL:

from pxr import Usd, UsdGeom

produces the following error

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/$USER/opb/USDefault/lib/python/pxr/Usd/__init__.py", line 25, in <module>
    Tf.PreparePythonModule()
  File "/Users/$USER/opb/USDefault/lib/python/pxr/Tf/__init__.py", line 88, in PreparePythonModule
    module = importlib.import_module(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
SystemError: type Boost.Python.enum has the Py_TPFLAGS_HAVE_GC flag but has no traverse function

At this point I walked away because I didn’t have the expectation that the Python scripts would with work this version of Python anyway. 3.11 is way different than the 3.7.7 Pixar mention on their build machine description page. Perusing the Python 3.11 docs on importing alone shows a myriad of changes in 3.10.

UPDATE: See Act 5

Act 2: USDZ Tools Hello World

UPDATE: The USDZ scrips work better when run using a fresh build of the Pixar tools in the path instead and then running the script with the preferred python explicitly: e.g. python3 /Applications/usdpython/usdzconvert/usdARKitChecker -v three_d_thing.usda

Step 1: Go Get Python 3.7 the hard way

The Apple USDZ Tools are comprised of the USD libraries (version 22.03), plus some other helper scripts pre-compiled using Python 3.7.9

So 3.7.9 it is.

Needing to manage multiple python environments happens all the time… HOWEVER

Why doesn’t pyenv work with the USDZ Tools? Since the Apple tools are pre-compiled, the environment needs to perfectly match the one it was built for and the build appears to have a dependency directly to /Library/Frameworks/Python.framework/Versions/3.7/Python, (thanks otool -L /Applications/usdpython/USD/lib/python/pxr/Tf/_tf.so for helping make sense of the error message)

So python.org download we go. https://www.python.org/downloads/release/python-379/ Python.org does not recommend or support 3.7 anymore, but there remains an installer on the page (20230629).

After running the installer, find the Python 3.7 folder in /Applications and run the Install Certificates.command file.

The installer seems to also update .zprofile (or respective shell profile) to contain the below lines. If it doesn’t, run the Update Shell Profile.command to do so. WARNING: This will now set python 3.7.9 as the default for EVERY shell window because it puts 3.7 at the front of the line. Fixed in step 4, but at this point I went with it.

PATH="/Library/Frameworks/Python.framework/Versions/3.7/bin:${PATH}"
export PATH

Step 2: Go get the precompiled files from Apple

As it turns out USDZ Tools provides a handy installer (20230629 - version 0.66, from 2019, Compiled for python 3.7.9, using USD version 22.03)

Step 3: Run them using their shell launcher

After the install, as recommended in the installer docs (DO NOT SKIP READING THEM), I double clicked on the USD.command file in /Applications/usdpython

At this point python3 --version did show 3.7.9 and echo $PYTHONPATH did include /Applications/usdpython/USD/lib/python

python3 /Applications/usdpython/samples/101_scenegraph.py produced

#usda 1.0
(
    defaultPrim = "scenegraph"
    upAxis = "Y"
)

def Xform "scenegraph" (
    assetInfo = {
        asset identifier = @scenegraph.usd@
        string name = "scenegraph"
    }
    kind = "component"
)
{
    def Scope "Geom"
    {
        def Cube "cube"
        {
        }

        def Sphere "sphere"
        {
            double3 xformOp:translate = (4, 0, 0)
            uniform token[] xformOpOrder = ["xformOp:translate"]
        }
    }
}

I had a little dance party in my soul after seeing this. It had been multiple hours over multiple days to get to this point. I’ve glossed over a number of failed builds. On the other hand I was also pretty annoyed being stuck having 3.7.9 as my main version of Python on the machine I wanted to do any USD python work on.

Step 4: Corral 3.7.9 into just the shell it’s wanted for

I brute forced returning to python 3.11 (brew install python, brew reinstall python@3.11, brew link --overwrite python@3.11). I was tired. It was late. That may not have been necessary. I also removed

PATH="/Library/Frameworks/Python.framework/Versions/3.7/bin:${PATH}"
export PATH

from .zprofile and added it to the top of /Applications/usdpython/USD.command (sudo nano /Applications/usdpython/USD.command because we aren’t in a user editable folder) and launched it again.

USD.command now looked like:

#!/bin/sh

# ADDED uncomment below for error checking
# env 

##### ADDED TO WORK WITH PYTHON 3.7
PATH="/Library/Frameworks/Python.framework/Versions/3.7/bin:${PATH}"
export PATH
##################################

BASEPATH=$(dirname "$0")
#Note: The double path is in provided USD.command file, I left it alone. 
export PATH=$PATH:$BASEPATH/USD:$PATH:$BASEPATH/usdzconvert;
export PYTHONPATH=$PYTHONPATH:$BASEPATH/USD/lib/python

# uncomment to set the PYTHONPATH to FBX Bindings here:
# export PYTHONPATH=$PYTHONPATH:"/Applications/Autodesk/FBX Python SDK/2020.2.1/lib/Python37_x64"

if [[ $PYTHONPATH == *"FBX"* ]]; then
    :
else 
    echo "For FBX support, edit PYTHONPATH in this file (USD.command) or your shell configuration file"
fi

$SHELL

python3 /Applications/usdpython/samples/101_scenegraph.py worked again! And I can still use Python 3.11 as my default install in other shells. Dance party for real.

ACT 3: Going back to the Pure Pixar Library

Launching a terminal using the USD.command file to take advantage of the quick load of python 3.7.9, I ran:

cd $DIR_WITH_CLONED_REPO
python3 USD/build_scripts/build_usd.py --no-usdview ~/opb/USD379NoView

Note: This version of the install requires no extra python dependencies, because it DOES NOT include usdview.

I had gotten to something like this point before, but now I:

The updated USD_pixar_noview.command:

#!/bin/sh

# ADDED uncomment below for error checking
# env 

##### ADDED TO WORK WITH PYTHON 3.7
PATH="/Library/Frameworks/Python.framework/Versions/3.7/bin:${PATH}"
export PATH
##################################

BASEPATH=$(dirname "$0")
# Removed duplication 
export PATH=$PATH:$BASEPATH/USD379NoView/bin;
export PYTHONPATH=$PYTHONPATH:$BASEPATH/USD379NoView/lib/python

# uncomment to set the PYTHONPATH to FBX Bindings here:
# export PYTHONPATH=$PYTHONPATH:"/Applications/Autodesk/FBX Python SDK/2020.2.1/lib/Python37_x64"

if [[ $PYTHONPATH == *"FBX"* ]]; then
    :
else 
    echo "For FBX support, edit PYTHONPATH in this file (USD_pixar_noview.command) or your shell configuration file"
fi

$SHELL

Now to check:

echo $PATH #looked good
echo $PYTHONPATH #looked good
python3 --version #looked good
cd $SOME_DIRECTORY_TO_WRITE_FILES_IN
python3 #open REPL

In the REPL, typed hello world example from https://openusd.org/release/tut_helloworld.html

from pxr import Usd, UsdGeom #This is the hard one... OMG IT WORKED!!!
stage = Usd.Stage.CreateNew('HelloWorld.usda') # No error
xformPrim = UsdGeom.Xform.Define(stage, '/hello') # No error again... could it be?
spherePrim = UsdGeom.Sphere.Define(stage, '/hello/world') # getting close
stage.GetRootLayer().Save() # Eyes closed.... AMAZING THE FILE APPEARS! 
# Cntrl-D to leave

Every line worked and resulted in a HelloWorld.usda!!!

#usda 1.0

def Xform "hello"
{
    def Sphere "world"
    {
    }
}

Act 4: Trying again with pyenv

Using pyenv does not work for the precompiled USDZ Tools, but lets give it another chance with Pixar’s repo.

brew install pyenv
# brew install openssl readline sqlite3 xz zlib
pyenv install --list
pyenv install 3.7.9
pyenv init #to get instructions on how to configure your shell

For me pyenv init said to:

# Load pyenv automatically by appending
# the following to 
# ~/.zprofile (for login shells)
# and ~/.zshrc (for interactive shells) :

export PYENV_ROOT="$HOME/.pyenv"
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"

# Restart your shell for the changes to take effect.

This means pyenv will take over managing Python in every zsh shell (my default) and that’s what I want.

Now to recompile with the new setup.

pyenv shell 3.7.9
python3 --version # expected: 3.7.9
which pip3 #expected: /Users/$USER/.pyenv/shims/pip3
pip3 install PyOpenGL #if prompted to update pip, go ahead.
# pip3 install PySide6 <- Bug in USD 23.05 PySide6 >= 6.5.0
pip3 install PySide6==6.4.3 # <- UPDATE see epilogue
cd $DIRECTORY_WITH_CLONED_REPO
python3 USD/build_scripts/build_usd.py ~/opb/USDefault379

Now for yet another .command file. This time to ask pyenv to set it’s version to 3.7.9 and load the paths like before. I’ve written this one to live in the folder WITH the build. There are pros and cons to that. In this case the pro is that I should be able to use it without change in any new build folder. I’m calling it USDBuildShellLauncher

Don’t forget to chmod 755 USDBuildShellLauncher.command once you set it up.

#!/bin/sh

## uses pyenv to control version of python
## after installing, run pyenv init in preferred
## shell for instructions to finish configuration.
export PYENV_VERSION=3.7.9

BASEPATH=$(dirname "$0")
export PATH=$PATH:$BASEPATH/bin;
export PYTHONPATH=$PYTHONPATH:$BASEPATH/lib/python

$SHELL

Use USDBuildShellLauncher.command to test it out

cd $DIRECTORY_WITH_FILES_TO_CONVERT
# Note with the PATH being set we don't need the full path in the command anymore
usdcat -o $OUTPUT.usda $INPUT.usdc
#cd $SOME_DIRECTORY_TO_WRITE_FILES_IN #if necessary
python3 #open REPL

In the REPL, once again typing in the example from https://openusd.org/release/tut_helloworld.html

from pxr import Usd, UsdGeom #This is the hard one... OMG IT WORKED!!!
stage = Usd.Stage.CreateNew('HelloWorld.usda') # No error
xformPrim = UsdGeom.Xform.Define(stage, '/hello') # No error again... could it be?
spherePrim = UsdGeom.Sphere.Define(stage, '/hello/world') # getting close
stage.GetRootLayer().Save() # Eyes closed.... AMAZING THE FILE APPEARS! 
# Cntrl-D to leave

Amazing!!!! It worked!!

Act 5

As it turns out…

If I hadn’t given up so easily, I might have gotten the USD python scrips kinda working in 3.11? I learned this by discussing the error I was getting with todbot. He’d duplicated my work on his machine and we were trying out multiple import statements to see if the errors changed.

They did, and then they just stopped coming…

I went back to my machine and lo and behold, when opening a repl, running an import line and just an import line (e.g. from pxr import Usd, UsdGeom) 3 times clears out the problems and then you can proceed. I haven’t run a script yet, but that’s interesting!

ALSO - this is the ONLY build I’ve done (with corrected PySide6) that usdview will work for me at all.

I do think my 3.7.9 builds are certainly more stable than the 3.11 build, so they were worth the effort.

But on that note, FIN.

Epilogue

I have not really mentioned usdview much even though some of the above builds have it. For one, I don’t really need it. usdchecker provides more of what I want. But really I’ve been having some trouble with it.

Once I get some more success on that front I’ll make a post about it.