​

Building the engine, and the toolchain

​

1. How the engine builds

​

1.1 There is one build path, and it is pip

cmake_minimum_required(VERSION 3.13)
project(${TARGET_NAME})
add_library(${TARGET_NAME} SHARED ${ALL_SRC})

​

1.2 The PEP 517 chain

# packages/engine/pyproject.toml
[build-system]
build-backend = "backend"
backend-path = ["_custom_build"]
requires = ["cmake>=3.13.3", "numpy>=2.0", "setuptools>=69.2.0"]

if self.config_settings:
    flags = [f"--{k}={v}" for k, v in self.config_settings.items()]
    sys.argv = sys.argv[:1] + ["build_ext"] + flags + sys.argv[1:]

​

1.3 What setup.py does, in order

1. Codegen at import time. create_all(generated_dir) runs at module scope, on every invocation — even --help. create_shadertext() slurps packages/engine/data/shaders/* into a const char* _shader_cache_raw[] string table plus #include/#ifdef dependency tables; create_buildinfo() shells out to git rev-parse HEAD (a non-git copy prints fatal: not a git repository and produces an empty SHA — harmless). Output goes to $PYMOL_BLD/generated, default build/generated, i.e. inside the source tree.
2. Option parsing — defaults in class options at setup.py:193-204.
3. Prefix search — get_prefix_path(), then each prefix is scanned for include, include/freetype2, include/libxml2, include/openvr, lib64, lib.
4. Two extensions are declared: pymol._cmd (the whole engine) and chempy.champ._champ (10 C files from contrib/champ).
5. build_cmake() makes build/temp.../<target>, os.chdirs into it, runs cmake <srcdir> -D... then cmake --build . --config Release -j<os.cpu_count()>. Note the chdir: the build is not reentrant.
6. Version comes from layer0/Version.h by regex. Currently 3.2.0a.

​

1.4 What actually gets compiled

​

1.5 Native dependencies

​

1.6 The failure catalogue

1. fatal error: 'mmtf.hpp' file not found. The default is use_msgpackc = "guess"; guess_msgpackc() finds Homebrew’s msgpack headers, returns "c++11", and MoleculeExporter.cpp then wants mmtf.hpp, which nobody installed. Any dev with msgpack-cxx in Homebrew hits this on the first pip install. The fix is to vendor mmtf-cpp, which is what the bootstrap does. Do not “fix” it with --config-settings use-msgpackc=no — see item 2.
2. use-msgpackc=no silently amputates MMTF and BCIF I/O, which are parity rows. It is not an acceptable default, and scripts/bootstrap.sh refuses to fall back to it.
3. libxml2 is keg-only. Without $(brew --prefix)/opt/libxml2 on PREFIX_PATH the build fails on missing libxml/parser.h. The alternative, --libxml=no, drops COLLADA export.
4. --osx-frameworks=false is broken in this tree. It appends the relative path usr/X11, which never resolves. Treat it as unsupported and keep the default True, which links -framework OpenGL.
5. pymol -c script.py exits 0 even when the script raises, and with -y it can produce zero output and exit 0. Upstream CI’s pymol -ckqy testing/testing.py --run all is therefore only weakly load-bearing. Assert on parsed output, not on exit code.
6. Ray-traced image-diff tests are not bit-stable on darwin/arm64. Do not gate CI on image tests on macOS.
7. pip install -e . writes a ~10 MB .so into the source tree, and upstream’s .gitignore does not ignore *.so. This is why the bootstrap installs the engine non-editable and why it appends the build’s leavings to .git/info/exclude (a local-only file that git never merges, so an upstream merge can never see it).
8. PYMOL_PATH moves between install modes. guess_pymol_path() resolves it to the parent of modules/ when running from a source tree, and to <site-packages>/pymol/pymol_path under a wheel install; PYMOL_DATA and PYMOL_SCRIPTS derive from it. Nothing may hardcode either layout.
9. Upstream’s macOS CI does not test arm64 — it downloads the x86_64 Miniforge on an Apple Silicon runner, so upstream builds PyMOL under Rosetta. This fork’s arm64 build is, as far as upstream CI is concerned, untested territory.

​

1.7 Build flags that matter

​

2. scripts/bootstrap.sh

1. Platform and native deps. macOS requires Homebrew and hard-fails listing whatever of cmake libpng freetype glew glm netcdf msgpack-cxx libxml2 is missing. Linux probes for the headers and warns rather than failing. Catch2 is deliberately not required.
2. Python venv at packages/bridge/.venv (override with --venv or $TENMOL_VENV). It prefers an explicit, non-shimmed interpreter — a pyenv shim or a conda python changes setup.py’s prefix search — and warns if the chosen one looks conda-flavoured. Build requirements (pip, numpy, setuptools, cmake) are installed before the engine build, because the build runs --no-build-isolation.
3. Vendor mmtf-cpp into packages/engine/.deps/mmtf-cpp (override with $TENMOL_DEPS_DIR or point $TENMOL_MMTF_INCLUDE at an existing copy). A --depth 1 clone of rcsb/mmtf-cpp, unpinned. It goes out of tree and onto PREFIX_PATH rather than into packages/engine/include/, which is upstream’s and must stay pristine.
4. Build the engine into the venv, non-editable, with --no-build-isolation --config-settings use-msgpackc=c++11 and PREFIX_PATH covering the brew prefix, the keg-only libxml2, and the vendored headers. On failure it tails the log and refuses to suggest use-msgpackc=no. It then removes the pymol.egg-info pip leaves in the upstream tree.
5. pip install -e packages/bridge[dev]. PySide6 from the engine’s [dev] extra is deliberately not installed: this client replaces the Qt GUI and PySide6 is ~200 MB.
6. .git/info/exclude gains the build’s leavings (packages/engine/modules/pymol.egg-info/, packages/engine/testing/timings.tab, _cmd*.so, _champ*.so). Local-only, never merged.
7. pnpm install, after checking node ≥ 22 and enabling pnpm through corepack if needed.

cmake --build packages/engine/build/temp.*/_cmd
cp packages/engine/build/lib.*/pymol/_cmd.cpython-*.so \
   packages/bridge/.venv/lib/python3.13/site-packages/pymol/

​

3. scripts/doctor.mjs

​

4. The JavaScript toolchain

• The upstream tree is never linted or formatted. eslint.config.js ignores every packages/engine/ subdirectory by name, plus docs/, and the Prettier scripts name their targets explicitly instead of using .. A pnpm format must never produce a diff inside packages/engine/.
• tsconfig.base.json does not set composite. A package that wants project references opts in locally. Setting composite in the shared base makes any app that imports package sources through a path alias fail with TS6307. declaration, declarationMap and incremental are in the base, so opting in is a three-line change.

​

5. CI