​

Spike 05 — State change detection for the web-client bridge

​

STATUS — re-read against the tree on 2026-08-02

§1–§7 are the design the bridge shipped and they still describe it. The two-channel split of §5 (blocking APIEnter queries on the engine thread, the three non-blocking drains anywhere else), the single-owner rule of §3, and the “_get_feedback() returns None, not []” trap are all live constraints, not history. §8 IS SUPERSEDED. Do not implement it. It was a ranked wish-list of C++ that had not been written; 08-native-changes.md then wrote something deliberately different, and §8’s top recommendation is the one that was rejected on evidence:

§8 item	what actually happened
8.1 patch SettingRec::setChanged() — “highest value, ~4 lines”	REJECTED, on purpose. 08 §1.3: Setting.h is included by ~everything (a full-tree rebuild at every upstream merge) and setChanged() fires for bg_rgb, ray_trace_mode and the movie panel, i.e. it manufactures false wake-ups. The 113 ExecutiveInvalidateRep call sites inside Setting.cpp are what actually rebuild a rep, and one bump at ExecutiveInvalidateRep covers all of them.
8.2 ExecutiveNamesVersion	Built, as m_web_panel_version — plus m_web_enable_version and m_web_name_version. 08 §1.1/§1.2. §8.2’s “two sites need their own bump” was right in substance and wrong in detail: the choke point for enable/disable is ReportEnabledChange, not ExecutiveSpecEnable.
8.3 ReprVersion / ColorVersion	Built, as m_web_rep_version + a 64-bit FNV-1a content signature per rep. 08 §2. §8.3’s five bump sites would have missed show/hide entirely — that path drives OMOP_VISI/ObjectSetRepVisMask directly and never reaches ExecutiveInvalidateRep (08 §1.2).
8.4 do not add a Notify bus; bound the feedback queue in the bridge, not in C++	still correct, still followed.

The consequence for §2/§6: the poll’s blind spots (per-atom colour, per-atom reps, alter, coordinate edits) no longer need the §7.3 command-echo channel to be correct — a per-rep version bump sees all four. §7.3 remains a useful latency optimisation, not the only defence. Everything here was measured with a session scratchpad venv that no longer exists; use packages/bridge/.venv/bin/python.

​

1. How the Qt GUI detects change today (the real polling model)

​

1.1 Two timers, that’s it

def update_feedback(self):
    self.update_progress()                       # cmd.get_progress() -> progress bar
    feedback = self.cmd._get_feedback()          # drains the log ring -> text pane
    ...
    for setting in self.cmd.get_setting_updates() or ():   # drains changed-setting flags
        ...                                                # -> re-checks menu items
    self.feedback_timer.start(500)

​

1.2 The object list is NOT polled from Python at all

​

1.3 The three existing consumers of the consume-once drains

packages/engine/modules/pmg_qt/pymol_qt_gui.py:944          self.cmd._get_feedback()
packages/engine/modules/pmg_qt/pymol_qt_gui.py:952          self.cmd.get_setting_updates()          # global
packages/engine/modules/pmg_tk/Setting.py:146               self.cmd.get_setting_updates(sele, state)  # per-object
packages/engine/modules/pmg_tk/skins/normal/__init__.py:489 self.cmd._get_feedback(self.cmd)

​

2. Measured cost of the polls a web client needs

cmd.get_names()                    median=      1.3 us  p95=      1.3 us
cmd.get_names('objects')           median=      1.2 us
cmd.get_names('all')               median=      1.3 us
cmd.get_names('public_selections') median=      1.0 us
cmd.get_view()                     median=      2.0 us
cmd.get_frame()                    median=      0.2 us
cmd.get_state()                    median=      0.2 us
cmd.count_atoms('sele')            median=    152.5 us   <-- 141-atom selection
cmd.count_atoms('all')             median=    269.2 us
cmd.get_setting_updates()          median=      1.0 us
cmd.get_vis()                      median=      3.1 us
cmd._get_feedback()                median=      0.7 us
cmd.get_progress()                 median=      0.3 us
cmd.get_movie_playing()            median=      0.8 us
cmd.get_scene_list()               median=      0.7 us
cmd.get_wizard()                   median=      0.9 us
--------------------------------------------------------
SUM of medians (full set):        437.4 us  =  0.437 ms
budget @30 Hz = 33 333 us  |  @20 Hz = 50 000 us
p.idle()                           median=      0.7 us
p.getRedisplay()                   median=      0.8 us
cmd.refresh()  (nothing dirty)     median=      1.5 us

atoms=  52569 objs= 11 | get_names  1.3us | get_view 2.0us | get_vis  3.1us | count_atoms(sele)  172us | count_atoms(all)   279us
atoms= 200718 objs= 42 | get_names  1.8us | get_view 2.0us | get_vis  9.5us | count_atoms(sele) 1100us | count_atoms(all)  1418us
atoms= 501795 objs=105 | get_names  3.0us | get_view 2.0us | get_vis 24.9us | count_atoms(sele) 5902us | count_atoms(all)  6739us
cmd.index('sele') at 500k atoms   = 11 796 us   (60 270 indices)
cmd.get_names('selections')       =      1.2 us  (does not scale)

​

2.1 Verdict on rate

tick cost median 67.7 us  p95 97.3 us  max 229.6 us
wall 11.00 s, process CPU 0.027 s -> 0.25% of one core
false-positive emissions with nothing happening: 0

​

3. Consume-once behaviour — confirmed, and it is worse than “one consumer”

for (int a = 0; a < cSetting_INIT; ++a) {
  if(I->info[a].changed) {
    I->info[a].changed = false;      // <-- destructive read
    result.push_back(a);
  }
}

--- get_setting_updates consume-once ---
after 2 sets: call1 = [155, 279]
             call2 = []

--- two independent 'consumers' interleaved (the failure mode) ---
consumerA saw: [468]  consumerB saw: []
=> EXACTLY ONE CONSUMER: True

--- _get_feedback consume-once ---
call1 = ["PyMOL>print('hello-from-pymol')"]
call2 = []

--- getRedisplay ---
getRedisplay call1=1 call2=0            (reset=True is the default)
getRedisplay(reset=False) twice: 1 1    (non-destructive variant exists)

1. The per-object setting drain is a separate channel. Setting a per-object value does not show up in the global drain:
   global drain after object-level set: []
   object drain call1: [155]
   object drain call2: []
   
   So the bridge must call cmd.get_setting_updates(name, state) for every object as well as the global one. Cost measured for 31 objects: 21.6 µs total (e13_vis.py) — cheap, do it every tick.
2. _get_feedback() returns None, not [], when it cannot get the lock (packages/engine/modules/pymol/internal.py:596-606, lock_attempt = acquire(blocking=0), packages/engine/modules/pymol/locking.py:29-30). Treating None as “no output” is silently correct today only because the queue is not drained; treating it as [] and also clearing local buffers would lose lines. Measured directly during a running ray: _get_feedback -> None.
3. get_setting_updates() returns [] on a lock miss (packages/engine/modules/pymol/setting.py:440-447), which is indistinguishable from “nothing changed”. This is safe (nothing was drained, so the flags survive to the next tick) but it means a lock miss looks like quiescence. Do not build “settle detection” on it.
4. The feedback queue is unbounded. OrthoFeedbackIn (packages/engine/layer1/Ortho.cpp:492-499) pushes into a std::queue<std::string> with no cap; OrthoFeedbackOut (:502-515) pops exactly one. Measured: 20 000 undrained lines → +2.98 MB RSS, all 20 000 returned in order (e8_fb2.py). If the browser disconnects and the bridge stops draining, PyMOL leaks. The bridge must drain unconditionally and buffer on the Python side with its own cap.
5. OrthoFeedbackIn is gated on G->Option->pmgui. It is true under pymol2.PyMOL() (verified: cmd.do("print('X')") → ["PyMOL>print('X')"]). But echo=0 suppresses it entirely, and raw Python print() only reaches the queue if pcatch._install() has redirected sys.stdout (that is what pymol_gl_widget.py:105 does). The bridge must call pcatch._install() or it will not see plugin/script print output.
6. A session load fires 798 setting updates in one drain (e6_misc.py) — usable as a “resync everything” signal. Scene recall fires a small set ([23, 152, 254, 396]).

​

4. Is there ANY push mechanism already? Yes — one, and it is unusable as-is

cWizEventPick 1 | Select 2 | Key 4 | Special 8 | Scene 16 | State 32 | Frame 64 | Dirty 128 | View 256 | Position 512

mutation                | during action | after p.idle() | after cmd.refresh()
------------------------+---------------+----------------+---------------------------
turn x 10               | []            | []             | ['do_view']
zoom chain A            | []            | []             | ['do_position','do_view','do_dirty']
frame(+1)               | []            | []             | ['do_dirty','do_frame:5']
set state               | []            | []             | ['do_scene']
load new object         | []            | []             | ['do_scene']
color red               | []            | -              | ['do_scene']
show / hide spheres     | []            | -              | ['do_scene']
disable / enable        | []            | -              | ['do_scene']
set sphere_scale        | []            | -              | ['do_scene']
scene recall            | []            | -              | ['do_scene','do_dirty']
set_name                | []            | -              | ['do_scene']
cmd.select(...)         | []            | -              | []          <-- MISS
cmd.delete(...)         | []            | -              | []          <-- MISS
cmd.ungroup(...)        | []            | -              | []          <-- MISS

1. It is not a push at all — it is draw-pumped. p.idle() delivers nothing. Only cmd.refresh() (packages/engine/layer4/Cmd.cpp:4715-4731 → SceneInvalidateCopy + ExecutiveDrawNow) or cmd.ray() deliver events. So you still need a loop; you have merely moved the poll into the render path.
2. The pump is expensive when it does anything. cmd.refresh() on the 52 k-atom scene: median 1.5 µs when clean, but median 38 313 µs (38 ms) after cmd.color("red","all"), because SceneUpdate rebuilds every rep. Pumping at 30 Hz during interactive editing forces a full rep rebuild 30×/s.
3. Object-list deletes and all selection changes are missed (see table).
4. Fatal: there is exactly one wizard stack and the user owns it. Verified:
   spy alone ->  ['spy.do_view'] | get_wizard: Spy
   after cmd.wizard('measurement'), get_wizard: Measurement
   stack: ['Spy', 'Measurement']
   spy events while user wizard active: []
   
   WizardGet returns the top of the stack, so the moment the user opens Mutagenesis or Measurement the bridge goes deaf. Default mask is pick+select only (packages/engine/layer1/Wizard.cpp:218-227), so real wizards would not re-emit them either.

​

5. The real hazard: the API lock, not the poll rate

probe                    block ms  ray done  value/err
get_progress                  0.1  False     -1.0
_get_feedback                 0.0  False     None
get_setting_updates           0.0  False     []
get_names                  3808.8  True      ['base', 'c1', ...]   <-- BLOCKED 3.8 s
get_view                      0.1  True      ...

ran 5.90s, 47 ticks (target 177), lock-miss=0
tick cost median 58.6 us  p95 182.0 us  max 4 390 701.2 us      <-- 4.39 seconds
schedule lateness median 3.81 ms  p95 5.02 ms  max 5.03 ms      (ray took 4419 ms)

​

6. What the poll cannot see (measured gaps)

1. Per-atom representation state is invisible. cmd.get_vis() reports the object-level visRep only. Proof:
   cartoon only      : [1, [], [5], 5]
   + spheres on CA   : [1, [], [5], 5]   changed: False   (574 atoms have rep spheres)
   + sticks on ALL   : [1, [], [0, 5], 5] changed: True
   
   show spheres, m and name CA is undetectable by any cheap poll. (PyMOL’s own object panel does not display per-rep state either, so this is not a parity regression — but any React “reps” indicator beyond upstream’s would need cmd.count_atoms("x and rep spheres") per rep, which is count_atoms-priced.)
2. Colour changes are invisible. cmd.color("red", sel) changes no polled field. cmd.get_object_color_index(name) (0.8 µs) catches only the object colour, not per-atom.
3. Coordinate / alter / alter_state changes are invisible.
4. Group membership needs an extra query: cmd.get_names("objects", selection="grp") (58.9 µs) — the group object itself survives ungroup, so the names tuple does not change.
5. cmd.set_name("base","BASE") is a no-op because ignore_case is on by default — an easy false “missed event” when writing tests.

​

Positive control — what the poll does catch, within one tick

turn x 10                    -> ['view']
zoom chain A                 -> ['view']
create newobj                -> ['names','enabled','view','vis']
delete newobj                -> ['names','enabled','vis']
select sele                  -> ['names','enabled','vis']
deselect                     -> ['enabled']
delete sele                  -> ['names','vis']
disable base / enable base   -> ['enabled']
set sphere_scale             -> ['settings(1)']
mset 1 x20                   -> ['settings(2)']
frame 7                      -> ['frame','settings(2)']
scene store SZ               -> ['scenes','settings(2)']
scene recall SZ              -> ['settings(4)']
group g1                     -> ['names','enabled','vis']

​

7. The design

​

7.1 Process/thread shape

 browser ──WebSocket──┐
                      │
        ┌─────────────▼──────────────────────────────────┐
        │ PyMOL process (single pymol2.PyMOL instance)   │
        │                                                │
        │  MAIN THREAD  ("PyMOL thread") — owns API lock │
        │    while True:                                 │
        │      drain command inbox (from WS)             │
        │      execute commands                          │
        │      p.idle()                                  │
        │      tick()  -> snapshot + diff + drains       │
        │      sleep to 33 ms cadence                    │
        │                                                │
        │  STATUS THREAD (10 Hz) — never takes API lock  │
        │    cmd.get_progress()                          │
        │    cmd._get_feedback()                         │
        │    cmd.get_setting_updates()                   │
        │                                                │
        │  WS I/O THREAD (asyncio loop)                  │
        └────────────────────────────────────────────────┘

• One PyMOL thread. All APIEnter calls (i.e. all of §7.2) happen on the thread that also executes commands, so a poll can never contend with a command. Contention is not a deadlock but a 4-second stall (§5).
• The status thread must only ever call the three non-blocking functions. Adding a cmd.get_names() to it re-introduces the stall.
• The bridge is the sole owner of the three drains. get_setting_updates() (global and per-object), _get_feedback() and getRedisplay() are destructive. No plugin, no pymol.rpc, no pymolhttpd may run alongside. pcatch._install() must be called at startup. Nothing else in the process may call them — enforce with a lint rule over packages/bridge/.
• _get_feedback() is None means “locked, retry”, not “empty”.
• Never let the feedback queue go undrained (unbounded, §3.4).

​

7.2 The tick (30 Hz, main thread)

def snapshot():
    return {
      "names":    tuple(cmd.get_names("all", enabled_only=0)),   #   1.3 us
      "enabled":  tuple(cmd.get_names("all", enabled_only=1)),   #   1.3 us
      "groups":   tuple(cmd.get_names("group_objects")),         #   1.3 us
      "view":     cmd.get_view(),                                #   2.0 us
      "frame":    cmd.get_frame(),                               #   0.2 us
      "state":    cmd.get_state(),                               #   0.2 us
      "scenes":   tuple(cmd.get_scene_list()),                   #   0.7 us
      "vis":      canon(cmd.get_vis()),                          #   3.1 us
      "movie":    cmd.get_movie_playing(),                       #   0.8 us
      "wizard":   wizard_ident(cmd.get_wizard()),                #   0.9 us
    }

su_global = cmd.get_setting_updates()                # 1.0 us
su_object = {n: cmd.get_setting_updates(n, 0)        # 21.6 us for 31 objects
             for n in snap["names"]}

• cmd.count_atoms(...) — 5.9 ms at 500 k atoms. Selection atom counts are a debounced request (client asks after the names/enabled diff settles for ~150 ms) or ride on the command-echo channel.
• cmd.get_names("objects", selection=grp) per group (58.9 µs each) — only re-query when groups or names changed.
• cmd.refresh() — see §7.4.

​

7.3 The command-echo channel (covers the §6 gaps)

{ "cmd": "color", "args": ["red", "chain A"], "invalidates": ["geometry:chain A", "color"] }

​

7.4 cmd.refresh() — pump it, but lazily

• nothing dirty → 1.5 µs (free, call it every tick)
• after cmd.color("red","all") → 38 313 µs (38 ms)
• full-surface build → 5.20 s

​

7.5 Message shape

S→C  state.delta   { seq, changed: {names?, enabled?, groups?, view?, frame?, state?,
                                    scenes?, vis?, movie?, wizard?},
                     settings: {global:[idx…], perObject:{name:[idx…]}} }
S→C  state.resync  { seq, full snapshot }        # session load, cmd.do, client reconnect
S→C  log.append    { lines:[…] }                 # 10 Hz status thread
S→C  progress      { value }                     # 10 Hz status thread, -1 == idle
C→S  poll.rate     { hz }                        # 30 when focused, 4 when hidden

​

8. What must be added in C++ (v2, optional — none of it blocks v1)

​

8.1 Non-destructive setting generation counter (highest value, ~4 lines)

void setChanged() {
  defined = true;
  changed = true;
}

​

8.2 ExecutiveNamesVersion (~3 lines)

1622  ExecutiveInvalidateGroups        3238  ExecutiveOrder
1684  ExecutiveUpdateGroups            5297  ExecutiveSetNamedEntries
1876  ExecutiveScrollTo               14455  ExecutivePurgeSpec        (delete)
14603 ExecutiveDeleteStates           14618  ExecutiveReAddSpec
14856 ExecutiveManageObject (create)  14927  ExecutiveManageSelection  (select)
17328 ExecutiveSetOrderOf

​

8.3 ReprVersion / ColorVersion (real new capability)

​

8.4 Explicitly NOT recommended

• Do not add a Notify/observer bus. The measured poll cost (67.7 µs at 30 Hz = 0.25 % core) does not justify threading an event system through five layers, and a C++→Python callback has to take the GIL anyway — which is exactly the WizardCallPython pattern that already exists and already has the re-entrancy hazard (WizardUpdate → SceneUpdate → WizardDoView, packages/engine/layer3/Executive.cpp:11533-11534; the wizard header itself warns event_mask_dirty = 128 # anything changed (BEWARE FEEDBACK!), packages/engine/modules/pymol/wizard/__init__.py:13).
• Do not bound the feedback queue in C++ to fix §3.4 — bound it in the bridge instead, so the fix does not have to ship in a rebuilt _cmd.so.

​

9. Consequences for other owners (reported, not applied)

1. docs/architecture.md — the bridge must be single-threaded for state queries with a separate non-blocking status thread. A “poller thread + command thread” design stalls for the full duration of ray/surface builds (measured 3.8 s and 5.2 s).
2. architecture.md — the bridge is the exclusive owner of get_setting_updates() (global and per-object), _get_feedback() and getRedisplay(reset=True). Any design that also runs pymol.rpc, pymolhttpd, or a Qt GUI in the same process is broken. Add a lint rule.
3. architecture.md / wizards.md — the wizard event mask cannot be used as the bridge’s change feed (single stack, user-owned, draw-pumped, misses delete/select). Wizards remain a proxied feature, not infrastructure.
4. feature-parity.md — the object panel, wizard panel and scene bin have no Python data feed today (they are C++ Block::draw surfaces). Those rows need explicit “new bridge endpoint required” flags, not “wire up existing API”.
5. Whoever owns internal-gui.md / geometry extraction: cmd.get_vis() is object-level only — per-atom rep state must come from the command-echo channel or from §8.3.
6. Build/bootstrap owner: pcatch._install() must be called by the bridge at startup, mirroring packages/engine/modules/pmg_qt/pymol_gl_widget.py:105, or script print() output never reaches the client.

​

10. Reproduction

cd <scratchpad>/events
PYTHONUNBUFFERED=1 <scratchpad>/venv/bin/python e1_timing.py       # §2 poll costs
PYTHONUNBUFFERED=1 <scratchpad>/venv/bin/python e2_consume.py      # §3 consume-once
PYTHONUNBUFFERED=1 <scratchpad>/venv/bin/python e3_wizard_push.py  # §4 wizard events + refresh cost
PYTHONUNBUFFERED=1 <scratchpad>/venv/bin/python e4_push_alt.py     # §4 load_callback/set_key/log/stack
PYTHONUNBUFFERED=1 <scratchpad>/venv/bin/python e5_dirtybit.py     # getRedisplay coverage
PYTHONUNBUFFERED=1 <scratchpad>/venv/bin/python e6_misc.py         # §2 scaling, movie, session
PYTHONUNBUFFERED=1 <scratchpad>/venv/bin/python e8_fb2.py          # §3.4 unbounded queue
PYTHONUNBUFFERED=1 <scratchpad>/venv/bin/python e9_loop.py         # §5 30 Hz poller vs ray
PYTHONUNBUFFERED=1 <scratchpad>/venv/bin/python e10_lockfree.py    # §5 blocking vs non-blocking
PYTHONUNBUFFERED=1 <scratchpad>/venv/bin/python e11_progress.py    # §5 progress during long ops
PYTHONUNBUFFERED=1 <scratchpad>/venv/bin/python e12_design.py      # §7 prototype + coverage
PYTHONUNBUFFERED=1 <scratchpad>/venv/bin/python e13_vis.py         # §6 get_vis / per-object drain
PYTHONUNBUFFERED=1 <scratchpad>/venv/bin/python e14_final.py       # §6 rename/groups/reps