hal: C++ API (hal.hh) and pybind11 bindings on the new HAL API

[grandixximo]

@rene-dev @BsAtHome - this is the unification pass we discussed in the #4099 meeting: Rene's C++/pybind11 interface rebuilt on Bertho's getter/setter API, satisfying both constraints (no HAL internals access; simple, low-maintenance, self-documenting C++ surface).

Based on master, fully additive: _hal/hal.py are untouched and nothing is replaced. The by-name query functions and HAL_PORT creation are compiled out when the query API header is absent and activate automatically when #4247 lands.

What this adds

• src/hal/hal.hh - type-safe C++ layer, header-only, in namespace linuxcnc::hal:
  • hal::pin<T> typed handles via traits<T> over the rtapi_ types: rtapi_s32/rtapi_sint are distinct C++ types, so 32/64 accessor selection happens at compile time with zero cost. Handles re-read the shmem slot per access, so hal_link() rewrites stay visible (same semantics as C pin pointers). Handles are move-only; HAL objects are unique.
  • hal::pin_t variant + hal::anypin: runtime-typed access multiplexes on the variant tag (the stored-type pattern required for name-keyed collections), with the tag supplied by the query API.
  • hal::component: pins + params, newpin/newparam (typed and runtime-typed), add_pin for the struct-member idiom, item access, prefix handling.
  • ULAPI by-name functions (get_value, set_value, set_signal, component_exists, pin_has_writer, signal management) implemented entirely on hal_get_p/hal_set_p/hal_get_s/hal_set_s/hal_comp_by_name. No hal_priv.h, no direct shmem access, no re-implemented parsers.
• src/hal/halpybind.cc - pybind11 module (halpp.so), adapted from @rene-dev's bindings, extended with params and error paths. Built alongside _hal/hal.py; module replacement is a separate (3.0) discussion.
• src/emc/task/taskclass.cc - converted to the new API (it was the only hal.hh consumer).
• tests/halpp/ - Python and native C++ smoke suites (both green), plus python3-pybind11 build dep.

Safety properties worth noting

• Query callbacks are strictly exception-free: conversion errors are reported by flag and thrown after the library releases the HAL mutex. (Throwing through a callback wedges the session-wide mutex; found the hard way, and tested.)
• Set-side coercion uses the item's type with range checks; an overflow raises instead of truncating.
• Typed access compiles to the same two-load sequence as the C API and the old pointer dereference, so there is no hot-path cost.

Relationship to the original pybind11 branch

Kept: the variant/map/PyPin structure and the binding shape.
Dropped: the private set_common string parser (superseded by hal_set_p), hal_mutex_guard, the get_info_signals stub, waitWritable (was non-functional).
Compatibility aliases (hal_comp, hal_pin<T>, hal_dir, PyPin) are provided so existing consumers of the old hal.hh keep working.

Test evidence

Full tree builds; tests/halmodule.0/.1 pass; new smoke suites cover component lifecycle, typed pins/params (incl. negative S32 and 64-bit values), signals, linking, by-name get/set with coercion, port pins, and error paths. The suites auto-skip query-API checks when built without the query API, so they run on both master and the #4247 tree.

[grandixximo]

This seems to be working, but I am not sure if it is the right shape we want, these are the direction I took on the issue raised in the meeting, subject to revision.

Problem	Solution
Handle is width-blind	Type lives in traits<T> (compile time) or the variant tag (runtime, from q->pp.type)
hal_link rewrites the handle slot	pin<T> stores a pointer to the slot and re-reads per access (C semantics, staleness impossible)
Throwing through query callbacks wedges the HAL mutex session-wide	Exceptions captured in the callback, rethrown after the library releases the mutex
Handle storage must be shmem	hal_malloc'd hal_refs_u slot per item (halmodule's own pattern)

@BsAtHome A bare hal_sint_t is width-blind, and wrapping foreign raw handles across widths is unsafe in the interim. But the C++ layer never selects from the handle alone. Component-created pins carry the type at construction via traits (int32_t vs int64_t are distinct C++ types, compile-time selection), and the runtime path uses q->pp.type from your query API to pick the variant alternative, the stored-type-tag multiplex you described in #4099. The only unsafe path is explicitly reinterpreting a raw handle, which can be guarded with a query lookup. So the layer works transitionally, it just gets simpler after the break.

[rene-dev]

oh, that was quick. I think this should be separate from the other big PR, as this can go in master fairly quickly. the only reason I stopped was that buster had no pybind11 package. this should also replace the current pyhal/halmodule instead of adding a third module. and, the same cpp api should be used in xhc, task and other places which are implemented in cpp.

[grandixximo]

Buster is basically not supported anymore with master, I will fix the review points tomorrow, thank you both ☺️
My bad on missing the include, getting late time for bed...

[BsAtHome]

There is another issue. The new hal_lib uses a reference counted init/exit and has separated out hal_lib_init() and hal_lib_exit(). The reason to take this out is that you do not need a component for pin/param/signal read/write or query interactions. You only need to have mapped shared memory.

Calling hal_init() to create a component (if you want to create pins) now calls the factored out hal_lib_init() and hal_exit()calls the factored out hal_lib_exit().

The python class must initialize using hal_lib_init() and terminate by calling hal_lib_exit(). If the user has not called hal_exit() on any created components, you will now get a proper error message (because the user forgot to terminate correctly).

[rene-dev]

can you rename the module to halmodule, so that the UIs and the tests that are already in place use the new bindings, and remove the old ones?

[grandixximo]

can you rename the module to halmodule, so that the UIs and the tests that are already in place use the new bindings, and remove the old ones?

Replacing halmodule is the goal, agreed, but renaming now would break things for two separate reasons.

First, a hard technical one: on master the new bindings cannot cover the existing Python surface. get_info_pins/signals/params, pin_has_writer and component_exists are used by qtvcp (qt_halobjects, hal_selectionbox) and hal_glib, and they need the query API (hal_list_*, hal_get_p) from #4247. Without it, the options are a strict subset module (GUIs break immediately) or private shmem access (exactly what this API change abolishes).

Second, the compatibility tail: hal.error exception type, KeyboardInterrupt on halcmd unload (documented in hal.py's docstring, user comps wrap their main loop in it), the exact get_info_* dict shapes (tests/halmodule.0 compares verbatim), the hal.stream object API (tests/halmodule.1), setitem coercion quirks, pin .dir/.type attributes, and the hal.py wrapper shims. Matching all of that is what makes the swap a user-visible event, and it deserves its own PR with a full parity pass, not a rename here.

So the proposal stands as: land this additive now (hal.hh + halpp alongside, nothing replaced), and do the rename + replacement as a follow-up once #4247 is in. Your old branch's consumer sweep (hal_glib, qtvcp core, raster, tests) is the starting checklist for that PR. Is this acceptable? Open for discussion...

[rene-dev]

I see that #4231 changes halmodule.cc, we should also add those functions via c++/pybind11

[BsAtHome]

I see that #4231 changes halmodule.cc, we should also add those functions via c++/pybind11

No, all of this should be in halquery.

[rene-dev]

I see that #4231 changes halmodule.cc, we should also add those functions via c++/pybind11

No, all of this should be in halquery.

What is halquery?

[BsAtHome]

I see that #4231 changes halmodule.cc, we should also add those functions via c++/pybind11

No, all of this should be in halquery.

What is halquery?

That is the new interface to all of HAL's internals nicely wrapped up from the hal query interface without someone needing to dig into HAL's internals. Hal_lib's internals are fully isolated in #4247 and nobody should be writing code that circumvents that. Inclusion of hal_priv.h is strictly forbidden in any code that is not part of hal_lib.
Halmodule deals with components and everything else was "bolted on". You do not want one big blob of code to do it all. You want modularization, which halquery provides.

[rene-dev]

ah, found it. I would much prefer if it would be a c++ api, which can be used to automatically generate the python bindings.
most of userspace code is c++ anyway. I have maintained the c bindings in the past(pyhton2->3 migration), and do not ever want to deal with them again.

[BsAtHome]

It is a pure C version, so it does not have the same problems as C++ interfacing C-based Python ;-)

[grandixximo]

This is exactly the split #4251 (this PR) implements: the query API stays pure C in the library (Bertho's layer, no bypass possible), and the C++ API sits on top of it (hal.hh), which is where Python bindings get generated (halpybind) , no hand-written CPython code. halquery.c proves the query API is sufficient; the same functions can be exposed as pybind wrappers over hal_get_p/hal_list_* once #4247 lands. So the endgame can be: C query API in hal_lib, C++ API above it, Python bindings generated from the C++ layer, and the hand-written C modules retired on the same schedule. Happy to demonstrate the halquery-equivalent functions in halpybind as a follow-up once the base is in.

[BsAtHome]

Fine by me to get it C++ wrapped. It does not yet have priority one.

There is one thing that should be fixed in both halmodule and halquery: use IntEnum types for both type and direction. It may be necessary to have a haltype module with the enums so you it can be properly shared between the other modules. Not sure how this can be handled properly in using consistent namespace names and hierarchy.

[grandixximo]

A shared pure-Python haltype.py with HALType(IntEnum) and HALDir(IntEnum) is the cleanest shape. IntEnum members are ints, so the C boundary stays plain int and everything interops both ways; hal, halquery and the pybind module import and re-export from it, keeping hal.HAL_FLOAT working unchanged. I would keep the namespace flat (sibling of hal.py) rather than a linuxcnc.* package, which breaks every existing import hal. One pybind11 note: py::enum_ does not emit real IntEnum objects, that needs py::native_enum (pybind11 >= 3.0, trixie ships 2.13), so for now the bindings take constants from haltype.py and we flip to native_enum when the distro catches up.

[BsAtHome]

A shared pure-Python haltype.py with HALType(IntEnum) and HALDir(IntEnum) is the cleanest shape. IntEnum members are ints, so the C boundary stays plain int and everything interops both ways; hal, halquery and the pybind module import and re-export from it, keeping hal.HAL_FLOAT working unchanged.

That would be the simple way. If it works, then it should be nice. The real benefit is in halquery to have the iteration result show the right name instead of a number. There it is big value. Probably also need to add the same for other constants.

I would keep the namespace flat (sibling of hal.py) rather than a linuxcnc.* package, which breaks every existing import hal.

Global namespace naming and fixing hierarchy is a problem we need to fix in a later version. This would be a version 3.0 thing, I guess ;-)

One pybind11 note: py::enum_ does not emit real IntEnum objects, that needs py::native_enum (pybind11 >= 3.0, trixie ships 2.13), so for now the bindings take constants from haltype.py and we flip to native_enum when the distro catches up.

Ha, and so much for "C++ is easy to maintain" ;-)
The version and support problem always pops up regardless of language and library. You always end in hell once you need to support over long periods of time and different distros and distro versions. At least it is warm in hell.