Release notes

Unreleased

Database migration v67 → v68 (one step, automatic on load, _migrate_v68_rp_group_flag). The new parameter is additive and defaults to prior behaviour, so a migrated database selects representative periods byte-identically until you opt into net-load clustering.

• Net-load representative-period clustering (--netload-clustering). The representative-period preprocessor can cluster on a real-MW net-load signal (demand − Σ VRE·availability over demand-match-sized capacities) instead of the default per-series normalized profile/inflow stack. VRE is any unit__node__profile arc whose profile_method is upper_limit (the schema default). --vre-penetration scales the demand-match energy-share target. The default (non---netload-clustering) path is byte-parity with prior behaviour.
• group.use_for_representative_periods (schema v68). A per-group yes_no flag that marks a region-group as a net-load aggregation unit: when one or more groups carry yes, net load is summed per flagged group; when none do, selection falls back to per-node net-load signals.
• Net-load iteration driver (python -m flextool.representative_periods.netload_iterate). An invest-only iterate-until-stable loop that feeds each iteration's solved investment capacities back into the next net-load selection until the representative set stabilises (early-stop on stability; bootstrap dispatch skipped). --keep-best additionally dispatches each mature iteration and keeps the lowest full-year dispatch-cost set. Subsets are run via a model.solves override alternative; a fail-closed guard rejects an active timed energy_margin_adder.
• Fixed representative-period blended-weights inter-period storage state being mis-scaled by unit size for storage nodes with unit size ≠ 1.

Release 4.0.0b26 (30.7.2026) — adequacy calibrator + Calibrate-investments GUI; additive energy-margin & hard node balance; near-optimal IPM acceptance

Database migration v65 → v67 (two steps, automatic on load). The polar-high floor moves to >=3.7.0 (the near-optimal solve-acceptance diagnostics below rely on it). Other floors unchanged (polars>=1.40, highspy<=1.14.0). Every new schema parameter is additive and defaults to prior behaviour, and the two migration steps preserve existing values, so a migrated database solves byte-identically until you opt into the new levers.

Adequacy — calibration & margins

• Adequacy-margin calibrator (new CLI python -m flextool.calibrate <db> <scenario>). An iterative helper that repeatedly solves a scenario and raises a per-node energy-margin adder until the system reaches an adequate capacity margin, so investments are neither over- nor under-built. Each iteration reads the previous solve's unserved-energy slack and resizes the margin; the result is written into a dedicated <scenario>_adeq_calib alternative appended to the scenario's stack (originals untouched — delete the alternative to undo). Controls include --iterations, --sizing (timed, placing margin only in the periods/steps that fell short, or uniform), a planning-safety --overshoot, first/remaining-iteration damping, and a resource-cap stall guard (--stall-fraction) that stops chasing a target a hard limit makes unreachable. Runs are warm-started between iterations for speed.
• node.energy_margin_adder + energy_margin_method = inflow_adder (schema v66). An additive companion to the multiplicative margin from b25: extra demand [MWh] added to a node in the investment solve only, raising adequacy headroom without changing dispatch demand. The b25 multiplicative energy_margin parameter is renamed to the self-documenting energy_margin_multiplier (a definition rename; existing values are preserved), and the previously reserved inflow_adder method value is now live.
• Representative periods — per-region budgeted force-include. The net-load force-include gains an opt-in per-region-group scope (--force-region-scope / --force-region-budget) that scores net load independently per region group and greedily forces each region's worst lull under a budget cap, instead of forcing a single system-coincident period. Default (no flags) is unchanged, byte-parity behaviour.

GUI — Calibrate investments

• New "Calibrate investments" dialog. A non-modal dialog (opened from the main window for the checked scenarios) that hosts two tools: a representative-periods builder and the adequacy calibrator above. It never renders its own command lines — the live CLI previews it shows are the exact commands it runs, with a Copy button each. Representative periods are written into a scenario-named alternative (<scenario>_rp_<n>rp_<len>h, de-duplicated on repeat builds) with a descriptive note, and an opt-in checkbox appends that alternative to the selected scenario(s). Every control carries a plain-English hover explanation; launching either tool opens the Execution window so the run is visible. The dialog is scenario-aware (each scenario is clustered from its own series and only receives the solves that belong to it) and fits small screens (bounded, scrollable solve list; the body scrolls within a height-capped window). xlsx-backed scenarios are excluded, since their database is regenerated on each run.

Node balance — hard-balance option

• node.penalty_method (schema v67). A per-node toggle for the balance slack. regular (default, byte-identical to before) keeps the penalty_up / penalty_down unserved-energy / over-supply slack variables; off removes them, making the node balance a hard equality so a node that cannot balance turns the solve infeasible instead of silently absorbing the shortfall at penalty cost. Useful on adequacy studies where that hidden slack would otherwise mask inadequate capacity.

Solver — near-optimal acceptance

• Accept in-practice-optimal crossover-off interior-point solves (requires polar-high>=3.7.0). When the interior-point method returns a feasible primal with a negligible primal–dual gap but the status stays "unknown" (crossover disabled), the solve is now accepted as optimal instead of being treated as a failure, and this acceptance is propagated to the multi-solve cascade's exit scan. Tunable via FLEXTOOL_ACCEPT_* knobs; default thresholds are conservative.

Fixes

• Reserve balance crashed on dynamic / N-1 reserves. The dynamic and N-1 reserve-balance right-hand side left the flow variable's native source/sink axis open, which could abort canonicalisation on some models; the axis is now tied correctly.
• Mixed-shape group floors were dropped in (period, time) broadcasting. A group with a mix of period-scalar and period-time floor Maps had the other groups' floors inner-joined away, letting the optimum fall below the intended floor; the broadcast now preserves every group's rows.
• Calibrator alternative write crashed on 0-based scenario stacks. The calibration-alternative append no longer collides on real models whose scenario_alternative ranks are stored 0-based.

Release 4.0.0b25 (15.7.2026) — per-solve autoscaler mode; energy/capacity margins; rep-period force-include; storage-handoff & scale-invariant-floor fixes

Database migration v63 → v65 (two steps, automatic on load). The polar-high floor moves to >=3.6.1 (the scale-invariant coefficient floor — without it, full-autoscale re-solves of some battery / demand-response and wide-coefficient-range models are mis-computed). Other floors unchanged (polars>=1.40, highspy<=1.14.0). The new schema parameters are additive and default to prior behaviour; the correctness fixes below (scale-invariant floor, storage-handoff, space-mangled names) change results only on the models that exercise those paths.

Numerical scaling — correctness

• Per-solve autoscaler mode: solve.scaling (schema v64). The LP autoscaler mode (off / solver_only / basic / full) is now a per-solve database parameter, not just the run-wide --scaling / FLEXTOOL_SCALING control, so a model can pin the mode that solves it correctly. Motivation: some models with a very wide coefficient range are mis-scaled by autoscale Layer 2 and return a grossly wrong all-slack solution even though the LP is correct (HiGHS solves it directly to the right answer); solve.scaling = basic lets the DB carry that workaround per model. Default is full (unchanged behaviour). Precedence: operator override (--scaling / env) > solve.scaling

  default.

• Scale-invariant coefficient floor (requires polar-high>=3.6.1). In full autoscale mode the coefficient floor previously floored structurally- essential matrix coefficients whose Layer-2-scaled magnitude fell below the threshold, silently mis-solving several battery / demand-response models. The floor is now applied in user-space, so full agrees with off / basic again. Five scenario goldens (dr_shift_demand, test_a_lot, multi_fullYear_battery_nested_*) were regenerated to the corrected solutions; off / basic outputs are byte-identical to before.

Adequacy — representative-period bias levers (opt-in)

• node.energy_margin + energy_margin_method (schema v65). An invest- stage-only lever that scales a node's demand inflow upward during the investment solve to offset representative-period VRE optimism (so the model builds enough firm capacity), while dispatch still runs on the true inflow. energy_margin_method: none (default, off) or inflow_multiplier (apply the energy_margin factor). Note FlexTool represents demand as negative inflow, so a factor > 1 correctly deepens demand.
• group.capacity_margin_method replaces has_capacity_margin (schema v65). The yes/no has_capacity_margin flag becomes a method enum none / manual (migrated yes → manual), consistent with energy_margin_method and leaving room for a future automatic mode.
• Representative-period net-load force-include. New opt-in CLI flags on the representative-period preprocessor force adequacy-critical base periods (peak- load / highest sustained net-load) into the representative set, fixing rep- period under-investment when the sampled periods miss a seasonal stress window (e.g. a coincident low-VRE / high-demand week). Supports grow / fixed count modes, a vg_weight knob, +f{n} naming, and optional node-group demand-weighting (--region-groups) so the net-load signal is weighted by regional demand. Default (no flags / no region groups) is the unweighted, byte-parity path.

Fixes

• Storage nested-fix handoff was inert / inverted. Three fixes to the multi-solve coarse→fine storage handoff: (1) the node__storage_nested_fix_method emit used the wrong column contract (storage_nested_fix_method vs the consumer's method), so every fix-node set was empty and the handoff did nothing; (2) the fix_usage throughput cap was collapsed to a single coarse window instead of summing over every coarse window overlapping the fine period (too tight for any roll spanning >1 coarse window); (3) the fix_price water-value reference price carried the AMPL/GLPK dual sign, which is inverted relative to HiGHS — the child was penalised for holding storage instead of rewarded. multi_fullYear_battery_nested_* goldens regenerated now that the handoff is active.
• Space-bearing entity names in output after an in-process solve. HiGHS' writeSolution permanently rewrites spaces to underscores in the live LP column/row names; the output extractors then failed to join a node whose name contains a space (e.g. Coastal Grid) against the originals (KeyError in calc_capacity_flows, silently-empty dual columns). The un-mangled names are now restored from the polars-rendered Solution.col_names / .row_names. Models without spaces in entity names are byte-identical.

Release 4.0.0b24 (9.7.2026) — warm-start basis cache; greenfield indirect-converter capacity fix; coarse rep-period storage fix

No schema change (still v63). The polar-high floor moves to >=3.6.0 (the warm-start basis cache and the Benders trust-region wiring call new 3.6.0 primitives). Other floors unchanged (polars>=1.40, highspy<=1.14.0). Two result-changing correctness fixes below (indirect converters, coarse rep-period storage) affect models that use those features; everything else is OFF by default and byte-identical on models that don't opt in.

Model — correctness (changes results on affected models)

• Greenfield indirect (process-node) converters now pay for their capacity. An indirect converter (H2 + elec -> [proc] -> NH3) took its capacity- enforcing maxFlow RHS from p_flow_upper (the existing + invest_max ceiling) on every arc, so a greenfield unit (existing = 0) got free, unpaid capacity up to the investment ceiling — the output constraint v_flow − Σ v_invest ≤ invest_max/unitsize was satisfied at v_invest = 0. The indirect maxFlow RHS is now split by arc side, mirroring the direct- unit maxToSink/maxFromSource semantics: output arcs bind with the existing-only RHS (any output flow forces paid v_invest > 0), while input/fuel and zero-coef aux arcs keep the loose bound (pinning them to existing = 0 would make fuel hard-infeasible). Direct units are byte- identical. Sink-less indirect invest units now raise a clear NotImplementedError instead of silently emitting free capacity.

Representative periods — correctness

• Coarse representative-period nodes keep their seasonal storage structure. Coarsening a rep-period run previously dropped the per-representative-period storage seams, collapsing the rep-period structure; storage now keeps per- rep-day seams with a free-start balance. Fixes a companion bug where representative-period labels were nulled on FlexData load (round-trip).

Warm-start — basis cache (opt-in, OFF by default)

• A solve can now reuse a related solve's HiGHS basis to warm-start. New, opt-in machinery keyed by a structural fingerprint of the model's name-set (so a cached basis is only injected into a compatible model), covering both solve paths:
• save-memory subprocess path: the basis is cached as a native HiGHS .bas file and injected into the child via a new cmd_solve_mps --warm-basis (readBasis) hook, with a per-transfer stats sidecar. No variable-name state crosses into the parent (positional-by-construction via the determinism wrapper).
• in-process WarmProblem path: the basis is cached in memory (get_named_basis / set_named_basis) and re-injected on the next compatible solve (sweep / resume — the "UC2/UC3" use cases).
• Safety gate: a first-transfer A/B harness measures the warm vs. cold run and disables warm-start on any regression, so a stale or unhelpful basis can never make a solve worse. All failure modes (added/dropped/ renamed column) fail safe to a correct cold solve. Also fixes a latent bug where the configured work_folder never reached the subprocess.

Solvers — configuration

• highs.opt is now honoured on the in-process solve path (it was silently ignored — the option-file directory was clobbered mid-orchestration). The work folder is also seeded with a solver_config/ skeleton, and the full merged effective solver options are logged before the solve (precedence: baseline < highs.opt < solver_arguments < CLI).

Decomposition (Benders) — mostly experimental, OFF by default

• Region-subproblem autoscale fixes a large-N decomposition crash. Layer-2 (+ Layer-3 HiGHS-native top-up) numerical scaling is now applied to the Benders region subproblems, with a region-solve retry. This resolves an N = 10 mid-solve crash that previously aborted the decomposition.
• Driver rewired onto the polar_high.benders coordinator (byte-parity with the previous in-line loop; pinned by a λ = 0 reference trajectory).
• Master-primal trust-region stabilizer wiring (an honest absolute box on the master coupling point; env-gated, mutually exclusive with in-out) and experimental master-hosted-node mode (split() 4-way arc classification, invest handoff, go/no-go pin diagnostic) — both opt-in and off by default.
• Interior region-dual mode (FLEXTOOL_BENDERS_REGION_DUALS, env-gated) no longer crashes at iteration 1 (barrier primal-dual tolerance).

Release 4.0.0b23 (3.7.2026) — commercial solvers handle spaces in names; opt-in Benders cut compaction

No schema change (still v63). Dependency floors unchanged (polar-high>=3.4.0, polars>=1.40, highspy<=1.14.0). The commercial-solver fixes below change no result on the default (HiGHS) path; the Benders additions are OFF by default.

Solvers — commercial CLIs (Gurobi / CPLEX / Xpress / COPT)

• Commercial solvers now handle entity names containing spaces (common in FlexTool models, e.g. a node named Battery Farm), matching the whitespace- agnostic in-process HiGHS path. The subprocess path previously (a) crashed in the parent with parent failed to read MPS back — it re-read the just-written MPS through highspy.Highs.readModel purely to recover variable names — and (b) once that was removed, could still return a silently wrong answer because free-format MPS is whitespace-delimited and a spaced name corrupts the column. The MPS is now written with generic, whitespace-safe names (write_mps(emit_names=False)) and the solver's solution is mapped back by index onto the real names held in memory (rebuilt from the released Problem's _vars / pre-release _cstrs). No parent-side readModel (which also spiked tens of GB of RSS on large LPs).
• CPLEX / Xpress / COPT .sol parsing fixes surfaced while adding real-solve coverage: COPT writes # Objective value <v> with no = (objective is now parsed); Xpress' default writesol emits an index-based .asc/.hdr CSV that was never parseable, so the Xpress path is switched to the MPS-like SLX solution (writeslxsol, name-based) for the primal plus writeprtsol for the objective. The per-solve temp dir is now removed wholesale so solver sidecars don't leak.
• New real-solve tests exercise the full subprocess path for all four commercial solvers on a model whose entity name contains a space, backed by each solver's Python wheel via a CLI-shaped shim (skip cleanly where a wheel / licence is absent).

Decomposition (Benders) — opt-in, OFF by default

• Periodic master cut compaction (FLEXTOOL_BENDERS_CUT_COMPACT_AT, unset/0 = OFF = byte-identical to before) prunes accumulated master cut rows via polar_high.WarmProblem.compact_cuts, with a slack (default) or non-default dominance selection policy (FLEXTOOL_BENDERS_CUT_POLICY). Note: this is an experimental knob — on load-bearing cut pools it does not reduce iterations.
• Capability guard: compact_cuts needs polar-high>=3.5.0. When the env flag is set but an older polar-high is installed (the floor still allows 3.4.0), compaction is disabled at run start with a clear needs >= 3.5.0 warning instead of crashing mid-solve. Default (OFF) solves are unaffected.

Database migration v62 → v63 (automatic on load). Dependency floors unchanged (polar-high>=3.4.0, polars>=1.40, highspy<=1.14.0). Additive and OFF by default — the converged solution is unchanged on every model.

Decomposition (Benders)

• The in-out separation weight is now a per-solve database parameter, solve.benders_in_out_weight (previously only the machine-local environment variable FLEXTOOL_BENDERS_IN_OUT_WEIGHT from b21). It is the weight λ in the in-out separation point f_sep = λ·centre + (1-λ)·f_out: 0.0 (the default) = off (exact Benders, byte-identical to before); values in (0, 1) turn the stabilisation on, larger = more. Only used when decomposition = 'benders'. The environment variable still works and, when set to a valid value, overrides the database parameter (machine-local wins), mirroring the worker-count knob; an out-of-range or malformed environment value is ignored with a warning and the database value is used. The v63 migration adds the parameter to the solve class (default 0.0, solve_advanced group) — no authored data changes, and a database with the parameter unset behaves exactly as before.

Release 4.0.0b21 (1.7.2026) — Benders in-out stabilization (degeneracy tail-off)

No database migration (schema stays v62). Dependency floor raised to polar-high>=3.4.0 (polars>=1.40, highspy<=1.14.0 unchanged). Opt-in and OFF by default — the converged solution is unchanged on every model.

Decomposition (Benders)

• Optional in-out separation to speed up a slowly-converging (tailing-off) spatial-Benders solve. When storage (or any inter-temporal coupling) makes timesteps fungible, the region recourse is flat in the per-timestep coupling flow and the cut slopes become basis-dependent; the master then wanders among cost-equivalent flow schedules and the lower bound closes very slowly even though the best feasible solution is near-optimal early. In-out separation (Ben-Ameur & Neto 2007) generates each cut at an interior point f_sep = λ·centre + (1-λ)·f_out instead of the extreme master vertex — better- centred cuts, no wandering, faster bound closure, at zero extra subproblem solves (the region is solved once per iteration either way, just at a better point). Enabled per solve via the environment variable FLEXTOOL_BENDERS_IN_OUT_WEIGHT (the weight λ): 0.0 (the default) is OFF and byte-identical to before; λ ∈ (0, 1) turns it on, larger = more stabilisation. On a hydrogen-trade tail-off benchmark, λ ≈ 0.3–0.7 reached the practical optimality gaps about 30% faster and closed the bound tighter than the un-stabilised run (which plateaued above the gap it could otherwise reach). The stabilisation math is the domain-free polar_high.decomposition.InOutStabilizer (3.4.0); FlexTool drives one instance per node group. Correctness is preserved regardless of λ: a cut generated at any point is a valid supporting hyperplane (valid lower bound), the interior point is clamped to the chosen capacity (valid upper bound), and the moment a region's cut fails to separate the master vertex the method falls back to an exact-Benders step for that region — so the optimum is unchanged and convergence is guaranteed. A future release will promote the knob to a solve database parameter once a default is settled.

Release 4.0.0b20 (1.7.2026) — Benders stall guard (fail fast with a diagnostic)

No database migration (schema stays v62). Dependency floor raised to polar-high>=3.3.0 (polars>=1.40, highspy<=1.14.0 unchanged) for the new generic tail-off detector. Detection only — no converged solution changes.

Decomposition (Benders)

• A stalled spatial-Benders solve now fails fast with a plain-English diagnostic instead of silently exhausting the iteration cap. When one node group is near-infeasible on its own (it can only meet its demand via the boundary/slack penalty), the master keeps proposing coupling flows that force the penalty regime, the best feasible cost freezes far above any sane value, and the run would otherwise burn all its iterations returning that garbage. The solver now detects this and aborts with a three-section message that names the worst-offender node group and recommends the fix (give that node group's import/boundary nodes a moderate import price rather than an extreme penalty — an over-large penalty is what inflates the recourse and freezes the bound). The detector is a domain-free tail-off monitor in polar-high 3.3.0 (decomposition.StallMonitor); FlexTool supplies the reference scale (the sum of the node groups' stand-alone costs) and the node-group diagnosis. A run is flagged stalled only when the best feasible cost is frozen for a window of iterations AND still far above that reference AND the gap is far from the tolerance — a conjunction that never trips a slow-but-genuinely-converging solve (validated against solves that converge in 8–28 iterations). The window defaults to 8 and is overridable with FLEXTOOL_BENDERS_MAX_STALL.

Release 4.0.0b19 (1.7.2026) — retiring-unit existing map read + Benders cut-tolerance fix

No database migration (schema stays v62) and dependency floors unchanged (polar-high>=3.2.0, polars>=1.40, highspy<=1.14.0). Two independent fixes: one corrects an output/dispatch parameter for retiring units; the other removes a spurious hard-failure in the spatial-Benders decomposition solver.

Outputs / dispatch

• Read a retiring unit's existing capacity by the per-period MAX, not the last row. _entity_unitsize_lf gated the existing-map cascade input on "period" in columns, but Spine names a Map's index column with its silent-default "x", so the gate never matched and every period row fell through to a unique(keep="last") that kept the last period's value. For a unit whose existing map decays to 0 at expiry, that collapsed the cascade input to 0, defaulted unitsize to 1000, and turned existing_count = existing/unitsize into a spurious fraction — capping continuous online at that fraction and making integer online impossible to commit. The reader now takes the per-entity MAX over whatever period rows exist, index-name-agnostically (CLAUDE.md Invariant #2). Byte-identical for scalar existing (a single-row group-by is a no-op) and for every existing test fixture; corrects the reported online/count for retiring-unit models. Adds regression tests in the silent-default-index coverage file and a dev-doc audit of the readers that live outside the _param_shapes resolver.

Decomposition (Benders)

• The spatial-Benders cut self-check no longer hard-fails on solver round-off. After each master solve, _check_cuts_satisfied asserts every just-appended optimality cut is honoured at the new master point. That check re-derives a row already present in the master LP, so it can only differ from the solved value by the solver's feasibility tolerance — which HiGHS measures on its internally-scaled matrix, making the unscaled slack scale with the cut row's coefficient magnitude, not its right-hand side. On early iterations a node group whose recourse cost overshoots produces large reduced-cost slopes, so cost_r and Σ slope·f̄ nearly cancel: the rhs collapses to O(1) while the coefficients stay at O(1e6). The old tolerance keyed off the cancelled rhs and demanded a precision the ill-conditioned row cannot deliver, aborting an otherwise-converging solve (observed on a 7-node-group hydrogen- trade model: a 7e-4 slack on a 2.66e6-scale row — 2.7e-10 relative). The tolerance is now keyed off the row magnitude, matching the fail-safe design of the sibling flow-clamp / lower-bound / sandwich guards: numerical noise is absorbed, a moderate gap is warned, and only a gross violation (a genuinely un-appended cut, whose recourse estimate sits near its large-negative floor) still hard-fails — now with the same three-section plain-English diagnostic as the other guards. The affected model converges to the monolith objective within tolerance.

Release 4.0.0b18 (30.6.2026) — fix zeroed existing capacity for scalar+Map entity classes

No database migration (schema stays v62) and dependency floors unchanged (polar-high>=3.2.0, polars>=1.40, highspy<=1.14.0). Output-only fix — the converged solution (dispatch, investments, optimal objective) is unchanged.

Outputs

• Fix zeroed existing capacity for a unit whose constant value shares an entity class with a period-Map sibling. When one unit carries a scalar (constant) existing capacity while another unit of the same class carries a period-indexed Map, the Spine reader returns a single frame with one shared index column (named x by Spine's silent default) in which the constant unit's row has a null index. The per-entity resolver classified rows by the presence of that column rather than per-row, so the null-index scalar became an explicit (entity, period=null) row that never joined the period grid and was silently filled with 0. Downstream this zeroed the output-facing existing/total capacity for such units — surfacing most visibly as negative VRE curtailment (potential − flow, with potential computed from a 0 capacity), and as 0 in the capacity, capacity-factor, and per-entity pre-existing fixed-cost reports. The resolver now detects scalars per row (a null index broadcasts the constant across the period universe), in both mirror implementations. The LP itself was never affected — its flow bound reads a separate, scalar-correct existing-count parameter, and existing fixed cost enters the objective only as an opt-in constant term — so this is purely a reporting fix; re-running output processing on an existing solve produces correct reports without re-solving.

Release 4.0.0b17 (30.6.2026) — autoscale centres the objective (no more "excessively small costs")

No database migration (schema stays v62). Dependency floor raised to polar-high>=3.2.0 (polars>=1.40, highspy<=1.14.0 unchanged). The converged solution is unchanged on every model.

Numerical scaling

• The "Problem has some excessively small costs" HiGHS warning is fixed at the source. FlexTool multiplies the whole objective by the legacy scale_the_objective = 1e-6, which drags every cost coefficient ~6 decades down (e.g. a typical operational-cost band lands around [5e-5, 6e-1], below HiGHS' 1e-4 floor). polar-high 3.2.0's autoscale Layer 3 now geometrically centres the cost band over HiGHS' comfort zone via a power-of-two user_objective_scale: it lands the band straddling 1.0, clearing the warning and giving the cheapest costs several decades of headroom above the dual-feasibility tolerance — which helps duals (node/reserve prices) converge on degenerate dispatch LPs. The exponent is a power of two and HiGHS unscales the objective and duals on output, so the reported solution is byte-for-byte unchanged — only the magnitudes the simplex pivots on. Bands already in HiGHS' zone are left untouched, so well-scaled models see no change.

Housekeeping

• Genericized a handful of test fixtures and source comments (synthetic entity names in place of model-specific ones); no behavioural change.

Release 4.0.0b16 (30.6.2026) — Benders fail-safe numerical guards + clearer failure diagnostics

No database migration (schema stays v62) and dependency floors unchanged (polar-high>=3.1.0, polars>=1.40, highspy<=1.14.0). Robustness fixes to the Benders regional-decomposition driver plus a docs note; a model that does not use regional decomposition is unaffected, and the converged solution is unchanged.

Benders regional decomposition

• Clamp the master coupling flow to capacity instead of tuning the tolerance. The post-master-solve f <= cap self-check kept tripping on benign solver slack: HiGHS enforces feasibility on the internally-scaled master, so the unscaled slack exceeds both the nominal tolerance and the reported max_primal_infeasibility, and it grows as cuts accumulate (~3e-6 at iter 4, ~1.2e-5 at iter 8) — a fixed tolerance chases a moving target (this supersedes the b14 tolerance-from-feasibility approach). Instead, the upper bound is now evaluated at the strictly capacity-feasible point (C, min(f, cap)): clamping flow down can only raise region recourse cost, so it stays a valid whole-problem UB and the optimality cut stays valid (a subgradient inequality holds at any linearization point). The clamped flow is used consistently downstream (region pin, next cuts, incumbent); only a gross overshoot (>1e-2 relative, or 1e3× the solver's reported infeasibility) still hard-fails as a genuine read/stale bug. Clamp magnitude logged at DEBUG.
• Fail-safe LB guards. The LB-monotonicity (1e-6) and LB <= best_UB sandwich (1e-9) self-checks no longer hard-crash the whole run on benign numerical noise (discarding a good feasible incumbent). Both now separate noise from a genuine invalid-bound bug via a unified gross band max(tol, 1e-3): a small LB dip pins LB back to the previous valid bound and continues; a small LB/UB crossing is treated as converged (stop on the incumbent); a gross dip or overshoot still hard-fails (stale basis / corrupted cut / invalid-bound pathology).
• Plain-English failure diagnostics. Every hard-failure site (master / node-group not optimal, LB dip, LB > UB, coupling overshoot) now emits a one-line summary plus "What this means" and "How to avoid it", using only FlexTool class vocabulary (node, group, connection, flow) — no model-instance terms — pinned by a static AST regression test.

Documentation

• Recommend setting an explicit Map index_name (period or time) for parameters that accept both a period map and a time series (e.g. flowGroup min_instant_flow / max_instant_flow), so the value-domain fallback never has to guess; cross-linked from the flowGroup flow-limit reference.

Release 4.0.0b15 (30.6.2026) — flowGroup instant-flow obligation fix

Fixes a flowGroup min_instant_flow / max_instant_flow limit being silently ignored — or crashing at load — depending on how the limit was authored. No database migration (schema stays v62; FLEXTOOL_DB_VERSION unchanged) and dependency floors are unchanged (polar-high>=3.1.0, polars>=1.40, highspy<=1.14.0). Models that set no instant-flow limit on a flowGroup are unaffected.

flowGroup flow limits

• Emit the instant-flow constraint for every authored shape. The min_instant_flow / max_instant_flow obligation's RHS cap was resolved correctly for all shapes (constant, period map, time map, period+time — including Spine's silent-default "x" Map index_name), but the constraint's (g, d, t) support was built by a separate raw-source projection that detected the index axis by column name. That projection returned an empty support for an "x"-labelled period map, a constant, or a time map — so the >= / <= constraint was never emitted and the limit was silently ignored — and it crashed (ColumnNotFoundError: "t", at apply_projection_params) on a pure period map, which has no t column. Spine Toolbox cannot cleanly distinguish period from time maps, so authors routinely leave index_name as "x", landing in the broken path. The support is now derived directly from the resolved cap and broadcast against the active (d, t) grid — exactly as the cumulative-flow limits already are — so support and RHS stay in lock-step and every authoring shape binds correctly. A regression test covers each cap shape plus the reported field case: an "x"-indexed period map with a non-zero first period and a zero later period under a single-period solve.

Release 4.0.0b14 (29.6.2026) — Benders master coupling self-check tolerance fix

Raises the solver-backend floor to polar-high>=3.1.0 (the Benders master coupling self-check now reads Solution.max_primal_infeasibility, added in 3.1.0). No database migration (schema stays v62); other floors unchanged (polars>=1.40, highspy<=1.14.0). A model that does not use regional (Benders) decomposition is unaffected.

Benders regional decomposition

• Size the master coupling self-check from the solver's achieved feasibility. The post-master-solve self-check that the chosen flow is supported by the chosen capacity (f <= cap) used a hard-coded 1e-6 absolute tolerance. For a unitsize-normalised coupling row C - f >= 0 with small capacity, HiGHS enforces primal feasibility on the internally-scaled row (default 1e-7), which maps to a larger unscaled slack on f <= cap — a normal solver artifact, not an invalid bound. On a full-scale model this tripped at iteration 4 (slack ~3e-6 > tol 1e-6) and aborted an otherwise-valid solve. The tolerance is now derived from the master solve's own Solution.max_primal_infeasibility (with a small relative floor), and the error message reports the actual slack vs. tolerance so a genuine, orders-of-magnitude coupling violation still surfaces loudly.

Release 4.0.0b13 (25.6.2026) — scenario-comparison & nodeGroup-dispatch plot fixes

Output/visualization-only bug fixes — no database migration (schema stays v62; FLEXTOOL_DB_VERSION unchanged) and dependency floors unchanged (polar-high>=3.0.0, polars>=1.40, highspy<=1.14.0). The LP, engine, and all solver outputs are byte-identical to b12; only scenario comparison and the nodeGroup dispatch plots change. Each fix carries a regression test.

Scenario comparison

• Retag single-level scenario columns to folder identity. combine_parquet_files re-tags each scenario's baked-in model tag to its folder name so two folders holding the same model scenario stay distinct — but the retag was gated on MultiIndex columns only. Entity-less, period-indexed variables (e.g. costs_discounted_p_) carry a single-level column Index named scenario and so skipped it; two compared folders sharing a baked tag (e.g. trade_only_shipping_4 / _2, both baked trade_only_shipping) then produced duplicate columns and crashed plot_outputs' stack() ("Columns with duplicate values are not supported in stack"). Adds the single-level retag branch.

NodeGroup dispatch plots

• Correct consumer sign + restore connections. Consumers no longer render as producers (the node→unit path double-negated the already-negated unit_inputNode_dt_ee); non-aggregated connections (battery/transport) load unconditionally instead of only when an aggregated connection exists; and the redundant not-in-aggregate "total" connection path that double-counted is dropped. Fixes the matching per-node consumer bug too.
• Keep both directions for same-node bidirectional units. A unit that both produces to and consumes from the same node (reversible/storage with input_node == output_node) no longer has its production side overwritten; the mixed-sign column is split into _pos/_neg so charge and discharge both show.
• Read connection flow by the node's physical end, not flow direction. connection_leftward/rightward are keyed by the connection's physical end node, but the plot picked a frame by the mapping table's flow direction, so a unidirectional connection whose in-group node sits on the "wrong" end vanished from the plot (while correct in the engine's authoritative nodeGroup_flows). Both paths now use a single end-correct lookup (_connection_series_at_node).

These plots now agree with the engine's authoritative group-flow output; real elec-group plots are byte-equivalent where the old code happened to capture everything, with regression tests added for the cases it dropped.

Release 4.0.0b12 (25.6.2026) — Benders regional decomposition (v62) replaces Lagrangian; greenfield cross-region trade; GUI fixes

Requires a database migration to v62 (FLEXTOOL_DB_VERSION 61 → 62) and raises the solver-backend floor to polar-high>=3.0.0 (3.0.0 removes the old LagrangianProblem driver and ships the Benders cut-append / warm-restart / parallel primitives this release uses); other floors unchanged (polars>=1.40, highspy<=1.14.0). This is a breaking schema change: the spatial regional decomposition is changed from the dual-subgradient (Lagrangian) scheme (introduced in the unreleased b11) to a Benders decomposition. Existing databases are auto-migrated by the v62 step; a model that does not opt into decomposition solves exactly as before (the monolithic LP and all outputs are unchanged).

Benders regional decomposition (schema v62)

• New algorithm. A coordinating master problem holds the inter-regional trade flows and the trade-connection investment plus one recourse variable per region; each region is an operational subproblem solved with the master's trade fixed, returning duals that become optimality cuts the master accumulates. The master objective is a valid lower bound and the loop converges on the optimality gap. This fixes the earlier subgradient scheme, which collapsed greenfield cross-region trade to autarky with an invalid bound above the true optimum.
• Schema renames. The decomposition_schemes value-list member lagrangian → benders, and the decomposition_methods member lagrangian_region → benders_regional (every authored solve.decomposition / group.decomposition_method value is rewritten in place by the migration).
• Knobs. The solve.lagrangian_alpha / lagrangian_max_iter / lagrangian_tolerance parameters are dropped (Benders has no subgradient step) and replaced by solve.benders_max_iter (default 50) and solve.benders_tolerance (default 1e-3, the relative optimality-gap threshold).
• Invest → dispatch handoff. A Benders invest solve assembles the owner-selected invested trade + in-region capacity into a whole-system handoff so a downstream rolling or monolithic dispatch solve consumes the realised capacity end-to-end.
• RP-weight consistency. The master is built by build_flextool over a network-only reduced model — the same emit the monolith uses — so the master trade-flow cost and the region recourse costs carry the same representative-period / timestep weights and sum to the monolithic objective.
• Warm master + parallel regions. The master is grown one optimality cut per iteration and warm-re-solved off the retained basis (WarmProblem.solve(retry_on_unknown=True)) instead of rebuilt, removing the super-linear cold-presolve cost at scale; the region recourse subproblems solve in parallel via polar_high.parallel. The worker count is the machine-local FLEXTOOL_BENDERS_WORKERS env var (0/auto = cpu − 1; no DB/schema knob, so a many-core author's count never travels to a small box) — superseding the b11 --lagrangian-workers flag / GUI knob, which never shipped.
• Implementation. Driver in flextool/engine_polars/_benders.py (master + multi-cut loop, solve_benders / BendersResult); region slicing in flextool/engine_polars/_region_filter.py. Uses the polar-high primitives WarmProblem.add_cut_row / add_recourse_col / solve(retry_on_unknown=…) and the polar_high.parallel helpers (floor raised to polar-high>=3.0.0). Benders is HiGHS-only (the master is a persistent WarmProblem).
• Stochastic branch weights reach the objective. Confirmed (and pinned by a regression test) that per-branch probability weights from solve.stochastic_branches flow through the derived cascade into the objective — the stochastic twin of the representative-period-weight fix above; the dense 1.0 baseline set in apply_derived_a is superseded by the real sibling-normalised weights in apply_derived_g.
• Clean Benders progress log. The per-sub-solve native HiGHS output (master + N region solves each iteration) is muted by default via WarmProblem.set_output_flag(False); instead the orchestrator streams a concise per-iteration LB / UB / gap line and a lock-guarded "iter k: done (obj=…)" line as each region finishes (fired from the parallel worker via a new solve_benders subsolve_callback). The start line carries the run config inside its tag so it stays visible past the long region list. Set FLEXTOOL_BENDERS_VERBOSE=1 to restore the full native HiGHS log.

Desktop GUI

• Fix a crash when the migration dialog's grab_set fires from a callback on a not-yet-viewable window.
• Smaller default code/log font, with the code font size now exposed in settings.
• Simplify the Add-to-execution button to a static label driven only by the checked rows (enabled when ≥1 scenario is checked, disabled otherwise), dropping the per-scenario "Create vs. Update" results-on-disk detection.

Release 4.0.0b11 (24.6.2026) — per-solve Lagrangian decomposition (v60/v61); thread-parallel subsolves; xlsx round-trip + output fixes; per-monitor GUI placement

Requires a database migration to v61 (FLEXTOOL_DB_VERSION 59 → 61, in two steps): v60 adds the per-solve solve.decomposition selector and its Lagrangian knobs; v61 adds timeset.representative_period_weights. Raises the solver-backend floor to polar-high>=2.9.0 (the per-solve decomposition path and the new --lagrangian-workers knob use its thread-parallel subsolves / per-subsolve callback); other dependency floors unchanged (polars>=1.40, highspy<=1.14.0). A model that does not opt into decomposition solves exactly as before — the LP and all outputs are unchanged for the monolithic path.

Per-solve Lagrangian decomposition (schema v60)

• DB-driven decomposition. A solve now selects its own scheme via the new solve.decomposition parameter (value list decomposition_schemes); set it to lagrangian to decompose that solve's investment problem by owner region. The Lagrangian behaviour is tuned per solve by lagrangian_alpha, lagrangian_max_iter and lagrangian_tolerance (grouped under solve_advanced). The old --decomposition* CLI flags are removed — the choice lives in the database, not the command line.
• Investment → dispatch handoff. A decomposed investment solve assembles the owner-selected invest/divest variables and deposits the resulting capacities for the downstream rolling dispatch solve to consume, so a Lagrangian invest stage hands its build decisions to an ordinary dispatch stage end-to-end.
• Live progress in the log. The Lagrangian driver now prints a per-iteration line plus a closing dual-gap summary block; the progress callback is forwarded to polar-high only when the installed version supports it.

Thread-parallel Lagrangian subsolves

• --lagrangian-workers N (CLI; mirrored to FLEXTOOL_LAGRANGIAN_WORKERS) and a machine-local "Parallel workers (Lagrangian)" spinbox in the GUI solver-options dialog (0 = auto = cpu − 1) cap how many region subproblems solve concurrently within each barrier. The count is plumbed to polar-high's max_workers and a per-subsolve progress callback, both version-gated via inspect.signature — an older polar-high (which floors at 2.9.0 here, but the guard is defensive) silently stays on the sequential path. The worker count is a per-machine setting (GUI YAML, not a DB/schema parameter), so a many-core author's count never travels to a small box. Default behaviour is unchanged — with no flag the solve stays sequential.

Representative-period weights (schema v61) & xlsx round-trip

• New timeset.representative_period_weights (rank-2 nested Map: base-period start → representative-period start → weight) produced by the representative-periods preprocess tool; its presence enables RP-aware storage binding (*_blended_weights), its absence silently downgrades those methods to non-RP behaviour. Mutually exclusive with timeset_weights on the same timeset. Its Map axes are named for a clean Excel export.
• Excel round-trip correctness. Several DB → xlsx → DB corruptions are fixed: constant/periodic-shared scalars no longer duplicate; float leaves and the period axis survive Map round-trips; ladder parameter identity, tier order and solve-parameter export are preserved; empty Array/Map values round-trip via a sentinel; and a generic multi-index nested-float Map path fixes a latent stochastic 3-D/4-D drop.

Output & plot fixes

• Node prices for coarse-resolution block nodes. Nodes on a coarser new_stepduration resolution land in the block balance equation; the per-(d,t) price output extracts and broadcasts that block dual instead of raising a node_prices_dt_e KeyError.
• Cost-by-entity comparison plot is now grouped + stacked correctly (and a frivolous scenario_rule is dropped).
• Dispatch plots: build each scenario's dispatch DataFrame once to avoid pandas fragmentation; silence a degenerate dispatch y-limit; correct the connection_dt_eee comparison column spec.

Input fix

• Cast p_flow_max values to Float64 on read, so an integer-typed limit no longer trips a dtype mismatch downstream.

Desktop GUI

• Per-monitor window placement. Child windows (Execution-jobs / Results) open on the main window's monitor and remember their geometry per monitor configuration; placement uses screeninfo for flicker-free, multi-monitor-aware positioning. Fixes the earlier single-monitor winfo_screenwidth() assumption that could strand a child window off-screen.
• Windows DPI font scaling, header style and the Add-input-source dialog sizing are corrected; startup logs ground-truth DPI / font-scaling for diagnosis.

Release 4.0.0b10 (16.6.2026) — never open child windows off-screen

GUI-only release — no database migration (schema stays at v59; FLEXTOOL_DB_VERSION unchanged) and dependency floors unchanged (polar-high>=2.6.0, polars>=1.40, highspy<=1.14.0). The LP, engine, and all outputs are byte-identical to b8.

Desktop GUI

• Never open the Execution-jobs / Results window off-screen. Both placed themselves to the right of the main window (main_x + main_w) whenever the screen was wider than 2400 px; with b9's full-screen main-window sizing (main_w ≈ screen_w) the child opened entirely off the right edge — which Windows renders as a black, unrecoverable alt-tab thumbnail — and winfo_screenwidth() only sees the primary monitor, compounding the bad offset. Side-by-side placement is now gated on real free space to the right (exec_x >= 0 and enough room for the minimum width); otherwise it falls back to the overlap layout that always stays fully on the primary screen.

Release 4.0.0b9 (16.6.2026) — live free-RAM execution admission; deterministic Windows tk scaling

GUI-only release — no database migration (input schema stays at v59; FLEXTOOL_DB_VERSION unchanged) and dependency floors unchanged (polar-high>=2.6.0, polars>=1.40, highspy<=1.14.0). The LP, engine, and all outputs are byte-identical to b8.

Desktop GUI

• Live free-RAM execution admission (replaces the static per-job reservation). Raising "max parallel executions" no longer fails to start new scenarios while wrongly reporting "memory limited": the old admission summed each running job's dispatch-time budget snapshot against total RAM minus a reserve, so with N jobs running the snapshots already filled the headroom and raising the worker count never freed room (a permanent false memory-limit). Admission now keys off actual live free RAM (vm.available), re-measured every scheduler tick, with a per-job estimate ladder (explicit cap > learned history ×1.05 > auto fair-share) and a predicted-available figure that subtracts only the unrealised growth of still-warming jobs.
• Deterministic Windows tk scaling = dpi/72. Tk on Windows does not auto-scale for display DPI (it stays pinned at the 96-DPI baseline), so point-sized fonts rendered tiny on a scaled screen while the old (dpi/96)*0.85 fudge plus a raise-only guard made the result wildly machine-dependent. With the process DPI-aware, tk scaling is now set to the documented pixels-per-point dpi/72 unconditionally on Windows (1.0 at 100 %, 1.5 at 150 %, …); Linux keeps its tuned 96-baseline raise-only behaviour.

Release 4.0.0b8 (16.6.2026) — small-number coefficient cutoff (v59); legacy Excel import; solver-options GUI

Requires a database migration to v59 (FLEXTOOL_DB_VERSION 58 → 59; adds the model.small_number_threshold parameter). Raises the solver-backend floor to polar-high>=2.6.0 (the cutoff sets its new coef_zero_threshold); other dependency floors unchanged (polars>=1.40, highspy<=1.14.0). LP-neutral by default — the cutoff is a no-op at its 0.0001 default unless a model carries coefficients below it, and the rest of the release is fixes and importer/GUI work.

Model-wide small-number coefficient cutoff (schema v59)

• New model.small_number_threshold (float, default 0.0001): LP matrix coefficients and RHS terms whose absolute value is below it are floored to 0.0 when building the problem, narrowing the LP's numerical range (conditioning). The floor is a generic polar_high feature (Problem.coef_zero_threshold) applied at every coefficient/RHS finalize point; FlexTool reads the parameter (spec → provider → FlexData) and sets it in build_flextool. Registered in autoscale PARAMETER_TYPES; v59 migration adds the parameter; schema, canonical DBs and fixtures regenerated.

Legacy (pre-v25) specification-Excel import

• The specification-format Excel importer now handles older template layouts (changelog v12+): coerces numpy scalars to native Python before to_database (fixes an int64 crash), builds an empty unstacked frame for header-only parameter sheets (instead of a pandas "need at least one array" raise) while still reporting genuine errors with the sheet name, remaps renamed legacy parameters (period_timeblockSet→period_timeset, block_duration→ timeset_duration, has_state→has_storage, variable_cost→ other_operational_cost) and aliases the pre-v25 time-structure sheets. The GUI conversion flow is hardened/clarified.

Desktop GUI — solver options

• Fix the solver-options dialog never opening (it referenced a non-existent self.root; MainWindow is the Tk root). Add a MIP relative gap control (a checkbox gates emission — off defers to the .opt baseline / solver default; default 0.001, 0 valid) routed GUI → --solver-mip-gap → FLEXTOOL_HIGHS_MIP_GAP, mirroring the time-limit knob. Presolve "choose" is now actually forwarded (was silently dropped to the engine's pinned "on"). Add runtime-file bootstrap. Clamp the main-window width to the screen (only height was clamped) so a DPI-aware natural width can't run off a small display.

CO₂ total-cap & engine fixes

• Report CO₂ for co2_max_total (total-cap) models: the emissions output set wired only the period-cap and price flow sets; the multi-period total-cap port (flow_from_co2_capped_total, group_co2_max_total) was never added, so a co2_method=total model (e.g. SouthAfrica) reported CO2 [Mt] = 0 despite a correct engine cap.
• Read the CO₂-accumulator unitsize on its actual entity axis — the rolling-accumulator preferred the e-keyed p_all_entity_unitsize carrier but unconditionally selected p, crashing the output stage on the first co2_method=total model after an Optimal solve.
• Key the profile-flow source on the real input-node arc: a direct (constant-efficiency) unit carrying an output-node profile keyed the constraint on (unit, unit, sink) instead of the real v_flow[unit, input_node, sink] arc, matching nothing → empty row → HiGHS presolve Infeasible (e.g. a nuclear unit with a fixed output profile).
• Make the v28 interest/discount rename collision-safe: it now checks the target name is free, so a DB that already carries both names (reachable via an interrupted migration + template re-apply) no longer raises.

Output fixes

• Empty the scenario output_parquet/<scenario>/ dir before writing so retired/ renamed outputs don't linger and trip the scenario-comparison "not registered" warning (done late, after results is built, so a failed run keeps prior outputs).

Release 4.0.0b7 (11.6.2026) — self-documenting output metadata; per-entity cost break-down; unit corrections

No database migration — the input schema stays at v58 (FLEXTOOL_DB_VERSION unchanged). The results SpineDB schema grows (six new per-entity cost parameters + back-filled metadata descriptions; pinned parameter-definition count 52 → 58), but results DBs are write-only outputs, so there is nothing to migrate. Dependency floors unchanged (polar-high>=2.5.0, polars>=1.40, highspy<=1.14.0). LP-neutral: the solve and every system-level total are byte-identical to b6 — the changes are output presentation, self-documenting metadata, additive per-entity outputs, and reported-unit corrections. One entity-first dt golden (group_flow__dt.csv) is regenerated for a fixed solve-column, and a handful of reported values are corrected to ground-truth units (CO₂, invest-marginal — labelling, not LP changes).

Self-documenting output metadata

Output values now carry their own unit + semantics, derived once from the output-stage transform and surfaced everywhere — so the meaning of a column can't drift from the code.

• A single-source _output_meta derives per-column (unit, semantics) from the transform each output applies; 84/84 processed outputs are declared, with per-column maps for mixed-unit tables. The ~22 inlined dt→d period- annualization expressions are centralized onto one annualize_dt_to_d helper (behaviour-preserving, golden-byte-identical including timestep_weight≠1).
• Metadata is embedded in the parquet footer and surfaced in every renderer: a CSV datapackage.json sidecar (the CSVs themselves stay byte-identical), an Excel metadata sheet + header-cell hover comments, Spine fills the empty parameter_definition descriptions at import (rich hand-written ones are never clobbered), and plots / the result viewer label the value axis and show a hover tooltip — including the on-the-fly a (total) and w (weekly) variants. result_key_summary() / result_variant_summary() back the hovers.
• Reported-unit corrections to ground truth: invest_marginal → CUR/kW(h) (a single ÷1000 at the dual-synthesis choke point); per-entity CO₂ is t/a (the schema's Mt was 1e6 too large) and the system total is Mt-over-horizon; dt group/node flows and raw inflow are the un-integrated MW level (not per-step MWh); startup_cumulative is units/a; and default_plots.yaml value-axis units are corrected (nodeGroup energy, marginal investment, entity_annuity = CUR/MW/a). Schema-vs-_output_meta drift is guarded by a CI ratchet rather than per-run log spam.

Per-entity cost break-down

Six additive, period-level outputs collapse every cost category per entity instead of system-wide: cost_{unit,connection,node}_{annualized,discounted}_d_ec (annualized M CUR/a + discounted M CUR). No existing system total changes.

• Fuel cost is attributed to the consuming process; CO₂ cost is summed over every priced group a process touches. Built by vectorized slice+scale (no per-entity / per-period loops).
• Emitted to CSV, the results SpineDB (Map(category → solve → period → value) on the unit/connection/node classes; six new schema params, count 52 → 58), and plots ("Cost by unit/connection/node" — stacked categories on period bars, one subplot per entity, annualized + discounted, comparison grouped by scenario). Registered in scenario-comparison loading.
• Reconciliation tests: the cross-kind per-entity sum equals the system summary per category and period (rtol 1e-9), with explicit non-zero value checks for starts / commodity_sales / fixed cost invested; 13-scenario goldens.

Plot & output fixes

• Solve-column injection is keyed off the named period / time index levels instead of positional slots, so an entity-first dt output (e.g. group_flow__dt with a (group, period, time) index) no longer emits a blank solve column. Behaviour-preserving for all period/time-first indexes; one affected golden regenerated (other values byte-identical).
• Bar value-label geometry: orientation-aware slot floor, thin single bars under labels, slot room scaled with label thickness; variant-aware units for the a / w plot variants.

Desktop GUI

• Open .sqlite results DBs in the Spine DB Editor (with a fallback popup when it is unavailable); DPI-aware sizing fixes; output-metadata hover in the result viewer.

Fixes & tests

• NaN-guard the capacity factor when capacity is zero.
• Migrate the input_entity_colors test fixtures to the v58 top-level flowGroup class, and regenerate the base_weighted node__d.csv golden for the timeslice-weight-aligned dt→d aggregation.

Release 4.0.0b6 (10.6.2026) — period energy balance; flowGroup carve-out; Spine Toolbox ↔ project interoperability

Requires a database migration to v58 (FLEXTOOL_DB_VERSION 57 → 58; carves flowGroup out of the group class — see below). Raises the solver-backend floor to polar-high>=2.5.0 (it routes the HiGHS log through Python's sys.stdout, so the solver log is visible under the Spine-Toolbox Basic Console on Windows — see the Toolbox section); other dependency floors unchanged (polars>=1.40, highspy<=1.14.0). The release is LP-neutral / byte-identical to b5 on all existing fixtures; the sole results change is the balance_within_period correctness fix, which only moves numbers for nodes that actually carry that method under non-uniform timeslice weights.

Period energy balance — balance_within_period nodes

Ports the legacy balance_within_period node type to the polars engine (nodeBalancePeriod_eq): a per-(node, period) energy balance for annual-budget nodes (e.g. a yearly gas-import node), closing a bug where such a node was a free unconstrained source.

• Emit nodeBalancePeriod_eq — one row per (n, period), each flow/slack term summed over t with RHS -Σ_t inflow, no storage / self-discharge terms, excluding nodes that also carry a state; the vq_state_up/down slack domain is widened to nodeBalance ∪ nodeBalancePeriod. Gated to a no-op (and collapses to the per-dt nodeBalance_eq) when no period nodes are present.
• Annualize every period-balance term (flows, sections, back-flow, slacks, and the inflow RHS) by the per-(d,t) timeslice representativeness weight that the cost objective, demand scaling, and the dt→d energy/emission outputs already use — so a representative/timeslice-weighted solve conserves physical annual energy, not the unweighted timeslice sum (verified: per-period import/draw 0.838 → 1.000000, slack 0). The per-dt balance correctly omits the weight; only a period balance annualizes.
• Rename the internal p_rp_cost_weight → p_timestep_weight (Python field, builder, output attribute, and wire-key): it is a temporal representativeness weight — neither RP-specific nor a cost weight — that drives demand, energy and cost annualization alike. No autoscale-registry or schema change.

flowGroup carve-out (schema v58)

Splits the overloaded SpineDB group class: group keeps the node-balance / system / investment / structural concerns, and a new flowGroup class (3-dim (process, node) membership) owns the flow limits and aggregation.

• New flowGroup entity class + group__unit__node / group__connection__node re-homed to flowGroup__*; the engine read/solve path (flow-limit readers, the 3-dim membership index-set populator, _param_shapes allow-list, autoscale PARAMETER_TYPES) now resolves the four flow-limit params against flowGroup.
• flow_aggregator becomes a method enum (none / dispatch_plots_only / standalone_aggregator_only / both) that subsumes the dropped boolean output_flowGroup_indicators; output_nodeGroup_dispatch / output_nodeGroup_indicators are renamed to print_dispatch / print_indicators.
• A double-count overlap warning: a flow arc bound into two dispatch-bound aggregators (dispatch_plots_only / both) is named loudly — this was the root-cause class that motivated the carve-out; overlap among standalone aggregators is legitimate and not warned.
• New signed group_flow__dt.csv standalone timewise output; fixes a latent group_process_node backfill bug; heals missing DC-power-flow parameter metadata. Migration authored with corrected flowGroup / print_* descriptions. Validated LP-neutral by the full scenario suite.

Spine Toolbox ↔ project interoperability

The Spine Toolbox workflow becomes a first-class peer of the FlexTool GUI: a Toolbox run now roots all its outputs at a user-chosen project folder, the per-project plot_settings.yaml is honored, and a results SpineDB can be produced from the processed parquet without re-solving. No forced migration — existing workflows keep working unchanged.

Project-folder output rooting

• A Toolbox run roots output_parquet/, results.sqlite, plots, and the per-project plot_settings.yaml at a project folder instead of the repo root. The folder is chosen by editing one line in the gitignored, user-local file templates/project_folder.txt (seeded by flextool-update). The path may be absolute or relative to the FlexTool root; blank / comment-only reproduces the old repo-root behavior. Editing the file never dirties git, so switching projects needs no committed change.
• This replaces the old flextool_location.txt mechanism. The FlexTool run Tool's command argument changed from --flextool-location templates/flextool_location.txt to --project-folder-file templates/project_folder.txt. The legacy --flextool-location CLI argument still works as a fallback for old / forked setups.

Results SpineDB from parquet replay

• output-spinedb=true in the Output settings DB writes <project>/results.sqlite (each scenario as a Spine alternative) from both workflow steps: the FlexTool run produces it from the live solve (with all params), and the "Re-create results" step (re)builds/augments it by replaying the processed parquet — no re-solve (e.g. to add the SpineDB afterwards without re-running, or to reduce the scenario set via its filter).
• Caveats on the replay path only: the two inflation/discount-factor params are omitted (they need the live solve); for a bidirectional connection the (source, sink) byname uses the parquet (node_1, node_2) geometry (1-way connections are exact). The native run path writes the full set.
• output-spinedb now defaults to true in the Toolbox output-settings template, so a fresh project gets results.sqlite without flipping the option.
• The cold-start results-DB build is cross-platform: the create-from-JSON is serialized by an exclusive sidecar lock (fcntl.flock on POSIX, msvcrt.locking on Windows) and the atomic publish retries the transient Windows "file still in use" (WinError 32) os.replace, so parallel scenario writers into one shared results.sqlite no longer race or fail on Windows.

Per-project plot_settings in Toolbox

• A plot_settings.yaml placed in the project folder is now honored by both the per-scenario plots and the comparison plots in the Toolbox track (previously it silently used the bundled default).

GUI interoperability

• Outputs under projects/<Name>/output_parquet/<scenario>/ are picked up automatically by the FlexTool GUI (the folder name is the result identity; a Toolbox run shows under "source 0" — cosmetic). Conversely, the Toolbox can run a GUI-made project by pointing "Input data" at the project's input DB and setting project_folder.txt to that project.
• Spine-Toolbox Basic Console (Windows): the run survives the Console's python -i REPL, and the full HiGHS solver log is now visible there — the latter via polar-high>=2.5.0, which re-emits the log through sys.stdout (the Console only captures the Python stream, not the native fd-1 write).

Migration

• Run flextool-update — it seeds templates/project_folder.txt and refreshes the settings DBs. The refresh now back-fills new parameter definitions (e.g. output-spinedb) into existing settings DBs that predate them — schema only, so existing user choices are preserved — so options added in later versions appear in the editor without rebuilding the DB by hand.
• Only if you forked / customized .spinetoolbox/project.json: update the FlexTool Tool's command argument from --flextool-location … to --project-folder-file <project>/templates/project_folder.txt.

Release 4.0.0b5 (8.6.2026) — multi-solve output union; plot colour/order system; timeslice-weight alignment; SpineDB output; Python 3.13

Requires a database migration to v57 (FLEXTOOL_DB_VERSION 56 → 57; timeslice-weighting doc descriptions). New dependency floor polars>=1.40 (older polars silently bypasses autoscale Layers 2/3 — see below); polar-high>=2.4.0 and highspy<=1.14.0 unchanged. This release contains model and constraint changes; they are byte-identical to b4 where the relevant inputs (timeset weights, capacity coefficients, availabilities) are uniform/absent.

Multi-solve output unioning

A nested / multi-solve run (e.g. invest → dispatch, or several rolling solves) now unions each sub-solve's realized outputs into one complete result set instead of letting the last writer win.

• A cross-solve OutputTimelineBundle (realized dispatch + invest union, with [d,t]→solve / period→solve maps) is built once from the authoritative realized timelines; output writers consume it to union per-roll param/set slices, oracle-parity with a single-solve run.
• A raising disjointness guard replaces the legacy silent earliest-wins trim: overlapping realized (period, timestep) or invest period across solves is now a FlexToolConfigError naming both solves, not a masked truncation.
• Deterministic period-share annualisation: the per-period dispatch denominator is reduced by max-per-period, fixing an order-dependent dedup that could intermittently inflate dispatch totals/costs (≈120× on short invest windows) depending on filesystem glob order.
• The realized invest-eligible set is unioned so the invested capacity breakdown survives a dispatch-only child roll; invest cost inputs union across sub-solves.

Plot colour & ordering system

A per-project, editable colour and stacking-order system for all plots, driven by <project>/plot_settings.yaml (falls back to the bundled default when absent; byte-identical output for projects without one).

• Colours/order resolve from the project file through both PNG export and the result viewer; entity colours are auto-seeded from the input DB at job start (additive, append-only) so the list is complete and editable while scenarios run.
• New GUI "Colors, order…" picker: a non-modal window with an embedded HSV colour picker (SV square + hue slider, no Pillow), side-by-side positive/negative swatches with an auto-breaking link, drag/keyboard row reordering, multi-select group moves, Refresh-from-DB, and Undo/Redo.
• Colour simple bars / time-series / dispatch by entity class (color_entity_class), including subplot- and file-split entities; the group class is split into nodeGroup / flowGroup by membership.
• Comparison plots colour their scenario series from an editable scenarios section; the dead config.yaml system is removed.

Timeslice / representative-period weighting alignment

One representative weight (p_rp_cost_weight, = 1 without timeset_weights) is now routed through inflow demand-scaling and every extensive dt→d energy/emissions output aggregation — matching the cost objective, which already used it — so served == reported == costed energy. Adds an f annualisation- divergence diagnostic + warning; missing years_represented is treated as unit weights (no noisy warning). Doc descriptions authored via the v57 migration. Byte-identical when weights are uniform/absent.

SpineDB results output

A new spinedb write-method dumps the processed result tables into a Spine database using the FlexTool results schema, alongside the CSV / parquet / Excel outputs.

• write_spinedb writer with round-trip tests + the checked-in results schema JSON; wired into the dispatcher and CLIs (--write-methods spinedb, or the output-spinedb setting in the output template). One Spine alternative per run; time series as nested Map values on the model entity classes.
• Desktop GUI: a "Comparison SpineDB" File-outputs row drives the new method. Documented in docs/results.md.

Investment duals

• Synthesize a complete signed effective investment dual (positive = more investment would lower total cost); extract the min-side duals; retire the redundant standalone dual_invest_*_d_e outputs. The sign convention is stated on the plot value-axis label.
• Fixes: 1/scale_the_objective on the v_invest reduced-cost writer; period- cap constraints aligned to coef = unitsize / RHS-in-MW; a guaranteed period axis for the sole-binding maxInvest_total; cover cross-solve invest candidates in the densify universe.

Model & constraint changes

• Densify availability in the profile/online and node-state RHS so default-1.0 units (wind / solar / coal, profile storage nodes) aren't inner-join-dropped to zero generation.
• Wire capacity_min_coeff into the minFlow_minload floor and apply capacity_max_coeff to direct arcs (both were previously ignored on those paths).
• Reverse flow for lossless 2-way connections (method_2way_1var_off) via a shared non-negative v_flow_back aux unifying DC and non-DC arcs (verified against a hand-calc) — a lossless 2-way connection could not previously carry reverse flow.
• Rename the four flow-bound constraints to per-edge names (maxToSink* → maxFlow*, minToSink_minload → minFlow_minload); LP-internal identifiers only, no output/golden/migration impact.
• INFLOW-1: derive the inflow signed-split on synthetic / rolling sub-solves so the non_sync and capacity_margin demand budget survives (was silently dropped, pinning VRE to zero).

Environment robustness — solver-stack auto-heal

Compiled-extension native crashes (a bare exit code — Windows 0xC0000005, POSIX SIGILL/SIGABRT/SIGSEGV — that no try/except can catch) are now diagnosed and, where fixable, auto-healed.

• flextool/env_check.py probes the whole solver stack in one child process — polars (a real SIMD op) → highspy (Highs) → polar_high (Problem) — flushing a marker after each so the surviving markers pinpoint the dead extension.
• Per-component remediation: polars → polars-lts-cpu; highspy 1.14.0 → highspy==1.13.1 (HiGHS#2964 import-crashes 1.14.0 on Server 2019 / Win10 1809-era machines); a clear "rebuild your environment" message when the fault is the Python environment itself (Anaconda-mixed venv, network drive), which no pip install can fix.
• Wired into self-update (auto-fix), GUI startup (one-click fix, fingerprinted OK-cache), and the scenario crash path. faulthandler enabled in cmd_run_flextool. Neither remediation is made the default; FLEXTOOL_NO_ENV_CHECK opts out. 31 tests in tests/test_env_check.py.

Dispatch, comparison & import

• Folder name is the scenario identity at the load boundary, so two output folders holding the same model scenario (different settings) stay distinct in legends/titles/filenames instead of collapsing under the shared in-data tag (guarded no-op — byte-identical — on the DB/CLI path).
• Slice dispatch / comparison by the in-data scenario tag, not the output-folder name; allow a comparison view via an override map_dimensions_for_plots.
• Old-FlexTool (FlexTool-2) import now opens the execution window and runs the migration as its own visible job; the importer is pinned to the frozen v56 schema and always migrates.

Desktop GUI

• Theme-aware run-log colouring (command echo, timer/memory rows, solve-start, HiGHS dump block, warnings/errors blended from the log widget's live colours); clearer solve-phase layout with an Autoscale by polar-high: checkpoint so the "Solver" row reports pure HiGHS time.
• Memory guard master switch (session-only) + per-job "Force start" override to admit a scenario past the memory limit; a jobs-list right-click menu (Force start / Pause / Kill / Remove).
• Track retired input sources and fix source-number drift.

Packaging — Python 3.13 + dependency floor

• Officially support and CI-test Python 3.13 (classifier + a 3.13 leg on Linux / Windows / macOS); the full suite passes on CPython 3.13 (polars 1.41, pandas 3.0). requires-python stays >=3.9.
• Pin polars>=1.40: 1.40 introduced pl.struct(...).is_sorted() (used by polar_high's _verify_dense_sorted); older polars raised on it and silently bypassed autoscale Layers 2/3.

Plot & output fixes

• Keep value labels on sub-pixel bars in comparison charts; avoid a stack + grouped-bar conflict; use the settings-group name as the default disk-plan title; apply category/entity colours regardless of legend layout.
• perf(outputs): densify entity columns in one reindex instead of a per-column loop.

Internal cleanup

• Remove the experimental --fast-single-solve engine path and a temporary diagnostic dump block.
• Stop emitting unconsumed scratch/middle-product CSVs and delete the dead derive_* emit wrappers across the per-roll emitters; fuse the inflow-peak stages into one closed-form graph; remove a dead duplicate _inflow_scaling.py and an orphaned golden; reconcile the emit manifest.
• Drop the dead group_label_width from the bar-chart layout (the expand-axis group was already folded into the bar tick label).
• The test suite now runs in parallel by default (pytest-xdist, -n 8; a [dev] dependency, wired into CI). Worker processes keep the per-test global-state resets isolated; -n 0 disables it for interactive debugging.

Release 4.0.0b4 (2.6.2026) — vectorized per-roll preprocessing; glpsol scaffolding removed

Per-roll preprocessing — vectorized

Every heavy per-roll preprocessing parameter/set emitter now derives its output frame with vectorized polars — a dense (entity × dt) or (entity × period) grid, a left-join for each parameter source, a priority coalesce across the cascade branches, and group-by-sum folds for stochastic / parent-period terms — once per roll over that roll's own window, replacing the per-cell Python loops. There is no cache and no full-domain frame. On the RETO-Africa DES scenario (9 rolling solves) the run is ~25% faster end-to-end, with every output byte-identical to the previous release and the global peak working set unchanged (~18 GB). No model or input changes.

• New shared module engine_polars/_vectorize.py: the entity×dt / entity×period grid builders (carrying row-order keys), the dict→lookup lift, the stochastic + parent-period fold, and repr-based value rendering.
• Converted families: pdtProcess, pdtNode, pdtProcess_source / pdtProcess_sink, pdtCommodity, pdtGroup / pdGroup / pdtProfile, the varCost pair + pssdt_varCost filters, pdtNodeInflow, and the inflow- and lp-scaling pipelines.
• Each heavy derive is gated by a per-family parity test against a reference implementation: strict byte-equality for the lookup/coalesce families, rtol≈1e-12 tolerance for the genuinely-summing families (inflow / lp scaling, multi-term folds).

Solver / packaging

• Removed the leftover scaffolding for the already-retired glpsol binary: the build-glpsol.yml workflow, the bin/glpsol CI permission step, and the GLPK clauses in LICENSE.txt. HiGHS (via highspy) is the only solver. The DB migration that rewrites a legacy solver=glpsol to highs is unchanged.

Docs

• docs/dev/engine_polars.md documents the vectorized per-roll emit; CONTRIBUTING.md and CLAUDE.md record the parity-test obligation and the byte-parity invariants.

Release 4.0.0b3 (1.6.2026) — solver backend: bounded-memory autoscale + block-COO; plot & GUI fixes

Solver backend — bounded-memory autoscale + block-COO

Requires polar-high >= 2.4.0.

FlexTool's automatic scaling (Layers 1-3) and LP coefficient build no longer materialise wide coefficient products to read magnitude statistics. polar-high's new block-COO evaluation slices the pre-sorted (period, timestep) dense axis as contiguous numpy blocks, and the autoscale range / bucket readouts walk the constraint spine in bounded row-batches. On the large RETO-Africa DES scenario (9 rolling solves): autoscale peak working set dropped ~46 -> ~23 GB, ~15% faster, with every solve objective byte-identical to the previous release. No model or input changes.

• engine_polars/autoscale/_layer2.py buckets coefficient magnitudes through the bounded walk; the size-blind family-row skip is gone (no family's range is silently dropped from the scaling decision).
• The scaling decision is cached per rolling-group LP shape, skipping the per-roll range traversal on shape-identical interior rolls; between-solve transient peaks are flattened and prior solve-level state is released before the next build.
• Requires polar-high >= 2.4.0 (declare_dense_axes + the bounded walk); the interim capability guards have been dropped.

Plot output

• Grouped-bar plots: deterministic y-axis label, aligned grouped-bar rows, value axis sized to label width, compact engineering value labels, and thicker bars when value labels are enabled.

Desktop GUI

• Update dialog: an OK button to save settings without updating; smarter handling of no-tracking branches and no-op updates.
• Execution log no longer unresponsive / unselectable (refreshes coalesced).
• Status icons, spinners and checkboxes use Windows-renderable glyphs.

Fixes

• Migration: the timeblockSet -> timeset timeline collapse is now deterministic.
• Input: p_section wired on the synthetic-solve early return.
• Input: an explicit null parameter value in a higher-priority alternative — the way a scenario clears a value set in the base alternative (e.g. nulling a node's constraint_invested_capacity_coeff Map) — now resolves to "unset" instead of crashing the solve with a misleading "Unrecognised Map index column(s) []" error.
• The test_a_lot scenario golden was regenerated — a benign alternate-optimum dispatch reshuffle (objective byte-identical), not a model change.

Release 4.0.0b2 (29.5.2026) — desktop GUI: in-app updates, safer migration, macOS fix

This beta focuses on the standalone Tkinter desktop application.

In-app "Update FlexTool"

• New Update FlexTool… button in the top-right of the main window (next to UI settings…). It detects how FlexTool is installed and runs the right upgrade automatically — git pull + editable reinstall for a git checkout, or pip install --upgrade for a PyPI install — streaming output to the Execution window and ending with a restart-required prompt.
• An Install Spine Toolbox checkbox (opt-in; large dependency, required for the Spine DB Editor; pre-ticked when Toolbox is already present).
• A lightweight startup update check highlights the button (blue) when a newer version exists. Disable it with the Check for updates on startup toggle in the dialog or the FLEXTOOL_NO_UPDATE_CHECK environment variable.

Database migration

• GUI database migration now streams step-by-step progress to the Execution window; the modal dialog is a clear "interface locked until done" gate.
• Migration is wrapped in a backup/restore: a database is always left either fully upgraded or exactly as it was found (a failed or cancelled migration is rolled back), with failures pointing to the issue tracker.

Launchers and diagnostics

• A bare flextool command launches the desktop GUI; flextool/flextool-gui are windowed launchers (no stray console on Windows), backed by a rotating flextool_gui.log so diagnostics survive when there is no console.
• When the Spine DB Editor is missing or fails to start, the GUI explains why (and offers Update FlexTool / the install command) instead of failing silently; the editor's traceback is captured to the Execution window.
• Execution-window log: keyboard selection (Shift+arrows), a Copy log text button, and a right-click Copy / Select-all menu.

Fixes

• GUI status icons, spinners and checkboxes now use glyphs that render in the default Windows Tk fonts (Geometric Shapes / check marks); the previous emoji/Math-symbol glyphs (hourglasses, ballot boxes, ⌀, ⊘) showed as missing-glyph boxes on Windows. (This is a font-coverage issue, distinct from the cp1252 console-printing fix — GUI glyphs are rendered by Tk, not printed.)
• Fixed a macOS crash (RuntimeError: main thread is not in main loop) during database migration: worker threads no longer call tkinter (including after) directly — all worker→main GUI updates are marshalled onto the main thread.

Release 4.0.0b1 (29.5.2026) — first beta: schema v56 finalised, input-shape correctness, autoscale hardening

The 4.0 line moves from alpha to beta. The Spine data schema is finalised at version 56, the polars engine's input derivation gained an authoritative parameter-shape resolver, and the auto-scaler's registry contract is now enforced in CI. Existing input databases migrate forward automatically (flextool-migrate-database / GUI auto-migration).

Schema finalised at v56 (parameter & method renames; solver-config consolidation)

• Coefficient parameters renamed for consistency: flow_coefficient → conversion_flow_coeff, max_capacity_coefficient → capacity_max_coeff, min_capacity_coefficient → capacity_min_coeff, constraint_*_coefficient → constraint_*_coeff.
• Method value-lists cleaned up: co2_methods.no_method → none; an explicit none off-member added to conversion_methods, profile_methods and ramp_methods; is_* / has_* flags retyped to the yes_no list; the redundant storage_nested_fix_method.no member dropped.
• is_enabled re-added on constraint and the reserve__upDown__{unit,connection}__node classes (read-time gate that drops disabled entities).
• Per-solve solver knobs consolidated: the individual solver_options, highs_method, highs_parallel, highs_presolve, solver_threads, solver_log_level, solver_time_limit, solver_io_api and use_row_scaling parameters are gone — solver options now live in the solver_arguments map, and the rest are CLI flags (--scaling, --solver-log-level, --solver-time-limit, --solver-io-api, --highs-threads). db_migration.py carries every rename/fold forward.

Input-shape correctness — authoritative parameter-shape resolver

• New engine_polars/_param_shapes.py resolver detects each DB-authored parameter's shape from its nesting depth + per-level index_name (validated against a per-parameter allow-list), with value-domain probing to disambiguate spinedb_api's silent-default "x" Map index.
• Efficiency / min_load / efficiency_at_min_load and other_operational_cost derivation now route through this resolver (p_slope, p_section, p_pssdt_varCost). This fixes a class of cross-join blow-ups on long timelines — a time-series parameter carrying the "x" index was mis-read as a scalar and broadcast against the full (period, time) grid (overflowing on multi-week/multi-period runs) — and the matching silent flattening of time variation on short timelines.

Numerical scaling / auto-scaler hardening

• The Layer-2 registries (_layer2_types.py / _quantity_types.py) are verified complete against the schema and every emitted constraint / variable; lookup_cstr now implements the documented prefix dispatch so dynamic constraint names (e.g. the ramp family) resolve.
• An unregistered family no longer silently degrades a solve to an un-scaled LP: FLEXTOOL_AUTOSCALE_STRICT=1 turns it into a hard error, and CI now runs the suite with strict mode on.
• Layer-2 scaling preserves the solver objective across the setSolution push-back; the fast single-solve path unpacks the Layer-2 result correctly.

Solver & I/O

• Requires polar-high >= 2.3.0 (Where-pushdown release).
• Self-describing Excel v2 round-trips cleanly; commodity price-ladder parameters use the facet-leaf (price/quantity) layout; the Spine reader no longer invents defaults (strict bool / header / dtype / alt handling).

Testing & developer docs

• Tests build their input DB from JSON/schema under tmp_path rather than depending on a checked-in SQLite (which lagged the schema); a session-scoped schema_db_url fixture and schema-driven registry coverage catch drift at the source.
• Per-scenario timing-budget floor raised to 10 s (machine/drive dependent; still catches gross regressions).
• Developer docs document the autoscale-registry, parameter-shape and build-from-schema invariants (docs/dev/architecture.md); CLAUDE.md is now tracked; ruff check . is required and clean.

Release 4.0.0a5 (28.5.2026) — GUI migration UX + side-menu polish

GUI-focused follow-on to 4.0.0a4. Two visible bugs and a sweep of side-menu housekeeping.

Automatic DB migration no longer freezes the GUI

Opening a project with outdated sqlite input sources used to lock the Tk main thread for the duration of the migration with no feedback at all. The migration now runs in a worker thread behind a modal progress dialog with a Cancel button.

• flextool/update_flextool/db_migration.py — migrate_database() gains keyword-only progress_callback and cancel_check hooks plus a MigrationCancelled exception carrying the last successfully completed version. Cancel is checked at the top of each step only — an in-flight commit always finishes before the exception is raised, so cancel does not corrupt the DB.
• flextool/gui/db_version_check.py — forwards the new hooks through check_and_upgrade_database(); adds public helpers needs_flextool_migration() and get_target_flextool_version() so the GUI can pre-check which files actually need migration without performing one. A dedicated except MigrationCancelled branch reports partial progress and documents that re-run is safe (steps idempotent).
• Externally-referenced sqlite files (registered via external_refs) now trigger a 3-button consent dialog before migration: Migrate in place, Copy all to project and migrate (copies into input_sources/ and drops the external_refs entry), or Cancel (skips externals; internal files still migrate).
• Two new dialogs under flextool/gui/dialogs/: migration_consent_dialog.py and migration_progress_dialog.py (modal with spinner, current-file label, thread-safe update_status / mark_finished, Cancel button).
• MainWindow._run_db_migrations_with_ui() drives the new flow: plan → consent → optional copy (persisted once before the worker starts) → progress modal → worker thread → wait_window → summary messagebox.

Per-project side-menu settings actually persist now

_load_auto_gen_vars() restored 12 Tk vars in sequence. Each var.set() immediately fired the _on_auto_gen_toggled trace, which read all vars and wrote settings.yaml — capturing the still-stale not-yet-restored values from the previous project (or fresh defaults on first load). Every project load silently overwrote the file with the wrong values within milliseconds of opening. A guard flag now suppresses the trace during the restore.

Stacked-area y-axis no longer clips peaks

flextool/plot_outputs/plan.py _compute_time_plan used the per-series min/max for subplot_y_ranges, but stacked-area charts render the column-wise sum — peaks would flat-top at the visual ceiling (orange wind_plant over blue coal_plant in the dispatch plot). For chart_type == 'stack' the range now uses num.clip(lower=0).sum(axis=1).max() and num.clip(upper=0).sum(axis=1).min(). Non-stack line charts keep the original formula. The shared-axis manifest unions already-correct numbers, so cross-scenario sharing is preserved.

Side menu / UI settings popup

• Theme radios moved from the side menu into the UI settings popup as a "Theme" cascade — they are a global setting and belong in the global menu.
• Button-label hygiene: Project → Projects…; Add → Add…; Png settings → Png settings…; Execution jobs → Execution jobs…; Results viewer → Results viewer…; UI settings → UI settings…
• Tooltips on every File outputs row (Scenario pngs / Excels / csvs / Comparison pngs / Excel) — short descriptions of what the action produces and where on disk.
• _styled_popup_menu() themes native tk.Menu popups to match the current sv_ttk theme (bg/fg/activebackground/selectcolor) and binds font="TkDefaultFont" so the menu font tracks the body font. Applied to the UI settings popup plus the three right-click context menus on the input-sources and scenarios trees — the radio bullet was invisible on dark themes and the menu font was smaller than the rest of the UI.
• _install_menu_hover_dismiss() auto-unposts the UI settings popup ~400 ms after the mouse leaves all menus; the delay lets the cursor briefly cross the parent border into a cascade submenu without dismissing.

Cleanup

• Removed plot_settings/{single_dataset,multiple_datasets}/ default_result_plots.json — superseded by flextool/schemas/default_plots.yaml, not referenced by any code path, not shipped via pyproject.toml package-data.
• default_plots.yaml — scenario_rule: l added to three sub-configs; one unit_time_plots variant flipped from h to l.

Release 4.0.0a4 (28.5.2026) — cross-solve memory hygiene + diagnostics

Follow-on to 4.0.0a3. The Stage A+B refactor exposed (but did not fix) a pre-existing cross-solve memory accumulation in the orchestration: each completed solve's OrchestrationStep held a full parent-side highspy.Highs instance plus per-variable polars frames plus the per-solve FlexData and FlexDataProvider, released only at the START of the iter-after-next (a one-iter lag) or — for the per-step provider/flex_data — only after the WHOLE cascade. On DES this meant the parent reached 47-48 GB by solve 8, and solve 9 froze the machine under swap pressure.

This release lands the targeted fixes plus the diagnostic infrastructure used to find them.

Cross-solve memory hygiene

• flextool/engine_polars/_subprocess_solve.py — the cold-path (--save-memory) read-back from the subprocess .sol file no longer re-loads the entire MPS into a parent-side highspy.Highs instance just to extract column names and dual arrays. A new _parse_highs_sol parser plus a duck-typed _SolHighsShim expose only what the downstream writers consume (getSolution, getLp().row_names_, passColName, getObjectiveValue). On DES this was the +33 GB sidecar per solve. As a side effect, a pre-existing latent bug is fixed: _v_obj__{solve}.parquet on cold runs used to write 0.0 for the objective (the prior getObjectiveValue() call had no value to read); the shim returns the value parsed from the .sol's Objective: line. If you have analysis built on v_obj parquets from any prior cold (--save-memory) run, those parquets are wrong. Warm-path runs are unaffected.

• flextool/engine_polars/_orchestration.py — the slim of the prior step's solution state moved from "start of next iter's handoff" to "end of current iter, immediately after Outputs written" (kills the one-iter retention lag). Extended to also null flex_data, flex_data_provider, and solution._vars on the cold path. The just-parked step survives so the LAST iter's outputs still write correctly.

• flextool/engine_polars/_native_run_model.py + _orchestration.py — on the warm path the slim is per-level: the most-recent Solution.highs and flex_data_provider of each level (solve_hydro, solve_dispatch, ...) survive until either (a) a fresher same-level step supersedes them, or (b) the pipeline has no more upcoming iterations of that level. Earlier same-level Highs instances and unrelated-level state are dropped. solution._vars and flex_data are dropped unconditionally on every prior step (warm-restart uses HiGHS' basis, not these frames).

• FLEXTOOL_COLD_KEEP_PROVIDER=1 — opt-in env knob. Cold path drops flex_data_provider by default (each cold solve re-reads the Spine DB). Set this to retain the provider across cold iterations — trades higher parent RSS for skipping the per-iter DB re-read. Useful for workloads where Spine DB re-read dominates wall time on large input DBs.

Diagnostics

• flextool/_mem_sampler.py (new module) — opt-in daemon-thread RSS / available / swap sampler. Activated by FLEXTOOL_MEM_SAMPLER=1. Writes timestamped TSV-ish lines to /tmp/flextool_mem_sampler_<pid>.log (override with FLEXTOOL_MEM_SAMPLER_LOG=path) at 100 ms cadence (override with FLEXTOOL_MEM_SAMPLER_INTERVAL_MS=N, clamped 20-10000). Per- sample explicit flush so the last samples before SIGKILL reach disk. Zero-overhead when off. Falls back to /proc parsing when psutil is unavailable on Linux.

• Memory/timer table redesigned in _MemoryRecorder: 4-column layout (time | RSS memory | system memory | system swap) where each cell renders <absolute> (<±delta>) in a unit consistent with the absolute value. Swap now reports a delta (previously swap_prev_mb was not tracked).

• "Solve cleanup" checkpoint row emitted at level-group boundaries (between consecutive solves in the cascade). Δrss vs. Outputs written makes any retained-across-the-boundary memory obvious in the standard log table — no separate diagnostic tool required.

• Textual log messages (autoscale console summary, save-memory / warm-disabled / soft-promote warnings) now wrap at 100 chars via a textwrap.fill helper so multi-sentence prose stays readable in narrow terminals.

Documentation

• docs/dev/env_vars.md (new) — comprehensive reference for every env var the flextool code actually reads. Grouped by Functional toggles / HiGHS tuning / Memory diagnostics / Memory tuning / Precision cleanup / Niche-test hooks. Cross-linked with polar-high's docs/guide/env-vars.md for the POLAR_HIGH_* side.

• docs/dev/inject_between_solves.md (new) — developer guide for the run_chain_from_db(..., override_provider=callable) parameter-injection mechanism. Covers the 11 whitelisted handoff carrier keys, lifecycle (per-iter, after handoff translation, before preprocessing), warm-path interaction (warm reuse happens BEFORE the override fires), audit hook (FLEXTOOL_AUDIT_SOURCES=1), security note, and a working Python wrapper example.

• docs/dev/scaling.md, engine_polars.md, architecture.md — fixed stale FLEXTOOL_AUTO_SCALE=0 references; the real variable is FLEXTOOL_SCALING=off.

• Two stale _orchestration.py comments referencing the Stage A6-deleted _layer2._rewrite_term_lazy helper updated to describe the actual side-vector mechanism that replaced it.

Maintenance

• polar-high>=2.0.2 → polar-high>=2.2.0 in pyproject.toml. The cross-solve hygiene code (and most of 4.0.0a3 itself) reads polar-high's canonical-matrix surface (Problem.canonicalise, Problem._matrix, _layer2_*_factor, _canonical_dirty), all introduced in polar-high 2.2.0.

• flextool/cli/cmd_run_flextool.py and flextool/engine_polars/_orchestration.py — ruff clean. Earlier the sampler import sat between the MALLOC_ARENA_MAX setdefault and the workload imports, triggering 15 E402 errors; moved the start_mem_sampler() call to the first statement after imports. Pre-existing F401 / F841 / F821 warnings in _orchestration.py also cleaned (unused imports/locals removed, polars as pl and Callable added to the TYPE_CHECKING block).

• _audit_reports/FLEXTOOL_PYRAMID_HANDOFF.md and _audit_reports/storage_binding_method_callsites.md removed from the git index (the _audit_reports/ and specs/ folders are gitignored; these two were the only tracked exceptions). Files preserved on disk.

Real-workload validation

The cross-solve memory cliff that froze the user's machine at DES solve 3 under --save-memory (parent at ~40 GB + subprocess trying to start + system swap-thrashing → kernel page-fault storm) is structurally addressed. Synthetic 4-solve fixture validates the per-solve RSS-at-Outputs written baseline shifts from a +9 MB-per-solve climb to a flat curve (modulo the intentional captured_vars retention for cross-solve writers, which is independent of the cliff). Full validation on a real DES run is recommended before relying on the workaround being durable.

tests/engine_polars/autoscale/test_layer2_roundtrip.py and tests/engine_polars/autoscale/test_h2_trade_e2e.py (the bit-for-bit safety nets) pass; full flextool autoscale suite 51 passed; polar-high 126 passed / 8 skipped.

Requires polar-high >= 2.2.0 (unchanged from 4.0.0a3).

Release 4.0.0a3 (28.5.2026) — GLPK-style autoscale refactor

Stage A of a two-stage refactor of the polar-high <-> flextool autoscale interface, paired with polar-high 2.2.0. Together they replace the pre-existing "rewrite every constraint's lazy plan to embed the scaling factors, then re-evaluate from scratch in every consumer" model with the GLPK textbook approach: scaling lives in two numpy side vectors on Problem, and a single canonical CSC matrix is built once and shared across consumers.

What changes in flextool

• flextool/engine_polars/autoscale/_layer2.py — apply_layer2 no longer mutates Problem._cstrs or Problem._obj_terms. It writes the per-column factor into Problem._layer2_col_factor (as 1 / cf_math, the inverse convention required by consumers that multiply rather than divide) and the per-row factor into Problem._layer2_row_factor (forward rf_math). Sets Problem._layer2_locked = True and Problem._canonical_dirty = True. Var.lower / Var.upper continue to be mutated in place — they are scalar per family, the cost is O(n_var_families), and they must be visible to any caller of Var (not just to the matrix consumers). Layer2Plan's public shape is unchanged; unscale_solution is byte-for-byte unchanged.

• The four now-dead helper functions are deleted: _rewrite_term_lazy, _rewrite_obj_term_lazy, _scale_rhs, _rhs_has_vars. ~250 LoC net reduction in _layer2.py.

• The POLAR_HIGH_RANGES_MAX_FAMILY_ROWS family-size skip added in 4.0.0a2 to apply_layer2's row-factor loop is no longer needed and was removed. (The equivalent skip still lives in bucket_coefficients — that one is for the in-bucket-walk pattern in Layer 1 detection, unrelated.)

• flextool/engine_polars/_orchestration.py — two warm-path comments updated to describe the side-vector design instead of the deleted _layer2._rewrite_term_lazy helper.

Real-workload validation

The original OOM trigger on DES (profile_flow_upper_limit's 1.5M-row multi-Param chain in apply_layer2) is gone. A full DES smoke run under the new architecture completed 8 of 9 dispatch solves with Model status: Optimal before OOM-killing at solve 9 — and that 9th-solve OOM is not a refactor regression but a pre-existing cross-solve memory build-up that the refactor exposed by getting further into the workflow than any previous run. At solve 9's start the process is at 47.6 GB RSS purely from accumulated state across solves 1-8; investigation deferred.

Bit-for-bit safety nets (test_layer2_roundtrip.py + test_h2_trade_e2e.py::test_h2_trade_autoscale_full_matches_solver_only_bit_for_bit) pass under the new code path. Full flextool autoscale suite: 51 passed. Full polar-high suite: 126 passed / 8 skipped (the one pre-existing-hung warm-rolling-speedup test deselected).

Requires polar-high >= 2.2.0 — the side-vector storage, the canonical matrix, and the _canonical_dirty flag all live on the polar-high Problem class.

Release 4.0.0a1 (26.5.2026) — first v4 alpha

First public alpha of the v4 line. The 3.x series ended at 3.47.0 on 24.5.2026; the v4 clean-slate orphan-root commit (5b8225a3, 25.5.2026) was the version-identity break. This alpha gathers the v4 engine swap, the output overhaul, and the recent storage_binding_method restructure into a single PEP-440-tagged pre-release so downstream users can pull 4.0.0a1 from pip / TestPyPI for trial integration.

Parity with the 3.x line is expected (modulo bugs surfacing during alpha testing). 4.0.0 final follows after the remaining minor-functionality additions, cleanup, and broader testing land. Subsequent alphas / betas / release candidates follow PEP 440 (4.0.0a2, 4.0.0b1, 4.0.0rc1, 4.0.0).

What v4 brings (recap from the orphan-root commit 5b8225a3)

• Engine: polar_high + highspy in-process; legacy GMPL pipeline, glpsol bundling, flextoolrunner package retired entirely.
• Input API: SpineDBBackend as the single Spine DB → polars layer; input_derivation/ as the per-solve preprocessing successor to flextoolrunner/preprocessing.
• Cascade transport: FlexDataProvider (in-memory) is the contract; CSV mirrors only emit under --csv-dump (debug flag).
• Outputs: canonical parquet under output_parquet//; intermediate output_raw/ deleted automatically on successful run unless --csv-dump preserves it.
• CLI: --engine flag retired (polar_high is the only engine); --highs-threads wired through to HiGHS; bin/ relocated to solver_config/.
• Tests: 521 MB of tracked fixture data reduced to ~5 KB; the scenario_workdir fixture builds workdirs on demand via the live cascade.

What lands on top in this alpha

• storage_binding_method restructure (six-phase migration, May 2026) — the parameter was previously list-valued with additive semantics in nodeBalance_eq, which silently double-counted state-change residuals for any node carrying multiple methods. Reverted to single-valued; the value_list is now an eight-member enum (seven cycle-scope methods plus bind_intraperiod_blocks as an orthogonal aggregation method); the three legacy names (bind_within_timeset, bind_using_blended_weights, bind_within_model) are migrated by DB schema v55 to their new names; the two new RP variants (bind_within_period_blended_weights, bind_forward_only_blended_weights) are fully implemented. Same storage entity can now drive an RP investment solve and a chronological dispatch solve back-to-back without parameter changes (per-solve silent degrade replaces the earlier strict check). See the storage_binding_method Phase A-F commits.
• DB schema v55 — migration auto-runs on DB load via migrate_database(path); bumps FLEXTOOL_DB_VERSION 54 → 55. Refreshes both the value_list members and the parameter description on existing DBs to match the new schema template.
• autoscale package (commits 92b98efd through d4a4d9df) — semantic per-type scaling layer between the cascade and HiGHS; the legacy scaling.py is fully retired. --auto-scale / --scaling CLI flag is on by default; HiGHS-native bound + objective scaling driven from polar-high.

Known caveats for alpha users

• This is an alpha — bugs are expected. Reports against any scenario that worked under 3.47.0 but misbehaves under 4.0.0a1 are especially welcome and feed directly into the cleanup before the 4.0.0 final.
• One known parity-tolerance gap remains (tests/engine_polars/test_phase_e_g_multi_roll.py::test_fullYear_roll_matches_v3320_golden, rel 1e-6); pre-existing, orthogonal to the alpha-cycle changes.
• Pre-v55 DBs that carried array-valued storage_binding_method parameter values must be migrated before use; see the v52→v55 migration chain in flextool/update_flextool/db_migration.py.

Release 3.47.0 (24.5.2026) — JSON-fixture architecture + Stage 4 test migration

Minor release on top of 3.46.0. Closes nine latent test failures surfaced during the Tier 4 sweep, lands the architectural overhaul of the JSON-fixture infrastructure that supports test data, and migrates 140 engine_polars test files off the gitignored disk fixtures that were generated externally by the legacy GMPL engine. Twenty-six commits across four themes.

Latent failures (Groups B / C / A)

The Tier 4 sweep surfaced nine failures that pre-existed the retirement but were masked by narrower test gates. All addressed:

• Group B (commit 7b119abd) — three tests in test_handoff_cumulative_carriers.py referenced pre-Phase-4.1a column names (value, mwh) that no longer exist; producers emit the canonical p_ladder_cum_sim_hours / p_ladder_cum_realized_mwh. Test-only fix.
• Group C-reader (commit 79dc699b) — three csv.reader sites in flextool/engine_polars/_blocks.py (_read_input_rows plus two header probes in emit_block_data_for_solve) read from disk directly, violating the Provider-only cascade contract. Routed through input._provider_open so the Provider serves the bytes in cascade mode and disk is consulted only as the non-cascade-caller fallback.
• Group C-writer (commit 460539b1) — gated the p_entity_period_existing_capacity.csv disk write on a new csv_dump flag threaded from state.csv_dump through write_outputs_for_solve → write_all_handoffs → write_p_entity_period_existing_capacity. The in-memory SolveHandoff.realized_existing already carries the data, and _load_prior_existing already prefers it over disk; the unconditional CSV write was a redundant dump.
• Group A (output path) (commit 6a05f5a4) — _entity_all_capacity in read_parameters.py needs each sub- solve's v_invest_p/n + v_divest_p/n long-form frames to compute the total column in unit_capacity__d.csv (and the sister CSVs). Polar-high's recent memory optimisation releases Solution._vars internally between sub-solves, so by the time the end-of-cascade writer runs only the last sub-solve's variables survive. Captured the four frames into OrchestrationStep.captured_vars immediately before the step is deposited and exposed them through a SnapshotSolution wrapper via a new effective_solution property. Fixes multi_year and wind_battery_invest_lifetime_renew_4solve.
• Group A (parity tests) (commit 905781e7, supersedes 993a0cde) — multi_fullYear_battery_nested_* parity failures were the same upstream cause (LP missed roll-state handoff + storage-pin constraints because the production loader reads handoff carriers exclusively from in-memory Provider keys and the isolated parity tests never run the orchestrator that populates them). The four replay-style tests have been deleted as redundant with test_scenarios.py (which runs the full cascade for the same scenarios and passes against tracked goldens).

Worktree merge — fix_price / fix_usage / upward storage handoff

Four commits cherry-picked from a parallel worktree session (f206272c, 1ab347e3, f420ce83, bb35cd54):

• fix_price — model.py:3427 used the wrong p_unitsize (process-indexed) instead of p_state_unitsize (node-indexed) on the use_reference_price objective term. Surfaced by a new γ-style child-only wiring test under tests/engine_polars/constraints/. The stale xfail at test_cost_aggregation_semantics.py:943 retired.
• fix_usage producer (input.py:5706-5837) promoted to the full formula (efficiency slope + min_load section + noEff branch), mirroring the LP constraint LHS at model.py:1500-1537. Closes the legacy producer/constraint asymmetry documented in specs/feature_fixes.md §3.
• Upward dispatch→storage state carrier (SolveHandoff.upward_roll_end_state + HANDOFF_UPWARD_ROLL_END_STATE provider key) made explicit. Sequential-prior orchestration produces identical numerical behaviour today; the carrier exists to make the upward path separable for future routing changes (specs/feature_fixes.md §1).

JSON-fixture architecture overhaul — single source of truth

Five commits land the architectural cleanup the user steered toward during the session. Today the test-fixture and canonical-database JSONs are kept consistent by tooling rather than convention:

• Stage 1 (commit 4e857535) — deleted three silently-divergent duplicates of tests/fixtures/{tests,stochastics,lh2_three_region}.json under tests/engine_polars/fixtures/. Same line counts, but drifted parameter defaults (e.g. null vs 1.0). The tests/engine_polars/fixtures/ directory's remaining role is exposing flex_toy_*.py modules on sys.path.
• Stage 2a (commit eb352d3c) — extended schema migration coverage to tests/fixtures/*.json via a new flextool.update_flextool.test_fixtures module mirroring the existing canonical_databases.migrate_all pattern. Brought the seven test fixtures up to FLEXTOOL_DB_VERSION=52 (tests.json went 213→234 parameter_definitions, stochastics 207→235, etc., counts preserved on entities / alternatives / scenarios / parameter_values / scenario_alternatives).
• Stage 2b (commits 3494d7d6 + c763e637 + 0ff95715) — built flextool/update_flextool/generate_canonical.py + flextool/schemas/canonical_databases/_recipes.yaml and wired templates_examples.json + howto_stochastics.json as generated projections of tests.json / stochastics.json. The remaining seven howto_*.json + templates_time_settings_only.json stay authoritative for themselves — purpose-specific minimal teaching examples whose curation goals differ from test-coverage curation. Generator surfaced one drift (parameter_group_name='reserve' lost on two reserve.is_active parameters) which was fixed at the source.
• Stage 2c (commit 79a02552) — flextool/update_flextool/ extend_tests_fixture.py adds an agent-friendly YAML-delta workflow for append-only additions to tests/fixtures/*.json (new entities + alternatives + scalar parameter values + scenarios). Validates against spinedb_schema.json with did-you-mean suggestions; rejects edits to existing entries; rejects structured (Map / time-series) values, pointing the caller at the SpineDB editor for those. 8 unit tests.
• Stage 5 (commit 4d73460b) — deleted four nested-battery replay-style parity tests + _handoff_seed.py. Same scenarios covered end-to-end via JSON-path in test_scenarios.py.

CI verify chain

The drift-detection pipeline is now CI-enforced (tests.yml::template-check, commit 6d5849ac):

• sync_master_json_template --verify — schema parity (pre-existing)
• test_fixtures verify — tests/fixtures/*.json at current schema
• generate_canonical --verify — generated canonical files in sync
• canonical_databases verify — authoritative canonical files at current schema (new; mirrors test_fixtures.verify_all)

Same commands run locally as pre-push sanity.

Stage 4 — engine_polars test migration off gitignored disk fixtures

Five phases (a8ce274f, 2e4f499d, 8f42313d, 840fc56c, 494046f9, 905781e7) migrate 140 test files off the tests/engine_polars/data/work_<scenario>/ gitignored snapshot pattern. Tests now build their work-folders on demand via a new session-scoped scenario_workdir fixture in tests/engine_polars/conftest.py:

• scenario_workdir(scenario_name, db_fixture="main") runs the full cascade with csv_dump=True + keep_solutions=True and snapshots the last step's flex_data_provider so the on-disk CSV layout matches what the legacy disk fixtures shipped.
• Session-cached per (scenario, db_fixture) — each scenario pays its ~2 s cascade cost once per test session, no matter how many tests use it.
• Seven db_fixture values map to per-fixture session DBs: main (tests.json), stochastic, lh2, case14, h2_trade_parity, multi_ts_branch1, stochastics_pbt_inflow, branch2_parent_period.
• tests/fixtures/_augment_phase3d.py adds five new scenarios to existing fixture JSONs via proper alternative overlays (coal_ladder_annual, coal_ladder_cumulative, wind_battery_invest_lifetime_renew_inflation_2pct, delay_source_coef, 2_day_stochastic_dispatch_no_storage). All idempotent.
• tests/fixtures/case14_dc_power_flow.json (8173 lines) is a committed export of the PGLib IEEE case14 MATPOWER fixture used by test_flex_dc_power_flow.py.
• Dynamic-discovery cluster tests (test_arithmetic_cluster, test_block_cluster, test_block_layout, test_npv_cluster, test_profile_cluster, the opt-in sweep in test_orchestration) replace discover_workdirs() glob with a curated PARITY_SWEEP_CASES list in a new tests/engine_polars/_parity_sweep.py.

The gitignored tests/engine_polars/data/work_*/ snapshot dirs remain on .gitignore for back-compat, but no migrated test depends on them.

One incidental production fix — e5b784ef

While exploring an unrelated issue an agent stumbled into a real LP bug: v_current was indexed on a larger union than current_idx in rolling / nested-multi-invest scenarios, causing v_current - v_forward to crash with "Array conditional must be same shape as self" inside the nodeState change emission. Reindexed v_current to current_idx like the other v_prev_* series.

Coverage

tests/engine_polars/ full sweep: 1824 passed, 538 skipped, 6 xfailed, 0 failed (~27 minutes). The four CI verify hooks report clean. No regressions in tests/test_scenarios.py (65 scenarios pass).

Release 3.46.0 (23.5.2026) — flextool/flextoolrunner/ retirement (Tier 4 sweep)

Minor release on top of 3.45.0. Completes the multi-tier sweep that started in 3.33.0 when the GMPL pipeline was retired: the flextool/flextoolrunner/ package itself is now gone. Every piece of the old solve-coordinator has been relocated into a home that reflects its current responsibility, and the top-level flextool.__init__ re-export of FlexToolRunner is dropped. Thirteen audit commits land the move end-to-end (8baab773..1776dd09).

Module moves

• precision.py → flextool/common_utils/precision.py (Tier 4 Commit 1, ec0dfacb).
• timing_recorder.py → flextool/cli/_timing.py (Tier 4 Commit 2, 0a620dbd).
• solver_runner.py → flextool/engine_polars/_solver_base.py; solve_handoff.py shim deleted (Tier 4 Commit 3, 4813b490).
• scaling.py / scaling_report.py / blocks.py / runner_state.py / solve_handoff.py consolidated into flextool/engine_polars/ — scaling.py + scaling_report.py merged with the engine-side copies, blocks.py relocated, RunnerState + PathConfig consolidated into engine_polars/_solve_state.py, runner-side scaling deleted as dead code (Tier 4 Commits 4-6, 53557311 / 37309a90 / 2f8ca342).
• region_decomposition.py + region_filter.py → flextool/decomposition/ (Tier 4 Commit 7, f862eeec).
• flextoolrunner.py → flextool/engine_polars/_db_loader.py (carries the FlexToolRunner class).
• db_reader.py → flextool/engine_polars/_db_reader.py.
• Dead modules deleted: solve_config.py, timeline_config.py, tests/test_commodity_ladder_smoke.py (smoke had no live coverage beyond what the integration suite already exercised).
• flextool/__init__.py no longer re-exports FlexToolRunner; callers that still need the class import it from flextool.engine_polars._db_loader directly. All in-tree call sites were updated; the only external surface affected is from flextool import FlexToolRunner, which is now from flextool.engine_polars._db_loader import FlexToolRunner.

Doc sweep

• Cleanup passes A-D refreshed flextool.flextoolrunner.* references across tests/, specs/, docs/, migration/, and the top of RELEASE.md. Current-tense references now point at the new homes; historical narrative ("ported from", "lived in flextoolrunner/ through 3.32") was preserved intentionally.
• docs/dev/architecture.md — the dedicated "Solve Coordinator" section, the top mermaid diagram, the repository layout block, and the public-APIs example all rewrite around the new module map.

Release 3.45.0 (22.5.2026) — RP-blended-weights state coupling + logging polish

Minor release on top of 3.44.0. Closes the long-standing bind_using_blended_weights gap in engine_polars — the representative-period storage binding declared by the schema and used by flextool.representative_periods.preprocess had no constraint emission, so any node with that binding silently lost state-continuity and the RP cost diverged from the full-model reference. Twelve commits land the full mechanism end-to-end, matching the deleted Pyomo flextool.mod reference (blob c04afa59, lines 1689, 1691, 2197-2200, 2965-2997). Three smaller commits polish run-log / memory-checkpoint output.

Representative-period state coupling — full implementation

• New LP variables (flextool/engine_polars/model.py):
• v_state_inter[n, b] — long-run state at each base-period boundary, indexed over nodeState_rp × rp_base_period_set.
• v_state_rp_start[n, d, t] — free starting state at each RP-block-first step, indexed over nodeState_rp × rp_block_first.
• Intra-period state change inside nodeBalance (mirrors .mod:2197- 2200): for n ∈ nodeState_rp, within-timeset lag for (d, t) ∉ rp_block_first, and (v_state_rp_start − v_state) · unitsize for the first step of each RP block.
• Inter-period constraints:
• rp_inter_period_balance (.mod:2965-2975) — couples v_state_inter along rp_base_chain via the weighted sum of intra-block state changes.
• rp_inter_period_cyclic (.mod:2978-2988) — closes the chain by equating the first-to-last v_state_inter delta to the same weighted sum keyed at b_first.
• rp_inter_period_max_state (.mod:2991-2997) — capacity bound on v_state_inter at every (n, b, d).
• maxState_rp_start — sibling of maxState that bounds v_state_rp_start (the .mod expressed this as a per-row Var.upper, which the polars engine doesn't yet support).
• New storage_bind_using_blended_weights projection helper + SIMPLE_PROJECTIONS mapping entry in _projection_params.py — mirrors storage_bind_within_timeset / storage_bind_within_solve.
• Eight new FlexData attributes (nodeState_rp, rp_base_period_set, rp_base_chain, rp_base_first, rp_base_last, rp_block_first, p_rp_last_step, rp_base__rep) wired through the loader, the warm-carry list, and the region filter. Load-time invariant fires a ValueError naming the missing field when nodeState_rp is non-empty but any of its tightly-coupled siblings is empty.

Supporting RP-set restoration

• Reverses the relevant parts of 752dff3f to bring back six derived RP sets/params (rp_base_chain, rp_base_first, rp_base_last, rp_block_first, rp_block_last, rp_block_start_last) inside _compute_rp_frames and moves the rp_base_period_set / rp_rep_period_set derivation into the same place so it can see rp_weights (the previous derivation lived in emit_per_solve_sets which runs BEFORE emit_rp_data in the cascade — always wrote empty frames).
• Adds nine K.SOLVE_DATA_RP_* / K.SOLVE_DATA_NODE_STATE_RP constants to _provider_keys.py (no .csv suffix, per the Phase 3b convention) and migrates the three existing RP emit sites to use them.

Decoder fix (_decode_rp_weights)

• The DB-side params_to_dict returns representative_period_weights as flat triples [base, rep, weight] when the 2-level nested Map is flattened via convert_map_to_table, not as [base, inner_map_or_list]. Both copies of _decode_rp_weights (engine_polars/_timeline.py and flextoolrunner/timeline_config.py) now detect and decode the flat-triple shape; the nested-list shape stays in place. Without this fix state.timeline.rp_weights was silently empty, the RP gate in _native_run_model fell through to emit_empty_rp_data, and bind_using_blended_weights silently degraded to no state-continuity.

Tests

• tests/test_representative_periods.py::TestRPAllRepresented::test_all_represented_matches_full — was strict-xfail at the original +5.84% cost gap; now passes within the 1% tolerance.
• New hand-calculable toy fixture tests/engine_polars/test_rp_blended_weights_minimal.py: 2 base periods (b1 → b2 chain), 1 rep block, 1 storage node, inflow configured so the cyclic constraint forces exactly 2 units of unavoidable slack. Cost golden 2.0 derived by hand and matched by HiGHS to 1e-6. Capacity-binding probe confirmed: dropping the storage cap from 100 → 0.5 raises cost 2.0 → 7.0, proving maxState_rp_start binds.

Logging / memory-checkpoint polish

• Solve start: lines now include a sub-solve counter [name, N/total] so cascade progress is visible at a glance.
• [mem] checkpoint table is narrower and uses a two-column process/system layout with a descriptive header. Adds system-swap column; drops the redundant HiGHS-internal column that was always zero outside the solve.
• Run-log entry gets "Run start" / "Available solvers" banner, blank line between solves, and several routine writer chatter lines moved to debug level.

UnboundLocalError fix in _emit_per_solve

• Drops a dead function-local from flextool.engine_polars import _provider_keys as K inside emit_per_solve_sets. Phase 1 of this release added a module-level K import whose use at line ~169 tripped UnboundLocalError because the dead function-local re-binding made K function-local across the whole body.

Release 3.44.0 (22.5.2026) — auto user_bound_scale + scaling cleanup

Minor release on top of 3.43.2. Bumps the polar-high dependency to v1.4.0 (breaking removal of Problem.peek_lp_ranges) and rewires flextool's user_bound_scale handling around the new stream-time scaling path. Five flextool commits + two polar-high commits land together.

polar-high opt-in auto-scaling (new default for flextool)

• polar-high v1.4.0 adds Problem(auto_user_bound_scale=True): during the streaming feed to HiGHS, accumulate LP coefficient ranges over the per-family arrays we already build (matrix, cost, col-bound, row-bound) and apply a user_bound_scale recommendation via setOptionValue before Highs.run(). The embedded heuristic is col-bound-only with a 6-decade gate and [-10, 0] clamp — identical policy to flextool's Rivendell-bug-1+5+6 fix, deliberately ignoring row-bound spread to avoid crushing tight column bounds on energy-system LPs with wide cumulative-resource RHS magnitudes. Ranges are exposed on Solution.streamed_lp_ranges for callers that want to inspect.
• All three real flextool Problem() construction sites opt in to auto_user_bound_scale=True: the cascade cold-build branch in _orchestration._drive_cascade, the standalone single-solve in _orchestration.run_single_solve_from_db, and the warm-LP first-build in _warm._build_warm_problem.

Input-data heuristic disabled by default

• recommended_highs_options(apply_user_bound_scale=...) defaults to False. The legacy input-data heuristic was firing on every solve and clamping to N=-10 on DES-class scenarios because RHS family ranges routinely reach 1e+6 — producing the "User-scaled problem has some excessively small bounds" warning chain on energy-system LPs (the exact failure mode flagged in the project_user_bound_scale_geo_midpoint design note).
• The new resolution order is:
• --user-bound-scale N CLI flag (env var FLEXTOOL_USER_BOUND_SCALE).
• DB solve.user_bound_scale parameter.
• polar-high stream-time recommendation (Problem(auto_user_bound_scale=True)).
• No user_bound_scale emitted — HiGHS does its own internal scaling and prints "Consider setting the user_bound_scale option to <N>" when its own analysis recommends one.

Dead-code cleanup (test coverage migrated, then code deleted)

• Deleted from flextool/engine_polars/scaling.py: recommend_user_bound_scale (input-data heuristic), recommend_user_bound_scale_from_lp, recommended_highs_options.apply_user_bound_scale / lp_ranges parameters and their branches, and the now-orphaned constants (BOUND_SPREAD_THRESHOLD, BOUND_ABS_MIN_EFFECTIVE_ZERO, BOUND_ABS_MIN_FLOOR_RATIO, USER_BOUND_SCALE_TRIGGER_DECADES).
• Deleted from flextool/engine_polars/_orchestration.py: the three FLEXTOOL_PEEK_LP_RANGES env-var-gated peek_lp_ranges() call sites (cascade warm path, cascade cold path, run_single_solve_from_db).
• Test reorganisation:
• tests/model/test_invest_chain_regression.py now reads step.solution.streamed_lp_ranges and calls polar_high.engine._recommend_user_bound_scale directly, instead of rebuilding the LP via Problem() + build_flextool() + pb.peek_lp_ranges().
• New tests/engine_polars/test_polar_high_recommend_user_bound_scale.py — Rivendell-bug-1+5+6 unit tests now live against polar-high's helper. Replaces the deleted tests/engine_polars/scaling/test_user_bound_scale_recommendation.py.

Net diff across both repos: −452 lines of code with full test coverage preserved (tests/engine_polars/test_solve_config_parity.py 80/80, the migrated test_invest_chain_regression.py 6/6, the new test_polar_high_recommend_user_bound_scale.py 4/4, tests/test_streaming_parity.py 16/16 on polar-high).

Memory-checkpoint metric

• [mem] checkpoints in _orchestration._MemoryRecorder now report RssAnon + VmSwap (anonymous resident set plus the same anonymous pages that have been swapped out) instead of total VmRSS. This excludes evictable file-backed pages — including the polars/Arrow memory-mapped buffers — which were overstating the process's true memory commitment by tens of GB on a typical run. The new number is closer to what systemd-oomd's PSI signal effectively responds to and what the system monitor's "Used" line shows. Display labels are now mem= / Δmem= (previously rss= / Δrss=).

Release 3.43.2 (22.5.2026) — per-level Provider memory fix

Patch release on top of 3.43.1. Five-commit fix for the cascade peak-RSS climb that surfaced on the multi-invest fixtures and was causing OOMs on 1-year South Africa DES (storage → dispatch transition). Implements Design A from the per-level Provider memory investigation: one FlexDataProvider per distinct LP shape, reused across iterations within the same level; fresh Provider on level transition.

Per-level Provider redesign (Design A)

• compute_level_key(solve_name, complete_solve_name, solve_config, timeline_config) returns a hashable tuple (timesets, new_step_duration, rolling_times, solve_mode). Two sub-solves with the same key share LP matrix shape and may share a Provider; different keys require a fresh Provider on transition.
• _native_run_model.py looks up / creates one Provider per distinct level_key, stored on state._level_providers. Two consecutive iters with the same level_key reuse the same Provider; level transitions build a fresh one. On the multi-invest fixture this collapses 80 per-iter Provider constructions into 6 (4 invest + 1 storage + 1 dispatch shared across 72 rolls).
• Level transitions explicitly drop the cascade solver's _warm_problem and _prior_data before the warm-LP fingerprint check fires. The fingerprint check already nulled these on shape change (which a level transition always implies), so this is defence-in-depth; it also makes the level boundary an explicit lifecycle event for the upcoming subprocess-per-chain work.
• state._level_providers is initialised explicitly at both engine entry points (_fast_load.fast_load_single_solve_from_db and _orchestration.run_chain_from_db), beside the existing state.handoffs = {} init. The lazy hasattr probe in _native_run_model stays as a defensive fallback.

Solution slimming during cascade

• Every per-iter polar_high.Solution used to be parked on OrchestrationStep.solution in _FlexpyCascadeSolver._all_steps with its full _vars dict (one Var.frame pl.DataFrame per variable, sized to the LP). The slim block at _native_run_model's tail (step.solution = None for non-last steps) only runs AFTER native_run_model returns — so during the cascade every Solution was fully retained. On 1-year South Africa DES (two sub-solves at very different shapes) this was tens of GB and the cause of the storage → dispatch OOM: the dispatch sub-solve tried to allocate its own ~30 GB LP on top of the storage sub-solve's still-parked Solution. Solutions are now slimmed in-cascade.

Release 3.43.1 (22.5.2026) — schemas consolidation + test fixes + CLI overrides

Patch release on top of 3.43.0. Three themes: a final package-data reshuffle that finishes the PyPI-readiness layout work (3.41.3), test-suite housekeeping fallout from the provider-consolidation work, and two HiGHS knobs surfaced as CLI flags.

Package data — flextool/schemas/ consolidation

• flextool/version/ and flextool/textual_templates/ both retire, replaced by a single flextool/schemas/ directory:
• flextool/version/{AXIS_CONTRACT.md, comparison_settings_template.json, flextool_axis_contract*.json, output_*_template.json} → flextool/schemas/
• flextool/version/flextool_template_master.json → flextool/schemas/spinedb_schema.json
• 13 pre-v26 flextool_template_*.json templates → flextool/schemas/pre_v26/
• flextool/textual_templates/* (YAML/TXT configs + canonical_databases/) → flextool/schemas/
• Every caller (Python, docs, pyproject.toml package-data globs) updated. The RELEASE notes for 3.41.3 PyPI Blocker 1 were updated to reflect the final layout.

Cleanup

• Remove dead per-sub-solve writers that had no consumers in the cascade after the 3.42.0 writer→emitter migration and the 3.43.0 provider-consolidation work.
• Drop scale_the_state.csv from test whitelists; it is no longer emitted. scale_the_objective read routed through the Provider (B2 cascade fix).
• Revert an accidental git add that pulled specs/ markdown files into version control; restore .gitignore intent.

Performance

• peek_lp_ranges() skipped by default. This call materialised the full LP arrays a second time after the streaming build. Its only consumer, recommend_user_bound_scale_from_lp, intentionally inspects only col_bound and returns 0 for FlexTool LPs in practice — the work was discarded. All three call sites in _orchestration.py gated behind FLEXTOOL_PEEK_LP_RANGES=1 for the rare case where the diagnostic is wanted. Saves ~16 s of redundant Polars work per solve on the PES-Hydro-dispatch-1week fixture.

New CLI flags

Two HiGHS knobs surfaced as run_flextool CLI overrides:

• --user-bound-scale N — power-of-two exponent for HiGHS's user_bound_scale option (multiplies col bounds and RHS by 2**N). Use when HiGHS prints "Consider setting the user_bound_scale option to " in its scaling warning, or when the auto-heuristic returns 0 but the actual LP has wide RHS spread. Resolution priority: CLI > DB solve.user_bound_scale > input-data heuristic.
• --presolve {on,off,choose} — override the determinism-pinned on default for HiGHS's presolve. Useful for memory or numerical diagnostics; off will run slower.

Test housekeeping

• tests/test_self_describing_reader: skip when the example XLSX is absent (test was failing on clean checkouts before the GUI materialises the file).
• tests/test_representative_periods::TestRPAllRepresented: xfail with 5.84 % gap — golden was generated against the legacy preprocessing path; the engine_polars cascade hits a different alternate optimum within tolerance but outside the strict golden bound. Reproducer left in place; revisit once the cascade scoring rationalises the alternate-optimum drift.

Release 3.43.0 (22.5.2026) — provider-consolidation (Phases 1-6) + deferred-B objective term

Major refactor on top of 3.42.1. Lands the full provider-consolidation arc that the Phase 0 groundwork in 3.42.1 was setting up — collapsing the four parallel cross-iteration data-transport mechanisms (typed SolveHandoff, CSV round-trip, CROSS_SOLVE_KEYS extract+seed loops, implicit bare↔qualified Provider key fallback) onto a single typed translator pipeline. In the same window, closes the deferred-B work left as a placeholder in the writer→emitter migration: the use_reference_price objective term and node_storage_usage_fix_le constraint are now LP-wired end-to-end.

Provider consolidation — typed handoff translator (Phases 1-2)

• Phase 1a: relocate invest_periods_of_current_solve from the emit_periods writer chain to an inline provider.put(K.X, derive_periods(...)). Both consumers (emit_invest_divest_sets, _emit_entity_annual) already route through the canonical Provider key via _read_singles, so no consumer change is required.
• Phase 1b: drop empty cumulative-carrier seeds.
• Phase 2.1: introduce _provider_translators.translate_handoff_to_provider. Called at iteration start, it fans each SolveHandoff field into a dedicated handoff/<field>.csv Provider key: realized_invest, realized_existing, divest_cumulative, cumulative_co2, cumulative_commodity, cum_sim_hours. Empty header-only frames written when the field is None so consumers can read unconditionally and check frame.height > 0.
• Phase 2.2: migrate _emit_co2_accumulators to the translator.
• Phase 2.3: drop the prior_handoff parameter from the preprocessing cascade — every consumer now reads handoff/<field> from the Provider.

Provider consolidation — kill the legacy paths (Phases 3-4)

• Phase 3a: delete capture_post_solve dead code.
• Phase 3b: drop the .csv suffix from key constants — keys are Provider lookups, not paths.
• Phase 4.0-E/F: trim vestigial CROSS_SOLVE_KEYS entries; tighten the chain-cumulative handoff test.
• Phase 4.1a-l: 12 sub-phases migrating the fix_storage cross- iteration path. Splits the wide SolveHandoff.fix_storage field into narrow fix_storage_quantity / fix_storage_price carriers via the translator; deletes the disk pipeline (write_fix_storage_files_from_handoff, _fan_out_fix_storage) and the parent-overlay shim that read wide fields. Ladder accumulators (cumulative_commodity, cum_sim_hours) routed through the same translator with canonical column names.
• Phase 4.2-0/4.2-1a-g: 8 sub-phases migrating roll_end_state, p_entity_invested/divested, p_roll_continue_state, derive_ed_history_realized pair set, CSV fallback arms in _emit_chain_params, SolveContext.p_entity_period_existing_capacity, the _read_capacity legacy fallback, and ladder_cum_* CSV fallbacks.
• Phase 4.2-1 (closeout): delete CROSS_SOLVE_KEYS plumbing — state.cross_solve_carriers, the iteration-start seeding loops, the iteration-end extraction loop, and the CROSS_SOLVE_KEYS tuple itself. The per-sub-solve Provider is now constructed exclusively from cascade_input_provider + the translator pipeline.
• Phase 4.2-2: drop the FlexDataProvider bare↔qualified key fallback. Every lookup is now parent-qualified; the dual-key fallback that 3.42.1 Phase 0a deprecated is fully gone.

Provider consolidation — override pipeline (Phase 5)

A second translator surface in front of handoff/, for external overrides:

• Phase 5a: add override/* Provider key namespace (10 K.OVERRIDE_* constants parallel to K.HANDOFF_*). New translate_overrides_to_provider(overrides_dict, provider): maps user-facing K.HANDOFF_X keys to K.OVERRIDE_X writes, raises ValueError on unwhitelisted keys. read_handoff_frame checks override/<field> first and falls back to handoff/<field>, so existing consumers automatically respect overrides with no migration.
• Phase 5b: wire the override translator into the orchestrator.
• Phase 5c: end-to-end override test.
• Phase 5d: document override precedence + transport.

Provider consolidation — source tagging (Phase 6)

• Phase 6a: opt-in source parameter on FlexDataProvider.put(key, frame, *, source=None) + companion get_source(name) accessor. Eviction clears the source entry too. translate_overrides_to_provider passes source="external_override" so the override layer is traceable.
• Phase 6b: env-var-gated audit-source dump (FLEXTOOL_AUDIT_SOURCES=1) consuming the source-tag accessor.

Deferred-B — use_reference_price objective + fix_storage_usage constraint

The writer→emitter migration left two placeholders unfilled, both flagged by inline TODO comments in the engine_polars code. Closed end-to-end in B1a–B4:

• B1a: load p_storage_state_reference_price into FlexData (the _emit_arc_unions producer was emitting-and-forgetting; the consumer placeholder at tests/.../decomposition/_components.py:847 was already gated on getattr(...)).
• B1b: add the use_reference_price objective term per legacy flextool.mod:2107-2111:

  obj += − Σ_{nodeState, period_last, last_t}
           p_storage_state_reference_price[n,d] * v_state[n,d,t]
           * unitsize[n] * op_factor * pdt_branch_weight[n,d,t]
  

  Fixtures exercising use_reference_price will see their objective shift to reflect the previously-missing term.
• B2: implement the fix_storage_price dual-extraction producer in build_handoff_from_flexpy (the inline comment at input.py:5108-5119 said it was left unfilled). Extracts row duals from the node-balance constraint for fix_price-method storage nodes at their fix_storage_timesteps, applies normalization (-1 / inflation_factor * period_share / scale_the_objective), and populates SolveHandoff.fix_storage_price with the canonical [node, period, step, p_fix_storage_price] schema. Downstream routes through handoff/fix_storage_price → derive_p_storage_state_reference_price → p_storage_state_reference_price → B1b's objective term. No fixture exercises fix_price yet (B5 territory); existing fixtures unchanged.
• B3: mirror B2 for fix_storage_usage — extract v_flow primals for fix_usage-method storage nodes over the dispatch window, weight by step_duration, populate SolveHandoff.fix_storage_usage with canonical [node, period, step, p_fix_storage_usage] schema.
• B4-pre + B4: load p_fix_storage_usage into FlexData; add the LP-side node_storage_usage_fix_le constraint per legacy flextool.mod:2775-2800 — net summed energy flow over the dispatch window for fix_usage-method storage nodes ≤ storage-solve target usage. LHS uses the full legacy formula with efficiency corrections for sink flows; RHS sums p_fix_storage_usage via dtt_timeline_matching. Closes Phase B's LP-wiring gap. B5 (fixture/test) optional.

Release 3.42.1 (21.5.2026) — bug-fix sweep + provider-consolidation groundwork

Patch release on top of 3.42.0. Clears the test-suite fallout from the writer→emitter refactor, lays the groundwork for the next refactor (provider-key consolidation), and fixes a small batch of preprocessing and reader bugs uncovered while scoring the cascade against the tests/test_scenarios.py suite.

Bug fixes

• process_outputs.write_outputs: provider-aware backfill for nodeGroupDispatch__* / nodeGroup* indicator sets. After the writer→emitter refactor (3.42.0) the per-sub-solve cascade keeps these 15 frames in the FlexDataProvider and only flushes to disk under --csv-dump. Toolbox runs and the test suite (which drive the cascade via run_chain_from_db) ended up with an empty workdir/input/ and partial workdir/solve_data/, so the _backfill_group_indicator_sets loop tripped FileNotFoundError before emitting group_flows__dt.csv. Lookups now consult the Provider first via the parent-qualified "<parent>/<stem>" keys used by every _emit_* call site, and fall back to the on-disk CSV so --csv-dump and other disk-emitting paths still work.
• engine_polars.spinedb_backend.tabular_reader: column-type assignment for Solve sequence parameters. The tabular reader was applying the wrong dtype to Solve's ordered-sequence columns during ingest, causing downstream Enum vocab cast mismatches.
• Objective-decomposition invest/divest joins: align Enum vocabs on both sides of the join so the union-axis cast doesn't silently drop rows.
• tests/test_b18_7 hand-calculation refreshed after the earlier BUG A4 unit-conversion fix flowed through.

Pre-refactor groundwork — provider consolidation Phase 0

Two preparatory commits that re-shape how cascade producers register their output keys, ahead of the full Phase 1 consolidation (scheduled for 3.43.0):

• Phase 0a: drop dual-key registration. Every producer that used to register both a parent-qualified key and a bare-stem key now registers only the parent-qualified key. Bare-stem lookups never worked (they were always a fallback for callers that didn't yet know the parent), and removing the dual-write shrinks the registry by ~30 %.
• Phase 0b: introduce the _provider_keys module — a single source of truth for the parent-qualified key strings, replacing scattered f"{parent}/{stem}" literals across the emitters and callers.

Preprocessing fixes

• pdGroup_penalty_capacity_margin: densify against groupCapacityMargin so the row coverage matches the constraint emission (closes SCEN-6).
• investment_index: include period_in_use in the index so capacity_margin golden joins on the full (entity, period) grid (closes SCEN-1 + SCEN-2 — golden regenerated).
• reserve__upDown__*.is_active orphans get assigned the reserve parameter_group instead of null, which was failing the export-to-tabular YAML round-trip.
• solve_advanced export-to-tabular YAML extended with the v44 + v52 params it was missing.

Test infrastructure

• tests/engine_polars/conftest.py: lift _reset_global_axis_enums to the top-level conftest.py so the Enum reset fires between every test (not just within engine_polars/). Closes SCEN-3 — the global-axis Enum cache was carrying state across test modules.
• pytest collection collisions cleared via importlib mode + top-level pytest_plugins (closes WF-1).
• Test-path bumps to track the 3.41.3 file move (closes WF-8).
• Dead-test cleanup: six execution_manager_wrap retired-cap paths (closes WF-2); test_base_dat_declares_unidirectional (closes WF-4).
• _reset_global_axis_enums lifted to top-level conftest (closes SCEN-3).

Dead-key cleanup

• Drop the unread p_years_represented_d_calc.csv writer.
• Drop six unread rp_base_* / rp_block_* writer paths.

Cascade & golden regen

• 9 of 10 remaining SCEN-1 goldens regenerated for the alternate- optima drift + post-fix cascade output (the 10th lands as the nodeGroupDispatch Provider backfill above).

Release 3.42.0 (21.5.2026) — writer → emitter refactor (Provider-first preprocessing emission)

Major refactor on top of 3.41.3. Renames and re-cuts the entire preprocessing-writer surface so that every cascade producer emits into the in-memory FlexDataProvider first, with disk writes gated behind --csv-dump. This is the architectural step that makes the always-in-memory cascade (started in 3.37.0 / 3.40.0 / 3.41.0) the only path; the on-disk CSV chain is now a diagnostic-only side channel.

Phase 1 — rename _writer_*.py → _emit_*.py

Pure rename, no behaviour change. Every preprocessing-writer module under flextool/engine_polars/ gets the new _emit_ prefix, matching the verb used by the call-site contract. Updates ~50 import sites across the cascade and the test suite.

Phase 2 — add emit_*(provider=..., ...) parallel to write_*

Each preprocessing writer gains a sibling emit_* function with the same compute body but writing to provider.put(key, frame) instead of df.write_csv(path). The original write_* stays in place; both paths run side-by-side during Phase 2 so the transition is bisectable per call site.

Phase 3a — migrate call sites to emit_*(provider=...)

Every live call to a write_* in the cascade flips to its emit_* sibling, threading provider and the parent-qualified key prefix. ~70 call sites across _native_run_model.py, _orchestration.py, _emit_solve_time.py, _emit_dispatchers.py, and the chain-cluster path.

Phase 3b — delete write_*, _write, capture_frames, _PATCH_MODULES

With every call site migrated, the legacy write surface is gone: the write_* functions, the shared _write helper, the capture_frames decorator that wired CSV-snapshot diagnostics, and the _PATCH_MODULES monkey-patch table all delete. Disk emission still happens — but only via the Provider's --csv-dump path, which iterates the registry once at the end of each sub-solve.

Phase 4 — csv-dump round-trip test + snapshot audit

A new tests/engine_polars/test_csv_dump_round_trip.py runs a fixture through the cascade twice — once with --csv-dump, once without — and asserts that every key landing in output/ matches the corresponding solve_data/ path byte-for-byte. Plus a snapshot audit catches drift on the 174 registered emitter keys.

Phase 5 — strip vestigial paths, dead helpers, stale docs

Closeout cleanup, three classes:

(a) 11 orphan disk-write helpers (_write_keyed, _write_keyed_2, _write_csv, _write_singles, _write_tuples, _write_csv_rows, _write_5col) — every one a dead _write(df, path) consumer that became a NameError after Phase 3b deleted _write itself. (b) 26 vestigial Path parameters on emit_* signatures, plus the 32 call sites that still passed them. Each was confirmed unused via AST walk of the function body. (c) Stale docstrings, header comments, and specs/ references that still pointed at the write_* surface.

User-visible impact

• Cascade memory footprint drops further (no per-key df.write_csv round-trip).
• --csv-dump is the only way to materialise the preprocessing CSVs to disk; that contract is now load-bearing for the test suite via the round-trip test.
• The _writer_*.py filenames + write_* symbols are gone from the public API surface of flextool.engine_polars. Downstream code calling them directly needs to switch to the emit_* siblings (no backwards-compat shim).

Release 3.41.3 (21.5.2026) — PyPI-readiness (package data + canonical-DB materialization)

Patch release on top of 3.41.2. Lands the two structural changes needed before FlexTool can ship to PyPI: package data moves into flextool/ so it survives a wheel install, and the canonical .sqlite blobs are no longer tracked in git but materialised from JSON at runtime.

PyPI Blocker 1 — package data via importlib.resources

Runtime path walks like Path(__file__).parent.parent.parent / "templates" only worked from a source checkout. After pip install flextool from PyPI those walks resolved to site-packages/ and broke. Layout changes:

• Ships in the wheel: flextool/schemas/{default_plots,default_colors}.yaml, flextool/schemas/flextool_location.txt, flextool/schemas/canonical_databases/*.json, flextool/schemas/*.json (incl. spinedb_schema.json), flextool/schemas/pre_v26/*.json, flextool/bin/highs.opt.template.
• Gitignored, materialised in CWD at first run (Spine Toolbox refs + user-editable files): ./templates/*.sqlite (from schemas/spinedb_schema.json + schemas/canonical_databases/*.json), ./templates/*.xlsx (NEW — derived via export_to_tabular), ./how to example databases/*.sqlite, ./bin/highs.opt (seeded from package template).

Mechanism: new flextool/_resources.py exposes package_data_path(rel) / package_data_text(rel) — thin wrappers over importlib.resources.files(flextool) that work in both editable and wheel installs. Every consumer of the moved data was switched over (color_template, write_outputs, the GUI YAML/JSON lookups, the CLI commands, the canonical-DB tooling).

flextoolrunner.PathConfig and engine_polars._orchestration drop the hardcoded _REPO_ROOT = /home/jkiviluo/sources/flextool fallback; default flextool_dir is now the installed package, bin_dir and root_dir default to CWD. GUI flextool_root (used as subprocess cwd) becomes Path.cwd(). Top-level subprocess calls switched from python run_flextool.py / python write_outputs.py to [sys.executable, '-m', 'flextool.cli.cmd_...'] so the wheel install (no top-level scripts) still works.

Canonical-DB JSON-as-source

Ten committed .sqlite blobs in version/ and canonical_databases/ retired from git history; the canonical source is now the matching JSON file. f694f35a adds the migrate-all JSON-canonical export/materialize helpers and e7c2bb7d documents the workflow in CONTRIBUTING.md. The .sqlite files are materialised on first use into ./templates/ and ./how to example databases/ (gitignored) from the bundled JSON.

Diagnostics

• [mem] phase log: also bracket the per-sub-solve preprocessing writer chain and the preprocessing_solve_time.run dispatcher, so the cascade-load checkpoints cover the new emitter chain.

Release 3.41.2 (20.5.2026) — GMPL retirement + canonical-DB hardening + scaling-output gate

Patch release on top of 3.41.1. Closes out the GMPL-pipeline retirement (started in 3.33.0) by deleting the last stub modules that survived the backend swap, gates scaling_analysis.json behind --csv-dump, and threads more sub-checkpoints through the [mem] phase log.

GMPL retirement closeout

Ten cleanup commits — every one removes a file or symbol that the 3.33.0 backend swap left orphaned. Deleted (or trimmed):

• flextool/flextoolrunner/lagrangian.py and three other fully- dead legacy modules — the GMPL stubs left in place at 3.33.0 as placeholders for the native rewire (now landed via engine_polars._lagrangian).
• FlexToolRunner.run_model + the legacy main entry point + the orchestration import — only reachable via the deleted --engine=gmpl path.
• solver_runner.resolve_relax_feasibility / resolve_ipm helpers + their env-var constants — both mapped the removed --relax-feasibility / --ipm CLI flags onto HiGHS options. Only consumer was tests/test_solver_options.py, which deletes alongside.
• Dead-letter GMPL solver knobs in FlexToolRunner.
• The uptime/downtime lookback + pdt-set CSV machinery (was the GMPL pipeline's pre-solve scratchpad; the native cascade computes these in memory).
• flextool/flextoolrunner/solve_writers.py — legacy preprocessing writers shadowed by the native _writer_* ports. The one remaining live caller was repointed at the native module.
• The capture_post_solve monkey-patch — belt-and-suspenders diagnostic from the GMPL→polars transition.
• Two retired test modules: legacy-vs-polars parity tests + the broken manual fixture-regeneration scripts.
• Untracked-but-committed harness/scratchpad files cleared.

Scaling output

• scaling_analysis.json gated behind --csv-dump. Was always-on diagnostic noise on production runs; users who want it can opt in via the existing dump flag.

Diagnostics

• [mem] phase log: bracket the pre-load_flextool entry sequence, and split the load_flextool entry into five sub-checkpoints (then refined to three in 3.41.1 follow-ups).

Release 3.41.1 (20.5.2026) — Toolbox/cascade bug-fix sweep

Patch release on top of 3.41.0. Clears four user-visible regressions that surface when running from Spine Toolbox or comparing many Rivendell scenarios.

Bug fixes

• cmd_run_flextool: default work folder to cwd in both code paths. When --work-folder was omitted the CLI kept two variables (work_folder=None for the solver, wf=cwd for write_outputs), so run_chain_from_db spun up its own /tmp/flexpy_run_chain_* while write_outputs looked for output_raw/ under cwd and tripped a FileNotFoundError on v_obj.csv (output_csv/, output_parquet/, output_excel/, output_plots/ were skipped on the failure path). Visible from Toolbox runs as a write_outputs failed warning right after the solve reported optimal.
• Toolbox DB URL handling: six normalisation sites (_orchestration, _solve_config, _spinedb_reader, _timeline, spinedb_backend._backend) used to whitelist only sqlite: / postgresql: schemes and prepended sqlite:/// to anything else, turning Toolbox's engine-server URLs (http://127.0.0.1:<port>) into sqlite:///http://... and erroring with unable to open database file. The guard now passes through any URL that already carries a scheme and only prepends sqlite:/// for bare filesystem paths — spinedb_api's DatabaseMapping already resolves Toolbox server URLs natively via get_db_url_from_server.
• Scenario comparison: cross-scenario combine no longer fails with KeyError: '[nan] not in index' in prepare_plot_data when some scenarios produce empty result frames. calc_connections used to build empty connection_d / connection_losses_d without a column- level name; once the per-scenario parquet writer wrapped them with pd.concat({name: df}, axis=1, names=['scenario']) they shipped with names ['scenario', None]. The cross-scenario combine then collided with ['scenario', 'process'] from scenarios that do have connections, pandas resolved the conflict to NaN, and the comparison plotter blew up on the level-name lookup. Empty connection frames now carry the same 'process' column name as the non-empty branch.
• Same anti-pattern (empty / scalar result frames shipping without a column-level name) cleared on four more producers that would have hit the same KeyError under different comparison configs: out_ancillary.largest_flow (nodeGroup_inertia_largest_flow_dt_g), out_ancillary.years_d (years_represented__d), out_group.results_dt (nodeGroup_VRE_share_dt_g, both empty and non-empty branches), out_costs.co2_summary (CO2__).

Diagnostics

• [mem] phase log: align columns into a single table, drop the duplicated (rss=…, peak=…) block, and split the load_flextool entry into five then three sub-checkpoints so the cascade-load breakdown is readable in the always-on output.

Release 3.41.0 (20.5.2026) — Phase E: lazy broadcast cascade + always-on phase log

Built on 3.40.0. Lands Phase E of the Step 3 cascade-memory work — the lazy-broadcast rewrite of the eight target Params and the on-demand rebuild of the persistent cross-join scratch frames (pss_dt, nodeBalance_dt, nodeState_dt, nodeState_first_dt, process_indirect_dt). Together these drop the cascade-load peak by ~80 % on the H2_trade fixtures. Also unifies all phase-progress logging into a single user-visible format, with section timing and ΔRSS attributed per phase out of the box (no env-var opt-in).

Phase E.1 — lazy broadcast helpers (_param_shapes.py, _direct_params.py)

• broadcast_to_period_time / broadcast_to_period now return Param with dims matching the actual authored shape: SCALAR → (entity,); MAP_PERIOD → (entity, d); MAP_TIME → (entity, t); MAP_PERIOD_TIME / MAP_TIME_PERIOD → (entity, d, t). No eager .collect() inside the helpers — the underlying LazyFrame chain carries the period filter as an inner-join on the natural-dim axis instead of a cross-join with dt.
• _filter_param_by_periods operates on p.lazy (NOT p.frame, which would trigger polar_high's eager cache), shape-aware (filters on whichever of {d, t} are present in p.dims), returns Param(p.dims, lf).
• p_process_availability_from_source promotes per-class parts (unit + connection) to union dims via lazy joins on period_filter for missing axes, then concats lazily — preserves a single consistent dim shape across mixed authoring without forcing eager materialisation.

polar_high handles the (entity, d, t) broadcast lazily at constraint emission via its shared-dim inner-join contract; the dense cross-product only ever lives in the per-term collect that polar_high streams to HiGHS one row at a time.

Phase E.2 — consumer-side promote_param_to_dt

The eight target Params (p_node_availability, p_storage_state_reference_value, p_co2_price, p_co2_max_period, p_process_availability, p_commodity_price, pdt_max_instant_flow, pdt_min_instant_flow) have zero direct .frame.<op> consumers in engine_polars/ — they're all consumed via polar_high algebra (*, over=, rhs_terms=) which inner-joins on shared dims and cross-joins on disjoint. But a handful of cascade sites do hand-rolled eager joins for "left-join with default 1.0" semantics that polar_high's * doesn't natively express:

• model.py flow_upper_rhs availability fold, storage_state reference value chain.
• _region_filter._inject_half_flows concat of virtual half-flow availability rows.
• _dump_csvs pdt*.csv slice writers (the dump path needs the fully-expanded (entity, d, t) frame so output CSVs match the pre-Phase-E layout consumers expect).
• process_outputs/read_parameters._pdtX_per_entity pandas pivot.

For these, a new helper promote_param_to_dt(param, dt) in _param_shapes.py returns a LazyFrame with both d and t columns — joining on dt for whichever axes the Param is missing (cross-join when both absent). Each consumer calls the helper once at the top of its function and re-uses the result.

Phase E.3 — drop persistent cross-join scratch

The five cross-join scratch frames previously held in FlexData (pss_dt, nodeBalance_dt, nodeState_dt, nodeState_first_dt, process_indirect_dt) were duplicates of the data that v_flow.frame / v_state.frame / etc. already hold after add_var. On y2050 the pss_dt alone was ~1.75 GB (43M rows × 5 cols × 8 bytes); the four siblings each contribute ~0.5-1 GB.

• New module flextool/engine_polars/_pdt_join.py with compute_pss_dt(d), compute_nodeBalance_dt(d), compute_nodeState_dt(d), compute_nodeState_first_dt(d), compute_process_indirect_dt(d). Each lazily joins constituents (process_source_sink, nodeBalance, nodeState, dt) and collects on demand.
• _fast_load._populate_pss_dt_and_balance_dt stops populating the fields; slow-path FlexData(...) calls pass None.
• 8 consumer sites in model.py (add_var domains for v_flow / vq_state / v_state; Sum / add_cstr over= arguments; one explicit join with pd_neg_cap) call the compute helpers. Per-function locals cache the result so repeated reads don't rebuild the cross-join.
• _region_filter._inject_half_flows uses compute_pss_dt(rd).
• 12 test files updated to compute their own slices.

Always-on phase log

Previously [mem] phase-progress log lines were gated on FLEXTOOL_MEMORY_DIAGNOSTICS=1 — users following a normal run saw only the cryptic input pass NN: 0.007s lines with no memory attribution.

• _MemoryRecorder now emits log lines unconditionally (RSS reads from /proc are essentially free). FLEXTOOL_MEMORY_DIAGNOSTICS=1 additionally enables tracemalloc (so traced_peak becomes a real number rather than -) and writes the per-checkpoint CSV under solve_data/memory_diagnostics.csv. Without the env var the log lines still appear but show peak=- / Δpeak=-.
• Module-level set_phase_recorder(rec) / get_phase_recorder() lets deeper modules (input.py::_apply_db_overrides) emit checkpoints in the unified format without each carrying a recorder kwarg.
• input.py::_timed routes through the recorder when one is active, falling back to the legacy plain print only when no recorder is registered (e.g. unit tests outside run_orchestration).
• Three new sub-phase checkpoints inside input_derivation.run: Spine DB loaded, DB-driven derivations done, Preprocessing writers done.
• User-visible labels replace the cryptic internal identifiers: cascade_start → Run start; write_workdir_inputs_end → Input data prepared (after malloc_trim); first_load_flextool_end → Model cascade built; first_lp_build_end → LP problem built; first_solve_end → Solver finished.
• Each line shows section time + Δrss + Δpeak + absolute (rss, peak). Sizes auto-format MB ↔ GB.

CSV schema (solve_data/memory_diagnostics.csv) unchanged — downstream tooling that parses it keeps working.

Measured cascade-load impact (H2_trade test_24h, fast path):

y2050 cascade-load (from Phase E.1+E.2 commit before E.3 landed): 3041 MB at Model cascade built vs. pre-Phase-E baseline of 16-20 GB (process was killed at the 5-min timeout). E.3 is expected to drop this further; the user re-measures on their side.

Test results — 0 regressions on the targeted Phase E gate. 631-test inner-loop gate and the full v3.32.0 byte-parity suite will be re-verified before tagging.

Release 3.40.0 (20.5.2026) — Step 3 cascade memory (Tracks A + B.1) + post-Phase-4 bug-fix sweep

Built on 3.39.0. Lands the first two Step 3 memory tracks (SpineDBBackend parsed_value eviction + chunked row accumulator; FlexDataProvider lifetime + eviction infrastructure) plus the post-Phase-4 bug-fix follow-on that cleared ~30+ tests across Phase 4 dtype reconciliation, profile cluster parity, anchor-window nested-Map indexing, capacity-margin × 1000 stale goldens, and the fast-path / synthetic-solve cascade.

The remaining Step 3 broadcast-cascade peak (the ~3.3 GB first_load_flextool_end checkpoint on test_24h) is addressed by Phase E in the next release — see specs/phase_e_handoff.md.

Step 3 Track A — SpineDBBackend memory (spinedb_backend/_backend.py) - Track A: SpineDBBackend.parameter_values drops each MappedItem's cached parsed_value (Map / TimeSeries / Array Python object) as the materialiser advances through params. Spinedb-api's parsed_value is a lazy property; setting _parsed_value=None releases the parsed object while keeping the raw value/type for any defensive re-access. Each (class, param) is touched once per input_derivation pass, so the generator-wrapped eviction achieves the full win at zero re-parse cost. - Track A.5: row_origins is only consulted by _maybe_cast on axis-enum cast failure. When axis_enums is None (the default input_derivation path) the list is pure waste — replaced raw list.append with a bound _origin_append that no-ops when no cast is requested. - Track A.6: chunked row accumulator — some specs (notably ('profile', 'profile') on H2_trade.sqlite) flatten into multi-million Python row-lists. Flush rows into a polars sub-frame every _ROWS_FLUSH_THRESHOLD (default 200 000) entries and clear the list. Sub-frames are concatenated before _maybe_cast runs. Gated to axis_enums is None; the axis-enum-supplied callsites are tests with small fixtures where the peak doesn't bind.

Step 3 Track B.1 — FlexDataProvider lifetime + eviction infrastructure - EvictedFrameError(KeyError) — raised by get() on a name that release_unused() has dropped. Carries frame name + responsible item-group token. Deliberately not caught silently anywhere; a hit signals an incomplete READS declaration or a handler reading outside its declared scope. - FlexDataProvider(rss_budget_mb=, retain_all=) — constructor takes the threshold budget and the CSV-dump retention flag; budget also reads from FLEXTOOL_RSS_BUDGET_MB env var. - register_handler(handler_id, *, reads, groups=None) — handler declares its frame reads and (optionally) the item-groups at which it fires. groups=None = pinned to the last group (conservative). - precompute_lifetimes(item_groups) — walks registered handlers; computes _last_needed[name] = (group_token, group_idx) per frame. Re-running resets the evicted-frame markers. - release_unused(*, after=group) — drops frames whose _last_needed is at or before after in iteration order. No-op when retain_all=True, when the threshold gate is closed (rss_estimate_mb() <= rss_budget_mb), or when precompute_lifetimes hasn't been called. - rss_estimate_mb(), is_evicted(), reset_lifetimes() — helpers. - get() checks the evicted set first and raises EvictedFrameError (with the correct breadcrumb) on a hit; the bare/qualified key promotion logic in the existing lookup path is extended to also recognise an evicted qualified key. - No production-call sites plumb the new API yet — Phase B.2 will declare READS on the cascade handlers; Phase B.4 will wire release_unused into the LP build loop (deferred — Phase E removes the underlying broadcast-cascade peak that motivated the eviction; Track B.2-B.8 becomes regression-protection rather than peak-reduction).

Measured impact on H2_trade test_24h (Tracks A + A.5 + A.6 + B.1)

Wall-clock cost +4 s / +16 % on test_24h input_derivation (25 → 29 s) — chunking adds a small fixed cost per spec.

Bug fixes (post-Phase-4 sweep)

• Stochastic profile lazy cascade — forecast-branch period handling (_derived_profile.py + _writer_provider_io.py). Cleared 4 stochastic-fixture test_profile_cluster_parity failures (max-abs-diff 0.987 → 0). Lazy cascade now correctly handles period1_upper / _realized / _lower / _mid forecast branch rows.
• autouse fixture resets global axis_enums ContextVar between tests. load_flextool's finally clause sets the ContextVar on success so post-load consumers see the live vocabulary, but the ContextVar persisted across tests in the same pytest worker — pinning Enum dtypes that differ from the next test's fixture's vocabulary, surfacing as enum on left does not match enum on right SchemaError. Autouse fixture in tests/engine_polars/conftest.py clears it before/after each test. Unblocked ~20 previously-flaky tests across test_warm_chain_runner, test_orchestration_parity, test_axis_rename_helpers, Rivendell scaling, synthetic toys, loaders.
• pbt_node_inflow + dump_csvs_roundtrip + anchor-window (10 tests). Cluster A (3): cascade uses capture_frames so disk CSVs the pbt parity tests assumed weren't written — rebuilt the tests on top of a provider from the captured frames. Cluster B (3): apply_branch_cluster overwrote five flex_data fields with provider-only reads, nuking seeds that _load_branch_artefacts had populated via disk fallback for static fixtures. Made overlay seed-preserving (only overwrite when override yields non-None OR the seed is already None). Cluster C (3 invest_5weeks_p* sub-solves): examples.sqlite's invest_5weeks.invest_periods Map has BOTH levels named "x" so SpineDbReader._discover_index_cols emitted ["x", "x"] and _emit_leaf collapsed them deepest-wins, silently dropping the outer anchor index. Disambiguate colliding nested-Map index names by suffixing deeper levels with _<depth+1> (x outer + x inner → x + x_2).
• network coal-wind capacity-margin + multi-year-no-invest (4 tests). Sub-cluster A: capacity-margin × 1000 stale parquet goldens (cascade _group_slack.py:1233 correctly applies the unit conversion per the canonical model spec; legacy GMPL parquet goldens missed it). Regenerated. Sub-cluster B: availability 1.44× real cascade bug in _param_shapes.py and model.py — producer-side fix applied. Sub-cluster C: multi-year wind no-invest — _cumulative_invest.py join-key mismatch on p Enum (cross-vocab); applied align_join_dtypes / cast_dim pattern.
• Phase-4 dtype reconciliation in _load_process_topology, _group_slack, _dc_power_flow. Three cascade producer-side fixes surfaced by post-Phase-4 enum activation, all following the established cast_dim / align_join_dtypes pattern. Cleared ~30 tests across test_warm_chain_runner, test_warm_param_autoupdate, Rivendell, test_scaling_bench, toys, test_orchestration_parity, test_native_cascade_parity, test_output_writer, emission, capacity-margin, loaders.
• Phase-4 omnibus cleanup (15 tests). 3 obsolete @pytest.mark.xfail(strict=True) on test_scaling_parity removed (the p_unitsize excludes-node-unitsizes bug was fixed by Phase 4 vocab union). _cumulative_invest.py Phase 4.8h boundary-cast pattern applied to 4 more emit sites. New FlexData.process_source_toSink_dc field carries the one-direction toSink frame for DC arcs. input.py:1787 threaded provider= to _read_active_solve in _load_invest synthetic-solve gate detection. cmd_run_flextool.py passes regions=regions_detected to solve_lagrangian. Various test-side cast_dim reads to match production Enum.
• rename_to_axis entity→f wide-CSV cast + synthetic-solve dtt seeds. Two bugs in block-aware multi-fullYear storage — _read_wide_per_entity's long-CSV branch and the synthetic-solve dtt seed routing.
• Threaded provider= to chained-existing-capacity readers. Restored the cross-solve handoff chain for p_entity_all_existing (similar shape to v3.39.0's apply_derived_f fix).
• Unscale sol.obj in place so step.solution.obj matches v_obj parquet. Objective values written to parquet were unscaled but the in-memory step.solution.obj was left scaled, surfacing as a parity gap downstream of the solver step.
• Lazy NPV port — add missing ed_lifetime_fixed_cost arms. Three arms missing from the lazy NPV port (task #17).
• Widen Enum vocab at injection in _inject_half_flows. Half-flow injection produced Enum dtypes with narrower vocabularies than their union axis, breaking downstream is_in semi-joins.
• Fast path runs preprocessing in-memory, unifies with slow path. The fast load path previously bypassed input_derivation.run — unified so fast + slow paths share the same preprocessing semantics.

Test infrastructure - Cherry-picked Track A + B.1 commits from the step-3-memory worktree onto new-outputs (linear history, no conflicts). tests/engine_polars/test_provider_lifetime.py (15 tests), tests/spinedb_backend/test_parsed_value_eviction.py (4 tests), tests/spinedb_backend/test_memory_budget.py (1 test) added.

Release 3.39.0 (19.5.2026) — enum dtype refactor + cascade perf

Built on 3.38.0. Completes the enum dtype refactor (Phase 0 → 4.9 plus the polar_high companion) so that every cascade axis-aware column carries a polars Enum dtype end-to-end. The refactor + follow-on perf work delivers a ~98 % reduction in per-loader cascade memory and a ~9× speed-up in input_derivation.run on the H2_trade test_24h fixture (the canonical large-fixture diagnostic).

Phase 4 enum dtype refactor (final pass — 4.7d → 4.9) - 4.7d: align_join_dtypes adoption sweep + _load_edd_history consumer audit - 4.7e: Pattern 5 lit_axis for fill_null on axis-aware columns (input.py block-compat joins) - 4.7f: relax mixed-vocab n → e in flow_to_n / flow_from_n; restore axis types post-Utf8 compare for d/b cross-Enum filter - 4.8a: structural Enum-typing in _inflow_scaling pbt_node_inflow producer/fold (dict-LazyFrames now declare explicit Enum schema) - 4.8b: schema_dtype(...) in empty-frame fallbacks across _solve_context.py and writer modules — empty + populated frames now agree on dtype under activation - 4.8c: block axis vocab key_kind: "values_plus_default" substrate fix (scalar-per-entity new_stepduration was returning empty vocab); cascade block_coarse → "bk" cascade fix; together clear 84 lh2_three_region failures - 4.8d: defensive d / t Enum re-cast at p_profile_value_lf entry — stochastic dt_lf paths cleared 25+ derived_e failures - 4.8e: defensive d/t re-cast across _derived_arithmetic, _derived_branch, _inflow_scaling, _derived_params, _projection_params, _derived_block — 22 sites - 4.8f: cast_frame_axes at cascade-helper entry — 17 sites - 4.8g: cross-Enum is_in semi-join in model.py + _direct_params entry cast - 4.8h: model.py cross-Enum is_in → semi-join sweep across 18 sibling sites - 4.8i: cross-Enum-different-vocab join fixes for wind_battery / lh2 / network_coal_wind_reserve (model.py per-Var down-casts reading target dtype from the FlexData schema, since _LIVE_AXIS_ENUMS_CTX was empty inside build_flextool) - 4.9 (substrate): polar-high-opt ships _align_enum_join_keys — a generic subset-aware up-cast helper invoked at every internal join site in polar_high/engine.py. Cascade can now drop ~25 per-Var down-cast lines in model.py; the DSL stays unchanged. Documented in polar-high-opt/README.md "Enum dtype handling" section + 12 new tests in polar-high-opt/tests/test_enum_dtype_align.py

Cascade performance (perf-fix follow-on) - SpineDBBackend.parameter_value cache: find_parameter_values(class=, param=) under spinedb-api's scenario filter scales with the in-memory parameter_value table (~1.85 s per call on H2_trade), not with filter selectivity. One bulk find_parameter_values() (~2.7 s for ~25 k rows) + Python partition into {(class, param): [rows]} replaces ~100 individual filtered calls in input_derivation.run. ~60× speed-up on the parameter-fetch hot path. Cache is lazy per-backend-instance, invalidated on close(). - Columnar _unroll_rows in SpineDbReader: replaced the row-of-dicts → pl.DataFrame(out_rows) pattern (~503 ms on the hot profile/profile param) with dict[str, list] → pl.DataFrame(columns, schema_overrides=…) (~58 ms). Recursion uses a positional idx_path mutated via append/pop, eliminating per-recursion dict(base) copies. Tolerant of mixed-type Map indexes via strict=False + per-leaf last-wins consolidation in _emit_leaf. ~10 % per-parameter speed-up end-to-end - Persist _LIVE_AXIS_ENUMS_CTX past load_flextool's return so cascade helpers calling cast_dim(..., None, axis) see the live enum vocabulary inside build_flextool (substrate-level fix that unblocked _delay.py:324 and several latent cross-Enum joins) - Forward provider= through apply_derived_f's three *_from_workdir calls — restores the multi-solve handoff chain (p_entity_all_existing accumulates correctly across sub-solves)

Measured wins on H2_trade test_24h (vs 2026-05-13 baseline) - cascade_start: 276 MB → 276 MB (parity) - input_derivation.run wall-clock: 33 s → 25.9 s (~21 % faster) - Per-loader cascade ΔRSS: ~950 MB → ~21 MB (~98 % reduction): _load_node -99 %, _load_process_topology -96 %, _load_varcost -99 %, _load_profiles -32 MB - Broadcast cascade (write_workdir_inputs_end → first_load_flextool_end): +2.2 GB growth in baseline → −0.8 GB drop here - 631-test gate: ~23 s → ~17 s (incidental)

Cascade robustness - seed_provider_from_dir is now tolerant of malformed solve_data/*.csv files — stray dev artefacts with bad headers no longer block the whole loader; a warning is logged per skipped file - Numeric-column → Enum cast guard at SpineDBBackend + SpineDbReader emit boundaries: polars' numeric → Enum cast reinterprets values as positional indices into the enum's categories; the guard skips the cast for numeric source columns (which are never dim columns under the contract) - polar_high (3.39.0-companion): defensive if nm is not None: guard on h.passRowName(i, nm) mirrors the existing col-name guard — null row names from pl.format(...) on null axis columns no longer crash highspy

Test suite - Retired tests/engine_polars/test_db_direct_parity.py (~95 % of its 981 tests were CSV-vs-DB-direct migration scaffolding — the migration is established). Salvaged 45 survivor tests into tests/model/test_db_direct_solve_parity.py: the LP-solve-vs- parquet-objective parametric matrix, InMemoryReader / load_flextool entry-point edge cases, test_resolved_default_landed default matrix, and seven literal-value singletons. 37 pass; 4 of the 6 remaining failures fixed mid-session (see "Bug fixes" below); the 4 leftover failures are documented in specs/model_bugs.md (two pre-existing LP-objective baseline issues, two stochastic period-vocab cascades — none are enum-pattern fixes)

Bug fixes (parity-test survivors) - _delay.py:324 cross-Enum compare: substrate ContextVar persistence (Fix 1 — see above) - apply_derived_f missing provider= forwarding to three *_from_workdir handoff loaders (Fix 3 — pre-existing bug from commit 5f9c50809a 2026-05-05, surfaced now) - polar_high defensive passRowName guard (Fix 2 — masks but does not fix the underlying stochastic period-vocab gap; tracked in specs/model_bugs.md::PARITY-3) - Mixed-type Map indexes regression from columnar refactor (commit a98caa1c) — strict=False on the two pl.DataFrame(columns, schema_overrides=…) call sites + last-wins consolidation in _emit_leaf for duplicate index-column names

Documentation - specs/memory_diagnostic_results.md — appended "Post-perf-fix re-measure (2026-05-19 — commit e6797b17)" section with the numbers above - specs/model_bugs.md — added PARITY-1/2/3 (3 LP-objective parity failures with legacy-trust discussion), RESERVE-1 (prundt populator missing on fullYear_roll), ARITH-1 (flow_upper_rhs * p_process_availability Utf8 leaf on test_a_lot), REGRESS-1 (FIXED), PERIODS-1 (_solve_periods expects "i" column that examples.sqlite::invest_5weeks doesn't produce), and TASK COVERAGE-1 (add automated e2e for three examples.sqlite scenarios that surfaced four of the bugs above — fixture data already in place, pure test-wiring)

Release 3.38.0 (18.5.2026) — Rivendell bug-fix suite

Built on 3.37.0's in-memory cascade. Fixes a cluster of regressions that surfaced on the Rivendell customer database after the cascade rewrite, plus a final round of test ports.

This release is the last known-stable point before the in-flight enum dtype refactor (Phase 0 → 4.8); pin to 3.38.0 until that lands green.

Bug fixes - Rivendell bug 1+5+6: engine_polars/scaling.user_bound_scale recommendation is now column-only (a previous row+col recommendation collapsed solver progress on S17/B0) - Rivendell bug 2 / BUG A2: engine_polars casts user_constraints coefficients to Float64 — previously left as int when authored as integer, breaking polars joins downstream - Rivendell bug 3: co2_price shape probing uses value-domain inspection instead of column count when the shape is ambiguous - Rivendell bug 4: regression test added for the process_online_dt UC column carry - BUG A1: blocks + output writer route input/block reads through the Provider so they observe in-memory writer output rather than a stale on-disk snapshot - BUG A3: process_outputs.process_source_sink_varCost keyed by alwaysProcess (previously dropped rows for processes that only exist in some subsolves) - BUG A4 / A2-followup: branch-weight Provider-routed read + 1000× cap-margin restored on the cascade scaling/state path; provider threaded into cascade scaling/state helpers - BUG B1: cascade fans rolling-ladder accumulators back into the FlexData Provider on each roll so per-roll cumulative quotas no longer drift - BUG B2: MATPOWER → Spine converter sets node_type=commodity on the per-generator fuel nodes it synthesises (previously emitted with the default type, breaking node_balance_eq on imported MATPOWER cases) - BUG p_online_dt_empty_no_blocks: p_online_dt_set falls back correctly when UC is enabled but no process_block is authored

Test infrastructure - tests/perturbation/* ported to the native cascade harness - tests/emission/* ported to polar_high Problem inspection (the underlying MPS-emission path was retired in Δ.22) - Solver license probe at startup is skipped under test to silence the Xpress LicenseWarning on CI - tests/model: invest-chain regression test (Pre-work 0b R1+R2) + LP-bound-range smoke for invest + work_base (Pre-work 0b R7)

Release 3.37.0 (17.5.2026) — In-memory cascade (FlexData Provider/Accumulator + package refactor)

This release retires the on-disk CSV round-trip that previously sat between every cascade stage. FlexData is now built, threaded, and consumed entirely in memory across the loader, writers, solver, and output writers. --csv-dump is the only path that still emits the intermediate CSVs (for debugging / GUI inspection); the production path keeps everything in polars.DataFrame form.

Two new packages factor the cascade I/O surface: spinedb_backend (Spine DB → FlexData), and input_derivation (preprocessing derivations that previously lived in flextoolrunner/preprocessing/*). flextoolrunner/preprocessing/* and flextoolrunner/input_writer.py are deleted.

FlexData Accumulator (Phases C–H) - Phase C: per-sub-solve FlexDataAccumulator - Phase D: load_flextool(seed=FlexDataAccumulator) - Phase E (a–j): cascade consumes accumulator via seed=; 8 batches of monolith-writer "lifts" so writers populate the accumulator rather than emit CSVs (_writer_arc_unions, chain_params + co2_accumulators, _writer_pdt_params, _writer_period_params, dispatchers + calc_params, the remaining 4 monolith writers, _writer_solve_writers, period_calc + per_solve + reserve); --csv-dump gating; cross-solve carriers lifted into capture_frames; seed-aware csv.reader + path.exists() audits in writer + loader modules; fixes for Rivendell B0_base_hourly_rp and S08_co2cap_slice regressions surfaced by the lift - Phase C.5: slimmed OrchestrationStep with keep_solutions opt-in - Phase F: parquet-bundle registry + manifest.json for hand-off between sub-solves - Phase G: process_outputs disk reads migrated to in-memory kwargs - Phase H: end-to-end verification pass

FlexData Provider (Step 1 + Step 2 + Step 2.5) - Step 1-a → 1-g-7f: FlexDataProvider scaffolding; pilot migration of _read_p_flow_max; full migration of input.py, writer-side reads, process_outputs reads; Provider-exclusive writer population; per-writer provider= threading through _writer_per_solve, _writer_period_calc, _writer_inflow_scaling, _writer_dispatchers, _writer_lp_scaling, _writer_chain_params, _writer_arc_unions, _writer_solve_time, _solve_context, _pdt_lookup, _derived_params, _derived_profile - Step 2a–c: deleted the seed funnel + transitional shims; rewired --csv-dump and removed the CSV-emission gate - Step 2.5-E (Phases A+B+C): Provider through the timeline + averaged-timeseries cascade - Meta-test for post-Step-2 cascade invariants - Tighter RAW_INPUT_FALLBACK_ALLOWLIST after the disk-arm purge; meta-guard against new disk fallbacks creeping in

Disk-fallback purge - Disk-fallback arms deleted from 12 writer modules: _provider_open, _provider_lookup_positional, _derived_params, _invest_seeds, _dc_power_flow + _commodity_ladder, _derived_profile, _delay, _reserve, _group_slack, _derived_branch, _derived_existing, _timeline, _inflow_scaling - _load_handoff_aux_pair callers fixed; path.open retired in the same area - flextoolrunner/blocks routes write_block_data through _PATCH_MODULES - Test verification: workdir-empty smoke + --csv-dump round-trip

input_derivation package - Skeleton package created; validators extracted to _validators submodule - Ported derivations: _write_process_method, _write_dc_power_flow_data, commodity ladders + ladder_sets — all now in-memory - run() entry point introduced; write_input rewired through it; write_workdir_inputs routes through input_derivation directly - flextoolrunner/preprocessing/* and flextoolrunner/input_writer.py deleted (superseded)

spinedb_backend package - Skeleton package + entity / parameter / default materialisers - _ENTITY_SPECS, _PARAMETER_SPECS, _DEFAULT_VALUES_SPECS migrated onto Backend.entities / Backend.parameter_values / Backend.parameter_defaults - Provider threaded into write_input, _writer_leaf_sets, _writer_mid_sets, _writer_calc_params - flextoolrunner plumbs Provider through solve_writers.write_all_branches, averaged-timeseries, and timeline_config.py disk read

In-memory region decomposition (item 2.6) - flextoolrunner region decomposition is now fully in-memory across all entry points

Tests migrated to native cascade - Migrated: test_commodity_ladder_smoke, test_commodity_ladder_rolling, test_years_represented, test_timings_csv, TestPGLibCase14Integration, test_cost_aggregation_semantics, test_non_anticipativity, test_lh2_three_region, test_obj_decomposition - Retired: test_mps_parity.py deleted (legacy MPS parity dead); tests/emission/* skipped (flextool.mps emission gone in Δ.22 — ported to polar_high in 3.38.0); perturbation/* skipped (harness incompatible — ported in 3.38.0); test_solve_handoff.py trimmed (9 retired PoC tests dropped) - test_integration_parquet_matches_csv_on_base_scenario skipped (legacy parquet/CSV parity is no longer meaningful)

Miscellaneous - Commodity ladder reads routed through the Provider - apply_npv signature mismatch fixed; provider threaded - engine_polars/scaling: geometric-midpoint user_bound_scale + 6-decade gate - GUI: Debug checkbox passes --debug --csv-dump to scenario runs - Dropped redundant _native_leaf_set_override in _drive_cascade - engine_polars: ported co2_max_total + fixed minimum_downtime invest tightening - read_parameters: densified reserve_upDown_group_reservation to LP domain - process_outputs / read_variables: stream parts via ParquetWriter row-groups

Release 3.36.0 (15.5.2026) — Output-writer hardening + LP determinism + Surface A audit

Hardening release. The engine_polars cascade is pinned to a deterministic HiGHS solve and the LP column/row ordering is now canonical, so re-runs and goldens are stable across PYTHONHASHSEED and across CI workers. Many process_outputs regressions surfaced during the polars-cascade adoption are fixed; golden CSVs that drifted on the glpsol→HiGHS transition are regenerated with REGEN_LOG documentation. Surface A loader audit lands 50+24+13 focused tests.

LP determinism - engine_polars/scaling: pin HiGHS to deterministic simplex options - Canonical LP column/row ordering via add_var / add_cstr wrappers - Post-unique() set frames sorted before LP id assignment - engine_polars keys _all_steps by per-roll solve name, not parent solve - tests/db_utils: numeric columns cast to float in round_for_comparison - assert_frame_equal gains atol=1e-4 to absorb the rounding step - scaling._scale_cache cleared between scenarios in tests - Two scenarios' time_budget bumped to absorb the determinism overhead (incl. wind_battery_invest_lifetime_renew_4solve 4.0 → 6.0 s)

Output writers (process_outputs) - costs_discounted: multi-roll inflation/lifetime values now correct - dtt MultiIndex sorted by (solve, period, time, t_previous) - Per-period years_represented threaded through FlexData - d_realize_invest filtered to realized periods only; divest periods included - summary_solve.csv: solve names natural-sorted; Investment discount factor written over period_in_use - VRE_potential + group_flows CSVs restored after a regression - process__ct_method built from process_min_load_eff - dt_realize_dispatch / d_realized_period use realized_dispatch - unit_capacity__d.csv lag in rolling solves fixed - (entity, period) lookups filtered to v.invest's actual index - par / s unioned over full_dt for rolling/multi-solve - Self-discharge multiply fixed on rolling/multi-solve scenarios - unit__{input,output}Node wide-pivot columns sorted - Both flow directions emitted for direct-method arcs - commodity_node sets derived from FlexData flow_* frames - nodeGroupDispatch arc-union sets backfilled for group_flows - Sub-femto HiGHS LP residuals clipped in out_flows before CSV emit - Native engine + process_outputs fix for nested invest+dispatch cascades - Rolling-storage hand-off fix for non-forward-only storages - Forecast-branch rows emitted when output_horizon=yes - Reserve None-handling hardened; process_reserve column emission gated on scheduled reservation - calc_group_flows: .squeeze() dropped — was crashing single-timestep solves - engine_polars: detects period column under user-renamed Map.index_name

Goldens regenerated (HiGHS-clean vs glpsol residuals) - 6 scenarios routed from retired glpsol to HiGHS - coal_co2_limit goldens regenerated against alt-optima - cost-penalty goldens regenerated (HiGHS-clean zeros vs glpsol residuals) - multi_year_wind_no_investment unit_capacity golden regenerated - 5 alt-optima goldens regenerated against canonical LP column ordering - test_a_lot + 5weeks_battery_intraperiod_blocks alt-optima goldens regenerated - 3 + 4 summary_solve.csv goldens regenerated (HiGHS precision + chronological Investment-factor order) - hyphenated_entity_names regen documented - tests/REGEN_LOG documents Phase-2 candidate verification (1 regen + 5 flags) - Test infrastructure handles two-row-header / Unnamed: CSV forms

Surface A audit fixtures + tests - Base fixtures for loader / constraint / objective audit - 50 focused loader tests for Surface A audit - 24 model-run constraint tests for B.1–B.15 - 13 cost-term isolation tests for B.16–B.20 - step_duration=3 derived-fixture regression guards - Hand-computed step_duration LP objective tests - engine_polars/scaling: two-sided guard on user_bound_scale recommendation - engine_polars: cascade-vs-seed parity diagnostic

plot_outputs performance - _plot_simple_bars vectorised; FixedLocator / FixedFormatter for bar tick axes - _sum_row_heights vectorised - Stacked & grouped bar paths vectorised - Sub-pixel-wide bars skipped in draw path - Single-level expand groups counted in pagination - Stale on-disk plans invalidated via schema_version - Phantom NaN rows/cols dropped after dim-rule pivot

Other - native_run_model: write timeline_matching_map.csv from returned dict - GUI: stable input-source numbering with orphan reuse + visual marker; source-number suffix always written in output folder names - GUI: ResultViewer style overrides scoped to its own plot tree - GUI: status / peak / timestamp columns widened in jobs tree - tests/scenarios wired to engine_polars cascade

Release 3.35.0 (13.5.2026) — Cascade memory + GUI font system + plot performance

Performance, GUI and developer-experience release on top of 3.34.0. Long cascade runs no longer balloon RSS across rolls; the GUI handles per-monitor DPI scaling and user-tunable font sizes; dump_csvs no longer writes 7 GB debug oracles by default. Includes preparatory work for the engine-wide dtype refactor (axis-vocabulary discovery + canonical pl.Enums) — the end-of-load Enum sweep was activated and then parked in the same release as a precaution.

Cascade memory - Scaling diagnostic gated to once per base solve — 38 % wall-clock speedup on large cascades - Opt-in tracemalloc + RSS checkpoints to localise OOM phases (env-var gated) - libc.malloc_trim called after heavy allocators — 38 % RSS drop - Heap trimmed at end of each iter too (multi-roll heap accumulation) - dump_csvs: skip 7 GB-scale CSVs by default; env var restores the debug oracle

dtype refactor — preparatory - Phase 1+2: axis-vocabulary discovery + canonical pl.Enums - Phase 3+4+5 (partial): load_flextool end-of-load pl.Enum sweep — activated, then disabled in the same release as a guard while downstream fallout was understood - Dtype-flexible scratch frames in engine_polars (preparatory for Enum activation) - cast_dim helper + populated-branch alignment; cast_dim sweep across cascade (3 files + remainder) - process_outputs: normalise enum/utf8 join keys in _entity_all_capacity

Path B (rolling-handoff perf) - Rolling-handoff CSV reads deduplicated (Path B, Category A) - WriterSnapshot for top-7 per-solve preprocessing (Path B, Cat B) - ctx threaded through period/branch helpers + _dt_period_active_steps + dtttdt

GUI font + DPI system - Phase 1: centralised font metrics in ui_metrics.py - Phase 2: per-role named fonts via setup_fonts - Phase 3: saved geometry + sash clamped on restore - Phase 4: rescale saved positions by font metric on restore - Phase 5: Reset window layout action - Phase 6: em-ified dialog widths - Phase 7 + 7c: user-tunable UI font size + UX size tweaks - Phase 8: per-monitor DPI font rescale - Header fonts bound to named fonts so live size changes propagate

Other - engine_polars: respect solve.new_stepduration in apply_derived_a - engine_polars: preserve legacy entity-level rows in dump_csvs slices - engine_polars: native p_inflow override in apply_derived_a disabled (regression guard) - engine_polars: end-of-load FlexData → Enum cast disabled (preparatory work parked) - tests: pbt_node_inflow multi-time_start + parent-period fixtures (Phase C)

Release 3.34.0 (12.5.2026) — Multi-solver support + documentation overhaul

This release ships the two user-facing additions on top of 3.33.0's engine swap: per-solve commercial-solver selection and a documentation refresh organised around the Spine Toolbox GUI as the primary FlexTool interface.

Multi-solver support - New per-solve parameters (v52 migration) on the solve entity: solver, solver_io_api, solver_options, solver_time_limit, solver_mip_gap, solver_threads, solver_log_level. HiGHS remains the default — existing scenarios with no solver parameters are unaffected. - Backends supported via polar-high: HiGHS (default, ships with FlexTool), Gurobi, CPLEX, Xpress, COPT. Each commercial solver requires its own Python wrapper + license; FlexTool itself never imports the commercial wrappers and never inspects licenses (vendor's discovery path handles it). - The three convenience knobs (solver_time_limit, solver_mip_gap, solver_threads) are normalised across solvers; FlexTool translates them to each backend's native parameter name. solver_options passes raw key/value pairs through unchanged. - Cascade startup probes each available solver with a trivial 1-var LP and logs a per-solver license status line, e.g. Solver license status: gurobi=licensed, cplex=licensed, xpress=licensed, copt=not-installed, highs=licensed. - Cold rebuilds in the cascade dispatch through polar_high.solvers.solve(...) and normalise the result via a new LiteSolution adapter so downstream output writers stay unchanged. Warm-start and Lagrangian decomposition remain HiGHS-only by polar-high design — selecting another solver on a warm cascade logs a warning and falls back to cold rebuilds; selecting another solver on a Lagrangian-decomposed scenario raises a clear configuration error. - Per-solver documentation pages under docs/solvers/ (install, licensing, common errors, how to set the solver in FlexTool).

Documentation overhaul - Site reorganised around the Spine Toolbox GUI as the primary FlexTool interface; CLI flows are documented but no longer the default story. - New developer guide and how-to recipes for the engine_polars-era codebase. - decomposition_method parameter description tightened; superseded GUI screenshot removed.

Bug fixes - _writer_mid_sets.derive_commodity_node_co2 no longer crashes with ComputeError: cannot compare string with numeric type (f64) on customer DBs where p_commodity.csv's value column is inferred as Float64 by polars. _read_csv now forces every column to Utf8 on read via infer_schema_length=0. - process_outputs.calc_storage_vre no longer raises KeyError when node_self_discharge_loss is authored on nodes that have no v_state LP variable (supply-curve / commodity nodes). The self-discharge multiply is now restricted to the intersection of authored nodes and storage nodes. - CLI cascade exception handler distinguishes FlexToolUserError (configuration problem — clean message, exit 1) from other exceptions (real flextool bug — full traceback). Users hitting unknown-solver / missing-license errors no longer see a stack trace. - update_flextool now refreshes declared dependencies for editable installs too — pip install -e . with new core deps in pyproject.toml would previously be missed, causing ModuleNotFoundError on the next solver invocation. The --upgrade is now passed without --upgrade-strategy=eager so transitive dependencies aren't churned.

Performance - Spine DB read: SolveConfig.load_from_db_url and TimelineConfig.load_from_db_url now pre-warm with db.fetch_all("entity") + db.fetch_all("parameter_value") before any find_* call, mirroring the legacy FlexToolRunner.__init__ pattern. Measured 5.5–6.4× speedup on large customer DBs (~1.8 s saved per cascade run); sub-MB test DBs see no change.

Other - v52 schema migration renames the legacy solver value list to solvers in place: members expanded from [glpsol, highs, cplex] to [highs, gurobi, cplex, xpress, copt]; pre-existing solve.solver = "glpsol" values are rewritten to "highs" (GLPK retired in Δ.22). Parameter_definition foreign key is preserved through the rename so no data is lost.

Release 3.33.0 (12.5.2026) — glpsol retired; HiGHS via polar-high becomes the sole LP backend

This release closes out the largest architectural change since FlexTool went open source: the GLPK/GMPL pipeline that built the LP via glpsol, wrote MPS files to disk, and re-loaded them into HiGHS is gone. The new path builds the LP in process via polar-high on top of the polars DataFrame engine, hands it to HiGHS through highspy, and stays in memory for the entire solve → handoff → output-writer chain.

There are no binary build artefacts left to ship: glpsol is gone, the bin/glpsol* binaries that the previous releases bundled have been deleted, and the flextool.mod GMPL model file is deleted. FlexTool is now a pure-Python install — pip install flextool is sufficient.

The version is held at 3.x rather than bumped to 4.0.0 until the new engine reaches full parity with the old one across the test suite. A 4.0.0 semantic bump will follow once that bar is met; this release ships the backend swap behind the existing 3.x API surface.

Breaking changes - bin/glpsol* removed; flextool/flextool.mod and flextool/flextool_base.dat removed - --engine=gmpl hard-rejected with a clear retirement banner - Legacy GMPL-pipeline CLI flags removed: --use-old-raw-csv, --ipm, --auto-scale, --relax-feasibility, --glpsol-timing, --report-near-duplicates - The stubbed flextool/flextoolrunner/lagrangian.py (legacy GMPL coordinator) is deleted; --decomposition lagrangian now drives the native polars coordinator (see below) - --highs-threads accepted as a no-op stub for GUI/Toolbox compat; the native cascade is single-threaded

New solver backend (engine_polars) - Whole-system FlexData is built directly from a Spine DB or pre-staged workdir CSVs; the LP build, solve, and output handoff all happen in memory through engine_polars.run_chain_from_db - HiGHS is consulted via polar-high.Problem / WarmProblem / LagrangianProblem; no MPS roundtrip - New experimental --fast-single-solve CLI path for simple single-solve workloads — bypasses write_input entirely and reads inputs from Spine via SpineDbReader - Native engine_polars/_writer_* ports of 150+ preprocessing sets and calculated parameters (Writer Phase 1-4: L0-L9, follow-ups 1-8, closeout, Phase 2 sub-dispatches 1-8, Phase 3 cascade adoption, Phase 4 Gap F handoff) - Automatic LP scaling — analyser, scaling-report, two-sided cost-band guard, geometric-centering fallback, objective and bound scaling - Δ.31: in-memory FlexData + solution threaded into process_outputs.write_outputs so the output writers no longer round-trip through solve_data/*.csv

Spatial Lagrangian decomposition — native rewire - --decomposition lagrangian CLI rewired onto engine_polars._lagrangian.solve_lagrangian: per-region builds via LagrangianProblem, damped subgradient outer loop, primal averaging on the cross-region pipeline flows - Smoke-test coverage on the LH2 three-region fixture pins the CLI contract within a 2 % gap-to-monolithic tolerance

Flex-temporal decomposition (carried over from new-outputs 3.29.0, fully integrated) - Per-entity temporal blocks make mixed-resolution dispatch possible (hourly power + daily hydrogen in the same solve) - v50 + v51 migrations land solve.new_stepduration and group-level new_stepduration + decomposition_method - Constraints made block-aware: storage, conversion, flow capacity, DC flow, UC, ramp, profile, reserve; node balance generalised via overlap-set aggregation - Output writers expand coarse-block variables back onto the fine timeline for the user

Test infrastructure - Layer-1/2/3 test pyramid scaffolded, with golden objectives pinned on 10 scenarios and per-scenario timing budgets - @pytest.mark.solver and @pytest.mark.smoke markers - GitHub Actions CI on the smoke tier - LH2 three-region JSON fixture + per-fixture native parity tests - MPS-parity harness retired alongside the GMPL pipeline

Packaging / installation - All binary dependencies retired — FlexTool is now a pure-Python install - polar-high is now a core dependency (was an optional engine-polars extra) - pip install flextool is sufficient on Linux, macOS, and Windows (HiGHS arrives via the highspy wheel) - PyPI release in preparation (see specs/pypi_release_checklist.md for the remaining steps)

Release 3.32.0 (5.5.2026)

Bug fixes - Excel template link-sheet retention: drop link sheets where the link's own class has no surviving data (e.g. connection_node when constraint is unselected) - group.{include_stochastics, new_stepduration} retagged from model/timeline to solve_advanced - timeset.{timeset_weights, representative_period_weights} retagged to solve_advanced - Quadratic linear scan in blocks::derive_blocks::_assign_entity_to_group replaced with a precomputed (entity → group) lookup

New features - --groups CLI flag on cmd_export_to_tabular to restrict the Excel template to a parameter-group set; sheets with no surviving columns are dropped - GUI parameter-group picker on "Add empty FlexTool input Excel" — DB-driven checkbox tree, required groups (timeline, model, solve_basics, basics) highlighted with hover tooltip; theme-aware colours - Default-value row in v2 Excel sheets where any parameter on the sheet has a default (Map/Array defaults stringified) - Representative-period clustering input drops scalar-valued profiles/inflows

Note — new-outputs also landed four hot-path optimisations on the legacy flextool/flextoolrunner/preprocessing/* writers (is_static fast-path on PdtLookup, sparse-emit + default 0 on the six dense pdt* CSVs, pandas.read_csv swap in _read_pdt_at_param, sparse iteration in pssdt_varCost_*, the O(N²×T²) → O(N + |pairs|) fix on node_inflow_scaling_params.has_node_time_inflow, and side-node indexing for process_arc_unions::write_node_group_dispatch_sets). Those edits target a code path that the engine_polars writer port (Writer Phase 1-4) is retiring, so they were not carried across in the merge — the performance lessons are preserved as actionable items for the native polars writers in specs/sparse_writer_lessons_for_engine_polars.md.

Release 3.31.0 (4.5.2026)

Bug fixes - Stochastic per-branch profile values previously lost in pdtProfile are restored - Plot dataframe fragmentation in dispatch column-alignment fixed - Dispatch view reads group flags from nodeGroupDispatch (not nodeGroupIndicators) - solve.timeline_hole_multiplier now declared in the v44 schema (default 1.0) - db_migration v42: robust legacy value-list cleanup via find_parameter_value_lists - Migration steps tolerate NothingToCommit so a second run doesn't abort the chain - RP clustering input: scalar-valued profiles/inflows dropped

New features - mod → Python preprocessing migration: ~150+ sets and calculated parameters previously declared in flextool.mod now live under flextool/flextoolrunner/preprocessing/, written per-solve to solve_data/ - Migration scaffolding: phase-0 inventory + MPS parity harness + lint rules + DAG of derivations - 70 incremental migration batches with MPS parity verified at 7-sig-fig precision across multiple baselines (rolling, multi-solve, contains_solve, h2_trade, test_a_lot) - Per-class param taxonomy in _param_taxonomy.py (PROCESS_TIME_PARAM, NODE_PERIOD_PARAM, etc.) - Order-determinism lint rule (tests/test_preprocessing_ordered_set_lint.py) — bare set() / set-literals / set-comprehensions blocked in preprocessing/ - templates/examples.sqlite rebuilt from tests.json and migrated to v51

Release 3.30.0 (28.4.2026)

Bug fixes - read_highs_solution: strip GLPSOL quotes from ISO 8601 timestamps in parsed variable names - handoff_writers: load resolved-set CSVs from solve_data/ (Phase A follow-up) - plot_canvas: cap rendered DPI to keep figure pixmaps under X11's limit - RP: representative-period indices sorted into chronological order - plot_bars: bar thickness constant per row regardless of bar count - GUI memory-watchdog: kill only when BOTH free-RAM reserve AND swap allowance are breached - GUI: refuse to open the same input source twice (flash row red)

New features - result_viewer Phases B–E: single rebuild path with filter-only tree toggles, per-scenario availability union, lazy plan-parquet union, comparison-only plans - GUI per-job memory budget + watchdog + admission control + status bar - GUI: drag-to-reorder + Alt+Up/Down on result-viewer scenarios tree - GUI: shared CheckTreeController for every check tree + generation tokens for comparison-data updates - GUI: View button always shown; opens single mode focused on the scenario - GUI: persist per-variant duration with template default + clamp-safe save - GUI: robust delete + prune dangling scenario state from settings.yaml - .mod-resolved set files moved from input/ to solve_data/; set_ prefix dropped - --glpsol-timing diagnostic flag for per-constraint matrix-gen timing - nodeBalance_eq and conversion_indirect use tuple binding instead of overlap equality-filters

Release 3.29.0 (24.4.2026)

Bug fixes - v50 item-lookup bug fixed while restoring unidirectional in the master template - export_to_tabular whitelists synced with master template schema - Lagrangian: defensive name-map consistency check per iteration

New features - Flex-temporal decomposition: per-entity temporal blocks for mixed-resolution dispatch (e.g. hourly power + daily hydrogen in the same solve) - v50 migration: new_stepduration moved from timeset to solve - v51 migration: group-level new_stepduration + decomposition_method - blocks.py: derive per-entity blocks + overlap set + block predecessors + boundaries - Generalised node balance via overlap-set (M-matrix) aggregation; constraints made block-aware (storage, conversion, flow capacity, DC flow, UC, ramp, profile, reserve) - Output writers expand coarse-block variables back to the fine timeline for the user - Spatial Lagrangian decomposition: regional input filter (cross-region flows as half-flows), lagrangian.py scaffolding + subgradient loop with primal recovery, CLI flag + docs - LH2 three-region fixture + golden integration test (Lagrangian vs monolithic) - HighsModelHandle persistence helper for repeated subgradient solves

Release 3.28.0 (24.4.2026)

Bug fixes - Default HiGHS to serial simplex (parallel=off, threads=1) to avoid non-determinism and occasional stalls on small models; defaults reinforced in solver_runner as belt-and-suspenders - Restore 0.05 discount_rate default and related advisory defaults that had drifted - Revert slack primary+escape split — back to single-variable slacks with penalty acting as a valve

New features - HiGHS upgraded to 1.14 (highspy>=1.14) - Auto-scaling of LP/MIP numerics: row scaling via node_cap/group_cap, objective scaling, and bound scaling with diagnostics printed - --auto-scale CLI flag that gates objective/state auto-apply behaviour - --highs-threads CLI flag - HiGHS mip_detect_symmetry enabled — ~15× speed-up on unit-commitment with identical units - virtual_unitsize documented as a speed lever for UC with identical units - unidirectional transfer method for connections - Cost-aggregation semantics fixed — weighting factors applied per variable class - Divest salvage included in objective; pre_existing renamed to all_existing; storage valuation documented - Auto-seed missing output_info, output_settings and comparison_settings databases on first run - Seed bin/highs.opt from tracked template on first run - scaling_benchmark harness relocated to benchmarks/scaling; unit tests moved to tests/ - min_load_efficiency section-term test promoted from xfail to passing

Release 3.27.0 (22.4.2026)

Bug fixes - Annualize nodeGroup_flows_d_gpe to match the node_d_ep convention - Column misalignment in ed_lifetime_fixed_cost[_divest].csv writers - Harden axis-manifest override against None active_scenarios - Sum only over stack_levels in shared axis-bound resolution - Reserve provision with a timeseries reserve requirement - Reserve provision active only when there is demand - Delete dead plot functions

New features - Parameter groups — every parameter now assigned to a group; "Outputs" renamed to "output" - Parameter-group colours re-tuned for readability in both light and dark theme - Colour template infrastructure for plots: category-based colouring (costs, node_flows) and entity_class.group colouring (flowGroup) - Cross-scenario axis-bounds manifest — viewer reads shared bounds so y-axes stay stable across scenarios - Per-scenario axis manifest with subset filter - Composite colour lookup for nodeGroup_flows plots - Colour template plumbed into the batch render path - Plot plan JSON cleanup (redundant timestep fields dropped) - Group-output parameters renamed; output_results split; flow-group indicator stub added - Alphabetical sorting of subplots - Periods ordered left-to-right in vertical p-variant bar plots

Release 3.26.0 (21.4.2026)

Bug fixes - Negative-remaining infeasibility in cumulative price-ladder cap - Bug in realized_invest uniqueness - Parameter_type_list corrections across migrations and master template

New features - Commodity price ladder — price_method, unitsize, price_ladder parameters for commodities - v_trade variable (MWh × unitsize) with tier caps and objective routing - Rolling cumulative-quota handoff for price_ladder_cumulative — per-period accumulators preserve remaining quota across rolls - Two-roll cumulative-ladder validation scenario - Rolling-aware co2_max_total with per-period accumulators - Node-type consolidation aligned with the ladder work

Release 3.25.0 (20.4.2026)

Bug fixes - Align fix_storage_*.csv header between init and phase 3 - Carry fractional tail in write_years_represented (tested at R=2.5) - canonical_sort sorts stably by solve_pos only, preserving within-solve order - Drop _round_to_sig from the parquet read path - Reindex extract_variable output to canonical phase-1 row order

New features - Direct HiGHS → parquet extraction replaces the phase-3 glpsol reader round-trip for solver outputs and solve handoffs - Phase-3 glpsol retirement — derived-parameter printfs moved pre-solve; readers repointed at input/ and solve_data/ - Canonicalise row order by solve-creation order across all readers - has_balance, has_storage and node_type consolidated into a single node_type enum - Parquet VariableSpec column names aligned with CSV readers - empty_variable_frame helper for same-shape empties

Release 3.24.0 (19.4.2026)

Bug fixes - storage_state_start first-timestep semantics corrected for cyclic bindings - Goldens regenerated for the sink-flow coefficient flip

New features - coefficient on capacity constraints split into flow_coefficient, max_capacity_coefficient, min_capacity_coefficient - Sink-side flow_coefficient flipped from division to multiplication — both sides of the balance now use the same convention - New pre_built capacity-coefficient term alongside the invested term - Growth-cap recipe documented - Lifetime-aware dispatch capacity bound - max_flow_for_unconstrained_variables model parameter replaces the previous 1e6 literal - CO2 duals exposed; rp_cost_weight threaded into constraints; horizon-vs-annual outputs distinguished - timeset_weights wired through the runner (populates rp_cost_weight.csv) - no_investment handled as a lifetime_method

Release 3.23.0 (17.4.2026)

Bug fixes - Pre-processing now also updates the solve period_timeset - Step duration correctly applied in period aggregations of flow-derived outputs - Reserve Excel handling fixed

New features - Greedy convex-hull clustering for representative-period selection - Representative-period storage binding via bind_using_blended_weights (renamed from bind_using_rp_weights) - bind_intraperiod_blocks storage binding method for LTS-style intra-period blocks (blocks defined by gaps in timeline indices) - Multi-year wind no_investment scenario test and goldens - Base-weighted scenario test and goldens - Delays for units and connections (constant or map) - 5weeks_battery_intraperiod_blocks scenario test and goldens

Release 3.22.0 (7.4.2026)

Bug fixes - Parquet loading strips the scenario column level - savefig works for Figure() objects not registered with pyplot - Threading errors from plt.figure() replaced with Figure() usage - Dispatch colors and nodeGroup filtering - Cancel pending draw_idle to prevent redundant re-renders - Numerous canvas / toolbar jitter fixes (freeze/thaw around draws, layout locking) - Legend clipping, tree navigation, canvas clearing - Stacked-area plot axis limits - Empty years_represented for rolling dispatch solves - CO2 emissions parquet key mismatch - Scenarios without connections

New features - PlotPlan — pre-computed plot plans saved to disk - Threaded figure building with prefetch cache and memory-based GB cache limit - Time-series downsampling via tsdownsample with a numpy fallback - Network graph visualisation from the Spine database - Cross-scenario dispatch y-limits and column consistency - Comparison-mode parquet pipeline - Availability manifest for three-level variant display - Redesigned three-level variant navigation (Shift+Up/Down to next row with data at focus column) - PlotPlan unit tests - YAML config restructured with entry names as top-level keys - Per-variant plot_name and explicit variant field in configs - Dispatch mode in Result Viewer - combine_scenario_parquets and comparison checkboxes in the viewer - Parquet file-size reduction by dropping DataFrame multi-index bloat

Release 3.21.0 (1.4.2026)

Bug fixes - Handle scenarios without connections - Connection output for method_2way_1var (DC power flow) - Guard against all-NaN values in plotting - Reserve provision active only when there is demand for the reserve - Mac glpsol binary naming and selection across platforms - Fixed and cleaned-up test suite

New features - DC power flow B-θ formulation in the GMPL model - Method resolution moved from GMPL to Python - DC power flow data pipeline and database migration - Output processing and parquet passthrough for DC power flow - MATPOWER import CLI wrapper - PGLib-OPF IEEE 14-bus integration test - 24-test DC power flow pytest suite - glpsol built for macOS, Linux and Windows via GitHub Actions (including macos-15 arm64)

Release 3.20.0 (29.3.2026)

Bug fixes - Excel migration edge cases - Direct execution from Excel (.xlsx) corrected - GUI updated to handle different input Excel versions

New features - FlexTool 2.0 importer — reads old FlexTool 2.0 Excel .xlsm files into a Spine database - Import of sensitivity scenarios from FlexTool 2.0 - CLI entry point for FlexTool 2.0 sensitivity import - Version-aware importer that handles older FlexTool 3.0 Excels as well - invest_method support in the FT2 importer - _inflow renamed to _node; _storage used where applicable - base used instead of Base as the default alternative - Stochastic sheets in Excel - Improved Excel read/write roundtrip, including DB-to-Excel export - Dialog to migrate Excel files - Single source of truth for the FlexTool input DB version (flextool/update_flextool/__init__.py)

Release 3.19.0 (14.3.2026)

Bug fixes - Execution runs from the FlexTool root with isolated work folders - Comparison Excel written to the correct directory - All scenarios no longer produce identical results - Output-actions column order, drag-select and spinbox sync - Foreground-color crash on start - Several threading and layout issues

New features - Result Viewer window with scenario listbox, plot tree and variant panel - PlotCache with memory-based GB limit - PlotCanvas with PNG display wired into ResultViewer - Dark mode with sv_ttk and a theme-toggle radio - Font-relative and DPI-aware GUI sizing - Plot-settings dialog with YAML config parsing - Execution menu window with subprocess pool and parallel workers - Database version checking and auto-upgrade from the GUI - Custom file picker with last-modified column and sorting - Visual feedback: green/grey/red highlights, boxed outputs, auto-check - Keyboard shortcuts (including Ctrl-A to select all) and rearranged move arrows - DB-editor integration with process tracking - --work-folder support for parallel scenario execution - --parquet-base-dir for comparison without a database - output_db_url optional in cmd_run_flextool - Dual variant state in the viewer: desired (dashed) + shown (solid) - Animated hourglass spinner on output-action buttons - Persisted checkboxes, View button, sizing adjustments - Parent-directory button in the file-picker dialog

Release 3.18.0 (27.2.2026)

Bug fixes - Fix reading input files in run_flextool.py - Many small issues surfaced by the refactor fixed so all examples execute

New features - Major refactoring into deep modules — package structure reorganised for separation of concerns - New subdirectories to keep the FlexTool root uncluttered - Initialisation now also creates settings databases - Updated update_flextool.py - Walked back global flextool CLI commands — they do not work with multiple installs - Added output_info_template.sqlite - flextool_location.txt for runner discovery

Release 3.17.0 (14.1.2026)

Bug fixes - Float-to-string handling in the import spec - Node-summary column alignment - nodeBalancePeriod uses pdtNodeInflow (sum over period) instead of annual_flow (annual_flow should not carry a sign) - Rolling-model fixes including p_roll_continue handling

New features - Excel import specification - Direct Excel-to-DB read pipeline - Calamine engine for faster spreadsheet reading - Full FlexTool execution from Python — no Toolbox required - Removed data checks from later solves (they belong in the first solve)

Release 3.16.0 (5.12.2025)

Bug fixes - Fixed costs split off from investment costs and their calculation corrected - years_represented with a discount factor >1 now calculated correctly - Timestep durations checked to be positive and non-zero

New features - Speed-up in Python output writing - Result list refactored - Old write_outputs.mod glpsol output code removed from flextool.mod - Further plotting enhancements and minor refactoring - solve_progress.csv with timings and scenario name - Vertical plots, sums and means, better subplot spacing - write_outputs can be used to plot a single ad-hoc time series

Release 3.15.0 (2.12.2025)

Bug fixes - costs_discounted.csv calculation for multi-year models - Empty sets and rolling-model crashes - Negative capacity units handling in outputs - Inertia calculation in outputs - Missing fix_storage_time_lists - Several output-processing bugs (ramp, investment, storage, group results)

New features - Python post-processing pipeline reaches parity with the old glpsol write_outputs.mod — all result CSVs replicated except node ramp envelopes - Scenario-comparison specification (scenario_comparison.json) and combined-scenario results framework - Removed redundant fix_storage constraint that used dt_realized_dispatch - First ugly-duckling plotting on top of the Python pipeline

Release 3.14.0 (6.8.2025)

Bug fixes - Fixing problems with group results (wrong signs) - Fix broken importer - Fix calculation of VRE shares in the results - Other operational costs for units had gone missing in 3.12.0 (22.5.). Now they are back.

New features - There is a new migrate button in the workflow to update the active input database to the latest version.

Release 3.13.0 (27.6.2025)

Bug fixes - Added missing fixed costs to operational models - Catching infeasible models better

New features - Time structure has been simplified, see ´https://irena-flextool.github.io/flextool/reference/#how-to-define-the-temporal-properties-of-the-model' - Changed the time structure to have timeset instead of timeblockSet - Added periods_available for the model entity, to have periods in the data that are not used without domain errors - Added assumptions about the model timestructure to ease the defining of it.

Release 3.12.0 (22.5.2025)

Bug fixes - Decimal points in node_balance_t were causing some grievance. - Issues with lower case / upper case filenames when importing results - Added missing parameters inertia_constant, ramp_cost, ramp_speed back to the model (caused by release 3.10.0) - Fixed a output processing crash in a rolling model without VRE

New features - Delays for units and one-way connections. New parameter delay, measured in time-steps (be careful), which can be a constant or a map of (integer timesteps, weight) where the weights should add to one. The map allows to spread the delay like it might happen in river systems. - An order of magnitude speed-up in reading inputs

Release 3.11.0 (4.3.2025)

Bug fixes - Preventing crashes on curtailment calucaltions to units without capacity and MIP stochastic solving - Correcting nested storage state passing

New features - Timeseries prices for commodity and co2

Release 3.10.0 (31.1.2025)

Bug fixes

• Prevent HiGHS from hanging after founding a solution
• Fix profile limits (issue when efficiency != 1)
• Made to work with new Toolbox DB API

New features

• New HiGHS versio (1.9.0)
• Script to see the differences in outputs (or inputs) between two model runs (meant for testing)
• FlexToolRunner.py reads input data directly from DB (speed-up)

Release 3.9.0 (28.11.2024)

Bug fixes

• Fixing Excel input template

New features

• Updated documentation to match Toolbox changes
• New HiGHS version (1.8.1), should help with model getting stuck.
• Reorganising file structure to remove clutter from FlexTool root
• Some name changes in the workflow
• FlexTool has requirements.txt and pyproject.toml (allows pip install -e ., but not in pypi)
• Improved installation instructions (switched from miniconda to venv)
• Linux (Ubuntu tested) should work out of the box (but FlexTool execution only in work directory one scenario at a time!)

Release 3.8.0 (29.8.2024)

Bug fixes

• Excel input file to use entity_alternative
• unit_curtailment_share fix to Excel input

New features

• Existing capacity can be set for periods
• Cumulative limits for investments in nodes, units and connections
• New node_type that has period limit for inflow
• New highs solver version
• All parameter now have valid types. These are not enforced, but Spine Toolbox highlights parameters with a wrong type.

Release 3.7.0 (24.6.2024)

Bug fixes

• Several bugs related to the migration to Spine Toolbox 0.8.
• CPLEX call without pre_command fixed
• Timeline aggregarion without all timesteps
• User constraint including capacity fixed to include unitsize
• User constraint including flows fixed to include step_duration
• Creating multiple invests with 2D maps fixed
• Update_flextool.py/ migrate_database.py crashing if nothing to commit
• Nested structure:
• Pass end state only from dispatch level
• Storage fix_quantity overrides other storage options

New features

• Added non-anticipatory constraint to stochastic branches. The brnaches now start from the first timestep.
• Storage_reference_value: Now possible to set seperately for multiple end times as time step map.
• Nested structure storage_nested_fix_method:
• Added fix_usage
• Fix_price changed to set the price to the objective function of the lower level solve

Release 3.6.0 (15.5.2024)

New features

• Upgraded FlexTool to Spine Toolbox v0.8
• Entity alternative replaces is_active parameter

Release 3.5.0 (29.4.2024)

Bug fixes

• Stochastic and rolling weights

Release 3.4.0 (13.4.2024)

Bug fixes

• Inertia with MIP units added missing unitsize
• Result ordering improvements
• Fixing how flow variables are limited
• Min_load to work with multiple inputs/outputs¨
• Capacity limit includes the efficiency

New features

• Output of inertia seperately for units
• Default penalty values for reserve, inertia, capacity margin and non-synchronous limits
• Large model data processing speed-ups

Release 3.3.0 (23.3.2024)

Bug fixes

• Investments: maxState to only sum over the investments in this node, not all the nodes
• Ramp constraint now correctly calculates the investments done in previous periods
• Min and Max Invest/Divest constraints to only affect investment periods
• Efficiency works with profile units
• Allow upward and downward slacks to be more than the inflow
• Potential VRE generation now includes efficiency and availability
• Non-sync group constraint now excludes the flows inside the group
• Two-way MIP connection to allow only one direction at a time
• Connection variable cost to the objective function
• Allow investing with just invest_periods without realized_periods
• unit_inputNode: coefficient applied correctly to units with multiple inputs

New features

• Two-stage stochastic modelling of future uncertainty
• Better infeasibility and required parameter checks to inform the user what is wrong with the model
• Small changes to some result data parameters
• Added availability (constant /timeseries) parameter for entities
• Several outputs to be optional to hasten the data transfer.
• Added group flow output parameter output_aggregate_outputs to give the flow of the desired nodes in a grouped format
• Option for loss of load sharing share_loss_of_load between nodes in a group to better describe the system where loss of load is present
• A workflow item to display the summary file from model runs
• Database with pre-made time settings for the users to start with time_settings_only.sqlite
• C02 max total costraint
• Default values for:
  • Penalty values
  • Efficiency
  • Constraint constant
  • Reserve reliability

Documentation

• Introduction
• Improved tutorial (including how-tos split of into their own section)
• Installation / update instructions
• Theory slides
• How to section to guide on making specific parts of the model

Building parts of the model:

• How to create basic temporal structures for your model
• How to create a PV, wind or run-of-river hydro power plant
• How to connect nodes in the same energy network
• How to set the demand in a node
• How to add a storage unit (battery)
• How to make investments (storage/unit)
• How to create combined heat and power (CHP)
• How to create a hydro reservoir
• How to create a hydro pump storage
• How to add a reserve
• How to add a minimum load, start-up and ramp
• How to add CO2 emissions, costs and limits
• How to create a non-synchronous limit
• How to see the VRE curtailment and VRE share results for a node

Setting different solves:

• How to run solves in a sequence (investment + dispatch)
• How to create a multi-year model
• How to use stochastics (represent uncertainty)

General:

• How to use CPLEX as the solver
• How to create aggregate outputs
• How to enable/disable outputs
• How to make the Flextool run faster

Release 3.2.0 (23.8.2023)

Bug fixes - Storage state, unit flows and ramps were not properly limited in multi-period investment models. - Removed limits from vq_state (they were causing infeasibilities in some case). - Added time changing efficiency to profile based flows. - Flows from node to unit in unit__node are marked negative (were positive).

New features - Breaking change: realized_periods (solve objects) is supplemented with a realized_invest_periods so that the user can state from which solves the results should take investment results and from which solves dispatch results. - New parameters for solve objects: solve_mode, rolling_duration, rolling_solve_horizon, rolling_solve_jump and rolling_start_time that enable to build rolling window models. See How to use a rolling window for a dispatch model. - New parameter contains_solves that enables nesting solves inside solves (e.g. to calculate shadow values for long term storages or to implement rolling dispatch inside a multi-year investment model). See How to use Nested Rolling window solves (investments and long term storage). - New outputs: For groups: VRE_share_t. For unit__nodes (VRE units): curtailment_share, curtailment_share_t. - Name changes to outputs: flow to flow_annualized, sum_flow to sum_flow_annualized. - Add migration for results database (parameter descriptions can be migrated). - Documentation updates: new how-to sections. - Faster outputting of ramp results. - update_flextool.py will leave the Toolbox project untouched. - Disable execute project button from the FlexTool workflow.

Release 3.1.4 (14.6.2023)

Bug fixes - Cancelling of plot_results will not freeze Toolbox

New features - Lifetime_method (reinvest_automatic and reinvest_choice) so that user can choose whether assets will be automically renewed at the end of lifetime or the model has to make that choice - Commercial solver support (CPLEX) - Database migration. init.sqlite and input_data.sqlite will be updated to the latest version when update_flextool.py is run. It is also possible to update any database using migrate_database.py. After git pull, run python -m update_flextool.py. In future, just python -m update_flextool.py is sufficient. Best to update Spine Toolbox before doing this.

Release 3.1.3 (1.6.2023)

Bug fixes - Use of capacity margin caused an infeasibility - Certain inflow time series settings crashed the model (complained about ptNode[n, 'inflow', t]) - Investments did not consider lifetime correctly - units stayed in the system even after lifetime ended - Lifetime is now calculated using years_represented - Fixed how retirements work over multiple solves

New features - Added support for commercial solvers (CPLEX explicitly at this point)

Release 3.1.2 (23.5.2023)

Bug fixes - Node prices were million times smaller than they should have been (bug introduced when scaling the model in 3.1.0) - Non-synchronous limit was not working.

New features - Documentation structure was improved - Model assumes precedence between storage_start_end_method, storage_binding_method, and storage_solve_horizon_method (in that order)

Release 3.1.1 (13.5.2023)

Bug fixes - The calculation of the p_entity_max_capacity did not consider which investment method was actually active. It also limited capacity in situations where investment method was supposed to be 'invest_no_limit'.

Release 3.1.0 (13.5.2023)

Bug fixes - Division by zero capacity for units with cf - Output fixed costs for existing units - Fix output calculation for annualized fixed costs - Fix error in importing CO2_methods for groups - Add other_variable_cost for flows from unit to sink

New features - New investment method with a limited horizon (model used to assume infinite horizon) - Braking change: new parameter 'years_represented', which replaces 'discount_years'. It is a map to indicate how many years the period represents before the next period in the solve. Used for discounting. Can be below one (multiple periods in one year). Index: period, value: years. - Added sidebar to documentation (gh-pages branch) - Default solver changed from GLPSOL to HiGHS - Documentation improvements - Scaling of the model resulting in very big performance improvement for linear problems - Improved plotting for the web browser version - In Spine Toolbox, plotting is based on the specification from the browser version - More results: group flows over time - More results: costs for whole model run - More result plots

Release 3.0.1 (17.3.2023)

Bug fixes - CO2_method for groups was not imported correctly from Excel inputs - Fix other_variable_cost in units for flows from unit to sink (source to unit was working)

New features - Output fixed costs for existing units - Added outputs to show average capacity factors for periods

Release 3.0.0 (11.1.2023)

All planned features implemented

TEMPLATE

Release (dd.mm.yyyy)

Bug fixes - foo

New features - bar