FlexTool Architecture

Project Purpose

IRENA FlexTool is an energy and power systems optimization model. It reads input data from a Spine database, derives per-solve preprocessing tables in polars, builds a linear/MIP program in memory via polar_high, solves it with HiGHS in-process via highspy, and writes results to parquet (with optional CSV, Excel, and plot renderings on top of the parquets). Inputs can be authored directly in the Spine database, populated from tabular input files (Excel / ODS / CSV), or imported from old FlexTool v2 .xlsm files. A Tkinter desktop application and a Spine Toolbox workflow provide the two user-facing GUIs.

Top-level data flow

flowchart LR
  excel["Excel / ODS / CSV
+ old FlexTool .xlsm"] --> spdb
  manual["Spine DB editor"] --> spdb[("Spine input DB")]
  spdb --> backend["SpineDBBackend
(EAV → polars frames)"]
  backend --> derive["input_derivation
(per-solve preprocessing)"]
  derive --> provider["FlexDataProvider
(in-memory transport)"]
  provider --> build["engine_polars.build_flextool
(polar_high.Problem)"]
  build --> solve["HiGHS via highspy
(Problem / WarmProblem)"]
  solve --> handoff["SolveHandoff
(Provider keys)"]
  handoff -->|next roll / sub-solve| derive
  solve --> outputs["process_outputs
(read solution → calc_* → out_*)"]
  outputs --> parquet[("output_parquet//")]
  parquet --> renderers["CSV mirrors / Excel /
PNG plots / GUI ResultViewer /
scenario_comparison"]

Key invariants:

• One backend. SpineDBBackend is the only input backend. Everything upstream (Excel, ODS, CSV, old .xlsm) is converted to a Spine DB before the engine sees it.
• Lazy polars, in-memory. Frames stay as LazyFrame until they reach polar_high, which assembles the HiGHS matrix. Solver variables / duals are read back from the live highspy.Highs instance — no intermediate solver files.
• Parquet is the canonical output. CSV / Excel / PNG / GUI plots all read from the parquets the cascade just wrote. Scenario comparison loads multi-scenario parquet sets and emits combined renderings.

CLI surface

flowchart LR
  root["Root wrapper scripts
run_flextool.py, write_outputs.py,
scenario_results.py, ..."] --> cli["flextool/cli/cmd_*"]
  cli --> engine["flextool/engine_polars
cascade + LP build + HiGHS solve"]
  cli --> backend["flextool/spinedb_backend"]
  cli --> derive["flextool/input_derivation"]
  cli --> decomp["flextool/decomposition"]
  cli --> inputs["flextool/process_inputs"]
  cli --> outputs["flextool/process_outputs"]
  cli --> plots["flextool/plot_outputs"]
  cli --> compare["flextool/scenario_comparison"]
  cli --> export["flextool/export_to_tabular"]
  cli --> update["flextool/update_flextool"]
  cli --> gui["flextool/gui"]

Each cmd_*.py is a thin argparse wrapper; the work lives in the linked packages. Every CLI command also has an installed entry point in pyproject.toml (the flextool-* names).

Repository layout

flextool/                          The installed Python package
├── __init__.py                    Public re-exports: write_outputs,
│                                    migrate_database, initialize_database,
│                                    update_flextool
├── cli/                           Argparse wrappers (cmd_*.py) +
│                                    cli/_timing.py (TimingRecorder)
├── spinedb_backend/               Spine DB → polars frames (the input API).
│                                    SpineDBBackend.find_entities /
│                                    parameter_values / default_values.
├── input_derivation/              Per-solve preprocessing computations
│                                    (the live successor to the retired
│                                    flextoolrunner/preprocessing).
├── engine_polars/                 Optimization engine + cascade coordinator:
│                                    FlexData, FlexDataProvider, derived
│                                    parameter / set emitters (_emit_*.py),
│                                    build_flextool, run_orchestration /
│                                    run_chain_from_db, SolveHandoff,
│                                    auto-scaling.
├── decomposition/                 Region decomposition + region filter
│                                    (Benders sub-problems).
├── common_utils/                  Cross-package helpers (precision rounding).
├── process_inputs/                Tabular / Excel / old-format readers →
│                                    Spine DB writers.
├── process_outputs/               Solution → parquet + downstream CSV /
│                                    Excel rendering pipeline.
├── plot_outputs/                  PNG / SVG plot rendering + plot-plan
│                                    persistence (the viewer-readable
│                                    parquet plans).
├── scenario_comparison/           Multi-scenario parquet load + dispatch
│                                    & summary comparison rendering.
├── export_to_tabular/             Spine DB → Excel (.xlsx) exporter.
├── update_flextool/               DB schema migration chain, canonical-DB
│                                    generation, settings-DB initialisation,
│                                    repo self-update.
├── model_builder/                 Synthetic / mock-up scenario generator.
├── representative_periods/        Greedy convex-hull representative-period
│                                    clustering.
├── gui/                           Tkinter desktop application (main_window,
│                                    result_viewer, plot_canvas,
│                                    network_graph, check_tree, dialogs/...).
├── helpers/                       Small standalone diagnostic utilities
│                                    (file compare, LP coefficient inspector,
│                                    schema converters).
├── solver_config/                 Solver options templates (HiGHS + the
│                                    commercial solvers; shipped in the
│                                    wheel as *.opt.template, seeded into
│                                    the project-root solver_config/ on
│                                    first run).
├── schemas/                       Schema JSONs, YAML templates,
│                                    canonical-database JSONs, pre-v26
│                                    migration snapshots.
└── lean_parquet.py                Tiny parquet loader used by the GUI
                                     ResultViewer.

Root wrapper scripts (kept for Spine Toolbox workflow compatibility):
  run_flextool.py, write_outputs.py, scenario_results.py,
  migrate_database.py, read_tabular_input.py,
  execute_flextool_workflow.py, update_flextool.py

solver_config/                     Project-root user-editable solver options
                                     (highs.opt + gurobi/cplex/xpress/copt.opt
                                     — all gitignored; each seeded from its
                                     bundled *.opt.template on first run).

templates/                         Example databases and project assets
                                     (examples.sqlite, input templates).

migration_artifacts/               (none — the migration scratchpad was
                                     retired with the GMPL pipeline.)

tests/                             Test suite (single root):
  tests/                             — top-level unit & integration tests
  tests/engine_polars/               — engine-internal unit tests (cluster,
                                       constraints, scaling, perturbation, ...)
  tests/spinedb_backend/             — backend unit tests
  tests/model/                       — model-level end-to-end suites
  tests/gui/                         — GUI startup tests
  tests/decomposition/               — Benders / region-filter tests
  tests/fixtures/*.json              — single authoritative source for all
                                       test data; sqlite DBs are built on
                                       demand
  tests/scenarios.yaml + tests/expected/  — the scenario integration suite
  tests/conftest.py                  — shared top-level fixtures
  tests/engine_polars/conftest.py    — engine-internal fixtures
                                       (scenario_workdir, ...)

execution_tests/                   Spine Toolbox subprocess execution tests.

docs/                              MkDocs documentation site:
  docs/dev/                          — developer reference (this file,
                                       decomposition, scaling, slack convention)
  docs/*.md + assets                 — user-facing tutorials and reference.

.spinetoolbox/                     Spine Toolbox project integration.
.github/workflows/                 CI/CD (tests + verify hooks).
pyproject.toml, mkdocs.yml,        Build config, doc config, citation,
CITATION.cff, LICENSE.txt,         license, readme, release notes.
README.md, CHANGELOG.md

Module details

flextool/spinedb_backend/ — the input API

SpineDBBackend is the sole materialiser between the Spine database and the rest of the engine. It owns the EAV-to-tabular transformation: every method returns a canonical-schema polars.DataFrame, and no method writes to disk. The caller stack is:

SpineDB → SpineDBBackend → input_derivation → FlexDataProvider → cascade

Key surface:

• SpineDBBackend.find_entities(entity_class, ...) — entity rows.
• SpineDBBackend.parameter_values(entity_class, parameter, ...) — parameter rows with parsed values (Map / time-series / array / scalar).
• SpineDBBackend.default_values(...) — default-value lookups for cascade fallbacks.

A handful of small caching layers (parameter-value row cache, parsed-value eviction, chunked accumulators) keep peak RSS bounded on continental-scale inputs.

flextool/input_derivation/ — per-solve preprocessing

The per-solve derivation pipeline. Each module is a focused port of a self-contained preprocessing step (commodity ladder, DC power flow, process methods, period sets, ...). The package's __init__.py orchestrates the three-step pattern:

1. Spec-driven entity / parameter spec loops over the backend.
2. Derivation helpers project the raw rows into FlexData fields.
3. Validators sanity-check shape / dtype invariants before handoff.

Output is an in-memory FlexData augmented with derived fields, exposed to the engine through FlexDataProvider.

flextool/engine_polars/ — the optimization engine

The dominant package. Builds a polar_high.Problem from FlexData, solves it with HiGHS, and threads cross-solve state through SolveHandoff (in memory; CSVs only emitted under csv_dump=True for debugging).

Public surface

Exported from flextool.engine_polars (__init__.py):

Solve modes

Same build_flextool body, three wrappers from polar_high:

• Monolithic — polar_high.Problem. One LP per call.
• Warm-start cascade — polar_high.WarmProblem. Rebuilds the LP shell once, then mutates marked Params (_apply_warm_updates in _solve_handoff.py) between rolls. The optimisation room for the future rolling-overlap work (only update changing coefficients) lives here.
• Benders (regional) — engine_polars._benders.solve_benders driven by the orchestrator. Slices FlexData by region via _region_filter, then runs a Benders decomposition: a coordinating master (a persistent polar_high.WarmProblem) holds the inter-regional trade flows and the trade-connection investment plus one recourse variable η_r per region; each region is an operational subproblem solved with the master's trade pinned, returning duals that the master accumulates as optimality cuts (WarmProblem.add_cut_row / add_recourse_col). The master objective is a valid lower bound and the loop converges on the optimality gap. This replaces the earlier dual-subgradient scheme, which collapsed greenfield cross-region trade to autarky with an invalid bound. See decomposition.md.

Feature blocks in build_flextool

LP construction is feature-by-feature; each block runs only when its switch field on FlexData is populated. Required-field lists are module-level tuples in model.py:

If a switch is set but a required field is missing, build_flextool raises ValueError rather than silently degrading.

Block-COO dense-axis contract

build_flextool declares FlexTool's dense trailing axes once, up front, via m.declare_dense_axes(("d", "t")) — (period, timestep). This is an unconditional call requiring polar-high>=2.4.0 (the interim hasattr capability guard has been dropped; the pin lives in pyproject.toml). With the axes declared, polar-high's canonical-matrix build slices the pre-sorted dense axis as contiguous numpy blocks and multiplies coefficient factors with ufuncs (the block-COO slice path) instead of wide relational joins. Sum-wrapped chains — FlexTool's Sum-heavy families (nodeBalance, reserves, co2) — evaluate in one pass via a captured SumBlockMeta recipe with a relabel fast-path. The block-COO output is bit-identical to the legacy polars path; the win is purely the peak-memory and wall-time of the build.

Cross-solve state (Provider keys + SolveHandoff)

Inside the cascade everything moves through FlexDataProvider. The relevant key families are declared in engine_polars/_provider_keys.py:

• HANDOFF_* — quantities the just-solved problem produces for the next iteration (fix_storage_quantity/price/usage, roll_end_state, realized_invest, realized_existing, divest_cumulative, cumulative_co2, commodity_ladder_cumulative, cum_sim_hours, ed_history_realized_first, edd_history).
• INPUT_* — backend-derived rows the cascade re-reads each iteration.
• SOLVE_DATA_* — per-solve sets and parameters that the derivation layer produces.

The Provider is the contract. Disk-side CSV mirrors in solve_data/<solve>/ exist only when csv_dump=True (a debug toggle) and are not load-bearing for any downstream phase.

File map (most important)

__init__.py                       Public re-exports (table above)
model.py                          build_flextool — LP build, feature blocks
input.py                          FlexData dataclass + loader entry point
chain.py                          run_chain + ChainStep dataclass
_orchestration.py                 run_orchestration / run_chain_from_db,
                                  native cascade
_solve_state.py                   RunnerState, PathConfig, SolveConfig,
                                  TimelineConfig dataclasses
_solve_handoff.py                 SolveHandoff carrier + capture wiring
_benders.py                       solve_benders + Coupling / BendersResult
                                  (master + multi-cut loop)
_region_filter.py                 region slicing + half-flow injection
_warm.py                          WarmProblem update routine
_provider_keys.py                 Canonical key names for FlexDataProvider
_flex_data_provider.py            In-memory transport surface
_param_shapes.py                  Canonical shape per FlexData field
_spinedb_reader.py                Spine DB → FlexData
_inmemory_reader.py               In-memory test-fixture loader
_input_source.py                  FlexInputSource / CsvSource / InputSource
_db_loader.py                     FlexToolRunner: DB-side state loader
_db_reader.py                     Spine DB read helpers
_recursive_solve.py               Rolling / nested solve tree builder
_stochastic.py                    Stochastic branch handler
_timeline.py                      Timeline / timeset construction
_block_layout.py / _blocks.py     Multi-resolution period-block layout
_solver_base.py                   Solver invocation surface
autoscale/                        Auto-scaling pipeline + YAML audit report
_derived_*.py                     Calculated-parameter helpers (NPV, branch,
                                  walks, profile, block, ...)
_emit_*.py                        Set/parameter emitters (period_params,
                                  arc_unions, calc_params, co2_accumulators,
                                  pdt_params, reserve, leaf_sets, mid_sets,
                                  inflow_scaling, lp_scaling, ...)
_native_run_model.py              Per-solve run helper (cascade-internal)
_output_writer.py                 Solution → output_parquet bridge

Parameter shapes & (d,t) broadcasting

_param_shapes.py is the authoritative resolver for how a DB-authored parameter maps onto the model axes. Never detect a parameter's period / time axis by column name. Spine writes a Map's index column under the author's index_name, defaulting to the silent "x" when unset — so a column-name check like ("t","time","step") misclassifies a time series as a scalar and broadcasts it with a how="cross" Cartesian product against the full (d,t) grid. On long timelines that overflows polars' 2³² row index; on short ones it silently flattens the time variation via a downstream unique([...]). Both bit us (the p_slope blow-up). Route every param→(d,t) broadcast through resolve_param_shape (DB-metadata shape + value-domain probing) → broadcast_to_period_time (keeps the Param at its authored shape, no cross-product) → promote_param_to_dt (narrowing inner-join on the carried axis). A new parameter that needs this must be added to PARAM_ALLOWED_SHAPES.

_param_shapes is the I/O contract only — it validates user-authored DB shapes and errors to the user on mismatch. Model-internal axes (e.g. the rolling-solve roll axis) are deliberately excluded; they are never DB-authored, never reach the resolver, and a stray one already fails loudly via _UnrecognisedIndex. Document such internal axes in schemas/flextool_axis_contract.json, not in the resolver's allow-lists.

Silent-default "x" index — readers outside the resolver

Not every DB read goes through resolve_param_shape. The per-entity cascades in _derived_params.py / _derived_existing.py / _derived_npv.py read raw entity params (source.parameter(...)) and must detect the period axis themselves. The failure mode is identical to the resolver's: Spine names a Map's index column with the author's index_name, whose spinedb_api default is the literal string "x" (not empty — real 2-D maps carry index_name='x', see _spinedb_reader._discover_index_cols). So a if "period" in cols gate misfires on the common "x"-indexed map and the code either drops the row or collapses it via a downstream .unique(subset=["e"], keep="last") — for a retiring existing map (nameplate → 0 at expiry) that keeps the last period's 0, defaulting unitsize to 1000 and turning existing_count = existing/unitsize into a spurious fraction (caps continuous online below 1; forbids integer online entirely).

Safe idioms (use one of these for any raw per-entity map read):

• period-agnostic aggregate — group_by("e").agg(pl.col("value").max()) (used by _entity_unitsize_lf: the cascade only needs "non-zero", and the per-period MAX is non-zero iff any period is);
• explicit dual check — period_col = "period" if "period" in cols else "x" / for cand in ("period","x") (used by _ed_explicit_period_param, _per_solve_sets);
• per-row scalar detection — treat any single non-name/value column as the period dim and classify scalar-vs-map by that column's per-row null-ness, so a constant value sharing a class with period-Map siblings (surfaces as x = null) broadcasts instead of zeroing (used by _per_entity_param_lf in both the existing and npv mirrors).

Audited state (see the coverage test tests/engine_polars/test_silent_default_index_pdt.py):

• Safe — availability, inflow, other_operational_cost, reservation (resolver-routed); the existing/lifetime/ virtual_unitsize per-period chains (_per_entity_param_lf, hardened); _ed_explicit_period_param, _per_solve_sets; profile in real solves (reads the engine-written canonical pt_profile.csv / pbt_profile, not the raw Map).
• Residual, latent — the _derived_profile.py raw-Spine fallback readers (_profile_time_lf matches only t/i/time; _profile_period_time_lf / _profile_period_only_lf gate on literal "period"). An "x"-indexed profile Map read via these returns empty (silent zero profile). Only reachable when the averaged-CSV primary path is absent (tests / runner bypass), so not a live-solve bug — harden if that path ever becomes load-bearing.
• Guarded by schema — invest_max_total and virtual_unitsize are declared float-only in parameter_types (no map row), so the keep="last" collapse in _e_explicit_param and in the virtual_unitsize sub-branch of _entity_unitsize_lf can only bite if a param is authored against its declared type. Value-type enforcement is not yet strict, so these are latent-until-misauthored, not safe by construction.
• Separate gap — min_load is declared map-capable (map,1 / map,3) but p_min_load_from_source reads it as a scalar Param(("p",)), and minFlow_minload consumes only that scalar. A period/time-varying min_load is therefore flattened regardless of the index name — an engine "consumes scalar only" gap, not just an "x"-detection one. Route through the resolver if per-period min_load is ever wired.

flextool/process_inputs/ — tabular → Spine DB

Reads tabular data and writes Spine DB. The output of this package is the Spine DB the engine consumes.

flextool/process_outputs/ — solution → parquet → downstream

Three-layer architecture:

Solution extraction (in-memory, from highspy.Highs):
  read_highs_solution.py         VARIABLE_SPECS — variables, row duals,
                                 column duals harvested directly from
                                 the live solver instance.

Post-processing (parquet-backed calculations):
  drop_levels.py                 Strip 'solve' level from time-indexed objects.
  calc_capacity_flows.py         Capacity, online status, flow_dt, ramps.
  calc_connections.py            Connection flows and losses.
  calc_storage_vre.py            Storage state changes and VRE potential.
  calc_slacks.py                 Reserve, slack, inertia quantities.
  calc_costs.py                  All cost aggregates.
  calc_group_flows.py            Group-level flow aggregations.
  process_results.py             Coordinator: calls calc_* in order.
  handoff_writers.py             SolveHandoff CSV mirrors (csv_dump only).
  solve_order.py                 Solve-sequence reconstruction.

Output formatting:
  out_capacity.py                Unit / connection / node capacity tables.
  out_flows.py                   Unit flow, capacity-factor, VRE, ramp.
  out_group.py / out_flowgroup.py NodeGroup flow, inflow, VRE-share, indicators.
  out_node.py                    Node summary and additional-results outputs.
  out_costs.py                   Cost summary, CO2, generic outputs.
  out_ancillary.py               Connection, reserve, inertia, slack, dual.
  write_outputs.py               ALL_OUTPUTS list + orchestrator.
  to_spine_db.py                 Result DataFrames → Spine output DB.

The cascade produces output_parquet/<scenario>/. CSV / Excel / plot renderers all consume those parquets — no separate solver-output extraction step.

flextool/plot_outputs/ — PNG / SVG plots and plot plans

Static plots (lines, bars, stacked area) plus a persistence layer that saves precomputed plot plans (one parquet per result key) so the GUI's ResultViewer can re-render without recomputing pivots / pagination.

flextool/scenario_comparison/ — multi-scenario analysis

Loads parquets from multiple scenario folders, builds combined views, and renders comparison plots.

flextool/export_to_tabular/ — Spine DB → Excel

Exports a Spine input DB to .xlsx in self-describing v2 or original v1.

flextool/gui/ — Tkinter desktop application

Spawns CLI commands as subprocesses; never imports the engine directly.

__main__.py                      Entry point → MainWindow.
main_window.py                   MainWindow (tkinter.Tk root).
project_utils.py                 Create / list / rename projects.
settings_io.py                   Load / save GlobalSettings + ProjectSettings.
data_models.py                   Settings + ScenarioInfo dataclasses.
input_sources.py                 InputSourceManager (file selection).
scenario_lists.py                Available / executed scenario managers.
execution_manager.py             ExecutionManager (threaded subprocess pool).
execution_window.py              ExecutionWindow (progress).
output_actions.py                Plot / export action wiring.
output_log_window.py             Log display.
db_editor_integration.py         Spine DB editor launcher.
db_version_check.py              Input-DB version validation + auto-upgrade.
error_handling.py                safe_callback decorator.
platform_utils.py                Platform-specific file/app opening.
result_viewer.py                 ResultViewer (parquet → plot canvas).
plot_canvas.py                   Matplotlib canvas with downsampling.
plot_cache.py                    Plot cache.
plot_config_reader.py            Plot-config YAML parser.
network_graph.py                 Network-topology figure builder.
check_tree.py                    CheckTreeController.
hover_tooltip.py / tree_reorder.py / downsampling.py / cli_format.py / scenario_key.py / config_parser.py
dialogs/                         Add / project / plot / file-picker dialogs.

flextool/update_flextool/ — migration and updates

CI-enforced verify chain (tests.yml::template-check):

• sync_master_json_template --verify — schema parity.
• test_fixtures verify — tests/fixtures/*.json at current schema.
• generate_canonical --verify — generated canonical files in sync.
• canonical_databases verify — authoritative canonical files at current schema.

flextool/decomposition/ — Benders region helpers

region_decomposition.py and region_filter.py slice FlexData by region group and prepare sub-problems for the Benders solve mode (the filter-only --region CLI entry point that materialises one region's standalone input directory). The in-engine slicer used by the live Benders driver lives at engine_polars/_region_filter.py; the master + cut loop is in engine_polars/_benders.py.

flextool/model_builder/ and flextool/representative_periods/

• model_builder — synthetic scenario generator used for benchmarking and stress-testing the engine. build_model.py is the entry point; encoding.py, names.py, profiles.py are helpers.
• representative_periods — greedy convex-hull clustering used at the input-preparation stage to pick representative weeks / days. preprocess.preprocess_representative_periods is the entry point.

Net-load clustering mode

By default the greedy hull clusters on the per-series normalized profile/inflow feature stack (preprocess._build_clustering_matrix). The --netload-clustering flag (schema wiring lives in representative_periods/netload_inputs.py for the DB read and representative_periods/netload.py for the pure math) instead clusters on a real-MW net-load signal per aggregation unit g:

net_load[g, h] = Σ_{demand node in g} demand_h
                 − Σ_{VRE unit in g} cap[u] · avail[profile(u)][h]

FlexTool's sign convention (shared with force_include) makes demand a negative inflow, so demand_h = −inflow_h + |scalar demand level|. VRE is any unit__node__profile arc whose profile_method is upper_limit; that is the schema default, so an arc that never sets profile_method explicitly (no row from the reader) is treated as VRE. Each aggregation unit's series is min-max normalized to [0, 1] exactly as the default feature builder does per feature.

An aggregation unit is either a region group or a single node. Granularity is chosen by the schema-v68 group.use_for_representative_periods (yes_no) flag: when one or more groups carry the flag yes the member nodes' net load is summed into one regional signal per flagged group; when no group carries the flag the selection falls back to per-node signals (each node its own aggregation unit). On a pre-v68 database the flag parameter is not yet defined — that genuinely-absent case is detected and quietly activates the per-node fallback, while a read failure on a defined flag is allowed to surface rather than silently degrade.

The iteration-0 VRE capacities come from a demand-match default: the investable VRE of each aggregation unit is sized so its energy plus the existing VRE energy just covers the unit's demand energy (pure energy balance, curtailment ignored). --vre-penetration scales that energy-share target (1.0 = full-energy match). The default (non---netload-clustering) code path is byte-parity with the prior behaviour.

Net-load iteration driver

representative_periods/netload_iterate.py (run_netload_iteration, NetloadIterConfig; also a main()/__main__ CLI) composes the single-shot net-load selector into an iterate-until-stable loop that feeds each iteration's solved investment capacities back into the next selection — sizing the VRE fleet from a solve's realised investments (instead of the demand-match default) sharpens which periods are net-load-critical.

• Invest-only loop. Iteration k in range(iterations + 1). k = 0 is the bootstrap: select on the demand-match caps and run an INVEST-ONLY solve. The bootstrap dispatch is skipped — default-cap investments are known-bad, so measuring their full-year cost is not worth the solve. k = 1..n select on the prior iteration's invested caps; if the selected representative set is unchanged from the previous iteration the loop converges and breaks before solving (early-stop on stability).
• Keep-best (opt-in). With --keep-best, each mature iteration (k ≥ 1) ALSO runs the full-year DISPATCH, reads its total system cost, and the loop keeps the representative set with the lowest full-year dispatch cost. A keep-best winner from an earlier iteration is deterministically re-materialised by re-running the selector on that iteration's input caps (greedy hull over a fixed matrix — no randomness). A final dispatch is run on the chosen set for output.
• Stable-alternative-name orphan guard. The full alternative_name is pinned once and passed as the explicit override to the selector every iteration, so the selector never derives a per-iteration name (whose force-suffix could shift with the forced count) and orphans timesets. A stable name lets the selector's purge+rewrite cleanly REPLACE the single net-load timeset each iteration, so N iterations leave exactly one.
• model.solves-override subset mechanism. cmd_run_flextool has no --solves flag; the DB-driven way to run a subset is the model.solves Array resolved under the scenario's alternative stack. The driver writes a dedicated model.solves OVERRIDE alternative at the top of the stack (mirroring calibrate._db_alt) — the invest solve(s) for an invest-only pass, the dispatch solves (or the scenario's full original chain) for a dispatch pass. A genuine subset run, not a silent full-chain fallback.
• Fail-closed guards. _guard_no_timed_energy_margin refuses to run when a node's energy_margin_method resolves to inflow_adder with a timed (2-D per-cell) energy_margin_adder — the timed adder and the net-load loop's cap feedback are mutually exclusive. Every iteration solve is assess_solve-gated on its required outputs; an untrusted solve raises NetloadIterError rather than feeding a bad cap set forward. Basis caches are cold per solve by default (the LP columns change with the representative set); --warm-start opts into one shared cache only when the grid is stable.

Public API surface

from flextool import (
    write_outputs,
    migrate_database,
    initialize_database,
    update_flextool,
)
from flextool.engine_polars import (
    FlexData,
    FlexDataProvider,
    SolveHandoff,
    build_flextool,
    load_flextool,
    run_chain_from_db,
    run_orchestration,
)
from flextool.engine_polars._db_loader import FlexToolRunner
from flextool.spinedb_backend import SpineDBBackend
from flextool.process_inputs import TabularReader, write_to_flextool_input_db
from flextool.process_outputs import write_outputs, post_process_results
from flextool.plot_outputs import plot_dict_of_dataframes
from flextool.scenario_comparison import get_scenario_results
from flextool.export_to_tabular import export_to_excel
from flextool.update_flextool import (
    migrate_database,
    initialize_database,
    update_flextool,
)
from flextool.helpers import (
    compare_files,
    find_largest_numbers,
    parse_mps_to_matrices,
    convert_schema,
)

Solver layout

HiGHS is a regular Python dependency (highspy); FlexTool no longer bundles solver binaries. The only solver-config artefact is the HiGHS options file:

flextool._resources.package_data_path("solver_config/highs.opt.template") returns the template; the runtime layer copies it to solver_config/highs.opt on first run if absent. Users edit the runtime copy; _orchestration._finalise_highs_options reads it and applies any CLI overrides (--user-bound-scale, --presolve, --highs-threads).

--save-memory (or FLEXTOOL_SAVE_MEMORY=1) is an opt-in peak-RSS strategy that disables warm-start and round-trips each sub-solve through an on-disk MPS file, trading ~5-10 GB peak RSS for slower per-solve build time. The orchestrator emits a one-time warning when it disables warm-start because of this flag.

Solver outputs: folder layout

work_folder/
├── solve_data/<solve>/        Debug-only mirrors of cascade state
│                                (csv_dump=True path). Not load-bearing.
├── output_parquet/<scenario>/ Canonical results — one parquet per output.
├── output_csv/<scenario>/     Optional CSV mirror of the parquets.
├── output_excel/              Optional summary workbook.
└── output_plots/              Optional PNG / SVG plots (default_plots.yaml).

Cross-solve state moves through FlexDataProvider keys; the solve_data/ CSVs exist only as a debugging aid.

Solve-to-solve handoff

When a solve is part of a rolling-horizon, nested, or sub-solve chain, the just-solved problem produces a SolveHandoff (typed in-memory carrier). Inside the cascade it's transported via the HANDOFF_* Provider keys — zero disk round-trips. The csv_dump=True mirror files retain the same names as before for debug inspection:

fix_storage_timesteps, ed_history_realized_first, and edd_history used to be SolveHandoff fields too, populated only by the now-removed capture_post_solve() disk-read constructor. They were retired when the cascade fell through to Provider/CSV reads in all production paths; see the docstring of _solve_handoff.py for the history.

Solution extraction

process_outputs.read_highs_solution.VARIABLE_SPECS declares every variable and dual to harvest from the live highspy.Highs instance: v_flow, v_state, v_invest, v_divest, vq_*, period / total / cumulative invest-cap duals, co2_max_* duals, the inflation-adjusted v_dual_node_balance, and so on. Each VariableSpec carries an unscale_by field so row / objective scaling is undone before the parquet is written.

To add a new variable or dual: append a VariableSpec to VARIABLE_SPECS. No other changes required.

Numerical scaling

FlexTool's LP/MIP can accumulate coefficients spanning many decades when users combine entities of different physical scale (10 kW heat pump alongside 10 GW continental grid). The scaling pipeline is layered so each mechanism targets one source of spread; see scaling.md for the user-facing guide and slack_convention.md for slack semantics.

Layers (bottom up)

1. Unitsize column normalisation. Every variable is written in units of p_entity_unitsize per entity, so the raw values stay near O(1).
2. Single-variable slack convention. Every vq_* is a single non-negative variable, relative to its row-scaler where one applies. See slack_convention.md.
3. Row scaling (opt-in, --scaling CLI flag / FLEXTOOL_SCALING). Node-balance and group-balance constraint rows get multiplied by node_capacity_for_scaling / group_capacity_for_scaling derived from connected-entity unitsizes; rounded to powers of 10 to preserve HiGHS symmetry detection.
4. Precision cleanup (--precision-digits). Every numeric value is rounded to 10 significant figures so benign precision artifacts don't trip near-duplicate detection.
5. Autoscale package (engine_polars/autoscale/). Three layers, on by default (gate via --scaling=off / FLEXTOOL_SCALING=off; intermediate modes are solver_only, basic, full):
6. Layer 1 — detect (_ranges.py). Walks the assembled LP and computes matrix / cost / bound / RHS log10 ranges plus a cross-group max-ratio. Always emitted to the YAML audit; the trigger flag tells the orchestrator whether Layers 2/3 should run. The range readout routes through polar-high's bounded_coefficient_walk (requires polar-high>=2.4.0): the constraint / column spine is iterated in fixed row-batches and the (rid/col_id, coef) product is rebuilt per batch, so the wide coefficient product is never materialised to read its min/max. The former size-blind family-row cap (POLAR_HIGH_RANGES_MAX_FAMILY_ROWS / _skip_unbounded_over_cap) is gone — every family's range is now bounded and folded into the report, so no shape is silently dropped.

7. Layer 2 — semantic per-type scaling (_layer2.py, _quantity_types.py). Buckets matrix columns by quantity-type (energy, capacity, monetary, …) and applies a power-of-2 scaler per type to compress the median-of-type onto O(1). Has structural limits: cannot compress within-type spread, and the Rivendell guard prevents over-shrinking columns with mixed-magnitude coefficients on a single row. Like Layer 1, the magnitude-bucketing readout walks the spine in bounded row-batches via bounded_coefficient_walk (the log2-magnitude histogram reducer) rather than collecting the wide product — and with the family-row cap retired, every family's magnitudes are folded into the scaling decision (no silent coverage gap).

   Registry contract — read this before adding any add_var / add_cstr / parameter. Layer 2 looks up the quantity-type of every variable (VARIABLE_FAMILIES), constraint (CONSTRAINT_FAMILIES via lookup_cstr) and parameter (PARAMETER_TYPES) in _layer2_types.py / _quantity_types.py. These registries refuse to default — an unregistered name raises KeyError in bucket_coefficients. That KeyError is caught in _orchestration._autoscale_apply_layer2_pre_solve and the solve silently reverts to an un-scaled LP ("autoscale Layer 2 apply failed … reverting"), so a missing registration does not crash — it quietly disables autoscaling. So: every new add_var(...) / add_cstr(...) literal and every new schema parameter MUST get a registry entry (flags / yes_no gates → DIMENSIONLESS). lookup_cstr resolves dynamic (f-string) constraint names by exact-match → _linear/_integer/_p/_n suffix strip → longest prefix; dynamic names are invisible to the static-literal grep in test_registry_coverage, so pin them in test_dynamic_constraint_names_resolve. Run autoscale-exercising tests with FLEXTOOL_AUTOSCALE_STRICT=1 to turn the silent degrade into a loud failure (off in production so a gap never blocks a user solve). - Layer 3 — HiGHS native + bound-scale folding (_layer3.py). Picks an integer user_bound_scale exponent (within HiGHS's safe range) and folds D's escape-tier valve into the same pass, so the LP arriving at HiGHS is comfortably within the solver's conditioning sweet spot. 6. YAML audit (engine_polars/autoscale/_report.py). After every solve, writes solve_data/autoscale_<solve>.yaml covering Layer 1 ranges, the Layer 2 plan (per-type exponents + post-L2 ranges), and the Layer 3 plan (user_bound_scale, escape-tier). A 1-line console summary echoes via format_console_summary; the non-optimal hint (format_nonoptimal_hint) fires only when a rejected solve's post-autoscale LP is still wide-ranged (i.e. the autoscaler could not tame the spread), and even then it is framed as a possibility — never as the certified cause. Accept/reject itself is decided by engine_polars/_solve_acceptance.py from the solver's own diagnostics (see "Solve acceptance" below), not by the range report. 7. Manual override. --user-bound-scale N (or DB solve.user_bound_scale) pins Layer 3's bound-scale exponent and disables the auto-pick. Useful when the audit hints at a value the recommender did not lock in. 8. Output un-scaling (process_outputs/read_highs_solution.py). Every VariableSpec carries an unscale_by field; the Layer 2 + Layer 3 reverse maps are applied in the writer so downstream consumers see absolute units regardless of which layers fired.

Solve acceptance

The cascade does not gate on a bare kOptimal. An interior-point solve run without crossover (the model generator's run_crossover choice) can return a genuinely feasible, in-practice-optimal primal that HiGHS refuses to certify — post-solve leaves the dual objective slightly inconsistent, so HiGHS reports kUnknown rather than a failure. FlexTool consumes the primal for every output, so such a solve is usable.

engine_polars/_solve_acceptance.py::classify_acceptance decides from the solver's own diagnostics (polar_high.Solution.solve_diagnostics(), a snapshot of getModelStatus() + getInfo()):

• kOptimal → accept.
• a genuine failure status (kInfeasible, kUnbounded, kTimeLimit, kSolveError, …) → reject, naming the precise cause.
• kUnknown → accept as near-optimal iff the primal is feasible (HiGHS' own verdict + a scale-invariant relative check + a tolerance-derived absolute backstop) and primal_dual_objective_error is within the accepted optimality gap; otherwise reject.

Thresholds default to primal-relative 1e-6, absolute margin 10× the solver's primal_feasibility_tolerance, and P-D gap 1e-2 (the same order as the mip_rel_gap models routinely accept); each has a FLEXTOOL_ACCEPT_* env override, and FLEXTOOL_ACCEPT_NEAR_OPTIMAL=0 restores the legacy hard-fail. The subprocess/commercial path exposes no queryable handle, so solve_diagnostics() returns None and the solve is never near-optimal-accepted — it falls back to the strict kOptimal gate.

Test layout and fixture architecture

JSON-fixture single source of truth

tests/fixtures/*.json is the only authoritative source for test data and canonical example databases. Schema migration covers them via flextool.update_flextool.test_fixtures.migrate_all, and the CI verify chain enforces drift detection.

No test reads a checked-in .sqlite. A test that needs a DB builds one from its JSON / schema source under tmp_path_factory at session start (e.g. json_to_db(...) for fixtures, or the schema_db_url fixture which runs initialize_database on spinedb_schema.json once per session). Checked-in DBs (templates/*.sqlite) are user-facing and go stale between regenerations: a registry-coverage test that read templates/input_data_template.sqlite stayed green through the v56 is_enabled add because the on-disk template lagged the schema. Build from the source of truth so a schema change is caught the moment it lands — and never mutate a user-facing DB from a test. Some downstream JSONs (e.g. templates_examples.json, howto_stochastics.json) are generated projections of authoritative fixtures via flextool.update_flextool.generate_canonical; the remaining howto_*.json files are individually authoritative.

For append-only edits to fixtures (entities / alternatives / scalar parameter values / scenarios), use flextool.update_flextool.extend_tests_fixture — it validates against spinedb_schema.json with did-you-mean suggestions and rejects edits to existing entries.

Test directory layout

tests/
├── conftest.py                  Shared fixtures (scenario_workdir, ...).
├── fixtures/                    Authoritative JSON fixtures.
├── scenarios.yaml               Integration-suite scenario definitions.
├── expected/                    Golden CSVs (per-scenario, regenerated
│                                  when the suite genuinely changes).
├── engine_polars/               Engine-internal unit tests.
├── spinedb_backend/             Backend unit tests.
├── model/                       Model-level end-to-end suites.
├── gui/                         GUI startup tests.
├── decomposition/               Benders / region-filter tests.
└── test_*.py                    Top-level unit & integration tests.

scenario_workdir(scenario_name, db_fixture="main") (defined in tests/engine_polars/conftest.py) is a session-scoped fixture that builds a per-scenario workdir on demand by running the full cascade with csv_dump=True. It replaced the historical bulky CSV snapshot pattern that previously sat under tests/engine_polars/data/work_*/ (~521 MB gitignored); the directories that remain contain only a single golden_obj.json golden file each. Tests do not depend on pre-built fixture directories.

Three-tier gate

• Inner loop — module-specific tests, run after every edit (typically tests/engine_polars/<area>/).
• Broader gate — tests/engine_polars/ full sweep, once per commit batch.
• Activation truth — full tests/ including the integration scenario suite, run rarely (overnight / release).

Key architectural patterns

• CLI → core delegation. Every user-facing entry point (CLI script or GUI) is a thin wrapper that delegates to a core library module. The Tkinter GUI is special: it spawns CLI commands as subprocesses rather than calling library functions directly, so user runs stay isolated.
• State containers. RunnerState, SolveConfig, TimelineConfig, PathConfig bundle related state for the cascade. The Provider is the transport surface between phases.
• Dataclass-driven module boundaries. FlexData, SolveHandoff, DatabaseContents, TimeSeriesResults, DispatchMappings, SheetSpec define data shapes where packages meet.
• Multi-format I/O at the edges only. Inputs (Excel / ODS / CSV / old .xlsm / MATPOWER) convert to Spine DB; outputs render from parquets. The engine itself is single-source / single-sink.
• Verify hooks instead of conventions. CI verifies that schemas, canonical DBs, and test fixtures stay in sync — drift is a failed workflow, not a code-review catch.