Engine Internals

Crate Structure

Crate	Purpose
api	Axum HTTP server, route handlers, request validation
engine	Engine trait, shared runtime, lifecycle hooks
engines/generic	Generic binary — reads config, delegates trait methods
engines/nextflow	Nextflow binary — baked-in config, lifecycle hooks
common	Shared domain types (RunState, RunId, etc.)
telemetry	Tracing/subscriber init

Engine Trait

New engines implement the following methods:

pub trait Engine: Send + Sync {
    fn new() -> Self;
    async fn get_workflow_results() -> Result<...>;
    async fn get_task_logs() -> Result<...>;

    // Lifecycle hooks (default no-ops)
    async fn pre_run(&self, ctx: &HookContext) -> Result<()>;
    async fn on_success(&self, ctx: &HookContext) -> Result<()>;
    async fn on_failure(&self, ctx: &HookContext) -> Result<()>;
    async fn post_run(&self, ctx: &HookContext) -> Result<()>;
}

HookContext provides run metadata, an HTTP client, and a ServiceTokenClient for authenticated S2S calls (e.g. DRS registration). Hooks are called by the runtime — engines only override what they need.

Everything else — NATS subscription, process execution, log capture, state transitions — is handled by the shared engine crate runtime.

Execution Pipeline

1. init → 2. build → 3. pre_run hook → 4. execute → 5. monitor
6. parse → 7. update_db → 8. lifecycle hooks → 9. cleanup

Hook failures are logged but do not fail the run.

Adding a Custom Engine

1. Create a new crate under crates/engines/ that depends on engine.
2. Implement the Engine trait for result parsing, task log extraction, and any lifecycle hooks.
3. Call engine::server::bootstrap(MyEngine::new(), "name") from main.
4. Write an engine-config.yaml. Config values support ${ENV_VAR:default} syntax — resolved from environment variables at startup. See Engine Configuration.
5. Bundle the config in the Docker image or mount it via ENGINE_CONFIG_PATH.

For engines that don't need custom parsing, metis-engine-generic with a config file is sufficient.

Environment Variable Substitution

Engine configs support ${VAR:default} placeholders. These are resolved from environment variables when the config is loaded. If the variable is unset and a default is provided, the default is used. This allows baking engine behavior into the Docker image while letting infra control paths via Helm env vars.