Skip to main content

CI/CD runner protocol — implementation

git-shark does not ship its own runner. It implements the server side of the Forgejo/Gitea runner.v1 protocol so stock forgejo-runner / act_runner binaries work unchanged. This document covers how the server side is built and what is / isn't done.

Component map​

ComponentFileRole
Proto definitionssrc/main/proto/{ping,runner}/v1/*.protoMIT-licensed copies of gitea.com/gitea/actions-proto-def, with java_package/java_multiple_files options added.
Generated messagesde.workaround.ci.proto.* (build output)Protobuf message classes generated by protobuf-maven-plugin at generate-sources.
Connect endpointci/ConnectRunnerResource.javaJAX-RS resource serving the Connect unary RPCs under /api/actions.
Registration/presenceci/RunnerRegistrationService.javaToken issue, runner register/declare/authenticate, list/delete.
Task dispatchci/TaskDispatchService.javaFetchTask: authenticate, claim the oldest PENDING task, flip task+run to RUNNING, set the runner ACTIVE and the task deadline — atomically.
Task progressci/TaskProgressService.javaUpdateTask (result → task status + run roll-up, runner back to IDLE) and UpdateLog (resume-safe log-row append, ack_index).
Zombie reclaimci/ZombieReclaimService.javaScheduled sweep failing RUNNING tasks past their deadline (vanished runner) and rolling up their runs.
Run controlsci/ActionRunService.javaCancel a run (settle run + unfinished tasks) and re-run a finished run (reset tasks to PENDING, clear logs/outputs).
Commit statusci/CommitStatusService.javaAggregate a commit's runs into one status; shown on commit/MR pages and via the Gitea commit-status API.
Actions UIweb/ActionResource.java + templates/ActionResource/Read-only per-repo run list + run detail (jobs and their log rows); sidebar Actions tab.
CI settings UIweb/ActionSettingsResource.java + ci/ActionSecretService.java + templates/ActionSettingsResource/Owner-only, at settings/actions: CRUD for secrets (write-only, encrypted) and variables, plus minting repo-scoped runner tokens and listing/deleting the repo's runners.
Entitiesmodel/CiRunner.java, model/CiRunnerRegistrationToken.javaRunner state (migration V19).
Run entitiesmodel/ActionRun.java, model/ActionTask.java, model/ActionLog.javaRun/job/log-row persistence (migrations V23–V29). ActionTask.seq (bigserial) = surrogate int64 Task.id; runs_on = matching labels; needs = job dependencies; outputs = reported job outputs (JSON); job_id = workflow job key (shared by a matrix job's cells).
Secret/variable entitiesmodel/ActionSecret.java, model/ActionVariable.javaPer-repo CI secrets (encrypted) and variables (migration V26), delivered to runners in FetchTask.
Workflow ingestci/WorkflowIngestService.java, ci/WorkflowRunFactory.javaPost-receive hook: parse .forgejo/.gitea workflows at the pushed head, evaluate on: push, persist a run + its PENDING tasks (drained by FetchTask).
Admin UIci/AdminRunnerResource.java + templates/AdminRunnerResource/Token generation, runner list, deletion.
Admin gateaccount/AdminAccess.javaConfig-driven instance-admin check.

Key decisions (and why)​

  • Reuse the Forgejo/Gitea runner, don't write our own. The wire protocol is small, open, and MIT-licensed; the runner ecosystem and GHA-compatible workflow format come for free. The real cost is server-side workflow semantics (trigger/DAG/matrix), not the transport — see the issue for the full rationale.
  • Connect unary over plain JAX-RS, not quarkus-grpc. Connect unary RPC is just an HTTP POST with a serialized-protobuf body and a serialized-protobuf 200 body. We only need the generated message classes; serving them from ordinary JAX-RS resources avoids pulling in a gRPC server (extra listener, HTTP/2) we don't use. Hence protobuf-maven-plugin (messages only) rather than the gRPC codegen. Errors use the Connect JSON error shape ({"code","message"}) with the matching HTTP status.
  • Two secret types, hashed at rest. A reusable, instance-scoped registration token (admin issues it; runner presents it once in the Register body) and a permanent per-runner secret (minted in Register, sent to the runner exactly once, presented in x-runner-token thereafter). Only SHA-256 hashes are stored — same model as AccessToken.
  • Instance-scope, config-based admin. No admin role in the schema yet; AdminAccess reads gitshark.admin.handles. Deliberately minimal so phase 1 doesn't drag a roles migration with it.
  • uuid distinct from the DB primary key. The runner echoes uuid in headers; keeping it separate from the internal UUID id means the runner never learns our primary key.

What works today​

  • protobuf-maven-plugin generates the ping.v1 + runner.v1 message classes at build time.
  • Ping health check.
  • Register: validates the registration token, creates a ci_runner, returns the per-runner secret once. Registration tokens are reusable.
  • Declare: authenticates by uuid + secret, refreshes version/labels/last_seen, returns the runner.
  • Admin UI: generate/delete registration tokens, list/delete runners; gated by AdminAccess and the /admin/* authenticated policy.
  • Run-persistence tables: action_run, action_task, action_log (migration V23) with their Panache entities. action_task.log_length is the durable log-row count that doubles as the UpdateLog resume/ack offset; action_task.deadline is the zombie-timeout anchor.
  • Workflow ingest on push: the post-receive hooks (HTTP + SSH) call WorkflowIngestService, which reads .forgejo/workflows/*.{yml,yaml} and .gitea/workflows/* at the new commit of each updated ref, parses them (Jackson YAMLMapper), and for those whose push trigger matches the ref persists one action_run (per-repo number, PENDING) with one PENDING action_task per job via WorkflowRunFactory (@Transactional). Handles the YAML-1.1 on:→boolean-true key coercion.
  • Ref-based trigger filters: a bare/list on: push triggers on any branch push (never tags); an on: { push: {...} } object honors branches/branches-ignore (branch pushes) and tags/tags-ignore (tag pushes) with GitHub-style globs (globToRegex: ** spans /, */? do not). A tag-only filter block excludes branch pushes.
  • Path filters: paths/paths-ignore are matched against the files changed by the push (changedPaths diffs old→new, or the empty tree for a new ref, capped at 5000 paths). paths runs when any changed file matches; paths-ignore runs unless every changed file is ignored.
  • FetchTask dispatch: a registered runner claims the oldest PENDING task (TaskDispatchService, one transaction) — task+run flip to RUNNING, the runner goes ACTIVE, action_task.deadline is set, and the task is delivered with its surrogate int64 seq id and workflow_payload. The candidate row is locked FOR UPDATE SKIP LOCKED (id-only select, to keep the lock off the nullable runner join) so concurrent fetchers never claim the same task. The delivered Task carries a github.* context (job, ref, sha, repository, run_id, …) built in ConnectRunnerResource.toProto — without it the runner cannot select the job from the workflow and nil-derefs. Auth failures return the Connect unauthenticated error. No long-poll; tasks_version is a coarse max-seq.
  • Secret & variable delivery: a claimed task is handed its repository's variables (plaintext) and secrets (action_secret, stored with the SecretCrypto envelope, decrypted at delivery — a value that fails to decrypt is dropped, never sent as ciphertext) in the FetchTask Task.secrets/vars maps. Same trust model as GitHub self-hosted runners: secrets go to whatever runner claims the task (over TLS). No repo/org scoping of secrets and no fork-PR guard yet (no PR triggers exist).
  • Per-job payloads & matrix: ingest builds each task a standalone single-job workflow_payload (the original name/on plus just that job). A job with strategy.matrix expands into one task per cross-product cell — display name job (v1, v2), shared action_task.job_id, and a payload whose strategy.matrix is reduced to that single cell so the runner resolves matrix.*. github.job is the job_id; needs and cascade-cancel group a job's cells by job_id (a dependent waits for all cells, and is cancelled if any cell fails). include/exclude not yet handled.
  • needs ordering: each task records the jobs it depends on (action_task.needs, parsed at ingest). Dispatch will not hand out a task until every needed job in the run has succeeded; when a needed job ends FAILURE/CANCELLED, rollUpRun cancels the dependents (to a fixpoint, so the cancellation cascades) and the run reaches a terminal state instead of hanging. A dispatched task carries its needs' results and outputs in Task.needs.
  • Job outputs: a job's outputs (UpdateTaskRequest.outputs, sent incrementally by the runner) are accumulated into action_task.outputs (JSON) and echoed back as sent_outputs; dispatch delivers a needed job's outputs to its dependents as needs.<job>.outputs. ActionOutputs (de)serializes the JSON, fail-safe to an empty map on a bad value.
  • Scoped runners: a registration token (and the runners it creates) may carry a repository_id (V30) or an organisation_id (V31) on ci_runner_registration_token/ci_runner; neither set = instance-scope. Dispatch's scopeAllows hands a repo-scoped runner only its repository's tasks, an org-scoped runner any task in a repo owned by that org, and an instance runner any task. Scoped rows cascade-delete with their repository/organisation. Repo owners mint a repo-scoped token and list/delete the repo's runners under Settings → CI (ActionSettingsResource); the instance-wide admin page (AdminRunnerResource) mints unscoped tokens. Org-scoped tokens exist in the model and are honored by dispatch, but there's no UI to mint one yet (service-only).
  • Label matching: a task carries its job's runs-on labels (action_task.runs_on, parsed at ingest). Dispatch scans PENDING tasks oldest-first and claims the first whose labels are all advertised by the fetching runner (empty runs-on = any runner); an incompatible task is left for a runner that can serve it. The claim still locks the chosen row FOR UPDATE SKIP LOCKED and re-checks PENDING, so label filtering doesn't weaken the no-double-dispatch guarantee.
  • UpdateTask / UpdateLog progress (TaskProgressService): UpdateTask records the reported result, sends a finished task's runner back to IDLE, and rolls the owning run's status up from all its tasks (RUNNING until every task is terminal, then the worst outcome). UpdateLog appends log rows with resume-safe contiguous semantics — rows below action_task.log_length are ignored, a gap above it stops the append, and the returned ack_index is the durable row count. Both reject a task not assigned to the calling runner (unauthenticated) and an unknown task id (not_found).
  • Tests: RunnerRegistrationServiceTest (service), ConnectRunnerResourceTest (protobuf-over-HTTP round-trip for Ping/Register/Declare + auth failures), AdminAccessTest (admin gate), ActionRunPersistenceTest (run/task/log persistence, per-repo run numbering, pending-task lookup), WorkflowIngestServiceTest (push → run/task creation, non-push trigger and no-workflow are no-ops), WorkflowTriggerFilterTest (branch/tag include+ignore globs, bare push branches-only, tag pushes), WorkflowPathFilterTest (paths runs on a matching changed file, paths-ignore skips only when all changed files are ignored), FetchTaskTest (claim oldest pending over the wire, empty queue, bad credentials, and two runners racing one task → claimed at most once), TaskProgressTest (UpdateTask success rolls up task+run and frees the runner, UpdateLog append + dedup/resume, cross-runner and bad-credential rejection), LabelMatchingTest (runner claims a compatible task and skips an incompatible older one, gets nothing when none match, unconstrained task runs anywhere), SecretDeliveryTest (claimed task receives repo secrets decrypted + variables; empty fetch carries none), SecretsSettingsTest (owner adds a secret stored encrypted and never shown, adds/deletes a variable, duplicate-name rejected, stranger/anonymous get 404), NeedsOrderingTest (dependent waits for its need then receives its result; a failed need cancels the dependent and ends the run), NeedsOutputsTest (dependent receives an upstream job's outputs; outputs accumulate across incremental UpdateTask calls), CancelRerunTest (cancel settles run+unfinished tasks, re-run resets a finished run, a cancelled task tells the runner to stop via UpdateTask) and ActionControlUiTest (owner cancels/re-runs over HTTP, a non-writer is refused), SupersededRunsTest (a new push cancels the branch's earlier running run but leaves other branches alone), MatrixExpansionTest (single- and two-dimension matrices expand to one task per cell with a reduced payload; a non-matrix job stays single), MatrixNeedsTest (a dependent waits for every cell of a needed matrix job, and one failed cell cancels the dependent), CommitCiStatusTest (commit page + MR page show the aggregate badge, the commit-status API reflects failure, and a commit with no runs stays all-clear), EphemeralRunnerTest (an ephemeral runner is removed after its task — on completion and on zombie-reclaim — and its credentials stop working), ScopedRunnerTest (a repo-scoped runner skips other repos' tasks and idles when only they have work; an instance runner claims across repos), SecretsSettingsTest also covers the repo Settings → CI runner UI (owner mints a scoped token, lists/deletes the repo's runners; a stranger is refused), OrgScopedRunnerTest (an org-scoped runner serves its org's repos and idles otherwise).
  • Ephemeral runners: a runner registered with ephemeral=true is one-shot — once its single task reaches a terminal state it is deleted (ci_runner row removed; the task's runner_id is ON DELETE SET NULL), so its credentials stop working and it never gets a second task. This holds on both completion (TaskProgressService) and timeout (ZombieReclaimService deletes rather than just flagging OFFLINE). The stock act_runner --ephemeral client exits on its own after the job.
  • Zombie reclaim (ZombieReclaimService): a scheduled sweep (gitshark.ci.zombie-reclaim-interval, default 1m) fails any RUNNING task whose action_task.deadline has passed — the runner is presumed gone — rolls its run up, and flags the runner OFFLINE. The deadline is set at claim time from gitshark.ci.task-timeout (default 1h). A late update from a runner cannot resurrect an already-terminal task. ZombieReclaimTest covers both the reclaim (overdue → FAILURE, in-deadline left RUNNING) and the anti-resurrection guard.
  • Cancel & re-run: ActionRunService.cancel settles a run and its unfinished tasks; a task still running on a runner keeps its assignment but is told to stop the next time it calls UpdateTask (the response's TaskState.result is flipped to CANCELLED). rerun resets a finished run's tasks to PENDING (clearing runner, timing, logs and outputs) so they are picked up fresh. Both re-fetch the run inside the transaction (the entity arrives detached from the resource) and are gated on repository write access at POST .../actions/{n}/cancel and .../rerun (buttons on the run page).
  • Superseded runs: after ingest creates the run(s) for a push, ActionRunService.cancelSuperseded cancels that branch's other still-active runs (keeping the just-created ones), so an in-flight run is abandoned when a newer commit lands on the same ref. Other branches are unaffected.
  • Commit / MR status: CommitStatusService.aggregate folds a commit's runs (ActionRun.Repo.findByRepositoryAndCommitSha) into one status (worst-of: FAILURE > RUNNING > CANCELLED > SUCCESS; no runs → none). Shown as a badge on the commit-detail page and on the MR page (for the source branch's head commit, resolved live). The Gitea commits/{ref}/status API now returns the real aggregate (mapped to success/failure/pending) with one entry per run — a commit with no runs still reports success so Renovate proceeds.
  • Actions UI: a read-only Actions tab on each repository — ActionResource renders a run list (workflow, run number, status, event, short commit) and a run detail page with each job and its streamed log rows. Read-gated like the rest of the repo UI (404 for a hidden repo). Tested by ActionUiTest (list shows runs + tab, detail shows jobs and logs, unknown run number → 404).
  • Real-runner round-trip: ForgejoRunnerRoundTripTest starts an actual gitea/act_runner container (Testcontainers, :host execution so no docker-in-docker), which registers, fetches a queued task over the Connect protocol, runs its run: step, and reports SUCCESS with streamed logs — exercising Register/Declare/FetchTask/UpdateTask/UpdateLog against the genuine client. Needs a Docker daemon; self-skips otherwise.

What still needs to be implemented​

  • Long-poll & real tasks_version: FetchTask returns immediately and tasks_version is a coarse max-seq (bumps on creation, not state change), so with several simultaneous PENDING tasks a runner may under-poll. Add server-side long-poll and a state-driven version counter.
  • Non-push events: only push is evaluated; pull_request, scheduled and manual triggers are not. (!-negation within a single pattern list is also not supported.)
  • Matrix advanced options: include/exclude and fail-fast/max-parallel are not honored (plain dimension cross-product only).
  • Org-scoped token UI: dispatch enforces org scope, but there's no page yet to mint an org-scoped registration token (repo-scoped tokens have one; org-scoped are service-only).
  • Later phases: artifacts (ACTIONS_RESULTS_URL), non-push events.

References​