Projects

Project · Completed MVP · Sep 2026 — Oct 2026

SIRD: Integrating a Distributed Document Recognition System

SIRD extracts an insurance number, a person's name and date of birth from Russian pension-insurance documents. As the team's DevOps & Systems Integration Engineer, I connected the applications and infrastructure into a Kubernetes environment with documented end-to-end recognition tests.

Role
DevOps & Systems Integration Engineer
Period
Sep 2026 — Oct 2026
Status
Completed MVP
Stack
k3s · Helm · GitHub Actions · GHCR · PostgreSQL · RabbitMQ · RustFS · Keycloak · Prometheus · Grafana · Loki · Alloy
Outcome
Documented recognition scenarios succeeded before and after recovery; overall system acceptance and the full workload target remained unconfirmed.
SIRD document-recognition interfaces and service pipeline
SIRD project overview with the recognition interface, developer portal and service pipeline.

Context and Scope

We built SIRD as a four-person university project at HSE University. The intended users were document-processing staff and developers connecting consumer applications to a shared recognition API. The MVP covered three SNILS document layouts, with explicit diagnostics for missing or uncertain fields. Human review and corrections belonged to the consumer application.

Recognition took place behind an asynchronous API: clients needed a task identity they could poll while processing continued. This required coordination between application services, messaging, databases, object storage and identity management. My responsibility was to make those boundaries work together and provide evidence of their behavior. Other team members developed the OCR engine, backend business logic and interfaces.

How the System Worked

A consumer application submitted files to Task Service over HTTP. Task stored the source document in RustFS through its S3-compatible API and persisted task metadata in PostgreSQL. An outbox supported publishing work to RabbitMQ over AMQP without relying on an immediate, uninterrupted broker connection.

Process Service consumed work, managed processing state and retries, and invoked OCR through a bounded unary gRPC call. OCR read the authorized source object from S3; Process then stored the recognition result and published an outcome event. Task consumed that event and exposed status and results to the owning client. The workflow was asynchronous between Task and Process, with a synchronous OCR call inside the processing stage.

Keycloak supplied identity and tokens, while ACL supported access management. Two interfaces served different purposes: an administrator/developer portal used ACL, and a consumer demo connected to Task through a backend-for-frontend. Private document payloads lived in object storage; PostgreSQL held metadata, states and references.

Consumer App submits to Task Service; RabbitMQ connects Task and Process; Process invokes OCR over gRPC, with PostgreSQL, RustFS and Keycloak/ACL dependencies
SIRD service architecture and protocols, including document processing, storage and identity management.

My Contribution

I configured and operated the integration environment on two Debian virtual machines using k3s, with one control-plane node and one worker. I worked on persistent storage, PostgreSQL databases and accounts, RabbitMQ users and queues, and RustFS configuration. Helm charts and environment values connected the applications to these dependencies through service addresses, Secret references and transport settings.

I contributed four merged pull requests establishing backend CI and image publishing across Task, Process and ACL. The workflows ran Gradle verification before publishing containers to GHCR, with publication restricted to pushes to the relevant main branch. Kubernetes deployment remained a separate operation. I also helped select compatible service images and configure OIDC, S3 permissions and the shared artifact-capability key used between Process and OCR.

For operational visibility, I deployed Prometheus/Grafana and later connected application and broker metrics. I coordinated the limited Loki/Alloy rollout and checked its network policy. My security work included enabling k3s Secrets encryption, creating encrypted datastore backups before and after that change, and configuring a successful RustFS SSE-S3 test. I used Codex for parts of configuration preparation, auditing, recovery and testing, while setting scope and reviewing results.

Engineering Decisions in Practice

Aligning service boundaries

Integration exposed mismatches that individual service startup checks could not resolve. OCR expected an object key in its signed artifact capability, while the earlier Process implementation did not supply the agreed data. Kubernetes configuration also contained an assumed storage hostname and an OCR network-policy selector that did not match Process pod labels.

I compared contracts, actual Services and chart selectors, configured the correct endpoint and labels, and deployed the Process version containing the developer’s capability fix. This kept responsibility clear: the developer corrected application behavior; I aligned its deployment with the receiving service. Successful startup established readiness for integration testing, which still required a real recognition request.

Preserving the effective release configuration

On 22 October, Helm reinstallation lost runtime overrides required by Task and Process. A replacement RabbitMQ volume also started without the application’s users and messaging topology. Running infrastructure pods therefore coexisted with a broken processing workflow.

I coordinated a read-only forensic investigation and recovery with Codex. Comparing effective values, manifests and logs revealed missing OIDC settings and incompatible S3, RabbitMQ and OCR transport defaults. Recovery restored application configuration, broker users, permissions, exchanges, queues and bindings. A subsequent synthetic task completed successfully through the restored chain. The older broker volume remained available, but complete recovery of its historical messages was not established.

Following telemetry to the point of use

Telemetry required checks across several boundaries. After developers exposed the necessary metrics endpoints, I connected Service labels, named ports and ServiceMonitor selectors, then checked Prometheus targets and queries. RabbitMQ’s existing metrics plugin supported both general and detailed queue scraping. Separately, Loki/Alloy delivered synthetic probes and an allowlist of infrastructure messages; application logs remained outside that collection scope.

I also investigated a portal counter that showed zero despite a recorded request. The observed Prometheus series stayed at one throughout its sampled history, so the portal’s increase() query correctly returned zero for that window. This localized an accounting limitation rather than a connectivity failure; it did not justify substituting an arbitrary counter calculation.

Validation and Results

Historical API tests recorded successful recognition before the incident and after recovery. In the latter run, task creation returned HTTP 201, processing reached a complete terminal outcome, and result retrieval returned HTTP 200 with all five public field values matching the synthetic fixture. Separate deployment records also describe a successful browser-based demo scenario.

Infrastructure checks covered RabbitMQ publish/get, basic S3 operations and an SSE-S3 object round trip. Following Grafana’s out-of-memory failure, I raised its memory limit from 512 MiB to 1 GiB and checked health and node metrics. Later checks showed successful Task, Process and RabbitMQ scraping. These results establish specific operational properties, rather than blanket acceptance of every subsystem.

The team’s OCR benchmark used 120 positive and 24 negative synthetic documents. Exact-match accuracy was 99.17% for the insurance number, 90.83% for the full name and 96.67% for birth date. The accuracy criterion passed, although the benchmark’s separate latency guard failed. A 20-task smoke test completed on a separate kind environment; it did not establish the full 5,000-document workload target.

SIRD administrator portal showing two active test accounts and the selected administrator account
Administrator portal for users and access management with test accounts on example.test.

Limits and Lessons

The environment remained an integration prototype. One control-plane node, local worker storage and single-instance data services limited availability. Backups were created, but complete restore and failover were not demonstrated. Automatic image publishing did not extend to a verified Kubernetes delivery pipeline, and differences between Git, Helm and runtime configuration remained. Full tracing, integrated Grafana panels and overall system acceptance were also unconfirmed.

The practical lesson was to treat effective configuration as part of the release and verify recovery through application behavior. A stateful service can start with the wrong state, and a metrics library can exist without producing useful operational data. The same discipline applies to both: follow the chain to its observable result, then state precisely what the check established.