FHIR Conformance And Benchmarks

sct keeps four different checks separate:

1. Repository regression checks: do the documented GET and POST operation forms still return the expected results over the committed synthetic fixture?
2. FHIR structural validation: are representative responses valid base-R4 resources according to the independent HL7 FHIR Validator?
3. Official terminology interoperability tests: does the server pass the selected released suites from the HL7 FHIR Terminology Ecosystem?
4. Performance benchmarks: once the relevant correctness profile passes, how fast is it compared with local SQLite and other terminology servers?

The distinction matters. A fast server that returns the wrong $expand result is not useful, a structurally valid response can still contain the wrong concept set, and a repository-owned suite is not independent conformance evidence.

Evidence, Not Certification

The current CI runner is a repository-specific smoke/regression profile aligned with these FHIR R4 terminology operations:

• /metadata
• CodeSystem/$lookup
• CodeSystem/$validate-code
• CodeSystem/$subsumes
• ValueSet/$expand
• ValueSet/$validate-code
• ConceptMap/$translate

It proves the local contract remains stable; it is not an official HL7 suite and not an HL7 certification badge. The independent layers are:

The reproducible baseline pins FHIR Validator 6.10.4 and hl7.fhir.uv.tx-ecosystem#1.9.3; never use the mutable current package in CI. The published test registry currently uses an unversioned page while the package coordinate and hash pin the executable corpus. The separate approval-page source above is explicitly an unreleased draft and must not be presented as a current published HL7 requirement.

The independent structural gate runs separately from the repository regression suite. It downloads the Validator and its complete package graph by immutable coordinate and SHA-256, extracts the packages into an isolated cache, disables HTTP access during validation, and checks eight representative responses. It fails on structural errors and on any change to the reviewed warning baseline:

cargo build --bin sct
benchmarks/fhir-validator-ci.sh

The complete local evidence is written to target/fhir-validator/. CI retains the same directory as the fhir-validator-evidence artefact on success or failure. A passing structural gate does not assert that an expansion contains the correct concepts; the repository semantic suite remains a separate gate.

The official command is:

java -jar validator_cli-6.10.4.jar txTests \
  -tx http://127.0.0.1:8080/fhir \
  -test-version 1.9.3 \
  -mode general \
  -output ./tx-results \
  -fhir-settings ./fhir-settings.json

The Validator blocks plain HTTP and private-network targets by default. For this loopback URL, use a narrowly scoped settings file rather than disabling checks globally:

{
  "servers": [{
    "url": "http://127.0.0.1:8080/fhir",
    "type": "fhir",
    "authenticationType": "none",
    "allowHttp": true,
    "allowPrivateNetwork": true
  }]
}

Retain the complete runner output directory for every evidenced R4 run, including test.log, test-results.json, report.json, actual/, expected/ and conversions/. The conversion output distinguishes server failures from failures while converting the test corpus between FHIR versions.

The first baseline is an interoperability diagnosis, not a semantic score. The general run executed 597 tests: 593 operation requests were refused at the standard POST Parameters boundary, two $batch-validate requests reached unsupported routes, and two metadata comparisons failed. The sct-ecl suite's 103 requests were also refused before ECL evaluation because they carry inline ValueSet bodies. They target a licensed SNOMED test subontology that is not the committed 22-concept synthetic fixture, so those results must not be reported as "0% ECL conformance".

That baseline predates the shipped standard POST boundary and supported inline SNOMED ValueSet subset. It remains historical evidence for why the boundary was built, not a current interoperability result; the official suite must be rerun against the exact licensed test subontology before reporting any semantic score.

The local runner remains useful because benchmark evidence needs a stable, reproducible workload on developer machines, deployments and CI. The detailed pins, licensing boundary, staged external-gate plan and exact claim language are in the R17 conformance evidence record.

Run Conformance First

Start a terminology server:

sct serve --db snomed.db --host 127.0.0.1 --port 8080 --fhir-base /fhir

Then run:

benchmarks/conformance.sh --server http://localhost:8080/fhir

The runner checks:

Write machine-readable output for CI or later reporting:

benchmarks/conformance.sh \
  --server http://localhost:8080/fhir \
  --output reports/sct-conformance.jsonl

If a target server does not advertise ConceptMap/$translate, the translate checks are skipped by default. Use --strict when comparing only servers that are expected to support the full sct surface.

Then Benchmark

After conformance passes:

benchmarks/bench.sh \
  --db snomed.db \
  --server http://localhost:8080/fhir \
  --runs 20 \
  --warmup 5 \
  --write-benchmarks

The existing benchmark covers:

• concept lookup
• free-text search
• direct children
• ancestor traversal
• subsumption
• bulk lookup

The benchmark reports wall-clock medians and standard deviation. Local SQLite timings include process startup. FHIR timings include HTTP overhead.

Compare Against Snowstorm Or Ontoserver

To make a credible public claim:

1. Load the same SNOMED CT release into every server.
2. Run each server on the same hardware class.
3. Warm the filesystem, JVM, Elasticsearch/Lucene and SQLite caches.
4. Run benchmarks/conformance.sh first.
5. Only publish benchmark results for servers that pass the relevant conformance profile.
6. Record exact versions, heap settings, database size, disk type, CPU, RAM, operating system and release package.

Example:

# sct
benchmarks/conformance.sh --server http://localhost:8080/fhir
benchmarks/bench.sh --db snomed.db --server http://localhost:8080/fhir --runs 20 --warmup 5

# Snowstorm Lite
benchmarks/conformance.sh --server http://localhost:8081/fhir
benchmarks/bench.sh --db snomed.db --server http://localhost:8081/fhir --runs 20 --warmup 5

# Ontoserver or another FHIR terminology server
benchmarks/conformance.sh --server http://localhost:8082/fhir
benchmarks/bench.sh --db snomed.db --server http://localhost:8082/fhir --runs 20 --warmup 5

The conformance checks are deliberately fixture based so the same request matrix can be used across implementations. The benchmark fixtures should be expanded over time with more high-fanout hierarchies, deep concepts, inactive concepts, refsets and cross-map workloads.

Public Methodology

When publishing results, include:

• exact sct version and git commit
• SNOMED CT edition and release date
• whether refsets and crossmaps were loaded
• server base URL path, for example /fhir
• hardware and operating system
• Docker image tags or binary versions for comparator servers
• cache state: cold start, warm cache, or both
• number of runs, warmup runs and timeout
• full conformance pass/fail output
• raw benchmark output

The headline number should be scoped. For example, "sct serve is faster for these read-only SNOMED CT terminology operations on this release and hardware" is defensible. A general claim that one terminology server is universally faster than another is not.