Skip to content

Benchmarks

Metriq-Gym provides a comprehensive suite of quantum benchmarks to characterize and compare quantum hardware performance.

Running Benchmarks

Single Benchmark

mgym job dispatch metriq_gym/schemas/examples/quantum_volume.example.json \
    --provider local --device aer_simulator

Poll for Results

mgym job poll <JOB_ID>

Configuration

All benchmarks use JSON configuration files:

  • Schemas: metriq_gym/schemas/*.schema.json - Define parameters, types, and allowed values
  • Examples: metriq_gym/schemas/examples/*.example.json - Generic, editable configurations for learning and ad hoc runs
  • Canonical score suite: metriq_gym/suites/metriq_score_1_0.json - Exact configurations and scale datapoints for Metriq Score 1.0

Example files are not canonical Metriq Score configurations.

Available Benchmarks

Benchmark Description
Mirror Circuits Tests state fidelity via forward/reverse Clifford layers
EPLG Error per layered gate across qubit chains
BSEQ Bell state effective qubits via CHSH violation
WIT Wormhole-inspired teleportation protocol
LR-QAOA Linear-ramp QAOA for Max-Cut optimization
QML Kernel Quantum machine learning kernel accuracy
QED-C Benchmarks Application-oriented benchmarks (BV, QFT, etc.)
CLOPS Circuit layer operations per second to measure device speed
GHZ (Experimental) GHZ state fidelity, at a fixed size or by searching for the largest one the device can build

mirror_circuits

Mirror Circuits benchmark implementation.

Summary

Generates randomly parameterized mirror circuits that apply layers of Clifford gates, add a middle Pauli layer, and then revert the forward layers to test how well a device preserves state fidelity across the forward and reverse halves of the circuit.

Result interpretation

Polling yields MirrorCircuitsResult with: - success_probability: fraction of runs matching the expected bitstring. - polarization: rescales success_probability to remove the uniform-random baseline; higher implies better performance. - binary_success: boolean indicating whether polarization exceeded 1/e.

References

eplg

EPLG (Error Per Layered Gate) benchmark implementation.

Summary

Measures layer fidelity across qubit chains using randomized benchmarking techniques. Computes EPLG scores at various chain lengths to characterize two-qubit gate performance across the device.

Result interpretation

Polling returns EPLGResult with: - chain_lengths: list of qubit chain lengths tested - chain_eplgs: EPLG values at each chain length - eplg_10/20/50/100: EPLG at standard reference points - score: average EPLG across reference points (lower is better)

References

bseq

BSEQ (Bell state effective qubits) benchmark implementation.

Summary

Evaluates how well a device generates Bell pairs that violate the CHSH inequality across its connectivity graph. Circuits are built per colouring of the topology and executed in four measurement bases to detect correlations.

Connectivity graph

The benchmark uses the device's native connectivity graph to determine which qubit pairs can be coupled. For superconducting devices (e.g., IBM), this reflects the physical coupling map with sparse connectivity. For trapped-ion devices (e.g., IonQ, Quantinuum) and simulators, all-to-all connectivity is assumed (complete graph). The graph structure affects edge coloring: complete graphs with n qubits require n-1 colors (optimal), while sparse topologies typically require fewer colors but test fewer qubit pairs.

Result interpretation

Polling returns BSEQResult with: - largest_connected_size: size of the biggest connected subgraph of qubit pairs that violated CHSH (> 2). Higher means entanglement spans more of the device. - fraction_connected: largest_connected_size normalised by the discovered qubit count, making it easier to compare devices of different sizes.

References

wit

WIT (wormhole-inspired teleportation) benchmark implementation.

Summary

Runs a six- or seven-qubit teleportation-inspired circuit that mimics the protocol from Shapoval et al. (2023) and reports a Pauli-Z expectation value with binomial uncertainty.

Result interpretation

Polling returns WITResult.expectation_value as a BenchmarkScore: - value: estimated Pauli-Z expectation (ideal teleportation trends toward +1). - uncertainty: binomial standard deviation computed from the observed counts. Compare value versus uncertainty to decide whether more shots are required or if noise is degrading the teleportation fidelity.

References

lr_qaoa

Linear Ramp QAOA (LR-QAOA) benchmark implementation.

Summary

Solves weighted Max-Cut instances with a linear-ramp parameter schedule and compares results against classical optima to estimate approximation ratios and optimal sampling probabilities.

For a deeper dive into results across various graph types, see the authors' benchmarking dashboard.

Result interpretation

Polling returns LinearRampQAOAResult with metrics including: - approx_ratio_mean / stddev: how close average costs are to the optimum. - optimal_probability_mean / stddev: frequency of sampling an optimal bitstring. - confidence_pass: boolean indicating whether results meet the configured confidence. Higher approximation ratios and optimal probabilities reflect better QAOA performance.

References
  • Montanez-Barrera et al., "Evaluating the performance of quantum processing units at large width and depth", arXiv:2502.06471.

qml_kernel

Quantum Machine Learning Kernel benchmark implementation.

Summary

Constructs a ZZ feature map kernel, computes the inner-product circuit, and measures the probability of returning to the all-zero state as a proxy for kernel quality.

Result interpretation

Polling returns QMLKernelResult.accuracy_score as a BenchmarkScore where: - value: fraction of shots measuring the expected all-zero bitstring. - uncertainty: binomial standard deviation from the sample counts. Higher accuracy suggests better kernel reproducibility on the selected hardware.

References

qedc_benchmarks

QED-C application-oriented benchmark wrapper.

Summary

Provides a generic dispatch/poll pipeline around the QED-C benchmark suite (Bernstein- Vazirani, Phase Estimation, Hidden Shift, Quantum Fourier Transform) via the QC-App- Oriented-Benchmarks submodule.

Result interpretation

Polling returns QEDCResult.circuit_metrics, a nested dictionary keyed by qubit count and circuit identifier, populated with the fidelity or related metrics computed by the QED-C analyser. Inspect the per-circuit entries to understand performance trends.

References

clops

CLOPS (Circuit Layer Operations Per Second) benchmark implementation.

Summary

Measures the throughput of a quantum system by timing the execution of parameterized quantum volume-style circuits. CLOPS captures end-to-end performance including compilation, communication, and execution overhead.

Result interpretation

Polling returns ClopsResult with: - clops_score: circuit layer operations per second (higher is better) as measured from time of submission to job completion as reported by the cloud platform. - steady_state_clops: circuit layer operations per second (higher is better), ignorning the first execution span to exclude pipeline startup costs, measuring sustained throughput. Only works for cloud platforms that provide execution span metadata (currently only IBM Runtime), and is None when spans are unavailable or there are fewer than two spans.

This metric reflects real-world workload performance rather than isolated gate speeds.

References

ghz

GHZ state preparation and verification benchmark.

Summary

Constructs a GHZ state on the device using a binary dissemination tree over the device's connectivity graph (logarithmic depth where connectivity allows), then estimates fidelity via one of four verification methods: direct fidelity estimation (DFE, uniform sampling of the GHZ stabilizer group), the two-setting Z/X lower bound, parity oscillation curve fitting, or compressed sensing (parity samples at random phases, sparse spectrum recovery by L1 minimization).

Experimental: available for standalone evaluation. The methodology and defaults may change before inclusion in the benchmark suite.

Two modes:

  • Fixed size (num_qubits): verify a GHZ state of one requested size.
  • Size search (size_search: true): build GHZ states at halving sizes from the device size down to min_qubits (e.g. 156, 78, 39, 19, 9, 5), all dispatched in a single job, and report the largest size whose fidelity clears fidelity_threshold by confidence_sigma standard deviations. Defaults to compressed sensing, which resolves the frequency spectrum using logarithmically many phase samples.
Result interpretation
  • population: probability of measuring all-zero or all-one bitstrings (Z basis).
  • coherence: off-diagonal element. DFE averages uniformly sampled off-diagonal stabilizers X^(x)N·Z_s, an unbiased estimate of 2·Re(rho[0...0, 1...1]) with pure finite-sampling error (the diagonal half of the stabilizer group averages exactly to the population, so the Z circuit covers it with no sampling). two_setting_bound takes the global X parity and subtracts the population outside the GHZ subspace, bounding the contribution of non-GHZ complementary pairs: X^(x)N connects every bitstring to its complement and not only 0...0 to 1...1; without that subtraction a product state such as |+>^(x)N reports coherence 1. The oscillation methods instead isolate the N-qubit Fourier component, by curve fit or by LASSO over random-phase parity samples.
  • fidelity: (population + coherence) / 2. DFE targets GHZ+; oscillation methods maximize over the relative phase, and two_setting_bound bounds that phase-optimized fidelity. These targets differ for coherent phase errors. Compare runs using the same method.
  • recovered_frequency (compressed sensing only): dominant frequency of the parity signal. This is a diagnostic of the spectrum, not by itself an entanglement certificate; use fidelity to assess the prepared state.
  • phase_offset (compressed sensing only): phase θ of the off-diagonal element, i.e. the state (|0...0> + e^{iθ}|1...1>)/√2. Zero for an ideal GHZ; a nonzero value exposes coherent errors (residual Z rotations, frame misalignment) that leave the coherence magnitude unchanged. None when the coherence is at the noise floor, where the angle is undefined.
  • largest_passing_size / device_fraction (size search only): the largest tested size that passes the fidelity margin, and that size divided by the device size. Per-size fidelities are reported in search_sizes / search_fidelities, with uncertainties in search_uncertainties, so no resolution is lost to the threshold.
  • score: fidelity in fixed-size mode; largest_passing_size (a qubit count) in size-search mode. Scores from these modes are not directly comparable.
Size search and uncertainty

A size passes when fidelity - confidence_sigma * uncertainty > fidelity_threshold. This margin uses estimated standard errors, not a rigorous confidence guarantee. Size search reports the largest tested passing size.

Compressed sensing assumes a sparse spectrum. Its uncertainty propagates shot noise through the recovered model. If the recovered spectrum needs more phase samples, increase num_phases or use uniform parity oscillation.

References
  • Cruz et al., "Efficient quantum algorithms for GHZ and W states, and implementation on the IBM quantum computer", Adv. Quantum Technol. 2, 1900015 (2019). arXiv:1807.05572.
  • Flammia & Liu, "Direct Fidelity Estimation from Few Pauli Measurements", Phys. Rev. Lett. 106, 230501 (2011). arXiv:1104.4695.
  • Labib et al., "Compressed Sensing for Efficient Fidelity Estimation of GHZ States" (2026). arXiv:2604.27824.

Adding Custom Benchmarks

See Adding New Benchmarks to contribute new benchmarks.