Home / Reproducibility

Computational Reproducibility & Standards

At AIAC, research credibility rests on uncompromising computational rigor. Every technical claim, throughput benchmark, and model checkpoint published by our squads must satisfy our public four-point reproducibility protocol.

01

Deterministic Seeds & Execution

All training and evaluation pipelines must explicitly fix random number generator states across Python, NumPy, PyTorch, and CUDA backends (torch.use_deterministic_algorithms(True)).

02

Hermetic Environments

Experiments must provide containerized Dockerfiles or pinned requirements.txt with exact SHA-256 hashes to eliminate dependency drift across host systems.

03

Disclosed Compute & Telemetry

Latency and throughput metrics must disclose physical GPU model, driver version, ambient temperature, batch size, and thermal throttling state. Simulated numbers are strictly barred.

04

Open Artifacts & Lineage

Code, model checkpoints, preprocessing scripts, and evaluation logs must be versioned in public git repositories under OSI-approved licenses (MIT, Apache-2.0, or CC BY 4.0).

Governance Process

The Four-Council Review Gate

Before any technical report or preprint is accepted for release in the AIAC series, it must pass mandatory sign-offs from our independent student councils:

Council 1

ML Rigor & Benchmarking

Verifies evaluation datasets, baseline comparability, statistical significance bounds, and ablation studies.

  • ✓ Minimum 3 random seeds evaluated
  • ✓ Confidence intervals or standard deviations reported
  • ✓ Train/test data contamination check verified
Council 2

Systems & Infrastructure

Audits hardware reproducibility, memory allocation peaks, and container runner performance.

  • ✓ Automated GitHub Actions runner verification
  • ✓ Peak GPU VRAM and execution wall-clock time logged
  • ✓ CUDA kernel compatibility matrix confirmed
Council 3

Research Ethics & Lineage

Inspects dataset copyright licenses, anonymization, consent, and MITRE ATLAS security risk profiles.

  • ✓ Training data commercial/research licensing vetted
  • ✓ PII redaction and synthetic generation documented
  • ✓ Red-teaming and safety guardrails evaluated
Council 4

Documentation & Reproducibility

Ensures external researchers can clone and reproduce results with a single shell script.

  • ✓ Verified README.md with reproduction steps
  • ✓ Input data download scripts provided
  • ✓ BibTeX citation metadata verified
Checklist

Repository Submission Requirements

Every project repository maintained by AIAC members must adhere to this standard layout:

aiac-project-repo/
├── README.md               # Overview, setup guide, and replication instructions
├── LICENSE                 # Apache-2.0 or MIT
├── CITATION.cff            # Standard citation metadata
├── environment.yml         # Conda environment definition with pinned versions
├── docker/
│   └── Dockerfile          # Reproducible container recipe
├── data/
│   └── download_data.sh    # Public dataset fetch script (never commit large raw data)
├── src/                    # Well-typed, modular source code
└── tests/
    └── test_reproducibility.py # Automated seed and numeric tolerance checks