DATA -> ANALYSIS -> REPRODUCIBILITY
ALL YEARS | PAGE 81B - PROVENANCE -> QUALITY -> EFFECT SIZE -> UNCERTAINTY -> REPRODUCIBILITY
C. DATA-INTEGRITY + REPRODUCIBILITY CONTROL PANEL
| Element | Minimum standard | Failure signal |
|---|---|---|
| Data dictionary | Define variable meaning, units, coding, allowable values, source and derivation. | Same variable interpreted differently across analysts. |
| Provenance | Preserve raw/source data; document extraction date, query and transformations. | No path from final table back to source. |
| Eligibility/exclusions | Operationalize criteria before outcome-driven filtering; log every removal. | Exclusions appear only after results look inconvenient. |
| Missing data | Quantify by variable/group/time; examine plausibility of missingness assumptions. | Complete-case analysis used automatically with no rationale. |
| Code + versions | Script transformations/analysis when possible; use stable filenames/version control; seed stochastic procedures. | Manual spreadsheet edits cannot be reproduced. |
| Quality checks | Range, logic, duplicates, time order, impossible values, cross-source reconciliation. | Analysis discovers basic data errors late. |
| Privacy + access | Use approved storage, minimum necessary access, de-identification/data-use controls as required. | Data copied to convenient but unapproved tools. |
| Sharing plan | For applicable funded work, plan what data/metadata/code can be shared, where, when and with what limitations. | Sharing is considered only after the project ends. |
D. INTERPRETATION: WHAT THE NUMBER DOES - AND DOES NOT - TELL YOU
| Question | Best evidence to report | Common overclaim |
|---|---|---|
| How large is the effect/association? | Effect estimate + units; absolute and relative measures when useful. | "Significant" used instead of magnitude. |
| How precise is it? | Confidence/credible interval and sample information. | Point estimate treated as exact. |
| Could bias/confounding explain it? | Design limits, measured/unmeasured confounders, selection/information bias. | Adjusted model presented as proof of causality. |
| Were many analyses tried? | Pre-specified vs exploratory labels; multiplicity strategy when relevant. | One favorable analysis reported without the search space. |
| Does it matter clinically? | Baseline risk, absolute change, harms, feasibility, patient relevance. | Small p-value equated with important care difference. |
| Will it generalize? | Population, setting, exclusions, context and transportability limits. | Single-site result assumed universal. |
| Is it robust? | Sensitivity/alternative specifications and assumption checks. | One model treated as the only possible truth. |
D. INTERPRETATION: WHAT THE NUMBER DOES - AND DOES NOT - TELL YOU — 2026 NIH DATA-MANAGEMENT NOTE
- The NIH 2023 Data Management and Sharing Policy remains in force for covered NIH-funded/conducted research that generates scientific data. - NIH introduced an updated DMS Plan format in 2026; the current NIH page states that the 2026 DMS Plan format is now required for competing and non-competing awards. Verify the exact sponsor/funding requirements for your project.
TOP 10 RESEARCH + QI CONCEPT - 4: REPRODUCIBILITY BEGINS WITH PROVENANCE - NOT WITH A FINAL CODE ZIP FILE
| DEEP EXPLANATION | WHY STUDENTS MISS IT | EASIEST CLUE |
|---|---|---|
| A reproducible project preserves the chain from source -> cleaning -> derivation -> analysis -> figure/table. | Reproducibility is often imagined as a last-step repository upload rather than a property of the workflow | Ask whether another analyst can trace every reported number back to its source and rerun the transformation |
| Code is valuable, but code applied to an undocumented dataset or manual edits does not reconstruct the study. | from day one. | without guessing. |
| TOP 10 RESEARCH + QI CONCEPT - 5: A P-VALUE IS NOT EFFECT SIZE, CLINICAL IMPORTANCE OR CERTAINTY | ||
| DEEP EXPLANATION | WHY STUDENTS MISS IT | EASIEST CLUE |
| Statistical evidence must be interpreted with magnitude, precision, design quality, assumptions, multiplicity and | Binary "significant/not significant" language is easy to communicate and is often reinforced by software | Lead with the estimate and interval. Then ask: "How big, how precise, how biased, and how meaningful?" |
| clinical context. A tiny effect can be statistically detectable; an important effect can remain imprecise in | defaults. | |
| a small study. | ||
| TOP 10 RESEARCH + QI CONCEPT - 6: MISSINGNESS AND EXCLUSIONS ARE ANALYTIC ASSUMPTIONS - NOT HOUSEKEEPING | ||
| DEEP EXPLANATION | WHY STUDENTS MISS IT | EASIEST CLUE |
| Who disappears from the dataset, when, and why can change the estimate. Missing-data handling and | Rows are easily dropped by software or cleaning rules, making consequential decisions look like routine | Create an inclusion flow and missingness table before celebrating the final model. If exclusions change the |
| exclusions should be described, justified and tested when they could materially alter the conclusion. | preprocessing. | result, show it. |
TOP 10 RESEARCH + QI CONCEPT - 4: REPRODUCIBILITY BEGINS WITH PROVENANCE - NOT WITH A FINAL CODE ZIP FILE
| NIH DMS policy | 2026 DMS Plan format |
|---|---|
| 2026/27 research + scholarship expanded edition - use the approved institutional data environment and current sponsor/data-sharing requirements |