Ask about absent outcomes before comparing means
Imagine receiving two folders of outcome records and being asked whether their averages differ. Before calculating anything, make an inventory of the people those folders are supposed to represent. A folder may contain a complete series for one person, an unfinished series for another, and no usable final measurement for a third. The arithmetic describes whichever records enter it. Establishing that membership is therefore part of defining the result, rather than an administrative detail to settle afterward.
Consider a hypothetical record with an initial measurement, a category label and an empty final cell. The label cannot fill that cell. The person's outcome might have increased, decreased or remained unchanged; each possibility is compatible with the information specified. Nor does a blank become a measured zero simply because a spreadsheet needs a number. Supplying zero would add an assumption to the record, and the resulting average would depend on that assumption.
A useful first exercise is to write the intended question beside the actual calculation. Does the question concern everyone initially listed, everyone measured at the end, or a population defined by another condition? These descriptions can designate different collections of people even when the outcome has the same name. Mark every step where the list changes. An intelligible comparison needs both a rule for selecting records and a rule for summarizing them. Knowing the second without the first leaves the meaning of the estimate unfinished.
Make the selection rule explicit
Post hoc NVD groups were not randomized; one event qualified. Endpoint weight was required; data after treatment discontinuation were excluded.[1]
Reading the outcome comparison
| NVD | Change (%) |
|---|---|
| Yes | −22.8 |
| No | −22.2 |
To inspect a selection rule, construct an explicitly hypothetical flow with two separate decisions. The first assigns a category; the second asks whether an outcome can be used. Give each fictional person a row with a category field and an availability field. Someone can have an unambiguous category but no usable outcome. Someone else can have a measurement while lacking the information needed for classification. Neither field logically supplies the other.
Now imagine that a third feature affects both classification and whether a record becomes available. For example, in this invented exercise, people with more frequent record submissions have more opportunities to receive a label and more opportunities to supply the final measurement. Restricting the calculation to complete rows would then also select on submission behavior. This possibility does not establish bias in any particular comparison. It shows what an audit would need to examine before treating completeness as an irrelevant detail.
One test is to reverse the order of the two filters while retaining each person's identifier. If the written rules define fixed conditions on unchanged records, their intersection should contain the same identifiers either way. A difference would reveal an implementation problem, a changing definition or a dependency that needs explanation. Matching results would verify that limited property, not prove that selection is harmless. When interpreting the adjacent figure, use its caption to identify the quantity; keep the separate question of record eligibility open until its rule is understood.
Identify the population behind the question
SURMOUNT-1 randomized adults without diabetes, with BMI ≥30 or ≥27 plus a weight-related condition, to weekly subcutaneous tirzepatide or placebo for 72 weeks.[2]
Eligibility describes entry into a research question. Generalization asks whether an answer can travel beyond that entry rule. These are different tasks. Imagine a reader who shares one listed characteristic with an eligible population. That match alone says nothing about the remaining characteristics, how outcomes would be measured for that reader, or whether the setting would reproduce the conditions needed for comparison. A resemblance is a starting point for an argument, not the argument's conclusion.
A practical transport exercise would first name the proposed destination population. Next, specify which differences could matter for the particular outcome and which could matter for obtaining a measurement. These lists need not be identical. A characteristic could affect record completion without changing the outcome itself. Conversely, a characteristic could alter the outcome while leaving measurement equally available. Each possibility calls for different supporting information.
Keep a further distinction between entry eligibility and eligibility for a later calculation. In a hypothetical protocol, a person could satisfy every entry rule but lack the information demanded by a particular summary. Removing that row changes the analyzed collection without retroactively changing who qualified to enter. To generalize responsibly, the reader would need a defensible connection from the observed collection to the intended population. Merely repeating an eligibility label would not supply that connection, and a larger number of complete records would not by itself establish it.
Membership can depend on the course of follow-up
Consider an invented timeline with an outcome recorded early and a classification event recorded later. At the early date, the later label does not yet exist. An analyst can sort the earlier outcomes using that eventual label, but doing so asks a retrospective question: how did earlier records differ among people who would later receive different labels? It does not recreate an experiment in which the label was assigned before the early outcome.
Several explanations could fit such a pattern. A preexisting characteristic might influence both the early outcome and the later label. The early outcome might influence subsequent behavior that changes whether a label is recorded. Or the observed pattern might be compatible with ordinary variation. These are competing logical accounts of the fictional timeline, not findings about the study. Naming one account does not identify which account generated the records.
A useful timing test moves the classification cutoff while leaving the outcome date fixed. In the imagined dataset, ask whether membership changes when labels recorded after the outcome are disallowed. If it changes, the analyst has learned that the original grouping used information from beyond the outcome date. That finding would require a different interpretation from a comparison based only on information available beforehand. It would not alone identify a causal effect.
An alternative prospective question could define groups using information already available at a specified starting point and then observe later outcomes. Even that ordering would need an argument about comparability. Temporal order excludes some explanations, but it does not make the remaining groups interchangeable. A sound analysis states which information was available when, then limits its claim to what that ordering can support.
What question can the available record answer
Two questions can use the same outcome scale while asking about different things. One might describe the records that satisfy an observed-data rule. Another might ask what would happen under a policy of assigning a particular intervention, allowing subsequent decisions to unfold. The first question concerns a defined collection of observations. The second requires an account of outcomes under the policy, including situations in which the desired observations are unavailable.
In a hypothetical policy comparison, write down how to handle later decisions before choosing a calculation. Would the question follow outcomes regardless of those decisions, describe a hypothetical world without them, or stop counting at a defined event? Each choice changes what is being asked. It is therefore possible for two correctly implemented analyses to disagree because they address different targets. Giving one estimate a more expansive label cannot resolve that difference.
To connect an observed description to a broader target, identify the information that would complete the argument. It might include outcomes collected under additional circumstances, reasons a measurement was not obtained, or evidence supporting a proposed relationship between observed and unobserved values. These are requests for evidence, not permission to substitute guesses silently. If the necessary relationship remains an assumption, conclusions should remain conditional on it.
Read the companion event analysis with this same question in mind: what exactly would have to change in the records or the target for the conclusion to change? That comparison of reading methods can clarify an argument without treating separate analyses as interchangeable evidence. The name of a statistic is less informative than the population, outcome and decision rule it is meant to summarize.
Explore assumptions without inventing outcomes
The following teaching example is entirely invented, uses arbitrary outcome units, and does not reconstruct study data. Suppose four records are intended for a calculation. Three contain the values two, four and six; the fourth value is unknown. The observed values sum to twelve. If the unknown value is called x, the average across all four intended records would be twelve plus x, divided by four.
Assume, solely for this exercise, that x lies between zero and eight. The possible average then runs from three to five. Suppose a hypothetical comparison value is four. An x below four places the average below that comparison; an x above four places it above. This identifies the assumption on which the direction of the comparison turns. It does not estimate x or assign probabilities to different values of x.
The range itself now deserves scrutiny. Why exclude values outside it? A narrower range would produce a narrower answer, but only because the analyst imposed a stronger restriction. More decimal places would improve the presentation of the arithmetic without providing evidence for that restriction. If a conclusion survives a wide range of credible assumptions, it has a different basis from one that survives only a narrow preferred range.
A useful sensitivity report would therefore show both the decision threshold and the reasoning behind the allowed range. It would separate a mathematical statement, such as which values reverse a comparison, from an evidential statement about which values deserve consideration. The missing outcome would remain missing throughout. This exercise demonstrates how to expose dependence on an assumption while resisting the temptation to turn the assumed value into an observation.
Ask who made the analytical decisions
Lilly funded this analysis; an employee-shareholder planned and conducted statistical analyses.[1]
An analytical role identifies responsibility; it does not determine whether a calculation is correct. A stronger audit would ask whether another analyst could reproduce the result from a defined version of the records using the same instructions. Before comparing answers, both analysts would need to agree on the intended target, the inclusion rule, the handling of ambiguous entries and the precision at which results are reported.
Imagine commissioning such a check. The first deliverable could be a record of the decisions taken before the final summary is produced: which version was used, which rows qualified, and how conflicting entries were resolved. If access were permitted, the second analyst could compare intermediate outputs rather than only the final number. Agreement on a final average can conceal different underlying row selections whose differences happen to cancel.
A separate exercise would deliberately change one analytical choice. For example, the auditor might alter a hypothetical rule for an ambiguous record and examine which conclusions move. This tests sensitivity, whereas reproducing the original choices tests implementation. Combining the two exercises into a single pass or fail label would lose that distinction.
Neither exercise requires assuming misconduct. Both require an answer that can be challenged on specific grounds. If a discrepancy appears, identify the earliest step at which the outputs diverge. That makes the disagreement investigable: it could concern the data version, the interpretation of a rule or the calculation itself. Each would demand a different correction.
Means leave individual outcomes unresolved
Imagine one fictional population in which every outcome equals ten arbitrary units and another split evenly between zero and twenty. Both averages equal ten, yet the individual experiences represented by the records are very different. A mean alone cannot tell these distributions apart. Additional information about spread and shape would be needed before using either description to discuss the range of individual outcomes.
There is also a distinction between uncertainty about an estimated average and variation among individuals. Repeatedly estimating an average from hypothetical samples could produce closely clustered estimates while the individual outcomes within those samples remain widely dispersed. A precise estimate of a population summary does not logically collapse that individual spread. Asking how confidently a mean is estimated therefore answers a different question from asking how accurately a person's outcome can be predicted.
To evaluate a prediction, specify what it promises before comparing it with new observations. A predicted value could be assessed by how far it falls from an eventual measurement. A predicted range could be assessed by whether outcomes fall inside it as often as promised, as well as by how wide the range is. Those checks should use observations that did not already determine the prediction rule.
Use the existing outcomes guide to practice separating those promises in written claims. Identify whether a sentence describes an average, a probability, a range or an individual expectation. The evidence needed to evaluate the sentence follows from that choice; the presence of a group summary cannot settle all four.
Design a more informative next record
A future observation plan could keep separate fields for when an event occurred, when it was recorded, whether the intended measurement was obtained and why a value was absent. That separation would allow an auditor to distinguish a late entry from a late event, or a missing value from a measured result. A reason for absence would inform an explanation without revealing the unmeasured outcome.
One falsifiable question for an explicitly hypothetical project is whether an additional data-collection contact changes completion. Assign otherwise comparable collection schedules in advance, define completion before examining results, and compare the proportion of intended measurements obtained under each schedule. This would evaluate a collection procedure. It would not establish that the newly obtained values resemble the values still absent, nor answer a clinical treatment question.
Plan the reading record with equal care. Consult the molecule profile to identify the subject, then read a provider document separately when examining dated service documentation. A provider description would need its own evidence for any service claim. Keep the analytical question attached to its observations and assumptions so that a change of document cannot silently change what the evidence is being asked to establish.
Keep the clinical boundary visible
Zepbound delays gastric emptying and is not recommended in severe gastroparesis. Contraindications include personal or family medullary thyroid carcinoma history, multiple endocrine neoplasia type 2, and serious tirzepatide or ingredient hypersensitivity. Persistent or severe abdominal pain warrants prompt assessment; gastrointestinal dehydration can injure kidneys.[3]
Compounded drugs lack FDA approval.[4]
An interpretation of group data cannot decide a person's prescribing question. The relevant boundary is the difference between explaining a calculation and assessing an individual clinical situation. Likewise, a provider document cannot inherit a research conclusion merely by discussing the same subject. Keep each claim attached to the particular evidence and purpose it can support.
Reader questions
Does a small distance between two means establish equivalence?
No. First define how large a difference would matter for the intended question. Then ask whether an appropriate uncertainty assessment can distinguish differences inside that boundary from differences outside it. Choosing the boundary after seeing a result would change the evidential test. Numerical proximity alone supplies neither the criterion nor that assessment.
Can an absent measurement be treated as no change?
In a hypothetical record, zero change would mean that two usable measurements establish equal values. An empty cell establishes no such equality. Replacing it with zero creates a completed dataset under an assumption. That replacement can be examined in a sensitivity exercise, but it should not be presented as a measurement.
What would an independent reanalysis need to check?
First check whether identical records and rules yield matching intermediate selections and final summaries within a specified rounding tolerance. Then change a consequential rule deliberately and trace the effect. The first exercise can reveal an implementation discrepancy; the second evaluates dependence on an analytical choice, even when both implementations are correct.
Editorial assessment
A persuasive interpretation should survive being restated as an explicit question about identifiable records. Specify the intended population and outcome, trace how an observation enters the calculation, and identify every assumption needed to extend the answer. These steps make disagreement useful because they locate what additional evidence would resolve it. A claim can be well supported for a narrow target while remaining unestablished for a broader one. The most informative next check is therefore the one that could change the conclusion: a missing measurement, a different timing rule, or a reproducibility discrepancy. Where alternative credible assumptions lead to different answers, retain that dependence in the conclusion. Where the answer remains stable, describe the range of assumptions tested without converting stability into proof of a mechanism or a prediction for an individual.
Limits of this analysis
This article explains how to examine an outcome comparison. Its invented records illustrate logical tests rather than supplying observations about participants. The analysis does not produce an individual prediction, resolve unobserved outcomes or evaluate a provider's results. Any extension beyond the stated evidence would require its own assumptions and supporting observations.
References
- Rubino (2025) · Diabetes Obes MetabSource checked 2026-10-28
- le Roux (2023) · ObesitySource checked 2026-10-28
- ZEPBOUND · NLMSource checked 2026-10-27
- FDA GLP-1 · FDASource checked 2026-10-27
About this publication · Commercial disclosure · Request a correction