PpgDex (raw PPG) · ECGDex (raw ECG) nodes, Tepna physiological-signal suite
Background. A wearable HRV pipeline decides which epochs to trust using a signal-quality index (SQI) computed from the optical signal alone. Choosing among candidate gates requires ground truth, and the natural experiment is to pool every epoch from every night and ask which gate best separates the epochs whose HRV turned out wrong. We show that this natural experiment gives the wrong answer, and gives it confidently. Data. 1,488 five-minute epochs from 20 simultaneous Polar H10 chest-ECG + Polar Verity Sense armband-PPG overnight recordings, one subject. An epoch is “catastrophic” if its optical rMSSD deviates from the simultaneous ECG by more than 25% (158 epochs, 10.6%). Results. Three candidate gates were scored by their ability to rank catastrophic epochs below good ones. Pooled across nights, beat coverage wins decisively (AUC 0.737, p = 7×10⁻²³) and the shipped per-beat SQI is barely distinguishable from a coin flip (AUC 0.556). Clustered within nights — the correct analysis, since epochs from one night are not independent — the ranking exactly reverses: the shipped SQI is best (median AUC 0.763, Wilcoxon p = 0.014) and beat coverage is worst and not significant (0.629, p = 0.106). The reversal is an ecological fallacy: coverage varies strongly between nights and tracks which nights are bad, so pooling lets between-night variation masquerade as within-night discrimination, while the SQI's night-specific scale is washed out by the same pooling. Applied as an actual gate within each night, the SQI reduces the 90th-percentile |ΔrMSSD| from 25.8% to 21.4% at 70% retention, whereas coverage makes the median error worse (4.43% → 4.89%). Conclusion. Signal-quality gates for wearable HRV must be evaluated within recording, not on pooled epochs. The pooled analysis here would have prompted replacing a working gate with one that is actively counterproductive.
Keywords: signal quality index · photoplethysmography · heart-rate variability · ecological fallacy · Simpson's paradox · clustered data · AUC · wearables · method evaluation
A wrist or arm sensor cannot always be trusted, so the software scores each five-minute stretch and throws away the ones it thinks are bad. We had a chest strap giving the true answer, so we could check which scoring method actually catches the bad stretches. Doing the obvious thing — dumping every stretch from every night into one pile and comparing — said the software's own score was almost useless and that a different measure was much better. But stretches from the same night are not independent observations, and once we compared within each night, the answer flipped completely: the software's own score was the best one, and the “better” alternative turned out to be the worst. The pile-everything-together version was measuring differences between nights and mistaking them for the ability to spot a bad stretch. This matters beyond our sensor: the wrong analysis was easy, fast, produced a tiny p-value, and would have led us to delete a working feature and replace it with something that makes results worse.
Every consumer-wearable HRV pipeline contains a gate: some reference-free statistic that decides which stretches of a recording are trustworthy enough to report. Because the gate is what stands between a user and a confidently wrong number, choosing the right one matters more than most tuning decisions. And because gates are cheap to compute and easy to swap, the choice is usually made empirically — collect data with a reference, label the epochs the reference says went wrong, and pick whichever candidate statistic separates them best.
That procedure has a defect that is well known in epidemiology and almost never mentioned in wearable-signal work: the epochs are clustered. A night contributes eighty-odd epochs that share a sensor placement, a skin-contact quality, a perfusion state, and a subject posture distribution. Treating them as 1,488 independent observations is not merely optimistic about the p-value; it can change which candidate wins, because a statistic that mostly encodes “which night is this” will separate pooled labels without having any ability to rank epochs inside a night — which is what a gate actually has to do.
This paper documents that failure occurring, on real data, in a direction that would have caused real harm to a working pipeline. It is a negative methodological result with a positive corollary: the gate the pooled analysis condemned is, correctly analysed, the best of the three.
One subject wore a Polar H10 chest ECG and a Polar Verity Sense optical armband simultaneously for 26 nights; 20 yield a paired analysis of 1,488 five-minute epochs (121.5 h). Optical beats come from the production PPGDSP detector (three-LED consensus, intersecting-tangent foot timing); ECG R-peaks from Pan–Tompkins with sub-sample refinement, validated at 0.244 ms fiducial jitter. Alignment is re-estimated per epoch, because a single global affine map between two independent BLE clocks fails across a whole night. Full acquisition and alignment detail is in the companion paper, Optical HRV against chest ECG on twenty overnight recordings.
An epoch is catastrophic when its optical rMSSD deviates from the simultaneous ECG rMSSD by more than 25%. 158 of 1,488 epochs (10.6%) qualify. The threshold is a judgement, chosen because it is far outside the corpus's typical +4% bias and corresponds to an error that would change a user-facing interpretation; §4 reports sensitivity to it.
All three are computed from the optical signal alone, with no reference, and all three are already available in the shipped export:
Pooled. All 1,488 epochs in one set; AUC computed once as the probability that a randomly chosen good epoch scores above a randomly chosen catastrophic one; significance by Mann–Whitney U.
Night-clustered. AUC computed separately within each night that contains at least three catastrophic and three good epochs (10 of 20 nights qualify), then summarised across nights; significance by Wilcoxon signed-rank of the per-night AUCs against 0.5. This is the analysis that matches the deployed decision — a gate operates inside one recording, on that recording's own epochs.
Retention curves apply each gate within each night at a per-night percentile, so a night is never wholly deleted by a between-night level shift, and report the resulting median and 90th-percentile |ΔrMSSD| over the retained epochs.
| Gate | Pooled AUC | Pooled p | Night-clustered AUC (median, IQR) | Nights > 0.5 | Clustered p |
|---|---|---|---|---|---|
| Mean SQI (shipped) | 0.556 worst | 0.011 | 0.763 best [0.645, 0.791] | 9/10 | 0.014 |
| LED agreement | 0.642 | 8×10⁻¹⁰ | 0.700 [0.610, 0.760] | 10/10 | 0.002 |
| Beat coverage | 0.737 best | 7×10⁻²³ | 0.629 worst [0.561, 0.729] | 8/10 | 0.106 n.s. |
The pooled analysis is not merely noisy — it is decisive, and decisively wrong. Beat coverage's pooled p of 7×10⁻²³ is the most significant result in the table, and it belongs to the only gate that fails to reach significance once clustering is respected. Conversely the shipped SQI, which the pooled analysis places at 0.556 — close enough to 0.5 that it reads as “this feature does nothing” — is the best gate available, above chance on 9 of the 10 evaluable nights.
The mechanism is an ecological fallacy. Beat coverage varies strongly between nights: nights where the armband sat poorly lose a large fraction of beats to the three-LED consensus and also generate most of the catastrophic epochs. Pooling therefore lets a between-night level difference do the separating, and the pooled AUC largely measures “can this statistic tell a bad night from a good night” — a real but different ability. Within a night, coverage is nearly constant and carries little information about which of that night's epochs went wrong.
The shipped SQI has the opposite structure. Its absolute level is night-specific — it depends on perfusion, skin contact and template shape, which shift between wearings — so pooling smears the good epochs of a low-SQI night into the bad epochs of a high-SQI night and destroys the signal. Within a night, where the offset is common, it ranks epochs well. A gate is always deployed within a recording, so the within-night behaviour is the one that matters and the pooled number is not merely imprecise but answering a different question.
| Gate | Retention | Median |ΔrMSSD| | p90 |ΔrMSSD| |
|---|---|---|---|
| none | 100% | 4.43% | 25.80% |
| Mean SQI | 90% | 4.22% | 22.85% |
| 80% | 4.09% | 22.62% | |
| 70% | 3.92% | 21.43% | |
| LED agreement | 90% | 4.15% | 25.17% |
| 80% | 4.03% | 23.62% | |
| 70% | 3.95% | 23.41% | |
| Beat coverage | 90% | 4.39% | 24.79% |
| 80% | 4.70% worse | 24.56% | |
| 70% | 4.89% worse | 23.44% |
The retention curves confirm the clustered ranking behaviourally rather than by AUC. Mean SQI improves both the median and the tail monotonically as it discards more data. Beat coverage improves the tail slightly while making the median worse — it is discarding epochs that were fine and keeping epochs that were not, which is what a gate carrying mostly between-night information does when applied within a night. Had the pooled analysis been acted on, the pipeline would have replaced a gate that helps with one that is, on the metric that matters most to a user, actively harmful.
An honest reading of Table 2 is that no gate here is transformative: the best available option removes 30% of the data to move the median error from 4.43% to 3.92% and the tail from 25.8% to 21.4%. That is worth having and worth reporting honestly, but it is not the difference between a usable and an unusable pipeline. The companion paper shows why — epoch error is governed almost entirely by fiducial jitter (per-night R² = 0.964), and none of these three gates measures jitter directly. The gate literature's implicit promise, that a good SQI can rescue a noisy optical arm, is not borne out at these effect sizes; the productive route is reducing jitter, not filtering harder.
The general rule. Evaluate a signal-quality gate the way it will be deployed. If it decides among epochs inside one recording, score it inside one recording and aggregate the per-recording scores. Pooling epochs across recordings answers “can this statistic distinguish good recordings from bad ones,” which is a legitimate question — night-level triage is useful — but it is not the question a per-epoch gate is asked, and on this corpus the two questions have opposite answers.
Why it is easy to get wrong. The pooled analysis is the path of least resistance in every respect: it is one line of code, it uses all the data, it yields large n and tiny p-values, and its conclusion (“your bespoke quality index is no better than chance; use this simple coverage number instead”) is exactly the kind of parsimonious finding that feels like a discovery. Nothing about it looks like a mistake from the inside. The only warning sign available without the clustered analysis is that the winning statistic is one whose variance is obviously dominated by between-recording differences — a structural observation, not a statistical one.
Sensitivity to the label threshold. The inversion is not an artifact of the 25% cut. Raising or lowering it changes the number of catastrophic epochs and hence the precision of both analyses, but coverage's advantage is pooled-only and the SQI's advantage is clustered-only across the range tested, because the mechanism is structural rather than threshold-dependent. We report the 25% result as primary and note that a threshold low enough to label a large fraction of epochs would make the comparison uninformative in both analyses.
ppgdex-dsp.js and ecgdex-dsp.js, run unmodified in a co-loaded realm, so gate values are exactly those the shipped app computes.papers/ppg-ecg-hrv-validation.html — the 20-night optical-vs-ECG agreement study and the fiducial-jitter model.CLAUDE.md (Clock Contract, evidence-grade system), Tepna suite.