fix(schedulers): prevent NaN in ancestral schedulers with squaredcos_cap_v2 - #14221
Conversation
f8ea78b to
cff1528
Compare
…cap_v2 EulerAncestralDiscreteScheduler.step() and KDPM2AncestralDiscreteScheduler.set_timesteps() compute sigma_down = sqrt(sigma_to**2 - sigma_up**2). sigma_up is mathematically <= sigma_to, but under beta_schedule='squaredcos_cap_v2' the sigma dynamic range is so large that in float32 sigma_up can round fractionally larger than sigma_to, making the radicand negative and producing NaN that propagates into the sample (EulerAncestral) or into self.timesteps via log(0) in sigmas_interpol (KDPM2Ancestral). For EulerAncestral, clamp the radicand to >= 0 before the square root; the sigma_up == sigma_to limit gives sigma_down == 0, which is the correct value. For KDPM2Ancestral, clamp the sigma_up radicand to >= 0, compute sigmas_down via the algebraically identical and numerically stable sqrt(sigmas_next**4 / sigmas**2), and express sigmas_interpol as the geometric mean sqrt(sigmas * sigmas_down) instead of the log-space lerp whose -inf end term produces NaN wherever sigmas_down == 0. The linear and scaled_linear paths are unaffected and produce identical output. Adds regression tests for both schedulers covering the squaredcos_cap_v2 + low-step-count regime. Fixes huggingface#14213
cff1528 to
46511b4
Compare
|
You already use the closed form in sigmas_down = sigmas_next**2 / sigmas.clamp(min=1e-30) # sqrt(sigmas_next**4/sigmas**2)The same identity applies to so The distinction matters beyond tidiness, because the clamp does not only suppress the NaN. It returns zero where the true value is small but non zero. In float32:
The closed form also makes Checked on CPython 3.12, torch float32, macOS arm64. Happy to send this as a patch against your branch if that is useful. |
EulerAncestralDiscreteScheduler and KDPM2AncestralDiscreteScheduler both produce all-NaN output when configured with beta_schedule="squaredcos_cap_v2" at low step counts (the standard ancestral fast-sampling regime). The root cause is that sigma_down is computed as sqrt(sigma_to2 - sigma_up2): sigma_up is mathematically <= sigma_to, but the huge sigma dynamic range from squaredcos_cap_v2 means float32 rounding can make sigma_up fractionally larger than sigma_to, driving the radicand negative. In EulerAncestral the NaN reaches the sample via step(); in KDPM2Ancestral it reaches self.timesteps during set_timesteps() because the same cancellation feeds log(0) into sigmas_interpol.
For EulerAncestral I clamp the radicand to >= 0 before the square root (the sigma_up == sigma_to limit correctly gives sigma_down == 0). For KDPM2Ancestral I clamp the sigma_up radicand, compute sigmas_down via the algebraically identical and numerically stable sqrt(sigmas_next4 / sigmas2), and express sigmas_interpol as the geometric mean sqrt(sigmas * sigmas_down) instead of the log-space lerp whose -inf end term produces NaN wherever sigmas_down == 0. The linear and scaled_linear paths are unaffected and produce identical output. I verified the fix against the reproduction in the issue and added regression tests for both schedulers covering the squaredcos_cap_v2 + 4-step regime.
Fixes #14213