The code currently switches to the alternative formula for lambda only if (R,M)= (0,0)
but the alternative formula works whenever M = 0: Specifically, M = 0
implies y1 = -y2. If x1 = x2, then a = -b this is the r = infinity case that we
handle separately. If x1 != x2, then the denominator in the alternative formula
is non-zero, so this formula is well-defined.
One needs to carefully check that the infinity assignment is still correct
because now the definition of m_alt at this point in the code has changed. But
this is true:
Case y1 = -y2:
Then degenerate = true and infinity = ((x1 - x2)Z == 0) & ~a->infinity . a->infinity is handled separately. And if ~a->infinity, then Z = Z1 != 0, so infinity = (x1 - x2 == 0) = (a == -b) by case condition.
Case y1 != -y2:
Then degenerate = false and infinity = ((y1 + y2)Z == 0) & ~a->infinity . a->infinity is handled separately. And if ~a->infinity, then Z = Z1 != 0, so infinity = (y1 + y2 == 0) = false by case condition.
Co-Authored-By: Pieter Wuille <pieter@wuille.net>
This is a backport of secp256k1#1078