Looks like unsigned long int may be better for nkeys. Bizarrely, size_t is allowed to be as small as 16 bits, in C99 standard.

https://stackoverflow.com/questions/22514803/maximum-size-of-size-t

(maybe matters if anyone ever throws this into a weird hardware wallet embedded system)

compressed serialization, just to be clear

Yes, I was wondering about this ordering issue too. Should be simple enough to just sort the pubkeys lexically since we don't require any fancy insert/remove behaviour.

Seems more ideal if this never fails. That was always one of the totally unnecessary warts that bothered me about BIP32 (failure to generate key). An alternative never-fail:

if tweak == 0:
  tweak = order-1
elif tweak == order:
  tweak = order-2
else:
  tweak = tweak % order

or, with slightly more bias,

tweak = tweak % order
if tweak == 0:
  tweak = order-1

Failure is ok for interactive MuSig key generation because you can restart session, but if it's interactive then you don't actually need MuSig to begin with. You could just use random pre-committed keys with simple A+B+C+D aggregation.

If you have a critical non-interactive key generation and restarting is not possible, then this could cause a deadlock with 2^-128 chance. In fact even doing just tweak = tweak % order (allowing for tweak=0 which unfortunately cancels someone's key, with 2^-255 chance) may be preferable to having a deadlock.

Hmm, on second thought, the entire MuSig can fail if the final key ends up being point at infinity. So maybe it does fundamentally need a counter-increment-retry mechanism for computing C.

If people want to do it, then good for them.