<div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif">OK, so I'd embarrassingly been looking at legacy code. There's now an extra step, in the *out++ line here:</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif"><div class="gmail_default">{</div><div class="gmail_default"> t_sample g = rsqrt_exptab[(u.l >> 23) & 0xff] *</div><div class="gmail_default"> rsqrt_mantissatab[(u.l >> 13) & 0x3ff];</div><div class="gmail_default"> *out++ = f * (1.5 * g - 0.5 * g * g * g * f);</div><div class="gmail_default">}</div></div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">This seems to remove some of the error; Pd says "about 120dB" or 20ish bits of accuracy, on average.</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">I made a program and found that reliably if we use 19 bits of input mantissa, we'll get a minimum of 14 bits of rsqrt accuracy in the mantissa and about 41 bits on average. For 16 bits of input mantissa, minimum is about 6 bits mantissa and 34-35 bits on average. This is all for random bit patterns where the input exponent = 0.</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">So, the exponent table is 2^11 = 2048 points long, the mantissa could IMO be anywhere between 2^11 and 2^20 with fully satisfactory results. I think maybe 2^16 would be a good starting point.</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">Matt</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Jun 18, 2018 at 12:42 AM Matt Barber <<a href="mailto:brbrofsvl@gmail.com">brbrofsvl@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif">Hi,</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">Still working on this. I was preparing for and attending a memorial for my dad this week.</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">Matt</div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Jun 11, 2018 at 9:11 PM Matt Barber <<a href="mailto:brbrofsvl@gmail.com" target="_blank">brbrofsvl@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Jun 11, 2018 at 5:51 PM Jonathan Wilkes <<a href="mailto:jon.w.wilkes@gmail.com" target="_blank">jon.w.wilkes@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Mon, Jun 11, 2018 at 5:14 PM, Matt Barber <<a href="mailto:brbrofsvl@gmail.com" target="_blank">brbrofsvl@gmail.com</a>> wrote:<br>
> Oops, that's 8th and 9th<br>
><br>
> On Mon, Jun 11, 2018, 4:38 PM Matt Barber <<a href="mailto:brbrofsvl@gmail.com" target="_blank">brbrofsvl@gmail.com</a>> wrote:<br>
>><br>
>> Good to 8 because of the sqrt operation. If you compare results of sqrt<br>
>> with 10 mantissa bits on and then with 23 mantissa bits on, they will differ<br>
>> at the 7th bit, or 8th when you count the implied bit.<br>
<br>
Ah, I see. So it shaves the input mantissa down to 10 bits to look up<br>
the result in the<br>
table, and therefore the result has a mantissa good to 8 bits.<br>
<br>
2048 makes sense for the exponent table.<br>
<br>
As for the mantissa-- is there some fancy DSP algorithm we can use to<br>
grope toward a sensible number of bits? Perhaps something that places<br>
a [sqrt~] upstream in a chain that reads indices in a really big table?<br>
<br>
Something where n bits sounds bad but n + x bits sounds acceptable...<br></blockquote><div><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">I worry a little about using "how it sounds" as a heuristic just because the use case is not always audio-domain. But with that, I think that we could shoot for "CD quality" and do well enough. The way floats work for normalized audio -1.0 to 1.0 is that the worst resolution is in the range 0.5 < abs(x) < 1.0, which is a 23-bit range. So if you imagine it being quantized to integers, the maximum resolution would be 2^23, times 2 for the 0 < abs(x) < 0.5 range (sacrificing some available precision in that range, but basically using 2^23 numbers there where some of the resolution is in the exponents instead of mantissa), and then then times 2 for the sign bit, or 2^23 * 2 *2 = 2^25, or 25-bit audio. So by analogy, if we shoot for "accurate through the 14th mantissa bit," that will give us 16 bits, of which we'll only be using 15 because the sign bit won't matter in sqrt(). So we could find where the cutoff is on the input side and use that as a starting point. I'll look at that shortly.</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">Matt</div></div></div>
</blockquote></div>
</blockquote></div>