[L2Ork-dev] Precision of [sqrt~] for double-precision

[Matt Barber]

On Mon, Jun 11, 2018 at 5:51 PM Jonathan Wilkes <jon.w.wilkes at gmail.com>
wrote:

> On Mon, Jun 11, 2018 at 5:14 PM, Matt Barber <brbrofsvl at gmail.com> wrote:
> > Oops, that's 8th and 9th
> >
> > On Mon, Jun 11, 2018, 4:38 PM Matt Barber <brbrofsvl at gmail.com> wrote:
> >>
> >> Good to 8 because of the sqrt operation. If you compare results of sqrt
> >> with 10 mantissa bits on and then with 23 mantissa bits on, they will
> differ
> >> at the 7th bit, or 8th when you count the implied bit.
>
> Ah, I see. So it shaves the input mantissa down to 10 bits to look up
> the result in the
> table, and therefore the result has a mantissa good to 8 bits.
>
> 2048 makes sense for the exponent table.
>
> As for the mantissa-- is there some fancy DSP algorithm we can use to
> grope toward a sensible number of bits? Perhaps something that places
> a [sqrt~] upstream in a chain that reads indices in a really big table?
>
> Something where n bits sounds bad but n + x bits sounds acceptable...
>

​I worry a little about using "how it sounds" as a heuristic just because​
the use case is not always audio-domain. But with that, I think that we
could shoot for "CD quality" and do well enough. The way floats work for
normalized audio -1.0 to 1.0 is that the worst resolution is in the range
0.5 < abs(x) < 1.0, which is a 23-bit range. So if you imagine it being
quantized to integers, the maximum resolution would be 2^23, times 2 for
the 0 < abs(x) < 0.5 range (sacrificing some available precision in that
range, but basically using 2^23 numbers there where some of the resolution
is in the exponents instead of mantissa), and then then times 2 for the
sign bit, or 2^23 * 2 *2 = 2^25, or 25-bit audio. So by analogy, if we
shoot for "accurate through the 14th mantissa bit," that will give us 16
bits, of which we'll only be using 15 because the sign bit won't matter in
sqrt(). So we could find where the cutoff is on the input side and use that
as a starting point. I'll look at that shortly.

Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://disis.music.vt.edu/pipermail/l2ork-dev/attachments/20180611/6657dd31/attachment.html>