[L2Ork-dev] Precision of [sqrt~] for double-precision

[Matt Barber]

Hi,

Still working on this. I was preparing for and attending a memorial for my
dad this week.

Matt

On Mon, Jun 11, 2018 at 9:11 PM Matt Barber <brbrofsvl at gmail.com> wrote:

>
>
> On Mon, Jun 11, 2018 at 5:51 PM Jonathan Wilkes <jon.w.wilkes at gmail.com>
> wrote:
>
>> On Mon, Jun 11, 2018 at 5:14 PM, Matt Barber <brbrofsvl at gmail.com> wrote:
>> > Oops, that's 8th and 9th
>> >
>> > On Mon, Jun 11, 2018, 4:38 PM Matt Barber <brbrofsvl at gmail.com> wrote:
>> >>
>> >> Good to 8 because of the sqrt operation. If you compare results of sqrt
>> >> with 10 mantissa bits on and then with 23 mantissa bits on, they will
>> differ
>> >> at the 7th bit, or 8th when you count the implied bit.
>>
>> Ah, I see. So it shaves the input mantissa down to 10 bits to look up
>> the result in the
>> table, and therefore the result has a mantissa good to 8 bits.
>>
>> 2048 makes sense for the exponent table.
>>
>> As for the mantissa-- is there some fancy DSP algorithm we can use to
>> grope toward a sensible number of bits? Perhaps something that places
>> a [sqrt~] upstream in a chain that reads indices in a really big table?
>>
>> Something where n bits sounds bad but n + x bits sounds acceptable...
>>
>
> ​I worry a little about using "how it sounds" as a heuristic just because​
> the use case is not always audio-domain. But with that, I think that we
> could shoot for "CD quality" and do well enough. The way floats work for
> normalized audio -1.0 to 1.0 is that the worst resolution is in the range
> 0.5 < abs(x) < 1.0, which is a 23-bit range. So if you imagine it being
> quantized to integers, the maximum resolution would be 2^23, times 2 for
> the 0 < abs(x) < 0.5 range (sacrificing some available precision in that
> range, but basically using 2^23 numbers there where some of the resolution
> is in the exponents instead of mantissa), and then then times 2 for the
> sign bit, or 2^23 * 2 *2 = 2^25, or 25-bit audio. So by analogy, if we
> shoot for "accurate through the 14th mantissa bit," that will give us 16
> bits, of which we'll only be using 15 because the sign bit won't matter in
> sqrt(). So we could find where the cutoff is on the input side and use that
> as a starting point. I'll look at that shortly.
>
> Matt
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://disis.music.vt.edu/pipermail/l2ork-dev/attachments/20180618/de1302a1/attachment.html>