[L2Ork-dev] Precision of [sqrt~] for double-precision

[Jonathan Wilkes]

On Mon, Jun 11, 2018 at 5:14 PM, Matt Barber <brbrofsvl at gmail.com> wrote:
> Oops, that's 8th and 9th
>
> On Mon, Jun 11, 2018, 4:38 PM Matt Barber <brbrofsvl at gmail.com> wrote:
>>
>> Good to 8 because of the sqrt operation. If you compare results of sqrt
>> with 10 mantissa bits on and then with 23 mantissa bits on, they will differ
>> at the 7th bit, or 8th when you count the implied bit.

Ah, I see. So it shaves the input mantissa down to 10 bits to look up
the result in the
table, and therefore the result has a mantissa good to 8 bits.

2048 makes sense for the exponent table.

As for the mantissa-- is there some fancy DSP algorithm we can use to
grope toward a sensible number of bits? Perhaps something that places
a [sqrt~] upstream in a chain that reads indices in a really big table?

Something where n bits sounds bad but n + x bits sounds acceptable...

-Jonathan