Quantcast

Re: Scale of number and precision on floating data types

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Scale of number and precision on floating data types

Mark Rotteveel-2
No, that is not how floating point works. The storage is different from integers or bigint, as it is, the text makes no sense at all.

Mark


----- Bericht beantwoorden -----
Van: "Helen Borrie" <[hidden email]>
Aan: "Chatter regarding Firebird documentation" <[hidden email]>
Onderwerp: [Firebird-docs] Scale of number and precision on floating data types
Datum: wo, aug. 10, 2016 22:35

Hello ,

Wednesday, August 10, 2016, 9:20:50 PM, Martin Koeditz wrote:

> I hang on a text part in fblangref25-datatypes.xml -> fblangref25-datatypes-floattypes:
> Floating-point data types are examples of data stored in the DBMS with the precision matching the scale of the number.
 
> What is the scale of number? Please make this clear. Otherwise I don’t get the correct translation.
 
I don't like this at all.  I think it needs some input from Paul and
Dmitry and we should rewrite it.

The way I understand the storage of floating point numbers:

FLOAT is stored as INTEGER (precision 9, scale 0) with the scale
(stored separately) depending on the position of the decimal point.

DOUBLE PRECISION is stored as BIGINT (18,0) with scale
depending on the position of the decimal point.

So (as I understand), floating point and fixed point numbers are
stored in the same manner.  The difference - understood by the engine
- is that the *accuracy* of floating point numbers is reduced as the
scale increases.  The *accuracy* of the last two digits is always
unreliable;  hence, the more the decimal point moves to the right, the
less accurate the number.

This assumption does need confirmation or correction, though.

The correctness of the statement aside, is your question about how to
translate the words "precision" and "scale"?  Possibly Thomas can help
with that, as he did the German translations of quite a few of the
earlier documents and often writes articles for German publications.

Helen


------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. http://sdm.link/zohodev2dev
_______________________________________________
Firebird-docs mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/firebird-docs

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. http://sdm.link/zohodev2dev
_______________________________________________
Firebird-docs mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/firebird-docs
Loading...