Firefox and Javascript Rounding Rules

Question

I don’t know if I am missing something obvious here, but ...

In IE, Opera, and Chrome, I get what I expect from rounding numbers ending in 5:

125 toPrecision(2) => 130 11.5 toPrecision(2) => 12 

This is what I expect.

Firefox, however, is a bit more "sophisticated", giving the following:

 125 toPrecision(2) => 120 //wtf!!! 11.5 toPrecision(2) => 12 

After a little scratching my head, I came to the conclusion that Firefox uses the “rounding” rule, where if a number up to 5 is equal to a number rounded down and if a number up to 5 is an odd number, it is rounded:

 0.5 => 0 1.5 => 2 2.5 => 2 3.5 => 4, etc. 

I use rounded results to test student engineering solutions with pseudo-randomly generated questions. The input question in Chrome may be h = 1020 mm, but h = 1030 mm in FF, Chrome or Opera.

I need a function to make rounding consistent, i.e. I want 0.0001235 to be rounded to 0.000124, and I want 1234 to be rounded to 1240, so I cannot use the simple num = Math.floor (num + 0.5); To complicate things a bit, I want the students' input variables and answers to be correct for 3 sig digs if the first digit is not 1, in which case I want 4 sig digs:

 234.5 => 235 134.5 => 134.5 

I hacked a solution for 3 or 4 sig dig depending on the first digit, converting the number to a string and checking the first non-zero, non-decimal point and non-negative character for “1” - not really, but it works. I could do something similar for the rounding problem by checking to see if the rounded digit is 5, but I am wondering if there is an elegant bitwise solution.

Answer 1

Please see the tests here.

http://yuiblog.com/blog/2009/03/10/when-you-cant-count-on-your-numbers/

JavaScript has one type of number: IEEE 754 with double precision dot. Having one type of number is one of the best features of JavaScripts. A source of several types can be a source of complexity, confusion, and error. One type simplifies and stabilizes.

Unfortunately, the binary floating-point type has some significant drawbacks. The worst part is that it cannot accurately represent the decimal fractions, which is a big problem because humanity has commercially decimal words for a long, long time. There would be advantages to switching to a binary number of the system, but this is not going to happen. As a result, 0.1 + 0.2 === 0.3 is false, which is a source of great confusion.

Also consider SO questions:

https://stackoverflow.com/questions/287744/good-open-source-javascript-math-library-for-floating-point-operations

and

https://stackoverflow.com/questions/744099/javascript-bigdecimal-library/1575569#1575569

Answer 2

O "I want 0.0001235 to be rounded to 0.000124": this does not make much sense, since unlike 0.5, 1.5, 2.5, etc. the decimal number 0.0001235 is not exactly represented in radix 2; therefore, its rounding may be slightly higher (A) or slightly lower than (B) the exact value, where (A) and (B) will give different roundings: 0.000124 for (A), 0.000123 for (B).

Even if you get a fractional number, such as 1.5, round up to an integer after some calculations, (1.5). toPrecision (2) will not necessarily give the value that is closest to the exact value. The reason is that the exact value may be slightly below 1.5, in which case the correct rounding is 1, not 2. This problem is called the Table Maker dilemma.

This also means that if two students use slightly different methods (both fairly accurate) and round their approximations, they will get different rounded results (although their approximations can be very close to each other). In the case of such a binding as 0.0001235 (rounded to 0.000124), the Table Maker dilemma is inevitable if all calculations are not performed accurately or the algorithm does not specifically check related cases.

However, if the exact result is a snap or very close to a tie, you can still check if the student answered either (A) or (B).