Simple calculation task with Flex

I am having trouble calculating flex, I am trying to convert the pixel value to mm and reverse it.

Starting at 69.8:

Calculating MM to pixel from: 69.8 mm 69.8*300 = 20940 20940 / 2.54 = 8244.094488188975 8244.094488188975 / 10 = 824.4094488188975 

And the calculation back:

 Calculating pixel to MM from: 824.4094488188975 824.4094488188975/300 = 2.7480314960629917 2.7480314960629917 * 2.54 = 6.979999999999999 6.979999999999999 * 10 = 69.79999999999998 

We wanted 69.8, but in the end it turned out 69.79999999999998. I tracked the process using simple window calculations, and the first place it does it wrong is 20940 / 2.54 = 8244.094488188975, which should be 8244,094488188976.

Anny help on this would be great.

+1
source share
2 answers

Well, this is not so much a Flex issue as a general programming issue. Have you ever wondered how exactly the numbers are stored on your computer? Details are more in the reality of mathematics, but I will try to make it simple. You know that there are an infinite number of different Real numbers (for example, as points on a continuous line), but on your computer you store everything as zeros and ones, and there are a limited number of them (32 or 64 "bits"). So now we have a problem - how to represent an unlimited number of numbers in a limited space. The idea used in a floating-point number (those that can have β€œdot” values, for example, 1.03 or 4.2232) is that since you cannot all of them, you round the number to the closest you have .

It reminds me a bit of the memory of how much sugar someone puts in their coffee, you don’t remember that he likes to have 1.1232 tablespoons of sugar, because a tablespoon is just not so good at measuring the exact amount of material on It. You go around it one at a time, and it works great in most cases.

So, with floating point numbers, a similar idea stands, with an additional twist - numbers are much denser around 0 than from it, where the "empty space" between them becomes really large. For example, if you subtract 10,000 from the maximum value that the number can get, it will still be the same, because next to it there is no number that matters when looking for the nearest one.

  trace (Number.MAX_VALUE == Number.MAX_VALUE-10000); // returns "true" trace (200000 == 200000 - 10000); // returns "false" 

So, your problem assumes that the numbers are completely accurate, but they are not, you always get rounded objects. The number in as3 adheres to the IEEE-754 standard with double precision and determines how it determines what number it has and which is rounded.

Look at this:

 trace (8244.094488188976); // returns "8244.094488188975" 

Further reading:

Floating point

IEEE_754-2008

+7
source

Adobe's answer in TechNote called Flash returns erroneous results for some mathematical calculations; it does this briefly:

This is a fact of life in applications that use floating point arithmetic.

0
source

Source: https://habr.com/ru/post/1339194/


All Articles