Tag precision - This is page 14 - GeneraCodice

Fortran improper output on AIX and Linux

https://www.generacodice.com/en/articolo/10237848/fortran-improper-output-on-aix-and-linux
  linux  -  aix  -  fortran  -  precision  -  intel-fortran
StackOverflow StackOverflow

Losing precision converting from double to int in Java

https://www.generacodice.com/en/articolo/10227795/losing-precision-converting-from-double-to-int-in-java
  java  -  precision
StackOverflow StackOverflow

Precision of reading in file

https://www.generacodice.com/en/articolo/10214898/precision-of-reading-in-file
  c++  -  precision
StackOverflow StackOverflow

Why IEEE754 single-precision float has only 7 digit precision?

https://www.generacodice.com/en/articolo/10152216/why-ieee754-single-precision-float-has-only-7-digit-precision
  floating-point  -  types  -  precision  -  ieee-754  -  single-precision
StackOverflow StackOverflow

Creating a long precision number in C#

https://www.generacodice.com/en/articolo/10149519/creating-a-long-precision-number-in-c-sharp
  c#  -  floating-point  -  .net  -  precision
StackOverflow StackOverflow

Number above ULONG_MAX during unsigned arithmetic

https://www.generacodice.com/en/articolo/10120536/number-above-ulong-max-during-unsigned-arithmetic
  c  -  precision
StackOverflow StackOverflow

Want to calculate 18 digits but will only calculate to 6? Using long double but still won't work

https://www.generacodice.com/en/articolo/10112502/want-to-calculate-18-digits-but-will-only-calculate-to-6-using-long-double-but-still-won-t-work
  floating-point  -  c++  -  precision  -  long-double
StackOverflow StackOverflow

What is the convenience method to compare two NSNumbers with float values using limited precision?

https://www.generacodice.com/en/articolo/10112076/what-is-the-convenience-method-to-compare-two-nsnumbers-with-float-values-using-limited-precision
  floating-point  -  objective-c  -  comparison  -  precision  -  convenience-methods
StackOverflow StackOverflow

Fixed-point scaling and accuracy in multiplication

https://www.generacodice.com/en/articolo/10085982/fixed-point-scaling-and-accuracy-in-multiplication
  c  -  precision  -  fixed-point
StackOverflow StackOverflow

Floating Point Arithmetic error

https://www.generacodice.com/en/articolo/10031373/floating-point-arithmetic-error
  floating-point  -  double  -  python  -  precision  -  ieee-754
StackOverflow StackOverflow
Results found: 928
scroll top