How to avoid rounding errors when outputting floats?

[SHughes_Fusion]

I'm hitting an issue when outputting a float that the result appears to be suffering from rounding errors.

A value of 123.125 will output correctly, however, 123.12 will output as 123.119.

I'm guessing this will be due to the way numbers are represented, but I'm wondering is there any way to avoid this?

I'm thinking that the obvious solution of adding 0.001 will simply mean other numbers will output values slightly higher than they should?

I spotted this as I'm inputting and returning a float and every time it was 0.01 low. Initially I thought the problem was my StrToFloat routine but now I'm suspecting the output routine..

Code: Select all

Dim 
   FDigitsIn As PREINC1,        // pre incrementing pointer to digits array
   FDigitsAddress As FSR1,      // address of digits array
   FDigitsOut As POSTDEC1,      // post decrementing pointer to digits array
   FTextAddress As FSR2,        // address of result string
   FTextIn As POSTINC2,         // post incrementing pointer to result string
   FTextDec as POSTDEC2         // post decrementing pointer to result string

Dim TStr As String(12)              // Temp string to hold output

    // Convert a number to a string. Set FTextIn = @TStr before calling these routines.
Sub Num2Str(pValue As Word)
  Dim Index As Byte
  Dim Number As Word
  Dim Digits(6) As Byte
  FDigitsAddress = @Digits
  Index = 0
  Repeat
    Number = pValue Mod 10
    FDigitsIn = Number + 48
    pValue = pValue / 10
    Inc(Index)
  Until pValue = 0
  While Index > 0               // String created will be inverted so this swaps it back to the correct order
    FTextIn = FDigitsOut
    Dec(Index)
  Wend
End Sub

Sub SendData(CmdRef As Byte, Data As Float)
  Dim TB As Byte
  Dim TW As Integer
  
  FTextAddress = @TStr              // Initialise pointer to start of the string
  AddCommandCode(CmdRef)
  FTextIn = GH_ETB                  // Send end of transmission block character
  If Data.IsSigned Then             // Is number negative?
    FTextIn = "-"                   // Add a minus sign
    Data = -Data                    // Invert
  EndIf 
  TW = Data
  NumToStr(TW)                      // Append number to string
  FTextIn = "."
  
  Data = Data - TW
  Data = Data * 1000
  TW = Data
  If TW < 10 Then FTextIn = "0" EndIf       // If less than 10 add a padding zero
  IF TW < 100 then FTextIn = "0" Endif      // If less than 100 add two padding zeros
  NumToStr(TW)                      // Append to string
  FTextIn = RS485EOT
  FTextIn = null                    // Terminate 

  RS485Comms.Write(TStr)
End Sub

[Jerry Messina]

I'm guessing this will be due to the way numbers are represented, but I'm wondering is there any way to avoid this?

Not really. A 32-bit float has something < 7 digits of precision (I think it's 6.9), so you can easily start to get "errors" as you do various math operations, esp if the range of the variables is more than a few digits difference.

The way to avoid it? Use a fixed-point representation. A 32-bit long gives you a lot of range if you only need 3 digits past the dec pt.

FWIW, I tried your code and got "123.120", so maybe you don't have quite what you think. the difference between 123.119 and 123.120 is 1 LSB in that 32-bit float.

[SHughes_Fusion]

Maybe worth going to fixed point, I'll investigate.

I actually only need 2dp, I'd gone to three to try and understand what was happening.

I've found a sort of fix. If I add 0.006 that seems to give the expected results - after all, 123.125 should really output 123.13 if you are rounding to the nearest.. I've not tried every variant yet but it should be the case that any two dp number sent will at least come back the same which is the most important thing. The numbers returned will be a mix of user-set values and process variables - for the latter the user won't know if there is a rounding error but it will stick out like a sore thumb for the former if they send one thing and get another back.