Re: Sybase convert oddity


Lee Fesperman wrote:
joeNOSPAM@xxxxxxx wrote:

rickpoleshuck@xxxxxxxxxxx wrote:
select convert(numeric(16,2), 5.325),convert(numeric(16,2), '5.325')

returns

5.32 5.33


I would like to understand the logic behind this behavior, if any.

Hi. This has nothing to do with Java, so you
should post it to comp.databases.sybase.
It probably has to do with the fact that a
conversion to numeric will use an exact
algorithm for it's input. The string input is
exact, but when you pass 5.325 directly,
the DBMS's SQL parser will convert that to
an IEEE float which may not be able to exactly
represent a decimal 5.325. The exact value
may be slightly less, leading to the round-down.

You've undoubtedly nailed the problem, here. I just wanted to point out that this
non-standard behavior by the DBMS. The SQL standard defines 5.325 as an exact numeric
literal. An approximate numeric literal (floating point) must contain an 'E' suffix.

--
Lee Fesperman, FFE Software, Inc. (http://www.firstsql.com)

Actually the OP said that 5.329 also becomes 5.32, so
it's something else...
Joe

.

Relevant Pages

  • Re: Sybase convert oddity
    ... The string input is ... exact, but when you pass 5.325 directly, ... the DBMS's SQL parser will convert that to ... non-standard behavior by the DBMS. ...
    (comp.lang.java.databases)
  • Re: Sybase convert oddity
    ... exact, but when you pass 5.325 directly, ... the DBMS's SQL parser will convert that to ... an IEEE float which may not be able to exactly ...
    (comp.lang.java.databases)
  • Re: Sybase convert oddity
    ... The string input is ... exact, but when you pass 5.325 directly, ... an IEEE float which may not be able to exactly ... may be slightly less, leading to the round-down. ...
    (comp.lang.java.databases)