Re: When is a string a number?

On Wed, 2006-10-04 at 14:00 -0500, David Nicol wrote:
On 10/4/06, Martin J. Evans <martin.evans@xxxxxxxxxxxx> wrote:

With DBI/DBD::Oracle all values read from the database are scalars. As
everyone will know, whether something read from the database is a string
or a number in Perl purely depends on the context it is used in so:

internally, there are flags in the SV structure that hint as to what conversions
have been done on the SV already, for efficiency's sake.

As I suspected.

I have to admit I don't know how the JSON module knows what is a number
and what is a string in Perl but I see the same issue with Data::Dumper
so I presume there must be some way to find out if a perl scalar is a
number or a string.

It seems like the pure perl Dumper tests values with a regex and optimizes
to numbers when the stringification matches a number template, while
the XS version checks the flags in the SV structure.

OK, I only introduced the Data::Dumper example as it seemed very similar
but read on - I over simplified a little. However, it seems strange to
me that the pure perl Data::Dumper does something different. If I was
using Data::Dumper for data interchange and switched from the pure perl
to the XS version (or vice versa) and then called another XS based
module with the results - they could be different.

The problem gets a lot worse for me since I do some arithmetic on values
pulled from the database before converting them to JSON and this is
where Perl seems to change them into numbers e.g.

I don't want to have to do ($var +0) on all the number fields I pull
from the database (to turn them into numbers) and neither do I want to
do a '$var .= ""' (to turn all the fields into strings).

you might have to do exactly that. "$var" will produce a string version. Sorry.

This seems amazing - I really did not want to do this. I guess what I'm
saying is that it would be really useful for a DBD to mark a number as a
number so it does not start as a string and mutate to a number when I do
arithmetic on it. Most DBDs must know if the column is a number one or
not (certainly DBD::ODBC does).

Apparently you can tell the JSON module to make everything strings:
http://search.cpan.org/~makamaka/JSON-1.07/lib/JSON.pm#AUTOCONVERT

Here was an over simplification. I moved from JSON to JSON Syck. The
JSON module is a purl perl module and does "the right thing" for me for
all the cases I tried but I changed to JSON Syck which uses the libsyck
library for speed. This explains the difference since JSON Syck is XS
and will know the flags in the SV structure and JSON won't.

As an aside (and probably a perl question rather than a DBI one) does
anyone know why the type of a scalar changes when you use it on the
right side of an assignment:

perl -MData::Dumper -le '$a="1"; print Dumper($a); $b += $a; print
Dumper($a);'
$VAR1 = '1';
$VAR1 = 1;

The has-been-evaluated-as-a-number flag got set on $a when it was
evaluated as a number; then Dumper, with both available, chose the
number format.

That's interesting to know, I'll look into that. I was rather surprised
the right hand side of an evaluation changed the type of a variable
on the right hand side.

How does JSON and Data::Dumper know whether Perl thinks something is a
number or a string?

inspecting the flags; except pure-perl Dumper apparently uses a
regular expression
to identify numbers. Those are guesses. The source is available for
your inspection.

I did look at it briefly but not extensively since it was not my primary
problem.

Thanks for your insights David, they have increased my understanding
although I still feel stuck. I could move back to using the JSON module
but I could run into problems with fields like house_name_or_number
which I'm now guessing JSON will serialise as a number when it looks
like a number and a string when not. If I stick with JSON Syck
(which is a lot faster and I need the speed), I'm
forced to do +0 on all database fields I know are a number and I don't
like this at all. I guess I'm rather surprised to hit this issue without
seeing anyone else with it after all this time using DBI.

Martin
--
Martin J. Evans
Easysoft Limited
http://www.easysoft.com

.

Relevant Pages

  • When is a string a number?
    ... the json to a file where it is picked up by some javascript code. ... With DBI/DBD::Oracle all values read from the database are scalars. ... whether something read from the database is a string ...
    (perl.dbi.users)
  • Database insertion, escape issue
    ... I have a Perl script where I try to strip some data from a web page ... I am capturing the paragraphs of text and inserting ... I am currently trying this with a SQLite database. ... # Create the file string ...
    (perl.beginners)
  • RE: When is a string a number?
    ... Perl users don't experience these issues on a regular basis if ever. ... the values computed from the DB values are required in the JSON ... When is a string a number? ... I moved from JSON to JSON Syck. ...
    (perl.dbi.users)
  • Re: ORA-12704: character set mismatch problem with DBD::Oracle
    ... It's a string as far as perl is concerned ... > it is set to the character string "0800690C0A" it fails. ... The database is also Oracle 8.1.7. ...
    (perl.dbi.users)
  • Re: When is a string a number?
    ... whether something read from the database is a string ... or a number in Perl purely depends on the context it is used in so: ... number or a string. ... anyone know why the type of a scalar changes when you use it on the ...
    (perl.dbi.users)