Changes between Version 1 and Version 2 of perlJsonEncoding


Ignore:
Timestamp:
03/05/18 14:08:42 (10 months ago)
Author:
niles
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • perlJsonEncoding

    v1 v2  
    11 
    2 Encoding JSON using perl 
     2= Encoding JSON using perl = 
    33 
    4 Perl is not a language with strict typing. This lack of strict typing can have ramifications for producing JSON output. In our case this matters because the vso_jsoc_fetch.cgi CGI produces JSON that is critical for NetDRMS systems. 
     4Perl is not a language with strict typing. This lack of strict typing can have ramifications for producing JSON output. In our case this matters because the vso_jsoc_fetch.cgi CGI produces JSON that is critical for NetDRMS systems, and the exact JSON that is produced depends on the perl "typing". The "typing" in perl in fact amounts to a set of internal flags associated with a variable, and these can be set in surprising ways. Worse, the flags can be interpreted in different ways by modules attempting to determine a variable type. All this can be changed by updates to the perl version and the JSON module in use. For instance, consider this code in vso_jsoc_fetch.cgi : 
    55 
     6{{{ 
     7my $totalSize=0; 
     8my $totalCount=0; 
     9for my $key (keys %{$data}) { 
     10  $totalSize += $data->{$key}->{'susize'}; 
     11  $totalCount +=1; 
     12} 
     13}}} 
     14 
     15For many years this worked fine. In the JSON output, every {{{ $data->{$key}->{'susize'}; }}} appeared in the JSON as something like {{{ "susize" : "10284610" }}} 
     16 
     17However, after cpanm was used to install a new module, it seems like the JSON perl module was updated. It turned out that using {{{ $data->{$key}->{'susize'}; }}} 
     18to do math with the line {{{ $totalSize += $data->{$key}->{'susize'}; }}} actually changes some of the internal perl flags around the {{{ $data->{$key}->{'susize'}; }}} so that it was treated as a numeric, rather than a string, type. As far as I can tell, the old JSON perl module did not look at those flags, but the new JSON module did. The result was that the variable was treated as a numeric rather than a string in the JSON output, so the output looked like this : {{{ "susize" : 10284610 }}} (note that there are no quotes around the number). 
     19 
     20This was very hard to debug, since the vso_jsoc_fetch.cgi CGI was producing JSON that seemed, at a glance, to be OK, but in fact did not parse. Worse, this affected not only the local NetDRMS node, but every NetDRMS node that attempted to download from the local node. 
     21 
     22One workaround is to modify the code to use a temporary variable to do the math with, like so : 
     23 
     24{{{ 
     25my $totalSize=0; 
     26my $totalCount=0; 
     27for my $key (keys %{$data}) { 
     28  my $tmpVar = $data->{$key}->{'susize'}; 
     29  $totalSize += $tmpVar; 
     30  $totalCount +=1; 
     31} 
     32}}} 
     33 
     34This means that the re-typing from string to numeric happens on {{{ $tmpVar }}} and the typing of {{{ $data->{$key}->{'susize'} }}} is left alone so that it remains a string. 
     35 
     36A more definitive solution, however, is to install the JSON::XS module and use it to definitively set the types for the JSON to use. This is probably best demonstrated by the script below. It is probably not optimal to install the perl JSON::XS module on a production NetDRMS machine, however it should be considered when setting up a new machine. 
     37 
     38{{{ 
     39#!/usr/bin/perl 
     40 
     41use warnings; 
     42use strict; 
     43use Cpanel::JSON::XS; 
     44use Cpanel::JSON::XS::Type; 
     45 
     46use Devel::Peek; 
     47 
     48# Insofar as there is a "type" for perl variables, it's controlled 
     49# by a set of flags associated with the variable. The Devel::Peek 
     50# module lets you peek at the flags. I know that IV is the integer flag, 
     51# and NV is the float flag, but beyond that, it's kind of opaque. 
     52# Also the flags are prone to change between versions of perl, and 
     53# different versions of the JSON module may look at different flags. 
     54 
     55# Just to use Devel::Peek to look at some flags : 
     56my $var = "3"; 
     57Dump $var; 
     58print "^^^^^^^^^^\n"; 
     59 
     60my $x = $var + 0; 
     61Dump $var; 
     62print "^^^^^^^^^^\n"; 
     63 
     64$var += 0; 
     65Dump $var; 
     66print "^^^^^^^^^^\n"; 
     67 
     68# The bottom line is that it's probably bad practice, when encoding JSON, 
     69# to rely on the flags. You're better off setting up a hash table that 
     70# lets you set the type for each hash entry by name. What's below will 
     71# ALWAYS print 'num' as an integer, 'amount' as floating point and 
     72# 'name' as a string. It depends on having JSON:XS installed, which at the 
     73# moment I don't, and I don't want to risk installing it on a 
     74# running system. But when I set up a new system, I'm going to move 
     75# vso_jsoc_fetch.cgi to using this methodology. 
     76 
     77my $json =  
     78    Cpanel::JSON::XS->new->allow_nonref->allow_unknown->allow_blessed->pretty(1); 
     79 
     80# Associate the hash names with types. 
     81my $type = { 
     82             'num' => JSON_TYPE_INT, 
     83             'amount' => JSON_TYPE_FLOAT, 
     84             'name' => JSON_TYPE_STRING 
     85            }; 
     86 
     87# Put all entries in as integers 
     88my $data1; 
     89$data1->{"num"} = 1; $data1->{"amount"} = 2; $data1->{"name"} = 3; 
     90my $body1 = $json->encode($data1, $type); 
     91print $body1; 
     92 
     93# Put all entries in as floating point. 
     94my $data2; 
     95$data2->{"num"} = 4.0; $data2->{"amount"} = 5.0; $data2->{"name"} = 6.0; 
     96my $body2 = $json->encode($data2, $type); 
     97print $body2; 
     98 
     99# Put them all in as strings 
     100my $data3; 
     101$data3->{"num"} = "7.0"; $data3->{"amount"} = "8"; $data3->{"name"} = "Niles Oien"; 
     102my $body3 = $json->encode($data3, $type); 
     103print $body3; 
     104 
     105# No matter what, 'num' is printed as an integer, 'amount' as float, 'name' as string. 
     106# For vso_jsoc_fetch.cgi we want to use string for everything. 
     107# Niles March 2018. 
     108 
     109exit 0; 
     110}}} 
     111 
     112Niles Oien March 2018