Technical Teleconference, 2005 October 28

Agenda / Context

Igor requested a meeting to discuss how to use VOTable within VSO to make it easier for integration with other Virtual Observatories.

Date: Thu, 27 Oct 2005 19:18:50 -0700
From: Igor Suarez-Sola 
To: Joe Hourcle, Karen Tian, Alisdair Davey
Subject: VOTable for VSO


Hi all,

I want to clarify that the adoption of the VOTable format is for our 
metadata transactions and not scientific use (e.g. handling of fits 
files etc) and that the driver for this change is mainly performance. 
Although if we can be compliant with the VO all the better.

Also , hopefully, the adoption of the VOTable would be used internally 
by the CORE, GUI, Catalogs, CART and providers ...
See attached examples of VOTable.xml and VSOTable.xml (the only 
difference being in the TABLEDATA part). Both formats can be handle 
internally. However the interfaces can switch to one another and that'll 
be toggled by passing a flag in the request message.  The VSOTable gains 
a little bit of performance over the <TR><TD> one .. mainly in memory 
usage for large sets.

The VOTable structure used in its simplest form E.g. "VOTable.xml" is 
quite straight forward. However I see how it can get awkwardly complex 
as we get into rough territory like keeping state, linking results set 
with Queries etc. For instance for catalog entries in one ResultSet 
could yield a new query and to a new result set. Although in principle 
all that could be achieved by sticking to the VOTable schema. It can get 
very twisted and I don't quite see the advantage of that ... and that's 
why I think the VOTable format should be used only for the search 
results, while everything else internal to the VSO like states, link to 
other queries and links to result set are handle as we do now in our own 
format. (mmmh ... it now sounds like the obvious thing to do ...  
probably you'll be thinking what the hell is he talking about ... )


Anyway going back to the VOTable.xml example ... things I want you to 
pay attention to ...
1. We can have multiple RESOURCE elements
           We can take the RESOURCE element as the equivalent of one 
provider "search results"
           a) use the DEFINITIONS element to specify the provider and 
version maybe other information related to the provider??
           b) The TABLE element will be the actual result set
                  I- Use the PARAM element to pass information relative 
to the search result namely no_of_records_found and no_of_records_returned
                  II - A number of info messages (we can't put much 
granularity there but might suffice). If we want something more complex 
like  message type , level of  debugging etc we would have to bend the 
schema a little bit.

2. Alternatively we could have one RESOURCE and multiple TABLE elements 
where each TABLE will be the equivalent to one provider "search results" 
see attached "VOTable_2.xml" and the version and provider information 
goes as PARAM sub-elements. Note that the xml schema attached is version 
1.0 and doesn't seem to allow that, however a pdf I have on VOTable 1.1 does

3. The other issue is if you want to pass or keep the 'query input 
parameters' in the VOTable ... probably not but I thought of asking ...


Unfortunately I can't show you the code at this time. I thought I could 
give it to you today but I lost some time thinking in generalizing  the 
decoding of the VOTable xml. To be honest is quite straight forward as 
long as we keep it simple . I.e the VOTable for search results only. 
However the handling on the provider side and the integration with the 
DataProvider.pm is mostly done as there weren't much to change there. So 
I'm finalizing the xml decoder for the VOTable .. and that will take 
tomorrow and then over the weekend I wrap up the test files and 
documentation ... yes Joe you heard/read well 'I am DOCUMENTing'  as I 
go along ... ;-)  ... so by moday you could have a look at it .. sorry 
for the delay!


Talk to you tomorrow!

Igor

Decisions

Should we extend / modify VOTable?

        - Don't modify the structure.
        - Maybe add attributes if needed for extra VSO information
        - Try to find some VOTable folks, so we can ask why they laid it
          out how they did, and if they have any special intent for
          'resource', etc.
                - Igor might have local folks to ask
                - Joe might go to the SPASE meeting next week, and has
                  some local folks at GSFC to look up
        - We should try to make our format readable by VOTable tools.
                - Need to find what VOTable tools exist
                  http://viewcvs.cacr.caltech.edu/us-vo/viewcvs.cgi/contrib/VOTable/

        - Try to come up with something workable, and see how well it
          performs (speed, memory usage, compatability w/ existing tools),
          rather than try to worry about it being perfect in the first
          revision.

Speed / Memory optimizations

        - Joe will look at the serializer bits, and see if we can avoid
          deserializing the content in the Core, so it's passed through as
          an xml string
                - May have to parse for version compatability issues (eg,
                  deal with outdated providers and/or clients)
        - Get something working first, then find bottlenecks & optimize.