== Data Provider Meta-Data ==

2017 Dec 22

Five new tables have been added to the VSO database. These tables are used to store DB login information, names and argument lists of stored procedures used.


  ||   ''Table''           ||        ''Column''          ||    ''Function''                                  ||    ''Comment''   ||
  ||=''' data_providers'''=||         PROVIDER           ||   Data provider  (eg. HAO)                       ||                  ||
  ||                       ||         SOURCE             ||   Source of data (eg. MLSO)                      ||                  ||
  ||                       ||         DATA_FEED_TYPE     ||   Type of feed (standalone or remote)            ||  ''Future Use''  ||
  ||                       ||         DETAIL_TREE_TYPE   ||   Key indicating type of detail page that is generated.  ||          ||
  ||                       ||                            ||   If populated, then an additional detail page is        ||          ||
  ||                       ||                            ||   available from the PROVIDER.                   ||                  ||
  ||                       ||         DP_METAKEY         ||   If DETAIL_TREE_TYPE is populated, then this    ||=Used to generate URL on-the-fly=||              
  ||                       ||                            ||   column contains the meta-key that is parsed    ||                  ||
  ||                       ||                            ||   and populated by the plugin named in           ||                  ||
  ||                       ||                            ||   column DP_PLUGINNAME.                          ||                  ||
  ||                       ||         DP_TYPE            ||   Type of code used by the Data Provider.        ||                  ||
  ||                       ||                            ||   "Python" means Data Provider is written in Python. ||=''Under Construction''=||
  ||                       ||                            ||   NULL means Data Provider is written in Perl.    ||                  ||
  ||                       ||         DP_SCRIPTNAME      ||   Python script name if DP_TYPE = "Python".       ||                  ||
  ||                       ||         DP_PLUGINNAME      ||   Name of Plugin script used to generate the      ||  eg. hao_instrument  ||
  ||                       ||                            ||   detail page using the meta-key in DP_METAKEY.   ||                      ||
  ||                       ||     DISCRETE_DATE_RANGE    ||   Boolean variable used to mark Data Providers    ||                      ||
  ||                       ||                            ||   whose data is spread out in discrete date ranges  ||=eg. Rocket flights such as EUNIS & VAULT=|| 
  ||                       ||                            ||   which may likely be missed during query selection.||                   ||
  ||                       ||     DATA_PAGE_AVAILABLE    ||   Boolean variable marking Data Provider that has    ||                   ||
  ||                       ||                            ||   a separate URL providing a calendar showing data   ||                   ||
  ||                       ||                            ||   availability. Used together with DETAIL_TREE_TYPE, ||                   ||
  ||                       ||                            ||   DP_METAKEY & DP_PLUGINNAME to generate the URL.    ||                   ||
  ||                       ||          ACTIVE            ||   Boolean variable marking Data Provider as active (1) or not (0). ||=''Future use''=||
  ||                       ||      LAST_MOD_DATE         ||   Date the last modification to the row was done.      ||                 ||
  ||                       ||                            ||                                                        ||                 ||
  ||='''data_feed_credentials'''=||    PROVIDER          ||   Data Provider                                        ||                 ||
  ||                             ||    SOURCE            ||   Source of data                                       ||                 || 
  ||                             ||    IGROUP            ||   Instrument Group Number                              ||=Used to group instruments into sets=||
  ||                             ||                      ||                                                        ||=such as mappings to different databases.=||
  ||                             ||    LOGIN             ||   Database User login name                             ||     ''Future Use''                       ||
  ||                             ||    PW                ||   Database User password                               ||                                          ||
  ||                             ||    HOST              ||   Hostname where database is located                   ||                                          ||
  ||                             ||    PORT              ||   Port number to which database is connected           ||    eg. 3306 for MySQL                    ||
  ||                             ||    MECHANISM         ||   Type of DB interface code used                       ||    ''Future Use''                        ||
  ||                             ||    DB_TYPE           ||   Type of Database (eg. MySQL, PostgreSQL)             ||                                          ||
  ||                             ||    DB_NAME           ||   Name of Database the instrument group is located in  ||                                          ||
  ||                             ||    SP_CNAME          ||   Name of stored procedure used to count available data||                                          ||
  ||                             ||    SP_QNAME          ||   Name of stored procedure used to query data          ||    main sp used to return data           ||
  ||                             ||                      ||                                                        ||                                          ||
  ||='''instrument_details'''=||       PROVIDER          ||   Data Provider                                        ||                                          ||               
  ||                             ||    SOURCE            ||   Source of data                                       ||                                          ||
  ||                             ||    INSTRUMENT        ||   Instrument (eg. K-Cor, mk4, chp)                     ||                                          ||
  ||                             ||    SWITCHOVER_DATE   ||   Date when instrument data location changed           ||                                          ||
  ||                             ||    HOST_BEFORE       ||   Hostname where database was located before switchover||                                          ||
  ||                             ||    HOST_AFTER        ||   Hostname where database is located after switchover  ||                                          ||
  ||                             ||    DB_BEFORE         ||   Database name before switchover                      ||                                          ||
  ||                             ||    DB_AFTER          ||   Database name after switchover                       ||                                          ||
  ||                             ||                      ||                                                        ||                                          ||
  ||='''sp_details'''=||               PROVIDER          ||   Data Provider                                        ||                                          ||
  ||                             ||    SOURCE            ||   Source of data                                       ||                                          ||
  ||                             ||    IGROUP            ||   Instrument group no.                                 ||                                          ||
  ||                             ||    DB_NAME           ||   Database name where stored procedures are located    ||                                          ||
  ||                             ||    SP_CNAME          ||   Name of stored procedure used to count available data||       eg. count_acos                     ||
  ||                             ||    SP_QNAME          ||   Name of stored procedure used to query data          ||       eg. query_acos                     ||
  ||                             ||    ARG_LIST          ||   Colon-delimited list of arguments used in stored procedures ||  query fields listed without prefix, fields not in query have ''nq'' as prefix  ||
  ||                             ||    SQL_LIST          ||   Colon-delimited list of SQL DB types in above ARG_LIST ||                                        ||
  ||                             ||                      ||                                                          ||                                        ||
  ||='''schema_mapping'''=||           PROVIDER          ||   Data Provider                                          ||                                        ||
  ||                             ||    SOURCE            ||   Source of data                                         ||                                        ||
  ||                             ||    INSTRUMENT        ||   Instrument (eg. chp)                                   ||=instrument whose DB schema has changed=||
  ||                             ||                      ||                                                          ||=and hence needs field mappings=||
  ||                             ||    DB_NAME           ||   Database name where field mappings are needed          ||                                        ||
  ||                             ||    MISSING_COLUMN    ||   Name of missing field in query as a column name        ||    ''Future Use''                      ||
  ||                             ||    ORIGINAL_COLUMN   ||   Original field name in old DB that needs mapping       ||                                        ||
  ||                             ||    MAPPED_COLUMN     ||   New field in new DB that ORIGINAL_COLUMN is mapped to  ||  ORIGINAL_COLUMN --> MAPPED_COLUMN     ||
  ||                             ||    ORIGINAL_LABEL    ||   Original label of instrument                           ||                                        ||
  ||                             ||    MAPPED_LABEL      ||   New label of instrument                                ||  ORIGINAL_LABEL --> MAPPED_LABEL       ||
  ||                             ||    PER_YEAR          ||   Boolean variable marking if DB_NAME uses a per-year    ||  eg. 0 for MLSO                        ||
  ||                             ||                      ||   schema or not                                          ||                                        ||
                                

Further details can be found in the attached document.