wiki:DataProviderMetaData

Version 1 (modified by ed, 7 years ago) (diff)

Initial version of DataProviderMetaData

Data Provider Meta-Data

2017 Dec 22

Five new tables have been added to the VSO database. These tables are used to store DB login information, names and argument lists of stored procedures used.

Table Column Function Comment
data_providers PROVIDER Data provider (eg. HAO)
SOURCE Source of data (eg. MLSO)
DATA_FEED_TYPE Type of feed (standalone or remote) Future Use
DETAIL_TREE_TYPE Key indicating type of detail page that is generated.
If populated, then an additional detail page is
available from the PROVIDER.
DP_METAKEY If DETAIL_TREE_TYPE is populated, then this Used to generate URL on-the-fly
column contains the meta-key that is parsed
and populated by the plugin named in
column DP_PLUGINNAME.
DP_TYPE Type of code used by the Data Provider.
"Python" means Data Provider is written in Python. Under Construction
NULL means Data Provider is written in Perl.
DP_SCRIPTNAME Python script name if DP_TYPE = "Python".
DP_PLUGINNAME Name of Plugin script used to generate the eg. hao_instrument
detail page using the meta-key in DP_METAKEY.
DISCRETE_DATE_RANGE Boolean variable used to mark Data Providers
whose data is spread out in discrete date ranges eg. Rocket flights such as EUNIS & VAULT
which may likely be missed during query selection.
DATA_PAGE_AVAILABLE Boolean variable marking Data Provider that has
a separate URL providing a calendar showing data
availability. Used together with DETAIL_TREE_TYPE,
DP_METAKEY & DP_PLUGINNAME to generate the URL.
ACTIVE Boolean variable marking Data Provider as active (1) or not (0). Future use
LAST_MOD_DATE Date the last modification to the row was done.
data_feed_credentials PROVIDER Data Provider
SOURCE Source of data
IGROUP Instrument Group Number Used to group instruments into sets
such as mappings to different databases.
LOGIN Database User login name Future Use
PW Database User password
HOST Hostname where database is located
PORT Port number to which database is connected eg. 3306 for MySQL
MECHANISM Type of DB interface code used Future Use
DB_TYPE Type of Database (eg. MySQL, PostgreSQL)
DB_NAME Name of Database the instrument group is located in
SP_CNAME Name of stored procedure used to count available data
SP_QNAME Name of stored procedure used to query data main sp used to return data
instrument_details PROVIDER Data Provider
SOURCE Source of data
INSTRUMENT Instrument (eg. K-Cor, mk4, chp)
SWITCHOVER_DATE Date when instrument data location changed
HOST_BEFORE Hostname where database was located before switchover
HOST_AFTER Hostname where database is located after switchover
DB_BEFORE Database name before switchover
DB_AFTER Database name after switchover
sp_details PROVIDER Data Provider
SOURCE Source of data
IGROUP Instrument group no.
DB_NAME Database name where stored procedures are located
SP_CNAME Name of stored procedure used to count available data eg. count_acos
SP_QNAME Name of stored procedure used to query data eg. query_acos
ARG_LIST Colon-delimited list of arguments used in stored procedures query fields listed without prefix, fields not in query have nq as prefix
SQL_LIST Colon-delimited list of SQL DB types in above ARG_LIST
schema_mapping PROVIDER Data Provider
SOURCE Source of data
INSTRUMENT Instrument (eg. chp) instrument whose DB schema has changed
and hence needs field mappings
DB_NAME Database name where field mappings are needed
MISSING_COLUMN Name of missing field in query as a column name Future Use
ORIGINAL_COLUMN Original field name in old DB that needs mapping
MAPPED_COLUMN New field in new DB that ORIGINAL_COLUMN is mapped to ORIGINAL_COLUMN --> MAPPED_COLUMN
ORIGINAL_LABEL Original label of instrument
MAPPED_LABEL New label of instrument ORIGINAL_LABEL --> MAPPED_LABEL
PER_YEAR Boolean variable marking if DB_NAME uses a per-year eg. 0 for MLSO
schema or not

Further details can be found in the attached document.

Attachments