Ticket #13 (closed problem: fixed)

Opened 13 years ago

Last modified 2 weeks ago

update MLSO data provider

Reported by: joe Owned by: ed
Priority: normal Milestone: Continuing Development
Component: DP:MLSO Version:
Severity: normal Keywords:
Cc:

Description

update MLSO data provider

Attachments

HAO_CustomErrorMsg.png (39.1 KB) - added by ed 3 weeks ago.
Screenshot of custom error message from MLSO on vso1 showing link to HAO calendar page

Change History

comment:1 Changed 13 years ago by joe

  • Owner changed from joe to anonymous
  • Status changed from new to assigned

I've talked to Tony Darnell about fixing the MLSO data provider, and hopefully, he'll have time at the end of this week for us to put in the new stuff, and I've prepped the code so it should be just a drop in...I hope

comment:2 Changed 13 years ago by joe

  • Owner anonymous deleted
  • Status changed from assigned to new

comment:3 Changed 13 years ago by joe

  • Owner set to joe
  • Status changed from new to assigned

comment:4 Changed 4 years ago by joe

Joe Gurman is in talks with Joan Burkpile, as they're in the process of getting another replacement.

Niles is nearby, and Alisdair often travels out there, so they'll likely take this over.

comment:5 Changed 4 years ago by joe

  • Owner changed from joe to niles
  • Status changed from assigned to new

comment:6 Changed 4 years ago by niles

  • Status changed from new to assigned

comment:7 Changed 4 years ago by niles

Talk this over with Alistair, try to make contact with HAO through Joan. Apparently someone called Leonard is leaving - Joe Gurman may be a good contact for checking up with Joan.

comment:8 Changed 3 years ago by joe

After way too long, we have now update the MLSO data provider so it's functional again.

OF course, now we're going to need to make a new one to search their currently active instruments, as they split it into two databases.

Still need to add 'near' and 'latest' support.

comment:9 Changed 4 months ago by ed

Sent the new code to Travis at HAO in Dec. 2017. He's moved it to his test environment. The new code allows querying of both HAO databases now.

It also adds support for the K-Cor data.

comment:10 Changed 4 months ago by ed

  • Owner changed from niles to ed

comment:11 Changed 5 weeks ago by ed

Got an account on 05/11/18 on the local HAO machine to help Travis get the new code installed. Completed the installation of the new code on 5/15/18.

Search for CHIP, DPM & MK4 data all work in the HAO test env. K-Cor search did not return data as expected. I'll check to see if the DB schema for K-Cor has changed.

New search mechanism uses stored procedures in DB instead of hard-coded in the Perl pkg MLSO.pm

Also, all instruments and DBs are handled by the single pkg MLSO.pm and registry entry hao.xml

comment:12 Changed 5 weeks ago by ed

K-Cor had a small update to it's DB. Once the stored procedure was changed to use the table "kcor_img" instead of "kcor_file" and the data in "kcor_img" was restored, the K-Cor query returned results as well.

Now updating vso1 to have the new HAO registry file and custom error message code so the same tests can be repeated.

comment:13 Changed 4 weeks ago by ed

As part of the move of the new code to vso1 for testing, I found that the MySQL/MariaDB on vso1 had 3 broken system tables in database "mysql". The key table was the "proc" table which contains the stored procedures for the server. The DB is also missing 5 other system tables entirely. The other 2 tables: "db" and "user" apparently were not broken quite enough to prevent the DB from operating.

Table "proc" is now fixed and stored procedures have been added to vso1:

MariaDB [(none)]> show procedure status; +-------+---------------------+-----------+----------------+---------------------+---------------------+---------------+---------+----------------------+----------------------+--------------------+ | Db | Name | Type | Definer | Modified | Created | Security_type | Comment | character_set_client | collation_connection | Database Collation | +-------+---------------------+-----------+----------------+---------------------+---------------------+---------------+---------+----------------------+----------------------+--------------------+ | mysql | get_sizes | PROCEDURE | root@localhost | 2018-05-24 13:36:49 | 2018-05-24 13:36:49 | DEFINER | | utf8 | utf8_general_ci | latin1_swedish_ci | | vso | read_all_dps | PROCEDURE | root@localhost | 2018-05-24 13:39:21 | 2018-05-24 13:39:21 | DEFINER | | utf8 | utf8_general_ci | latin1_swedish_ci | | vso | read_dp_credentials | PROCEDURE | root@localhost | 2018-05-24 13:39:21 | 2018-05-24 13:39:21 | DEFINER | | utf8 | utf8_general_ci | latin1_swedish_ci | | vso | read_mapping | PROCEDURE | root@localhost | 2018-05-24 13:39:21 | 2018-05-24 13:39:21 | DEFINER | | utf8 | utf8_general_ci | latin1_swedish_ci | | vso | read_sp_details | PROCEDURE | root@localhost | 2018-05-24 13:39:21 | 2018-05-24 13:39:21 | DEFINER | | utf8 | utf8_general_ci | latin1_swedish_ci | | vso | read_swover_dates | PROCEDURE | root@localhost | 2018-05-24 13:39:21 | 2018-05-24 13:39:21 | DEFINER | | utf8 | utf8_general_ci | latin1_swedish_ci | +-------+---------------------+-----------+----------------+---------------------+---------------------+---------------+---------+----------------------+----------------------+--------------------+ 6 rows in set (0.00 sec)

The rest of the code needed to test HAO will now be added.

Since the issue of the MySQL/MariaDB system tables only affects SDAC, I'm not sure if a ticket is warranted. I'll fix the rest of the tables only this ticket is closed.

Side Note: The issue with /var being at 92% filled on vso1, resulting in other error messages in the DB log file has also be fixed (Marc moved some other files from /var, freeing up space).

Changed 3 weeks ago by ed

Screenshot of custom error message from MLSO on vso1 showing link to HAO calendar page

comment:14 Changed 3 weeks ago by ed

Added the rest of the new code to vso1 and tested both the stored procs on vso1 for CHP data for a date range that returns data from both DBs (ACOS & MLSO). Also tested a data range that has no data. Displayed new custom error message for HAO that has a link, dynamically rendered from the HAO plugin, that points to the calander page for the instrument searched (CHP).

Barring problems from dynamic loading (see below), vso1 is ready for Travis to test on.

NOTE: Ran into a problem in getting the HAO plugin dynamically loaded on vso1. What worked on my test environment did not work on vso1. Switched to a block version of eval to catch the error. Something is up with Perl finding and loading the CPAN module DateTime? on vso1. Will test on another linux machine to see if the issue can be reproduced. New dynamic loading of the DP module from UserInterface::GQuery may be getting messed up from the parent module Grid's use of AUTOLOAD. This issue seems specific to vso1, but it may be reproducible in other set-ups.

comment:15 Changed 2 weeks ago by ed

Added the key code to vso today (6/7/18) so CHP, K-Cor and the rest of the HAO instruments are now active and searchable from SDAC. The new code will search both HAO DBs depending on the query parameters for CHP, MK4 and DPM, and the MLSO DB for K-Cor.

A registry entry for CoMP is in place as well, just commented-out until the repair/upgrade to the instrument is completed later in 2018.

List of Committed Files:

MLSO DP specific: (relative path is in CVS)


1) DataProviders/MLSO/MLSO.pm 2) DataProviders/MLSO/sqls/acos_sps.sql --stored procedures (only used at HAO)

mlso_sps.sql

3) Registry/lib/Physics/Solar/VSO/Registry/data/hao.xml --new registry file

4) Config/bin/plugins/PLUGINS/HAO.pm --new plugin for HAO (used only for error messages)

General code: --supports meta-data tables and custom error messages


5) Config/lib/Physics/Solar/VSO/Config.pm --reads in meta-data tables

6) Config/lib/Physics/Solar/VSO/sqls/vso_sps.sql --stored procedures for meta-tables in VSO

7) Utils/lib/Physics/Solar/VSO/Utils.pm --new method to call a plugin

8) DataProviders/Inheiritable/Perl?/lib/Physics/Solar/VSO/DataProvider.pm --new generic custom

message method

9) UserInterfaces?/lib/Physics/Solar/VSO/UserInterface/GQuery.pm --call to new custom error

if one exists

10) FormBuilder?/cgi/show_details.pl --details page update to permit use of plugin if one

exists.

I'll close this ticket out now that the core new HAO code is active. The custom error message code needs a little more looking into to make sure it will work at NSO and elsewhere.

I'll create new ticket for the custom error message code and push that update out on Monday, together with the registry update for HAO.

comment:16 Changed 2 weeks ago by ed

  • Status changed from assigned to closed
  • Resolution set to fixed
Note: See TracTickets for help on using tickets.