Ticket #273 (closed problem: fixed)

Opened 5 months ago

Last modified 5 months ago

fix_mission_fits.pl debug

Reported by: jacob Owned by: jacob
Priority: normal Milestone: Continuing Development
Component: DataProvider Version:
Severity: minor Keywords: data provider


This ticket contains debug information regarding running fix_mission_fits.pl, the perl script which creates and populates a mysql database defined by the mission schema and config files, and corrects the provided .fits files. See the perldoc for more detailed usage information:

perldoc fix_mission_fits.pl

Env. Variables:

The script itself uses several environment variables containing filepaths to various required resources. The following environment variables must be exported prior to script execution:

CONFIG_BASEPATH -points to the directory containing the config and schema files for the database being set up. Files must be named {mission}_schema.tmpl and {mission}_config.tmpl.

SITE_PERL -points to an installation of perl on the filesystem. It's probably best to use a version of perl install with brew, activeperl, etc. instead of the system installation.

MISSION_CODES -points to the location of a number of resources, such as missionDB.pm, used by the script. Likely the same directory which houses fix_mission_fits.pl.

Additional environment variables may be used to expedite access to the data directory (see typical usage in the perldoc)

Dependencies: There are about a dozen perl modules in at the top of fix_mission_fits.pl that must be installed prior to successful execution, though there are also a few embedded in the modules which the main script uses. I would recommend installing the ones which are obviously listed in the use statements at the top of the main script and then calling the script and figuring out the few that remain from the error messages. This script draws from several resources and it would likely prove more time consuming to try and locate those use statements within all the files that it draws from.

Mysql version issues:

For reference, my experience with this installation has been with mysql 5.7.21. From what I gather, pre-5.7 distributions may or may not prone to the same problems, these are just the specific speed bumps I ran into during my installation and how I solved them.

1.: Data type byte limits: When defining stored procedures and functions in mysql, keep in mind the byte limits for each data type. VARCHARs, for example, are limited to 21845 bytes total (21844 available for use) when encoded with UTF-8. This can be problematic when parsing very large datasets, so if 21844 bytes is not sufficient for your purpose, use a TEXT or BLOB datatype.

2.: Empty string is not interpreted as 0 in mysql 5.7.x Relatively self-explanatory issue once you determine what is wrong, and easily catchable by a quick one-liner. Apparently there is a variable in the mysql configuration files called sql_mode that you can alter to prevent this from tossing an error, I found no elegant way to alter that variable on Mac OSX and I can't think of any reason why it would be better than just catch that blank string with an if.

I will close this ticket once I update the perldoc for fix_mission_fits.pl

Change History

comment:1 Changed 5 months ago by jacob

  • Status changed from new to closed
  • Resolution set to fixed

fix_mission_fits.pl committed with updated documentation

Note: See TracTickets for help on using tickets.