Changes between Initial Version and Version 1 of jsocOffline

Timestamp:

09/10/20 15:34:35 (4 years ago)

Author:

niles

Comment:

--

jsocOffline

@@ +1 @@
+
+=== When data at the JSOC are offline ===
+
+Occasionally data will go offline at the JSOC. This may happen for a variety of reasons, but the end result is the same. Visiting the jsoc_fetch web site at the JSOC will return single character status code of something other than 'Y' ('Y' meaning that the data are online).
+
+The JMD will see that the data are offline and then make a note to re-try later, saving the sunum in its Derby database to facilitate retries. This often is not useful. If the data are offline for a period that is long enough that the original user request for the data is no longer relevant, the retries are a waste of computational cycles that can cause the JMD to become hopelessly overloaded over time.
+
+The Derby database (which can be accessed with the "ij" command at the line prompt as the production user) can be queried as to how many SUs it has in each state :
+
+{{{
+ij> select STATE, count(*) FROM drms.su_cache GROUP BY STATE ORDER BY STATE;
+STA&|2          
+----------------
+DONE|1
+OFLN|305
+}}}
+
+The following perl script, cleanDerbyOfflineStates.pl is something that could be called by cron to clean out data in an OFLN state from the Derby database :
+
+{{{
+#!/usr/bin/perl
+
+
+#### Things you may need to change - most can be ascertained from the JMD.cfg file
+
+my $DERBYDB="/internalRaid/derby/derbyDB";
+my $DERBYDBCONNECT="'jdbc:derby://localhost:31001/$DERBYDB'";
+my $IJ="/opt/JMD/bin/jmdij";
+my $DERBY_HOME="/opt/derby";
+my $JETTY_LOGS_DIR="/opt/JMD/log";
+my $PSQL="psql -At -U slony data";
+my $site = 'NSOCU';
+
+#### End of things you may need to change
+
+#############################################
+
+print `export DERBY_HOME=$DERBY_HOME; echo \"connect $DERBYDBCONNECT; DELETE from drms.su_cache WHERE STATE='OFLN'; exit;\" | $IJ`, "\n";
+
+exit 0;
+}}}
+
+
+This can be called by cron with this crontab entry :
+
+{{{
+*/2 * * * * /home/production/vsoDailyChecks/cleanDerbyOfflineStates.pl &>/home/production/vsoDailyChecks/cleanDerbyOfflineStates.log
+}}}
+
+This manipulation of the Derby database reflects the fact that when the JMD was written, it wasn't entirely clear what should be done with data in the OFLN state. Similar issues have arisen for data in the EROR state. It has since become clear that very often holding on to these data in the Derby database to retry data download just causes problems, so scripts like the above can be useful.
+
+