Subscription and slony logs

Subscription

When a site 'subscribes' to a series, the process is initiated by the site sending a message to JSOC telling them that they're interested in that series. JSOC sends back the commands to build the table, and populate it to the current slony checkpoint. Then JSOC adds the table name to their list of what's being subscribed to by that site.

The only way to know if you're subscribed is to actually look in the lists at JSOC, which are named according to the convention:

/data/pgsql/slon_logs/live/etc/(site).lst

Slony logs

At JSOC, the logs are written to the directory :

/data/pgsql/slon_logs/live/site_logs/(site)

The slony logs have names according to the convention :

slony1_log_2_00000000000000690325.sql

Where the number 690325 is a counter that increments for each file. The term "log" is perhaps somewhat misleading with respect to these files, as they are not warnings or errors printed as the database operates. Rather, they are lists of database operations that have taken place at JSOC. If other sites are to keep in sync with JSOC, they must apply the same operations to their local databases. A typical log file may have entries like :

insert into "hmi"."rdvpspec_fd05" ("recnum","sunum","slotnum","sessionid","sessionns","ln_source_isset","ln_source_carrrot","ln_source_cmlon_index","ln_source_lonhg_index","ln_source_lathg_index",
"ln_source_loncm_index","cparms_sg000","logp_bzero","logp_bscale","carrrot","cmlon","lonhg","lathg","loncm","crpix1","crpix2","cdelt1","cdelt2","cdelt3","delta_k","delta_nu","d_omega","module",
"source","input","created","bld_vers","log_base","datamin","datamax","apode_f","apod_min","apod_max","cmlon_index","loncm_index","lathg_index","lonhg_index","sg_000_file","history") values
('9919708','528962345','0','27939659','su_rsb','1','2146','32','48','41','-32','','-8.48432540893554688','0.000635004483736478385','2146','200','120','-12.5','80','64.5','64.5',
'0.101023711','0.101023711','0.000181805124','0.101023711','28.9351845','0.000181805124','pspec3 v 1.1','hmi.rdVtrack_fd05[2146][200][120.0][-12.5][-80.0]',
'hmi.rdVtrack_fd05[2146][200]','1171214459','V8R2','2.71828182845904509','-29.1219711','12.1533213','0.96875','0.9765625','1','32','-32','41','48','logP.fits','');

insert into "hmi"."rdvpspec_fd05" ("recnum","sunum","slotnum","sessionid","sessionns","ln_source_isset","ln_source_carrrot","ln_source_cmlon_index","ln_source_lonhg_index","ln_source_lathg_index",
"ln_source_loncm_index","cparms_sg000","logp_bzero","logp_bscale","carrrot","cmlon","lonhg","lathg","loncm","crpix1","crpix2","cdelt1","cdelt2","cdelt3","delta_k","delta_nu","d_omega","module",
"source","input","created","bld_vers","log_base","datamin","datamax","apode_f","apod_min","apod_max","cmlon_index","loncm_index","lathg_index","lonhg_index","sg_000_file","history") values
('9919709','528962345','1','27939659','su_rsb','1','2146','32','48','42','-32','','-9.26125526428222656','0.000644568281907301633','2146','200','120','-15','-80','64.5','64.5',
'0.101023711','0.101023711','0.000181805124','0.101023711','28.9351845','0.000181805124','pspec3 v 1.1','hmi.rdVtrack_fd05[2146][200][120.0][-15.0][-80.0]',
'hmi.rdVtrack_fd05[2146][200]','1171214461','V8R2','2.71828182845904509','-30.2097244','11.6872139','0.96875','0.9765625','1','32','-32','42','48','logP.fits','');

After a while (daily?), these slony log files are archived into a bundle, named like so :

slony_logs_688467-689890.tar.gz

And then after a longer period of time - two weeks? - these archives of slony log files are deleted. It is critical that the slony logs are applied at remote sites before they deleted at JSOC.

Slony and the DRMS database

The small table _jsoc.sl_archive_tracking contains information about the most recent slony log ingested :

prompt> psql nso_drms

nso_drms=# select * from _jsoc.sl_archive_tracking;
 at_counter |         at_created         |        at_applied
------------+----------------------------+---------------------------
     886400 | 2014-06-27 09:36:17.452634 | 2014-06-27 16:37:03.71184

Subscription at your site

At your site will be a file named subscribe_series - it will be in the same directory as the get_slony_logs.pl script (at NSO it is in /datarea/production/subscribe_series).

To subscribe to a series you need to edit the config file - etc/subscribe_list.cfg at NSO - and then run it like so :

./subscribe_series ./etc/repclient.live.cfg ./etc/subscribe_list.cfg ~/.ssh-agent_rs.sh

The format of etc/subscribe_list.cfg is :

hmi.rdvflows_fd30_frame subscribe
hmi.rdvflows_fd15_frame subscribe
hmi.fsvbinned_nrt subscribe
hmi.rdVfitsc_fd05 subscribe
hmi.rdVfitsc_fd15 subscribe
hmi.rdVfitsc_fd30 subscribe
hmi.rdVfitsf_fd05 subscribe
hmi.rdVfitsf_fd15 subscribe
hmi.rdVfitsf_fd30 subscribe

And at NSO the complete listing of ./etc/repclient.live.cfg is :

# This template file contains configurable parameters used by replication client code. The client
# code includes subscribe_series, run_get_slony_logs.sh, and get_slony_logs.pl.
#
# Each NetDRMS site using the replication code should copy this file to a writeable location
# and edit it as appropriate. Each of the client-side replication scripts sources this configuration 
# file to set environment variables needed by that script.

#######################################################
# client-side-wide configurable parameters
#######################################################

node=nso2

# Client-server communication
kRSServer=solarport.stanford.edu
kRSTriggerDir=/data/pgsql/slon_logs/live/triggers
kRSUser=jsocexp
kRSPort=55000
ingestion_path=/home/production/subscribe_series/trigger

# Client database information
pg_host=vso2.tuc.noao.edu
pg_port=5432
pg_user=slony
pg_dbname=nso_drms
slony_cluster=jsoc


#######################################################
# subscribe_series configurable parameters
#######################################################

# file paths
kLocalLogDir=/home/production/subscribe_series/log
kLocalWorkingDir=/home/production/subscribe_series/work
kSQLIngestionProgram=/home/production/subscribe_series/bin/get_slony_logs.pl
kDeleteSeriesProgram=/opt/drms/bin/linux_x86_64/delete_series

attempts=2000

# XXX This is not correct - archive, retention, and tapegroup should be per-series, not global.
archive=0
retention=21
tapegroup=1


#######################################################
# get_slony_logs.pl configurable parameters
#######################################################

scp_cmd=/usr/bin/scp
ssh_cmd=/usr/bin/ssh
rmt_slony_dir=/data/pgsql/slon_logs/live/site_logs
slony_logs=/home/production/subscribe_series/slon_logs
PSQL=/export/vso2.0/usr/bin/psql
email_list=igor@noao.edu