== General Description of NetDRMS software and discussion of some issues ==

=== 1.0 Introduction ===

The Solar Dynamics Observatory (SDO) spacecraft, see https://sdo.gsfc.nasa.gov was launched in 2010. As of this time (April 2021) it is still observing the sun with several instruments, including the Atmospheric Imaging Assembly (AIA) and the Helioseismic and Magnetic Imager (HMI). The data rate is prolific, with approximately 70,000 images being acquired daily. These data are stored, and served out from, The Joint Science Operations Center (JSOC) at Stanford University, see http://jsoc.stanford.edu where they are managed by software developed at Stanford known as NetDRMS. This document was written to give a broad overview of NetDRMS, and present some issues that the system has.

SDO data are stored on disk at the JSOC in the Flexible Image Transport System (FITS) file format. This format supports storage of both the image data and meta data header information. As stored on disk, the image data have only very minimal header information, such as the dimensions of the image. The information needed to provide a complete FITS header is stored in a separate database.

When a user requests an image, possibly through a Common Gateway Interface (CGI) script available on the internet, the NetDRMS software combines the stored FITS image data with the information in the database to produce a FITS file with a complete FITS header that has all relevant meta data pertinent to the image. This process is know as '''exporting''' the image. This approach allows for edits to the header information to take place without having to overwrite existing FITS files.

SDO data are organized by '''series''' with a series typically being data from a certain instrument that has been processed in a certain way. Popular series include '''aia.lev1''' (level one AIA data) '''hmi.m_45s''' (HMI 45 second magnetograms) and '''hmi.ic_45s''' (HMI 45 second continuum images).

There is a difference in the way that AIA and HMI data are organized.


=== 2.0 Remote NetDRMS sites ===

To try to spread the load of network requests, in addition to the NetDRMS system at the JSOC in California, the Virtual Solar Observatory (VSO) operates remote NetDRMS sites. At the time of writing there are two remote sites, one at the National Solar Observatory (NSO) in Boulder, Colorado and one at the NASA Goddard Solar Data Analysis Center (SDAC) located in Greenbelt, Maryland. While these remote sites do have significant storage attached to them, they do not store the complete set of SDO images on disk as the JSOC does. Rather, they store a buffer of SDO data, with data aging off over time. The remote sites do maintain a database that is a mirror of the complete database at the JSOC used to generate FITS header information for exporting FITS files.

Thus the remote sites have the complete database of information used to export FITS data to users, but only a subset of stored image data. Image data can be copied from the JSOC to the remote sites in one of two ways.

The first way that data are copied is known as a '''mirror''' copy. This is simply an attempt to mirror recent data at the remote sites on the assumption that recent data will be more likely to be downloaded by users, and so it is desirable to have it "staged" at the remote sites for users to download.

The second way that data are copied from the JSOC to the remote sites is known as a '''user''' copy. This occurs when a user requests data from a remote site that is not in the remote site buffer of data. In this case, a copy is initiated, and the user who made the request is then obliged to wait while the copy of the data from the JSOC to the remote site takes place, and then wait for the export of the FITS file(s) requested to take place before the download can begin.