HAO search engine broken?

[rick]

I am not getting any record returns for any data selections from provider HAO. Any idea what's wrong?

[igor]

only happens at the NSO instance. It must be a registry problem I'll have a look

for the time being use

 http://vso.nascom.nasa.gov/cgi-bin/search?server=sdac

or

 http://vso.nascom.nasa.gov/cgi-bin/search?server=stanford2

[igor]

it's fixed at NSO now. My registry and the CVS registry are the same ... so I think I might have missed some email .. anyway I change the HAO version from 0.8 to 0.6 and it's getting data now

[joe]

I noticed some 'version' complaining from HAO last week. It might be more of the same (I guess this goes in the 'make error messages more obvious' pile).

By changing the version from 0.8 to 0.6 in the registry, GetData? will automatically fail. (because in 0.6, there was no GetData?, so the system was doing a check to make sure that it didn't call GetData? on systems that didn't have it implemented.)

I can strip that bit out of the core, as it's no longer needed anymore.

I'm getting inconsistent results back from HAO -- searching the same '4hr' time range, searching just for HAO gave 100 results; 10830Angstroms, 205 results; searching for HAO again -- 0 results (but _no_ message in the status window)

[rick]

I am still getting no results returned for a 10-year search of all HAO data (I started with smaller intervals of course). This time the instance is vso2.stanford.edu.

[igor]

a 10 - year search from HAO? you'll be lucky if you can get one full day without timing out. I think we really need these guys to tune up their database.

anyway if you click the search status "[show]" link on the left menu on your screen you should see a box comming up telling you it timed out. Let me know if it doesn't

[rick]

Give me a break - I didn't *start* with a 10-year search! I started with a day. And it didn't time out, the status bar says nothing to report. So then I went to 10 days - still nothing to report. Then 9 months. At that point the status is indeed a read timeout. But if I start at the time the registry says the data starts and turn up no records for successively longer periods, how am I supposed to know when the searching begins to fail (except that it takes a whole lot longer), and how am I supposed to actually find any data? I can take a MLSO/chp search from 1996.04.20 (nominal start) through 1996.08.31 and turn up no records. Proceeding incrementally, I find that the actual start of observations is 1996.09.29. So if the registry were cleaned up I would not have experienced this problem. But it's a bit of a problem if HAO cannot provide any useful information on a search extending over more than 7 days of actual data (~1250 records) when they do not even operate every day. As to the interface, it seems to me that the difference between a legitimate 0 records returned and one due to a server timeout or other error is sufficient that it should be automatically flagged, not require a query of search status as to why no records were found. 0 returns is a legitimate result in far too many cases to expect people to be suspicious and question it.

[joe]

I think that some modifications were made in how 1.2 presents information back to the searching user, that need to be undone.

We seem to be returning 0/0 data products from a provider when there was an error generated. This _may_ be my fault, as we don't actually connect directly to HAO these days -- we connect to vso1.nascom.nasa.gov, which proxies the query to HAO.

If HAO throws an error, it's possible that the proxy may end up masking it in the case of a transport error (and if the proxy times out before the core times out, it'd be a transport error)

Currently, however, when searching for large time ranges from MSLO, I get:

HAO

VSO-C500 :SOAP-ENV:Server.Transport : 500 read timeout

In the search status window.

I'm of the opinion that we should probably have the little box on the left changed from:

Search Status [show]

to something like:

Search Status [show] 3 messages.

So it's more apparent when an error has occurred.

[it's also possible that HAO is more likely to time out with the proxy, because VSO1 doesn't stream the data as it receives, I don't think. I'm not even sure if it's possible for me to do so.]

I'll contact Tony Darnell again, and see if I can get some time from his system administrator to update the code on their server, so I don't need to continue with the proxying.

[igor]

>Give me a break - I didn't *start* with a 10-year search!
>
ok, point taken! :-)

>  I started
>with a day.  And it didn't time out, the status bar says nothing to
>report.  So then I went to 10 days - still nothing to report.  Then
>9 months.  At that point the status is indeed a read timeout.  But
>if I start at the time the registry says the data starts and turn up
>no records for successively longer periods, how am I supposed to know
>when the searching begins to fail (except that it takes a whole lot
>longer), and how am I supposed to actually find any data?  I can take
>a MLSO/chp search from 1996.04.20 (nominal start) through 1996.08.31
>and turn up no records.  Proceeding incrementally, I find that the
>actual start of observations is 1996.09.29.  So if the registry were
>cleaned up I would not have experienced this problem.  But it's a bit
>of a problem if HAO cannot provide any useful information on a search
>extending over more than 7 days of actual data (~1250 records) when they
>do not even operate every day.
>
I'm not sure how the registry got that value and I don't know how their 
database is managed ... It could be that they  have that data but they 
move it to another table or something ... again I don't think there is 
much comunication between the HAO and us.

>  As to the interface, it seems to me that
>the difference between a legitimate 0 records returned and one due to
>a server timeout or other error is sufficient that it should be
>automatically flagged, not require a query of search status as to why
>no records were found.  0 returns is a legitimate result in far too
>many cases to expect people to be suspicious and question it.
>
Actually the logic originally flag this automatically but for some 
reason it stop doing it. I'll have to look at it again

[karen]

I did a quick count. Here is what I got:

chp: registry 19960420000000
     actual   19960929000000
dmp: registry 19940220000000
     actual   19970512000000 (still many no observation days after this)
mk4: registry 19981001000000
     actual   19981028000000

[rick]

Thank you, Karen; I hope that info will be reflected in the registry. One addendum: for the HAO->MLSO->chp observable:doppler data, the start date seems to be 1998.08.24. That is an isolate. The next dates for data are 1999.01.20 and 1999.12.30, after which it seems to start as a regular product. For data with such a high number of records per day and such a low frequency of days of oservation, we may need a more sophisticated registry for staged queries.