Tennessee Leeuwenburg wrote:
Hi John,
So long as you don't break simple stateless requests, I'm happy with
whatever you choose in order to provide stateful behaviour also -- I can
see the need for this now. The rest of this email contains my thoughts
about other ways to look at the problem.
I wonder why this is the case. What is the new file being added? Could
you please explain exactly what's being added to me? Why are new files
being added?
Its a realtime data feed from the IDD. In this particular dataset, its
satellite files that come in every 15 minutes, constituting a time series of
satellite images.
As for the deletion at 0Z, I would ask whether the request is for
"Latest" or for a specific date. I don't see why files for a specific
date would be removed, for example.
We only keep 7 days worth, so each night we delete the oldest day's worth.
However, I'll just assume you do need to do what you're doing for the
moment. I guess a session is the only way. However, not all clients are
going to be aware, so you might still need some way to handle things on
the server when people *do* ask for things without a session.
If the client doesnt assume any state, I can easily fulfill the request with
the dataset's state at the moment the request comes in.
What about introducing an "unlimited" vector into the aggregation? If
you joined along a new dimension, then requests (for example) the first
10 records would always come back with the same thing, even though more
data might now be available "at the other end". If you know your data is
going to be highly dynamic, then this doesn't seem unreasonable. You
might even implement a new kind of request for new additions to an old
aggregation.
yes, thats exactly the situation, i have an "unlimited" time dimension, and usually the
file is just growing, so the client with the "old" DDX doesnt get any bad data.
However, in principle the data might arrive out of order, and for sure we have
the problem when the files get deleted. Its really these cases, that happen
less frequently, that need special attention.
I may implement an operation that says "check to see if the dataset has grown along
the unlimited dimension".
I've quoted a bit from your other email, which prompted me to go an
re-examine this one.
>The problem is; everytime you do a data request, will you examine the
entire DDX and possibly some of the data like coordinate systems to make
sure nothing has changed?
I wouldn't bother usually, but if I was setting something up to track
the changes I *could* do it.
Yes, you could do it, but most clients wont.
I'm also thinking of the scenario where a
user is asking for the file as NetCDF and saving to local disk. If the
information only exists in the DDX, might not that be a problem?
Well, you culd ask for the data in a single request, but it will likely fail
because currently things get buffered in memory.
So if you break the data requests into seperate requests, you have the
possibility of things changing while youre bringing the data over.
Here's another option -- what if each DDX contained a unique identifier
-- such as the date and time to high precision? Further requests would
include this as a "currency" indicator. The server would then only
aggregate files which themselves have a creation date before that
indicator.
hmmm, sounds like a variation of the checksum idea. ill have to think about
that.
thanks for your ideas!
