Initial Draft: 2017-08-05
Last Revised: 2017-08-05
Author: Dennis Heimbigner, Unidata
Table of Contents
- Initial Event Set
- Topic Space Design
- Relation to LDM
- Performance Costs
- Appendix A. Miscellaneous Notes
Periodically some of the Thredds servers run by Unidata get seriously overloaded. One cause is because external users poll the Thredds server to see what has changed. If the polling rate is too high then the performance of the Thredds server can seriously deteriorate.
I am proposing here to mitigate this problem by allowing Thredds servers to generate events that signal changes that might be of interest to users. Then, instead of polling, these users can watch for specific changes events and use that information to update their local databases (or whatever).
The cost tradeoff for Unidata is the cost of periodic "hammering" versus the maintainence of an event server to distribute change events to users.
Looking ahead, it is also possible that this proposal can facilitate inter-server communications. This means that multiple Thredds servers could communicate useful information. This is speculative for now, but should be kept in mind.
I are proposing a pretty standard publish-subscribe system for use by a Thredds server. In this architecture, there are hooks in various places in the Thredds code that send short messages to a separate "broker" server.
On the client (user) side, each client registers with the broker to tell it the kinds of messages in which it is interested.
So the flow is:
- The server generates a change message
- The message is received by the broker
- The broker forwards the message to all clients that are registered as interested in that kind of message.
In order to be useful to Unidata and its community, I require certain capabilities for the event system.
In event systems, there are typically two ways to identify messages: by queue and by topic.
A topic based message is one that has an associated structured string used to classify the message. Often, the structure of the string is a tree represented by the format field.field.field... where each field is some identifier. This format can be used, for example, to mark the message as referring to some file in a tree structured file system. Thus a file /f1/f2/f3 might be mapped to the topic f1.f2.f3.
A queue-based identification is one in which a message is sent to a specific named queue. It is isomorphic to a topic system as far as a sender is concerned because each distinct topic string can be the name of a queue. I will not consider queue-based system further.
On the client side, the client must be able to register for messages by specifying a pattern indicating the message topics in which it is interested. It is desirable to allow a client to register for a number of different topics by specifying a pattern containing wildcards (as is common in e.g. Unix file specifications).
If, for example, our client was interested in events about all files within /f1/f2, it should be possible to specify a topic pattern such as "f1.f2.*".
Suppose a client is not active or not registered with the broker at the time an event is received by the broker. If later the client registers, it will not see that previously generated message. This is a problem because a client will be forced to again access the Thredds server to see what happened while it was offline.
To deal with this problem, I require that our broker support "durable" messages. In the event community, this means that the broker will cache messages for some period of time. When a client registers, it will receive any cached messages that match its pattern. Supporting durability is tricky because of issues such as message retention policies and message duplication. Nontheless, this is an essential requirement in order to avoid polling as much as possible.
In this context, persistence means that cached messages are maintained even if the broker crashes. It is closely related to durability if the durable messages are stored in some kind of file-based database system. I do not require persistence, although it may come for free with some brokers.
Transactions (Not Required)
Transactions in event systems are similar to those of database systems. It essentially means that a message is guaranteed to be delivered or it appears as if it is not delivered at all. It is unlikely that transactions will be required for our event system.
The initial set of events of interest will be catalog change events. Specifically these. * Insert - when a catalog has a new entry * Delete - when a catalog has an existing entry removed * Modify - when a catalog has an existing entry modified.
Modification is tricky since it effectively tested by looking at the modification date for a file. This may not be an actual change. Note also that it may turn out that modifications will actually look like a deletion followed by an insertion.
Note that modification false positives are generally acceptable if not too frequent. This is because the cost to the client is that it retrieves a catalog entry that has in fact not been modified.
False negatives are much less acceptable. This will occur if an insertion, deletion, or modification occurs but no corresponding event is generated. If a client cannot trust the server in this regard, then the event system will not be used.
Note also that change events should also apply to the creation of new catalogs as well as the files within a catalog.
Topic Space Design is an important aspect of this proposal. That is, we need to set of a topic tree such that clients can specify what they want with a reasonable degree of specificity.
From LDM, we know that this is important in order to avoid clients just subscribing to everything. I would hope that using a wildcard system -- as opposed to e.g. regular expressions -- is sufficiently simple that clients will not be tempted to ask for everyting. Realistically, this is probably a forlorn hope because it is likely that pollers want to know everything about a specific server.
My initial thought is that the root of our topic space is "Unidata.Thredds". From there, I would like to specify a particular server via a DNS name + port. There is a problem since DNS names contain dots. It may be necessary to use encoding, the url %xx for example, to change the dots and semicolons in the DNS name So one might say "Unidata.Thredds.motherlode%2eucar%2eedu%3a8080."
From there, the obvious choice is to encode the catalog path as the rest of the topic. So, for example:
The character escaping issues needs some thought.
Currently, Thredds does not immediately detect changes to its underlying file system. Rather, it dynamically rebuilds some catalogs when accessed. The dynamically generated catalog will, of course, reflect changes to that catalog since it was last accessed.
John Caron left nascent code (CatalogWatcher) in Thredds to actually operate at the time a change occurred (as opposed to when a catalog is retrieved). This is more or less the eager vs lazy issue.
So, in order to make this proposal work, I need to do (at least) the following. * Complete the existing code pieces such as CatalogWatcher * Add code to convert a file system reference to a catalog system url. * Add an event sender to Thredds
This appears to be a straightforward set of modifications. Of course Murphy's law says that I am forgetting something.
My current broker of choice is Apache ActiveMQ. But I cannot help but notice that LDM functionality is related to this proposal. So is fair to ask if LDM could be adapted to serve as the broker. Here are some issues that would need to be considered. Note that my knowledge of the current capabilities of LDM may be out of date.
- Message Size: LDM ships files, not short messages. In this sense, LDM is overkill.
- Volume: my speculation is that the volume of small messages would not be all that large; it might even be similar to the volume of distinct files shipped by LDM
- Multiple Languages: whatever broker we use, it must be possible to write clients in a variety of programming languages: C, Python, Java at least.
While it would be nice to make use of other Unidata technology, I currently do not plan to pursue this path. As noted, my current target is an ActiveMQ broker with JMS publishers and subscribers.
There is a performance cost in having the Thredds server generate events. But it is difficult to assess these costs without knowing how many events are being generated.
It should be possible to experimentally measure the number of events by adding counters to the CatalogWatcher code and running it for an extended period of time. Periodically, the counters can be dumped to the logs.
Ideally we would have R X 3 x 24 x 31 counters so we could compute the number of events per hour, day, week, and month per catalog root (assuming R roots). This is not a large amount of space . So at the end of every hour, day, and month, a log entry would be generated with a terse listing of the counters. The counters would then be reset.
The big problem is that these counters need to be executed on one of the Unidata productions systems (aka motherlode) in order to get realistic numbers. It is not clear if this is possible.
Note that the measurements would be taken over all the catalog roots. It is possible that other parts of the file system such as the GRIB indices would also need to be included.
The question is: how open should our broker be to arbitrary clients? The short answer is that it probably should have the same access controls as the associated Thredds server. Again, it is not clear what the requirements here should be.
The original motivation for this proposal was to try to mitigate clients hammering our production servers. The idea is that if they have access to changes events, they do not ned to poll our servers (or least quite so often).
Will it work? It depends on several factors:
- Is is easy for clients to use?
- Are they willing to change?
- Is it sufficiently reliable that clients will not feel they are loosing events?
Multiple Servers Per Broker
The above discussion assumes that there is the server - broker association is one to one. Other arrangements are possible such as multiple servers sharing a single broker or a server using multiple brokers. The relative merits of these alternatives is unclear to me, but the possibility is worth noting.
I could implement durability in the Thredds server
by keeping a queue of changed directories
and allowing clients to ask for everything since