6.6. Adaptive Spools

Historically, UseNet cannot be effectively cached without large amounts of disk. What is a news server if not a cache of Usenet anyway? The best that can be done is to set aside huge amounts of disk and "cache" an entire day, well not anymore. StormCellar allows a kind of reverse caching. Iinstead of caching everything that might be read often, it caches things that are almost never read. Doing so allows the entire Tornado Back End to be a cache, de facto, without any of the additional I/O associated with "rewriting" an article.

There are two flavors of adaptive spooling. The first is what is sent to the backend. Tornado Back End collects reader profiles from all of its front ends, and knows which groups are being read and how much, using this information it can only cascade posts form read groups. This methodology represented a 30% saving on both of the large production installations it was tried in. In other words, 30% of their Usenet feed was never read, ever, and therefore never needed to be sent to the StormCellar.

The benefits from this were immediate, instead of a Terabyte a day, the StormCellar only needed to store 700 Gigabytes, with the resultant gain in retention. The next obvious step became artificially expiring posts from the front end from groups that were rarely read, leaving more room for the primary groups. Once a StormCellar has been deployed, the subscriptions on all the Tornado Backend spools can be automated, and will allocate more spools for hard-hit groups. Since the StormCellar "catches" anything that is artificially expired, retention is unaffected, but the Tornado Backend is able to catch more of the load.