16.7. Putting it all Together
So you have your Erector set spread all over the floor. Different components to function in unique ways. We've gone over a few here, and you'll note a few we haven't gone over displayed in Figure 6. Figure 6 depicts what is often the challenge of a Usenet architect or administrator… building it so it works.
Follow the yellow numbers in the diagram to examine each section of the diagram. We've provided high resolution images if you wish to print them out.
Cyclones peer with external feeders.
Border routers are set up to peer with an arbitrary number of external peers. Depending on your network and hardware availability, each border router may be set up to take only a portion of your total peers or a portion of the total feed.
We use -alias and Aliases on all of our border routers to assure we don't loopback articles.
Intra-POP routers feed each other bi-directionally with a 15 second delayed feed. What this does is create a small time buffer so when an article is accepted on one host it isn't immediately offered to the second. Since an article from a host that does not use Aliases is likely to be fed to border routers at the nearly the same time, you want to avoid a race condition where one border router offers the article to the other when it's in the process of accepting it. Not a problem with IHAVE, since the Message-ID is logged to the history and future offers are declined, however, TAKETHIS presents a problem. Besides that, it's generally good practice to do what you can to stabilize the protocol and prevent race conditions where you can.
Inter-POP routers feed each other bi-directionally with a 30 second delay. You want to ensure articles get to all of your POPs, but you want to give articles a chance to arrive locally and minimize needless transit.
Split feeds to your Spam filtering Cyclones. Filtering Spam is often a resource intensive process. To divide the workload, we implement multiple servers and only send them 1/2 of the feed.
We redundantly feed Spam boxes with a delayed, split feed to allow for outage and redundancy.
Filtering Both Spam and Viruses are becoming bigger problems. Usenet is becoming more and more affected by bulk postings and virus types that are more underhanded. Generally, clients can avoid viruses where they don't open the file, but more and more viruses are increasingly harder to visually detect. Providers are being requested to solve the problem.
Spam filtering and Virus filtering are both highly intensive processes, so the work is divided over multiple servers with multiple paths for any one article to take. Since you don't want to waste resources scanning for viruses on a message you'll throw out as Spam anyway, the Spam filters receive articles first.
Local posts from the Tornado Front Ends are routed to the Filtering battery. Doing this acts like a global post filter and manages the amount of Spam and Viruses you allow to proliferate outside and back onto your network.
Cleaned feed goes to border Cyclones for "retail" feeds. If your organization sells news feeds, you can command a higher value for a lower Spam and lower Virus feed.
Numbering boxes aggregate all the cleaned feeds. Cyclones that perform article numbering via Hurricane receive feeds from all the filtering battery and aggregate it back into a full feed.
Internal numbering routers feed each other bi-directionally with an ample, 60 second delay to ensure articles that may have been lost or never fed on one POP are shared with the other.
Each Cyclone sends a split feed to the local Tornado Back Ends and a delayed split feed to the remote Tornado Back ends. Again, redundancy is cheap to attain.
Tornado Back Ends receive a local, split feed to divide the storage requirement
Each Tornado Back End ends up with 1/2 of a full feed. The articles are divided up at each site exactly the same, so rather than having a random distribution of 1/2 of the articles, each Tornado BE has a redundant copy at the other site(s).
Tornado Front Ends aggregate all Tornado Back Ends on the network, providing a full server via local Back Ends and a redundant copy via remote Back Ends.
