2.4. Operations Guide
The following sections provide a guide to the operation of the system.
Standard Operation Starting, restarting, stopping, logfiles, statistics, adding and removing groups, checkpoint messages, active file, the Top 1000, and Multi-Homed machines.
Monitoring the System Descriptions of syslog and other system information.
A Day in the Life of an Article Detailed description of how an article moves through the system.
Emergency Guide What to do when the unexpected happens.
2.4.1. Standard Operation
In the bin directory, you will find the start, stop, restart, rotate, and statsnow command scripts. These scripts control the cycloned binary contained in that directory. By default, Cyclone should be started by having the "root" user run the start command script.
Cyclone will bind to the privileged NNTP port and become the user "news" (Cyclone will refuse to continue if it cannot relinquish its "root" privileges). You can easily modify the way Cyclone operates by modifying these command scripts. Every Cyclone binary provides the -help option which explains program options.
2.4.1.1. Reconfiguring the System
The validate command script can be used to ensure the integrity of the Cyclone configuration files after editing (try the -detailed option).
To install a new configuration:
Edit the Cyclone configuration files, run validate to check the configuration, and finally run the restart command script to install the new configuration.
 | Run-time Feed Reloading |
|---|---|
As of January, 2003, all Highwinds Software products support run-time feed reconfiguration |
2.4.1.2. Statistics
Cyclone computes and updates raw statistics files (by default) to the log (specified by -stats) directory on an hourly basis. You can modify where and when the statistics get updated by using the -stats and the -update options as well as the statsnow command script. The specific format of Cyclone statistics are documented in the etc/README.stats file.
If instead of a filename, you specify "local0", "local1", ..., "local7", or "localnews" to the -stats option the system will use the syslog facility the statistics data to the named facility at LOG_INFO priority.
The quick.pl and summarize.pl PERL scripts produce formatted summaries of Cyclone statistics. These scripts can be run from the command line on both Incoming and Outgoing statistics files. Comments at the top of the summarize.pl script explain a very simple modification that can be made to turn it into a CGI program.
To rotate your statistics files on a daily basis, we recommend setting up a cron(1M) entry to run the rotate command script daily at 12:01AM.
2.4.1.3. CheckPoint Messages
The checkpoint command script, causes Cyclone to syslog connection statistics to the LOG_NEWS facility at LOG_INFO priority. By using this command script, you can obtain instantaneous status information without interfering with normal Cyclone statistics reporting.
You must run Cyclone with the "-detailed" option if you wish to use this feature.
2.4.1.4. Active File
Although Cyclone doesn't need an "Active File", by providing a file name to the "-active" option of cycloned, you can limit the articles that you receive to the Newsgroups listed in the specified file.
The Active File should contain one Newsgroup per line (Cyclone accepts active files from other news systems). For example:
comp.lang.tcl 0000000001 0000000000 y comp.lang.c.moderated 0000007595 0000007577 m |
If you provide an Active File, any article not posted to at least one Newsgroup listed in the file will be treated as "junk" and will not be sent to an outgoing feed unless that feed has the AllowJunk directive set to "Yes".
You can use the XrefAction directive in the cyclone.conf file along with an Active file to Generate Xref: header lines. This is an easy way to allow downstream News systems to synchronize article numbers by simply parsing the Xref: header line.
Finally, if your active file specifies that a specific group should be "moderated" and the DropUnapprovedArticles directive is set to "True", any articles appearing in those groups that do not have an Approved: header line will not be propagated.
2.4.1.5. Adding/Removing Groups
When using the -active mechanism to run Cyclone with an active file, it is useful to be able to add groups to and remove groups from the active file of a running system.
To add or remove a group on a running system, create a file in the same directory as the active file called active.control. In this file, list the changes you wish to make to the active file.
To remove a group, add a line to active.control consisting of the group name preceded by a dash ('-').
To modify an existing group, add an active file entry line preceded by an equals sign ('=').
To add a group only if it doesn't already exist, prepend a plus ('+') character to the active file entry line. To add a group if it doesn't exist or to modify its bounds if it does, simply place the active file entry line for this group in active.control as you would in the active.
When using the equals sign ('='), specifying zero ('0') for one of the article bounds leaves the existing bound unchanged.
For instance, the following lines would:
Add the group comp.compilers if it didn't already exist.
Adjust the low bounds on the group rec.much.traffic to 10000 if it exists.
Remove the group talk.binary.inane.chatter.
Add or modify the group comp.lang.tcl
Change the moderation status of the group comp.moderated without touching its high and low article bounds.
Add the group comp.lang.java if it didn't already exist ensuring that it is a moderated group.
Example 2-2. Sample active.control entry
+comp.compilers =rec.much.traffic 0 10000 -talk.binary.inane.chatter comp.lang.tcl 100 1 y =comp.moderated 0 0 m +comp.lang.java 0 0 m |
After creating the active.control file, tell the server to process the file by running the bin/reload command script.
2.4.1.6. Control Messages
Because each news system may have different policies with regard to adding and removing groups, the system queues up to ControlDepth newgroup, rmgroup, and checkgroups messages into the control directory for later processing. We recommend that you use a simple cron(1M) job to automatically process those messages according to your local policy. The Highwinds Contributed Software site contains a number of programs that implement different automatic control message processing policies. All of these programs use the active.control file detailed earlier.
2.4.1.7. Usenet Top 1000
The very generous folks at Freenix provide a monthly service that tracks the "Top 1000" sites on Usenet. As a part of this service, they provide the source code to a program you can use to prepare data from your site for submission to their service.
To send your contribution to the Usenet Top 1000, visit the Top 1000 site at http://www.freenix.org/reseau/top1000/, download and compile their inpaths.c program, and add the -paths ../log/paths option to cycloned inside the start command script. This option instructs Cyclone to save the Path: field of every message it routes to the argument filename.
Once a month, use the UNIX mv(2) command to move ../log/paths aside and run the bin/restart command script. Finally, run the compiled Freenix program (with the -p option) against the saved ../log/paths file and mail the results off to Freenix <pathsurvey@pathsurvey.eu.org>.
Highwinds Software is committed to submitting ACCURATE Top 1000 reports. If for any reason you think your Cyclone data is inaccurate, contact Highwinds Software Support.
2.4.1.8. Log Files
By adding the -log option to the start command script, Cyclone will create logfiles containing the article id of every article sent or received. These logfiles can be rotated by renaming them with the UNIX mv(2) command and running the statsnow command script. Since Cyclone provides system status information in other ways, we recommend against using logfiles except when tracking problems or trying to compile detailed statistics.
The logfile will note which spool each article is stored in (see the file etc/README.stats), but will also use special spool names for articles which are dropped. The codes below indicate the reason an article was dropped.
Table 2-1. Cyclone Spool Codes
| Code | Description |
|---|---|
| 0f | The article was dropped by the -program filter. |
| 0i | The article was dropped due to an IncomingGroupFilter, IncomingSubjectFilter, or IncomingPathFilter directive. |
| 0m | The article was dropped because it was posted unapproved to a moderated group and DropUnapprovedArticles was set. |
| 0t | The article was dropped because it was either too old or too new for any of the outgoing feeds. |
2.4.1.9. Group Statistics
If the -loggroups option is specified in addition to the -log option, then the Newsgroups line from each article is included in another field at the end of each log line. This can be used to generate statistics about traffic by group.
2.4.1.10. Multi-Homed Systems
For incoming feeds, adding the -interface option to cycloned inside the start command script, will instruct Cyclone to listen on a specific local interface for incoming connections.
For outgoing feeds, you can use the OutgoingInterface directive to select among local interfaces.
2.4.2. Monitoring the System
Cyclone uses the UNIX syslog system to report system status. Cyclone logs all system alerts with the LOG_NEWS facility and the severity levels detailed below. By default, most operating systems send LOG_ERR (and higher) severity levels to the system console. Keeping this default is recommended.
If you wish to obtain additional information about the running system, you should add the -detailed option to the start command script. In addition, you should modify your syslog configuration file to record LOG_INFO severity levels. Refer to your UNIX documentation regarding syslog.conf(4) for more details.
Table 2-2. Cyclone Syslog Severity Levels
| Level | Description |
|---|---|
| LOG_ALERT | This message is sent when an unrecoverable error occurs. This happens when a critical resource outside the control of the system has misbehaved. |
| LOG_CRIT | This happens when a serious error occurs. The system (in general) will make a best effort to recover from these errors. These types of errors may require intervention. |
| LOG_ERR | This happens when a normal error occurs. These errors are (in general) recoverable and do not require intervention. However, large numbers of these errors should not be a "normal" situation. |
| LOG_INFO | This happens when the system finds some information that it believes an observing administrator might find interesting or useful for statistical purposes. To enable these messages, you must run Cyclone with the -detailed option. |
2.4.3. A Day in the Life of an Article
What follows is a high level summary of the path an article takes through Cyclone meant to give you a brief look into some of the things Cyclone does while processing articles.
- First Contact
A peer system opens a TCP connection to the local NNTP port. This connection is accepted by cycloned if there is an IncomingHostName entry for the peer host in the feeds.conf file.
- Article Transmission
The peer system sends an article to the local system using either normal or streaming NNTP. cycloned checks the history database (defined by -history) to see if it has seen the article before. If not, the article is accepted by the local system.
- Parsing and Saving
The article is parsed and checked to ensure that the required header lines are intact and that the article's date stamp is recent. The Path: header line is preceded with the name of the local host (or the names given by the -masquerade and -alias options to cycloned). The article is then saved into one of the spool's specified in the cyclone.conf file.
- Distribution
After saving the article, cycloned checks it against each feed which has an OutgoingHostName entry. If the article meets all the qualifications of the outgoing feed, an entry is made in a file in a subdirectory of the spool directory named after the OutgoingFeedName of that feed.
- Re-Transmission
cycloned processes the articles stored in the outgoing queues by opening up a connection to the port and host specified by the PortNumber and OutgoingHostName entries for each of the feeds. cycloned sends the article to the remote host and the cycle repeats.
2.4.4. Emergency Guide
- Turn on Detailed Logging
By adding the -log option, you will see more details of the system's operation. You may wish to use the -log option to have Cyclone log the article ids of every message processed. Be sure to refer to Monitoring the System for more details.
- Look at the Statistics
Statistics reports and Checkpoints detail article and connection statistics. This data is a great starting point for all debugging. If you are having trouble, you may wish to use the checkpoint and statsnow command scripts to get data out of the system more frequently.
- Turn off DNS
If you have an unreliable DNS, modify the bin/start command script to use the -nodns option. This will require you to modify the feeds.conf file to use ONLY numeric IP addresses for all host values. After making the modifications, run the bin/restart command script.