SEA Technical Memorandum #0301, Technical Aspects of Implementing GroupMail
Last updated: October 5, 1989
Copyright 1988,89 by System Enhancement Associates, Inc.



                             Technical Aspects
                                     of
                           Implementing GroupMail


In the documentation for the GROUP conferencing system, we talked about how 
to use GROUP to interface other types of systems into GroupMail.  Our 
example showed how to do it with TBBS, but there are many other types of 
systems that are in the same boat.  

The best way to do it, of course, is for GroupMail handling to be 
integrated into the system directly, using that system's own native message 
base format.  

For any given system in any given conference, the system can be in any one 
of three positions: a leaf, a middle star, or the top star.  Also, group 
mail messages can arrive on the system in any one of three ways: by being 
entered locally, or by network mail, or inside of a GroupMail archive.  The 
proper action to take for any given message depends on these factors as 
follows: 

                  entered          via netmail        via GroupMail
                +----------------+------------------+--------------------+
    leaf        | send to uplink | send to uplink   | unpack to msg area |
                +----------------+------------------+--------------------+
    middle star | send to uplink | send to uplink   | copy and unpack    |
                +----------------+------------------+--------------------+
    top star    | pack           | toss to msg area | can't happen       |
                +----------------+------------------+--------------------+

We'll explain those actions in a bit more detail: 

    Send to uplink;  The destination address in the message header should 
    be changed to that of the uplink.  The origin address should NOT be 
    altered.  

    Unpack to message area;  The received GroupMail archive should be 
    unpacked, and all of the messages in it should be placed in the 
    appropriate message base.  The "name" portion of the archive filename 
    identifies which group the messages belong to.  The archive should be 
    deleted once it has been unpacked.  

    Copy and unpack;  This is the same as unpacking to a message area, 
    except that the GroupMail archive is not deleted.  Instead, it is moved 
    to a "holding area" where other systems can request it.  Presumably 
    some regular process deletes archives in the holding area after they 
    have been around long enough (however long that is).  

    Pack;  The message should be added to a GroupMail packet, which is then 
    archived into a GroupMail archive.  The archive should be placed in a 
    "holding area" where other systems can request it.  Presumably a 
    regular process cleans out the holding area, as with a midstar's "copy 
    and pack" process.  Due to the fact that GroupMail archive names "wrap" 
    every month, a topstar should not retain GroupMail archives for longer 
    than 28 days.  

    Toss to message area;  The message should be moved into the appropriate 
    message base.  

    Can't happen;  The top star of any given conference does not request 
    GroupMail archives for that conference from other systems.  Instead, 
    GroupMail messages are sent to him via network mail.  Thus, the top 
    star never receives any GroupMail archives for his own conference.  So 
    of course this will happen eventually.  


Details, details.


Group names may be up to eight characters, including A to Z, 0 to 9, and 
underscore.  GROUP massages any group name it is given to match this, by 
truncating at eight characters and changing anything that is not a letter 
or a digit into an underscore.  Thus, "GZORNIBLATZ" becomes "GZORNIBL" and 
"TK!43*" becomes "TK_43_".  The group archive extension is the base 36 
minute of the month.  For those who insist that we send code, it can be 
calculated easily like so: 

    static char *chset = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"
    unsigned x;
    char ext[4];

    x = minute + 60 * (hour + 24 * (day-1));

    ext[3] = 0;
    ext[2] = chset[x%36]; x /= 36;
    ext[1] = chset[x%36]; x /= 36;
    ext[0] = chset[x%36];


Group archives contain FidoNet standard "type 2" packets.  The packets are 
named as follows:

    ddhhmmss.PKT

Where "dd" is the day of the month that the packet was created, and 
"hhmmss" is the time of day.  However, almost anything ending in ".PKT" 
will do.  


GroupMail messages that are moving "up" the chain to the top star contain a 
group designator.  This is a line in the text of the message of the form: 

    ^aGROUP: <group>

where "^a" is an ASCII "start of header" (control A, decimal value 1), and 
"<group>" is the group name.  This field may be anywhere in the message 
text.  



Ping/Pong:

GroupMail incorporates automatic topology mapping in the form of "Ping!" 
and "Pong!" messages.  From time to time a topstar may request topology 
data for his conference by issuing a "Ping!" message.  This is simply a 
message in the conference where the message subject is the literal string 
"Ping!", where case is important.  The text of the message is irrelevant to 
the ping/pong process, but is usually used to distribute a "topology map" 
giving the results of the previous ping.  

To prevent false pings, the topstar should never import a message with a 
subject of "Ping!".  In the case of GROUP, messages received by the topstar 
with a subject of "Ping!" have the subject field altered to read "ping!" 
(i.e. the uppercase 'P' is turned into a lowercase 'p').

When a midstar or leaf node encounters a downward-bound "Ping!" message 
(i.e. when unpacking a GroupMail archive) it should respond by generating a 
"Pong!" message.  This is simply a message addressed to the uplink with the 
proper ^aGROUP: identifier, a subject of "Pong!", and one line of pong data 
(see below).  

When a midstar encounters an upward-bound "Pong!" message (i.e. when 
scanning the network mail area) it should readress it to the uplink as with 
normal upward-bound GroupMail, and it should add one line of pong data (see 
below) to the END of the message text.  Pong data added by a midstar should 
ALWAYS go at the end, as order is significant in determining the actual 
conference topology.

When a topstar encounters a "Pong!" message it should extract whatever data 
is desired and then discard the message.  Pong messages should NEVER be 
posted in the GroupMail conference, as they are not designed to be directly 
meaningful to conference participants.  

Pong data added by a leaf or midstar always takes the same form.  One pong 
data entry consists of one line of message text with the format:

    PONG <addr> <uplink> <keep> <progid> <eol>

where:

    PONG      is the literal string "PONG"

    <addr>    is the network address of the system that generated this 
              entry

    <uplink>  is the network address of the uplink for the system that 
              generated this entry

    <keep>    is the number of days that this system retains this 
              conference (if a midstar), or zero (if a leaf node)

    <progid>  is the name of the program used to process GroupMail at this 
              location 

    <eol>     is a literal end of line (carriage return/linefeed)

Each of these fields is a single word.  Spaces are not allowed because they 
are used to separate fields.


For example, the text of a typical pong message might look like this:

    ^aGROUP: BLATZ
    PONG 520/911 520/901 0 GMAIL_2.00
    PONG 520/901 520/1015 5 GMM
    PONG 520/1015 520/583 3 GROUP_2.1


The network addresses given in <addr> and <uplink> are unevaluated text 
strings, and may be in any format.  For example, "107/542", "1:107/542.6", 
"1:107/542@FIDONET", and even "1-201-472-8065" are all legal.



Implementation notes:

GROUP creates and maintains a "target file" for every conference.  This is 
a file placed in the group directory where the filename matches the group 
name, and the extension is an exclamation point.  For example, the target 
file for the BLATZ conference would be named "BLATZ.!".  This file is zero 
bytes in length, and thus does not occupy any disk space.  GROUP maintains 
the MS-DOS date/time stamp on the target file to match that of the newest 
group mail archive that has been unpacked for that conference.  Thus, 
requesting an update of "BLATZ.*" will receive only those GroupMail 
archives that are newer than the target file, and hence newer than the 
latest archive received.  


GROUP also creates and maintains several files that act as "high water 
marks".  These are named "GROUP.!" and are placed in the message 
directories.  They are marked as hidden system files so that they do not 
get mangled by other processes.  GROUP only examines messages that are 
newer than the associated high water mark, and then stamps the high water 
mark to be as new as the newest message examined.  If the "/S" (for "slow 
scan") option is specified, then GROUP will examine all messages.  

When a GroupMail archive is unpacked, GROUP sets the date/time stamp of the 
messages that are imported to match that of the high water mark.  Also, 
each message is marked as "sent", so that they do not get scanned out 
again.  

The top star receives messages by network mail instead of as GroupMail 
archives.  In this case the messages are NOT marked "sent".  
