Creating a Mail App

Creating a MailApp

I intend to collate my various mail archives in a single location and have chosen Foswiki as the repository. After some correspondence on the mailing list I decided to use the ClassificationApp as the basis for the implementation. I knew nothing about Foswiki applications, and was thoroughly confused at the beginning. I now have a working application, which turned out to be surprisingly simple. If only I had known at the start.

Introduction

The essence of the solution has nothing to do with the ClassificationApp. All I needed to do was parse the email and write Foswiki topics with additional fields in the meta data. I choose the following email attributes to be stored as meta data:

META:FIELD{name="To" attributes="" title="To" value="Cheryl Bond"}
META:FIELD{name="From" attributes="" title="From" value="Tracey Hall"}
META:FIELD{name="CC" attributes="" title="CC" value="Scott Raindeer, Adriaan Van Oosterhout"}
META:FIELD{name="Date" attributes="" title="Date" value="Mon, 19 Jan 2015 23:33:10 +0000"}
META:FIELD{name="Subject" attributes="" title="Subject" value="FW: Boeing CBT"}

Creating the input

I worked out that the ClassificationApp created ClassifiedTopics with standard metadata as follows:

META:FORM{name="Applications/ClassificationApp.ClassifiedTopic"}
META:FIELD{name="TopicType" title="TopicType" value="ClassifiedTopic, CategorizedTopic, TaggedTopic, WikiTopic"}
META:FIELD{name="TopicTitle" attributes="" title="<nop>TopicTitle" value="Mail1 from JB"}
META:FIELD{name="Summary" attributes="" title="Summary" value="subject from the mail1 from JB"}
META:FIELD{name="Tag" attributes="" title="Tag" value="Judy Bond"}
META:FIELD{name="Category" attributes="" title="Category" value="TestMailInboxCategory"}

So in the end I wrote each email topic with the meta data of a classified topic (with some specific email data as part of the ClassifiedTopic meta data), followed by the meta data I wanted to use:

META:TOPICPARENT{name="ClassifiedTopic"}
META:FORM{name="Applications/ClassificationApp.ClassifiedTopic"}
META:FIELD{name="TopicType" title="TopicType" value="ClassifiedTopic, CategorizedTopic, TaggedTopic, WikiTopic"}
META:FIELD{name="TopicTitle" attributes="" title="<nop>TopicTitle" value="FW: Boeing CBT"}
META:FIELD{name="Summary" attributes="" title="Summary" value="Mon, 19 Jan 2015 23:33:10 +0000"}
META:FIELD{name="Tag" attributes="" title="Tag" value="Tracey Hall"}
META:FIELD{name="Category" attributes="" title="Category" value="TestMailInboxCategory"}
META:FIELD{name="To" attributes="" title="To" value="Cheryl Bond"}
META:FIELD{name="From" attributes="" title="From" value="Tracey Hall"}
META:FIELD{name="CC" attributes="" title="CC" value="Scott Raindeer, Adriaan Van Oosterhout"}
META:FIELD{name="Date" attributes="" title="Date" value="Mon, 19 Jan 2015 23:33:10 +0000"}
META:FIELD{name="Subject" attributes="" title="Subject" value="FW: Boeing CBT"}

I used the following makefile to populate the Foswiki web. I created the content in /tmp/data/Sandbox and /tmp/pub/Sanbox. Note that I had to make sure that the correct security attributes were on the file and the cache required an update.

TEMP = /tmp
WEB = TESTK3
FOSWIKI = /var/lib/foswiki

installmaildata:
        cp ${TEMP}/data/Sandbox/* ${FOSWIKI}/data/${WEB}/.
        chmod 644 ${FOSWIKI}/data/${WEB}/*
        chown www-data:www-data ${FOSWIKI}/data/${WEB}/*

installmailpub:
        cp -r ${TEMP}/pub/Sandbox/* ${FOSWIKI}/pub/${WEB}/.
        chmod -R 755 ${FOSWIKI}/pub/${WEB}/*
        chown -R www-data:www-data ${FOSWIKI}/pub/${WEB}/*

updatecache:
        cd ${FOSWIKI}/bin; sudo -u www-data './rest /DBCachePlugin/updateCache'

installmail: installmaildata installmailpub updatecache

clean:
        rm -r ${TEMP}/data/Sandbox/*
        rm -r ${TEMP}/pub/Sandbox/*

cleanmail:
        rm -r ${FOSWIKI}/data/${WEB}/Inbox*

Using the ClassificationApp

After the installation of the email topics in ClassificationApp (web = TESTK3) I could see the topics as classified topics. But I could not see the attributes that I had supplied in addition to the ClassifiedTopic attributes (To, From, CC, Date, Subject). After some investigation I extended the form definition in the Applications/ClassificationApp.ClassifiedTopic to:

---++ <nop>%TOPIC% form
| *Name:* | *Type:* | *Size:* | *Values:* | *Description:* | *Attributes:* |
| TopicType | label | 1 | ClassifiedTopic, CategorizedTopic, TaggedTopic | Document type  | |
| <nop>TopicTitle | text | 75 | | title of this topic | |
| Summary | text | 75 | | short description of the role of this topic | |
| Tag | tag | 30 | | keywords for this topic | |
| Category | cat | 1 | | categories this topic is field to | |
| To | tag | 150 | | list of recipients | |
| From | tag | 150 | | list of senders | |
| CC | tag | 150 | | list of copied recipients | |
| Date | text | 50 | | date email was sent | |
| Subject | text | 150 | | Subject of the email | |

I started out with all the new fields as type text. That made them visible using the %META{"form"} in the installed email topics. But it did not allow me to manipulate them as tags when editing an installed email topic. In retrospect I am unlikely to ever want to do that, but it drew my attention to the "tag" type in the form definition. So I changed the type of To, From and CC to "tag" and that rendered them like any other tag. I am interested to find some documentation about this feature.

Displaying topics with a particular tag type and value

I set up a page to display the topics that were tagged by a specific value (the searchvalue) for a specific tag (the searchtag). I used the search expression:

%SEARCH{"%URLPARAM{"searchtag"}%~'*%URLPARAM{"searchvalue"}%*'" type="query" format="| $topic | $formfield(Date) | $formfield(Subject) |"}%

Using the url

http://192.168.1.170/foswiki/bin/view/TESTK3/SearchTaggedTopics?searchtag=To;searchvalue=Alex%20Kenley

the expected email topics were returned:

Listing the tagvalues for a particular tag

The final step is to create a list of tag values for a particular tag. For instance show me all email names that have sent me email. To do that I need a list of all the "From" tags. It took me a while to figure this out, but I ended up creating a new page ListOfTagValues with the content copied from Applications/ClassificationApp.RenderKnownTags:

%DBSTATS{
  "TopicType=~'\b(TaggedTopic|ClassifiedTopic)\b'"
  %IF{"defined TAG" then="include=\"^%TAG%\""}%
  field="Tag"                                                   ## modification 1
  web="%IF{"defined THEWEB" then="%THEWEB%" else="%BASEWEB%"}%"
  header="[$n"
  format="{\"value\":\"$key\"}"                                 ## modification 3
  separator=",$n"                                               ## modification 2
  footer="$n]"
  limit="%IF{"defined LIMIT" then="%LIMIT%" else="10"}%"
  pattern="^(.*)$" 
}%

I made a few modifications to the line above:

Modification	Changed to	Rationale
1	field="%TAGFIELD%"	I can choose the tag I want to select with a "Set TAGFIELD =" macro definition
2	separator="$n"	I want a bulleted list, not a comma separated list.
3	format=" * [[https://foswiki.org/Support/SearchTaggedTopics?searchtag=%TAGFIELD%;searchvalue=$key][$key]] ($count)"	This creates the url to access the SearchTaggedTopics pages created previously, passing the searchtag and searchvalue parameters.

With these changes i get the desired result: A list of tag values for a particular tag, with a reference to the search page that shows me what emails are associated with this tag:

Conclusion and questions

The worked example above shows how Foswiki can support an email archive. the ClassificationApp is a useful vehicle to serve with examples, but even now I have a workable solution, I am puzzled by the value of the ClassificationApp as a vehicle. I expected more and I suspect there is more. Here are some questions that I believe would help me to achieve the same result in a more coherent way.

1. I extended the form definition in the ClassificationTopic to include the fields: From, To, CC, Date, Subject. I expected to do that using the TopicType creation link in the ClassificationApp, inheriting from ClassifiedTopic but I have been unable to work out how to proceed from there.
   • For instance, what are the implications of the inheritance. I expected the form to be copied, but that does not appear to be the case
2. I copied and modified the Applications/ClassificationApp.RenderKnownTags page to get what I wanted. But the ClassificationApp has other Render... pages that would be nice to use. I do not understand the application well enough to decide whether
   • I should continue to use the ClassificationApp as an example and modify, OR
   • should change my approach regarding the meta fields so that I can use the Tag field for all the things I want to do
3. Considering the possibility that I create and submit this functionality as an Foswiki application, should I:
   • Create a new application (MailApp), separate from the ClassificationApp, OR
   • Create additional pages in the ClassificationApp and repackage that application.

I will post a summary of this post on the mailing list. -- BramVanOosterhout - 16 Jul 2015

I would recommend creating a new application. Just go to the Application.WebHome and use the form for adding an application there. If you want to extend, or use an existing application topic, just copy the original into your new application web and modify it there. The main benefits of this approach are:

• You won't risk over-writing your local modifications if you update ClassificationPlugin or WikiWorkbenchContrib.
• You can package up your application as it's on Contrib.

One caveat about copying a topic from one of the existing apps. You may need to review it for references back to the original web, either via and INCLUDE or VIEW_TEMPLATE and make sure they point back to the correct web. Be aware also that these referenced topics may change themselves in the future and cause issues. If that happens or if you're concerned about it, you might consider also copying over those dependency-topics into your own app web.

-- LynnwoodBrown - 17 Jul 2015

Update 14 August 2015 - an early implementation

I have a very rudimentary implementation together and the code is attached here.

There are 2 parts:

• a MailArchiveContrib.tgz. A Foswiki application that displays

correspondence to and/or from correspondents.

• A mbx2wiki.tgz. A perl script and make file that take a pst or mbx format mail file and populate the mail archive.

They constitute a work in progress and are by no means production ready. But as I will be travelling the east and west coast of the USA for the next three months, consider this a progress update.

In the MailArchiveContrib.tgz file you find 5 pages of notes that describe my journey in developing the app. I learned a lot of new things and they may be helpful to others that want to develop foswiki apps. There is a lot of good thinking in the WikiWorkbench. But the tool needs more expectation setting as part of the documentation. The 5 pages may help doing that. Although I believe an edit, if not a rewrite is essential. I will get back to that, but playing with the MailArchive, I get more ideas on making that more functional and approachable. The edit too, will have to wait until after my trip.

Hope this is useful to you. And if not; wait for the next instalment!

-- BramVanOosterhout - 14 Aug 2015

Commenting is disabled while not logged in