ActivityStreamPlugin Development

This is the topic to discuss development of download ActivityStreamPlugin

help If you need support, go to Support.ActivityStreamPlugin where you can ask questions and find answers to previously asked questions. warning If you want to report a bug, or a feature request, go to Tasks.ActivityStreamPlugin where you can see already submitted issues and where you can submit a new bug report or feature request.

Active Items

Id Summary Priority Current State Creation Date Last Edit
Item11964 initial work on an activity stream and microblogging system for foswiki Enhancement Being Worked On 21 Jun 2012 - 15:13 12 Dec 2017 - 17:00

Examples

feed.pngactivity-stream.jpgactivity-feed.pnguntitled-1.png notification-dropdown.jpgdribbble2.pngbig.pngyammer activity streams ss.jpg Socialcast20-20Activity20Stream.jpgjira-42-browse-project.pngActivityStreamGadget display.pngfull-view.jpg tanveer-activity.png

Foswiki events and how they are processed in the ActivityLogger

Raw event Message Mapped action Rewritten Message
Core
* scumbag/AUTOINC/XXXXXXXXX/ ignored  
attach   ignored  
changes   ignored  
edit   ignored  
edit (not exist) new  
login SUCCESS - user login  
  FAIL - user loginfailure  
logout LOGOUT - user logout  
move web.topic.attachment move(d) to web.topic.newattachment renameattachment target=newattachment@web.topic
move web.topic.attachment move(d) to otherweb.othertopic.attachment moveattachment target=attachment@otherweb.othertopic
move source move(d) to trash web deleteattachment  
rdiff int int compare rev1=int rev2=int
register user@domain register user
rename move(d) to web.topic rename target=web.topic
rename move(d) to trashweb.topic delete  
renameweb move(d) to web renameweb target=web
renameweb move(d) to trashweb deleteweb  
rest   ignored  
save   edit  
save attachment editattachment  
search pattern search pattern
upload filename upload  
view      
view (not exist) ignored  
viewfile   ignored  
MetaCommentPlugin
comment state=... title=... text=... comment state=... title=... text=...
commentapprove state=... title=... text=... commentapprove state=... title=... text=...
commentdelete state=... title=... text=... commentdelete state=... title=... text=...
commentupdate state=... title=... text=... commentupdate state=... title=... text=...
CopyContrib
copy      
CompareRevisionsAddOn
compare   compare  
SqlPlugin
sql sql statement    

Grouping messages in the ACTIVITYSTREAM widget

activities are extracted from the logs data base grouping them by

  • user
  • action
  • timerange (1 hour by default)

Actions of the same user ocurring in the given timerange are pooled and formatted indepentently. Some actions don't require a "..._multi" variation. Some actiosn are ignored by the ACTIVITYSTREAM widget. These kind of informations are still kept in the logs database and may be reused otherwise. For instance view actions are not displayed as part of the activity stream, although a view statistics is interesting to display such as:

  • per page: how often has this page been visited (within a certain timerange)
  • hot topics: which topics are visited most often

However this is out of the scope of the ACTIVITSTREAM widget itself and requires a different widget to be implemented.

Ignored actions:
  • compare
  • loginfailure
  • register
  • view

Friday, 11. February 2011

add John Doe created Some Topic 5 minutes ago.
user John Doe logged in 5 minutes ago.
key_go John Doe logged out 5 minutes ago.
%MIMEICON{"pdf" size="16"}% John Doe renamed attachment foo.pdf to someattachment.pdf 5 minutes ago.
application_edit John Doe renamed someattachment.pdf and 5 other attachments 5 minutes ago ...
%MIMEICON{"pdf" size="48"}% someattachment.pdf
comment ...
%MIMEICON{"pdf" size="48"}% someattachment.pdf
comment ...
%MIMEICON{"gif" size="48"}% someattachment.pdf
comment ...
%MIMEICON{"gif" size="48"}% someattachment.pdf
comment ...
%MIMEICON{"zip" size="48"}% someattachment.pdf
comment ...
%MIMEICON{"zip" size="48"}% someattachment.pdf
comment ...
%MIMEICON{"pdf" size="16"}% John Doe moved attachment someattachment.pdf to Some Topic 5 minutes ago.
application_go John Doe moved around someattachment.pdf and 5 other attachments 5 minutes ago ...
%MIMEICON{"pdf" size="48"}% someattachment.pdf
now at Some Topic
%MIMEICON{"pdf" size="48"}% someattachment.pdf
now at Some Topic
%MIMEICON{"gif" size="48"}% someattachment.pdf
now at Some Topic
%MIMEICON{"gif" size="48"}% someattachment.pdf
now at Some Topic
%MIMEICON{"zip" size="48"}% someattachment.pdf
now at Some Topic
%MIMEICON{"zip" size="48"}% someattachment.pdf
now at Some Topic
bin John Doe deleted attachment foo.pdf from Some Topic 5 minutes ago.
bin John Doe deleted attachment foo.pdf and 5 others 5 minutes ago ...
%MIMEICON{"pdf" size="48"}% foo.pdf
was at Some Topic
%MIMEICON{"pdf" size="48"}% foo.pdf
was at Some Topic
%MIMEICON{"gif" size="48"}% foo.gif
was at Some Topic
%MIMEICON{"gif" size="48"}% foo.gif
was at Some Topic
%MIMEICON{"zip" size="48"}% foo.zip
was at Some Topic
%MIMEICON{"zip" size="48"}% foo.zip
was at Some Topic

rename

rename_multi

delete

delete_multi

renameweb

renameweb_multi

deleteweb

deleteweb_multi

editattachment

editattachment_multi

search_multi

upload

upload_multi

comment

comment_multi

commentapprove

commentapprove_multi

commentdelete

commentdelete_multi

commentupdate

commentupdate_multi

-- MichaelDaum - 21 Jun 2012

Discussion

I'm concerned that Foswiki logging is not sufficient. I've been finding places where modifications are completely missed in the logging. For ex. the Groups API did no logging at all in the events log. I suspect core really needs an in-depth audit to make sure that things don't slip through the cracks:
  • Low level Store event logging
    • Record every modification to store, topics and attachments
    • Optional - access logging of ever read of a store object
  • Event logging
    • Audit all APIs and scripts, ensuring that every possible path for logical modificatiions of topics is recorded.

-- GeorgeClark - 21 Jun 2012

Nifty!

-- PaulHarvey - 22 Jun 2012

Agree with George about the audit. The starting point is a comprehensive list of "interesting events" - which Michael has created above, and I think we should adopt as a gold standard.

-- CrawfordCurrie - 22 Jun 2012

Oliver and I met in Hamburg 2 days ago and discussed this in great detail.

Of course event logging isnt enuf by far. These are only a quick way to get any content into a database. The events that the core emits will be added to the activities database as only one "collector" of many. The plugin will offer abstract classes and interfaces to register collectors for different kind of events and actions.

Likewise the way things are verbalized right now wont stay as it is. This too has to be made pluggable.

So we will have:

  • (a) collectors: base classes and an api to implement custom event types to be added to the database
  • (b) verbalizer: an api to produce a localized human readable representation of a single event or a grouped bunch of events that end up as one new row in the activity stream.
  • (c) search & filter: assess raw data in database, apply filtering and grouping and call the appropriate verbalizer for the type of event to be rendered
  • (d) renderer: produce the desired output format: html, json, rss, atom

The database will log events and actions in a kind of tripple as well as a set of meta data describing properties of object and subject (work group, team, category, tag, acl):

  1. subject: the person that produced that event
  2. verb: the action that the subject performed on the object
  3. object: the affected object (e.g. a topic, attachment, comment, tag, category, formfield, a custom foo-bar thing)

The verbalizer will have to be able to cope with different kinds of data being grouped together to be uttered in one. These are:

  • singular event: something happened only once
  • multi event: a couple of events happened in a short period of time done by the same subject, or the same object. (e.g. "You and five others commented on topic x.", "John doe approved the invoice FooBar and five others ... check out details.")
  • second person: "You did this"
  • third person: "John Doe did this."
  • stress a specific object in a multi-event: "John Doe uploaded this attachemnt and five others ... click here for details".

There are two kinds of activity collectors that one might think of implementing:

  • automatic events: these are detected automatically as a consequence of the user interacting with the system independent of any direct interaction with an activity stream
  • manually produced events: like prominently microblogging -> a user manually records an event that happened somewhere else outside of the system

Putting up a poll or asking a question is a manual event, as well as responding to it. These require a verbalizer of their own as well ("John doe asked a question/ answered a question/put up a survey/added his 2cent to the poll).

The key is that the base is pluggable to add things like polls, commenting, question-answering as an add on that can hook into the activity stream to be recorded and rendered appropriately.

A side issue is user profiles. Foswiki doesn't formalize what the avatar/photo of a user is, nor what the thumbnail representation of any other topic type is. We need this visualizing search results not only in activity streams. User photos are very important with regards to the way people perceive and interact with an activity stream. You will virtually see who did something on the wiki. That's quite a different feeling towards the system and the stuff happening. It is much more oriented towards the human beings behind the data being stored rather than the data that by the way was changed by someone. So it is in some sense the inverse of WebChanges, properly grouped and "humanized" the way the data is presented.

This kind of architecture isn't part of the current implementation by far. It is more a kind of getting the feet wet thing to get a feeling about the domain.

The plugin does work and is fully functional in that you can see data being displayed in System.SiteActivityStream and WebActivityStream per web. There's some more filtering already there to display only a certain group's activities or types of activities. Looks like the mocked up example above. Existing core events can be imported using the import rest handler. There's a Foswiki::Logger::ActivityLogger impementation that feeds data to the database while still having the plain file logger as a secondary one to bootstrap the database from scratch any time.

With regards to the current way to "reuse" the Foswiki::Logger gathering what is there, there's a lot of clean up required to make any sense of the data coming in that way. The most valuable source of information which kind of event actually stands behind a certain log message is ... as usual ... stored in the comment field. Some of this information might probably be too unspecific to really deduce the kind of event that actually happened. I did not make it the full way down all of those log messages as you can see above.

So it would very well help to rationalize the format of the log messages that we have right now, and even add some more here and there. Some are quite pointless and completely useless from the POV of an activity stream (see the ignores above). Well, but we can live with the current kind of log message too that the core emits right now. If there isn't an event being recorded then it wont be part of the activity stream. Most of the important stuff is already in there like create, delete, edit, move things around. Point is that we can isolate these problems in their own activity collector classes feeding data to the database. When the log format of the core events changes, only one class has to be fixed but not the complete plugin architecture.

-- MichaelDaum - 22 Jun 2012

http://activitystrea.ms/ and http://activitystrea.ms/specs/json/1.0/ may be a worth reading.

It describes an exchange format for activity streams in json and/or xml.

-- OliverKrueger - 22 Jun 2012

Maybe of interest aswell: http://pump.io/

-- OliverKrueger - 15 Jan 2013
 
Topic attachments
I Attachment Action Size Date Who Comment
ActivityStreamGadget_display.pngpng ActivityStreamGadget_display.png manage 119 K 21 Jun 2012 - 15:43 MichaelDaum  
Socialcast20-20Activity20Stream.jpgjpg Socialcast20-20Activity20Stream.jpg manage 343 K 21 Jun 2012 - 15:41 MichaelDaum  
activity-feed.pngpng activity-feed.png manage 58 K 21 Jun 2012 - 15:37 MichaelDaum  
activity-stream.jpgjpg activity-stream.jpg manage 104 K 21 Jun 2012 - 15:36 MichaelDaum  
big.pngpng big.png manage 89 K 21 Jun 2012 - 15:40 MichaelDaum  
dribbble2.pngpng dribbble2.png manage 16 K 21 Jun 2012 - 15:40 MichaelDaum  
feed.pngpng feed.png manage 68 K 21 Jun 2012 - 15:35 MichaelDaum  
full-view.jpgjpg full-view.jpg manage 228 K 21 Jun 2012 - 15:38 MichaelDaum  
jira-42-browse-project.pngpng jira-42-browse-project.png manage 287 K 21 Jun 2012 - 15:42 MichaelDaum  
notification-dropdown.jpgjpg notification-dropdown.jpg manage 77 K 21 Jun 2012 - 15:37 MichaelDaum  
tanveer-activity.pngpng tanveer-activity.png manage 131 K 22 Jun 2012 - 09:51 MichaelDaum  
untitled-1.pngpng untitled-1.png manage 32 K 21 Jun 2012 - 15:37 MichaelDaum  
yammer_activity_streams_ss.jpgjpg yammer_activity_streams_ss.jpg manage 170 K 21 Jun 2012 - 15:40 MichaelDaum  
Topic revision: r9 - 15 Jan 2013, OliverKrueger - This page was cached on 13 May 2020 - 15:32.

The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy