Item10780: Plugins need a working/temp file cleanup mechanism

Garbage collection for plugin-created files is a sore point. Right now, every site is on its own coming up with some maintenance script - and administrators pretty much have to read the code of every plugin that they use to find out what they have to do. And do it again with every plugin release.

Some plugins create files in pub/; others in the working area. Some of this data is intended to persist; other data is per-session. File naming conventions vary. But if you do nothing, sooner or later either your disk fills, or you delete something you shouldn't have.

This is an unreasonable burden for the site administrator - especially if you aren't a perl coder.

Here's an idea that I came up with over on http://twiki.org/cgi-bin/view/Plugins/GaugePluginDev:

Extend tick_foswiki to plugins. Something like (this needs to be Foswikified):

# Run plugin garbage collectors

use warnings;
use strict;

foreach my $plugin ( @{$twiki->{plugins}{plugins}} ) {
    next if( $plugin->{disabled} );

    local $TWiki::Plugins::SESSION = $twiki;

    my $cleanup = $plugin->{module} . '::pluginCleanup';

    if( defined( &$cleanup ) ) {
	no strict 'refs';
	&$cleanup( $twiki, $now );
	use strict 'refs';
    }
}

where plugins can have a hander somewhat like this proposed default:

# Garbage collection function called by the tick_foswiki.pl cron script
# Your plugin should cleanup any old/orphaned files that your plugin creates
# in your working area (or elsewhere).

sub DISABLED_pluginCleanup
{
    my( $foswiki, $now ) = @_;

    my $wa = Fosiki::Func::getWorkArea($pluginName);

    my $oldest = $now - (24*60*60);  # One day - adjust to suit or use a config variable (e.g. $Foswiki::cfg{$pluginName}{MaxAge})

    # Age-based cleanup

    foreach my $wf ( glob( "$wa/*" ) ) {
      unlink $wf if( (stat( $wf ))[9] < $oldest );
    }

    # Orphan cleanup.  Suppose your working files are "$web_$topic_data" 
   # but .conf files are never to be deleted:

    my $webRE = Foswiki::Func::getRegularExpression('webNameRegex');
    my $topicRE = Foswiki::Func::getRegularExpression('wikiWordNameRegex');

    foreach my $wf ( glob( "$wa/*" ) ) {

      my( $web, $topic, $type ) = $wf =~ m!^($webRE)_($topicRE)_.*(?:\.(.*))?$!;

      next unless( defined $web && defined $topic && defined $type && $type ne 'conf' );

      unlink $wf unless( Foswiki::Func::TopicExists( $web, $topic ) )    }
    }
} 

I prototyped this on twiki - here's a working example for the (recently patched) VarCachePlugin:

# =========================
sub pluginCleanup
{
    my( $twiki, $now ) = @_;

    my $webRE = TWiki::Func::getRegularExpression('webNameRegex');
    my $topicRE = TWiki::Func::getRegularExpression('wikiWordRegex');

    my $wa = TWiki::Func::getWorkArea($pluginName);

    foreach my $wf ( glob( "$wa/*/*_cache.{head,txt}" ) ) {

      my( $web, $topic, $type ) = $wf =~ m!^$wa/($webRE)/($topicRE)_cache\.(head|txt)$!;

      next unless( defined $web && defined $topic );

      unlink( $wf ) unless( TWiki::Func::topicExists( $web, $topic ) );
    }

}

I didn't cleanup directories in the case of web deletion for this quick prototype, but that's an easy exercise for the reader. A production version would probably also read the topics found & unlink cache files whose topic no longer contains %VARCACHE(?:{(.*?)})?%. I left these out to make the mechanism clear.

Obviously, plugins are free to use different schemes and may have to deal with directories - but at least this would provide a common mechanism.

Whatever API is added should be TWiki-compatible - hopefully this is one where both groups can get along.

Thoughts?

-- TimotheLitt - 23 May 2011

I think it's a great idea.

I think though, and I'm sure the TWiki crowd would also agree, it'd be nice to keep the plugin handlers named as a sort of fooHandler (Eg. beforeSaveHandler, preRenderingHandler, etc). So something like Foswiki::Plugins::EmptyPlugin::tickHandler.

We shouldn't have to worry about which plugins are disabled and which aren't - triggering with Foswiki::Plugins->dispatch('tickHandler') should take care of all that (I probably have the call signature wrong, I guess it's $session->{plugins}->dispatch('tickHandler') or so) but you get the idea

Needs a feature proposal.

-- PaulHarvey - 23 May 2011

I put a feature proposal in http://foswiki.org/Development/PluginGarbageCollection; the corresponding TWiki proposal is http://twiki.org/cgi-bin/view/Codev/PluginGarbageCollection.

I don't have a strong opinion about how to spell hander names. I did choose not to use Handler because unlike all the others it isn't used in any operation run by the webserver (rendering, save, attaching). It seemed to belong to a different family of things. So along the lines of initPlugin, I made this pluginCleanup.

For the same reason, I didn't put it into the registered handlers list and all that - this way there's no change to the core (except the tick script), no runtime overhead (except for compiling the unused subroutine). I didn't see a callable "for each plugin" iterator, and calling {plugins}->_dispatch required all that hookup.

But I can be had on implementation detail. The key thing is to do something to get control of the plugins' (semi-?)stable storage. This is my baseline...

-- TimotheLitt - 23 May 2011

Valid points about the handler name. I set back to New.

-- PaulHarvey - 23 May 2011

I've just commited Timothe's periodic.tgz (v2.0-005) to TasksPlugin and ported and bug fixed the Configure UI.

for my needs I can't have a daemon running, so I'll be poking around to see what I can do otherwise (and v3 looks like a massive beast!)

-- SvenDowideit - 06 Oct 2012

You want V3. I'ts not as massive as it seems; a lot of small files for modularity. V2 is dead - it was an interesting prototype, but nothing more. Since it was really only in my lab, I removed the compatibility API. So please, don't use it.

V3 addresses all the feedback from before. Hopefully the doc will get extracted from the sources and put into a topic so it's accessible.

I'm hoping for some help from the other wiki folks. I've been off-line for quite a while.

-- TimotheLitt - 07 Oct 2012

(I also comited the v3 code, and converted the configure portion so I could see the UI - and have been talking to Timothe about it )

-- SvenDowideit - 07 Oct 2012

The checkins all seem to be in a unreleased TasksPlugin, so I've moved this from a core feature to an extension for now. This needs a FeatureRequest if it's going to get into core.

-- GeorgeClark - 19 Jun 2014

See PluginGarbageCollection

-- GeorgeClark - 19 Nov 2015

Commenting is disabled while not logged in