Foswiki Developer Environment
Automates build and packaging process, including installer generation, for extension modules.
Summary of Contents
BuildContrib can be used to create a build script for your extension.
It is inspired by the Java ANT build tool developed by the Apache project,
but is targeted specifically at building Foswiki extensions. It is also
used for Foswiki release builds. The advantage of using BuildContrib is that it
dictates a standard structure and build procedure, which makes your extension
easier for you, and others, to build and maintain.
- not just for building code modules, can also be used to package WikiApplications, and even pure documentation packages.
- automatically generates an installer script that can help simplify end-user installation.
- includes a wizard script for creating a new extension.
- supports creating collections of extensions.
- This is a build and packaging module for use by developers, not an install module for end users (though it does build an installer script, among other things).
- The module has only been tested on Linux, but should work with Cygwin OK.
If you don't like reading documentation, and just want a fast route to creating a new extension, then:
- Install the contrib using the instructions below
cd to the root of your installation
perl create_new_extension.pl extension_name
then modify the extension files as required (including MANIFEST). Then when you are ready to create archives:
perl build.pl extension_name release
- Archives (.zip, .tgz) will be created in
The build module assumes:
- two kinds of extension modules; 'Plugins', and 'Contribs' (everything else e.g Skins, WikiApplications etc),
- you have some passing familiarity with build tools such as
- you are not developing your extension in a 'production' installation (something that is usually a really bad idea), but are instead doing the sensible thing and developing in a separate directory tree
- usually - but not always - a subversion checkout area.
Standard directory structure
BuildContrib is used to build the Foswiki core, as well as most extensions. This
document will focus on its use for building extensions. See the
file in a subversion checkout for information on building
Extensions are developed in subdirectories of the checkout.
For example, BathPlugin will be developed in
This directory is called the root directory
for the extension.
The standard directory structure under a root directory mirrors a standard
installation tree. Every plugin has some key files
name.pm - code file for the plugin, usually derived from EmptyPlugin
name/ - directory containing sub-modules used by your plugin, and for storing your
build.pl script and other support files. It is referred to as the module directory
build.pl - build script for this extension
MANIFEST - list of files to be installed
DEPENDENCIES - list of modules this extension depends on
configure setup for this extension
name.txt - your plugin/contrib topic
name/ - directory containing unit tests for the extension
Contribs are held in the
directory instead of
but otherwise in exactly work the same way.
Other directories normally found in a Foswiki installation may also exist under
a root directory e.g.
While development in a subversion checkout is strongly
recommended, it is also possible to develop in a normal Foswiki install. To do this, simply install the BuildContrib.
Setting up for Development
The first thing to do is to either
- check out a development environment from subversion, or
- create a separate, fresh, install for development.
Configure this install so it's a running installation; we'll refer to this as your
Now install the BuildContrib. In a subversion checkout,
to the installation root and
perl pseudo-install.pl BuildContrib
. In a non-subversion environment, install the
package. Make sure that all installed files are readable by the webserver user.
Your build script has to know how to find the Foswiki libraries, so it can
pick up the components of the build system. Set
(which is a
path, same as
) to point to your
directory in your development
is used to extend @INC _for the duration of the build
only_, so it won't mask problems during testing.
The approach we recommend
is to set
in your login script (e.g.
depending on what shell you prefer).
build.pl does not read
bin/setlib.cfg. It uses
$FOSWIKI_LIBS only to find the modules for the BuildContrib.
Each individual extension has its own build script, called
, in its
module directory. A build script is a perl script that takes a number of
as its parameters. For example,
perl build.pl test
will run unit
perl build.pl release
will build a new release.
The build script also accepts the following options:
| Do nothing; just print what you would have done
| Be verbose
| with target
upload, only upload the topic (not the archives)
Build targets are Perl functions, which operate on various data defined
in control files to build the various targets. Perl is used rather than
for portability reasons.
The targets you will normally use are:
| perform basic build steps
| Run Perl::Tidy on all perl modules listed in the MANIFEST
| run unit tests
pod and package a release zip
release and upload
| print a guess at the MANIFEST
| Generates a list of svn checkins with comments suitable for use in the history section of the plugin/contrib topic.
| Find and print a best-guess dependencies list (for DEPENDENCIES)
The default target is
. The BuildContrib is designed so that most common behaviour is catered for. It is also easy to override
any of the default targets in your
and add extra behaviours.
Does nothing by default. This is the first target executed, and can be overridden by your build.pl to do something unusual - for example, executing an ANT file to build some Java.
files listed in MANIFEST that have a corresponding
in the directory structure. When it finds one, it automatically compresses the
file to create/refresh
. If you are using Subversion, the generated file should then by checked in.
Also works on CSS files, for the file extensions
to perform the compression
This target runs Perl::Tidy (with default formatting options) over your
source code. This reformats the code consistently with the Foswiki
The test target is designed for use with extensions that have unit tests written using the UnitTestContrib. It automatically runs the unit tests found in the
The results of the
- a Zip format archive,
- a gzipped tar archive,
- a md5 checksum,
- the extension topic,
- an installer script
The archives will each contain the following:
- All the files listed in the
- Another copy of the install/uninstall scripts
This target builds a release, and then tries to upload it to a target repository. The target uploads all the files in the release, and also tries to upload any attachments to the extension topic (as found by scanning the topic for META:FILEATTACHMENT).
You can control what server the upload is done to. This lets you - for example - set up your own corporate extensions server.
These are used when you are unsure of the correct contents of MANIFEST and DEPENDENCIES. They make a best-guess at the required contents for these files.
is the precursor of Foswiki, and some support
for building extensions for TWiki is built in to the BuildContrib
Firstly, extensions that were developed for use with the old TWiki
BuildContrib can be built using the Foswiki BuildContrib. Just modify
to load the Foswiki build system rather than the old TWiki
has a special target,
, which can be used
with a Foswiki build script to generate a TWiki directory structure and build
script, that can then be used to build an extension targeted at TWiki. The
files in the extension are run through a number of "mapping rules" that
will map much of the Foswiki namespace to TWiki. This transformation is not
complete, because Foswiki has many more features than TWiki, and because
extensions will work in TWiki after this transformation, and for others it
can be used as a launchpad for further manual mapping steps.
$ cd EditRowPlugin/lib/Foswiki/Plugins/EditRowPlugin
$ perl build.pl twiki
$ cd ../../../TWiki/Plugins/EditRowPlugin
$ perl build.pl release
Building a release for Version 0 (15 Feb 2009) of EditRowPlugin
MD5 checksums in EditRowPlugin/TWiki_EditRowPlugin.md5
.tgz in EditRowPlugin/TWiki_EditRowPlugin.tgz
.zip in EditRowPlugin/TWiki_EditRowPlugin.zip
WARNING: no .txt was generated
WARNING: no _installer was generated
There is no TWiki-specific topic generated. The Foswiki topic should suffice.
Installer generation is also disabled using
!option installers none
the tranformed MANIFEST. Users must install the generated TWiki packages
manually from the command-line. This is required due to bugs in TWiki.
prefix on the archive names. This is useful to avoid naming
clashes with the standard Foswiki release of the same package. It is defined
!option archive_prefix TWiki_
in the tranformed MANIFEST.
Extension authors are strongly recommended to check the functioning of the
TWiki versions of their extensions very carefully.
is a trademark of Peter Thoeny.
file contains a list of all the files that are wanted in the
package. Each line is a file path, relative to the root of the installation.
Wildcards may NOT be used.
If the path contains spaces it must be enclosed in double-quotes.
Each file path has an optional octal permissions mask and a description.
data/System/BathPlugin.txt 0664 Plugin description topic
lib/Foswiki/Plugins/BathPlugin.pm 0444 Plugin code module
If no permissions are given, permissions are guessed from the permissions on
the file in the source tree. These permissions are used by the installer
script to set file permissions in the installation.
The following permissions are recommended, and will be applied by default if
you don't specify anything different:
| File type
|| Anyone can read, but cannot write or execute
|| Anyone can read, user and group can also execute
|| Anyone can read, only owner can write
| File in
| File in
|| Anyone can read or execute, but not write
| Anything other file
|| Anyone can read, but cannot write or execute
|| default directories to traversable
,v files. If you include a
,v file it will overwrite any existing
,v file when an extension is upgraded, potentially wiping out local changes on the end users installation.
MANIFEST, or any other side file used by the build process.
- unit tests
MANIFEST files can contain a number of directives that are used to control
aspects of the build process. These directives always start with an
exclamation mark (!) and must be on a line on their own.
By default, files in the data and pub directories are
automatically checked in to Foswiki when the installation script is run
(for example, when installing from
). This is useful when you
expect users to customise your files locally and you don't want to risk
overwriting their customisations. If you want to suppress this checkin
behaviour for individual files, you can add the string
in the description of the file. If you want to suppress it for larger
numbers of files, you can use the
directives in the
MANIFEST. Any files listed after a
directive, up to the next
directive or the end of the file, will not be checked in when installing
to Foswiki 1.0.1 or later
MANIFESTs can also include other extensions that have been packaged using
BuildContrib. For example,
This will include the WysiwygPlugin in the release package.
Note that there is a script in the Foswiki
that can be run at any time to check the contents of your MANIFEST against what is checked into Subversion.
is a general directive used to define global options. Currently
supported options are:
!option archive_prefix String_ will prefix the name of generated archive files with =String_
!option installer none will suppress the generation of an installer script.
file specifies dependencies on other extensions and
perl modules. Each line of the file is a single dependency:
name, version, type, description
- name is the name of the module,
- version is the version constraint (e.g. ">1.5"),
- type is its type (cpan, perl, C etc) and
- description is a short description of the module and where to get it.
The installer script written by the build process uses the dependency type
to decide how to install dependant modules. 'cpan' means 'get the module
from CPAN' and 'perl' means 'get the module from the Plugins web on Foswiki.org'
(or whatever other repositories the admin has specified using
When your module (the depender
) depends on another module (a dependee
), it is important to think carefully about what version of the dependee your module requires.
When you are working with Foswiki modules (such as contribs and plugins) you should list the version number of the module that you tested with. Normally you will want to use a
condition, so that more recent versions will also work. If a dependency on a Foswiki module fails (because the module isn't installed, for example) then the installer script will pull the latest version
of the module from Foswiki.org, whether that is the required version or not. This is a limitation of the way plugins are stored on Foswiki.org.
When you are working with CPAN modules, you need to take account of the fact that there are two types
of CPAN modules; built-ins
are perl modules that are pre-installed in the perl distribution. Since these modules are usually very stable, it is generally safe to express the version dependency as ">0" (i.e. "any version of the module will do").
Note however that the list of built-in modules is constantly growing with each new release of perl. So your module may be installed with a perl version that doesn't have the required module pre-installed. In this case, CPAN will automatically try to upgrade the perl version
! There is no way around this, other than for the admin on the target system to manually
install the module (download frm CPAN and build locally). You can help out the admin by expressing the dependency clearly, thus:
File::Find,>0,cpan,This module is shipped as part of standard perl from perl 5.8.0 onwards. If your perl installation is older than this, you should either upgrade perl, or manually install this module. If you allow this installer to continue, it will automatically upgrade your perl installation which is probably not what you want!
A dependency may optionally be preceded by a condition that limits the cases
where the dependency applies. The condition is specified using a line that
ONLYIF ( condition )
, where condition
is a Perl
conditional. This is most useful for enabling dependencies only for certain
versions of other modules. For example,
ONLYIF ($File::Munge::VERSION < 1.025)
MyPackage::FixOldFileFind, >=1.000, perl, Optional, only required if we have an old version of File::Munge.
only applies to the next dependency in the file.
Writing a build script
The easiest way to write a new build script is to use the
script, which is part of the BuildContrib.
- Create your plugin source tree
perl create_new_extension.pl BathPlugin
- if you are working in a Subversion checkout, move the directory just created up one level to be at the same level as
lib/Foswiki/Plugins/BathPlugin.pm as required to create your plugin functionality
lib/Foswiki/Plugins/BathPlugin/MANIFEST and make sure it lists all the files you want to include in the release package
During development we recommend you use the
soft-link your development code into your dev install. This script uses the
MANIFEST you write and creates softlinks (copies, on Windows) in your dev install that allow
you to run your test code without having to do a full re-install each time you
make a change.
If you have a pre-existing extension, and you want to package it for use with
BuildContrib, then you (may) need to create the module directory and write the
files. The easiest way to do this
is to copy those files from an existing extension in subversion, and modify
them for your use.
Preparing the Installer
The installer script generated by the builder when target
is used is based on a template. This template is populated with lists of files and dependencies needed to make the extension-specific installer script.
You can extend this script by providing
files in the module directory.
These optional files are embedded into the template install script
at the appropriate stage of the installation. Read
(in the BuildContrib)
to see how they fit in. The POD comments in that module indicate the functions
that are most likely to be useful to anyone writing a script extension.
You are stongly recommended to develop a unit test suite for your extension.
Unit tests are kept in the
directory for each extension.
To run the unit tests you will need to set up the test environment as described
The easiest way to generate tests for your extension is to copy the approach taken in another extension. See for example ActionTrackerPlugin and CommentPlugin, which both have extensive test suites.
Tests are run using
Building a release
When you are almost ready to release, you should
- Build a release package and installer
perl build.pl release
- Remove the softlinked version from your dev install
cd dev install
perl pseudo-install.pl -uninstall BathPlugin
- Install the release package you just built:
cd dev install
target automatically expands certain tokens in
and in the installer script. The following tokens are supported:
%$MANIFEST% - table of files in MANIFEST
%$FILES% - hash keyed on file name mapping to permissions i.e. 'data/System/ThsiTopic.txt' => 0664, 'lib/Foswiki/Plugins/BlahPlugin.pm' => 0775
%$DEPENDENCIES% - list of dependencies from DEPENDENCIES
%$VERSION% subversion number and the date of the most recent checkin
%$RELEASE% value of the
$RELEASE perl global variable from your master perl module
%$DATE% - local date
%$POD% - POD documentation for the package, excluding test modules.
%$PREINSTALL% - contents of PREINSTALL
%$POSTINSTALL% - contents of POSTINSTALL
%$PREUNINSTALL% - contents of PREUNINSTALL
%$POSTUNINSTALL% - contents of POSTINSTALL
%$BUGSURL% - URL of bugs web
%$INSTALL_INSTRUCTIONS% - basic instructions for installing
| Version: |
row in the table in the extension topic to determine what version of the package is installed. In the sources this is normally set to
| Version: | 3227 (2009-03-20) |
. When you
perl build.pl release
, 3227 (2009-03-20) is computed
by finding the most recent subversion checkin of any file listed in MANIFEST, so it's very reliable and low maintenance. You don't have
to use 3227 (2009-03-20) in the
| Version: |
row of the extension topic - you can use your own version string if you want, or you can use Foswiki-1 which will take whatever value you have assigned to the $RELEASE variable in the extension master perl module.
supports all of subversion revision numbers, manually generated triples (1.2.3), ISO dates, and
dd Mmmm yyy
format dates as valid revision identifiers.
When you are happy the release package is built correctly, you can
perl build.pl upload
By default the
target will upload to Foswiki.org. You will be prompted
to enter an alternate upload target, should you require it (e.g. to upload to
private corporate repository). The upload updates the topic and any associated
Var topics published by the extension, and uploads zip, tgz, md5 and installer
Installer scripts build by BuildContrib are important for the full
functioning of the extensions installer in
The installer script shipped with the package is very simple. By default all
it does is to check the dependencies you list, and if necessary download and
install any missing Foswiki and CPAN modules. Other dependencies are simply
checked. Topics shipped with the
module are automatically merged into any existing local copies, ensuring
histories are preserved.
If you want your installer to do anything else then you will need to write a POSTINSTALL script
You are strongly recommended to use this Contrib to help split your code development away from your live Foswiki environment, as described above
You do not need to install anything in the browser to use this extension. The following instructions are for the administrator who installs the extension on the server.
Open configure, and open the "Extensions" section. Use "Find More Extensions" to get a list of available extensions. Select "Install".
If you have any problems, or if the extension isn't available in
, then you can still install manually from the command-line. See http://foswiki.org/Support/ManuallyInstallingExtensions
for more help.
Another great Foswiki extension from the WikiRing
- Working together to improve your wiki experience
|| Crawford Currie
| Copyright ©:
|| 2004-2009, Foswiki Contributors
|| 3227 (2009-03-20)
| Change History:
| 20 Mar 2009
|| Foswikitask:Item1338: added SHA1 checksum generation - Foswiki:Main.WillNorris
| 5 Mar 2009
|| Foswikitask:Item1198: Improved support for %$VERSION% (made it much more accurate) and changed the generated date format to ISO. Also added support for %$RELEASE%, an optional release identifier taken from the master perl module.
| 15 Feb 2009
|| Foswikitask:Item1079: Added twiki target
| 31 Jan 2009
|| Macro expansion works even in non-english locales (Foswikitask:Item924)
| 03 Dec 2008
|| Re-release for Foswiki; copyright assigned to Foswiki Contributors
| 31 Aug 2008
|| TWikibug:Item5971 BuildContrib broken upload on 4.2.2 with template login. Also adding query to optionally skip topic attachment upload.
| 22 Apr 2008
|| TWikibug:Item5556 BuildContrib defaults all unspecified permissions to 664 && TWikibug:Item5455 BuildContrib doesn't cope with larger numbers of files - TWiki:Main.SvenDowideit
| 31 Jan 2008
|| TWikibug:Item4751 support spaces in MANIFEST TWikibug:Item4990 removed dependency on external md5sum program TWikibug:Item5309 added tidy target
| 10 Sep 2007
|| TWikibug:Item4600 upload VarXXX topics to Plugins web, to support pluggable documentation architecture. Minor doc improvements. TWikibug:Item3839 add build script at the root to help with building several plugins in one session TWikibug:Item4601 remove duplicate installer script (.pl)
|| TWikibug:Item2006 fixed default permissions to allow group write, and rewrote the main (this) doc to be more useful. TWikibug:Item3624 added a pause between uploads (20s)
|| TWikibug:Item3118 Remember last upload target for each extension you upload. Handle upload to sites that use Template login. TWikibug:Item3445 Carry attributes over for files that are uploaded with the topic, so they remain hidden. Also added date to default version string.
|| TWikibug:Item3597 Duplicated _installer in _installer.pl, so that 4.1 extensions installer can use it
|| Fixed attachment upload. Will now upload attachments attached to the main topic
|| Install made easier to use for developers and end users; will now download an archive if it can't find one locally. Added dependencies target.
|| TWikibug:Item1718 discovered that svn log doesn't work recursively, so more code required to find changes. Added new inline tokens for various URLs. Updated history to 8894.
|| TWikibug:Item1718 added rel=nofollow to generated links to ~develop
|| TWikibug:Item1718 Added
history target to support extraction of history from SVN logs.
|| TWikibug:Item1527 BuildContrib: Remember original filedates for pub files
|| TWikibug:Item1527 BuildContrib: Oops, some (very few) plugins are not ActionTrackerPlugin
|| TWikibug:Item1527 Generated install scripts now really attaches files in pub to topics (means less of these couldn't update history error messages)
|| TWikibug:Item1527 BuildContrib: Execute topic saves in the TWiki web as AdminGroup by default
|| TWikibug:Item1347 made build.pl tidy up it's mess; and switched buildcontrib over to using push and pop on dirs instead of cd, so we don;t get lost so easily. Oh, and corrected the permissions on the generated viewauth and rdiffauth files
|| TWikibug:Item1393 For release topic name, removed upper case change of suffix. E.g. 'TWiki-4.0.0-beta6' becomes 'TWikiRelease04x00x00beta06'
|| TWikibug:Item1393 added final parameter to build.pl (release name) and added processing to derive topic name from release id.
|| TWikibug:Item663 fixing typos, etc
|| TWikibug:Item1374 BuildContrib wasn't deleting its temporary files staging area
|| TWikibug:Item663 fixing formatting/typos
|| TWikibug:Item1347 changes to support arbitrary release naming, to better support TWiki builds. Should not affect extension builds.
|| TWikibug:Item1285 Removed TWiki_installer.pl from release package
|| TWikibug:Item956 skins have their MANIFEST in lib/TWiki/Contrib/...Skin/
|| TWikibug:Item437 Build.pm reverted - sorry people, it seems this broke the build
|| TWikibug:Item437 Revised BuildContrib to build MD5s. Crawford - feel free to ditch anything you don't like. (I sent you email about this a couple of days ago). MD5 files are generated for each package and then aggregated during a hands-off install to collect all dependent MD5s into package/DEPS.md5
|| TWikibug:Item569 added default RELEASE to everything that had a version, and removed a load of dead code that was getting in the way
|| TWikibug:Item598 removed comment that was getting added to txt files outside data
|| TWikibug:Item569 computed version numbers for plugins from the repository rev they were built from.
|| TWikibug:Item559 fixed permissions
|| TWikibug:Item562 added ability to update VERSIOn number to current repository head
|| TWikibug:Item561 added a script to build and upload all the plugins, so we can have releases of them from SVN. Presently uploaded to the same topic as the beta releases.
|| TWikibug:Item432 applied the final polish (I hope) before beta release
|| TWikibug:Item437 generated md5 sums for packages built using build.pl, and added them to the upload
|| TWikibug:Item436 incremented vernos of all changed plugins
|| TWikibug:Item421 polishing up installation, trying to catch gotchas, improving docs
|| TWikibug:Item143 more apache warnings; and a silly error in comment plugin tests eliminated
|| TWikibug:Item404 removed CHANGELOG from MANIFEST and added build scripts for all plugins and contribs, so they can be used with the main build.
|| TWikibug:Item384 As excellent as Antonio's solution is, it breaks all the save script tests and is incompatible with previous releases. Need to fall back on the old 'action' parameter if the new parameters are not available - even though it is really bad news (it blocks the use of 'action' as a form-field name :-(. Also updated the script documentation for Antonio's changes.
|| TWikibug:Item380 do as the man says; make all $/ local
|| TWikibug:Item196 Extracted the Manifest File processing from Build.pm to a module that can be used by other Build implementations. Needed for a CommandSet of TWikiShellContrib
|| TWikibug:Item196 more plugin and contrib fixes for develop; mainly just moving tests around and making sure they all pass.
|| TWikibug:Item243 simplified gendocs to stop it generating ,v files, since Sven is going to solve that. Coorected MANIFEST for source code documents.
|| TWikibug:Item243 added build scripts for all plugins shipped with TWikiForGenericUse
|| TWikibug:Item244 extracted target_stage and target_archive from target_release to allow me to add functionality to target_stage
|| TWikibug:Item237 removed requirement to set TWIKI_LIBS for core build
|| TWikibug:Item237 now supports automatic instantiation of plugins and contribs in the release tree. Install scripts are not run.
|| TWikibug:Item237 MANIFEST build based on BuildContrib. Very few extensions to BuildContrib were required to do it. Things that are missing are: 1. generation of ,v files 2. packaging up and inclusion of plugins.
|| TWikibug:Item229 added support for sub-headings in
configure. Added support for adding comments to LocalSite.cfg to BuildContrib. Can now create sections in
configure for individual plugins.
|| TWikibug:Item181 new version, supports DEVELOP and Cairo, and adds support for attachments (you can now mail attachments to topics)