BuildContrib Cookbook

This topic has been partially updated for using git, but needs validation and further updates.

This is a cookbook of recipes for using BuildContrib.

BuildContrib is a powerful, multi-faceted build system, used to build Foswiki extensions and the Foswiki core itself. It started out as a simple perl clone of ANT, but has gained a diverse range of other functionality since then.

Extensions should be checked into git and pushed to github, so that's assumed here. Anyone who develops outside git is asking for trouble, and is pretty much on their own. See HowToStartExtensionDevelopmentInGit for help getting started with it.

Insert the name and type of your module below, and hit "rewrite instructions" to rewrite the instructions for your module. See the Extensions web for a description of the different types of extension module. However note that AddOn is no longer used and should be avoided.

Root of git checkout area
Extension name e.g. "Electric"
Extension type	PluginContribSkinAddOnWikiApp

Recipe 1: Setting up your dev environment

The best way to develop is in a cloned git repository. See GitRepository for help in checking one out. (Subversion has been frozen and should not be used.) Once you have a checkout area, follow the installation instructions to configure it as a running Foswiki so you can test your code in the browser. Then set these environment variables (adjust for your shell). The 2nd one is a path with two directories. The last one will pick up the recommended version of perltidy

export FOSWIKI_HOME=%GIT%/core
export FOSWIKI_LIBS=$FOSWIKI_HOME/lib:$FOSWIKI_HOME/lib/CPAN/lib
export PERL5LIB=%GIT%/BuildContrib/lib 

The beauty of using a checkout area is that if you make a mistake, it's easy to revert your changes, and you are always testing against the latest code. By installing the "distro" github repository, you already have the pieces needed for development.

Use pseudo-install.pl to install all the developer components into the core, thus.

cd /var/www/foswiki/core && ./pseudo-install.pl developer 

If you intend to create a "Contrib" or "Skin" you'll also need to install the EmptyContrib

cd /var/www/foswiki/core && ./pseudo-install.pl EmptyContrib 

If you want to be able to check your work into github directly, you need to be authorized. See GettingStarted

Recipe 2: Creating a new extension

This recipe describes the process of building a new extension called ElectricPlugin, using the create_new_extension.pl script installed with BuildContrib. (Before you go any further. If you intend to create a Contrib or Skin, pseudo-install EmptyContrib )

1. cd /var/www/foswiki
2. perl core/create_new_extension.pl ElectricPlugin

This will create the ElectricPlugin file hierarchy, including build.pl, DEPENDENCIES and MANIFEST. At this point, the Extension Topic (and all other files) are copied from Empty(Plugin/Contrib), and you will need to change those to reflect what you are working on.

Recipe 3: Making MANIFEST and DEPENDENCIES files

The MANIFEST file lists files that are released with the extension. Not all files in the source tree will be released; for example, it is unusual to release developer tests and support scripts such as those used for benchmarking.

The DEPENDENCIES file lists external dependencies, such as those on CPAN modules, or other Foswiki modules. The format of the DEPENDENCIES file is one dependency per line; dependency, version, type, and comment are comma-separated fields specified per dependency, defined below.

Storable,>=0,cpan,Required for saving attachment hash values and parameters
htmldoc,,external,Required

• Dependency name Package or module name to be installed. For Foswiki modules it is a fully qualified name, for example Foswiki::Plugins::SafeWikiPlugin, For CPAN modules, this should be the name as installed from the CPAN shell. For external packages, and for best compatibility with the .deb Debian packager, the name should have a unique hit when searched with apt-cache search <name>
• Version If a minimum version of the package is required, this should be a simple expression >10.1 for example
• Type This is one of perl, cpan or external. perl dependencies are only used for other Foswiki modules. Note that the Foswiki installer will not install external packages, however if the dependency matches a debian package, it will be added to the .deb package as a dependency.
• Description Description of the dependency. If the description starts with the word Optional, it will never be automatically installed.

After you run create_new_extension.pl the initial MANIFEST and DEPENDENCIES files are empty. You can generate draft contents thus:

1. cd /var/www/foswiki/ElectricPlugin/lib/Foswiki/Plugins/ElectricPlugin
2. perl build.pl manifest
3. perl build.pl dependencies (needs the CPAN module B::PerlReq)

These targets "guess" the content of the files; you need to apply some intelligence to derive the actual content.

Caution The dependencies build target assumes that all dependencies are already installed on the system. The tool will not detect unresolved dependencies that cause compile errors in the modules.

You can run these targets whenever you want, even after you have generated what you believe are correct MANIFEST and DEPENDENCIES files.

Recipe 4: Making unit tests

We strongly recommend using test-first development, and ensuring that you have unit tests in place for all your extensions. Unfortunately many people are too lazy, or too stupid (like the author of BuildContrib), to write tests for their extensions. If you are one of the wise:

1. mkdir -p /var/www/foswiki/ElectricPlugin/test/unit/ElectricPlugin
2. Create a testcase. The easiest way to do this is to base it on an existing testcase. For an example, see /var/www/foswiki/CommentPlugin/test/unit/CommentPlugin. You can find the complete library of unit tests for core and all pseudo-installed extensions in /var/www/foswiki/core/test/unit

Recipe 4b: Running the unit tests

• do not ignore test failures - maintaining a quality test suite is a major help to yourself and others in maintaining extensions

There are two ways to run the tests. You can run them from within the extension directory, or run them from within core. Running from within core simplifies some things.

1. cd /var/www/foswiki/core/test/unit
2. perl ../bin/TestRunner.pl ElectricPluginSuite (or if you didn't create the suite)
   • perl ../bin/TestRunner.pl ElectricPluginTests

Alternative:

1. To run the tests, cd /var/www/foswiki/ElectricPlugin/lib/Foswiki/Plugins/ElectricPlugin and perl build.pl test This method will only run Test "Suite" modules.
   • build.pl will print the command line used to run the tests, so you can easily copy-paste-modify the command line to home in on a failing test

Recipe 5: Debugging unit tests

Sometimes a unit test will fail (that's why you have them, after all) and you will need to debug. This is not so easily achieved with build.pl since it fires off separate processes to do the hard work. Fortunately there are some simple tactics you can use to help debug a test. Say that you want to know why, in your ElectricPlugin, test_Something reports a failure though it is supposed to work.

Recipe 5a: Isolate the failing test

At the bottom of the test run is a short TML report:

---

Module Failure summary

FoswikirefsPluginTests has 1 unexpected results (of 2):

• F: FoswikirefsPluginTests::test_GITREF

---

Here is where running the tests from within core is helpful. (The build.pl test cannot run individual tests.) Copy/paste the failing test line to run it again. (Note always include -clean when re-running tests).

1. cd /var/www/foswiki/core/test/unit
2. ../bin/TestRunner.pl -clean FoswikirefsPluginTests::test_GITREF

From here you can add print statements, or make other changes to isolate the issue.

Recipe 5b: Use a debugger

Needs updating

The following recipe will guide you to a Perl debugger session which stops at the first line of sub test_Something. It is less scary than it looks at first glance, you only need to type in what's in boldface.

1. $ cd /var/www/foswiki/ElectricPlugin/lib/Foswiki/Plugins/ElectricPlugin
2. $ export PERL5OPT=-d
   • ...or the equivalent for your shell. This environment variable contains options which well be passed to all invocations of perl from now on.
3. $ perl build.pl test
   • Fire off the tests.
4. Loading DB routines from perl5db.pl version 1.28
(...)
main::(build.pl:48): $build = new BuildBuild();
DB<1> c
   • Just continue, we are not going to find bugs in build.pl (we hope!)
5. Running tests in /var/www/foswiki/ElectricPlugin/test/unit/ElectricPlugin/TemplateToolkitPluginSuite.pm
(...)
main::(/var/www/foswiki/test/bin/TestRunner.pl:3):
3: require 5.006;
DB<1> b postpone ElectricPluginTests::test_Something
   • Daughter DB session, maybe in an new xterm. At the current line, your test suite has not yet been compiled, that/home/foswiki/core/test/bin/TestRunner.pl /var/www/foswiki/ElectricPlugin/test/unit/ElectricPlugin/ElectricPluginSuite.pm's what postpone is for.
6. DB<2> c
   • Start TestRunner.pl. Some test cases may run before test_Something (the ordering is hardly predictable), but eventually...
7. ElectricPluginTests::test_Something(ElectricPluginTests.pm:57):
57: my $this = shift;
DB<3>
   • Eventually, you're where you can start tracing. Phew! After you're done, you have to quit ( q) twice, for both parent and daughter session.

Final hint: If all your test cases fail, but perl never stops at the break points you set with this recipe, set a breakpoint at the init routine: b postpone ElectricPluginTests::set_up

Recipe 6: Testing your release package

You build and test your release package as follows.

1. If you followed recipe 5, remember to ./pseudo-install.pl -uninstall ElectricPlugin first, or bad things will happen
2. cd /var/www/foswiki
3. perl ElectricPlugin/lib/Foswiki/Plugins/ElectricPlugin/build.pl release
4. Copy package files into Foswiki root
   cp ElectricPlugin/ElectricPlugin* core/
5. cd core/
6. perl ElectricPlugin_installer -r install
7. Test in the browser, check the documentation etc etc

Recipe 7: Uploading your work

Please don't skip Recipe 6! When an extension is uploaded, it becomes visible to the configure "Find Extensions" interface. Extensions that Foswiki advertises in the installer should at least install and run without errors using the Plugin topic. Extensions that don't cleanly install or function reflect badly on overall Foswiki quality. Note: If your extension will not work or is not tested in some environments, note this in the package form. And if it has not been thoroughly tested, or has known issues, please upload to the Extensions Testing Web.

The best way to upload your release package is to use the upload target.

cd /var/www/foswiki/ElectricPlugin/lib/Foswiki/Plugins/ElectricPlugin

and then

perl build.pl upload

It will offer to upload to foswiki.org (you can change this to upload to another repository, for example if you are using BuildContrib to build company-specific plugins). Use your Foswiki username and password. It will cache them to a local, unencrypted file, $HOME/.buildcontrib! If you can't use the upload target, then manually upload/attach the files that were built when you did perl build.pl release.

If your upload is large, or your upload link is very slow and takes longer than 3 minutes, it may fail with a 500 Internal Server Error. This is because of the 3 minute timeout on the fcgi process. Change the upload to use plain CGI by setting the Scripts path to http://foswiki.org/cgi

Note that the perl build.pl upload command will attempt to upload any attachments in the metadata of the Plugin topic. If your plugin has attached any files that you don't intend to upload, delete the metadata or the upload will fail with missing file errors.

-- Contributors: CrawfordCurrie, SvenDowideit, TWiki:Main.HaraldJoerg, GeorgeClark, NicolasWrobel

Discussion

In Tasks.Item8531, we finally stopped checking in .gz and compressed versions of js and css files, which is a great step forward.

But now, autogenerated MANIFESTs are much less useful than they used to be, and large plugins like TinyMCEPlugin, JQueryPlugin, anything with many .css and .js files is very tedious to confidently come up with a comprehensive MANIFEST.

So below are a couple of vim tricks that I finally got working, so that I can reliably rebuild the manifest automatically and apply the following regexes to get the minified/gzipped versions where appropriate

1. Assume extension directory has all minified and gzipped files removed
2. Generate a new MANIFEST using ./build.pl manifest
3. Insert .gz entries for any .css or .js file that does NOT have .uncompressed in their name:

    %s/^\(.*uncompressed.*\)\@!\(.*\)\.\(css\|js\)\(.*$\)/\1\2.\3\r\1\2.\3.gz/

4. Insert .gz AND minified entries for all .uncompressed.files:

   %s/^\(.*\)\.\(uncompressed\)\.\(js\|css\)\(.*\)$/\1.\2.\3\4\r\1.\3\4\r\1.\3.gz\4

Added a full recipe for updating TinyMCEPlugin at Development.TinyMCEPlugin

Probably, this is a sign that the gzipping/minificaiton work in BuildContrib is incomplete

-- PaulHarvey - 17 Mar 2010

You might want to have a look at depgen, which I recently developed to generate DEPENDENCIES, the related challenge with large plugins. It actually scans sources, so I suppose it could be adapted to find css/js references if you aren't too cute about how they are generated. In any case, it does solve the DEPENDENCIES challenge. (The off-line tasks package has 50 dependencies scattered across about 140 source files.) Run it --man for docs (but not as root).

Of course it would be great if build.pl handled all this correctly - I couldn't figure out how to get the TWiki version to do the right things, especially for a complex object. Thus this tool.

Enjoy.

-- TimotheLitt - 08 Oct 2012

build.pl has been able to do this for a long time (perhaps even pre-fork?); perl build.pl dependencies

Again it doesn't handle JS or CSS dependenices; that's really tricky to do.

-- CrawfordCurrie - 08 Oct 2012