How to Build a Release

This manual process describes the autoBuildFoswiki.pl script

THIS PROCESS REQUIRES PICKING OVER WITH A FINE-TOOTHED COMB.

It is an important principle that anyone can build a release using the automatic tools, given nothing more than some standard F/OSS tools and an GIT checkout area. The following sections describe the stages that need to be gone through to correctly build a Beta or Final release.

The contents of a release are specified by the contents of the master lib/MANIFEST file, that defines the release as core plus default extensions. Core files are listed at lib/Foswiki/Contrib/core/MANIFEST. !include statements are used to make it possible to have a release composed by the lists in many MANIFEST files.

The instructions are for building a standard release using the tools/build.pl build script, on Linux. It should be possible to generate a release on Windows, but it's untested.

Nightly build process (Run for the "master" and "Release02x01" branches)

Note: when there is an official beta release, it will be uploaded

This will create tgz and zip files named using the version number in Foswiki.pm, and appending the last git commit abbreviated hash to get files named like Foswiki-2.2.0_Alpha-auto2ba0daa.tgz.

Note: The script will abort if there are unit test failures. (sending an email to the developer events mailing list)

The builds and test results will be uploaded to master build area and Release02x01 build area (not reachable)

---

Build procedure

Note: at this time, this document and the build script is heavily biased toward creating official releases

Before generating a release package:

1. Make sure that the javascript and css compression tools are installed on your build system. A production release requires:
   • https://nodejs.org/
   • https://www.npmjs.com/package/terser
   • https://npmjs.org/package/cssmin
2. Get a git clone (see GitRepository)
   • Checkout core to a clean build area
     • git clone https://github.com/foswiki/distro
   • once you have a clean checkout area you can re-use it for subsequent builds. just
     • git pull --rebase
     • git status -uno should show no modified files
     • git clean -f -d (removes untracked files and directories. Add -x option to remove "ignored" files)
3. Install default plugins (symbolic-link)
   • perl -wT pseudo-install.pl developer to install the plugins specified in MANIFEST and the developer tools
4. Run the /bin/view script from the browser and follow the bootstrap process to make your system functional
   • ensure that your apache has sufficient file and directory permissions for data and pub
5. Run the unit tests. DO NOT RUN TESTS AS THE SUPER-USER. Some tests will be designed to test boundary conditions where the current user is denied write access to files. Since the superuser cannot be denied write access, you may get false failures.

   export FOSWIKI_HOME=`pwd`  # the core directory
   export FOSWIKI_LIBS=$FOSWIKI_HOME/lib:$FOSWIKI_HOME/lib/CPAN/lib:your_own_perl_lib_directories_here
   cd $FOSWIKI_HOME/test/unit
   perl ../bin/TestRunner.pl -clean FoswikiSuite.pm

   • If possible, run unit tests using the oldest supported perl (5.8.8) as well as current perl-blead. This is relatively easy to do using perlbrew or plenv.
6. Run the TestCaseAmISane sanity check manual testcase
7. Install the TestFixturePlugin: perl -wT pseudo-install.pl TestFixturePlugin
   • Do not enable TestFixturePlugin in configure
   • CPAN prerequisites: CPAN:HTML::Tagset CPAN:HTML::Parser
8. Run all the automatic testcases in the TestCasesweb (use firefox, and open each 'run' link in a new tab)
   • LOG IN FIRST (admin with configure pwd?)
   • Make sure there is at least one user registered on the site.
   • Enable $Foswiki::cfg{INCLUDE}{AllowURLs} = 1; in the Security settings | Miscellaneouss area of configure in order to run Develop:TestCases.TestCaseAutoIncludeAttachment
   • You should see ALL TESTS PASSED on all test pages. If you see the test itself, something is not configured correctly (check CPAN dependencies).
9. check for memory leaks by running perl -Ibin tools/MemoryCycleTests.pl
   • CPAN prerequisites: CPAN:Devel::Monitor
10. Check the manifest: cd lib/Foswiki/Contrib/core; sh tools/check_manifests.sh
    • Some files (top level .html, *DotPm.txt in the system web, and AUTHORS are automatically generated by the build process in the staging area, and do not appear in the MANIFEST.
    • Expect to see the following warnings; they are OK:

      Processing manifest /var/www/foswiki/distro/core/lib/Foswiki/Contrib/core/MANIFEST
      The following 5 files files were found in git, but are not in MANIFEST:
      lib/Foswiki/Configure/Checkers/Plugins/README
      pseudo-install.pl
      pub/System/PreferenceSettings/prefs-stack.odp
      pub/System/ProjectLogos/foswiki-logo.svg
      pub/System/TopicsAndWebs/diagram.odp
      ... (continues for every extension)

11. Check that all doc topic have correct TOPICINFOs
    • git: (from root of repo) git ls-files | grep \.txt$ | xargs grep '%META:TOPICINFO{' | grep -v TestCases | grep -v 'author="ProjectContributor".*version="1"'
      • if some topics are not 'valid' edit the TOPICINFO by hand
12. Fix up release notes with new changelogs, and make sure title is correct. See:
    • https://foswiki.org/Tasks/FixesReleaseNotesTml
    • Note that the release note is edited by editing the topic data/System/ReleaseNotes02x01. The build script creates a file in the root of the zip called ReleaseNotes??x??.html, and the build script needs your installation to be running to look up the release note topic and show it with the simple text skin.
    • update the global release notes at data/System/ReleaseNotes.txt, i.e. make sure that the numbers of fixes match and the release date matches the day of the final release
13. Make sure root level index.html points to the correct release notes.
14. Add the new release to the System.ReleaseHistory topic.
15. Verify that any updated extensions have had their Version and Release strings updated:
    • git --no-pager log --pretty=format:"%h %an %s" HEAD...refs/tags/[last-release-tag] SomeExtension
16. Double check for any topics with non-ascii characters. ( perl -nlE 'say if /\P{Ascii}/' < FileToCheck )

Build a test release package

Do this from a freshly cleaned checkout - see step one above

1. Install default plugins (hard copy)
   • perl -wT pseudo-install.pl default to install the plugins specified in MANIFEST
2. Make system functional, so that the INSTALL.html guide can be generated
   • perl -wT pseudo-install.pl -A to build a default LocalSite.cfg (Use -force to overwrite an existing config)
3. from command line (using /var/www/Release01x01 as example)
   • export FOSWIKI_LIBS=/var/www/Release01x01/core/lib:/var/www/Release01x01/core/lib/CPAN/lib
   • export FOSWIKI_HOME=/var/www/Release01x01/core
   • cd /var/www/Release01x01/core/lib
   • perl ../tools/build.pl release
   • When asked if you want to name version, answer No
4. The release tgz and zip is now in /var/www/Release01x01/core/ with the name Foswiki.tgz and Foswiki.zip

After the package is built, but before it is uploaded

1. unarchive the .tgz in a clean area
2. make sure that the generated files are all correct
   • Check the generated .html files in the root, and make sure that there are no occurrences of foswikiNewLink (INSTALL.html, ReleaseHistory.html, UpgradeGuide.html and ReleaseNotes02x00.html)
   • Verify that JQueryPlugin/pub/System/JQueryPlugin/plugins/foswiki/jquery.foswiki.js does not contain any utf-8 characters - should be only ASCII.
3. configure the web server and run configure, make sure it works
   • you may have to change the permissions or ownerships of the files to match your web server setup ( chown -R www-data.www-data * works on my debian system)
4. Verify that all expected plugins are installed and functioning

Build a final release package and upload to Download web

1. Clean the build area again (see above)
2. Install default plugins
   • perl -wT pseudo-install.pl -link default to install the plugins specified in MANIFEST
3. Make system functional, so that the INSTALL.html guide can be generated
   • perl -wT pseudo-install.pl -A to build a default LocalSite.cfg (Use -force to overwrite an existing config)
4. from command line (using /var/www/Release01x01 as example)
   • export FOSWIKI_LIBS=/var/www/Release01x01/core/lib:/var/www/Release01x01/core/lib/CPAN/lib
   • export FOSWIKI_HOME=/var/www/Release01x01/core
   • cd /var/www/Release01x01/core/lib
   • perl ../tools/build.pl release -commit
   • When asked type of build
     • First alpha or beta: enter nextmajor, nextminor, or next
     • Next production "GA" release, enter manor, minor, or patch
   • When asked if you want to name version:
     • For alpha, beta, RC enter y, and then specify the suffix, alpha Beta or RCn
     • For a production "GA" release, press enter.
5. The release tgz and zip is now in /var/www/Release01x01/core/ with the name Foswiki-X.X.X.tgz and Foswiki-X.X.X.zip

• Check the Foswiki.pm VERSION and RELEASE tags - make sure they are correct.

Beta and release candidates are uploaded

• ..to foswiki.org to Download/FoswikiReleaseXXxXXxXXBetaX or Download/FoswikiReleaseXXxXXxXXRCX

Make upgrade packages

• Upgrade package is a full release package with a few files removed.
• We do not make upgrade packages for new major and minor releases. And not for beta or RC releases either.
• Unpack the release tgz to a fresh directory. Rename the directory to Foswiki-upgrade-X.X.X
• The following files are not distributed in the upgrade package

Filename	Manually excluded from Upgrade Package	Skipped by foswiki-upgrade-check
data/_default/WebHome.txt	X	X
data/_default/WebLeftBar.txt	X
data/_default/WebPreferences.txt	X	X
data/Main/AdminGroup.txt	X	X
data/Main/AdminUser.txt	X
data/Main/NobodyGroup.txt	X
data/Main/PatternSkinUserViewTemplate.txt	X
data/Main/ProjectContributor.txt	X
data/Main/RegistrationAgent.txt	X
data/Main/SitePreferences.txt	X	X
data/Main/UnknownUser.txt	X
data/Main/UserHomepageHeader.txt	X
data/Main/UserListByDateJoined.txt	X
data/Main/UserListByLocation.txt	X
data/Main/UserListHeader.txt	X
data/Main/UserList.txt	X
data/Main/WebAtom.txt	X
data/Main/WebChanges.txt	X
data/Main/WebCreateNewTopic.txt	X
data/Main/WebHome.txt	X	X
data/Main/WebIndex.txt	X
data/Main/WebLeftBarExample.txt	X
data/Main/WebNotify.txt	X	X
data/Main/WebPreferences.txt	X	X
data/Main/WebRss.txt	X
data/Main/WebSearchAdvanced.txt	X
data/Main/WebSearch.txt	X
data/Main/WebStatistics.txt	X	X
data/Main/WebTopicList.txt	X
data/Main/WikiGuest.txt	X
data/Sandbox/CommentPluginExampleComments.txt	X
data/Sandbox/CommentPluginExamples.txt	X
data/Sandbox/CommentPluginTemplateExample.txt	X
data/Sandbox/CompareRevisionsAddOnDemoTopic.txt	X
data/Sandbox/CompareRevisionsAddOnDemoTopic.txt,v	X
data/Sandbox/PluginTestEmptyPlugin.txt	X
data/Sandbox/WebAtom.txt	X
data/Sandbox/WebChanges.txt	X
data/Sandbox/WebCreateNewTopic.txt	X
data/Sandbox/WebHome.txt	X	X
data/Sandbox/WebIndex.txt	X
data/Sandbox/WebLeftBarExample.txt	X
data/Sandbox/WebNotify.txt	X	X
data/Sandbox/WebPreferences.txt	X	X
data/Sandbox/WebRss.txt	X
data/Sandbox/WebSearchAdvanced.txt	X
data/Sandbox/WebSearch.txt	X
data/Sandbox/WebStatistics.txt	X	X
data/Sandbox/WebTopicList.txt	X
data/System/InterWikis.txt	X
data/System/NewUserTemplate.txt	X
data/System/WebPreferences.txt	X	X
data/System/WebTopicEditTemplate.txt	X
data/Trash/TrashAttachment.txt	X	X
data/Trash/WebPreferences.txt	X	X
data/TWiki/WebPreferences.txt	X	X
index.html	X

• Manually remove the files listed above.- See attached file: clean-upgrade.sh: Shell script removes files not distributed in upgrade package
• Create the upgrade zip zip -r Foswiki-upgrade-1.1.1.zip Foswiki-upgrade-1.1.1
• Create the upgrade tgz tar -zcf Foswiki-upgrade-1.1.1.tgz Foswiki-upgrade-1.1.1
• Generate md5 sums of the two and add them to the already generated md5 file.

GPG sign the files

• Release manager signs the 4 files (2 for new minor/major) with their private GPG signature
• Example: gpg -s -b -a -u email-address Foswiki-1.2.0_Beta_1.tgz
• Result is 4 (2) .asc files.

Create a release tag, and if it's a new major release, a release branch

• create an git tag from the released version
  git tag -a FoswikiRelease01x02x00 -m "Item000: Foswiki Release 1.2.0"
git push --tags
• If you need to remove a tag due to an error:
  git tag -d FoswikiRelease01x02x00 (Delete tag locally)
  git push origin :FoswikiRelease01x02x00 (Remove tag from Github)

Git tags can also be signed with GPG.

Building an updated VM

See VirtualMachineImages.

Final releases are uploaded to Sourceforge and GitHub

Sourceforge

• https://sourceforge.net/projects/foswiki
• Project File Manager
• Create new folder in Foswiki folder. Use name X.X.X
• Upload both tgz and zip and md5 and the .asc files. Choice of Browser upload or use sftp, which seems to be more reliable for large files.:

• sftp <user>@frs.sourceforge.net
  Password: 
  Connected to frs.sourceforge.net.
  sftp> cd /home/frs/project/foswiki/foswiki/2.1.4
  

• Make the zip default for Windows and Others
• Make the tgz default for Unix/Linix/Solaris/BSD etc
• OBS! Do not mark any other uploads like VMs or debs or windows installers as default for anything. Only ONE file can be the default for a given OS and it should always be .zip or .tgz. It is this setting that defines which download a visitor sees on the Foswiki Sourceforge landing page.

Github

• https://github.com/foswiki/distro/tags (If the new tag is not there, did you tag, and push --tags?)
• Edit some release announcement text for the new tag.
• Drag/drop upload the release files. tgz. zip, md5, sha1 and asc files

Create a release branch if necessary

• Branch the release as well, so that urgent fixes can be made to the release without disrupting the HEAD development.
  git checkout -b FoswikiRelease01x02x01
git push origin FoswikiRelease01x02x01
git branch --set-upstream-to=origin/Release02x01 Release02x01

• If creating a new branch Be sure to fully configure it!

  • Add it to the FoswikiOrgPlugin regex on https://foswiki.org/bin/configure so that push requests get applied to tasks
  • Add it to the form on ItemTemplate as a new form field.

Build and upload all default extensions that have changed version since last Foswiki release

• Make a normal release with perl build.pl upload from the lib/Foswiki/Plugins|Contrib/NameOfExtention directory.
• If you are in doubt anything has changed, re-upload. It cannot harm uploading the same again.
• You can check in a Foswiki installation in configure if the version installed matches the version on foswiki.org.

Announce the release

• If System.ReleaseHistory is out of date - need to update and rebuild!
• Email to foswiki-announce mailing list
  • If there are security aspects, Include a please keep confidential until date
  • Use the SourceForge admin interface to hid the emails until public.
• Update the Download.WebPreferences to set the current release.
• Update About.ProjectUpdate
• Update Extensions.ExtensionNews
• Update Tasks.ReleasedIn and Tasks.Codebase and Extensions.IncompatibleWith to adjust older, current and future releases.
• Use Tasks.BulkCloseTasks to close the tasks in WaitingForRelease status. DANGEROUS - Admin use only
• Manually close the Documentation task for the release, create a new one and update the Tasks.WebPreferences banner to point to it.
• Update Tasks.ReleaseBlocker
• Update Support.QuestionForm
• update irc /topic
• https://libera.chat/
• blog about it at https://blog.foswiki.org
• Social Networks
  • LinkedIn
  • fosstodon
  • facebook
  • wikimatrix
  • cmsmatrix
• Press Releases
  • http://www.newswiretoday.com/