Feature Proposal: Move foswiki core lib/CPAN into its own CpanContrib
first up, I don't want it, I don't need it, and i always worry about what is and isn't up to date.
Given that the released default distro is made up of lots of contribs, I'd like to do the same for this.
Description and Documentation
Make a CpanContrib
, make a build.pl script that gets / updates the CPAN libs we want to ship in foswiki, and then include it in the foswiki core MANIFEST.
the net result is no apparent change for users of the zips, tgz, or normal
use, while minimal installers can thus get much closer to minimal.
-- Contributors: SvenDowideit
- 06 Oct 2012
Good idea. Also merge in the DateTimeContrib
created. No sense having multiple CPAN library contribs.
- 09 Oct 2012
And we should review and add any other CPAN dependencies that are not shipped with our minimum perl - 5.8.8, and add them to the CPAN contrib. I don't see how it makes sense to ship only a subset of CPAN that leaves a default install not fully functional.
Rather than shipping the compiled packages, which will cause issues with XS modules, could we work out something to ship the distribution files and a build process:
foswiki/lib/CPAN/sources containing the dependency tarballs
- Include an operational cpanm for simple installation
- Run cpanm to install each package using:
cpanm -L lib/CPAN/lib lib/CPAN/sources/HTML-Parser-3.69.tar.gz
If a network connection is available, we could offer to download the latest versions.
tries to do this dynamically, and also tries to use the OS package manager or cpanplus if it's unavailable. I think we'd be better off to just use cpanminus or cpanplus, and use this for non-package-manager based installations. The package manager really ought to be resolving dependencies without need for help from dependencies_installer.
- 23 Oct 2012
Florian's unit tests issue shows that this really needs to be dealt with. Conflicting versions in Foswiki/lib/CPAN/lib and System libraries of Class::Load and dependencies results in failure due to missing XS modules.
- Ship pre-installed CPAN deps by default. (current solution)
- Change default lib order so local Foswiki/lib/CPAN/libis first.
- Makes it difficult to install newer libraries locally
- Ship pre-installed CPAN deps in a separate (optional) contrib.
- Ship a deplist and tool to build missing deps with cpanm in a private lib.
- Do we need to ship the dist. files to accommodate sites installing without an Internet connection
- Ship a dependency installer like
tools.dependency_installer.pl it tries to install using multiple tools - .deb, cpanplus (deprecated), rpm ...
- Simplify / remove CPAN deps so that core doesn't need them.
- 19 Apr 2013
The rationale for shipping CPAN libs was "to avoid DLL hell" (not my words). If we don't ship CPAN libs, then we are throwing users to the lions - perhaps not a bad thing, but it will reflect badly on us even though it's really CPAN's fault.
- I am against building tools to resolve CPAN dependencies.
- I am in favour of simplifying things so that barely-required CPAN modules are not-required.
- I am in favour of not shipping any CPAN libs, but instead check, and report, if the libs found are inadequate.
- The doc on where it looks for libs really needs to be beefed up.
- 19 Apr 2013
+1 for not shipping any CPAN libs and instead informing the admin what's missing or too old. And it's only fair to provide basic pointers how to get those modules from one's distro or directly from CPAN, but no custom CPAN installer or contrib - IMHO plain 'cpan' works well enough and is included with Perl.
And yes, the library path thing is too complicated, and I fail to see the benefit of all the different variants (append, prepend, defined but empty...). I'd say if someone installs something locally, that's because they want that to be used in preference to what's in the system path. So all that needs to be done is prepend Foswiki/lib and $PERL5LIB to @INC, and since $PERL5LIB might not be set in the web server environment, one might want to set it in LocalLib
.cfg... or perhaps not, and it has to be set in the (web server or cli) environment, thus removing the need for LocalLib
- 20 Apr 2013
The library path defaults to "Append". AFAIK the reason is because the CPAN modules we ship are intended as a "Last Resort" so that Foswiki will work "out-of-the-box" with the system perl. If a user installs newer versions using CPAN or system package managers, we want to use the system version and allow ours to be ignored. This seems to be good 99% of the time.
The "prepend" is used for when the system supplied version is too old, and cannot be easily updated. (some hosting sites, IT groups, etc.). I don't recall needing to tell users to do this. I guess it's pretty rare.
Developers should occasionally test with Prepend - so that bugs due to our bundled CPAN can be detected. The unit tests might
do that automatically - I'm not sure.
The defined but empty and other variants ... I have no idea.
Regarding removing LocalLib.cfg, it's mostly never needed anyway. It's very rare that I've ever run into needing to set the lib. Foswiki generally figures it out. The only time I've ever touched LocalLib
.cfg was to enable ASSERTs, and to change to prepend to be sure I was testing with the code we ship.
As far as not shipping any CPAN modules, CPAN can be very painful. The only way I've found that is reliable is CPAN:App::cpanminus
. The new dependency on "version" has caused some issues. Class::Load is looking to be a disaster I'll bet. You cannot get it on Debian unless you pull it from Sid.
- 20 Apr 2013
Ditto with respect to only using LocalLib
.cfg for developer things, like ASSERT. I would be surprised if the nightly tests look at the lib setup at all - I don't recall ever having seen anything like that.
- 21 Apr 2013
I've been experimenting a bit with the
tool, and found some suggestions through various google searches. Here is a revised proposal to address the following considerations:
- For sites that have "disconnected servers"
- Ship a mirror directory with the distribution files for all Foswiki dependencies
foswiki/lib/CPAN/lib with =cpanm --mirror <mirror-directory> -L foswiki/lib/CPAN/lib
- For hosted sites that can't locally build perl modules.
- Ship a populated
lib/CPAN/lib directory that can be moved into the distribution
- Dealing with modules that use XS and PurePerl implementation.
- Build any local modules with --pureperl flag
- Ship with an empty
- Change the default @INC so
lib/CPAN/lib is at the head of the list, overriding any local modules
- Either include this contrib "on the side" as part of our normal tarball / zipfile, or as a completely separate download
A new installation would have 4 options, in order of preference:
- Use system provided packages. They don't need this extension, so just remove it if it we include it in the tarball.
- OR Use
cpanm to build the dependencies.
cpanm will fetch the latest versions from CPAN.
- OR Use
cpanm with --mirrorto build from the mirror we ship with the contrib
- Either of these options get the XS optimized versions when available
- OR Copy the lib/CPAN/lib directory from the contrib into foswiki.
- Only pureperl is available, but no building is necessary
cpanmirror: Directory containing mirrored distribution files
/lib/CPAN/lib: Directory with pre-build
Pure Perl modules
tools/cpanm: Installation of cpanminus
- shell script / bat file to execute each strategy
There is a bit of a chicken & egg problem in that foswiki configure requires some of the dependencies. I think this should be a separate part of the distribution, either a separate directory, or a separate download. The user would expand the
file, and then follow instructions in the install guide to install the dependencies.
- Populating the mirror
- cpanm -L /dev/null --save-dists cpanmirror --scandeps Log::Log4perl::DateFormat This command will find all the dependencies of the named package and download all of the distribution files into the
- Building a self contained library with pure perl versions
- cpanm -L lib/CPAN/lib --pureperl --mirror ~/cpanmirror Class::Load This builds Class::Load and all dependencies as a self-contained
- Building only modules not locally up-to-date
- cpanm -l lib/CPAN/lib --no-man-pages --mirror ~/cpanmirror Class::Load This builds Class::Load and missing dependencies into
Noticed two separate CPAN contribs out in svn. CpanInstallerContrib and CpanContrib. Neither of which are released.
- 13 Mar 2014
Sounds good. Only problem I see is changing the order in
to the top. For now, if you'd like to have your own CPAN libs being preferred over any system-installed ones you'd put them into
- 14 Mar 2014
- 14 Mar 2014
The @INC issue is what we discovered with
. I know it's dropped, but it exposed an issue. It had complex dependencies including some XS code. The local site provided out-of-date deps, and us including a subset of them only made it worse, since we couldn't override without modifying the complicated
I think that if a site chooses to use the CPAN libs we provide, the ones the user builds into
need to be ahead of the System perl libs. The old way, we were last as a fallback. New way, we are first, but (nearly) empty
and we will override out-of-date local packages if the admin chooses to activate them. Good example beyond the XS packages is version, which ships with some perl but too old for our use.
- 14 Mar 2014
IRC discussion, MichaelDaum
comments that we need both
a last-resort and an override. We have that today by putting the CPAN package directly into lib. for ex.
would override the system package, but putting it in
is a last resort.
This is a good call-out as it eliminates the need to edit
to override the system libs. We run without that file 99% of the time. Looking at what cpanm does building a "self contained" library, it creates
, so it's worth testing if we could we use an optional
structure as the override libs, and a default but nearly empty
as the "last resort. This needs to be tested.
The hosted user who cannot build dependencies and is unable to get the hosting site to install dependencies would move the
and hopefully be done. And a user who needs to override a single package could run
cpanm -L foswiki
One possible issue, cpanm appears to populate bin as well, creating two files:
. Not sure what to do with them, but I doubt we want them executable from the browser.
Ug... I can't find any way of blocking the update of bin. Based on this, I think it's best to recommend any use of
be run into a temp location and then copy the directory into foswiki
- 14 Mar 2014
As of 3 June 2014. Here are the install experiences on a new perlbrew 5.20.0 installation:
- Error: Missing Dependency:
- Net::SMTP and IO::Socket::SSL are required to autoconfigure mail
The email auto-configure requires SSL, especially if it's going to work on Google's gmail.
- IO::Socket::SSL requires Net::SSLeay, but that requires OpenSSL and is unavailable in a pure perl implementation.
- Add Net::SSLeay as an optional dependency.
Other missing dependencies
And Foswiki won't run with Asserts enabled.
CGI will be removed from the Perl core distribution in the next major release. Please install it from CPAN. It is being used at /var/www/foswiki/trunk/core/lib/Foswiki.pm, line 56.
And configure is very noisy, logging the following repeatedly:
CGI::Carp will be removed from the Perl core distribution in the next major release. Please install it from CPAN. It is being used at (eval 1), line 1.
CGI will be removed from the Perl core distribution in the next major release. Please install it from CPAN. It is being used at (eval 8), line 1.
CGI::Util will be removed from the Perl core distribution in the next major release. Please install it from CPAN. It is being used at /home/gac/perl5/perlbrew/perls/perl-5.20.0/lib/5.20.0/CGI.pm, line 29.
CGI::Cookie will be removed from the Perl core distribution in the next major release. Please install it from CPAN. It is being used at (eval 46), line 2.
These can probably be avoided if our CpanContrib
ships all these, and puts them at the front
of the @INC lib path.
- 03 Jun 2014
HTML::Parser doesn't seem to want to build in a pureperl version. It needs a compiler so we can't distribute it.
- 04 Jun 2014
With the latest version of Configure - it's now just another UI method. The rationale to continue to ship CGI::Session doesn't seem to make sense. We should just eliminate the rest of lib/CPAN/lib. I've reopened Tasks.Item12475
- 06 Nov 2014