Feature Proposal: Wrap all outgoing mails into uniform and safe MIME envelope.

Motivation

Present implementation of mail sending is hacky, sometimes fails on sending utf-8 containing mails and puts extra burden of forming proper mail format on template writers.

Description and Documentation

It is all started when I needed to reset a password while having Ukrainian language chosen for the interface. That way it was discovered that Net::SMTP doesn't tolerate utf8 flag being set on a mail text. Tracing down the bug led to a conclusion that Foswiki::Net does too much hand work on a message headers instead of relying on some widely accepted module which would do the same thing with higher level of confidence that the job would be done properly. Same time Foswiki code would become way more readable and consequently more manageable. Yet using utf8 charset combined with quoted-printable encoding for the text parts of a message makes it both much safer for transporting (I know that a mail relay without utf8 support is a nonsense nowadays but life is so surprising sometimes!) and for handling by a mail client.

And after all isn't it a good idea to offload as much work as possible from template writers and let them focus more on the content of a message rather than on details of its format?

Sure, the task requires additional CPAN dependency. One with a relatively small number of dependencies has been chosen for the task – Email::MIME. It's rather widely recommended and it does what is needed for the task.

I have a ready to test patch.

Examples

Impact

We Need to know the minimum version of Email::MIME we can use. Currently the code relies on Email::MIME version 1.936. Please add to the following table, and indicate if you would be unable to update to a newer version.

Distribution	Version	Comments
Debian 6.0.5	1.903	Obsolete stable. - Debian 8 is current	edit
FreeBSD 8.4	1.929	Obsolete release. 9.x is legacy, 10.x is production	edit
FreeBSD 9.1+	1.936	9.1 is out of production but still has the newest ports tree.	edit
Red Hat epel	1.926		edit
Ubuntu 12.04 LTS.	1.910	Supported through 2017	edit
Ubuntu 14.04 LTS.	1.925	Current LTS release	edit
Ubuntu 15.04	1.926		edit

Implementation

-- Contributors: VadimBelman - 03 Nov 2015

Discussion

The patch has been tested with different mail transports and for SMIME compatibility. Except for Net::SMTP SSL and TLS method everything else works perfectly find. These two are failing on both patched site and my standard production one. So I suspect either a problem of my mail server setup or some deeper internals of code dealing with SSL connections. Further investigation required but it doesn't affect the basic idea.

-- VadimBelman - 03 Nov 2015

In my quick testing, TLS over submission port is working for me. This needs some review by CrawfordCurrie, who did the bulk of the utf-8 work. Of course as a complete patch it also needs changes to the DEPENDENCIES file, documentation, etc.

-- GeorgeClark - 04 Nov 2015

To get this feature requests properly considered, please add a CommitedDeveloper and date of commitment. December 1st is feature freeze for 2.1, so this needs to get the 14-day timer started asap.

-- GeorgeClark - 04 Nov 2015

Thanks for the testing and the advises. Fixed and updated the patch.

-- VadimBelman - 04 Nov 2015

It has a cost

This is not a standard CPAN lib getting added to the standard Foswiki distro.

Yet another problem for people installing on shared hosts.

I just tried to see how many dependencies there are installing this on a RedHat

Dependencies Resolved

 Package                                           Arch                         Version                              Repository                  Size
Installing:
 perl-Email-MIME                                   noarch                       1.926-1.el7                          epel                        45 k

Installing for dependencies:
 perl-Email-Address                                noarch                       1.898-3.el7                          base                        39 k
 perl-Email-Date-Format                            noarch                       1.002-15.el7                         epel                        17 k
 perl-Email-MIME-ContentType                       noarch                       1.017-1.el7                          epel                        19 k
 perl-Email-MIME-Encodings                         noarch                       1.315-1.el7                          epel                        18 k
 perl-Email-MessageID                              noarch                       1.404-1.el7                          epel                        18 k
 perl-Email-Simple                                 noarch                       2.203-1.el7                          epel                        33 k
 perl-MIME-Types                                   noarch                       1.38-2.el7                           epel                        38 k

I am worried that Foswiki is getting more and more difficult to get installed. We don't even ship the needed CPAN libs with Foswiki any longer. We have already seen several people in the IRC channel having trouble when installing on shared hosts where they need to beg the host provider to install CPAN libs. Now we have a proposal that adds 8 new CPAN dependencies just to enable sending emails.

Isn't there are an alternative way to resolve the issue of sending emails in UTF8?

-- KennethLavrsen - 05 Nov 2015

Kenneth, my bigger concern would be distributions where the packages are unavailable. I've checked Ubuntu back to 9.04, and Debian back to 6.0.5 and it's in the standard repos. I also built it with cpanm, and it builds without any xs modules. So we can easily add this to the CpanContrib for use by the hosted sites. It appears that this is a fairly simple requirement as cpan modules go.

The only issues with shipping CPAN in the CpanContrib are the modules that require xs versions. And Foswiki core would have the same issue. Unfortunately that includes version and some of the HTML parsing modules.

-- GeorgeClark - 05 Nov 2015

Kenneth, I would like to add to the above that UTF8 in not the only reason for the patch. That issue could be fixed with a single encode_utf8 call. Beside of that I wanted to clean up the code and make the overall mail handling more reliable.

As to additional dependencies – some times they're inevitable. As to my point of view it's no good to do a job someone has done already; yet it's almost 100% guaranteed that he's done it better that we could – simply because he spent more time on it.

-- VadimBelman - 05 Nov 2015

Unicode in the body text is one thing. The other is the Subject header: current impl can't deal with international chars in there. This should be fixed along the way, not sure the patch does that, Vadim?

-- MichaelDaum - 06 Nov 2015

Michael, it shall do it but I haven't checked this. Will try later today.

-- VadimBelman - 06 Nov 2015

Ok, Michael was totally right – Email::MIME doesn't handle headers encoding by default, need to be done manually. Sorry for missing that point.

The patch has been updated.

-- VadimBelman - 06 Nov 2015

Hm, can't really see the patch actually allows unicode in subjects, that is encoding it as '?utf-8?B?'.MIME::Base64::encode($subject).'?='= (RFC 1342)

Could you create a git branch Item13699 to implement this in a more comprehensible way? That would be cool. Thanks.

-- MichaelDaum - 08 Nov 2015

Vadim, I added you to the CoreDevelopers list so that you can create an Item branch on the distro repository. If you are uncomfortable doing that, you can fork the repository, create a branch in your own repo and then send in a pull request.

-- GeorgeClark - 08 Nov 2015

You should surely add all the dependencies to CpanContrib so a shared host user can install this from the browser and configure Foswiki to use it. Being pureperl CPAN libs is a good thing in this case.

But I still want to ask you all to always - and also here - consider carefully the price of adding more dependencies to make code "nicer" or "more cool" if the only win is for the developers. And seeing Michael's updates it seems the libs do not do everything anyway. If the statement "That issue could be fixed with a single encode_utf8 call" is really true - maybe that is the best solution even if it is ugly?

Note that I have not - and will not - raise a bloking ConcernRaisedBy on this. I just want you all to think one extra time and remember that each time we have added new dependencies we have had more people installing Foswiki in the IRC chat asking for help. And they are only the tip of the iceberg. Most just silently give up, curse at us, and move on to some other tool with a cool easy installer.

-- KennethLavrsen - 09 Nov 2015

Commenting definitely needs branches...

Michael,

Email::MIME does the encoding but it's a little bit obscure and is not automated. The chunk starting with '# MIME-reencode headers' comment and ending with loop containing header_str_set() does the job of encoding all email headers. Subject is not the only one where UTF8 could be met. There are To, CC, etc too.

Clark, thanks! I made a fork but since started working on the patch never managed to switch over to it. Yet I didn't like the fact that hooks work on the foswiki repository only.

-- VadimBelman - 09 Nov 2015

@Kenneth: this is not an issue of coolness or a nice-to-have. Current code is severely limitted. It actually can't send unicode emails. This can't be fixed with a call to encode_utf8 as things are different in email land. These Cpan libs deal with it and we should build on them as much as possible (instead of reinventing the wheel or something). Adding them to CpanContrib seems straight forward as they don't come with an XS part.

-- MichaelDaum - 09 Nov 2015

Kenneth, your point is clear to me. The subject is big, I'll reply separately.

Sure enough, the user base could be reduced slightly by making the installation more complex. But bugs do the same. The more we bury problems under hacks similar to encode_utf8 – the more we'll be fighting with them.

Just in few words: I hate the hacks! And as a cyrillic alphabet user I know exactly one thing: one would never be sure about what stupid software can do to a charset/encoding it doesn't support. Therefore anything beyond 1-127 range is at potential risk when off to the outer world. And if a user hits into a problem with mail delivery we'd better know that it's not because foswiki might have sent a malformed message.

This is not just about developers and nice code. The nice code usually means working code; hence it means happy users, as I see it. Does it worth a sysadmin spending a little bit more time on installing one more module? I think it does. Especially if we take this issue in our hands and teach the configure script about installing modules from CPAN. Did anybody ever thought about this?

Sorry about being a little bit too philosophical and muddled but I tried to make practical conclusions.

PS. Thanks for pointing me to the CpanContrib.

-- VadimBelman - 09 Nov 2015

On configure installing modules from CPAN.

No thank you. The servers I run are all Redhat (Centos) and everything is management by the rpm management system. To install rpms into the system perl tree you need root access.

And there is no way a typical Linux admin will install a parallel Perl and keep this manually up to date. Only perl developer geeks do that.

So to maintain a clean rpm based system that keeps it self auto updated by the yum jobs that run regularly I do not want my perl poluted by CPAN based installs. And I think we all agree that no job run in configure should run with root prevs.

The CpanContrib idea is to be a contrib you can install if you have touble installing CPAN libs. This is usually the case for people running their server in the cloud on some shared server. And this is getting more and more popular. Unfortunately these shared hosts are usually more focused on PHP, Java and the like. Perl is just the default perl that comes with the distro.

The only problem with CpanContrib is that we cannot put binaries in these. Only pure perl libs. In this case they are all pure perl so the problem is acceptable. Try to avoid binary CPAN libs. Just remember to get the libs for this feature added to CpanContrib.

-- KennethLavrsen - 10 Nov 2015

Foswiki does not have enough developers to implementing every needs directly in the Foswiki:: codebase. (Not to mention that the re-inventing of the wheel does not always mean an better wheel).

Using modules from some "reputable" and true perl monks is like as these programmers have worked directly on the Foswiki - that is, we get a lot of programmer's hours for "free" and (usually) better code. So, I am for the use of the CPAN modules.

Yes, the admin must do more installation work. Currently.

But, this can be fully eliminated by creating an Task:: type CPAN module, e.g. Task::Foswiki, or if you want differentiate major versions then Task::Foswiki::V1, Task::Foswiki::V2 and so on...

Examples of such Task:: modules already existing on CPAN:

• https://metacpan.org/pod/Task::WebGUI8
• https://metacpan.org/pod/Task::OTRS
• https://metacpan.org/pod/Task::Sympa
• https://metacpan.org/pod/Task::Latemp
• https://metacpan.org/pod/Task::Jenkins
• ... etc ...

such Task:: modules doesn't contains any code, and their sole purpose is installing all needed CPAN dependencies with one command - like cpanm Task::Foswiki.

Benefits of such Task:: type CPAN module:

• the user need install only one module - e.g.: Task::Foswiki (which will install the all needed CPAN dependencies)
• distros usually already have some guidelines about: How to package CPAN Modules - so, here is a bigger change (and easier way) to get the Task::Foswiki into the distros, as an pre-packaged modules - which means the user could install all CPAN-deps directly from their "cpanel" or such with one click...

So if some FW developer is enough skilled to create an CPAN module - the Task::Foswiki could be an right and beneficial step. (also, see: http://www.effectiveperlprogramming.com/2011/08/use-a-task-distribution-to-specify-groups-of-modules/ )

Ps: Partally offtopic comment, only as a reaction to Kenneth's "cpan concerns"...

-- JozefMojzis - 10 Nov 2015

Kenneth, I agree to Jozef that it's dubious what's worse: to reinvent the wheel or create some troubles to the admins. Your point about clean rpm base is understandable but I have so many objections to it:

• A server could be fully or partially dedicated to hosting Foswiki. And I'm not speaking of a hardware server – some kind of a virtual machine is much preferable. It solves root issue too because the VM could be fully under control of the person responsible of handling the web.
• Following the previous item I would prefer a cloud VM server in favor of a shared hosting. It's a somewhat more expensive but mostly worth it.
• Neither of the above is a viable solution? Modules could be installed in a local path without CpanContrib.

The last item is the most interesting here. By briefly inspecting the docs and the web I came up with the following proof of concept:

#!env perl

use warnings;
use strict;
use local::lib "$ENV{HOME}/tmp/perl5";
use CPAN;

CPAN::Shell->install('Perl::Version');

exit;

My point is that something like this shall be a part of the configure script. So, instead of complaining that a module is missing (after installing an extension, for example) configure would try installing it from CPAN first and only report a error when installation fails. With this we could get the burden of manual module installation off the shoulders of an admin! Yet, I no reason why it should

Unfortunately, this code still uses an external module local::lib. But it's very small, doesn't have dependencies and thus could be built directly into Foswiki infrastructure with no pain.

Yet the last option would be for those kind of shared hosting services where even that simple code won't work. This is the case where I'd reconsider the CpanContrib approach a little bit to transform it into a repository of pre-installed modules bundled with a records holding information about required files for each dependency and supported by the configure script too.

Thanks for pointing to all these problems. I hope to have my hand on this some day. Meanwhile I'm off to work with CpanContrib.

-- VadimBelman - 10 Nov 2015

We Need to know the minimum version of Email::MIME we can use. Currently the code relies on Email::MIME version 1.936. Please add to the following table, and indicate if you would be unable to update to a newer version.

-- GeorgeClark - 17 Nov 2015

George, I copy your comment to the Impact section as this is exactly where it belongs to in my opinion. It's also would be difficult to locate a table buried deeply in a pile of comments.

-- VadimBelman - 17 Nov 2015

Commenting is disabled while not logged in