Item2315: Net.pm appears to be unable to fetch Foswiki extensions if LWP not installed

See http://irclogs.foswiki.org/bin/irclogger_log/foswiki?date=2009-07-26,Sun&text=on

also http://irclogs.foswiki.org/bin/irclogger_log/foswiki?date=2009-11-01,Sun&sel=5#l1

It appears that if libwww-perl is not installed, the web installation fails to download the file, and then fails on the tar command when it is unable to open the missing file. No data could be read from file at /var/www/webs/foswiki/lib/Foswiki/Configure/UIs/EXTEND.pm line 346

-- GeorgeClark

no answers, but some pointers to when i came across this problem earlier:

• http://foswiki.org/Support/Question176 - describes when another user had the same problem and "fixed" it by installing CPAN:LWP
• http://irclogs.foswiki.org/bin/irclogger_log/foswiki?date=2009-06-02,Tue&sel=54-194#l51 - discussion that raw socket code no longer seems to work
• http://foswiki.org/Tasks/Item1678 - supposedly fixing raw socket code (released in 1.0.6)

what worries me is that otherwise perfectly good raw socket code just stopped working at some point---including plenty of examples all over the internet. i guess some version of perl broke backwards compatibility? or ??

-- WillNorris - 01 Nov 2009

Somebody added LWP as a core dependency in core/lib/DEPENDENCIES.

LWP,>=0,cpan,Optional, needed for the Configure Extensions installer, and for external URL based INCLUDEs

Looks like it was pre-Foswiki

-- PaulHarvey - 01 Nov 2009

If we can't resolve the raw socket issues, we should probably check for LWP and disable or present a warning with the Find Extensions function. A few people have run into this now. At least Net.pm should report that it was unable to fetch the file rather than failing with a TAR error.

-- GeorgeClark - 01 Nov 2009

I can't actually reproduce this locally, except by corrupting the LWP install. AFAICT the raw sockets code is working fine. I guess we need a more specific debug.

The core dependency is an option, BTW.

George, throwing it back to you for more analysis. Sorry.

-- CrawfordCurrie - 07 Nov 2009

I had not recreated this either. I opened this to document issues that folks were having on IRC - I think we now have 3 reports of problems in this area. So something should probably be done.

Instead of fixing the issue, maybe we can add code to Net.pm to detect/report that a fetch failed instead of failing on the misleading unable to read tar file. Error could include instructions to install libwww-perl, or possibly include further debugging information for us. I'll take a look at adding something to either better detect the failure or at least report that the file was not returned. Suggestions appreciated.

-- GeorgeClark - 07 Nov 2009

I've been able to recreate the problem. Running a wireshark trace while installing a simple plugin without libwww-perl installed, the package appears to be fetched correctly. The connection is opened, a GET issued, the file transmitted and the connection closed.

The problem is that the file is corrupted. The foswiki provided Net/HTTPRequest.pm is too aggressive in fixing newlines, and results in corrupting binary attachments and the uncompress fails.

-- GeorgeClark - 09 Nov 2009

I had wrongly advised George to use \r\n for clarity, and as it's not portable, I've fixed the code to use \015\012, just like LWP does.

As it's a fallback module, we can't really say we should use some more clever ways like some HTTP:: something module, but I would strongly advise on putting LWP as a core requirement. We have some features which depend on it, such as INCLUDE with http:// URL, and maybe some others. Or at least put a huge warning in configure when LWP is NOT present.

-- OlivierRaginel - 10 Nov 2009