Feature Proposal: Foswiki as a Standalone Server

Please migrate information from this page to the FoswikiStandAlone page to make it easier for developers to learn about the Foswiki Stand Alone architecture, without having to read through a historical view of the activity.

Motivation

Foswiki runs as CGI script by default, what leads to some problems like:

• Big overhead of loading, compiling and initializing for every request
• As many copies in RAM as there are simultaneous requests

Alternatives such as ModPerl or SpeedyCGI helps a lot with the first point, but their use is not simple. Also there is no native support to FastCGI, an interesting execution mechanism that is available on many hosting services, web servers and operating systems.

ModPerl helps with performance, but increase memory use: it keeps a perl interpreter loaded with each Apache's child and possibly all Foswiki modules, depending on how it was configured. Some problems of this setup are:

• Forking a child is expensive (using worker or event MPM's reduces the impact of this problem)
• Every apache child has a copy of Foswiki with it in memory, even if it's not serving a Foswiki request
• This way is not possible to distinguish Foswiki requests from static files requests
• Memory usage grows a lot with many simultaneous requests

Depending on the RAM available and the concurrency level of requests, the Operating System may be forced to use swap, what degrades performance a lot. It can be explored in DoS attacks. To avoid that, we need to put hard limits on the number of simultaneous requests the webserver can handle. But, as pointed, it's not possible to distinguish twiki requests from static file requests, so users can face very big latency times to load a complete page (with images, css, etc).

SpeedyCGI and FastCGI keeps Foswiki in memory, as ModPerl does, but separated from the webserver process, so it's possible to configure Foswiki independently. To achieve better performance results it's often needed to keep more than one backend process (again: many copies of Foswiki in RAM).

Also, it's not possible to use Foswiki without a web server, what turns complicated things like FoswikiOnMemoryStick.

Description and Documentation

What I wanted before start coding (Original Idea)

Performance raise when keeping Foswiki in memory is noticeable. A brief look at source code shows that many operations are common for all requests (Users loading, preferences parsing, etc) and the results could potentially be reused among requests.

If Foswiki could run as a Standalone Server it's performance would be comparable with achieved by ModPerl, SpeedyCGI or FastCGI, since the overhead of loading and compiling would be eliminated and it would also be possible to avoid performing many initialization operations repeatedly. Memory usage would be improved as well cause it would be possible to share memory used to store preferences (and whatever is common) between requests. That would be necessary some mechanism to forward requests from webserver to Foswiki and to receive responses.

Besides, Foswiki could run in some jail/chroot environment, reducing the impact of possible security problems.

What I Achieved

When I realized how flexible and easy to use Catalyst framework is, I decided that Foswiki should be like that. So I started to modify Foswiki core to be independent of execution mode: created all stuff needed, updated all necessary core code. Then I wrote some engine mechanisms, and it's done!

Now it's possible to run Foswiki as CGI, under FastCGI, ModPerl and as an HTTP standalone server. And it's very easy! ( at least I think it is )

No dependencies were added to run Foswiki as CGI and no configuration changes are needed.

Many hosting services provide FastCGI or ModPerl support or even let users run their own processes. There are many Codev topics with hacks to use these features. Most of them require source code changes, are not updated (or there is no feedback of successful use with newer versions) and can break Foswiki somehow (I had a very bad experience with edit + SpeedyCGI...). With this new mechanism it's possible to fully take advantage of what a hosting service provides without fears.

FastCGI and HTTP engines lets administrators to control how many copies to run simultaneously, making it easier to balance performance and memory usage, independent of webserver limits.

Current Status

There are engines to support CGI, CLI, FastCGI, ModPerl, and a HTTP stand alone.

CGI implements the default Foswiki behavior. CLI engine is automatic loaded when Foswiki is used from command line.

FastCGI pre-compile and loads how many copies of Foswiki as specified and forwards requests from webserver to them. This leads to a sensible performance increase, that is about 140% for simple topics, like Main.WebHome (on a fresh install, using pattern skin).

HTTP engine turns Foswiki into a HTTP server itself, so it's possible to use it without a "real" webserver. It's very useful for Foswiki development and also for FoswikiOnMemoryStick and FoswikiForWindowsPersonal (I didn't test it on windows, neither I tried to install in a memory stick, but I think HTTP engine make theses things much easier). It also has the same benefits FastCGI has and very similar performance (actually they are based on the same idea: keep things in memory, and differ in the protocol used to communicate).

ModPerl engine makes use of Foswiki with mod_perl very easy, and actually doesn't suffer from historical problems with global variables (thanks to pos Foswiki-4 code). It's a lot faster than CGI, but not as fast as FastCGI or HTTP. Each apache child (or thread, if you're using some multithreaded mpm) has it's own copy of Foswiki, so be careful with MaxClients config option to avoid sending webserver to swap. If you can, use FastCGI or HTTP engines instead (see observations above)

In summary:

Engine	Memory Usage	Speed	Support	CPAN Dependencies
CGI	As many copies of Foswiki as there are simultaneous requests.	Very Slow	Very good. Almost all hosting services and webservers support CGI	None
ModPerl	As many copies of Foswiki as there are apache children (or threads)	Fast	It only works with Apache webserver	HTTP::Body
FastCGI	Reasonably constant for each backend	Very Fast	Good. Many hosting services provide support for it	FCGI and maybe FCGI::ProcManager
HTTP	Reasonably constant for each instance	Very Fast	It probably runs on many operating systems and can be used if hosting service let users run their own processes	HTTP::Headers, HTTP::Request, HTTP::Response, LWP::MediaTypes, HTTP::Daemon (all those are part of libwww-perl), HTTP::Body and maybe File::Monitor, File::Spec and File::Find

Shortcut: Go directly to Next Steps or Try it out if you're not interested on technical details.

Impact

WhatDoesItAffect:

Install, Performance, Refactoring, Security %WHATDOESITAFFECT%

edit

Implementation Details

An engine framework, similar to catalyst's, is suggested. There is a Foswiki::Request object, that gets filled by engine's methods. Then Foswiki fills a Foswiki::Response object, instead of writing answer to STDOUT directly. This object is passed to engine, that finally sends answer to client.

In this scenario CGI is simply one more engine, that uses %ENV and a CGI query object to populate Foswiki::Request and that takes Foswiki::Response to print its info to STDOUT.

Dynamic is as follows:

Engine's corresponding script gets executed, adjusts $Foswiki::cfg{Engine} and loads Foswiki

$Foswiki::engine->run. The engine performs its specific setup tasks and waits for connections

$Foswiki::engine->prepare fills a Foswiki::Request object

prepareConnection fills remoteAddr, method and secure request fields

prepareQueryParameters fills request's queryParam field with data passed within query string, if any

prepareHeaders fills request object with connection headers available/pertinent and remoteUser field

prepareCookies fills cookies field with data available

preparePath fills pathInfo and uri fields

prepareBody performs any task related to process request body, if present

prepareBodyParameters fills request object with parameters passed within request body

prepareUploads fill request object with info about uploaded files, if any. Info is encapsulated in Foswiki::Request::Upload objects

Foswiki::UI::handleRequest: similar to old Foswiki::UI::run. Gives Foswiki::Request to Foswiki and gets a Foswiki::Response

$Foswiki::engine->finalize: Sends to client data within Foswiki::Response object and performs cleanup tasks

finalizeUploads deletes any temporary files created in prepare phase.

finalizeHeaders sends response headers to client

finalizeBody sends response body

Both Foswiki::Request and Foswiki::Response are reasonably compatible with CGI, but refer to documentation in code.

No dependencies were added to run Foswiki with CGI engine. I won't promise that for other's engines (Actually IO::Handle is used, but since it's part of core perl distribution I think that's ok)

Changes

There were a lot of code changes and there must be a few about programming style. Main code changes were:

• Created Foswiki::Request, Foswiki::Request::Upload, Foswiki::Response, Foswiki::Engine::.* and Foswiki::LoginManager::Session classes
• Modified Foswiki::UI, based on bin/twiki and catalyst framework.
• Updated Foswiki::UI::View and Foswiki::UI::Statistics to work well with engine mechanism
• Created Foswiki::UI::Rest class, adapted from bin/rest.
• Eliminated almost all direct accesses to %ENV
  • $ENV{SCRIPT_NAME} replaced by $request->action(). They are not strictly the same, but as far as I could notice Foswiki needs only $request->action()
• Replaced print calls with Foswiki::Response calls
• cgiQuery field of Foswiki class renamed to request (and it's a Foswiki::Request object instead of a CGI)
• Added response field to Foswiki class
• raw_cookie() not implemented and replaced by $request->headers('Cookie') in Foswiki::LoginTemplate
• Foswiki::generateHTTPHeaders() fills $twiki->{response} object instead of returning HTTP header string
• Added $Foswiki::cfg{BehindProxy} config option
• Unit tests:
  • new CGI replaced by new Foswiki::Request
  • $twiki->{cgiQuery} replaced by $twiki->{request}
  • $query->path_info("/Web/Topic") replaced by $query->path_info("/script/Web/Topic")
  • Added $Foswiki::cfg{Engine} = 'Foswiki::Engine::CGI' to BEGIN block of test/bin/testRunner.pl
  • Some tests needed some little specific updates

How to Configure

Configuration examples assumes Foswiki ins installed at /var/www/twiki. You have to configure an alias to Foswiki pub if you have installed on another directory.

CGI Engine

It replaces traditional execution mode.

Apache

Alias /twiki/bin /var/www/twiki/bin

<Directory "/var/www/twiki/bin">
    Options +ExecCGI +FollowSymLinks
    SetHandler cgi-script
</Directory>

Lighttpd

    alias.url += ( "/twiki/bin" => "/var/www/twiki/bin" )
    cgi.assign = ( "" => "" )

And don't forget to load mod_cgi.

FastCGI Engine

It depends on FCGI CPAN perl module and webserver must support FastCGI. I used libapache2-mod-fastcgi and libapache2-mod-fcgid from debian to test.

Apache + mod_fastcgi

There are two ways to use FastCGI engine with mod_fastcgi. First one is to let twiki_fastcgi.pl to control servers and second to let apache.

First mode

This mode requires also FCGI::ProcManager CPAN perl module.

$ cd /var/www/twiki/bin
$ ./twiki_fastcgi.pl --listen /var/run/twiki/fastcgi.socket --daemon --nproc 1

• --daemon is optional.
• Adjust --nproc to satisfy your needs

And within config files:

FastCgiExternalServer /var/www/twiki/bin/twiki_fastcgi.pl -socket '/var/run/twiki/fastcgi.socket'
Alias /twiki/bin/configure /var/www/twiki/bin/configure
Alias /twiki/bin /var/www/twiki/bin/twiki_fastcgi.pl

<Directory "/var/www/twiki/bin">
    Options +ExecCGI
    SetHandler cgi-script
</Directory>

Second mode

FastCgiServer /var/www/twiki/bin/twiki_fastcgi.pl -processes 1
Alias /twiki/bin/configure /var/www/twiki/bin/configure
Alias /twiki/bin /var/www/twiki/bin/twiki_fastcgi.pl

<Directory "/var/www/twiki/bin">
    Options +ExecCGI
    SetHandler cgi-script
</Directory>

Again: adjust -processes to satisfy your needs

Apache + mod_fcgid

Alias /twiki/bin/configure /var/www/twiki/bin/configure
Alias /twiki/bin /var/www/twiki/bin/twiki_fastcgi.pl
DefaultMinClassProcessCount 1
DefaultMaxClassProcessCount 1

<Directory "/var/www/twiki/bin">
    Options +ExecCGI
    SetHandler fcgid-script
    <Files "configure">
        SetHandler cgi-script
    </Files>
</Directory>

Adjust DefaultMinClassProcessCount and DefaultMaxClassProcessCount to satisfy your needs. These parameters are global: apply to all FastCGI applications.

Lighttpd

$HTTP["url"] =~ "^/twiki/bin/configure" {
    alias.url += ( "/twiki/bin" => "/var/www/twiki/bin" )
    cgi.assign = ( "" => "" )
}
else $HTTP["url"] =~ "^/twiki/bin/" {
    alias.url += ( "/twiki/bin" => "/var/www/twiki/bin/twiki_fastcgi.pl" )
    fastcgi.server = ( "/twiki_fastcgi.pl" =>
        (
            (   "bin-path"  => "/var/www/twiki/bin/twiki_fastcgi.pl",
                "max-procs" => 1,
                "socket"    => "/var/www/twiki/working/tmp/twiki.socket"
            )
        )
    )
}

And don't forget to load mod_cgi and mod_fastcgi.

ModPerl Engine

There is support for versions 1.24+, 1.9 and 2.0+.

It depends on HTTP::Body CPAN perl module.

In order to use it, create a file named mod_perl_startup.pl in tools directory with the following contents: ( Adjust your paths )

$ENV{MOD_PERL} =~ /mod_perl/ or die "mod_perl_startup called, but mod_perl not used!";
use lib qw( /var/www/twiki/lib );
require '/var/www/twiki/bin/setlib.cfg';
1;

It's also cool to load most Foswiki modules, so that some memory is saved.

Apache 1.x + ModPerl 1.24+

Alias /twiki/bin/configure /var/www/twiki/bin/configure
PerlTaintCheck On
PerlRequire /var/www/twiki/tools/mod_perl_startup.pl
PerlModule Foswiki::Engine::Apache
<Location "/twiki/bin">
   SetHandler perl-script
   PerlHandler Foswiki::Engine::Apache
</Location>
<Location "/twiki/bin/configure">
    SetHandler cgi-script
    Options +ExecCGI
</Location>

Apache 2.x + ModPerl 1.9+

Alias /twiki/bin/configure /var/www/twiki/bin/configure
PerlSwitches -T
PerlRequire /var/www/twiki/tools/mod_perl_startup.pl
PerlModule Foswiki::Engine::Apache
<Location "/twiki/bin">
   SetHandler perl-script
   PerlResponseHandler Foswiki::Engine::Apache
</Location>
<Location "/twiki/bin/configure">
    SetHandler cgi-script
    Options +ExecCGI
</Location>

HTTP Engine

I think this one is the most interesting

It depends on HTTP::Headers, HTTP::Request, HTTP::Response, HTTP::Daemon and HTTP::Body CPAN perl modules. If you want to use the auto-restart feature, then you also need File::Monitor, File::Spec and File::Find

Run:

$ cd /var/www/twiki/bin
$ ../tools/twiki_http.pl --host localhost --port 3000 --detach --restart --rdelay 1

And enjoy: http://localhost:3000/bin/view/Main/WebHome

• --binurl and --puburl defaults to $Foswiki::cfg{ScriptUrlPath} and $Foswiki::cfg{PubUrlPath}, so it's safe to omit them. (Remember to run configure script to adjust Path Settings)
• --host defaults to all (listen on all interfaces available)
• --port defaults to 3000
• --detach may not work on windows.
• --restart re-executes itself if any file that is depended on changes, or if files are created or deleted under lib/ (feature specially designed for developers)
• --rdelay delay between check for changes (default: 1 second)
• --help show usage information and exit

There is primitive support for static files, mainly added to help twiki development. In production, it's really recommended to run HTTP stand alone server behind a proxy. Some reasons are:

• Proxy webserver handles static files more efficiently
• Proxy webserver can be configured to force mime types of some dangerous files (.php, .html, ...) to text/plain
• Many instances of twiki can be executed, with proxy webserver acting as a load balancer

If, and only if, you are running behind a proxy, check {BehindProxy} option within configure script (Expert mode => Security Setup => Miscellaneous). If you run http stand alone behind a proxy and don't check {BehindProxy}, then %ENV{"REMOTE_ADDR"}% will always return proxy's IP address and {Sessions}{UseIPMatching} option would be useless. On the other hand, if you check {BehindProxy} without a proxy, then an attacker could easily perform an IP spoofing attack by sending a fake X-Forwarded-For HTTP header. So, adjust {BehindProxy} according to your setup (unless you are a Foswiki developer, there is no reason to not use a proxy).

The following configuration assumes you are running two twiki_http.pl instances, on ports 3000 and 3001.

Apache

Alias /twiki/bin/configure /var/www/twiki/bin/configure

<Directory "/var/www/twiki/bin">
    Options +ExecCGI
    SetHandler cgi-script
</Directory>

ProxyRequests Off
ProxyPreserveHost On
ProxyPass /twiki/bin/configure !
ProxyPass /twiki/bin balancer://twiki

<Proxy balancer://twiki>
    BalancerMember  http://127.0.0.1:3000/twiki/bin
    BalancerMember  http://127.0.0.1:3001/twiki/bin
</Proxy>

If you are running only one twiki_http.pl instance then you can do something like:

Alias /twiki/bin/configure /var/www/twiki/bin/configure

<Directory "/var/www/twiki/bin">
    Options +ExecCGI
    SetHandler cgi-script
</Directory>

ProxyRequests Off
ProxyPreserveHost On
ProxyPass /twiki/bin/configure !
ProxyPass /twiki/bin http://127.0.0.1:3000/twiki/bin

Don't forget to load mod_proxy, mod_proxy_http and mod_proxy_balancer (this one can be ignored if the last example is used).

Lighttpd

proxy.balance = "fair"

$HTTP["url"] =~ "^/twiki/bin/configure" {
    alias.url += ( "/twiki/bin" => "/var/www/twiki/bin" )
    cgi.assign = ( "" => "" )
}
else $HTTP["url"] =~ "^/twiki/bin/" {
    proxy.server = (
       "/twiki/bin" => ( ( "host" => "127.0.0.1", "port" => 3000 ),
                         ( "host" => "127.0.0.1", "port" => 3001 ) )
    )
}

Don't forget to load mod_cgi and mod_proxy.

Refer to mod_proxy documentation. Adjust proxy.balance for your needs.

Other configuration tips

{StoreImpl} and {RCS}{SearchAlgoritm}

If you're using some persistent engine (FastCGI, ModPerl or HTTP stand alone), it's better to set {StoreImpl} to RcsLite and {RCS}{SearchAlgorithm} to Foswiki::Store::SearchAlgoritm::PurePerl or Foswiki::Store::SearchAlgorithm::Native. The last one is a distributed as a Contrib: NativeSearchContrib.

ApacheLogin

In order to use ApacheLogin as login manager there are some extra configurations needed.

Note:: Please, consider switching to TemplateLogin.

Apache

<LocationMatch "/twiki/bin/(attach|edit|manage|rename|save|upload|mail|logon|.*auth).*">
   AuthUserFile /var/www/twiki/data/.htpasswd
   AuthName 'Enter your WikiName: (First name and last name, no space, no dots, capitalized, e.g. JohnSmith). Cancel to register if you do not have one.'
   AuthType Basic
   require valid-user
</LocationMatch>

Lighttpd

Refer to observation at FoswikiOnLighttpd#User_Authentication.

$HTTP["url"] =~ "^/twiki/bin/(attach|edit|manage|rename|save|upload|mail|logon|.*auth).*" {
    auth.backend = "htpasswd"
    auth.backend.htpasswd.userfile = "/var/www/twiki/data/.htpasswd"
    auth.require += ( "" => (   "method" => "basic",
                                "realm"  => "Enter your WikiName: (First name and last name, no space, no dots, capitalized, e.g. JohnSmith). Cancel to register if you do not have one.",
                                "require" => "valid-user"
                            )
    )
}

Don't forget to load mod_auth.

Known Issues

• configure script doesn't work under engine mechanism (but it can be configured as default CGI and it's good to do that way, since it is a very special case)
• OopsExceptions leads to Access Denied error with HTTP engine, cause http://localhost:3000 doesn't match $Foswiki::cfg{DefaultUrlHost}. It's not really a bug, but I think it's cool to register this. It doesn't show up when running behind a proxy. If you want to use it without a proxy (to develop, for example), then configure $Foswiki::cfg{DefaultUrlHost} to the correct url
• HTTP engine doesn't support yet https ( Really needed? Probably not, since it's recommended to run behind a proxy. Then configure https on proxy)

Legend:

	Solved
	Work in progress
	No action needed

Pending tasks

As soon as I can I'll work on the following tasks:

• Write unit test cases (and evaluate them with Devel::Cover)
• Request Dependet/Independent tags: What is common between requests? And what is specific?
  • Memory usage analysis
  • Singletons rethink (Potentially they will be split between request and execution singletons)
• To implement classes that take advantage of persistent execution (Put preferences in a DBM and loads it in a shared memory block, so reducing processing time and memory usage. Maybe something similar to users, forms...)
• Plugins port guide, possibly suggestions of hooks
• Analise performance
• ... ( suggestions welcome! )

Try it out!

Note: Use it for test purposes only. All existing unit tests are OK and as far as I could test (not too far...) things work fine, but probably there are some bugs around. If you find any, please report them at Tasks:Item4662 (add bug description as a comment).

Version	Description	Download
First release, from svn rev 15428	Independence from execution environment, support for CGI, CLI, FastCGI, ModPerl and HTTP stand alone	https://foswiki.org/pub/Development/FoswikiStandAloneFeature/FoswikiStandAlone-4.3.0-auto15428.tgz

Refer to FoswikiInstallationGuide for general tips and to above configuration examples, run the configure script and have fun

: If you're using some persistent engine (FastCGI, ModPerl or HTTP without --restart), you need to restart webserver (or HTTP instances) if you update configuration.

If you want to run unit tests, refer to UnitTests

-- Contributors: GilmarSantosJr

Discussion

Actually I'm already working on this as my Undergraduation Final Project. Currently, I'm analyzing what needs to be changed, what is common for all requests and how could it be possible to handle more than one request at a time, sharing common objects. The next step is to implement a mechanism similar to Apache's MPM, define how Foswiki and webserver would communicate and finally implement, test and compare multithread and multiprocess with shared memory approaches.

Maybe multithread is not a good idea, since Perl Threads doesn't share data between threads by default, but I think it's worth a try.

My goal is to achieve similar performance results to ModPerl, SpeedyCGI and FastCGI, but without the problem of excessive memory usage.

-- GilmarSantosJr - 21 Aug 2007

Which CPAN web server / web app are you thinking of using as the base?

-- SvenDowideit - 22 Aug 2007

I'm taking a look at Catalyst::Engine::* to figure out how it works. I want something as flexible, extensible and easy as that!

-- GilmarSantosJr - 23 Aug 2007

I've just created a branch to start playing

-- GilmarSantosJr - 23 Aug 2007

First commited version show blank pages with CGI or FastCGI engines. Corrected and commited.

-- GilmarSantosJr - 16 Sep 2007

Just to let you know it is great you are pursuing this!

-- ArthurClemens - 18 Sep 2007

I would like to point a very nice possibility this opens to us: a "real" webserver is not needed anymore to develop Foswiki. We just need to do a checkout, run the HTTP standalone engine and browse to e.g. http://localhost:3000/.

-- AntonioTerceiro - 19 Sep 2007

Thanks, Arthur!

I've just finished the "restart" feature on HTTP engine: it's no longer necessary to restart twiki_http.pl when a change is made to any file. It helps development a lot

-- GilmarSantosJr - 15 Oct 2007

For the non-techies among us: What would be a typical use case for this proposal?

Could someone explain in "real words" what this is all about? Thanks, I'm quiet shure this is useful

-- CarloSchulz - 19 Oct 2007

Hi Carlo,

This proposal increases twiki's execution mode flexibility. So it would be easier to run twiki in many hosting services (If the service provides FastCGI, ModPerl or lets users to run their own processes) and take more advantage.

Execution under FastCGI or as HTTP stand alone raises performance a lot, and it's possible to better control memory usage. It also permits some more code improvements to reduce resource usage even more (Less resource usage => cheaper hosting services => more twiki installations) .

HTTP stand alone could be run under a chroot jail, reducing impact of possible security holes.

HTTP stand alone (or some other persistent engine [but I think HTTP stand alone is the "best"]) together with DatabaseStore can be used to build very scalabe Foswiki-based sites using more than one machine to load balance.

There are some more advantages/usages for developers too.

-- GilmarSantosJr - 19 Oct 2007

Thanks for claryfiing

-- CarloSchulz - 22 Oct 2007

This looks very useful indeed - would be great to make this really easy to install (i.e. a FoswikiFor distribution, as in FoswikiForWindowsPersonal) so you can simply unzip the FoswikiStandalone distribution into a directory, start the HTTP server, and go.

Performance and WYSIWYG are the last two hurdles to Foswiki adoption in my view. and this helps enormously with performance. The FastCGI support is really interesting for web hosting, where Foswiki is quite popular, and for any Linux distros that already support FastCGI in their Apache versions.

-- RichardDonkin - 30 Oct 2007

Just wanting to let you know that I am still interested in this:

Warning: Can't find topic Blog.2007-10-26-twiki-stand-alone-projectComments

-- MichaelDaum - 14 Nov 2007

I've been using MediaWiki quite a lot recently, and it really is blindingly fast - they do a lot of caching, and of course PHP in most web servers runs as a module. FoswikiStandalone is important to catching up with this, as are both FastCGI and SpeedyCGI support.

-- RichardDonkin - 14 Nov 2007

I will have to have a play with this - I have FoswikiOnUbuntu installed via the twiki package for Ubuntu, but its speed on my old PCs is quite leisurely. FastCGI seems the way to go for me. Also, it would be great to have this up on Dreamhost (see DreamhostSetupNotes), who do support FastCGI.

-- RichardDonkin - 22 Nov 2007

Anyone get fastcgi working on Sun Web server?

-- DwayneLee - 04 Dec 2007

This sounds very interesting, unpack and run directly without apache or other web server? I tried to download but could not make it work. Am I doing it wrong or what is missing? I'm running Ubuntu 7.10.

• download the tgz from this page (15428)
• install cpan modules, HTTP::Headers..etc as specified above
• unpack to /var/www/twiki
• cd to /var/www/twiki/bin
• run ../tools/twiki_http.pl arguments...as specified above

I get something like:

Content-type: text/plain
Perl error when reading LocalSite.cfg: 
Please inform the site admin.
BEGIN failed--compilation aborted at /var/www/twiki/lib/Foswiki.pm line 484.
Compilation failed in require at ../tools/twiki_http.pl line 53.
BEGIN failed--compilation aborted at ../tools/twiki_http.pl line 53.

-- LarsEik - 14 Feb 2008

I haven't tried it yet, but perhaps you need to create a LocalSite.cfg in lib/ ? I just tried to "echo "1;" > ../lib/LocalSite.cfg" and it eliminates the error you get.

-- KoenMartens - 15 Feb 2008

Thanks Koen, after copying LocalSite.cfg from a debian server with the same paths it started and greets me with http://localhost/twiki/bin/view/Main/WebHome when I use port 80. but I cannot get configure it just goes "Not found". The configure script is there with same props as view. Docs say it dont work default but can be configured?

-- LarsEik - 15 Feb 2008

Hi Lars,

configure script is very special, since it must work even if Foswiki is not configured yet, so it doesn't make sense to get it working under this mechanism. The solution is to configure Foswiki manually (copy lib/Foswiki.spec to lib/LocalSite.cfg and change the desired settings) or to set up apache and run configure script as a CGI.

Thanks for the feedback. I'll work on a better out-of-the-box experience

-- GilmarSantosJr - 17 Feb 2008

Are there plans to incorporate this work into the upstream release?

-- MichaelDaum - 17 Feb 2008

I hope so and I'm working with that in mind

-- GilmarSantosJr - 17 Feb 2008

So basically one could copy a LocalSite.cfg from a server with the same paths. I also copied plugins but suddenly I copied too much.... It's really great and I look forward to test updated version, hope this is something that will be standard in Foswiki.

I hope the core developers are following this one.

-- LarsEik - 17 Feb 2008

GilmarSantosJr, please file a new proposal so that your work can enter the roadmap for Foswiki-5.x

Great work! Go ahead!

-- MichaelDaum - 18 Feb 2008

Thanks, Michael!

Actually this topic is a feature request, proposed for GeorgetownRelease (I think this one will be 5.x, wont it?) and UnderInvestigation. According to diagram at CurrentState we need to reach consensus or decide about it on some release meeting to get this accepted and then figure out how to merge with trunk.

I'll sync TSA scratch branch with trunk, update docs and wait for more feedback (maybe we got a consensus ), or the next release meeting.

-- GilmarSantosJr - 18 Feb 2008

This is all kinds of awesome! Finally Foswiki can run as a different user than Apache and it's fast to boot! I hope it gets merged soon... Can you outline what the big changes were that you needed to do for this?

-- WoutMertens - 27 Feb 2008

I cannot see this one listed under FeatureProposals, am I missing it? Maybe it is just because Legacy.DateOfCommitment is missing?? I just afraid it could be missed on the agenda since it's not listed with the other feature requests.

-- LarsEik - 27 Feb 2008

lets find out I've made Gilmar the commited developer - hope thats true

-- SvenDowideit - 28 Feb 2008

Yeah, that's true

Actually I was waiting to finish some architecture enhancements and update docs (especially for developers). I'm working on this right now, so I think it's OK to have 2008-02-28 as DateOfCommitment

-- GilmarSantosJr - 28 Feb 2008

I've just finished enhancements and docs update that I mentioned previously. Now Foswiki::Engine and Foswiki::UI are more independent from each other, the former is more simple and, besides to the lack of additional dependencies, no configuration change is needed to update from the traditional method (cgi script) to CGI engine.

Thanks Wout! Take a look at implementation details section

-- GilmarSantosJr - 28 Feb 2008

Date of Commitment is the date that you as a developer commit to be the driver and implementer of the feature proposed and it is what makes the 14 day clock tick.

The community now has 14 days to accept or reject or raise concern.

I have a question. You have now checked in a lot of code in a unique branch. How to you suggest merging this into the trunk?

-- KennethLavrsen - 28 Feb 2008

I could keep using SVK the same way I use it to get changes from trunk. But I think it's better to merge incrementally: one commit for each logical change (there could be one bug corresponding to each one). The big picture is:

• Add Foswiki::Request, Foswiki::Response and Foswiki::LoginManager::Session (abstraction layer created)
• Add Foswiki::Engine*, tools/twiki_http.pl, bin/twiki_cgi.pl, bin/twiki_fastcgi.pl and update lib/Foswiki.spec
• Merge changes to Foswiki.pm
  • Add response field
  • Rename cgiQuery field to request (and update it everywhere it's used within Foswiki.pm as well other classes)
  • Update some methods: UTF82SiteCharset, generateHTTPHeaders, etc
  • Eliminate direct %ENV accesses
• Update Foswiki::UI::* as needed (replace print 's and %ENV reads)
• Update Foswiki::Func
• Update Foswiki::LoginManager
• Update Foswiki::Sandbox
• Update lib/MANIFEST
• Update UnitTestContrib and default plugins unit tests (refer to implementation details above)
• Merge above documentation about configuration to official docs

Some of these changes also fix minor bugs that apply to 4.2 (I didn't have the time to describe them at bugs web, but I will)

After these first steps trunk will have engine mechanism merged, but I'd like to discuss about some changes to make architecture simpler:

• Split Foswiki class into Foswiki::Util, Foswiki::DefaultHandlers and Foswiki (much small and simple)
• Let Foswiki::generateHTTPHeaders responsibility with engine mechanism
• Rethink about classes relationships and singleton design (the idea is to avoid performing common-to-all-requests tasks when persistent engines are used [and it must work for non-persistents too])
• Add plugin hooks to give them the chance to take advantage of persistent execution (something like a globalInit and globalFinish)
• We will need to hunt memory leak sources (Yeah, they still exist... ), or add options/doc to restart persistent engines regularly.

Also I have some more enhancements in mind (these ones will direct affect performance and resource consumption):

• Implement a prefs mechanism that uses less RAM, that can be shared among many processes and that doesn't need to parse topics for every request (If what I'm thinking about works, then it would be a drop-in replacement for the actual Foswiki::Prefs, but would have some more dependencies, so it could exist a cfg{PrefsMechanism} to let users choose the more indicated for their needs).
• Try to make something similar to Forms and Users...

-- GilmarSantosJr - 28 Feb 2008

I am absolutely happy to see someone new that wants to address the core code.

Naturally we will need to create proposal topics for each of your ideas beyond the stand alone changes but it is encouraging to see new ideas related to the core.

Gilmar have you reviewed the Foswiki:Codev.FoswikiRoadMap? Do you agree with it. DO you want to add new things (anyone is encouraged to maintain and evolve the roadmap)?

-- KennethLavrsen - 03 Mar 2008

I'm monitoring not only FoswikiRoadMap, but also FoswikiRoadMapDiscussionFiveDotZero, DatabaseStore and TopicObjectModel. I'm happy to see that many ideas I had are already in progress!

I'll mature the ideas beyond changes already done and file new feature/refactoring proposals as soon as I fell I'm ready.

-- GilmarSantosJr - 11 Mar 2008

I am having some troubles:

1. The version in svn stops at a symlink bin/oops - it just does not update svn any further
2. I can't get the downloadable version to work when following the instructions in revision 64. There is no ScriptAlias, just 2 times Alias, and that does not make the bin folder executable. Could you add a complete httpd.conf example?

-- ArthurClemens - 12 Mar 2008

I don't know what caused the problem with svn, but I removed all links and added them again, it's ok now.

Note that there is an Alias to the configure script, and other to the twiki_cgi.pl script (instead of the bin directory. This was changed in svn, and a simple ScriptAlias is enough, like the standard way to run as CGI)

Alias /twiki/bin/configure /var/www/twiki/bin/configure
Alias /twiki/bin /var/www/twiki/bin/twiki_cgi.pl  # <--- Note this line makes "/twiki/bin" path point to twiki_cgi.pl script!!

There are also some options to the /var/www/twiki/bin directory, that make it executable:

<Directory "/var/www/twiki/bin">
    Options +ExecCGI
    SetHandler cgi-script
</Directory>

It works well with Apache 2.2.8-1 from Debian (I have this version now, but I remember it always worked at least with 2.x)

This way you'll run Foswiki as a CGI script. Try to use other engines as well (other engines may have CPAN dependencies. All of them are supposed to work!)

-- GilmarSantosJr - 13 Mar 2008

We have passed the 14 day timeout and I have only seen positive responses to I am happy to flip this into an accepted proposal.

-- KennethLavrsen - 15 Mar 2008

• Can you please add a logging facility in the httpd mode? It took me a while to figure out how to fix it just giving me 404 errors
• In Engine/HTTP.pm, please change all references to binUrl to scriptUrl. I couldn't make it work otherwise.
• For some reason, I had a lot of references to $cfg{} in my LocalSite.cfg file. I had to change all of them to $Foswiki::cfg{} before the standalone would work.
• Looks like the standalone httpd doesn't support ShorterUrlCookbook . I'm investigating if I can make the apache proxy do all the legwork.

-- WoutMertens - 22 Apr 2008

Ok got it working - wow that's fast, I love it! TagMePlugin is failing though, at this code:

    my $query = $Foswiki::Plugins::SESSION->{cgiQuery};
    my $tagMode = $query->param('tagmode') || '';

The query object in there is undef which makes the param() call fail. Does it not get filled in?

I also discovered that it won't handle subwebs exactly like plain Foswiki: if you go to twikihost/Web/Subweb it will say it can't find the topic "Subweb" while plain Foswiki will instead show twikihost/Web/Subweb/WebHome.

-- WoutMertens - 22 Apr 2008

Hi Wout, thanks for your feedback!

I'm pretty busy now (and that's why it's not merged yet), but I have many enhancements to HTTP engine in mind. Some kind of logging is one of them

binUrl is set to scriptUrl unless you specify them at command line, so it should work. Can you give more details about your configuration and problem? (Please, post the details at Tasks:Item4662)

ShorterUrlCookbook is something that I'll review latter. Probably, there must be explicit support for it at HTTP engine.

The problem with TagMePlugin is that it doesn't use plugins API, so it was expected to break sometime. Please, file a bug report to TagMePlugin: it should use Foswiki::Func::getCgiQuery instead of taking it directly from core.

I'll look at subwebs issue carefully and fix it if something is broken.

-- GilmarSantosJr - 24 Apr 2008

Actually I'm not sure if the binUrl/scriptUrl thing is a bug. I hadn't properly configured the twiki install because I thought copying localsite.cfg was enough; however my current install is 4.0.5 and that one had a twiki.cfg with defaults. Your tarball doesn't and therefore it was missing the default value for binUrl. Running configure fixed it.

Aside from TagMePlugin failing: If I "fix" the $query problem by removing the call and use PatternSkin, server response is insanely slow. Setting skin=plain works fine. When trying to run perl profiling on the code, I noticed that for loading 1 page, Foswiki::Users::FoswikiUserMapping::isGroup gets called 55000 times! Skin=plain has normal behaviour. Does PatternSkin work for you? Should I use the svn version instead of the tarball? Should I wait until you merged?

-- WoutMertens - 24 Apr 2008

I just fixed TagMePlugin and I didn't notice any problem with PatternSkin. I made many enhancements to svn version since I released the tarball, so I recommend to use the svn version. I'll add support to run configure script from HTTP engine before merge, cause it's a important feature. I'll come back to work on this as soon as I have the time for it.

I also tested the subweb issue with both FoswikiStandAloneFeature and trunk latest svn revision and they have the same behavior: I created a Main/Test web and requested /view/Main/Test and /view/Main/Test/. I got a "this topic doesn't exist yet" for the first from by both TSA and trunk, and I got Main.Test.WebHome for the second request. By looking at Foswiki.pm:1348, I see:

    if( $pathInfo =~ /\/((?:.*[\.\/])+)(.*)/ ) {
        # is 'bin/script/Webname/SomeTopic' or 'bin/script/Webname/'
        $web   = $1 unless $web;
        $topic = $2 unless $topic;
        $web =~ s/\./\//go;
        $web =~ s/\/$//o;

This regex always interpret Main/Test as $web = "Main" and $topic = "Test". Maybe it should verify if hierarchical webs are enabled and if the last name exists as a sub web...

-- GilmarSantosJr - 27 Apr 2008

Argh, in 4.0.5 that code is immediately followed by

    # Check to see if we just dissected a web path missing its WebHome
    if($topic ne "") {
      if(!$this->{store}->topicExists($web,$topic)) {
        if($this->{store}->webExists("$web/$topic")) {
          $web .= '/'.$topic;
          $topic = "";
        }
      }
    }

Which checks if the topic is actually a subweb, provided the topic doesn't exist. This code works even if hierarchical webs are disabled, for instance if someone tries to load "Main".

Looks like someone didn't like that feature? I can't see anything in the svn version doing the same thing...

-- WoutMertens - 28 Apr 2008

I searched for this change and found Tasks:Item2998. Since 4.1.0 it's needed to access subwebs by, at least, appending a trailing '/' to URL.

-- GilmarSantosJr - 30 Apr 2008

Heya Gilmar - do you have any info about scheduling bringing this work into the trunk? I've played with it a little, and really would like to see it go in before we do big work on the trunk, so that we can be sure that it gets testing. I'm even somewhat tempted to just get it merged now, in one big line, just because delaying isn't getting it done.

-- SvenDowideit - 19 Jun 2008

Hi Sven,

I will work on this as much as I can from now on (except this weekend, cause of São João's party). I hope to have it almost finished within the next two weeks. Almost finished because the merge will be relatively easy and fast, but it may cause some issues that were not caught by unit tests, neither by my use experience. And there is still some work to do: new unit tests, high level documentation and updates of the official documentation.

I didn't merge it before because it would (and will) probably break some things and I couldn't deal with that before (I didn't have free time for a while). But now I can

-- GilmarSantosJr - 20 Jun 2008

One thing we need to work out, is how to make sure that Plugins work for both trunk and FoswikiRelease04x02 - It shouldn't be hard, but pretty much all the Plugins will need work.

Because the trunk is used to publish Plugins for released versions - this migration needs to be done extremely carefully.

great to see progress Gilmar

-- SvenDowideit - 29 Jun 2008

Which plugins do need work and what kind of work?

-- MichaelDaum - 30 Jun 2008

Well behaviored plugins doesn't need work (perhaps only in its unit tests, since the test framework had a few changes).

Plugins not well-behaviored are the ones that:

• print data directly
• read from %ENV directly
• call die or exit
• accesses core variables without Foswiki::Func interface

I didn't test yet many plugins other than default ones.

-- GilmarSantosJr - 30 Jun 2008

Gilmar, did you modify any plugins in your branch? If so could you please make a list of them so that we know which changes and possible merges of your code and code in development environments not yet checked in are ahead.

-- MichaelDaum - 01 Jul 2008

I updated unit tests of:

• CommentPlugin
• EditTablePlugin
• PreferencesPlugin
• TablePlugin
• FoswikiUserMappingContrib
• WysiwygPlugin

And the only one I had to modify was WysiwygPlugin, but there is no consensus yet about this (I hope to have it today). I also changed TagMePlugin, but I already fixed it in trunk some time ago. By now, I tested only core plugins and TagMePlugin. I didn't tested others, cause I have no idea about plugins "popularity" and I couldn't test all of them.

-- GilmarSantosJr - 01 Jul 2008

That's okay (only 194 plugins remaining). Hey, this is five dot oh stuff, so things are allowed to break

-- MichaelDaum - 01 Jul 2008

I merged it to core today. bugs web is already running with the changes (using CGI engine). However some work is still needed:

• Write unit tests to the new code
• Figure out and implement something to make plugins (and tests) to work with trunk and older releases (currently, at least tests, work only with trunk)
• Add support to run configure script from HTTP engine
• Release FastCGIEngineContrib, ModPerlEngineContrib and HTTPEngineContrib (see EnginesAsContribs)

Now let's play with it!

-- GilmarSantosJr - 21 Jul 2008

Has anyone got WebChanges working with FastCGI? I keep getting "Could not perform search. Error was: Not a GLOB reference" - I suspect this has something to do with forking grep (on Debian linux).

-- PeterPayne - 25 Jul 2008

Hi PeterPayne,

Thanks for the feedback! There are still many small issues, including extensive testing, to be solved and I'm working on them. I didn't merged engines yet (and they won't be merged to core, but as contribs). After that, I'll post news here. Fell free (and invited!) to report bugs. Please, include:

• Svn revision you are using
• Which web server, version, fastcgi implementation...
• Operating system (and version)

-- GilmarSantosJr - 25 Jul 2008

Hi GilmarSantosJr, I was using a merge between

• http://svn.twiki.org/svn/twiki/trunk/core rev 17083, and
• http://svn.twiki.org/svn/twiki/scratch/FoswikiStandAlone/core rev 17066

Using the command su www-data -c "perl -T twiki_fastcgi.pl -l /tmp/twiki.socket" to start FastCGI (using FCGI::ProcManager).

Using Linux version 2.6.11.12 (root@colinux) (gcc version 3.4.4 20050314 (prerelease) (Debian 3.4.3-13)) #2 Mon Apr 3 09:12:20 EST 2006

-- PeterPayne - 01 Aug 2008

I've tracked down what is preventing the FastCGI version of FoswikiStandAloneFeature to display the WebChanges page. In file lib/Foswiki/Sandbox.pm there is a function called sysCommand(). Inside this function there is a block that goes:
  if ( $this->{REAL_SAFE_PIPE_OPEN} ) {
    ...
    my $pid = open($handle, '-|');
    if ( $pid ) {
      # parent ...
    } else {
      # child
      open (STDERR, '>'.File::Spec->devnull()) || die "Can't kill STDERR: '$!'";
      ...
    }
    ...
  }

The line in bold causes the child to die which results in the parent die-ing when it attempts to read $handle with the message "Not a GLOB reference". By simply commenting out the line the search works as expected. But of course what if the process did try to write to STDERR, then we'd have another issue.. so anyway there's the root cause of the problem for me.

-- PeterPayne - 08 Aug 2008

Sorry for the delay, Peter. I'm still working on getting TSA stable on trunk. After solving Tasks:Item5911 and Tasks:Item5846, I'll work on other engines: fix, update and release them (I mean, put them at svn trunk).

-- GilmarSantosJr - 11 Aug 2008

Thank you GilmarSantosJr, I wasn't expecting a quick response, so thank you. If I wasn't lazy I would work out how to redirect STDERR to nul under FastCGI myself. Maybe I'll figure it out and post the answer here. The main thing is knowing where the problem is (and now I know, as posted on 08 Aug 2008).

-- PeterPayne - 13 Aug 2008

I've just tested this issue and I could confirm this behavior. Peter, your trace was perfect! The problem is that FCGI ties STDERR and doesn't seem to handle open correctly. Anyway, within children process it doesn't make sense to STDERR be tied, so the solution is to untie STDERR before redirecting it do /dev/null. I'll fix this before FastCGI engine is released.

-- GilmarSantosJr - 14 Aug 2008

Whatever happened to this project and/or how can I get it to work? Below it says it's merged to core but when I download the latest foswiki then there is no tools/*-http.pl

-- SimonHardyFrancis - 29 Nov 2009

See FastCGIEngineContrib, ModPerlEngineContrib and HTTPEngineContrib.

-- OliverKrueger - 29 Nov 2009

Thanks Oliver. The first two aren't standalone, or? And the last one sounds like it might be standalone but the topic doesn't exist...

-- SimonHardyFrancis - 29 Nov 2009

Hi Simon,

All the infra-structure needed to make it possible is merged to core. HTTPEngineContrib is under development and the others are ready. See Tasks/Item1471 and EnginesAsContribs.

-- GilmarSantosJr - 29 Nov 2009

Thanks for the reply. How long do you think it'll be before it's working?

-- SimonHardyFrancis - 29 Nov 2009

I've been very busy for a while and probably it'll take me some time to go back to foswiki development. I hope it to be released and stable within the next six months. If I get enought time, it'll be sooner. Watch Tasks/Item1471 for news.

-- GilmarSantosJr - 30 Nov 2009

Commenting is disabled while not logged in