Item141: Rewrite the web DAV plugin (37h)

The WebDAVPlugin has some limitations; as an Apache module, it is a PITA to install, a PITA to debug, only works on 1.3, uses TDB. It needs modernising.

Background

In 2004 I implemented the WebDAVPlugin module to extend HTTP to support the DAV (Distributed Authoring and Versioning) protocol with TWiki content. DAV is a protocol extension to HTTP. It defines a set of requests that can be sent to a server to manage a remote filesystem. On the server side, DAV is installed as an Apache module called twiki_dav, which intercepts DAV requests and handles them. twiki_dav module was a c-code program based on the original mod_dav written for Apache 1.3, coupled with a TWiki plugin and external maintenance scripts.

Features of the module included:

• Support for TWiki ACLs in limiting access to TWiki topics and attachments
• Presentation of pub and data areas of TWiki topics for access via DAV
• Simplistic locking mechanism, for locking during updates
• No requirement to run mod_perl

Limitations were:

• No support for Apache 2 - tightly coupled to Apache 1.3
• No support for locking
• No support for Delta-V (the versioning API) or DASL (the proposed searching API)
• No subwebs
• No integration of pub and data views of topics
• Hard to test
• Tricky to install on "untested" platforms (OSX, Windows, VMS etc)

R1. Basic Requirements

1. Must support Apache 2.x
2. Must present database as a WebDAV resource.
3. May (preferred) present the database as a hierarchical file system, where webs (directories) contain subwebs (subdirectories) and files (topics). Each topic must have a directory entry (for the topic text), and may have a second entry with a _files suffix that is a sub-directory, if there are any attachments on the topic. This sub-directory must contain entries for the attachments in the topic.

      Tasks/
         Item141.txt
         Item141_files/
            logo.gif
      

4. OR may present the database as a hierarchical file system with two roots, pub and data, where data contains webs (directories) which contain subwebs (subdirectories) and files (topics), and pub which contains webs (directories) which contain subwebs (subdirectories) topics (subdirectories) which contain attachments (files)

      data/
         Tasks/
            DAV/
               Item141.txt
      pub/
         Tasks/
            DAV/
               Item141/
                   logo.gif
      

5. All attachments found in the database for a topic must be included, even if not present in the topic meta-data
6. PROPFIND should support return of named topic meta-data properties from topics (TWiki properties)
7. PROPFIND may support return of named topic meta-data properties from attachments (TWiki properties)
8. Must work on TWiki 4.2.3
9. Must work with the Internet Explorer webdav redirector
10. Windows Vista support may require Web Folders.
11. Should work in Firefox (with https://addons.mozilla.org/en-US/firefox/addon/540)
12. Access to topics and attachments must be controlled by standard TWiki topic access controls.
13. The user must be authenticated using their TWiki username and password before any WebDAV directories are visible
14. Must support versioning of topic changes, a la TWiki
15. Should support SSL access to DAV resources
16. Must support Neon

A decision must be made between R1.3 and R1.4

R2. Web DAV specifics

1. Should support DAV (RFC4918).
   • Strict compliance is not required, however the server must work with Internet Explorer and the Windows redirector and shuld work with Firefox.
2. Could support Delta-V (RFC3253). No design features may preclude this.
3. Could support DASL (http://www.webdav.org/dasl/. No design features should preclude this.

Design

The module will be based on the CPAN:Apache::WebDAV module. This is a Perl module that performs the same functions as mod_dav, but does it in Perl. The module is layered on top of Filesys::Virtual. The goal is to create a Filesys::Virtual layer over the TWiki store. This idea presents some interesting possibilities:

• Not only would the TWiki store be available as a DAV store, but it would also be available to other users of Filesys::Virtual (and would make it all a lot easier to unit test).
• Because it is a pure perl module, it should be installable in any Apache that supports mod_perl (as long as all the many dependencies can be resolved!)

On the negative side:

• Apache::WebDAV is Perl, and performance may not be great (though it should be good enough)
• Requires mod_perl, incompatible with Speedy CGI and FastCGI
• Written to use mod_perl 1, and must be ported to mod_perl 2
• It has not seen any maintenance since 2006, which suggests that either the 0.01 version is perfect (unlikely) or that the usage has not been heavy enough to debug it. Indeed, that is not surprising given the easy availability of the mod_dav module for Apache 2.
• Still no Delta-V or DASL (this is no great loss unless someone has a specific requirement for this)
• May be limited to Apache login (tbd)

The other possible choice, of implementing a mod_dav provider, would probably have delivered better performance, but it also creates a huge maintenance issue for the predominantly Perl-based TWiki community.

Estimates

Apache::WebDAV

Step	Estimate (hours)	Status
Research:	5	100%
Port Apache::WebDAV to mod_perl 2	8	70%
Create Filesys::Virtual::TWiki	12	60%
Testing	12	20%%
Total	37

Status

• The table above gives progress on the major tasks.
• As of 12/11/08, I have the basic WebDAV server going, and have a partial Filesys::Virtual::NextWiki that I can use to browse around from a WebDAV client.

Open issues

1. TWiki::Func is brain dead. It's impossible to get a list of attachments from the data store; you can only list the subset of attachments actually recorded in the topic meta-data. Short term, going to have to hack around TWiki::Func. Long-term, need to fix this in NextWiki.

-- CrawfordCurrie - 11 Nov 2008 - 11:25

This is so cool. I envied the confluence-users for so long for their WebDAV-Plugin. Maybe you want to have a look at this and this. -- MartinSeibert - 11 Nov 2008 - 21:25

---

This has been put on the back burner at the client's request.

• CrawfordCurrie - 27 Nov 2008

Any news on this development yet? -- MartinSeibert - 22 Apr 2009

Done, for a client who is currently testing. However it is not on public release. Contact AndreUlrich for availability.

-- CrawfordCurrie - 22 Apr 2009

So, this is closed source to buy?

-- MartinSeibert - 19 May 2009

The implementation comprises two distinct parts, a file system module that interfaces to the Foswiki store, and a generic Apache module that is independent of Foswiki. The file system module is open source, and is released under the GPL. The Apache module (the hard part) is a generic work, written ground-up by me without reference to Foswiki, and is licensed differently. The combined work is not being made publicly available. Please contact AndreUlrich for availability.

Because insufficient people committed to supporting the work, we took the risk and went ahead at our own expense (a significant investment for two small businesses).

-- CrawfordCurrie - 30 May 2009

Are there any movements in this direction? Any hints about how long it will be on the "Waiting for Release" stage? Because it would be really nice to have a possibility to edit attachments directly.

-- YuryGolev - 11 Apr 2011

The implementation is complete, and we are just waiting for enough people to help use recover some of the (significant) development costs by buying a copy before we open source it.

If you want to discuss this further and perhaps buy a license or two, contact AndreUlrich.

-- CrawfordCurrie - 11 Apr 2011

The suite now supports FastCGI, so lighttpd and nginx and IIS can all be used. Please contact me for availability.

-- CrawfordCurrie - 20 Feb 2014