Feature Proposal: Add a Perl API for manipulating "addresses" as Foswiki::Address objects rather than just ($web, $topic, $attachment, $rev, ...) tuples

Motivation

Re-worked this proposal to focus on acceptance of Foswiki::Address perl code only. There was a lot of other motivation/noise/requirements generally about missing features in Foswiki wrt addressing/atomising/accessing address bits from querysearch, macros, format strings... The old proposal can be found at TopicAddressing.

Description and Documentation

The new object class, Foswiki::Address, solves the following problems:

• instances identify a 'Foswiki resource'. Webs, topics, attachments, and (perhaps controversially) parts of topics - optionally of a specific revision. More on the "... and parts of a topic" controversy:
  • It has an underlying assumption that the TopicObjectModel can be thought of as atoms arranged in a hierarchical/nested containers, "Russian dolls" fashion (meta contains types contains members contains keys).
  • It has an underlying assumption that these atoms want to be individually addressed.
  • It's not finished yet, this is partially due to lack of feedback. Foswiki::Address has a dumb regex that pretends to parse a subset of QuerySearch (that which you'd throw at VarQUERY), whereas it should really be using the Foswiki::Query parser (and/or its getField() thing).
  • I've written some code that looks like addressGet($addrObj) and addressPut($addrObj, @stuff), but I'm not sure where that code should live. Foswiki::Meta ? (Foswiki::Meta seems fundamentally web/topic-objects only, which doesn't sit perfectly with Foswiki::Address).
• introduces the '@123' revision specifier syntax from LoadDifferentTopicVersions. Without this, we can't stringify an address which points to a specific revision.
• knows how to stringify all the address forms it supports.
• knows how to parse all the stringifications it can produce. See AddressTests.pm to confirm.
• knows that Web.Topic/Attachment.pdf looks like an attachment, and uses web/topicExist heuristics to try to do-what-I-mean
• offers getters/setters to programatically update an individual atom of the Foswiki::Address (eg. change the web part, change the attachment part, etc).
• introduces some opinion about TopicObjectModel terminology (see the Foswiki::Address->type() method)
• offers a way of testing if an instance is of a certain type
• offers a way of comparing whether two Foswiki::Address instances are equivalent
• has unit tests measuring performance. Indicates that creating a new Foswiki::Address is about half as fast as creating a pure hashref.
• designed so that you can bless a hashref that looks like {web => 'Web', topic => 'Topic', rev => 3}, for those who really think OO will slow things down. You can also promote a hashref to a Foswiki::Address instance by passing this kind of hash into the Foswiki::Address->new() constructor.

#MissionStatement

Mission statement

package Foswiki::Address

This class is used to handle pointers to Foswiki 'resources', which might be webs, topics or parts of topics (such as attachments or metadata), optionally of a specific revision.

The primary goal is to end the tyranny of arbitrary (web, topic, attachment, rev...) tuples. Users of Foswiki::Address should be able to enjoy programmatically updating, stringifying, parsing, validating, comparing and passing around of "address objects" that will (eventually?) be understood by the wider Foswiki universe, without having to maintain proprietary parse/stringify/validate/comparison handling code that must always be considerate of any recipient receiving traditional (web, topic, attach, rev..) tuples.

For example, Foswiki::Address already happily parses the following addresses (and can also stringify them again, perhaps after being built/updated piecewise/programmatically):

• Web.Topic/Attachment.jpg
• Web.Topic/Attachment.jpg@3
• Web.Topic
• Web/
• Web/SubWeb/
• 'Web.Topic'/text
• 'Web.Topic@2'/preferences[name='WEBBGCOLOR']
• 'Web.Topic@2'/preferences
• 'Web.Topic@2'/META:PREFERENCES
• 'Web.Topic'/Colour
• 'Web.Topic'/MyForm.Colour
• 'Web.Topic'/META:TOPICINFO.author

The interface does not offer any interaction with resources themselves; rather, functionality is provided to create, hold, manipulate, test and de/serialise addresses

Fundamentally, Foswiki::Address can be thought of as an interface to a hash of the components necessary to address a specific Foswiki resource.

Examples

Foswiki::Address

The equivalent of Foswiki::Func's normalizeWebTopicName is

Foswiki::Address->new(web => 'Web/SubWeb',         topic => 'Topic');
or
Foswiki::Address->new(webpath => [qw(Web SubWeb)], topic => 'Topic');
or
Foswiki::Address->new(string => 'Web/SubWeb.Topic', isA => 'topic');
or
Foswiki::Address->new(string => 'Web/SubWeb.Topic', isA => 'topic', web => 'DefaultWeb/DefaultSubWeb');
or
Foswiki::Address->new(string => 'Web/SubWeb.Topic', isA => 'topic', webpath => [qw(DefaultWeb DefaultSubWeb)]);

Impact

WhatDoesItAffect:

%WHATDOESITAFFECT%

edit

Implementation

Already implemented. Please see http://trunk.foswiki.org/System/PerlDoc?module=Foswiki::Address

TODO

• Add an ->attachment() get/setter. Having to use the tompath() is probably just too annoying. I'm also concerned that Crawford & Sven are concerned that they won't like this tompath thing at all

-- Contributors: PaulHarvey - 08 Feb 2011

Discussion

This is still poorly thought out. Please help flesh it out properly.

-- PaulHarvey - 08 Feb 2011

I was pondering something from the other end that might have relevance:

internally we have difficulty with passing around addresses too -

Crawford refactored the horrid pain we used to have passing web,topic,attachment parameters to passing around instantiated Meta objects. But this leaves us with a pretty heavy hammer, especially when we really want to store / list items.

So - I'd like to propose we create a lightweight 'address' object, that can contain (or return) any one of the forms of address that can happen - unparsed string (what the user asked for), validated string address (what we show the user when we fulfil the request), url (what the address of this element is), display name (what the user would see in the browser), parsed address in array form (somewhat legacy, but often used) and normalised parsed and separated hash (what we'd pass to store).

The address parser is the query TOM parser, and these address objects can be passed to getField() for evaluation....

to the expose this to users, we would provide the address 'part' requests - internally, those would be the keys in the normalised parsed and separated hash

-- SvenDowideit - 09 Feb 2011

Nice proposal. It covers all the use cases I can think of.

One thought: do we have list operators already? It would make sense to replace

• parentweb with concat(webs[0,last()-2], '.'),
• containingweb with webs[last()-1] and
• rootweb with webs[0]

With regards to implementing a lightweighted address object, I'd make it a hashref only without any other package definition. As these little beasts will float around a lot, real objects are too expensive.

{
  webs => ['rootweb', 'web', 'subweb', 'subsubweb'],
  topic => 'Topic',
  rev => 123,
  attachment => 'filename',
}

-- MichaelDaum - 09 Feb 2011

I'm far from convinced by this idea that "real objects are expensive". I don't know all the details of the current perl implementation, but as I understand it an object is just a blessed hash i.e. is represented in memory by a hash with one additional address field, so is hardly more expensive than an unblessed hash. Of course there is a cost if you use the $object->method() syntax to call methods on it, but that's a very small price to pay and a choice the caller can make, compared to the pain of having to deal with an inextensible address object. I haven't seen any credible evidence of the supposed cost of perl objects; of course if you create a lot of small "things" there will be a cost, but whether they are objects or unblessed hashes makes little difference AFAICT.

Having said that, I like the idea of an address object; it reduces the complexity of a lot of code, especially Foswiki::Meta. It's an awful lot of work, though. Some considerations:

• Can an address object that refers to an attachment also be taken as a reference to a topic (or a web)?
• Are address objects and Foswiki::Meta (or whatever the "loaded topic" object is called) freely convertable? Meta is clearly a specialised address object.....
• How are these address objects communicated across the Foswiki::Func interface? Is it time for a "new" plugins interface?
• Is an address object extensible to address other (as yet unthought of) subdivisions of a topic (e.g. sections, tables, paragraphs)? How?
• How are these objects serialised/deserialised (important for REST)?

-- CrawfordCurrie - 09 Feb 2011

Compare these two things:

Plain hashref:

my $objAddr = {
  webs => ['rootweb', 'web', 'subweb', 'subsubweb'],
  topic => 'Topic',
  rev => 123,
  attachment => 'filename',
};

Object:

require Foswiki::ObjectAddress;
my $objAddr = Foswiki::ObectAddress->new(
  webs => ['rootweb', 'web', 'subweb', 'subsubweb'],
  topic => 'Topic',
  rev => 123,
  attachment => 'filename',
);

Now imagine 10k objAddrs being created. What's the difference between those two? Is the additional fluff worth it?

-- MichaelDaum - 09 Feb 2011

In any language except perl I would say immediately "yes, without a shadow of a doubt". However perl seems to have problems with OO - or at least, so you keep telling me, though I have yet to see any convincing evidence.

-- CrawfordCurrie - 09 Feb 2011

if you're creating 10k objAddrs, then you are doing something wrong - in that case, store the 10k whatever format you get them in, and use the objAddrs methods to manipulate them as needed.

I do not think that is going to be an important issue, compared to making an API that allows us to simplify the codebase, rather than re-re-re-(un)parsing addresses and separated addresses.

can we remove the ToOOOrNotToOO form this topic as its really not relevant here?

-- SvenDowideit - 09 Feb 2011

Why delete. Keep it, please. It is part of our discussion.

Anyway.

May the coder of this feature decide how lightweight the objAddr should be. I'd rather prefer to focus more on the actual proposal, i.e. how to address parts of the full qualified web.topic address.

-- MichaelDaum - 09 Feb 2011

As soon as you need methods on the topic data, you wished you had used OO.

-- ArthurClemens - 19 Feb 2011

Actually, may not need the macro if VarFORMAT's $addresspary() is expressive enough

-- PaulHarvey - 14 Mar 2011

Why not Site:Web.Topic@rev with WIKITOOLNAME as Site?

-- OliverKrueger - 15 Mar 2011

Are you proposing we incorporate interwiki links into the address specification?

-- PaulHarvey - 16 Mar 2011

oh yes please - Foswiki::Address should implement interwiki links

-- SvenDowideit - 17 Mar 2011

These timing tests were obtained from one of my VMs on distro:6458c91c8b2b

AddressTests::... and FOSWIKI_ASSERTS=0

Test	Note	Throughput	%
test_timing_normaliseWebTopicName_default	normalizeWebTopicName('Foo', 'Bar')	222222.22/s (n=20000)	100
test_timing_normaliseWebTopicName_equiv_default	Foswiki::Address->new() equivalent	34482.76/s (n=10000)	15.5
test_timing_reparse_default	Foswiki::Address->parse() equivalent	42857.14/s (n=15000)	19.3
test_timing_normaliseWebTopicName	normalizeWebTopicName('', 'Foo.Bar')	166666.67/s (n=15000)	100
test_timing_normaliseWebTopicName_equiv	Foswiki::Address->new() equivalent	37037.04/s (n=10000)	13.8
test_timing_reparse	Foswiki::Address->parse() equivalent	42857.14/s (n=15000)	25.7
test_timing_hashref_creation	Pure hashref creation	476190.48/s (n=200000)	100
test_timing_creation	Foswiki::Address creation	238095.24/s (n=100000)	50

-- PaulHarvey - 18 Mar 2011

pure hashref is twice as fast as Foswiki::Address->new(); which probably makes sense (there's an if ($opts{string}) statement to check if we need to parse).

-- PaulHarvey - 18 Mar 2011

I've cancelled the clock; after discussing on IRC, there are some pretty fundamental unresolved issues we need to work through. Even if there were no issues, I've also not stabilised the Foswiki::Address API (as I've started to work with it in my own plugins), so I hope this process will achieve something useful someday realsoonnow.

Added a #MissionStatement that I hope will answer the "Why not use lightweight/unloaded/justuseit Foswiki::Meta objects" and make the goals more clear.

Unresolved issues:

• Do we really need this, or should we make a cleaner, lighter Foswiki::Meta2 - where 'unloaded' objects are no heavier than Foswiki::Address and provide the same functionality?
• Should the physical file attachment be treated separately to the metadata view of the file attachment(s)? Litmus test: can we pass an address object pointing to "an attachment" to unambiguously get a filehandle/iostream; can Foswiki calculate an http URL for the resource, given the Foswiki::Address? Can we also use a pointer to an attachment to obtain the properties of the file attachment? Can we address those properties unambiguously?
• Duplicating QuerySearch parser functionality. 80% of the code in this class is related to parsing "string forms" of addresses of Foswiki resources... querysearch parser needs some refactoring so we can delete the parser code here.
• API usability - can we stop passing around (web, topic, attachment, rev) tuples - will the ->new() constructor make sense to plugin authors, core hackers? FEEDBACK WELCOME, please comment at Foswiki::Development.TopicAddressing

-- PaulHarvey - 21 Mar 2011

Given that we're starting to use Foswiki::Address in the core, and it's been implemented, I guess I'd better get a proposal accepted.

This shouldn't pressure anybody into thinking that it's "too late", quite the opposite. I think I had to implement this before I was sure that we needed it. Now I need to know how upset this thing will make everybody feel...

-- PaulHarvey - 05 Jul 2011

We need to make sure that this is flexible enough to permit SupportDynamicGeneratedAttachments and EnableCloudStorageForAttachments to be implemented.

• "cached" attachments such as image gallery thumbnails need to be addressable.
• Attachments need to be "distributable" across external content providers.

Can the address have an indicator that the attachment is a cached vs. real attachment?

-- GeorgeClark - 05 Jul 2011

Well, Foswiki::Address is a pointer to a Foswiki resource - something that Foswiki::Meta, Foswiki::Func can interact with.

How this resource can be accessed (how its URI might be calculated) in a given context for a given action with certain parameters, is up to the user of Foswiki::Address. In the case of cloud storage, I'd imagine a %URL{"Foo/Bar/Cat.jpg"}% doing something like:

   my $addrObj = Foswiki::Address->new(string => 'Foo/Bar/Cat.pdf');
   my $cloud = Which::Cloud::is_it_on($addrObj);
   my $url = $cloud->genURL($addrObj, $params->{action} || 'view');

As for coping with cache objects: I think that's beyond the scope of Foswiki::Address. Foswiki itself can't address them: /pub/cache is not a real web (so it cannot be searched), those objects may or not be versioned, and the overall usage of this subdirectory is really left to the whim of plugin authors... In other words it's not managed data that wants to be manipulated with $topicObj->putKeyed() or $topicObj->haveAccess(), AFAICT.

Maybe we could implement some managed /pub/cache and that would make use of Foswiki::Address (assuming those objects have a relationship to 'Foswiki' resources), I just don't know how to extend Foswiki::Address at this point to cover that type of thing, and I suspect that responsibility for managed cache objects should probably be delegated to a different class anyway.

-- PaulHarvey - 06 Jul 2011

So, to clarify: Foswiki::Address doesn't know or care about URLs... I hope nobody puts any code into it that generates a URL to a resource.

-- PaulHarvey - 06 Jul 2011

I've yet to get some improvements merged in from github, but nobody has raised any concerns about the concept, though Crawford and Sven seem to have reservations about the tompath aspect.

-- PaulHarvey - 21 Sep 2011