Release Meeting: 08 Feb 2016

(07:51:49 AM) vrurg entered the room.
(07:56:19 AM) jomo entered the room.
(07:56:36 AM) jomo: hijall :)
(07:56:49 AM) gac410: hi jomo
(08:03:00 AM) gac410: Hi everyone ... Good morning.    Are we ready to start?   
(08:03:08 AM) vrurg: G'morning
(08:03:24 AM) jomo: probably need w8 for others a bit more...
(08:03:55 AM) vrurg: Is it only 3 of us alive?
(08:03:57 AM) ***uebera|| is listening, but busy in another window; not much to add from my side anyway today
(08:05:34 AM) gac410: Okay ...  Let's start.  
(08:05:58 AM) MichaelDaum: hey guys
(08:06:24 AM) gac410: Howdy MichaelDaum
(08:07:35 AM) gac410: First on agenda: Urgent tasks ...    There were a couple of minor breakages in 2.1 already reported.   Both already fixed.
(08:08:21 AM) CDot entered the room.
(08:08:25 AM) MichaelDaum: which bug items are these?
(08:08:32 AM) gac410: Older task  Item13795,  redundant params in ScriptURL.  I'd like to table for a discussion of a proposal
(08:09:17 AM) MichaelDaum: http://foswiki.org/Tasks/Item13795
(08:09:43 AM) jomo: good bot ;)
(08:09:46 AM) MichaelDaum: http://foswiki.org/Development/ContinueCanonicalSCRIPTURLDev
(08:10:26 AM) gac410: Item13945 - Actually a perl bug in Perl 510.   Unicode class \p{Ascii} is case sensitive - needs to be ASCII 
(08:11:05 AM) MichaelDaum: Item13945 seems to be a simple fix
(08:11:25 AM) gac410: and http://foswiki.org/Tasks/Item13944   I missed a deprecated function in the zones refactoring.
(08:11:35 AM) gac410: Both already have fixes committed
(08:11:39 AM) MichaelDaum: _we_ all missed it
(08:12:03 AM) gac410: okay.   Thanks  ;)
(08:12:23 AM) ***gac410 was taking the bullet for that one  :D
(08:13:20 AM) MichaelDaum: I've got lots of changes to JQueryPlugin
(08:13:26 AM) gac410: And http:://foswiki.org/Tasks/Item13941   Pretty straight forward.
(08:13:43 AM) fredericd entered the room.
(08:14:15 AM) gac410: So that's it for urgent task review,  unless anyone has any new tasks to escalate
(08:14:44 AM) jast: I haven't had time yet to upgrade anything to 2.1 :/
(08:16:03 AM) gac410: So #2 on agenda is the feature review.    There are a couple of minor ones,  and I expect we'll spend the majority of time on vrurg's  Mooification work
(08:16:07 AM) MichaelDaum: so 2.1.0 seems to follow the tradition of all Dot-Oh releases 
(08:17:28 AM) gac410: Oh... before we move onto feature reviews.     2.0.4. ...   Should we bother releasing it?    2.1 is fairly minor as feature releases go.
(08:17:38 AM) JulianLevens entered the room.
(08:18:12 AM) gac410: I can build it .. not a big deal,   but wonder if releasing it will just confuse the community
(08:18:22 AM) MichaelDaum: y
(08:18:35 AM) MichaelDaum: it makes more sense to go with a 2.1.1
(08:18:43 AM) ***uebera|| seconds this
(08:18:49 AM) ***vrurg too
(08:19:41 AM) gac410: Okay.    I'll mark all the pending tasks in 2.0.4  as closed / fixed in 2.1.0     And probably wait a couple of weeks before working on 2.1.1.   The urgents so far are not major
(08:19:58 AM) gac410: Now...  Features.    
(08:20:27 AM) MichaelDaum: +1
(08:20:47 AM) gac410: (foswiki.org can be really slow at times  :( )
(08:21:25 AM) jomo: yes - too many times is slow...
(08:21:46 AM) gac410: Proposals that tripped the 14-day clock recently:
(08:22:25 AM) gac410: http://foswiki.org/Development/UseRelativeLinksByDefault ...  One of mine.   fairly straight forward.   Changing all the %SCRIPTURL% macros to %SCRIPTURLPATH% when practical.
(08:23:18 AM) JulianLevens: Would that be %SCRIPTURLPATH{ ... }%
(08:23:28 AM) jast: sure
(08:23:51 AM) gac410: Yes.   At same time I'll change to that form ... except in extensions that need to be backwards compatible with Foswiki 1.x
(08:24:56 AM) gac410: Somewhat related is http://foswiki.org/Development/ContinueCanonicalSCRIPTURLDev ...  that one is only 4 days old and needs some discussion.  But there are some current issues wiith the SCRIPTURL{} form ...
(08:25:23 AM) gac410: But Let's not discuss it here today -    Table til next release meeting - plenty of time on that one.
(08:27:18 AM) gac410: And http://foswiki.org/Development/SupportAllMacrosInTemplateTopics    That one is CDot's ...  Looks like it can go to ready to implement.  
(08:27:49 AM) ***CDot has nothing more to add to it
(08:30:30 AM) gac410: http://foswiki.org/Development/AdoptSASS ... has passed it's time,  but there really doesn't seem to be any action in the proposal, so I don't know how we could flip it to accepted.
(08:31:23 AM) MichaelDaum: this is under investigation
(08:31:29 AM) gac410: MichaelDaum:   Your "Redundant Reads in Store"   ... could that one be parked?
(08:31:45 AM) MichaelDaum: sass will have quite an impact on the way we deal with css 
(08:32:09 AM) MichaelDaum: in BuildContrib, PatternSkin and JQueryPlugin and other plugins
(08:32:34 AM) gac410: MichaelDaum:   on the Adopt SASS,  might want to remove the commit date,  until it's a proposal that we can action.   Once date is set,  it automatically gets approved in 14 days.  (in theory)
(08:32:50 AM) MichaelDaum: ah ok will remove the date
(08:33:09 AM) MichaelDaum: and then what comes out of it in a feature branch
(08:33:30 AM) gac410: I didn't understand that 
(08:33:49 AM) MichaelDaum: and then see what comes out of it in a feature branch
(08:34:25 AM) gac410: Or,   add some meat to the proposal where we could agree with or raise concerns against  ;)
(08:35:11 AM) gac410: we are at +34 minutes.   Let's move onto vrurg's  prposal.  and his Mooification work.
(08:35:32 AM) gac410: He added a number of things to the release meeting agenda that he wanted to review / get feedback on.
(08:36:44 AM) gac410: vrurg:    Do you have a particular order that you'd like to get feedback on?   
(08:37:21 AM) vrurg: Except for the first item on the agenda which is better go first.
(08:37:28 AM) vrurg: Otherwise - no
(08:37:35 AM) jomo: http://foswiki.org/Development/ReleaseMeeting02X02_20160208
(08:38:12 AM) vrurg: Anyway what I'm doing right now is just the base for these proposal.
(08:39:02 AM) vrurg: And it's gonna take some time untils is ready for real redesign task.
(08:39:08 AM) jast: I'm not sure I understand which problem the OO refactoring is intended to solve
(08:39:27 AM) jast: so I'm hard-pressed to say anything relevant
(08:39:46 AM) jomo: basic question (at least for me) - the 3.0 will be some "new (read fully OO) API" or the new API will be in 4.0 and the 3.0 will be "only" an Moo-fication of the current codebase?
(08:40:28 AM) gac410: I don't think we are even close enough to put a release number on it.   
(08:40:47 AM) MichaelDaum: moo-ification will also impact any re-design of the user code
(08:41:25 AM) MichaelDaum: basically it does not make much sense to even start on a new user code given that all of that code would look quite different under moo
(08:41:31 AM) vrurg: jomo: OO API has to be in 3.0 in my view. Alongside with traditional Foswiki::Func.
(08:42:33 AM) gac410: So maybe jast's question is the most important.    What does such a major refactoring buy us?     OO is nice to do,  but how does this help "our customer"?
(08:42:48 AM) jast: ... and future development, I guess
(08:42:50 AM) CDot: +1
(08:44:18 AM) CDot: there are a number of things that would be good to do that are made harder by not having a robust OO architecture - such as cleaning up the store/meta divide
(08:44:20 AM) vrurg: It gives more reliable code.
(08:44:21 AM) jomo: probably isn't worth list here the benefits of the using Moo* - for the develoeprs. It is hidden to the customers (mean for the non-developing ones) - so...
(08:44:30 AM) CDot: Moo-ifying may motivate those changes to happen
(08:44:53 AM) jast: well, OO doesn't intrinsically make anything more reliable
(08:44:59 AM) jast: okay, store/meta divide
(08:45:13 AM) vrurg: jast: I mean Moo in particular.
(08:45:31 AM) jast: CDot: what aspect of OO do you think will help us with that?
(08:45:49 AM) CDot: jast: encapsulation, principally
(08:45:50 AM) vrurg: Exception handling – it does need rehinking and reorganizing.
(08:46:01 AM) CDot: a the moment it's too easy to leak across the interfaces
(08:46:31 AM) jast: these OO kits for Perl tend to just hide their internals slightly better, but encapsulation is still leaky
(08:46:48 AM) CDot: indeed, it's a fundamental problem with perl
(08:47:04 AM) CDot: but hiding internals is good
(08:47:37 AM) vrurg: jast: But the toolkit still imposes some rules which are easier to follow. This is one of the ways it motivates a developer.
(08:47:48 AM) JulianLevens: We are in need of code clean-up and better separation of concerns regardless of Moo-ification or not
(08:48:16 AM) MichaelDaum: in general using a proper oo system will result in better code quality
(08:48:17 AM) JulianLevens: Clears the way for more user improvements in the long term
(08:48:35 AM) jomo: JulianLevens++ - the Moofication just helps (a lot) write cleaner and more easily readable code...
(08:48:36 AM) MichaelDaum: sure, you could still code crap in oo
(08:48:41 AM) vrurg: For example, it's no a big deal to have weaken a reference – but having `has session => (weak_ref=>1)' – is preferable.
(08:49:03 AM) CDot: the way I look at is this. I personally don't think Moo-ification is a high priority, so I'm not motivated to work on it. But Vadim does see it as a priority, and I know it needs doing, so I'm not going to get in Vadim's way.
(08:49:05 AM) jast: fair arguments
(08:49:27 AM) MichaelDaum: however, a properly designed oo architectore is a lot more robust and future prove than following the current path. some areas of the foswiki core are really shitty, not even oo-ish as perl allows it to do ootb
(08:49:37 AM) jast: though I'd say at this point refactoring Foswiki::Func doesn't make a lot of sense if we don't do it completely
(08:50:02 AM) jast: right now plenty of places that need a session object (and right now use Foswiki::Func instead) don't actually get one
(08:50:08 AM) vrurg: jast: Agree.
(08:50:55 AM) jast: just one thing: I think you're underestimating the complexity of maintaining a glue/compatibility layer
(08:51:07 AM) ***CDot agrees with jast here
(08:51:55 AM) MichaelDaum: while that is true in general, we don't have enuf experience and data to says so in a more informed way.
(08:52:03 AM) jast: it's universally true
(08:52:07 AM) MichaelDaum: i.e. what exactly will have to be incompatible
(08:52:07 AM) vrurg: This is where I will need a lot of help from the community. When I'm done with the current process the code will remain pretty much the same. What I need is help in designing new OO/Exception model. 
(08:52:10 AM) CDot: I do kinda like the idea of OO-ifying the core to the point where plugins cease to exist as a concept
(08:52:21 AM) gac410: So one thing vrurg is proposing is to split the Foswiki "Session"   into two pieces.   A true "Session" object  ( login -> logout lifetime )  and a "Transaction" object   Today we mix the two.
(08:52:37 AM) CDot: that makes sense
(08:52:39 AM) jast: that's why I generally prefer not making huge breaking changes to the API... but you all make some good points, so what I'd really like to see is a more nuanced consideration of trade-offs for individual areas
(08:53:21 AM) MichaelDaum: gac410, we only mix terminology but not the objects in the code
(08:53:50 AM) jast: agreed
(08:53:51 AM) JulianLevens: but confuses newcomers :(
(08:53:52 AM) gac410: Right,  but it is terribly confusing for a new developer
(08:53:59 AM) jast: 'session' is not the ideal name for the Foswiki instance
(08:54:05 AM) jast: I struggle to find something better, though
(08:54:11 AM) MichaelDaum: $foswiki
(08:54:25 AM) CDot: I was originally going to call it 'twiki'. Just be grateful.....
(08:54:30 AM) gac410: :P
(08:54:39 AM) jast: let's compromise on $mediawiki
(08:54:40 AM) MichaelDaum: $foswiki = Foswiki-new()
(08:54:42 AM) vrurg: gac410: It is also about separating static and object methods into two modules. 
(08:55:05 AM) jast: why?
(08:55:22 AM) jomo: or just "$f" as uses "$c" in catalyst, "$m" Mason ...
(08:56:12 AM) ***MichaelDaum prefers $variablesThatTalkToMe
(08:56:13 AM) vrurg: jast: It should have been 'so-called static' . For example, what is the point of keeping load_package together with real methods?
(08:57:01 AM) vrurg: Foswiki has a number of subs which are more like API not session (in its current meaning) related stuff.
(08:57:01 AM) MichaelDaum: vrurg, you mean something like Foswiki::Utils for everything static
(08:57:20 AM) jast: or Foswiki::Func ;)
(08:57:20 AM) vrurg: MichaelDaum: Right. Minor but necessary in my view. 
(08:57:24 AM) gac410: All the load "bootstrap" methods should have been in a separate module.    since they are never used once the LocalSite is saved.  
(08:57:30 AM) vrurg: jast: Not all of them are for Func.
(08:57:45 AM) MichaelDaum: gac410, +1
(08:57:52 AM) vrurg: MichaelDaum: And I'd better keep them in Foswiki and introduce Foswiki::Transaction or whatever name it gets.
(08:58:05 AM) MichaelDaum: Foswiki::Core
(08:58:28 AM) gac410: vrurg.   Are there any particular decisions you need to make that you are looking on advice on.     "Should I do X or Y ..."
(08:58:30 AM) MichaelDaum: a transaction is something that comes and goes and is able to be rolled back
(08:58:31 AM) vrurg: Foswiki::load_package must shorter to write. :)
(08:58:56 AM) MichaelDaum: the $foswikiCore might be there to stay in memory between calls
(08:59:24 AM) MichaelDaum: initialized only once during compile time
(08:59:30 AM) vrurg: gac410: Not for this stage, as I said above. But everything I have in the agenda is what I need to hear about.
(09:00:13 AM) vrurg: MichaelDaum: let's leave this aside. These a really minor issues we can always resolve later.
(09:00:32 AM) gac410: So the agenda items vrurg wanted to discuss are here:  http://foswiki.org/Development/ReleaseMeeting02X02_20160208#A_4_Planning_for_release_3.0
(09:00:33 AM) vrurg: But what is needed is the Plan. 
(09:00:58 AM) gac410: Though a better title might have been planning for Mooification
(09:01:14 AM) vrurg: Splitting of Foswiki.pm is just a part of bigger task of preparing to PSGI. 
(09:01:25 AM) MichaelDaum: y
(09:02:17 AM) jomo: if you could eliminate the monstroous BEGIN block from the Foswiki.pm - means BIG step forwand... ;)
(09:02:18 AM) MichaelDaum: when following the moo path, we need a stage in between
(09:02:27 AM) vrurg: This is about what was discussed earlier – encapsulation and layering.
(09:02:47 AM) vrurg: jomo: Started with this.
(09:02:48 AM) MichaelDaum: what is the minimal effort to get to a running moo core while leaving all redesign aside 
(09:03:00 AM) gac410: maybe something else we should briefly address is Lavr's objection to Moo.   Obviously a community vote can override. 
(09:03:53 AM) MichaelDaum: the biggest hassle seems to be to even get started with Moo while leaving most of the code intact as far as possible
(09:04:10 AM) gac410: The well used issue,  is Centos  is horribly out of date ... and missing packages ...   again ...  
(09:04:19 AM) MichaelDaum: once that is achieved we could merge to head
(09:04:23 AM) vrurg: MichaelDaum: Not sure if I got you right... But this is my plan for now: no redesign (expect of convertions of some classes to Roles where it's the solution floaring atop), get it working as is.
(09:04:37 AM) MichaelDaum: vrurg, okay so we are on the same page.
(09:05:16 AM) MichaelDaum: the only part missing to make it a plan is an agreement by the community to merge your work to head
(09:05:50 AM) vrurg: The only major issue we might hit into is that I'm getting rid of new (this is well within Moo approach, of course) and move as much as possible to lazy attributes with their own default/builder.
(09:06:42 AM) vrurg: This may make code like $obj->{attribute} in a, say, plugin, buggy because attribute might happen to be unitialized yet.
(09:07:15 AM) MichaelDaum: still this is straight forward to fix isnt it
(09:07:49 AM) MichaelDaum: compared to any redesign or splitting up current classes
(09:07:50 AM) vrurg: Sure it is. Actually this is ~85% of all work I do know: search this->{ -> replace with this-> ;)
(09:08:11 AM) gac410: So for development strategy. ... do we hold off on merging features into master?    or proceed with a 2.2 for July,  recognizing that it will make the Moo Merge harder.   Or...
(09:08:11 AM) gac410: We could eventually rename the MooMaster -> master    and cherry pick or merge features into vrurg's  Mooified code.
(09:08:17 AM) jomo: i understand it me - it is straight forward in our codebase - but not in the user's codebase...
(09:08:50 AM) MichaelDaum: jomo, the user code is missing essential concepts to make it work properly
(09:09:27 AM) MichaelDaum: e.g. there are no Foswiki::User or Foswiki::Group classes
(09:09:58 AM) CDot: I like vrurg's approach.
(09:10:15 AM) vrurg: There is no clear structure of exception classes which would make error processing transparent. Currently it a terrible mess.
(09:10:54 AM) CDot: vrurg: there *is* a clear structure - just it's a very flat one.
(09:11:26 AM) CDot: all exceptions inherit from Error::Simple - can't get much flatter than that.
(09:12:02 AM) vrurg: CDot: have you ever tried to track down a syntax error in a dinamically loaded module? ;) This is why load_package is my word for today – I'm throwing away eval's. 
(09:12:35 AM) vrurg: But most of the time it's pure Error::Simple. 
(09:12:43 AM) CDot: indeed I have. And I have various techniques for doing it that I should probably have published.
(09:12:46 AM) gac410: +1    yay!    I have spent more time trying to find that 
(09:13:03 AM) CDot: the most obvious being to ensure all modules are unit tested
(09:13:08 AM) vrurg: Because Error is deprecated now I replace it with Try::Tiny and Throwable.
(09:13:21 AM) CDot: no problem with that, if it works.
(09:14:23 AM) CDot: and there is no doubt a strong case for handling exceptions better. But I don't think it's a "terrible mess".
(09:14:28 AM) vrurg: CDot: it does. Actually, I was wrong about 85% of the time – approx. another 15% is about replacing try/catch. Try::Tiny is very different in both syntac and approach.
(09:15:06 AM) CDot: as long as we have catch and otherwise, I'm happy.
(09:15:31 AM) vrurg: CDot: Perhaps I have this feeling because for the moment it's a mixture of different subsystems. So, sometimes it's a hell.
(09:15:44 AM) CDot: indeed it is.
(09:16:15 AM) jomo: not need "defend" the current codebase. Foswiki is great. vrurg just want make it even more better and cleaner... ;) :)
(09:16:50 AM) ***CDot wants to make sure that the real problems are addressed.... eventually
(09:16:54 AM) vrurg: Ok, I didn't expect any practical ideas on what is in the agenda. Perhaps we need another topic for architechtural discussion? 
(09:17:14 AM) CDot: go for it
(09:17:24 AM) vrurg: jomo: It is. I agree – otherwise I wound't be here. 
(09:17:37 AM) ***vrurg is tired and getting rather emotional sometimes. :)
(09:18:20 AM) gac410: Okay.    T  +1:15      Anything you still need to cover from your points in the agenda vrurg?
(09:18:30 AM) vrurg: Shall it be another proposal? 
(09:18:49 AM) jomo: +1 for the new topic - my be - is should contain a list of objects and their public methods... or not?
(09:19:02 AM) MichaelDaum: vrurg, I wished you could explain your notion of "one-topic, one-plugin" paradigm. this I did not understand in your proposal.
(09:19:03 AM) vrurg: gac410: No, thanks! The subject is too big for IRC chat.
(09:19:47 AM) vrurg: jomo: I will start it with items from the agenda and see what it develops into.
(09:20:47 AM) vrurg: MichaelDaum: I'd gladly explain what I have in mind.
(09:21:05 AM) MichaelDaum: :
(09:21:07 AM) MichaelDaum: )
(09:21:13 AM) ***gac410 takes as next actions:   1)  Close 2.0.4 tasks as fixed in 2.1.    2) Branch FoswikiRelease02x01  so we can start taking features into master ..
(09:21:16 AM) vrurg: But IRC isn't really appropriate place for this. Generally, it's still more of a wish than actual plan.
(09:21:32 AM) jast: gac410: good plan
(09:22:13 AM) vrurg: But think of full-text SEARCH which would load topic, expand macros, search in what it gets – it doesn't fir into one request-one topic paradigm either.
(09:22:27 AM) vrurg: s/fir/fir/
(09:22:35 AM) vrurg: fit
(09:23:14 AM) MichaelDaum: I am really not sure what you are aiming at
(09:23:45 AM) ***gac410 needs to step away for a few minutes.  Carpenter arrived.   biab... keep talking :D
(09:24:02 AM) vrurg: Shall we proceed in #foswiki or here?
(09:24:04 AM) JulianLevens: vrurg: Do you mean something like returning a bunch of partially loaded topics from the search?
(09:24:44 AM) ***vrurg is trying to formulate...
(09:25:41 AM) vrurg: Just to make sure I didn't miss something. Now when SEARCH is looking for topic content does it expand non-core macros?
(09:26:17 AM) MichaelDaum: SEARCH itself doesnt expand any macros
(09:27:02 AM) vrurg: So, to make it relly simple – it just loads the .txt file and seers for a substring inside, right?
(09:27:14 AM) jomo: expanding macros in every topic in SEARCH would be probably too expensive...
(09:27:27 AM) MichaelDaum: vrurg, right
(09:27:39 AM) vrurg: jomo: Sure it will. But what if this is exactly what a user needs? 
(09:28:05 AM) jomo: thats a question :) :)
(09:28:06 AM) MichaelDaum: %SEARCH is executed as any other macro. it is called by the TML parser which in turn picks up its output and proceeds parsing other macros
(09:28:35 AM) vrurg: But eventual page content may heavily depend on the result on macro expansion and end user doesn't really understand this aspect. He wants to find out this info no matter what!
(09:29:05 AM) JulianLevens: vrurg: remember there are database Stores but logically it's something like retrieving a sub string from the txt file
(09:29:07 AM) CDot: vrurg: so you want to enable the searching of rendered page output, and tracking that back to the original topic?
(09:29:27 AM) CDot: it's usual to use caches for that sort of thing
(09:29:31 AM) vrurg: CDot: exactly. I didn't have the right word to express it.
(09:29:55 AM) vrurg: CDot: Of course, this must be optional behaviour. And of course it will require caches.
(09:29:56 AM) MichaelDaum: vrurg, here is the difference: %SEARCH does not expand content _before_ checking its content. as a consequence you can't match on dynamic content produced by other macros, just net data as stored in the txt file
(09:30:19 AM) CDot: MichaelDaum: no, but you could search a rendered cache
(09:30:32 AM) vrurg: Even more: there must be a way for a plugin to decalre a page uncachable if it knows that this is the way it has be.
(09:30:48 AM) CDot: there is
(09:31:03 AM) vrurg: What if a topic is not rendered yet?
(09:31:07 AM) jomo: the dirtyblocks definition...
(09:31:16 AM) vrurg: SEARCH may render it on demand and cache.
(09:31:23 AM) gac410: This is also an extremely difficult challenge.   As a topic contents when rendered may be different based upon the user viewing the topic.   Conditional includes,  Search output,  ACL checking, etc.
(09:31:33 AM) vrurg: jomo: Not only. There various aspects.
(09:32:09 AM) JulianLevens: vrurg what is this trying to achieve: what are the benefits? Can it be done another way?
(09:32:09 AM) CDot: it's an interesting idea, but very difficult. I did at one point consider a "taint" mode for this sort of thing
(09:32:28 AM) CDot: where rendered output was "tainted" with the original source of the data
(09:32:44 AM) CDot: however it was such a major re-engineering project, i didn't take it any further
(09:32:48 AM) vrurg: gac410: Excatly the point! This is why having SEARCH capable of rendering might be useful. Yet, this makes caching more complicated...
(09:33:32 AM) vrurg: JulianLevens: I don't know if there is another way. I'm too busy with coding. Most of the time I think about these things are either on IRC or while driving the car.
(09:33:46 AM) JulianLevens: I don't like the separation of concerns here that's all
(09:34:26 AM) jomo: so, something as (in very simple terms) passing an "render="yes" argument to the SEARCH macro, and this will cause searching in the rendered content?
(09:34:27 AM) vrurg: CDot: Let a plugin decide and inform the engine, have a flag in the META – and it's a big part of the job already done.
(09:34:27 AM) JulianLevens: The Store could do more work esp with database backends - but let it process the raw data before returning
(09:35:00 AM) vrurg: jomo: Right.
(09:35:02 AM) MichaelDaum: the reason why it probably makes not much sense to %SEARCH for pre-rendered content is that we don't have a separate search-index.
(09:35:48 AM) MichaelDaum: a full-text search engine can shift lots of work from query time to indexing time ...and the latter can then be arbitrarily complex not harming query time
(09:35:54 AM) MichaelDaum: and query time is what users experience
(09:36:27 AM) jomo: btw, the SEARCH should be the fature of the Store - currently the search pulls every topic into the perl-memory - which is waste for example in DB based Stores... (imho)
(09:36:32 AM) MichaelDaum: so if we had a search index, then storing a pre-rendered version of a topic makes sense, and actually a client asked for it for SolrPlugin
(09:36:47 AM) jomo: s/fature/feature/
(09:37:16 AM) vrurg: MichaelDaum: This is why it has to be optional feature. If user check 'full text' on – he knows it's a time consuming process. Even more – it might be ran as a background task while user proceeds with other tasks.
(09:37:36 AM) JulianLevens: jomo: exactly that's why I'd like to propose partially loaded topics, but it needs quite a bit of thinking thru
(09:37:56 AM) MichaelDaum: vrurg, I'd rather not implement this in the Foswiki core and use a different subsystem for it ... fit for the job
(09:38:00 AM) vrurg: jomo: Not possible, I'm afraid, unless you forget about rendering.
(09:38:46 AM) jomo: vrurg: yes - sure for the rendered search it is needed to pull to perl and render...
(09:38:46 AM) MichaelDaum: think of foswiki as an aggregation step of different information coming from other sub-systems that do the hard work (sql, full-text indexing, web-service yadda, ...)
(09:38:50 AM) CDot: jomo: only on the basic engines. The DBIStoreCOntrib, for example, uses the DB to perform search filtering
(09:39:00 AM) vrurg: MichaelDaum: doesn't matter how to implement it. Still, per-user cache might prove to be very useful and may solve a lot of issues.
(09:39:05 AM) CDot: it only pulls in topics that match the filters that the DB supports.
(09:39:18 AM) MichaelDaum: vrurg, we do have a per user cache of pre-rendered pages alrady ;)
(09:39:42 AM) CDot: in the case of psql, for example, the DB does 99% of the search work.
(09:39:52 AM) vrurg: MichaelDaum: ok, half of the job is done. This is, perhaps, the next step in my work because last time SEARCH was failing with metacache.
(09:40:00 AM) jomo: CDot: okay, i didn't digged deeply into. So, it overrides the "default" search method?
(09:40:16 AM) CDot: no, it completely replaces it :-)
(09:40:22 AM) jomo: :)
(09:40:40 AM) CDot: architecturaly, separating "search" and "store" was never a good idea.
(09:40:49 AM) MichaelDaum: problem with %SEARCH atm is that it does all of the acl checking instead of the DB 
(09:40:52 AM) jomo: ++++++
(09:40:53 AM) vrurg: But back to per-topic plugins: this is a reason I consider this feature.
(09:41:07 AM) CDot: MichaelDaum: +1
(09:41:20 AM) vrurg: Besides of the idea that a plguin must encapsulate its data and thus be an object.
(09:41:23 AM) MichaelDaum: nother problem with %SEARCH is its filtering per web
(09:41:28 AM) CDot: +2
(09:42:19 AM) CDot: I do have an "ACL pre-filter" in a branch somewhere - a cache of topic permissions (in a Storable IIRC) that I used to filter the topics passed to the search engine
(09:42:27 AM) MichaelDaum: then %SEARCH does sorting as well...
(09:42:43 AM) CDot: y, and it's horribly complex
(09:42:55 AM) jomo: i just want use $wiki->search(for => sub { ..... }). - and any details are completly hidden... :) ;)
(09:42:59 AM) CDot: pushing SEARCH functions down into the DB was too much for my small brain
(09:43:37 AM) vrurg: Split API for low and high level searches, perhaps?
(09:43:44 AM) MichaelDaum: this all should be done by the DB: sort+filter+limit search results (including ACL) ... ALL of this is done by %SEARCH coded in perl. As long as it stays like this, no real performance is achievable no matter what DB is used behind %SEARCH
(09:43:52 AM) CDot: oh, I remember where it is; in the WebDAVPlugin (yes, it's very old!)
(09:43:55 AM) vrurg: Either by a search() parameters or separate methods.
(09:44:59 AM) JulianLevens: What are you trying to achieve? I'm struggling to see the benefit to users
(09:45:12 AM) CDot: MichaelDaum: you are right. there are two problems I found, though. the first was the complexity of the sorting/filtering ops in the current SEARCH. the second was canonicalising the capabilities of the various DB backends
(09:45:24 AM) CDot: ended up with lowest common denominator, which is not good.
(09:45:57 AM) CDot: really the place to start is to redesign SEARCH (c.f. DBQUERY)
(09:46:32 AM) JulianLevens: Whay not faster SEARCH with faster rendering
(09:46:45 AM) CDot: vrurg: that split is pretty much was SEARCH & DBQUERY do.
(09:47:02 AM) CDot: no, it's not, ignore me.
(09:48:32 AM) gac410: Okay all ... I'm baaack ...   ;)
(09:48:58 AM) vrurg: JulianLevens: Imagine a topic which is by 80% is a result of plugins macro expansion. End user doesn't know about all our internals – he sees a page. Later he recalls that he saw thar info somewhere but don't remember the location. He start to search – and fails. And gets upset. ;)
(09:50:04 AM) MichaelDaum: vrurg, exactly
(09:50:09 AM) gac410: Hm   Searching cache might not do it either, ... considering "dirty" areas and non-cachable topics
(09:50:45 AM) JulianLevens: And searches covering many pages, are you going to cache 50 pages?
(09:50:46 AM) ***vrurg was working a lot with people which do not understand explanations. Either do completely or don't do it at all.
(09:51:14 AM) MichaelDaum: vrurg, but it may find the original place of the information that have been aggregated from somewhere else ... given a %SEARCH it might not find the location where the %SEARCH itself is located but the topic that %SEARCH found somewhere else
(09:51:51 AM) MichaelDaum: WebSearch itself is implemented by means of %SEARCH
(09:52:09 AM) vrurg: JulianLevens: If there is enough disk space I might even propose a background low-prio cron job which would crawl all topics and render them in name of each user and cache result where it's possible.
(09:52:13 AM) MichaelDaum: it would not make much sense to list it as a result hit for its own search
(09:53:08 AM) vrurg: MichaelDaum: many sources, remember? The page content might not come from a wiki topic.
(09:53:13 AM) JulianLevens: vrurg: That's a better option using something like SOLR
(09:53:19 AM) MichaelDaum: as long as the information is located within the wiki stored in some topic, %SEARCH will find it _there_ ... not where the %SEARCH itself is seemingly materializing it
(09:53:54 AM) MichaelDaum: things are different when displaying external sources, e.g. via %SQL
(09:54:07 AM) MichaelDaum: %SEARCH is not able to query results of %SQL
(09:54:13 AM) vrurg: Yet if the other 20% of that topic is imporant too the user would have to issue another search to locate them? :(
(09:54:28 AM) JulianLevens: vrurg: cannot ignore the need for better explanations, although it is hard
(09:54:43 AM) JulianLevens: we cannot ..
(09:55:01 AM) jomo: how i see the whole problem now: We all are too constrained with the "current" design... . Maybe(?) could help vrurg (and all of us) talk about the future Foswiki - (where it should be in the next 10 years) - without any concerns about the backward compatibility and other "limiting" factors. 
(09:55:03 AM) jomo: So we could design an new object hierarchy (methods) for such brand new foswiki - and when the design is done, we can start talking about how to implement it in backward-compatible way... 
(09:55:31 AM) vrurg: MichaelDaum: but it must not know about storage, sources and this kind of staff. Load topic, not meta – consider it – procceed further. Again – optionally, when user knows what is he doing.
(09:56:29 AM) jomo: mean - only create a list of wanted methods and object - not the code itself ;) :) ofc.
(09:56:48 AM) vrurg: jomo: Or move toward it step-by-step, depracting pieces of code.
(09:57:04 AM) MichaelDaum: vrurg, I'd consider this more an academic question. but not an approach that can be prolonged towards a production-ready design of how to pipleline information
(09:58:07 AM) MichaelDaum: when a page doesn't load within 5 seconds people won't come back and TTT (toss the tool)
(09:58:09 AM) jomo: vrurg - okay - but in this approach im unable to see the "big picture" - where we want to go... 
(09:59:14 AM) CDot: does it matter? vrurg wants a platfrom which he can experiment on. So long as that platform meets some of the requirements of ongoing Foswiki, isn't that a good thing?
(09:59:31 AM) gac410: vrurg:   What you'll find in practice in deprecation.   Is we actually seldom remove the deprecations.  Especially API.   They sit relegated to 2nd class,  but old plugins continue to function.
(09:59:48 AM) vrurg: jomo: Make the big picture – and move toward it. This is what I mean. Unfortunately it's much more difficult to implement compatibility of a brand new code.
(10:01:00 AM) CDot: gac410: it's a truism that the code continues to be constrained by features that I bet no-one is using. earlyinitHandler, for example.
(10:01:07 AM) vrurg: gac410: I know. :) It's not a problem unless it prevents new things to come.
(10:01:42 AM) ***CDot loves that attitude
(10:02:01 AM) jomo: exactly - therefore i said - design (in paper) the new objects hierarchy (and refine it until all of us ok with it) and AFTER start coding them (and also start coding the compatibilit layer) - with this way - we can follows the golden rules of the right brainstorming... (where the main rule is: no idea is wrong - at the brainstorimg phase) :)
(10:02:29 AM) gac410: So one think to do is to clearly identify 1)  new deprecations,  and 2) Deprecations that will need to be actually finalized/removed.    And communicate that back to the greater community as this gets a bit closer.
(10:02:36 AM) gac410: s/think/thing/
(10:02:48 AM) jast: the still-unreleased new version of SafeWikiPlugin uses early init ;)
(10:03:05 AM) gac410: And /me can't count.   2 things to do   1) and 2)  
(10:03:29 AM) jast: 1) and 2) are sub-things of 'identify'
(10:03:53 AM) jast: and the rest is simply an addendum. no miscounting detected. :}
(10:03:59 AM) vrurg: jast: Inheritance in action. :)
(10:04:37 AM) gac410: I'd recommend pulling out your changes into discrete proposals.    Deprecations for Foswiki 3.0    "X needs to be deprecated for future removal    and Y needs to be actually removed.
(10:05:14 AM) gac410: So for the things that have concrete actionable items,  they can be more clearly communicated.
(10:06:00 AM) jomo left the room (quit: Quit: jomo).
(10:06:10 AM) vrurg: This is the way to go in future. There is nothing to depreacte yet. 
(10:06:13 AM) gac410: Generally,  internal refactoring that doesn't directly impact the API, or requirements / dependencies,  100% compatible ...   doesn't need a proposal.
(10:06:15 AM) jomo entered the room.
(10:06:28 AM) ***vrurg would love to see proposal of Foswiki::Func deprecation...
(10:06:47 AM) gac410: ALL of it  ???   yikes
(10:06:59 AM) vrurg: gac410: 90%, I thinks.
(10:07:21 AM) gac410: That would be one of those  "deprecate,  but leave it around so that legacy continues to function,  me thinks ...
(10:07:22 AM) vrurg: Some calls are not related to OO in neither way.
(10:07:41 AM) vrurg: Yes, it is.
(10:08:08 AM) vrurg: Discourage developers of using it but have it around for long time.
(10:09:01 AM) vrurg: Though I think that by the time the code will be PSGI ready most of old plugins won't survive it. To drastic change.
(10:09:36 AM) ***jomo needs learn much much more perl and "creativity" to be able to talk about the partial (smaller) things without seeing the "big picture" ... :|
(10:11:49 AM) gac410: Killing off older plugins will be I suspect a huge issue.   The ones in github, we can fix with time.  It's the "private" extensions that will be a problem    We may end up with a "FoswikiNG"   
(10:12:11 AM) jomo: PSGI - nobody works on it (no myself) because it is imho again one too complex task - and im unable to __design__ it myself.. the design mean - drastic changes in the core...
(10:13:06 AM) vrurg: gac410: I thought about it. Would like to avoid it but ...
(10:13:17 AM) gac410: Okay everyone.   We are at 2 hours 15 minutes.    Can we close the meeting? 
(10:13:28 AM) gac410: Excellent discussion everyone!
(10:13:33 AM) vrurg: gac410: Yes, it's time too.
(10:14:00 AM) gac410: vrurg,  are you happy a way forward on your imminent decisions?
(10:15:08 AM) vrurg: Sorry, don't understand this. But not really happy yet. ;) 
(10:15:10 AM) jomo: this is the hardest question for him...
(10:15:20 AM) jomo: :)
(10:15:52 AM) vrurg: Let's go back to #foswiki.
(10:18:22 AM) JulianLevens: thanks everyone excellent stuff
(10:18:27 AM) JulianLevens left the room.
(10:18:58 AM) vrurg: Really, thanks a lot! Good discussion.
(11:03:58 AM) MichaelDaum is now known as MichaelDaum_
(11:29:23 AM) jomo left the room.
(12:18:37 PM) MichaelDaum_ left the room (quit: Quit: quit).