Release Meeting: 02 Nov 2015

Details

1. Task review

  • NEW13519 (Closed): Foswiki should clearly report missing dependencies. Don't crash with 500 Internal Server Error.

2. Development Discussion

Performance

MichaelDaum brought up some work he has been doing investigating optimizations in Store. Some observations:
  • Focused on RcsStoreContrib.
  • Several optimizations improve speed considerably, but disable some checks that are important.
  • noCheckinPending checks are the cause of lots of slowdown
  • any r1 - 04 Nov 2015 - 05:29:01 - GeorgeClark will considerably slow down things even with n = max revs
  • the non-macro %REVISIONS% implemented in UI::View has got similar problems
  • Further discussion on some optimization strategies. Possibly fodder for 2.1.

Feature proposals

  • GeorgeClark noted a new proposal for 2.1: AddValidationsToURLPARAMMacro
  • Also noted timer running on MakeItEasierToBlockSystemWebGuestAccess. This is also proposed for 2.1, but could be applicable to 2.0, to simplify some updates. There was discussion of possibly including it into 2.0, but we need the ACL * wildcard for backwards compatibility on 2.0
  • We need to review and clean up feature proposals That will be primary agenda item of the next Release Meeting

Following feature proposals have git branches
  • Item13525 - Branch for Store performance, needs work. MichaelDaum has patches planned.
  • Item13594 - Param+= set support. No word from Julian if ready to merge.
  • Item13740 - Security headers - ready for 2.1

Release plan

Planning for 2.0.3: Patch release
  • One outstanding urgent bug:
  • %Item% (): Fails on Foswiki.org, unable to recreate locally.
  • Plan for 2.0.3 by mid November.

Planning for 2.1: Feature proposal review

  • Feature Freeze: 1 Dec 2015 - All proposed features tested and merged from Item* branches.
  • Release from: master
  • Branch date: TBD
  • Beta start: TBD
  • Release target: 4 Jan 2015

Next meeting - - Monday 16 Nov 2015 1300Z — ReleaseMeeting02x01_20151116

IRC Log

Conversation with #foswiki-release at Mon 02 Nov 2015 08:00:10 AM EST on gac410@chat.freenode.net (irc)
(08:00:48 AM) gac410: Good morning. 1300z is early now .. Daylight savings ended in the US, ugh.
(08:18:53 AM) JulianLevens  entered the room.
(08:37:33 AM) gac410: Hi all... Is there interest in a release meeting? I was contemplating another possible feature, didn't write it up yet.
(08:39:01 AM) gac410: add a "validate="regex" option to URLPARAM. So pages could validate/filter the input from the url rather than just grabbing it and hoping the filtering is enough
(08:42:31 AM) gac410: So you could do something like %URLPARAM{"effect" default="fade" validate="(show|fade|slide|blind)"}%'}" or something like that.
(08:44:14 AM) gac410: I'm getting closer on a PatchRelease01x01Contrib that will backport the major backwards compat issues to 1.1.x ALLOW = *, etc.
(08:45:05 AM) gac410: Also going to include all the perl deprecation fixes. so sites have a bail-out if they update their server and kill their wiki
(08:48:17 AM) Lynnwood [~Lynnwood@foswiki/developer/lynnwood] entered the room.
(08:56:32 AM) jomo  entered the room.
(08:57:01 AM) jomo: hiall
(08:57:36 AM) gac410: hi jomo. ... so far no activity. I guess the daylight savings time change caught everyone.
(08:58:22 AM) jomo: :)
(08:58:46 AM) MichaelDaum: hi all
(08:59:03 AM) gac410: Hi MichaelDaum
(09:00:12 AM) MichaelDaum: been working on JQueryPlugin before leavign into vacations ... which I just returned from
(09:00:45 AM) gac410: Ah. I wondered ... you've been a bit scarce lately :) Hope you had a good vaca
(09:00:46 AM) MichaelDaum: some 3rd party libs received updates. more will probably follow.
(09:01:32 AM) gac410: I've been focused on the PatchRelease01x01Contrib Try to make it easier to install new extensions on older foswiki.
(09:01:34 AM) gac410: https://github.com/foswiki/distro/blob/Release01x01/PatchRelease01x01Contrib/data/System/PatchRelease01x01Contrib.txt
(09:01:35 AM) MichaelDaum: ya. but now I need the vacation from the vacation
(09:02:11 AM) gac410: Ya. catching up on email after a couple weeks away somewhat ruins the benefits ;)
(09:03:38 AM) MichaelDaum: got some important fixes for NatEdit as well: it does not run jquery.validate on invisible formfields ... which is the default in this lib
(09:05:13 AM) gac410: I figure a 2.0.3 could go out pretty much any time.
(09:05:20 AM) MichaelDaum: then ran a few nyt profiles to improve rcs contrib
(09:06:32 AM) MichaelDaum: and improved speed by a margin ... alas it disables noCheckinPending by default
(09:06:53 AM) MichaelDaum: as it is the culprit for lots of slowdown
(09:06:58 AM) gac410: minor tweaks to RcsStore ? ah... answered that. Won't fly for mainline I don't think.
(09:07:32 AM) MichaelDaum: y
(09:07:37 AM) gac410: We can't break foswiki for sites who edit files offline. Even foswiki.org counts on that.
(09:08:00 AM) gac410: We could maybe make that configurable?
(09:08:16 AM) MichaelDaum: what I've found out is that any %REVINFO{rev="n"}% will considerably slow down things even with n = max revs
(09:09:05 AM) MichaelDaum: NatSkin has got three calls to %REVINFO{rev="maxrev"}% all of which for an rcs to count the number of revisions.
(09:09:18 AM) MichaelDaum: ^all of which fork^
(09:09:31 AM) CDot [~crawford@foswiki/developer/cdot] entered the room.
(09:09:54 AM) gac410: CDot ... MD is discussing some RcsStore performance issues
(09:11:35 AM) MichaelDaum: the non-macro %REVISIONS% implemented in UI::View has got similar problems
(09:11:41 AM) ***CDot sees no discussion.....
(09:12:04 AM) CDot: ah, revisions. Yep, ongoing problem with RCS
(09:12:46 AM) MichaelDaum: there are some optimizations in Foswiki::Store ... in layers above the store impl that functions called _within_ the store impl itself wont profit from
(09:13:06 AM) MichaelDaum: such as ways to get the max num of revs
(09:13:27 AM) MichaelDaum: theres some reorg needed here
(09:14:52 AM) MichaelDaum: see_numRevisions() within and outside of the store impl
(09:15:42 AM) MichaelDaum: to fix %REVINFO{rev="maxrev"}% you'd need to know the num recs within the store layer to prevent the slow path, alas getting to know the max num revs is on a slow path itself as things are atm
(09:16:05 AM) MichaelDaum: ^recs^revs
(09:17:21 AM) MichaelDaum: so my current optimizations will potentially affect any store impl
(09:18:18 AM) CDot: A lot could be achieved by pucshing those optimisations down into the store impl itself. For example, a DB store has no need of embedded meta-data, but embedded metadata is essential for optimising a text-based store
(09:18:57 AM) gac410: I assume that this is 2.1 feature discussion and not a 2.0.3 patch discussion ???
(09:19:19 AM) MichaelDaum: these little info bits need to be available as easy as can be. even an sql query would be too much.
(09:19:38 AM) MichaelDaum: gac410, y
(09:20:35 AM) MichaelDaum: I started investigating as even on my new system with lots of cpus, ram and ssd foswiki did not feel as snappy as I was expecting.
(09:20:38 AM) CDot: MichaelDaum: hmmm. You are assuming that there will continue to be a "meta object" or equivalent in memory. A fast DB would be better serving *everything* using queries (though not necessairly SQL!)
(09:21:26 AM) MichaelDaum: yea well sort of
(09:21:41 AM) MichaelDaum: see REVINFO for example
(09:22:29 AM) MichaelDaum: it is okay to pay for the first call, but any additional expansion of REVINFO increases runtime linearly ... not good.
(09:23:16 AM) MichaelDaum: especially REVINFO with a rev param
(09:24:00 AM) MichaelDaum: to be precise we have got two problems here
(09:24:14 AM) MichaelDaum: (1) the first call to %REVINFO{rev="..."}% is too expensive
(09:24:20 AM) CDot: MichaelDaum: can you identify the choke points? Last time I looked (and it has been a while) it was a moving target. Some pages it's the store, some it's rendering. Some it's the sheer volume of crap sent to the browser.
(09:24:22 AM) MichaelDaum: (2) no caching
(09:25:33 AM) MichaelDaum: CDot, yes: getLatestRevisionID, _numRevisions, getRevision.
(09:26:09 AM) CDot: .... which are not an identifiable problem on DBStoreContrib.... did you look at PFS as well?
(09:26:22 AM) MichaelDaum: nope
(09:28:04 AM) CDot: ok. So, spending a lot of time firtling the core parts of the store is likely to be wasted.... though moving functionality into RCS and firtling it *there* is likely to bear fruit.
(09:28:48 AM) CDot: or, not wasted. The right phrase is "likely to f**k up other store implementations"
(09:30:43 AM) ***MichaelDaum not sure what firtling means ... my dict neither ;)
(09:31:06 AM) MichaelDaum: atm all optimizations I did are rcs contrib only
(09:34:14 AM) CDot: sorry, "firtling" means "poking around" or "hitting with a hammer until it works"
(09:34:59 AM) CDot: I'm not criticising, just observing that any core optimisations need to consider the other store impls.
(09:35:30 AM) CDot: I'd like to think we have pushed as much as is appropriate into the store impl, but there are still thinks in Meta.pm that make me uncomfortable.
(09:35:50 AM) CDot: .... starting with that horrendous hash
(09:39:57 AM) ***gac410 just created a new feature proposal. http://foswiki.org/Development/AddValidationsToURLPARAMMacro
(09:41:22 AM) gac410: Also added http://foswiki.org/Development/MakeItEasierToBlockSystemWebGuestAccess ... timer ticking for a 2.1 enhancement.
(09:42:10 AM) gac410: The latter could possible slip into a patch release as it would make it easier to update sites that have already locked down system web, and has no adverse impact that I can think of anyway
(09:43:34 AM) gac410: Other than as Michael pointed out - backwards compat with 1.1 which I'm working on a patch contrib for.
(09:45:21 AM) gac410: MichaelDaum: Can you put your performance recommendations into a feature proposal? ... maybe make some of the shortcuts configurable?
(09:46:11 AM) MichaelDaum: gac410, in general I am fine with MakeItEasierToBlockSystemWebGuestAccess however we need to be aware that we force both groups of users into an upgrade of any sort: (1) those that stay with pre 1.1.10 yet installing new plugins (2) those on post 2.0.2 installing old plugins that have System web topics
(09:46:38 AM) MichaelDaum: both will face problems
(09:47:00 AM) MichaelDaum: we need a kind of survey of plugins using System web topics for configuration
(09:47:09 AM) gac410: This proposal is NOT locking down the System web So old plugins on new systems would have no impact at all.
(09:47:34 AM) MichaelDaum: WikiGuests will
(09:47:46 AM) gac410: It is only adding the ALLOW rules to topics where a locked system web would break the wiki. And that needs a backwards compat change.
(09:47:57 AM) gac410: No. Michael, no impact if user does not change System.WebPreferences.
(09:48:07 AM) MichaelDaum: oh okay
(09:48:09 AM) MichaelDaum: well
(09:48:33 AM) gac410: I am NOT proposing changing the default access to system. Only make it easy, so if admin choses to lockdown, the wiki still works.
(09:48:40 AM) MichaelDaum: what I mean is: when wiki guest is denied view in System web, then any plugin using sttings in there needs an upgrade as well.
(09:49:13 AM) MichaelDaum: right? and when that does not happen the wiki will be broken
(09:49:20 AM) gac410: Yes. I don't know how we can address that for old / private plugins.
(09:49:48 AM) MichaelDaum: all we can do is list those plugins that need some love&care in that respect
(09:50:03 AM) gac410: What I'm proposing, is that Default, out-of-the-box, foswiki will continue to be fully functional if WebPreferences on System is set to deny guests. Right.
(09:51:26 AM) gac410: Also, if a user extracts Foswiki-upgrade-2.0.1.zip, it won't clobber the ALLOW settings required for a locked down System web.
(09:51:26 AM) gac410:
(09:52:21 AM) gac410: And based upon that, ... no changes to WebPreferences, it is probably safe to put this in a patch release rather than defer for 2.1
(09:53:10 AM) MichaelDaum: that I am not sure of: does the 1.1.x branch have a migration path at hand for ALLOW = * topics?
(09:53:47 AM) gac410: Yes. Not *quite* yet released. But I've got a PatchRelease01x01Contrib in the Release01x01 branch just about ready.
(09:53:50 AM) MichaelDaum: we really need to make sure people dont get into a catch22 situation
(09:54:12 AM) gac410: See https://github.com/foswiki/distro/blob/Release01x01/PatchRelease01x01Contrib/data/System/PatchRelease01x01Contrib.txt
(09:54:26 AM) MichaelDaum: okay.
(09:54:49 AM) MichaelDaum: as soon as we've got that in the wild will we be able to lock down System for guests....but not before ;)
(09:56:06 AM) gac410: I think I've got all the major backwards compat issues covered. I need to pick up a couple of your 1.1.10 proposed fixes though. The CLI and REST fixes probably are worth backporting to a pach.
(09:56:20 AM) gac410: patchContrib
(09:56:38 AM) MichaelDaum: xecellent
(09:57:55 AM) gac410: We really need to make a pass of our FeatureProposals, and either park the dead ones, or find someone active to pick them up. And then slot them into 2.1 or a future release.
(09:59:39 AM) gac410: http://foswiki.org/Development/ReleasePlan
(10:00:28 AM) gac410: 2.0.3 ... lets plan for a mid-November. And that's hopefully the final 2.0.x release
(10:01:33 AM) gac410: In the last release meeting, we decided Feature Freeze: 1 Dec 2015 - All proposed features tested and merged from Item* branches.
(10:02:00 AM) MichaelDaum: okay I make sure my jquery and natedit fixes are ready to go by then
(10:03:03 AM) gac410: For feature branches currently in github ...
(10:03:31 AM) MichaelDaum: could you update ReleasePlan, gac410? I'll then copy that into the FoswikiCalendar http://foswiki.org/Development/FoswikiCalendar
(10:03:40 AM) gac410: Okay. will do.
(10:04:01 AM) gac410: We have Item13594 branch. for the Param+= set implementation
(10:04:37 AM) gac410: Item13740 - add security headers
(10:05:20 AM) MichaelDaum: 2.1
(10:05:26 AM) gac410: Michael, your Item13525 branch with the Store cache ... is that defunct now?
(10:06:15 AM) MichaelDaum: I've got patches that should go in there extending this patch bit more
(10:07:17 AM) gac410: Do you still need another cache? I'm concerned that we might end up with too many caches. MetaCache StoreCache ...
(10:07:39 AM) gac410: JulianLevens: Is your Item13594 branch close to ready to merge?
(10:08:14 AM) MichaelDaum: gac410, yea right. current approach looks different.
(10:08:52 AM) MichaelDaum: it adds a {numRevs} hash to cache the max num revs of a topic.
(10:09:06 AM) MichaelDaum: as thats supprisingly expensive in rcs store
(10:09:23 AM) MichaelDaum: topic => int
(10:09:25 AM) gac410: Okay ... That seems pretty small. Vs. a full topic cache.
(10:09:33 AM) MichaelDaum: yep
(10:10:25 AM) gac410: So back to 2.0.3. ... I've got one task waiting for feedback from CDot ... but I think its good to go. We MUST encode # when in a urlparam. That's a possibly major breakage on 2.0.x
(10:11:00 AM) MichaelDaum: how is it potential breakage?
(10:11:01 AM) CDot: gac410: give me a link, I'll have a look now
(10:11:15 AM) gac410: http://foswiki.org/Tasks/Item13837
(10:11:35 AM) gac410: That is the ActionTracker bug. It really was broken. Param order was not a red herring :D
(10:11:53 AM) gac410: Any param with # would truncate the query string.
(10:11:57 AM) CDot: if it's ATP it can't possibly block a core release....
(10:12:05 AM) gac410: No.... It's a core bug
(10:12:08 AM) CDot: k
(10:12:27 AM) CDot: oh hell, yes, I guess it would
(10:12:30 AM) ***CDot reads on
(10:13:03 AM) gac410: A commit you did fixing unit tests for some reason removed # from urlEncode ...
(10:13:22 AM) gac410: But they all pass when I revert that change, so ... no idea what you were thinking there
(10:13:22 AM) CDot: I think # was added by default. I was probably reading a RFC that listed it.
(10:13:40 AM) CDot: me neiother. :-(
(10:14:03 AM) gac410: The RFC is confusing. It doesn't list # in the paragraph quoted. But elsewehere in text it says # must alsways be encoded.
(10:14:18 AM) gac410: Anyway... if fixored it.
(10:15:34 AM) gac410: Anyway, so with the http://foswiki.org/Tasks/PatchReleaseBlocker list. I don't see any reason not to just go with 2.0.3. I cannot figure out how to recreate the Extension installer issue.
(10:15:53 AM) gac410: Though it's *still* broken on foswiki.org. I cannot get MultiTopicSave to update the System topic.
(10:16:56 AM) gac410: There are enough tasks in 2.0.3, that I think we should get it out soon, http://foswiki.org/Tasks/TasksByRelease?release=2.0.3&checkins=All
(10:18:27 AM) gac410: So http://foswiki.org/Tasks/Item13785 is the only thing IMO blocking 2.0.3 ... and I'm at a loss.
(10:24:48 AM) gac410: MichaelDaum: You asked "how is that potential breakage" ... If A # appears in a query string un-encoded, it truncates the rest of the query string since it's the "fragment delimiter" aka Anchor
(10:25:02 AM) gac410: 2.0 no longer encodes # in urlparams
(10:26:12 AM) MichaelDaum: ah okay
(10:26:14 AM) MichaelDaum: sure
(10:26:24 AM) MichaelDaum: anything after some # is client side
(10:26:51 AM) gac410: It's been broken since 2.0.0 but only ActionTracker has tripped over it
(10:27:09 AM) gac410: Michael, how do I update that calendar again?
(10:27:50 AM) gac410: I updated ReleasePlan
(10:30:26 AM) MichaelDaum: see the info at http://foswiki.org/Development/FoswikiCalendar
(10:31:05 AM) MichaelDaum: but NM. I've got the calendar in my google calendar already. easy to add events to it.
(10:31:54 AM) gac410: Everyone. Please review http://foswiki.org/Development/ReleasePlan. If a feature is ready to merge, please set the CurrentState to ReadyToMerge.
(10:32:25 AM) gac410: If you don't think a feature will make the 1 Dec. feature freeze, please mark it for 2.2
(10:33:43 AM) gac410: Michale, all that page says is: Members of the Community.AssociationBoard have write access. I clicked edit, but there is no calender there to edit.
(10:33:53 AM) gac410: er. Michael (sorry)
(10:35:50 AM) ***MichaelDaum added the dates to the google calendar
(10:37:09 AM) gac410: Ah,... okay, and finally it shows up ... I had to use Chrome, not Firefox when I clicked the google calendar icon to get it iadded.
(10:38:19 AM) gac410: Anyway. EVERYONE ... homework. Please review the ReleasePlan and FeatureProposals. And make sure the 2.1 features are ready for the Dec. 1st feature freeze
(10:38:58 AM) MichaelDaum: Is Paul still interested in MakeZonesLessIntrusive?
(10:39:13 AM) gac410: I have not heard from him in a long time.
(10:39:50 AM) gac410: I suspect Sven, Paul, Arthur, features all need to be parked or otherwise put up for adoption,
(10:41:26 AM) jomo: if i read right - the patch is already included in the http://foswiki.org/Development/MakeZonesLessIntrusive - or it isn't ok?
(10:43:05 AM) gac410: Vladm is looking into changing Foswiki email to use Email::MIME so that the encoding for utf-8 can be better handled.
(10:43:28 AM) gac410: jomo, yes. I added myself to that particular one. even I can cut/paste :D
(10:44:43 AM) gac410: Ah yeah... So I guess unless someone else (michael?) picks it up, I'll get it ready.
(10:48:37 AM) gac410: I added a new FeatureProposal state. "ReadyToMerge"
(10:49:31 AM) gac410: So UnderConstruction says the Item/Feature branch is still being worked on. ReadyToMerge, ... safe to merge into master once it's ready to start accepting features.
(10:50:41 AM) MichaelDaum: okay
(10:51:22 AM) MichaelDaum: guys got a call comming in.
(10:55:02 AM) MichaelDaum is now known as MichaelDaum_
(11:06:22 AM) gac410: Let's wrap it up. Thanks everyone. See you in 2 weeks
(11:06:40 AM) jomo: cu :)
(12:09:59 PM) jomo left the room (quit: Quit: jomo).
(12:19:28 PM) MichaelDaum_ left the room (quit: Quit: quit).
(02:13:34 PM) CDot left the room (quit: Quit: Leaving.).
(03:51:46 PM) JulianLevens left the room (quit: Ping timeout: 272 seconds).
(06:20:12 PM) Lynnwood is now known as Lynnwood_
(07:15:13 PM) Lynnwood_ is now known as Lynnwood
(09:24:15 PM) Lynnwood left the room (quit: Quit: My Mac has gone to sleep. ZZZzzz…).

Topic revision: r1 - 04 Nov 2015, GeorgeClark
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy