Release Meeting: 27 July 2015

Details

1. 2.0.1 blocker review

  • 13525 (Waiting for Feedback): Store reads the same file repeatedly from disk
  • NEW 13537 (No Action Required): Cache failures on Foswiki.org

These two items are deferred 2.0.2. Considerable discussion of Configure experiences, and new blockers. All will defer to 2.0.2, as the bulk_copy.pl fixes are so critical.

  • 13561 (Closed): PERL type configuation settings lose their formatting when LocalSite.cfg is saved. Uglify happening to perl structures in the configuration
  • 13560 (Closed): configure does not set initial values when extensions are installed with an external package manager. Install by way of unzip or git has issues - config not correctly populated. Andreli and MichaelDaum are very concerned. Installing from outside of configure seems to be a common approach.

Jast also brought up issues reported in a catch-all task - 13552 (Confirmed): UX for web-based extensions installer is confusing/error-prone, but it really needs to be split up into fixable chunks.

Jast reported that the latest bulk_copy is still skipping files. It turns out we had forgotten that the Foswiki store will not list files that are "hidden" by way of a prefix scan. Any file beginning with [_.*] will be omitted by store, so bulk_copy can't see them. Jast tried a quick patch to see if he removed the filter, they would be copied. bulk_copy indeed reported success - files were copied. But when checking, they were not there. So there are problems here.

In the meantime for 2.0.1, George flagged this limitation in the Release Notes and Upgrade Guide.

More discussion on Item13525, and the performance issues. Michael asks others to please benchmark.

Also this covers a another apparent issue "3rd finding: Foswiki::Store::Rcs::Handler::getRevision($version) ignores the $version argument" ... we really need Crawford to review this - is it a bug, or a documentation issue. Do we have a unit test to demonstrate an issue with this?

2. 2.1.0 Development process review

  • Is there agreement that we proceed with a branch / merge?
No. we have enough 2.0.x issues to stay in bugfix mode. Also branches are really not ready to merge yet

If we are ready to start merging 2.1 features:
  • Need to branch Release02x00 On hold
  • JanKrueger add new branch into Weblate Yes when we are ready
  • ...

3. Proposal review

  • Need to update the backlog of proposals.
    • 21 Proposals stuck in "14-day". We should review and accept them. They've met the 14 day threshold ... some by years.
    • 8 with concerns. We should review and "vote".
    • 30 "Accepted" proposals need review
    • 12 "In process" need some disposition.

Possible proposals for 2.1

See also ReleasePlan

The remainder of these didn't get any discussion.

Next meeting - - Monday Aug 10, 2015 1300Z — ReleaseMeeting02x01_20150810

IRC Log

(09:00:16 AM) jomo entered the room.
(09:00:35 AM) MichaelDaum: hi everybody
(09:00:45 AM) gac410: Hi everyone ...    Welcome to Foswiki 2.1 kick-off
(09:00:48 AM) jomo: hi
(09:01:07 AM) MichaelDaum: gac410, how about discussing 2.0.1 first
(09:01:23 AM) gac410: yes indeed.  First on the agenda. 
(09:02:01 AM) gac410: I've got the RC built  but have not installed it yet on foswiki.org.    There are 168 file changes between 2.0.0 and 2.0.1
(09:02:23 AM) MichaelDaum: in sum, what are the issues fixed?
(09:02:50 AM) gac410: There are 20 fixes and 3 minor enhancements.    
(09:03:05 AM) gac410: Most important are the bulk_copy fixes  
(09:03:05 AM) MichaelDaum: does it fix missing {Module} entries in LSC?
(09:03:16 AM) andreli entered the room.
(09:03:23 AM) gac410: No task on that one.   I've never encountered that unless you pseudo-install
(09:03:50 AM) MichaelDaum: Ive installed a fresh 2.0.0 and ran thru normal configure. 
(09:04:00 AM) MichaelDaum: I then installed more plugins using unzip on the local filesys
(09:04:10 AM) MichaelDaum: then ran configure to enable those plugins
(09:04:17 AM) MichaelDaum: which all went fine ... alas
(09:04:18 AM) gac410: Oh.. unzip....     do we still support htat.
(09:04:35 AM) MichaelDaum: the error.log listed lots of {Module} entries being guessed
(09:04:38 AM) gac410: I did change core to warn but operate ... uses a sane default if the {module} is missing
(09:04:48 AM) jast: side note, I went through web-based extension install again today... installed a plugin that I knew had no spec issues, and the install went through and told me to hit 'save' to activate the defaults. save wasn't visible, though.
(09:05:07 AM) Lavr entered the room.
(09:05:14 AM) MichaelDaum: ... which is fine. I just removed the warning msg for now as there is no penalty in not having them in LSC
(09:05:23 AM) gac410: If we don't get tasks opened,  things will never get fixed. 
(09:05:26 AM) Lavr: Lurking while being on a conference call ;-)
(09:05:37 AM) gac410: okay   Welcome back Kenneth !
(09:06:00 AM) jast: I did open a task :-)
(09:06:07 AM) MichaelDaum: also: lots of plugin defaults werent propagated from their Config.spec into LSC ... with lots of error reports all over
(09:06:16 AM) jast: it's the one about install UX in general, I believe it covers that
(09:06:33 AM) MichaelDaum: jast, this one needs splitting up
(09:06:34 AM) gac410: Michael  if you don't use the installer,  the config will be a mess.
(09:06:56 AM) MichaelDaum: gac410, when did we decide not to support unzip?
(09:07:10 AM) MichaelDaum: I would have cried out loud
(09:07:27 AM) andreli: so would I
(09:07:46 AM) andreli: still using unzip to install plugins
(09:07:47 AM) MichaelDaum: my clients get their update via git push to their repo
(09:07:51 AM) gac410: Probably an oversight,  but this is the first time anyone has brought it up.  It's been that way since the new configure was built back before first beta
(09:08:15 AM) MichaelDaum: gac410, thats why it is called 2.0-oh-oh
(09:08:16 AM) jast: so we need a way to apply defaults
(09:08:36 AM) JulianLevens entered the room.
(09:09:01 AM) jast: probably not big deal implementing that, but the code is a bit difficult to follow at times
(09:09:11 AM) gac410: Michael, That's also why we run alpha, beta and RC's    ...   For defaults Run installer.   That's the only way currently.   we need a blocker for 2.0.2 
(09:09:26 AM) MichaelDaum: okay
(09:09:33 AM) MichaelDaum: I am all for rapid patch releases!
(09:09:34 AM) gac410: The bulk_copy is too important to get out IMO  ... 
(09:09:39 AM) MichaelDaum: agreed
(09:09:51 AM) jast: yeah, we ran with the current bulk_copy today and it skipped only part of the data this time ;-)
(09:10:11 AM) gac410: Still skips data ???   yikes.    
(09:10:42 AM) jast: it's on our list for investigating
(09:10:47 AM) gac410: jast what were the conditions for that.    IMO that's a blocker.
(09:11:32 AM) jast: don't know exactly, another guy in the office did it
(09:11:49 AM) gac410: for those using unzip,   I think that running    tools/configure -save     will merge in the Config.spec files.    But needs checking
(09:11:55 AM) jast: at first glance, nothing extra special
(09:12:33 AM) jast: the copy in distro master is up-to-date, right? (from 6 days ago)
(09:12:45 AM) gac410: bulk_copy will definitely skip.    System web.    Any subdirectories of an attachment dir,    and any attachments hidden behind corrupted meta
(09:13:08 AM) gac410: master is up to date.   I build 2.0.1-RC1 from it last night.   But not loaded anywhere yet.
(09:13:33 AM) jast: I believe the only special thing about the attachments was that they had the H flag
(09:13:50 AM) jast: not necessarily all of the ones that got skipped, I only looked at this very briefly
(09:13:58 AM) gac410: They definitely should have been copied. ... the tool even logs that it is copying a hidden attachment.
(09:14:18 AM) jast: according to the guy who ran it, the script claimed to have copied files that it didn't actually copy
(09:14:29 AM) gac410: oh.. .that's not good.
(09:15:31 AM) gac410: darn.   IMO that's enough to block 2.0.1 ...  please open an urgent task.   
(09:18:11 AM) gac410: The current contents of 2.0.1 is http://trunk.foswiki.org/System/ReleaseNotes02x00#Foswiki_Release_2.0.1_Details
(09:18:23 AM) gac410: Okay ... anything else regarding 2.0.1    
(09:19:00 AM) gac410: And also someone please open a task about installing from zipfiles.   thanks.
(09:20:26 AM) gac410: Anything else  2.0.1 ...  going  .. going    
(09:20:31 AM) MichaelDaum: hm
(09:20:43 AM) MichaelDaum: not much feedback on http://foswiki.org/Tasks/Item13525
(09:21:01 AM) MichaelDaum: I am probably the only one to run nyt profile on foswiki ?
(09:21:53 AM) MichaelDaum: theres a script attached to the item to ease the process.
(09:21:59 AM) andreli: It's a while back we ran nyt.
(09:22:09 AM) MichaelDaum: please do run it and see what happens on your platforms
(09:22:10 AM) jast: we're running nyt on our configuration, but haven't managed to make it play nice with 2.x so far
(09:22:13 AM) gac410: I have not done it.  I did make one change for 2.0.1 that should have improved TipsContrib, by restricting the searches.
(09:22:28 AM) andreli: We concluded, the performance is mor CPU limited then IO
(09:22:49 AM) jomo: andreli: +1
(09:22:56 AM) gac410: Just general feel,  2.0 seems much "crisper" than 1.1. was
(09:23:07 AM) MichaelDaum: true.
(09:23:12 AM) jast: we managed to shave off >95% of overhead seen on a wiki with huge LDAP by patching LdapContrib ;)
(09:23:48 AM) MichaelDaum: the question on Item13525 is not about LDAP
(09:23:53 AM) MichaelDaum: it is about System.WebHome
(09:23:58 AM) MichaelDaum: what does it take to render it
(09:24:01 AM) jast: I know, it was meant as a side note
(09:24:26 AM) andreli: jast: Is your LdapContrib public?
(09:24:27 AM) gac410: The TipsContrib had a rather poor search.   It should be better now.
(09:24:39 AM) gac410: Used on System.Webhomme
(09:24:41 AM) MichaelDaum: it should be removed from System.WebHome
(09:24:46 AM) MichaelDaum: ^Tips
(09:25:13 AM) jast: andreli: it is, but we removed some features to make it faster
(09:25:27 AM) MichaelDaum: EARCH is the no.1 resource drain on System.WebHome ...
(09:25:32 AM) MichaelDaum:  % SEARCH is the no.1 resource drain on System.WebHome ...
(09:25:40 AM) jast: SEARCH almost always is the #1 resource drain :)
(09:26:04 AM) gac410: And the search was particularly bad.   Un-anchored *TipTopic* search across System and Main webs.
(09:26:17 AM) MichaelDaum: 2nd finding: readFile("foobar.txt") was called repeatedly for no good reason
(09:26:46 AM) MichaelDaum: 3rd finding: Foswiki::Store::Rcs::Handler::getRevision($version) ignores the $version argument
(09:27:34 AM) gac410: That sounds like 3 tasks.     #2 is a feature for 2.1 ... a new low level cache.    #3 ... is it a bug.   Needs a unit test.  what are side effects?
(09:27:56 AM) andreli: Heavy searches over seldom changing public content: Cron over to Topic with %SEARCH -> write output in a new Topic -> include the new topic
(09:28:04 AM) MichaelDaum: dunno. it seems to be an api+docu error. 
(09:29:12 AM) MichaelDaum: this needs Cdot
(09:29:38 AM) gac410: So searches.  The point of TipsContrib is a randomly changing tip.   caching it rather makes it less useful.   But System.WebHome ... how often is that used?
(09:30:06 AM) MichaelDaum: it is visited by everybody looking up Documentation
(09:30:09 AM) gac410: y.    we need separate tasks for these things.  
(09:30:18 AM) gac410: And tips can be useful. 
(09:30:28 AM) gac410: Though ours are a bit stale  :)
(09:30:37 AM) MichaelDaum: y
(09:31:09 AM) MichaelDaum: well back to the real issue in Item13525
(09:31:30 AM) MichaelDaum: I counted 118 readFile() operations readign System.WebHome
(09:31:33 AM) gac410: I'd say this is another feature request.  Redesign TipsContrib to not be resource intensive.   (Maybe cache complete list in a single topic and random pick from witthin the topic.
(09:31:58 AM) gac410: You've implemented a new feature as a solution.  A new low level cache.   2.1
(09:31:58 AM) MichaelDaum: PatterSkinNavigation.txt opened 18 times approx
(09:32:01 AM) andreli: An other approach for Tips: Cache the complete list and create the randomness in Javascript.
(09:32:44 AM) MichaelDaum: andreli, can you come up with a patch?
(09:32:47 AM) MichaelDaum: good idea
(09:32:50 AM) jast: I'm not sure caching all file reads is correct
(09:33:09 AM) gac410: So to wrap up 2.0.1.    We have a possible blocker.   bulk_copy lies.   jast,   can you focus on getting that documented.    That one will block 2.0.1
(09:33:26 AM) jast: will do
(09:34:18 AM) gac410: caching low-level reads.   Another new feature.   Needs a request, if jast is raising a concern.   Needs discussion.
(09:34:18 AM) MichaelDaum: especially people with their data dir on a remote filesystem will suffer from Item13525
(09:34:55 AM) jast: I haven't thought about it properly, so I'm not formally raising a concern
(09:34:56 AM) gac410: And RcsHandler ignoring version.   Needs a task, and if possible a unit test,   and/or review with CDot.
(09:35:33 AM) jast: just going with my general experience that caching adds just as many problems as it solves ;)
(09:35:41 AM) jomo: jast: +1
(09:35:45 AM) jomo: :)
(09:36:09 AM) gac410: thats why I'm pushing that it needs to be a feature for 2.1,  and not a quick& dirty tweak to 2.0.x
(09:37:11 AM) gac410: Okay...   Other than Jast's possible blocker to 2.0.1    Anything else before we move to 2.1  development process?
(09:38:02 AM) MichaelDaum: http://foswiki.org/Development/PreventRedundantFileReadsInStore
(09:39:00 AM) MichaelDaum: to all: please run NYT yourself before and after applying this patch
(09:39:06 AM) MichaelDaum: add your findings, please
(09:39:36 AM) gac410: Summary.  Needs tasks  for   zipfile installls (bug 2.0.2),  bulk_copy issues (blocker 2.0.1)    rcs api: possible (bug 2.0.2 unless proven urgent) 
(09:39:43 AM) MichaelDaum: I'll keep the item branch in sync with master in the meantime.
(09:39:51 AM) gac410: thanks Michael
(09:41:04 AM) gac410: So for 2.1 ... JulianLevens   Item13135 branch... seems to be stale from .gitignore changes ...   we can drop that one
(09:41:15 AM) JulianLevens: y
(09:42:17 AM) gac410: HTMLtemplates branch.    really needs firming up of spec in http://foswiki.org/Development/ReduceImpactOfCGIDotPMinFoswiki   and a new task.
(09:42:45 AM) MichaelDaum: y
(09:43:01 AM) MichaelDaum: let's try an impl and see where we get with that one
(09:43:49 AM) gac410: Since neither feature branches are ready to merge into 2.1   I'm suggesting we delay doing the branch for 2.0.x maintenance. and stay on master for a bit longer.
(09:44:39 AM) MichaelDaum: yea. no reason create a 2.1 branch next to master atm. 
(09:44:59 AM) MichaelDaum: best would be to only have two active branches at any time
(09:45:17 AM) gac410: andreli: MichaelDaum:   regarding zipfile install.  would it acceptable to run tools/configure -save    as a "merge" of the Spec files,  and if missing, add the {Module} ?
(09:45:18 AM) MichaelDaum: with item and feature branches merged to them
(09:45:33 AM) gac410: Agreed.
(09:46:01 AM) gac410: Jast.    when we do decide to branch out 2.0.x  for patching,  would you be able to add it into Weblate as another branch?
(09:46:32 AM) MichaelDaum: gac410, let's call it file-level installs. unzip or git pull ... whatever is used to materialize a plugin on the filesystem.
(09:46:41 AM) andreli: gac410: fine with me.
(09:46:54 AM) MichaelDaum: best would be to fix configure
(09:47:08 AM) gac410: yes.   fine Michael.    It applies to pseudo-install as well.
(09:47:11 AM) MichaelDaum: it propagated Config.specs to LSC before 2.0
(09:47:30 AM) MichaelDaum: we basically need that behavior back
(09:48:24 AM) gac410: If you "install" while bypassing configure,   I think that running tools/configure fixes it up.   IMO patching stuff in from the GUI is fraught with danger.
(09:48:42 AM) gac410: Every time I touch it,  "bad things happen"  
(09:48:46 AM) gac410: ;)
(09:48:56 AM) MichaelDaum: not really: clicking on "Enable" for a plugin should do required steps.
(09:49:40 AM) jast: gac410: sure
(09:49:49 AM) MichaelDaum: as things are right now, configure ui does not. instead the core warns about missing {Module} entries.
(09:50:05 AM) MichaelDaum: btw an even simpler fix would be to remove the warning message
(09:51:48 AM) gac410: hm.   I can't remember.   It used to just crash.   I changed it to a warning. 
(09:52:02 AM) Lynnwood entered the room.
(09:52:13 AM) MichaelDaum: _that_ would have been a blocker
(09:53:38 AM) MichaelDaum: okay. I ran configure -save.
(09:53:52 AM) jomo: getting the bulk_copy out is imho the priority - the configure issues are fixable by admin - the bulk_copy isn't
(09:53:54 AM) MichaelDaum: now all perl hashes are uglified again
(09:53:56 AM) gac410: Okay..    again everyone.  we need specific tasks for issues.   "Catch all's"  are nice, but get overlooked.
(09:54:09 AM) MichaelDaum: and it did not create missing {Module} entries
(09:54:22 AM) MichaelDaum: so no go
(09:54:33 AM) gac410: MichaelDaum:    Then that's exactly what would happen with gui install.    configure -save runs the same save wizard.
(09:54:56 AM) jast: gui install does stage the {Module} entry
(09:55:14 AM) jast: or at least it claims it does in the installer output
(09:55:18 AM) jast: but I couldn't get it to save that
(09:55:49 AM) gac410: Yes   gui install works fine.    Gui save after a unzip install    is what I'm mentioning.     Save in gui depends upon the DOM knowing that something changed.
(09:56:15 AM) gac410: If the dom doesn't know,  then you won't be able to trigger the Save wizard.     
(09:56:48 AM) gac410: So need more tasks.     Ugly hashes   separate bug from missing module.   
(09:57:49 AM) gac410: Right.  IMO the only blocker for 2.0.1  is bulk_copy.  And it's BADLY broken in 2.0.0,  so that needs some serious attention.    Sure wish CDot was here.
(09:58:25 AM) MichaelDaum: reported as http://foswiki.org/Tasks/Item13560
(09:59:44 AM) gac410: Thanks Michael.   but re:  This especially applies to extensions installed via unzip or git,  ...   really it *only* applies in that case, right?   Hopefully using the install extension wizard is completely correct.
(10:00:10 AM) MichaelDaum: gac410, not sure
(10:00:28 AM) MichaelDaum: what I am sure of is that clicking on "Enable" does not fully enable plugins.
(10:00:47 AM) gac410: I've not seen any instance of install using the wizard, not completely configure it.  
(10:00:49 AM) MichaelDaum: which would include: (1) create a {Module} entry (2) propagate Config.spec
(10:00:52 AM) MichaelDaum: defaults
(10:01:16 AM) gac410: A premise of new configure is   ABSOLUTE.    Changing one setting  should NEVER change a 2nd setting.    Clicking enable may NOT set a module.  
(10:01:19 AM) MichaelDaum: what happend to the issue that congigure uglifies perl hashes in LSC?
(10:02:10 AM) MichaelDaum: this renders {FromTypes} and the like unmaintainable
(10:02:13 AM) gac410: Needs at task.  I have not see that.   I had opened a task about that ages ago and verified that it was fixed. So you are hitting a different path.
(10:02:27 AM) gac410: I agree.   It was a blocker and CDot had fixed it.
(10:02:54 AM) MichaelDaum: Foswiki.spec -> {FormTypes} is nice perl. LSC -> {FormTypes} is uglified
(10:03:12 AM) gac410: the normal path  using the extension_installer,  or the InstallExtension wizard    doesn't corrupt these things I don't believe.
(10:03:26 AM) MichaelDaum: this is {FormTypes} ... a core setting
(10:03:56 AM) gac410: Then his fix got broken.    It was working at one point.
(10:05:02 AM) gac410: It was a difficult fix,  and he changed Spec processing from an eval to a line/by/line interpreter / copy   to fix the issue.
(10:06:16 AM) MichaelDaum: reoported as http://foswiki.org/Tasks/Item13561
(10:06:51 AM) gac410: All I can say is please open a task.   But it won't block 2.0.1 . ...  (Thanks).. if it makes it, great.  But though very annoying,  it's  not on the scale of bulk_copy corrupting your store ... silently no less.
(10:06:59 AM) MichaelDaum: feel free to reasign it to another patch release in case you'd prefver to get 2.0.1 out without fixing these
(10:07:45 AM) gac410: y.    we need bulk_copy fixed.   That's critical.   It breaks store by missing data,  and if jast is right, lying about it no less.
(10:07:47 AM) MichaelDaum: 50 shades of ugrgent.
(10:08:42 AM) gac410: JulianLevens:   Would you have some time to further validate bulk_copy?    
(10:10:59 AM) gac410: okay ... 1 hour into meeting.   Next on agenda was 2.1 feature review.   The bottom line is our proposals system is a mess of stale stuff.   Needs a bit of a cleanup.
(10:11:23 AM) ***MichaelDaum just added a new proposal for 2.1
(10:11:43 AM) gac410: The items in 14 day need to be either voted, rejected or accepted. 
(10:12:27 AM) gac410: JulianLevens:  Your proposal http://foswiki.org/Development/AddBacklinksToQuery    I escalated that a bit and slotted it for 2.1.    
(10:12:29 AM) MichaelDaum: ... and committed developers known to have left the project need to be removed from the proposal
(10:14:02 AM) gac410: The issue is that on i18n systems,  backlinks is totally broken.   So we need a fix for 2.1, be it a new backlinks feature embedded into QUERY   or a new BACKLINKS macro
(10:14:12 AM) JulianLevens: I'll look at that and provide feedback ASAP. I know the principle but it's been a while
(10:14:21 AM) gac410: Thanks JulianLevens
(10:15:20 AM) gac410: I've fixed http://foswiki.org/Development/ReleasePlan ...  it has 3 accepted proposals for 2.1,  3 needing investigation,  and a whole mess of stuff that could be picked up.
(10:15:20 AM) JulianLevens: Hmm, do you have time frame for 2.1?
(10:15:49 AM) gac410: That's something we can discuss. ... not really ye 
(10:15:51 AM) gac410: t
(10:16:07 AM) gac410: afk - back in a moment...
(10:17:21 AM) JulianLevens: Can I ask everyone here to vote on the options within http://foswiki.org/Development/AddConcatOptionToAttrs
(10:19:31 AM) gac410: back
(10:20:05 AM) gac410: will do JulianLevens.  
(10:20:34 AM) ***MichaelDaum clicks, JulianLevens
(10:23:26 AM) MichaelDaum: I still prefer param="string1" + "string2" as well as param="string1" .... param+="string2"
(10:23:50 AM) jast: same here
(10:23:55 AM) MichaelDaum: as this is what most programming languages and i.e. perl have
(10:24:25 AM) JulianLevens: Then please vote accordingly :)
(10:24:43 AM) gac410: so that's a vote for "both" ???  
(10:25:01 AM) JulianLevens: yes that's both
(10:25:15 AM) MichaelDaum: JulianLevens, what exactly is the use case for a -=
(10:25:27 AM) gac410: okay ...  got more confused the more I read and then skipped ahead to the bottom line  :)
(10:25:45 AM) gac410: ah... prepend  vs append
(10:26:32 AM) gac410: hm    indeed why would you need to prepend,  vs just list the values in the right order?
(10:26:48 AM) jast: yes, I find -= confusing, too
(10:26:56 AM) jomo: also
(10:26:58 AM) MichaelDaum: -= for prepend?
(10:27:04 AM) jast: voted
(10:27:07 AM) gac410: That's what it says
(10:27:11 AM) JulianLevens: There are use-cases for prepend but they do tend to be contrived
(10:27:27 AM) MichaelDaum: woops my comment has been deleted
(10:27:52 AM) jast: and I'd like to note that the voting table took a while to understand :)
(10:28:14 AM) JulianLevens: The implementation allows easy addition of new operators; so prepend could be added later very easily
(10:28:26 AM) gac410: lets use comment plugin and then summarize into table.      Otherwise stepping on each other.
(10:28:30 AM) jast: oh, I've figured out why bulk_copy skipped a lot of attachments here
(10:28:47 AM) gac410: jast  wonderful   Is it bad news?
(10:28:50 AM) jast: the attachments had a leading '_', and the RCS iterators in 1.1.x (not sure about 2.x) skip filenames starting with _
(10:29:06 AM) gac410: Yes   thats true.    _ is defined as a hidden attachment iirc.
(10:29:26 AM) jast: but there's already a FILEATTACHMENT attr for that
(10:29:40 AM) jast: and you can't tell the store to not skip over _ :(
(10:29:43 AM) gac410: hidden from store,  vs hidden from user
(10:30:07 AM) jast: so I can upload _foo via the store but then not find it...
(10:30:07 AM) gac410: How did they get there?    Manual upload? 
(10:30:17 AM) jast: plugin upload using store API
(10:30:51 AM) jast: Func API, in fact
(10:31:03 AM) gac410: hm   yeah that sounds like a bug then.     I think historically     _blah was used to hide webs     like _default.     
(10:31:10 AM) gac410: but not sure about attachments
(10:31:13 AM) jast: it's specifically in the getAttachmentList function
(10:31:57 AM) gac410: MichaelDaum:  Lavr: ... any old timer info about why RcsStore  hides filenames with leading underscore?
(10:31:58 AM) jast: at least it's not an issue in bulk_copy itself
(10:32:11 AM) gac410: y.   And cannnot be fixed.    
(10:32:15 AM) MichaelDaum: gac410, my 2cent: OMG
(10:32:34 AM) gac410: easily.  ...    as it would need to patch 1.1.x source store.
(10:32:48 AM) jast: 2.x has the same behaviour
(10:33:11 AM) MichaelDaum: never knew about _underscore_attachments being treated special
(10:33:59 AM) jast: seems like the iterator is the only place that happens
(10:34:19 AM) jast: FILEATTACHMENT entries with _ prefix in the name are not skipped
(10:34:24 AM) ***MichaelDaum pictures Foswiki becoming an archeological artifact of research sooner than later
(10:35:23 AM) gac410: skips any files named with dot, asterisk or underscore prefix.       grep { !/^[.*_]/ && !/,v$/ && -f "$ed/$_" } readdir($dh);
(10:35:48 AM) jast: so how do we address this? conceivably bulk_copy could be rewritten to combine info from store iterator *and* FILEATTACHMENT entries
(10:35:59 AM) jast: but it's probably not going to be trivial
(10:37:42 AM) gac410: Not fixable in bulk copy.   It is by design using the Store API.     Better would be to patch the source store.
(10:38:39 AM) gac410: bulk_copy is intended to handle DB based stores, etc.   fully portable.    
(10:39:00 AM) gac410: If it starts digging into the file system that all falls apart.    
(10:39:51 AM) jast: sure, that's why I suggested something different ;)
(10:40:21 AM) jast: but it's still not really a change worth making for this, probably
(10:40:31 AM) jast: so, perhaps we should just document this?
(10:41:23 AM) gac410: yes.   Document for sure.    And maybe include a suggested patch.    And maybe a task consider this hidden-from-list  a possible 2.0 bug.
(10:42:31 AM) gac410: PlainFileStore behaves the exact same way:      grep { !/^[.*_]/ && !/,pfv$/ && ( -f "$ed/$_" ) } readdir($dh);
(10:43:12 AM) gac410: the "dot" i understand.  Don't want to reveal possible .htaccess files as attachments.   
(10:43:24 AM) gac410: I don't understand the asterisk or underscore restriction
(10:44:26 AM) JulianLevens: neither do I and Store.pm does not document this behaviour for eachAttachment
(10:44:34 AM) gac410: But imo this is still possibly a nasty inconsistency.   I believe PlainFile is much more tightly coupled to the attachment %META:FILE...  
(10:48:18 AM) JulianLevens: My 1st thought is to add include and exclude regex parms to eachAttachment with defaults as-is
(10:49:13 AM) jast: we're calling this through 2-3 indirections
(10:49:35 AM) JulianLevens: bulk_copy could pass more inclusive values. Still like to understand why * & _ are excluded though
(10:49:36 AM) jast: so that's a lot of changes, and probably not urgent enough to do on 01x01
(10:50:00 AM) gac410: y.   This is a mess.      And really needs a 1.1.9 patch to make a conversion work.
(10:50:34 AM) ***MichaelDaum backing off ... still listening but ...
(10:50:37 AM) MichaelDaum is now known as MichaelDaum_
(10:50:38 AM) JulianLevens: Which is the achiles heel of bulk_copy
(10:51:56 AM) gac410: Same code is in twiki.   Omits _, . and * prefixes
(10:52:13 AM) gac410: So this dates way back.
(10:53:04 AM) JulianLevens: bulk_copy should use latest code with all appropriate fixes in place - or we'll be forever applying back patches
(10:53:31 AM) gac410: Okay everyone.   Lets wrap this up.    ...  it sounds like 2.0.1 can go ahead with a very strong warning about bulk_copy and hidden files.   
(10:54:00 AM) gac410: with suggestion of a "find" command to find them  and a one-off patch to copy them
(10:55:25 AM) gac410: And for the next meeting on the 10th.   lets plan a feature review for 2.1.    I will unfortunately have to either bow out early,  or suggest moving to Tuesday.  I have a 10am Dr's  appt.   
(10:56:48 AM) gac410: We clearly have some discussion points to hash out with CDot  on configure and store.
(10:57:16 AM) gac410: Any other last minute.   Or declare the meeting closed?   
(10:57:50 AM) jomo: imho the restriction reason for the * (asterix) is - the windows doesn't allows asterix in the filenames. And probably we don't want have attachments what arent downloadable to ANY OS....
(10:58:42 AM) jomo: just guessing...
(10:58:49 AM) gac410: okay,,  but that ought to be handled in NameFilter    but good guess for sure.
(10:58:55 AM) jast: there's a whole number of other characters that aren't allowed on windows
(10:59:01 AM) jast: those are not excluded in this list
(10:59:06 AM) jomo: < > : " / \ | ? *
(10:59:39 AM) gac410: the underscore   used to be the documented way to hide attachments.   ChartPlugin  caches generated graphics using _ChartPlugin as the prefix
(11:00:46 AM) andreli left the room (quit: Ping timeout: 240 seconds).
(11:02:45 AM) andreli entered the room.
(11:03:41 AM) gac410: yeah it's a twiki historical thing.   For ex.  in the docs: 
(11:04:39 AM) gac410: Prefix the filename with an underscore (the leading underscore avoids a name clash with files attached to the same topic)
(11:05:02 AM) jomo: IIRC the ImagePlugin uses the _filename for the cached (generated) thumbnails too...
(11:05:06 AM) gac410: It's describing how to handle generated cached type files.  
(11:05:13 AM) jomo: :)
(11:06:02 AM) gac410: In theory cached files should be relatively safe to omit,  as they get recreated on view.   But if it's (ab)used for other stuff. ... oops
(11:07:33 AM) gac410: DirectedGraphPlugin ... same thing.
(11:08:08 AM) gac410: well.  old versions.  Newer code doesn't do that.
(11:08:10 AM) jomo: main problem - anyone could upload a file _something - and it isn;t a cached data. so two possibilites: 1.) deny allow upload such files and use _file for the temporary-cached files or 2.) allow underscore on the filenames, but need another mechanism to detect the temporary-caches files...
(11:09:01 AM) gac410: With the move away from file based store,   web accessible cache needs to eventually end up somewhere outside of the Store managed structure.
(11:11:00 AM) gac410: DojoToolkitContrib .. ships lots of _ prefixed files too.
(11:12:51 AM) gac410: jast,   in your testing,  if you could see what happens if the _ is removed from the getAttachmentList filter... that would be helpful
(11:13:03 AM) gac410: as a temporary patch to 1.1.9 source store.
(11:16:25 AM) jast: I tested that... the attachments get copied
(11:25:43 AM) gac410: okay great.   So we can include a suggested patch.    I suppose another option is to monkey-patch the old version  :(
(11:26:21 AM) gac410: jast ... did they end up polluting the META on the target system?
(11:29:40 AM) gac410: though that will get complex.  Multiple versions to patch :(  source could be 1.1 or 2.0   RcsLite, RcsWrap or PlainFile   
(11:30:43 AM) jast: RcsLite and RcsWrap share the same code for this
(11:30:52 AM) jast: in 1.1.x, Foswiki::Store::VC::Handler
(11:31:00 AM) gac410: okay that's good 
(11:31:09 AM) jast: same in 2.0, only it's Foswiki::Store::RCS::Handler
(11:31:24 AM) gac410: Could also be source from PlainFile 
(11:31:30 AM) jast: right
(11:32:25 AM) jast: as for "polluting the META"... there were META entries in the original topics, so "polluting" isn't exactly the right word
(11:33:34 AM) jast: in fact it seems to be copying the embedded store form with no changes whatsoever
(11:33:56 AM) gac410: okay that's fine.   I guess it will depend on the original file source.   Attached with API,  or external to the store.
(11:34:03 AM) gac410: good.
(11:37:07 AM) jast: oh, dang
(11:37:09 AM) jast: correction
(11:37:19 AM) jast: bulk_copy claims the files got copied but they're not there
(11:37:48 AM) gac410: ug.  that's even worse.
(11:38:20 AM) jast: looking into this
(11:38:34 AM) gac410: thanks.
(11:38:38 AM) andreli left the room (quit: Remote host closed the connection).
(11:39:38 AM) gac410: Also by not copying files like .htaccess,  sites that might have applied custom apache access will have a miess.
(11:39:41 AM) gac410: mess
(11:50:56 AM) jast: I can't figure this out right now
(11:51:13 AM) jast: and my mind is pretty much gone... I'll look again tomorrow
(11:51:18 AM) gac410: In the release notes, I'm adding:  +<div class="foswikiHelp foswikiAlert">%X% *Caution:*  There are known limitations to  =bulk_copy.pl=.  See the Foswiki:System.UpgradeGuide for details. Hidden files  (filenames with underscore (_) or dot (.) prefix) will not be copied.</div>
(11:51:29 AM) gac410: Okay thanks.   I'll shoot an email to CDot as well.
(11:51:48 AM) gac410: And I'm adding detailed information to the UpgradeGuide as well.
 

Topic revision: r6 - 10 Aug 2015, GeorgeClark - This page was cached on 11 Sep 2019 - 08:32.

The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy