Foswiki on GitHub is open for business! Next release meeting: Monday September 1, 1300Z

Item10545: Webnotify for Abc*Xyz? doesn't send out news on change.

Priority: CurrentState: AppliesTo: Component: WaitingFor:
Urgent Closed Extension MailerContrib  
Hello folks, I'd like to receive your help regarding managing a newsletter for the public from within foswiki 1.1.2. I try to create a newsletter topic with a unique name by using 30 Aug 2014 - 02:09 like NewsLetter2011x03x25Topic So Over the time I have several newsletters handy.

I only want to send the latest one.

For publishing I use a button sending this URL: http:///Foswiki-1.1.2/bin/rest/MailerContribPlugin/notify?nochanges=&webs=Private

In WebNotify i entered this entry:
   * PublicUsersGroup: NewsLetter*Topic?
But I can't make it work sending in news mode.

When I do it with WebNotify having this line:
   * PublicUsersGroup: NewsLetter*Topic!
it works. But it sends out the email each time I call notify.

I also observe that the selection of the Web works, but it sends notifications and newsletter by the same call. frown

Any idea?

Regards

Martin Raabe

-- MartinRaabe - 25 Mar 2011

I have tried to recreate your issue on 1.1.3-beta2, and have not been successful.

Processing Sandbox
   * %USERSWEB%.SumNut: NewsLetter*Topic?
        Last notification was at 2011-03-26T01:28:44Z
        Change to NewsLetter123456Topic at 2011-03-26T01:30:50Z. New revision is 1
Sent newletter for Sandbox to gac@private.net
  • Ran tools/mailnotify a 2nd time - debug showed:
Processing Sandbox

Also changed from use to group notification, and it still worked fine. Finally tested publishing using the rest interface:

Processing Sandbox
   * %USERSWEB%.BlergBloog: NewsLetter*Topic?
   * %USERSWEB%.SumNut: NewsLetter*Topic?
   Last notification was at 2011-03-26T01:43:40Z
   Change to NewsLetter123456Topic at 2011-03-26T01:49:45Z. New revision is 3
Sent newletter for Sandbox to gac@private.net
Set-Cookie: FOSWIKISID=5c7e09c7cc83e7e904d1a71b81ec4dc4; domain=private.com; path=/; HttpOnly

So all seems to be working correctly here.

By the way, you can use operands -nonews to do a changes only run and -nochanges to do a news only run. But the "last run date" will still be reset, so I don't believe you can then go back and rerun the news notification.

-- GeorgeClark - 26 Mar 2011

Hello George, thank you for the reproduction trial. Obviously I left out one detail. This is not the command line tool I use. I use the rest interface to be able to call the notification through a button. I'll transfer the topics to the Sandbox and give it a try. Can't say, when this will happen. I hope to do it over the weekend.

-- MartinRaabe - 26 Mar 2011

Hello George,

I now have transferred the topic to the Sandbox:

I added me to WebNotify by the line:
   * Main.MartinRaabe: NewsLetterMRe*Topic?

I'm not an AdminGroup member.

Unfortunately I can't add my Form so I commented it out in the NewsLetterMReList.

When I create a new newsletter in the NewsLetterMReGenerator (like I did with NewsLetterMRe2011x03x26Topic) then I can press the "Publish" Button to call the notify. But since I'm not an AdminGroup member, my result is: Only administrators can do that

I hope this helps you to reproduce my situation here.

THX for helping me out here.

-- MartinRaabe - 26 Mar 2011

I've installed your test topics including the registered Form on my local system so I can fully test the email generation. I've created a newsletter topic, and then pressed the Publish button. Here are the results:

Processing Sandbox
   * %USERSWEB%.MartinRaabe: NewsLetterMRe*Topic?
   Last notification was at 2011-03-26T01:56:03Z
   Change to WebNotify at 2011-03-27T01:49:17Z. New revision is 3
   Change to NewsLetterMRe2011x03x26Topic at 2011-03-27T01:47:21Z. New revision is 1
   Change to WebPreferences at 2011-03-27T01:46:30Z. New revision is 3
   Change to NewsLetterMReList at 2011-03-27T01:44:22Z. New revision is 1
   Change to NewsLetterMReGenerator at 2011-03-27T01:43:33Z. New revision is 1
   Change to NewsLetterMReTemplate at 2011-03-27T01:42:08Z. New revision is 1
   Change to NewsLetterMReForm at 2011-03-27T01:41:04Z. New revision is 1
Sent newletter for Sandbox to martinraabe@mydomain.com
Set-Cookie: FOSWIKISID=56b6e355d1d51d004aea29e3e667b135; domain=mydomain.com; path=/; HttpOnly

Then created another newsletter using the Generator, and hit the publish button again.

Processing Sandbox
   * %USERSWEB%.MartinRaabe: NewsLetterMRe*Topic?
   Last notification was at 2011-03-27T01:49:17Z
   Change to NewsLetterMRe2011x03x26bTopic at 2011-03-27T01:59:42Z. New revision is 1
Sent newletter for Sandbox to martinraabe@mydomain.com
Set-Cookie: FOSWIKISID=56b6e355d1d51d004aea29e3e667b135; domain=mydomain.com; path=/; HttpOnly
It appears to work to me. I've tried it on both 1.1.3 branch and trunk. I also did a checkout to the 1.1.2 commit and it also worked on a 1.1.2 checkout. I'm not sure what else to suggest.

Except I just created another newsletter topic on foswiki.org Sandbox, and it appeared to not send. Now I'm baffled Also switched over to the compatibility logger by adding LogFile names to LocalSite.cfg. ... still works for me.

-- GeorgeClark - 27 Mar 2011

Greg,

thx for reporting.

I'm a bit confused.

Maybe the checked in stuff behaves different than the downloadable release?

What do you suggest me to do to find out? Would the Localsite.cfg file help? Might a plugin disturb here?

-- MartinRaabe - 27 Mar 2011

Crawford, Do you have any ideas? This seems to fail on Foswiki.org - but is working fine on my local test system. It's also failing on Martin's 1.1.2 system. I've checked out 1.1.2 from git, and it still works for me.

-- GeorgeClark - 27 Mar 2011

Is the task waiting for me?

-- MartinRaabe - 29 Mar 2011

No, it's waiting for me.

I have no idea why it would work in one install, but not in another, but bear in mind that the mailer maintains a token file on disc to indicate when the last notification was sent. This can cause confusion sometimes when things appear to "suddenly not work". I tried it on my local system, and it also works as advertised. Without a detailed debug on martin's system, it's impossible to say what is going wrong.

-- CrawfordCurrie - 29 Mar 2011

Martin, The normal cron job that runs emails updates the Web token regardless of whether it sent news or change summaries. The token is stored in working/workareas/MailerContrib/[webname] and contains a timestamp of the last mailnotify run.

  • If you do a force new revision save to the topic, and then publish, ensuring that there is no intervening cron triggered run, does your NewsLetter get published.
  • If you manually edit the MailerContrib/[My web] token and set it to a lower number. (It's a timestamp in Unix epoch seconds) does that allow it to publish.
  • If you remove the MailerContrib/[My web] token for your web, does your newsletter get published? (That should cause all newsletters to be sent!)

If any of these help, then I suspect it's something to do with mailnotify detecting the changes.

-- GeorgeClark - 29 Mar 2011

One more question ... you by any chance haven't modified the default for the following?
Default state of the Minor Changes, Don't Notify (DontNotify) check box in preview. Check box is initially checked if Set DONTNOTIFYCHECKBOX = checked="checked", or unchecked if empty: (can be overwritten by user preferences)

* Set DONTNOTIFYCHECKBOX =

Another thing to look at are the log entries written after you save your newsletter. Does the log entry contain "minor" in the description, which would prevent notification: (emphasized here)

2011-03-29T15:03:28Z info SomeUser save TestTopic3 minor Firefox 127.0.0.1

Also compare the message Last notification was at 2011-03-26T01:56:03Z from your mailnotify log against the timestamps in the log.

-- GeorgeClark - 29 Mar 2011

Hello George, thank you for the ideas. I attached the log of the call of mailnotify from the shell. In between both calls I changed the NewsLetter2011x03x29Topic. The log shows, that it is detecting the change, but doesn't find, that the change is triggering a news email. It is the "?" which doesn't work. If I do the same with a "!" in WebNotify, it works. So it's not about Minor Changes, since the topic has just been created, isn't it. If you like, we can do a netviewer session, where you can take a look over my shoulder and you can get new ideas of what is working here... Se my contact info under my website in MartinRaabe

-- MartinRaabe - 29 Mar 2011

Martin, Looking at the attached log, it did find a change!
   * %USERSWEB%.WikiGuest: NewsLetter*Topic?
        Last notification was at 2011-03-29T07:32:47Z
        Change to NewsLetter2011x03x29Topic at 2011-03-29T12:34:12Z. New revision is 1

But for some reason is not sending the email. Do you know, are you using Sendmail or Net::SMTP for sending mail. If the latter, could you go one step further and enable $Foswiki::cfg{SMTP}{Debug} = 1; in your LocalSite.cfg (It's an expert parameter under the Mail tab). and then run the test again. That will trace all of the e-mail handling as well. You should sanitize the output - it might include passwords, and url's you don't want revealed.

If you can join the #foswiki IRC channel, we could work a bit easier on this.

Note that WikiGuest user is ignored for email. I added the WikiGuest line to my WebNotify for the NewsLetter topic and it's silently ignored.

-- GeorgeClark - 29 Mar 2011

To explain the lack of WikiGuest email, WikiGuest is handled in the Base user mapping, and is available regardless of any other installed mappers. This mapper does not supply an e-mail address for the guest user regardless of the configuration for the user.

-- GeorgeClark - 29 Mar 2011

I've managed to recreate the issue - finding several things wrong:

  • The "nochanges" and "nonews" parameters need to have a value passed in the rest interface. nochanges=1, where as the documented cli interface is to pass -nochanges
  • The script appears to optimize the wildcards and throws away news subscriptions if a more generic wildcard matches. So Coding either on separate lines or on a single statement Main.MyUser: * News*Letter? - the Newsletter subscription is discarded.
  • The nochanges option appears to also disable the newsletter. nonews prevents the newsletter and sends changes. nochanges doesn't send either.

There is still something more going on that I don't understand, even accounting for these two things, there are still times where the news is not sent.

-- GeorgeClark - 29 Mar 2011

MailerContrib::WebNotify::processChange appears to either add an email address to the changesSet or allSet of email addresses. So if an email is going to get a change notification it will never get a newsletter for the same topic. I've added debug print statements and see that the email address cannot be added to both the allSet or the changeSet - but don't quite see why/how to fix this. But definitely broken!

I think the issue is down in the optimization. Some optimizing appears to not consider the type of subscription. So * supersedes Blah?. I have no idea how to fix this ... Crawford, any ideas.

-- GeorgeClark - 29 Mar 2011

Crawford, I have a fix maybe that Martin is testing. It's a stab in the dark on some code I don't really understand. Please review.

diff --git a/MailerContrib/lib/Foswiki/Contrib/MailerContrib/Subscription.pm b/MailerContrib/lib/Foswiki/Contrib/MailerContrib/Subscription.pm
index f1ec18a..93100a7 100644
--- a/MailerContrib/lib/Foswiki/Contrib/MailerContrib/Subscription.pm
+++ b/MailerContrib/lib/Foswiki/Contrib/MailerContrib/Subscription.pm
@@ -119,16 +119,27 @@ specified by another subscription. Thus:
 sub covers {
     my ( $this, $tother, $db ) = @_;
 
-    #* should win always.
-    return 1 if ( $this->{topics} eq '*' );
+    print "COVERS - checking $this->{topics} $this->{options} against $tother->{topics} $tother->{options} \n";
 
     # Does the mode cover the other subscription?
-    return 0
-      unless (
-        ( $this->{options} & $tother->{options} ) == $tother->{options} );
+    unless (
+      ( $this->{options} & $tother->{options} ) == $tother->{options} ) {
+        print " .. RETURN 0 - options mismatch\n";
+        return 0;
+    }
+
+    # * should win always unless 
+    if ( $this->{topics} eq '*'
+      ) {
+      print " .. RETURN 1 - covered\n";
+      return 1;
+      }
 
     # do they match without taking into account the depth?
-    return 0 unless ( $this->matches( $tother->{topics}, undef, 0 ) );
+    unless ( $this->matches( $tother->{topics}, undef, 0 ) ) {
+        print " .. RETURN 0 - topic regex mismatch\n";
+        return 0;
+    }
 
     # if we have a depth and they don't, that's already catered for
     # by the matches test above
@@ -138,7 +149,12 @@ sub covers {
 
     # if we have a depth and they have a depth, then there is coverage
     # if our depth is >= their depth
-    return 0 unless ( $this->{depth} >= $tother->{depth} );
+    unless ( $this->{depth} >= $tother->{depth} ) {
+        print " .. RETURN 0 - depth exceeded\n";
+        return 0;
+    }
+
+    print " .. RETURN 1 - default\n";
 
     return 1;
 }
x

Bumping to urgent to get visibility

-- GeorgeClark - 29 Mar 2011

The NewsLetter*Topic! is now detected as a newsletter request. But still not working is the NewsLetter*Topic?.
Processing Private
   * Main.OttoPublic: NewsLetter*Topic?
	Last notification was at 2011-03-29T21:34:53Z
	Change to NewsLetter2011x03x29Topic at 2011-03-29T21:37:55Z. New revision is 13
Set-Cookie: FOSWIKISID=178d3e3c06573b45985b5a086b9ffad3; path=/; HttpOnly

-- MartinRaabe - 29 Mar 2011

Only a partial fix - Solves the issue with * deleting all other subscriptions, but still issues.

-- GeorgeClark - 29 Mar 2011

Yes, definitively NewsLetter*Topic? doesn't work, NewsLetter*Topic! works.

Good progress George!

-- MartinRaabe - 29 Mar 2011

At this point with the latest change to Subscriptions.pm, it is working for me in all cases. I need a detailed example of what else fails.

-- GeorgeClark - 29 Mar 2011

So the changes also work for me in the first run. I'll keep it running and test it further.

-- MartinRaabe - 30 Mar 2011

Crawford. I believe that this change is safe for 1.1.3. The unit tests still pass with these changes applied. Other than lots of debug prints, the primary change is to move the test for mis-matched options ahead of the test for the * wildcard. Otherwise * trumps and removes all of the Newsletter subscriptions.

-- GeorgeClark - 30 Mar 2011

I guess we should have this checked in and tested in a release candidate 1

-- KennethLavrsen - 04 Apr 2011

Looks like the error crept in just days before the big bust-up with (tm)wiki; probably got lost in the wash.

The problem arises because the optimisation code, which looks for expressions that "cover" other notifications, were extended to special-case '*' on it's own. Unfortunately the special case didn't account for the different notification modes.

-- CrawfordCurrie - 05 Apr 2011

 

ItemTemplate edit

Summary Webnotify for Abc*Xyz? doesn't send out news on change.
ReportedBy MartinRaabe
Codebase 1.1.2
SVN Range
AppliesTo Extension
Component MailerContrib
Priority Urgent
CurrentState Closed
WaitingFor
Checkins Foswikirev:11306 Foswikirev:11308
TargetRelease patch
ReleasedIn 1.1.3
Topic attachments
I Attachment Action Size Date Who Comment
mailnotify.loglog mailnotify.log manage 1.0 K 29 Mar 2011 - 16:36 MartinRaabe Log of calling mailnotify from the shell
Topic revision: r26 - 16 Apr 2011, KennethLavrsen
 
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. see CopyrightStatement. Creative Commons License