You are here: Foswiki>Tasks Web>Item14105 (25 Sep 2016, GeorgeClark)Edit Attach

Item14105: Could not create statistics for all webs using cron

pencil
Priority: Normal
Current State: Waiting for Feedback
Released In: n/a
Target Release: n/a
Applies To: Engine
Component: FoswikiUIStatistics
Branches:
Reported By: Ehj52N
Waiting For: Ehj52N
Last Change By: GeorgeClark

Metadata

  • Foswiki: 2.1.2
  • Perl: 5.14.2
  • Apache: 2.2.22
  • RCS: 5.8.1

Problem

I want to update the statistics on a regular basis using cron. I executed the following call for testing on the command line:
user@host:/srv/foswiki/bin$ sudo su www-data -c './statistics -autocreate 1 -subwebs 1 -user (UserName|WikiGuest|AdminUser)'
with and without -user.

  • I get the following error message for WikiGuest|AdminUser:
    [Mon Jul  4 13:38:56 2016] statistics: Use of uninitialized value in string eq at /srv/foswiki/lib/Foswiki/User
    s/TopicUserMapping.pm line 574, <DATA> line 558.
    Foswiki: Create Usage Statistics
    !Do not interrupt this script!
    (Please wait until page download has finished)
    * Statistics for Jul 2016
    * Reporting on Web1 web
      - view: 1266, save: 1, upload: 0
      - top view: 370 Topic1
      - top contributor:   1 Main.UserName2
    !  - ERROR: no permission to CHANGE statistics topic in Web1
    * Reporting on Web1/SubWeb web
      - view: 0, save: 0, upload: 0
    !  - ERROR: no permission to CHANGE statistics topic in Web1/SubWeb
    [...]
    End creating usage statistics
  • Using an user from the admin group results in the following:
    Foswiki: Create Usage Statistics
    !Do not interrupt this script!
    (Please wait until page download has finished)
    * Statistics for Jul 2016
    * Reporting on WebA web
      - view: 618, save: 0, upload: 0
      - top view:  20 WebHome
      - Topic WebStatistics updated
    * Reporting on WebB web
      - view: 22, save: 0, upload: 0
      - top view:   9 WebHome
      - Topic WebStatistics updated
    * Reporting on WebC web
      - view: 908, save: 0, upload: 0
      - top view: 179 TopicC
    [Mon Jul  4 13:41:46 2016] statistics: Automatic recovery: breaking lock for /srv/foswiki/data/WebC/WebStatistics.txt at /srv/foswiki/lib/Foswiki/Store/Rcs/RcsWrapHandler.pm line 526.
    RCS: rcs  -l %FILENAME|F% failed:  at /srv/foswiki/lib/Foswiki/Store/Rcs/RcsWrapHandler.pm line 539.
    Foswiki detected an internal error - please check your Foswiki logs and webserver logs for more information.
    
    RCS: rcs  -l %FILENAME|F% failed: 
    The according error.log entry is:
    | 2016-07-04T13:41:46+00:00 warning | RCS: rcs  -l %FILENAME&vbar;F% failed:  at /srv/foswiki/lib/Foswiki/Store/Rcs/RcsWrapHandler.pm line 539.
    When using rlog to look at the history of the file, I get the following error:
    rlog: ../data/Documentation/WebStatistics.txt,v:4768: missing author
    rlog aborted
    The according line in the file is the following:
     4766 next    1.382;
     4767 1.382
     4768 date    70.01.01.00.00.00;      author ;        state Exp;
     4769 branches;
     4770 next    1.381;
     4771 1.381
    It seems that the date and author values are somehow broken. Change the values of the broken history file to something in between the other revisions, e.g.
     5958 1.383
     5959 date    2007.01.29.21.03.42;    author TWikiGuest;      state Exp;
     5960 branches;
     5961 next    1.382;
     5962 
     5963 1.382
     5964 date    2007.01.29.16.00.00;    author TWikiGuest;      state Exp;
     5965 branches;
     5966 next    1.381;
     5967 
     5968 1.381
     5969 date    2007.01.29.15.52.18;    author TWikiGuest;      state Exp;
     5970 branches;
     5971 next    1.380;
    Now, I can build statistics for all webs using my user account. When using something useful like WikiGuest, I get again the
    ERROR: no permission to CHANGE statistics topic in
    messages.

What parameters should I use to get this running without using my user account as -user parameter?

Comments

-- Main.Ehj52N - 18 Aug 2016 - 09:10

I now get the following error
ERROR: no permission to CHANGE statistics topic in
for my user, too, when executing the following command line using cron
((echo "\nStart: $(date)\n\n" && cd /srv/foswiki/bin && ./statistics -u MyUser -subwebs 1)  >> /tmp/foswiki-statistics.log)
. The output in the log:
Start: Thu Aug 18 06:42:01 UTC 2016


Foswiki: Create Usage Statistics
!Do not interrupt this script!
(Please wait until page download has finished)
* Statistics for Aug 2016
* Reporting on ExampleWeb web
  - view: 978, save: 1, upload: 0
  - top view:  78 WebHome
  - top contributor:   1 Main.MyUser
!  - ERROR: no permission to CHANGE statistics topic in ExampleWeb
End creating usage statistics

I can create the statistics manually for all webs when using the web interface from /System/SiteToolStatistics. Hence, I am confused because my user seems to have the right permissions from web but not from command line.

-- Main.Ehj52N

Where did you find the example of using a string like (UserName|WikiGuest|AdminUser) for the -user argument. You should pick one. Typically Statistics should be run as the admin user, which ensures that they have full access to the WebStatistics topics. Allowing guests or other users to update statistics is rather unusual. I'm pretty sure the default user for cli scripts is admin, so you should not need the -user parameter.

As far as the corrupted history, I'm guessing that the very unusual (UserName|WikiGuest|AdminUser) user string somehow broke the topic history. Probably the easiest thing to do is to just remove the RCS file for your WebStatistics topic. We recommend doing that occasionally anyway. RCS processing with 1000's of revisions is extremely slow.

-- GeorgeClark - 18 Aug 2016

I also believe the problem is with your command syntax. To run the statistics script from cli, I'd use simply:
sudo -u www-data ./statistics -autocreate 1 -subwebs 1
And for the cron job syntax, assuming you create the cron job for www-data user, would be (as per System.SiteToolStatistics#Configuring_for_automatic_operation):
0 0 * * * (cd /path/to/bin; ./statistics -subwebs 1 >/dev/null 2>&1)

One problem that may have triggered your errors above is that recent versions of Apache don't actually allow you to su to www-data user, however you can run a command as that user using sudo -u www-data command...

-- LynnwoodBrown - 18 Aug 2016

  • Regarding the -u|user parameter: I was playing around with the command and when being used, the script worked but as I stated before, it broke.
  • Regarding the corrupted history: that issue seems to be fixed, because the statistics can be generated from the WebUI.
  • @ LynnwoodBrown Your first command resulted in the error message I posted in my first comment: !  - ERROR: no permission to CHANGE statistics topic in ExampleWeb
    The permission for the WebStatistics topic are always set to default.
  • What additional information do you need or may help here?

-- Ehj52N - 19 Aug 2016

I am afraid that I really don't have any ideas where to go from here. The Command scripts should always run as the admin user by default, which will always have authorization to read / save all topics. If something is not working in that area and we have been unable to recreate it, we'll have to work on some debugging print statements in the perl code to try to figure out why your shell scripts are not getting admin authority.

-- GeorgeClark - 25 Sep 2016
 

ItemTemplate edit

Summary Could not create statistics for all webs using cron
ReportedBy Ehj52N
Codebase 2.1.2
SVN Range
AppliesTo Engine
Component FoswikiUIStatistics
Priority Normal
CurrentState Waiting for Feedback
WaitingFor Ehj52N
Checkins
TargetRelease n/a
ReleasedIn n/a
CheckinsOnBranches
trunkCheckins
masterCheckins
ItemBranchCheckins
Release02x01Checkins
Release02x00Checkins
Release01x01Checkins
Topic revision: r6 - 25 Sep 2016, GeorgeClark - This page was cached on 14 Jan 2020 - 13:59.

The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy