← Index
NYTProf Performance Profile   « line view »
For ./view
  Run on Fri Jul 31 18:42:36 2015
Reported on Fri Jul 31 18:48:16 2015

Filename/var/www/foswikidev/core/lib/Foswiki/Store/SearchAlgorithms/Forking.pm
StatementsExecuted 203681 statements in 907ms
Subroutines
Calls P F Exclusive
Time
Inclusive
Time
Subroutine
40111.42s48.8sFoswiki::Store::SearchAlgorithms::Forking::::_searchFoswiki::Store::SearchAlgorithms::Forking::_search
401124.1ms49.1sFoswiki::Store::SearchAlgorithms::Forking::::_webQueryFoswiki::Store::SearchAlgorithms::Forking::_webQuery
11123µs31µsFoswiki::Store::SearchAlgorithms::Forking::::newFoswiki::Store::SearchAlgorithms::Forking::new
11116µs29µsFoswiki::Store::SearchAlgorithms::Forking::::BEGIN@13Foswiki::Store::SearchAlgorithms::Forking::BEGIN@13
11111µs16µsFoswiki::Store::SearchAlgorithms::Forking::::BEGIN@14Foswiki::Store::SearchAlgorithms::Forking::BEGIN@14
11110µs38µsFoswiki::Store::SearchAlgorithms::Forking::::BEGIN@15Foswiki::Store::SearchAlgorithms::Forking::BEGIN@15
1119µs40µsFoswiki::Store::SearchAlgorithms::Forking::::BEGIN@34Foswiki::Store::SearchAlgorithms::Forking::BEGIN@34
1115µs5µsFoswiki::Store::SearchAlgorithms::Forking::::BEGIN@17Foswiki::Store::SearchAlgorithms::Forking::BEGIN@17
1115µs5µsFoswiki::Store::SearchAlgorithms::Forking::::BEGIN@36Foswiki::Store::SearchAlgorithms::Forking::BEGIN@36
1114µs4µsFoswiki::Store::SearchAlgorithms::Forking::::BEGIN@21Foswiki::Store::SearchAlgorithms::Forking::BEGIN@21
1114µs4µsFoswiki::Store::SearchAlgorithms::Forking::::BEGIN@30Foswiki::Store::SearchAlgorithms::Forking::BEGIN@30
1114µs4µsFoswiki::Store::SearchAlgorithms::Forking::::BEGIN@20Foswiki::Store::SearchAlgorithms::Forking::BEGIN@20
1114µs4µsFoswiki::Store::SearchAlgorithms::Forking::::BEGIN@29Foswiki::Store::SearchAlgorithms::Forking::BEGIN@29
1114µs4µsFoswiki::Store::SearchAlgorithms::Forking::::BEGIN@25Foswiki::Store::SearchAlgorithms::Forking::BEGIN@25
1113µs3µsFoswiki::Store::SearchAlgorithms::Forking::::BEGIN@24Foswiki::Store::SearchAlgorithms::Forking::BEGIN@24
1113µs3µsFoswiki::Store::SearchAlgorithms::Forking::::BEGIN@22Foswiki::Store::SearchAlgorithms::Forking::BEGIN@22
1113µs3µsFoswiki::Store::SearchAlgorithms::Forking::::BEGIN@31Foswiki::Store::SearchAlgorithms::Forking::BEGIN@31
1113µs3µsFoswiki::Store::SearchAlgorithms::Forking::::BEGIN@26Foswiki::Store::SearchAlgorithms::Forking::BEGIN@26
1113µs3µsFoswiki::Store::SearchAlgorithms::Forking::::BEGIN@27Foswiki::Store::SearchAlgorithms::Forking::BEGIN@27
1113µs3µsFoswiki::Store::SearchAlgorithms::Forking::::BEGIN@23Foswiki::Store::SearchAlgorithms::Forking::BEGIN@23
1113µs3µsFoswiki::Store::SearchAlgorithms::Forking::::BEGIN@28Foswiki::Store::SearchAlgorithms::Forking::BEGIN@28
1113µs3µsFoswiki::Store::SearchAlgorithms::Forking::::BEGIN@32Foswiki::Store::SearchAlgorithms::Forking::BEGIN@32
Call graph for these subroutines as a Graphviz dot language file.
Line State
ments
Time
on line
Calls Time
in subs
Code
1# See bottom of file for license and copyright information
2package Foswiki::Store::SearchAlgorithms::Forking;
3
4=begin TML
5
6---+ package Foswiki::Store::SearchAlgorithms::Forking
7Implements Foswiki::Store::Interfaces::SearchAlgorithm
8
9Forking implementation of flat file store search. Uses grep.
10
11=cut
12
13231µs243µs
# spent 29µs (16+13) within Foswiki::Store::SearchAlgorithms::Forking::BEGIN@13 which was called: # once (16µs+13µs) by Foswiki::Store::Rcs::Store::query at line 13
use strict;
# spent 29µs making 1 call to Foswiki::Store::SearchAlgorithms::Forking::BEGIN@13 # spent 13µs making 1 call to strict::import
14225µs220µs
# spent 16µs (11+4) within Foswiki::Store::SearchAlgorithms::Forking::BEGIN@14 which was called: # once (11µs+4µs) by Foswiki::Store::Rcs::Store::query at line 14
use warnings;
# spent 16µs making 1 call to Foswiki::Store::SearchAlgorithms::Forking::BEGIN@14 # spent 4µs making 1 call to warnings::import
15229µs266µs
# spent 38µs (10+28) within Foswiki::Store::SearchAlgorithms::Forking::BEGIN@15 which was called: # once (10µs+28µs) by Foswiki::Store::Rcs::Store::query at line 15
use Assert;
# spent 38µs making 1 call to Foswiki::Store::SearchAlgorithms::Forking::BEGIN@15 # spent 28µs making 1 call to Exporter::import
16
17234µs15µs
# spent 5µs within Foswiki::Store::SearchAlgorithms::Forking::BEGIN@17 which was called: # once (5µs+0s) by Foswiki::Store::Rcs::Store::query at line 17
use Foswiki::Store::Interfaces::QueryAlgorithm ();
1818µsour @ISA = ('Foswiki::Store::Interfaces::QueryAlgorithm');
19
20220µs14µs
# spent 4µs within Foswiki::Store::SearchAlgorithms::Forking::BEGIN@20 which was called: # once (4µs+0s) by Foswiki::Store::Rcs::Store::query at line 20
use Foswiki::Store::Interfaces::SearchAlgorithm ();
21219µs14µs
# spent 4µs within Foswiki::Store::SearchAlgorithms::Forking::BEGIN@21 which was called: # once (4µs+0s) by Foswiki::Store::Rcs::Store::query at line 21
use Foswiki::Search::Node ();
22217µs13µs
# spent 3µs within Foswiki::Store::SearchAlgorithms::Forking::BEGIN@22 which was called: # once (3µs+0s) by Foswiki::Store::Rcs::Store::query at line 22
use Foswiki::Search::InfoCache ();
23218µs13µs
# spent 3µs within Foswiki::Store::SearchAlgorithms::Forking::BEGIN@23 which was called: # once (3µs+0s) by Foswiki::Store::Rcs::Store::query at line 23
use Foswiki::Search::ResultSet ();
24218µs13µs
# spent 3µs within Foswiki::Store::SearchAlgorithms::Forking::BEGIN@24 which was called: # once (3µs+0s) by Foswiki::Store::Rcs::Store::query at line 24
use Foswiki();
25219µs14µs
# spent 4µs within Foswiki::Store::SearchAlgorithms::Forking::BEGIN@25 which was called: # once (4µs+0s) by Foswiki::Store::Rcs::Store::query at line 25
use Foswiki::Func();
26217µs13µs
# spent 3µs within Foswiki::Store::SearchAlgorithms::Forking::BEGIN@26 which was called: # once (3µs+0s) by Foswiki::Store::Rcs::Store::query at line 26
use Foswiki::Meta ();
27217µs13µs
# spent 3µs within Foswiki::Store::SearchAlgorithms::Forking::BEGIN@27 which was called: # once (3µs+0s) by Foswiki::Store::Rcs::Store::query at line 27
use Foswiki::MetaCache ();
28218µs13µs
# spent 3µs within Foswiki::Store::SearchAlgorithms::Forking::BEGIN@28 which was called: # once (3µs+0s) by Foswiki::Store::Rcs::Store::query at line 28
use Foswiki::Query::Node ();
29219µs14µs
# spent 4µs within Foswiki::Store::SearchAlgorithms::Forking::BEGIN@29 which was called: # once (4µs+0s) by Foswiki::Store::Rcs::Store::query at line 29
use Foswiki::Query::HoistREs ();
30218µs14µs
# spent 4µs within Foswiki::Store::SearchAlgorithms::Forking::BEGIN@30 which was called: # once (4µs+0s) by Foswiki::Store::Rcs::Store::query at line 30
use Foswiki::ListIterator();
31218µs13µs
# spent 3µs within Foswiki::Store::SearchAlgorithms::Forking::BEGIN@31 which was called: # once (3µs+0s) by Foswiki::Store::Rcs::Store::query at line 31
use Foswiki::Iterator::FilterIterator();
32222µs13µs
# spent 3µs within Foswiki::Store::SearchAlgorithms::Forking::BEGIN@32 which was called: # once (3µs+0s) by Foswiki::Store::Rcs::Store::query at line 32
use Foswiki::Iterator::ProcessIterator();
33
34256µs271µs
# spent 40µs (9+31) within Foswiki::Store::SearchAlgorithms::Forking::BEGIN@34 which was called: # once (9µs+31µs) by Foswiki::Store::Rcs::Store::query at line 34
use constant MONITOR => 0;
# spent 40µs making 1 call to Foswiki::Store::SearchAlgorithms::Forking::BEGIN@34 # spent 31µs making 1 call to constant::import
35
36
# spent 5µs within Foswiki::Store::SearchAlgorithms::Forking::BEGIN@36 which was called: # once (5µs+0s) by Foswiki::Store::Rcs::Store::query at line 41
BEGIN {
3715µs if ( $Foswiki::cfg{UseLocale} ) {
38 require locale;
39 import locale();
40 }
411844µs15µs}
42
43=begin TML
44
45---++ ClassMethod new( $class, ) -> $cereal
46
47=cut
48
49
# spent 31µs (23+8) within Foswiki::Store::SearchAlgorithms::Forking::new which was called: # once (23µs+8µs) by Foswiki::Store::Rcs::Store::query at line 660 of /var/www/foswikidev/core/lib/Foswiki/Store/Rcs/Store.pm
sub new {
50117µs18µs my $self = shift()->SUPER::new( 'SEARCH', @_ );
# spent 8µs making 1 call to Foswiki::Store::Interfaces::QueryAlgorithm::new
5114µs return $self;
52}
53
54# Search .txt files in $dir for $searchString. This is the 'old' interface
55# to search algorithms (prior to Sven's massive search refactoring. It was
56# simply called 'search')
57
# spent 48.8s (1.42+47.4) within Foswiki::Store::SearchAlgorithms::Forking::_search which was called 40 times, avg 1.22s/call: # 40 times (1.42s+47.4s) by Foswiki::Store::SearchAlgorithms::Forking::_webQuery at line 248, avg 1.22s/call
sub _search {
584053µs my ( $searchString, $web, $inputTopicSet, $session, $options ) = @_;
59
60 # SMELL: I18N: 'grep' must use locales if needed,
61 # for case-insensitive searching.
624027µs my $program = '';
63
6440118µs if ( $options->{type}
65 && ( $options->{type} eq 'regex' || $options->{wordboundaries} ) )
66 {
67 $program = $Foswiki::cfg{Store}{EgrepCmd};
68 }
69 else {
70 $program = $Foswiki::cfg{Store}{FgrepCmd};
71 }
72
734047µs if ( $options->{casesensitive} ) {
7440233µs $program =~ s/%CS\{(.*?)\|.*?\}%/$1/g;
75 }
76 else {
77 $program =~ s/%CS\{.*?\|(.*?)\}%/$1/g;
78 }
794048µs if ( $options->{files_without_match} ) {
8040189µs $program =~ s/%DET\{.*?\|(.*?)\}%/$1/g;
81 }
82 else {
83 $program =~ s/%DET\{(.*?)\|.*?\}%/$1/g;
84 }
854031µs if ( $options->{wordboundaries} ) {
86
87 # Item5529: Can't use quotemeta because $searchString may
88 # be UTF8 encoded
89 # TODO when testing UTF-8 code, try quotemeta. It should
90 # work with a decent perl
91 $searchString =~ s#([][|/\\\$\^*()+\{\};@?.\{\}])#\\$1#g;
92 $searchString = '\b' . $searchString . '\b';
93 }
94
954074µs if ( $Foswiki::cfg{DetailedOS} eq 'MSWin32' ) {
96
97 #try to escape the ^ and "" for native windows grep and apache
98 $searchString =~ s/\[\^/[^^/g;
99
100 # Fix escaping and quoting for Windows
101 $searchString =~ s#\\#\\\\#g;
102 $searchString =~ s#"#\\"#g;
103 $searchString = q(") . $searchString . q(");
104 }
105
1064025µs my $matches = '';
107
108 #SMELL, TODO, replace with Store call.
1094051µs my $sDir = $Foswiki::cfg{DataDir} . '/' . $web . '/';
110
111 # process topics in sets, fix for Codev.ArgumentListIsTooLongForSearch
1124013µs my $maxTopicsInSet = 512; # max number of topics for a grep call
113 # SMELL: the number is actually dependant on the length of the path
114 # to each file
115 # SMELL: the following while loop should probably be made by sysCommand,
116 # as this is a leaky abstraction.
117 # heck, on pre WinXP its only 2048, post XP its 8191 -
118 # http://support.microsoft.com/kb/830473
1194023µs if ( $Foswiki::cfg{DetailedOS} eq 'MSWin32' ) {
120
121 #tune the number based on the length of "$sDir/WebSearchAdvanced.txt"
122 #30 is a guess - wotamess
123 $maxTopicsInSet =
124 ( ( 8191 - ( length($program) + length($searchString) + 30 ) ) /
125 ( length("$sDir/LongWebSearchAdvanced.txt") + 10 ) );
126
127 #print STDERR "++++++++++++ $maxTopicsInSet \n";
128 }
129
130 # while (my @set = splice( @take, 0, $maxTopicsInSet )) {
131 # @set = map { "$sDir/$_.txt" } @set;
1324015µs my @set;
1334056µs404.25ms $inputTopicSet->reset();
# spent 4.25ms making 40 calls to Foswiki::Iterator::FilterIterator::reset, avg 106µs/call
1344084µs404.87ms while ( $inputTopicSet->hasNext() ) {
# spent 4.87ms making 40 calls to Foswiki::Iterator::FilterIterator::hasNext, avg 122µs/call
1354604076.7ms46040391ms my $webtopic = $inputTopicSet->next();
# spent 391ms making 46040 calls to Foswiki::Iterator::FilterIterator::next, avg 8µs/call
1364604088.9ms46040493ms my ( $Iweb, $tn ) =
# spent 493ms making 46040 calls to Foswiki::Func::normalizeWebTopicName, avg 11µs/call
137 Foswiki::Func::normalizeWebTopicName( $web, $webtopic );
1384604070.0ms push( @set, "$sDir/$tn.txt" );
13946040182ms920002.09s if (
# spent 2.09s making 92000 calls to Foswiki::Iterator::FilterIterator::hasNext, avg 23µs/call
140 ( $#set >= $maxTopicsInSet ) #replace with character count..
141 || !( $inputTopicSet->hasNext() )
142 )
143 {
1441202.30ms12044.4s my ( $m, $exit ) = Foswiki::Sandbox->sysCommand(
# spent 44.4s making 120 calls to Foswiki::Sandbox::sysCommand, avg 370ms/call
145 $program,
146 TOKEN => $searchString,
147 FILES => \@set
148 );
14912014.0ms @set = ();
150
151 # man grep: "Normally, exit status is 0 if selected lines are found
152 # and 1 otherwise. But the exit status is 2 if an error occurred,
153 # unless the -q or --quiet or --silent option is used and a selected
154 # line is found."
155120153µs if ( $exit > 1 ) {
156
157 # TODO: need to work out a way to alert the admin there is
158 # a problem, without filling up the log files with repeated
159 # SEARCH's
160
161 # NOTE: we ignore the error, because grep returns an error
162 # if it comes across a broken file link or a file it does
163 # not have permission to open, so throwing here gives wrong
164 # search results.
165 # throw Error::Simple(
166 # "$program Grep for '$searchString' returned error")
167 }
1681202.94ms12027.6ms $matches .= Foswiki::Store::decode($m);
# spent 27.6ms making 120 calls to Foswiki::Store::decode, avg 230µs/call
169 }
170 }
1714038µs my %seen;
172
173 # Note use of / and \ as dir separators, to support Winblows
174 $matches =~
17517560443ms s/([^\/\\]*?)\.txt(:(.*))?$/push( @{$seen{$1}}, ($3||'') ); ''/gem;
176
177 # Implicit untaint OK; data from grep
178
17940872µs return \%seen;
180}
181
182#ok, for initial validation, naively call the code with a web.
183
# spent 49.1s (24.1ms+49.1) within Foswiki::Store::SearchAlgorithms::Forking::_webQuery which was called 40 times, avg 1.23s/call: # 40 times (24.1ms+49.1s) by Foswiki::Store::Interfaces::QueryAlgorithm::__ANON__[/var/www/foswikidev/core/lib/Foswiki/Store/Interfaces/QueryAlgorithm.pm:119] at line 110 of /var/www/foswikidev/core/lib/Foswiki/Store/Interfaces/QueryAlgorithm.pm, avg 1.23s/call
sub _webQuery {
1844061µs my ( $this, $query, $web, $inputTopicSet, $session, $options ) = @_;
185 ASSERT( !$query->isEmpty() ) if DEBUG;
186
187 #print STDERR "ForkingSEARCH(".join(', ', @{ $query->tokens() }).")\n";
188 # default scope is 'text'
1894074µs $options->{'scope'} = 'text'
190 unless ( defined( $options->{'scope'} )
191 && $options->{'scope'} =~ m/^(topic|all)$/ );
192
1934020µs my $topicSet = $inputTopicSet;
1944016µs if ( !defined($topicSet) ) {
195
196 #then we start with the whole web
197 #TODO: i'm sure that is a flawed assumption
198 my $webObject = Foswiki::Meta->new( $session, $web );
199 $topicSet =
200 Foswiki::Search::InfoCache::getTopicListIterator( $webObject,
201 $options );
202 }
203 ASSERT( UNIVERSAL::isa( $topicSet, 'Foswiki::Iterator' ) ) if DEBUG;
204
205 #print STDERR "######## Forking search ($web) tokens "
206 # .scalar(@{$query->tokens()})." : ".join(',', @{$query->tokens()})."\n";
207 # AND search - search once for each token, ANDing result together
20840196µs40120µs foreach my $token ( @{ $query->tokens() } ) {
# spent 120µs making 40 calls to Foswiki::Search::Node::tokens, avg 3µs/call
209
2104021µs my $tokenCopy = $token;
211
212 # flag for AND NOT search
2134022µs my $invertSearch = 0;
2144034µs $invertSearch = ( $tokenCopy =~ s/^\!// );
215
216 # scope can be 'topic' (default), 'text' or "all"
217 # scope='topic', e.g. Perl search on topic name:
2184013µs my %topicMatches;
2194036µs unless ( $options->{'scope'} eq 'text' ) {
220 my $qtoken = $tokenCopy;
221
222# FIXME I18N
223# http://foswiki.org/Tasks/Item1646 this causes us to use/leak huge amounts of memory if called too often
224 $qtoken = quotemeta($qtoken) if ( $options->{'type'} ne 'regex' );
225
226 $topicSet->reset();
227 while ( $topicSet->hasNext() ) {
228 my $webtopic = $topicSet->next();
229 my ( $itrWeb, $topic ) =
230 Foswiki::Func::normalizeWebTopicName( $web, $webtopic );
231
232 if ( $options->{'casesensitive'} ) {
233
234 # fix for Codev.SearchWithNoPipe
235 #push(@scopeTopicList, $topic) if ( $topic =~ m/$qtoken/ );
236 $topicMatches{$topic} = 1 if ( $topic =~ m/$qtoken/ );
237 }
238 else {
239
240 #push(@scopeTopicList, $topic) if ( $topic =~ m/$qtoken/i );
241 $topicMatches{$topic} = 1 if ( $topic =~ m/$qtoken/i );
242 }
243 }
244 }
245
246 # scope='text', e.g. grep search on topic text:
247401.10ms unless ( $options->{'scope'} eq 'topic' ) {
24840188µs4048.8s my $textMatches =
# spent 48.8s making 40 calls to Foswiki::Store::SearchAlgorithms::Forking::_search, avg 1.22s/call
249 _search( $tokenCopy, $web, $topicSet, $session, $options );
250
251 #bring the text matches into the topicMatch hash
2524013.3ms if ($textMatches) {
253 @topicMatches{ keys %$textMatches } = values %$textMatches;
254 }
255 }
256
25740131µs my @scopeTextList = ();
25840435µs if ($invertSearch) {
259 $topicSet->reset();
260 while ( $topicSet->hasNext() ) {
261 my $webtopic = $topicSet->next();
262 my ( $Iweb, $topic ) =
263 Foswiki::Func::normalizeWebTopicName( $web, $webtopic );
264
265 if ( $topicMatches{$topic} ) {
266 }
267 else {
268 push( @scopeTextList, $topic );
269 }
270 }
271 }
272 else {
273
274 #TODO: the sad thing about this is we lose info
275403.35ms @scopeTextList = keys(%topicMatches);
276 }
277
278 # reduced topic list for next token
279404.56ms40218ms $topicSet =
# spent 218ms making 40 calls to Foswiki::Search::InfoCache::new, avg 5.45ms/call
280 Foswiki::Search::InfoCache->new( $Foswiki::Plugins::SESSION, $web,
281 \@scopeTextList );
282 }
283
28440484µs return $topicSet;
285}
286
28713µs1;
288__END__