← Index
NYTProf Performance Profile   « line view »
For ./view
  Run on Fri Jul 31 18:42:36 2015
Reported on Fri Jul 31 18:48:16 2015

Filename/var/www/foswikidev/core/lib/Foswiki/Store/QueryAlgorithms/BruteForce.pm
StatementsExecuted 71079 statements in 289ms
Subroutines
Calls P F Exclusive
Time
Inclusive
Time
Subroutine
4011477ms108sFoswiki::Store::QueryAlgorithms::BruteForce::::_webQueryFoswiki::Store::QueryAlgorithms::BruteForce::_webQuery
1112.25ms5.88msFoswiki::Store::QueryAlgorithms::BruteForce::::BEGIN@29Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@29
11123µs32µsFoswiki::Store::QueryAlgorithms::BruteForce::::newFoswiki::Store::QueryAlgorithms::BruteForce::new
11114µs27µsFoswiki::Store::QueryAlgorithms::BruteForce::::BEGIN@26Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@26
11110µs42µsFoswiki::Store::QueryAlgorithms::BruteForce::::BEGIN@53Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@53
11110µs15µsFoswiki::Store::QueryAlgorithms::BruteForce::::BEGIN@27Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@27
1115µs5µsFoswiki::Store::QueryAlgorithms::BruteForce::::BEGIN@32Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@32
1114µs4µsFoswiki::Store::QueryAlgorithms::BruteForce::::BEGIN@38Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@38
1114µs4µsFoswiki::Store::QueryAlgorithms::BruteForce::::BEGIN@40Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@40
1114µs4µsFoswiki::Store::QueryAlgorithms::BruteForce::::BEGIN@33Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@33
1114µs4µsFoswiki::Store::QueryAlgorithms::BruteForce::::BEGIN@43Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@43
1113µs3µsFoswiki::Store::QueryAlgorithms::BruteForce::::BEGIN@44Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@44
1113µs3µsFoswiki::Store::QueryAlgorithms::BruteForce::::BEGIN@46Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@46
1113µs3µsFoswiki::Store::QueryAlgorithms::BruteForce::::BEGIN@35Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@35
1113µs3µsFoswiki::Store::QueryAlgorithms::BruteForce::::BEGIN@41Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@41
1113µs3µsFoswiki::Store::QueryAlgorithms::BruteForce::::BEGIN@34Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@34
1113µs3µsFoswiki::Store::QueryAlgorithms::BruteForce::::BEGIN@39Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@39
1113µs3µsFoswiki::Store::QueryAlgorithms::BruteForce::::BEGIN@42Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@42
1113µs3µsFoswiki::Store::QueryAlgorithms::BruteForce::::BEGIN@36Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@36
1113µs3µsFoswiki::Store::QueryAlgorithms::BruteForce::::BEGIN@37Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@37
Call graph for these subroutines as a Graphviz dot language file.
Line State
ments
Time
on line
Calls Time
in subs
Code
1# See bottom of file for license and copyright information
2package Foswiki::Store::QueryAlgorithms::BruteForce;
3
4=begin TML
5
6---+ package Foswiki::Store::QueryAlgorithms::BruteForce
7Implements Foswiki::Store::Interfaces::QueryAlgorithm
8
9Default brute-force query algorithm. Works by hoisting regular expressions
10out of the query tree to narrow down the set of topics to be tested. Then
11uses the query 'evaluate' method on each topic in turn to fully evaluate
12the remaining query.
13
14Not sure exactly where the breakpoint is between the
15costs of hoisting and the advantages of hoisting. Benchmarks suggest
16that it's around 6 topics, though this may vary depending on disk
17speed and memory size. It also depends on the complexity of the query.
18
19=cut
20
21# TODO: There is an additional opportunity for optimisation; if we assume
22# the grep is solid, we can cut those parts of the query out for the full
23# evaluation path. Not done yet, because CDot strongly suspects it won't make
24# much difference.
25
26229µs241µs
# spent 27µs (14+13) within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@26 which was called: # once (14µs+13µs) by Foswiki::Store::Rcs::Store::query at line 26
use strict;
# spent 27µs making 1 call to Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@26 # spent 14µs making 1 call to strict::import
27232µs219µs
# spent 15µs (10+4) within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@27 which was called: # once (10µs+4µs) by Foswiki::Store::Rcs::Store::query at line 27
use warnings;
# spent 15µs making 1 call to Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@27 # spent 4µs making 1 call to warnings::import
28
292104µs15.88ms
# spent 5.88ms (2.25+3.64) within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@29 which was called: # once (2.25ms+3.64ms) by Foswiki::Store::Rcs::Store::query at line 29
use Foswiki::Store::Interfaces::QueryAlgorithm ();
3017µsour @ISA = ('Foswiki::Store::Interfaces::QueryAlgorithm');
31
32223µs15µs
# spent 5µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@32 which was called: # once (5µs+0s) by Foswiki::Store::Rcs::Store::query at line 32
use Foswiki::Store::Interfaces::SearchAlgorithm ();
33218µs14µs
# spent 4µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@33 which was called: # once (4µs+0s) by Foswiki::Store::Rcs::Store::query at line 33
use Foswiki::Search::Node ();
34218µs13µs
# spent 3µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@34 which was called: # once (3µs+0s) by Foswiki::Store::Rcs::Store::query at line 34
use Foswiki::Search::InfoCache ();
35219µs13µs
# spent 3µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@35 which was called: # once (3µs+0s) by Foswiki::Store::Rcs::Store::query at line 35
use Foswiki::Search::ResultSet ();
36218µs13µs
# spent 3µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@36 which was called: # once (3µs+0s) by Foswiki::Store::Rcs::Store::query at line 36
use Foswiki();
37218µs13µs
# spent 3µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@37 which was called: # once (3µs+0s) by Foswiki::Store::Rcs::Store::query at line 37
use Foswiki::Func();
38218µs14µs
# spent 4µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@38 which was called: # once (4µs+0s) by Foswiki::Store::Rcs::Store::query at line 38
use Foswiki::Meta ();
39218µs13µs
# spent 3µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@39 which was called: # once (3µs+0s) by Foswiki::Store::Rcs::Store::query at line 39
use Foswiki::MetaCache ();
40218µs14µs
# spent 4µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@40 which was called: # once (4µs+0s) by Foswiki::Store::Rcs::Store::query at line 40
use Foswiki::Query::Node ();
41219µs13µs
# spent 3µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@41 which was called: # once (3µs+0s) by Foswiki::Store::Rcs::Store::query at line 41
use Foswiki::Query::HoistREs ();
42218µs13µs
# spent 3µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@42 which was called: # once (3µs+0s) by Foswiki::Store::Rcs::Store::query at line 42
use Foswiki::ListIterator();
43218µs14µs
# spent 4µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@43 which was called: # once (4µs+0s) by Foswiki::Store::Rcs::Store::query at line 43
use Foswiki::Iterator::FilterIterator();
44239µs13µs
# spent 3µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@44 which was called: # once (3µs+0s) by Foswiki::Store::Rcs::Store::query at line 44
use Foswiki::Iterator::ProcessIterator();
45
46
# spent 3µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@46 which was called: # once (3µs+0s) by Foswiki::Store::Rcs::Store::query at line 51
BEGIN {
4715µs if ( $Foswiki::cfg{UseLocale} ) {
48 require locale;
49 import locale();
50 }
51121µs13µs}
52
532662µs273µs
# spent 42µs (10+31) within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@53 which was called: # once (10µs+31µs) by Foswiki::Store::Rcs::Store::query at line 53
use constant MONITOR => 0;
# spent 42µs making 1 call to Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@53 # spent 31µs making 1 call to constant::import
54
55=begin TML
56
57---++ ClassMethod new( $class, ) -> $cereal
58
59=cut
60
61
# spent 32µs (23+9) within Foswiki::Store::QueryAlgorithms::BruteForce::new which was called: # once (23µs+9µs) by Foswiki::Store::Rcs::Store::query at line 648 of /var/www/foswikidev/core/lib/Foswiki/Store/Rcs/Store.pm
sub new {
62116µs19µs my $self = shift()->SUPER::new( 'SEARCH', @_ );
# spent 9µs making 1 call to Foswiki::Store::Interfaces::QueryAlgorithm::new
6314µs return $self;
64}
65
66# Query over a single web
67
# spent 108s (477ms+108) within Foswiki::Store::QueryAlgorithms::BruteForce::_webQuery which was called 40 times, avg 2.70s/call: # 40 times (477ms+108s) by Foswiki::Store::Interfaces::QueryAlgorithm::__ANON__[/var/www/foswikidev/core/lib/Foswiki/Store/Interfaces/QueryAlgorithm.pm:119] at line 110 of /var/www/foswikidev/core/lib/Foswiki/Store/Interfaces/QueryAlgorithm.pm, avg 2.70s/call
sub _webQuery {
684059µs my ( $this, $query, $web, $inputTopicSet, $session, $options ) = @_;
69
7040183µs40905µs my $resultTopicSet =
# spent 905µs making 40 calls to Foswiki::Search::InfoCache::new, avg 23µs/call
71 Foswiki::Search::InfoCache->new( $Foswiki::Plugins::SESSION, $web );
72
73 # see if this query can be fasttracked.
74 # TODO: is this simplification call appropriate here, or should it
75 # go in Search.pm
76 # TODO: what about simplify to constant in _this_ web?
774010µs my $queryIsAConstantFastpath; # undefined if this is a 'real' query'
7840118µs40374µs my $context = Foswiki::Meta->new( $session, $session->{webName} );
# spent 374µs making 40 calls to Foswiki::Meta::new, avg 9µs/call
794074µs408.79ms $query->simplify( tom => $context, data => $context );
# spent 8.79ms making 40 calls to Foswiki::Query::Node::simplify, avg 220µs/call
80
814070µs40349µs if ( $query->evaluatesToConstant() ) {
# spent 349µs making 40 calls to Foswiki::Query::Node::evaluatesToConstant, avg 9µs/call
82 print STDERR "-- constant?\n" if MONITOR;
83
84 # SMELL: use any old topic
85 my $cache = $Foswiki::Plugins::SESSION->search->metacache->get( $web,
86 'WebPreferences' );
87 my $meta = $cache->{tom};
88 $queryIsAConstantFastpath =
89 $query->evaluate( tom => $meta, data => $meta );
90 }
91
924028µs if ( defined($queryIsAConstantFastpath) ) {
93 if ( not $queryIsAConstantFastpath ) {
94 print STDERR "-- no results\n" if MONITOR;
95
96 #CONSTANT _and_ FALSE - return no results
97 return $resultTopicSet;
98 }
99 }
100 else {
101 print STDERR "-- not constant\n" if MONITOR;
102
103 # from here on, FALSE means its not a constant, TRUE
104 # means is is a constant and evals to TRUE
1054018µs $queryIsAConstantFastpath = 0;
106 }
107
108 # Try and hoist regular expressions out of the query that we
109 # can use to refine the topic set
110
11140178µs403.18ms my $hoistedREs = Foswiki::Query::HoistREs::hoist($query);
# spent 3.18ms making 40 calls to Foswiki::Query::HoistREs::hoist, avg 80µs/call
112 print STDERR "-- hoisted " . Data::Dumper->Dump( [$hoistedREs] ) . "\n"
113 if MONITOR;
114
115 # Reduce the input topic set by matching simple topic names hoisted
116 # from the query.
117
1184033µs if ( ( !defined( $options->{topic} ) )
119 and ( $hoistedREs->{name} )
120 and ( scalar( @{ $hoistedREs->{name} } ) == 1 ) )
121 {
122
123 # only do this if the 'name' query is simple
124 # (ie, has only one element)
125 my @filter = @{ $hoistedREs->{name_source} };
126
127 #set the 'includetopic' matcher..
128 $options->{topic} = $filter[0];
129 }
130
131 # Reduce the input topic set by matching the hoisted REs against
132 # the topics in it.
133
1344019µs my $topicSet = $inputTopicSet;
1354030µs if ( !defined($topicSet) ) {
136 print STDERR "-- new topic Set from $web\n" if MONITOR;
137
138 # then we start with the whole web?
139 # TODO: i'm sure that is a flawed assumption
14040126µs40390µs my $webObject = Foswiki::Meta->new( $session, $web );
# spent 390µs making 40 calls to Foswiki::Meta::new, avg 10µs/call
14140171µs402.79s $topicSet =
# spent 2.79s making 40 calls to Foswiki::Search::InfoCache::getTopicListIterator, avg 69.9ms/call
142 Foswiki::Search::InfoCache::getTopicListIterator( $webObject,
143 $options );
144 }
145
146 # TODO: how to ask iterator for list length?
147 # TODO: once the inputTopicSet isa ResultSet we might have an idea
148 # TODO: I presume $hoisetedRE's is undefined for constant queries..
149 # if (() and ( scalar(@$topics) > 6 )) {
150407.94ms if ( defined( $hoistedREs->{text} ) ) {
15140110µs my $searchOptions = {
152 type => 'regex',
153 casesensitive => 1,
154 files_without_match => 1,
155 web => $web,
156 };
1574073µs my @filter = @{ $hoistedREs->{text} };
15840461µs803.20ms my $searchQuery =
# spent 2.90ms making 40 calls to Foswiki::Query::Node::toString, avg 72µs/call # spent 307µs making 40 calls to Foswiki::Search::Node::new, avg 8µs/call
159 Foswiki::Search::Node->new( $query->toString(), \@filter,
160 $searchOptions );
161
162 #use Data::Dumper;
163 #print STDERR "--- hoisted: ".Dumper($hoistedREs)."\n" if MONITOR;
164
1654098µs40429µs $topicSet->reset();
# spent 429µs making 40 calls to Foswiki::Iterator::FilterIterator::reset, avg 11µs/call
16640466µs400s $topicSet =
# spent 49.2s making 40 calls to Foswiki::Store::Rcs::Store::query, avg 1.23s/call, recursion: max depth 1, sum of overlapping time 49.2s
167 $session->{store}
168 ->query( $searchQuery, $topicSet, $session, $searchOptions );
169 }
170 else {
171
172 # TODO: clearly _this_ can be re-written as a FilterIterator,
173 # and if we are able to use the sorting hints (ie DB Store)
174 # can propogate all the way to FORMAT
175
176 print STDERR "WARNING: couldn't hoistREs on " . $query->toString()
177 if MONITOR;
178 }
179
18040162µs local $/;
18140271µs40894µs $topicSet->reset();
# spent 894µs making 40 calls to Foswiki::Iterator::FilterIterator::reset, avg 22µs/call
18240130µs40259ms while ( $topicSet->hasNext() ) {
# spent 259ms making 40 calls to Foswiki::Iterator::FilterIterator::hasNext, avg 6.48ms/call
183876014.7ms876071.5ms my $webtopic = $topicSet->next();
# spent 71.5ms making 8760 calls to Foswiki::Iterator::FilterIterator::next, avg 8µs/call
184876016.1ms8760134ms my ( $Iweb, $topic ) =
# spent 134ms making 8760 calls to Foswiki::Func::normalizeWebTopicName, avg 15µs/call
185 Foswiki::Func::normalizeWebTopicName( $web, $webtopic );
186 print STDERR "-- $Iweb, $topic\n" if MONITOR;
187
188876019.5ms876027.4s if ($queryIsAConstantFastpath) {
# spent 27.4s making 8760 calls to Foswiki::Iterator::FilterIterator::hasNext, avg 3.13ms/call
189 print STDERR "-- add $Iweb, $topic\n" if MONITOR;
190 if ( defined( $options->{date} ) ) {
191
192 # TODO: preload the meta cache if we're doing date
193 # based filtering - else the wrong filedate will be used
194 $Foswiki::Plugins::SESSION->search->metacache->get( $Iweb,
195 $topic );
196 }
197
198 # TODO: frustratingly, there is no way to evaluate a
199 # filterIterator without actually iterating over it..
200 $resultTopicSet->addTopics( $Iweb, $topic );
201 }
202 else {
203876029.8ms2628025.9s my $meta =
# spent 25.9s making 8760 calls to Foswiki::MetaCache::addMeta, avg 2.95ms/call # spent 14.7ms making 8760 calls to Foswiki::search, avg 2µs/call # spent 12.5ms making 8760 calls to Foswiki::Search::metacache, avg 1µs/call
204 $Foswiki::Plugins::SESSION->search->metacache->addMeta( $Iweb,
205 $topic );
206 print STDERR "-- evaluate $Iweb, $topic\n" if MONITOR;
20787601.13ms next unless ( defined($meta) ); #not a valid or loadable topic
208
209 # this 'lazy load' will become useful when @$topics becomes
210 # an infoCache
211876010.1ms876010.7ms $meta = $meta->load() unless ( $meta->latestIsLoaded() );
# spent 10.7ms making 8760 calls to Foswiki::Meta::latestIsLoaded, avg 1µs/call
212 print STDERR "Processing $topic\n"
213 if Foswiki::Query::Node::MONITOR_EVAL;
214876014.0ms87601.20s my $match = $query->evaluate( tom => $meta, data => $meta );
# spent 1.20s making 8760 calls to Foswiki::Query::Node::evaluate, avg 137µs/call
2158760170ms8760618ms if ($match) {
# spent 618ms making 8760 calls to Foswiki::Search::InfoCache::addTopic, avg 71µs/call
216 $resultTopicSet->addTopic($meta);
217 }
218 }
219 }
220
221401.56ms return $resultTopicSet;
222}
223
22413µs1;
225__END__