Filename | /var/www/foswikidev/core/lib/Foswiki/Store/QueryAlgorithms/BruteForce.pm |
Statements | Executed 71079 statements in 289ms |
Calls | P | F | Exclusive Time |
Inclusive Time |
Subroutine |
---|---|---|---|---|---|
40 | 1 | 1 | 477ms | 108s | _webQuery | Foswiki::Store::QueryAlgorithms::BruteForce::
1 | 1 | 1 | 2.25ms | 5.88ms | BEGIN@29 | Foswiki::Store::QueryAlgorithms::BruteForce::
1 | 1 | 1 | 23µs | 32µs | new | Foswiki::Store::QueryAlgorithms::BruteForce::
1 | 1 | 1 | 14µs | 27µs | BEGIN@26 | Foswiki::Store::QueryAlgorithms::BruteForce::
1 | 1 | 1 | 10µs | 42µs | BEGIN@53 | Foswiki::Store::QueryAlgorithms::BruteForce::
1 | 1 | 1 | 10µs | 15µs | BEGIN@27 | Foswiki::Store::QueryAlgorithms::BruteForce::
1 | 1 | 1 | 5µs | 5µs | BEGIN@32 | Foswiki::Store::QueryAlgorithms::BruteForce::
1 | 1 | 1 | 4µs | 4µs | BEGIN@38 | Foswiki::Store::QueryAlgorithms::BruteForce::
1 | 1 | 1 | 4µs | 4µs | BEGIN@40 | Foswiki::Store::QueryAlgorithms::BruteForce::
1 | 1 | 1 | 4µs | 4µs | BEGIN@33 | Foswiki::Store::QueryAlgorithms::BruteForce::
1 | 1 | 1 | 4µs | 4µs | BEGIN@43 | Foswiki::Store::QueryAlgorithms::BruteForce::
1 | 1 | 1 | 3µs | 3µs | BEGIN@44 | Foswiki::Store::QueryAlgorithms::BruteForce::
1 | 1 | 1 | 3µs | 3µs | BEGIN@46 | Foswiki::Store::QueryAlgorithms::BruteForce::
1 | 1 | 1 | 3µs | 3µs | BEGIN@35 | Foswiki::Store::QueryAlgorithms::BruteForce::
1 | 1 | 1 | 3µs | 3µs | BEGIN@41 | Foswiki::Store::QueryAlgorithms::BruteForce::
1 | 1 | 1 | 3µs | 3µs | BEGIN@34 | Foswiki::Store::QueryAlgorithms::BruteForce::
1 | 1 | 1 | 3µs | 3µs | BEGIN@39 | Foswiki::Store::QueryAlgorithms::BruteForce::
1 | 1 | 1 | 3µs | 3µs | BEGIN@42 | Foswiki::Store::QueryAlgorithms::BruteForce::
1 | 1 | 1 | 3µs | 3µs | BEGIN@36 | Foswiki::Store::QueryAlgorithms::BruteForce::
1 | 1 | 1 | 3µs | 3µs | BEGIN@37 | Foswiki::Store::QueryAlgorithms::BruteForce::
Line | State ments |
Time on line |
Calls | Time in subs |
Code |
---|---|---|---|---|---|
1 | # See bottom of file for license and copyright information | ||||
2 | package Foswiki::Store::QueryAlgorithms::BruteForce; | ||||
3 | |||||
4 | =begin TML | ||||
5 | |||||
6 | ---+ package Foswiki::Store::QueryAlgorithms::BruteForce | ||||
7 | Implements Foswiki::Store::Interfaces::QueryAlgorithm | ||||
8 | |||||
9 | Default brute-force query algorithm. Works by hoisting regular expressions | ||||
10 | out of the query tree to narrow down the set of topics to be tested. Then | ||||
11 | uses the query 'evaluate' method on each topic in turn to fully evaluate | ||||
12 | the remaining query. | ||||
13 | |||||
14 | Not sure exactly where the breakpoint is between the | ||||
15 | costs of hoisting and the advantages of hoisting. Benchmarks suggest | ||||
16 | that it's around 6 topics, though this may vary depending on disk | ||||
17 | speed and memory size. It also depends on the complexity of the query. | ||||
18 | |||||
19 | =cut | ||||
20 | |||||
21 | # TODO: There is an additional opportunity for optimisation; if we assume | ||||
22 | # the grep is solid, we can cut those parts of the query out for the full | ||||
23 | # evaluation path. Not done yet, because CDot strongly suspects it won't make | ||||
24 | # much difference. | ||||
25 | |||||
26 | 2 | 29µs | 2 | 41µs | # spent 27µs (14+13) within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@26 which was called:
# once (14µs+13µs) by Foswiki::Store::Rcs::Store::query at line 26 # spent 27µs making 1 call to Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@26
# spent 14µs making 1 call to strict::import |
27 | 2 | 32µs | 2 | 19µs | # spent 15µs (10+4) within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@27 which was called:
# once (10µs+4µs) by Foswiki::Store::Rcs::Store::query at line 27 # spent 15µs making 1 call to Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@27
# spent 4µs making 1 call to warnings::import |
28 | |||||
29 | 2 | 104µs | 1 | 5.88ms | # spent 5.88ms (2.25+3.64) within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@29 which was called:
# once (2.25ms+3.64ms) by Foswiki::Store::Rcs::Store::query at line 29 # spent 5.88ms making 1 call to Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@29 |
30 | 1 | 7µs | our @ISA = ('Foswiki::Store::Interfaces::QueryAlgorithm'); | ||
31 | |||||
32 | 2 | 23µs | 1 | 5µs | # spent 5µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@32 which was called:
# once (5µs+0s) by Foswiki::Store::Rcs::Store::query at line 32 # spent 5µs making 1 call to Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@32 |
33 | 2 | 18µs | 1 | 4µs | # spent 4µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@33 which was called:
# once (4µs+0s) by Foswiki::Store::Rcs::Store::query at line 33 # spent 4µs making 1 call to Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@33 |
34 | 2 | 18µs | 1 | 3µs | # spent 3µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@34 which was called:
# once (3µs+0s) by Foswiki::Store::Rcs::Store::query at line 34 # spent 3µs making 1 call to Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@34 |
35 | 2 | 19µs | 1 | 3µs | # spent 3µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@35 which was called:
# once (3µs+0s) by Foswiki::Store::Rcs::Store::query at line 35 # spent 3µs making 1 call to Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@35 |
36 | 2 | 18µs | 1 | 3µs | # spent 3µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@36 which was called:
# once (3µs+0s) by Foswiki::Store::Rcs::Store::query at line 36 # spent 3µs making 1 call to Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@36 |
37 | 2 | 18µs | 1 | 3µs | # spent 3µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@37 which was called:
# once (3µs+0s) by Foswiki::Store::Rcs::Store::query at line 37 # spent 3µs making 1 call to Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@37 |
38 | 2 | 18µs | 1 | 4µs | # spent 4µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@38 which was called:
# once (4µs+0s) by Foswiki::Store::Rcs::Store::query at line 38 # spent 4µs making 1 call to Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@38 |
39 | 2 | 18µs | 1 | 3µs | # spent 3µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@39 which was called:
# once (3µs+0s) by Foswiki::Store::Rcs::Store::query at line 39 # spent 3µs making 1 call to Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@39 |
40 | 2 | 18µs | 1 | 4µs | # spent 4µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@40 which was called:
# once (4µs+0s) by Foswiki::Store::Rcs::Store::query at line 40 # spent 4µs making 1 call to Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@40 |
41 | 2 | 19µs | 1 | 3µs | # spent 3µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@41 which was called:
# once (3µs+0s) by Foswiki::Store::Rcs::Store::query at line 41 # spent 3µs making 1 call to Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@41 |
42 | 2 | 18µs | 1 | 3µs | # spent 3µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@42 which was called:
# once (3µs+0s) by Foswiki::Store::Rcs::Store::query at line 42 # spent 3µs making 1 call to Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@42 |
43 | 2 | 18µs | 1 | 4µs | # spent 4µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@43 which was called:
# once (4µs+0s) by Foswiki::Store::Rcs::Store::query at line 43 # spent 4µs making 1 call to Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@43 |
44 | 2 | 39µs | 1 | 3µs | # spent 3µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@44 which was called:
# once (3µs+0s) by Foswiki::Store::Rcs::Store::query at line 44 # spent 3µs making 1 call to Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@44 |
45 | |||||
46 | # spent 3µs within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@46 which was called:
# once (3µs+0s) by Foswiki::Store::Rcs::Store::query at line 51 | ||||
47 | 1 | 5µs | if ( $Foswiki::cfg{UseLocale} ) { | ||
48 | require locale; | ||||
49 | import locale(); | ||||
50 | } | ||||
51 | 1 | 21µs | 1 | 3µs | } # spent 3µs making 1 call to Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@46 |
52 | |||||
53 | 2 | 662µs | 2 | 73µs | # spent 42µs (10+31) within Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@53 which was called:
# once (10µs+31µs) by Foswiki::Store::Rcs::Store::query at line 53 # spent 42µs making 1 call to Foswiki::Store::QueryAlgorithms::BruteForce::BEGIN@53
# spent 31µs making 1 call to constant::import |
54 | |||||
55 | =begin TML | ||||
56 | |||||
57 | ---++ ClassMethod new( $class, ) -> $cereal | ||||
58 | |||||
59 | =cut | ||||
60 | |||||
61 | # spent 32µs (23+9) within Foswiki::Store::QueryAlgorithms::BruteForce::new which was called:
# once (23µs+9µs) by Foswiki::Store::Rcs::Store::query at line 648 of /var/www/foswikidev/core/lib/Foswiki/Store/Rcs/Store.pm | ||||
62 | 1 | 16µs | 1 | 9µs | my $self = shift()->SUPER::new( 'SEARCH', @_ ); # spent 9µs making 1 call to Foswiki::Store::Interfaces::QueryAlgorithm::new |
63 | 1 | 4µs | return $self; | ||
64 | } | ||||
65 | |||||
66 | # Query over a single web | ||||
67 | # spent 108s (477ms+108) within Foswiki::Store::QueryAlgorithms::BruteForce::_webQuery which was called 40 times, avg 2.70s/call:
# 40 times (477ms+108s) by Foswiki::Store::Interfaces::QueryAlgorithm::__ANON__[/var/www/foswikidev/core/lib/Foswiki/Store/Interfaces/QueryAlgorithm.pm:119] at line 110 of /var/www/foswikidev/core/lib/Foswiki/Store/Interfaces/QueryAlgorithm.pm, avg 2.70s/call | ||||
68 | 40 | 59µs | my ( $this, $query, $web, $inputTopicSet, $session, $options ) = @_; | ||
69 | |||||
70 | 40 | 183µs | 40 | 905µs | my $resultTopicSet = # spent 905µs making 40 calls to Foswiki::Search::InfoCache::new, avg 23µs/call |
71 | Foswiki::Search::InfoCache->new( $Foswiki::Plugins::SESSION, $web ); | ||||
72 | |||||
73 | # see if this query can be fasttracked. | ||||
74 | # TODO: is this simplification call appropriate here, or should it | ||||
75 | # go in Search.pm | ||||
76 | # TODO: what about simplify to constant in _this_ web? | ||||
77 | 40 | 10µs | my $queryIsAConstantFastpath; # undefined if this is a 'real' query' | ||
78 | 40 | 118µs | 40 | 374µs | my $context = Foswiki::Meta->new( $session, $session->{webName} ); # spent 374µs making 40 calls to Foswiki::Meta::new, avg 9µs/call |
79 | 40 | 74µs | 40 | 8.79ms | $query->simplify( tom => $context, data => $context ); # spent 8.79ms making 40 calls to Foswiki::Query::Node::simplify, avg 220µs/call |
80 | |||||
81 | 40 | 70µs | 40 | 349µs | if ( $query->evaluatesToConstant() ) { # spent 349µs making 40 calls to Foswiki::Query::Node::evaluatesToConstant, avg 9µs/call |
82 | print STDERR "-- constant?\n" if MONITOR; | ||||
83 | |||||
84 | # SMELL: use any old topic | ||||
85 | my $cache = $Foswiki::Plugins::SESSION->search->metacache->get( $web, | ||||
86 | 'WebPreferences' ); | ||||
87 | my $meta = $cache->{tom}; | ||||
88 | $queryIsAConstantFastpath = | ||||
89 | $query->evaluate( tom => $meta, data => $meta ); | ||||
90 | } | ||||
91 | |||||
92 | 40 | 28µs | if ( defined($queryIsAConstantFastpath) ) { | ||
93 | if ( not $queryIsAConstantFastpath ) { | ||||
94 | print STDERR "-- no results\n" if MONITOR; | ||||
95 | |||||
96 | #CONSTANT _and_ FALSE - return no results | ||||
97 | return $resultTopicSet; | ||||
98 | } | ||||
99 | } | ||||
100 | else { | ||||
101 | print STDERR "-- not constant\n" if MONITOR; | ||||
102 | |||||
103 | # from here on, FALSE means its not a constant, TRUE | ||||
104 | # means is is a constant and evals to TRUE | ||||
105 | 40 | 18µs | $queryIsAConstantFastpath = 0; | ||
106 | } | ||||
107 | |||||
108 | # Try and hoist regular expressions out of the query that we | ||||
109 | # can use to refine the topic set | ||||
110 | |||||
111 | 40 | 178µs | 40 | 3.18ms | my $hoistedREs = Foswiki::Query::HoistREs::hoist($query); # spent 3.18ms making 40 calls to Foswiki::Query::HoistREs::hoist, avg 80µs/call |
112 | print STDERR "-- hoisted " . Data::Dumper->Dump( [$hoistedREs] ) . "\n" | ||||
113 | if MONITOR; | ||||
114 | |||||
115 | # Reduce the input topic set by matching simple topic names hoisted | ||||
116 | # from the query. | ||||
117 | |||||
118 | 40 | 33µs | if ( ( !defined( $options->{topic} ) ) | ||
119 | and ( $hoistedREs->{name} ) | ||||
120 | and ( scalar( @{ $hoistedREs->{name} } ) == 1 ) ) | ||||
121 | { | ||||
122 | |||||
123 | # only do this if the 'name' query is simple | ||||
124 | # (ie, has only one element) | ||||
125 | my @filter = @{ $hoistedREs->{name_source} }; | ||||
126 | |||||
127 | #set the 'includetopic' matcher.. | ||||
128 | $options->{topic} = $filter[0]; | ||||
129 | } | ||||
130 | |||||
131 | # Reduce the input topic set by matching the hoisted REs against | ||||
132 | # the topics in it. | ||||
133 | |||||
134 | 40 | 19µs | my $topicSet = $inputTopicSet; | ||
135 | 40 | 30µs | if ( !defined($topicSet) ) { | ||
136 | print STDERR "-- new topic Set from $web\n" if MONITOR; | ||||
137 | |||||
138 | # then we start with the whole web? | ||||
139 | # TODO: i'm sure that is a flawed assumption | ||||
140 | 40 | 126µs | 40 | 390µs | my $webObject = Foswiki::Meta->new( $session, $web ); # spent 390µs making 40 calls to Foswiki::Meta::new, avg 10µs/call |
141 | 40 | 171µs | 40 | 2.79s | $topicSet = # spent 2.79s making 40 calls to Foswiki::Search::InfoCache::getTopicListIterator, avg 69.9ms/call |
142 | Foswiki::Search::InfoCache::getTopicListIterator( $webObject, | ||||
143 | $options ); | ||||
144 | } | ||||
145 | |||||
146 | # TODO: how to ask iterator for list length? | ||||
147 | # TODO: once the inputTopicSet isa ResultSet we might have an idea | ||||
148 | # TODO: I presume $hoisetedRE's is undefined for constant queries.. | ||||
149 | # if (() and ( scalar(@$topics) > 6 )) { | ||||
150 | 40 | 7.94ms | if ( defined( $hoistedREs->{text} ) ) { | ||
151 | 40 | 110µs | my $searchOptions = { | ||
152 | type => 'regex', | ||||
153 | casesensitive => 1, | ||||
154 | files_without_match => 1, | ||||
155 | web => $web, | ||||
156 | }; | ||||
157 | 40 | 73µs | my @filter = @{ $hoistedREs->{text} }; | ||
158 | 40 | 461µs | 80 | 3.20ms | my $searchQuery = # spent 2.90ms making 40 calls to Foswiki::Query::Node::toString, avg 72µs/call
# spent 307µs making 40 calls to Foswiki::Search::Node::new, avg 8µs/call |
159 | Foswiki::Search::Node->new( $query->toString(), \@filter, | ||||
160 | $searchOptions ); | ||||
161 | |||||
162 | #use Data::Dumper; | ||||
163 | #print STDERR "--- hoisted: ".Dumper($hoistedREs)."\n" if MONITOR; | ||||
164 | |||||
165 | 40 | 98µs | 40 | 429µs | $topicSet->reset(); # spent 429µs making 40 calls to Foswiki::Iterator::FilterIterator::reset, avg 11µs/call |
166 | 40 | 466µs | 40 | 0s | $topicSet = # spent 49.2s making 40 calls to Foswiki::Store::Rcs::Store::query, avg 1.23s/call, recursion: max depth 1, sum of overlapping time 49.2s |
167 | $session->{store} | ||||
168 | ->query( $searchQuery, $topicSet, $session, $searchOptions ); | ||||
169 | } | ||||
170 | else { | ||||
171 | |||||
172 | # TODO: clearly _this_ can be re-written as a FilterIterator, | ||||
173 | # and if we are able to use the sorting hints (ie DB Store) | ||||
174 | # can propogate all the way to FORMAT | ||||
175 | |||||
176 | print STDERR "WARNING: couldn't hoistREs on " . $query->toString() | ||||
177 | if MONITOR; | ||||
178 | } | ||||
179 | |||||
180 | 40 | 162µs | local $/; | ||
181 | 40 | 271µs | 40 | 894µs | $topicSet->reset(); # spent 894µs making 40 calls to Foswiki::Iterator::FilterIterator::reset, avg 22µs/call |
182 | 40 | 130µs | 40 | 259ms | while ( $topicSet->hasNext() ) { # spent 259ms making 40 calls to Foswiki::Iterator::FilterIterator::hasNext, avg 6.48ms/call |
183 | 8760 | 14.7ms | 8760 | 71.5ms | my $webtopic = $topicSet->next(); # spent 71.5ms making 8760 calls to Foswiki::Iterator::FilterIterator::next, avg 8µs/call |
184 | 8760 | 16.1ms | 8760 | 134ms | my ( $Iweb, $topic ) = # spent 134ms making 8760 calls to Foswiki::Func::normalizeWebTopicName, avg 15µs/call |
185 | Foswiki::Func::normalizeWebTopicName( $web, $webtopic ); | ||||
186 | print STDERR "-- $Iweb, $topic\n" if MONITOR; | ||||
187 | |||||
188 | 8760 | 19.5ms | 8760 | 27.4s | if ($queryIsAConstantFastpath) { # spent 27.4s making 8760 calls to Foswiki::Iterator::FilterIterator::hasNext, avg 3.13ms/call |
189 | print STDERR "-- add $Iweb, $topic\n" if MONITOR; | ||||
190 | if ( defined( $options->{date} ) ) { | ||||
191 | |||||
192 | # TODO: preload the meta cache if we're doing date | ||||
193 | # based filtering - else the wrong filedate will be used | ||||
194 | $Foswiki::Plugins::SESSION->search->metacache->get( $Iweb, | ||||
195 | $topic ); | ||||
196 | } | ||||
197 | |||||
198 | # TODO: frustratingly, there is no way to evaluate a | ||||
199 | # filterIterator without actually iterating over it.. | ||||
200 | $resultTopicSet->addTopics( $Iweb, $topic ); | ||||
201 | } | ||||
202 | else { | ||||
203 | 8760 | 29.8ms | 26280 | 25.9s | my $meta = # spent 25.9s making 8760 calls to Foswiki::MetaCache::addMeta, avg 2.95ms/call
# spent 14.7ms making 8760 calls to Foswiki::search, avg 2µs/call
# spent 12.5ms making 8760 calls to Foswiki::Search::metacache, avg 1µs/call |
204 | $Foswiki::Plugins::SESSION->search->metacache->addMeta( $Iweb, | ||||
205 | $topic ); | ||||
206 | print STDERR "-- evaluate $Iweb, $topic\n" if MONITOR; | ||||
207 | 8760 | 1.13ms | next unless ( defined($meta) ); #not a valid or loadable topic | ||
208 | |||||
209 | # this 'lazy load' will become useful when @$topics becomes | ||||
210 | # an infoCache | ||||
211 | 8760 | 10.1ms | 8760 | 10.7ms | $meta = $meta->load() unless ( $meta->latestIsLoaded() ); # spent 10.7ms making 8760 calls to Foswiki::Meta::latestIsLoaded, avg 1µs/call |
212 | print STDERR "Processing $topic\n" | ||||
213 | if Foswiki::Query::Node::MONITOR_EVAL; | ||||
214 | 8760 | 14.0ms | 8760 | 1.20s | my $match = $query->evaluate( tom => $meta, data => $meta ); # spent 1.20s making 8760 calls to Foswiki::Query::Node::evaluate, avg 137µs/call |
215 | 8760 | 170ms | 8760 | 618ms | if ($match) { # spent 618ms making 8760 calls to Foswiki::Search::InfoCache::addTopic, avg 71µs/call |
216 | $resultTopicSet->addTopic($meta); | ||||
217 | } | ||||
218 | } | ||||
219 | } | ||||
220 | |||||
221 | 40 | 1.56ms | return $resultTopicSet; | ||
222 | } | ||||
223 | |||||
224 | 1 | 3µs | 1; | ||
225 | __END__ |