blob: ff032fe0a7fe1cd2ca93bc6973a19c0204cfccd2 [file] [log] [blame]
Pierre De Rop3a00a212015-03-01 09:27:46 +00001/*
2 * Licensed to the Apache Software Foundation (ASF) under one or more
3 * contributor license agreements. See the NOTICE file distributed with
4 * this work for additional information regarding copyright ownership.
5 * The ASF licenses this file to You under the Apache License, Version 2.0
6 * (the "License"); you may not use this file except in compliance with
7 * the License. You may obtain a copy of the License at
8 *
9 * http://www.apache.org/licenses/LICENSE-2.0
10 *
11 * Unless required by applicable law or agreed to in writing, software
12 * distributed under the License is distributed on an "AS IS" BASIS,
13 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 * See the License for the specific language governing permissions and
15 * limitations under the License.
16 */
17
18Installation:
19============
20
21- see toplevel README on how to import dependencymanager into bndtools
22
23How to launch the stress test under bndtools:
24============================================
25
26The stress test performs two kind of tests on DM and parallel DM.
27
281) first kind of tests: starts/stops several times each tested bundle (DM/Parallel DM). When
29started, the test bundle is expected to register/unregister several services. And no processing is
30done at all in each component start methods).
31
322) second kind of tests: same as before, but some processing is done in each component start methods.
33
34To launch the stress test under BndTools, click on the noindex.bndrun file of the
35"org.apache.felix.dm.benchmark" project, then click on "Run", then in "Run OSGi".
36
37You should see something like that in the eclipse console:
38
39>> --------------------------------------------------------------------------------------------------------------
40g! Starting benchmarks (each tested bundle will add/remove 630 components during bundle activation).
41
42 [Starting benchmarks with no processing done in components start methods]
43
44Benchmarking bundle: org.apache.felix.dependencymanager.benchmark.dependencymanager ....................
45-> results in nanos: [189,130,687 | 205,730,144 | 312,092,102 | 357,470,857 | 871,419,487]
46
47Benchmarking bundle: org.apache.felix.dependencymanager.benchmark.dependencymanager.parallel ....................
48-> results in nanos: [85,158,366 | 103,439,337 | 122,633,515 | 157,082,407 | 284,332,202]
49
50 [Starting benchmarks with processing done in components start methods]
51
52Benchmarking bundle: org.apache.felix.dependencymanager.benchmark.dependencymanager .....
53-> results in nanos: [2,748,431,149 | 2,750,475,610 | 2,756,254,193 | 2,772,447,115 | 2,774,345,245]
54
55Benchmarking bundle: org.apache.felix.dependencymanager.benchmark.dependencymanager.parallel .....
56-> results in nanos: [687,259,058 | 696,725,568 | 700,220,615 | 704,310,739 | 740,325,481]
57-----------------------------------------------------------------------------------------------------------------
58
59You can also possibly run the same test using optimized DM filter indices.
60To do so, run "index.bndrun"
61but using DM filter indices has a CPU cost and are useful if you have many service dependencies.
62To test filter indices, first increase the number of components created/removed during bundle
63startup. To do so, edit the Artist.java and change the "Artists" when is by default set to 30, and set it to 300.
64
65You should then observe some significant performance improvements:
66
67for example, with Artist.ARTISTS=300, you should observe the following:
68
69noindex.bndrun (no filter indices used):
70
71>> --------------------------------------------------------------------------------------------------------------
72 g! Starting benchmarks (each tested bundle will add/remove 6300 components during bundle activation).
73
74 [Starting benchmarks with no processing done in components start methods]
75
76Benchmarking bundle: org.apache.felix.dependencymanager.benchmark.dependencymanager .....
77-> results in nanos: [17,436,869,644 | 17,525,534,346 | 18,080,624,001 | 18,246,597,908 | 20,715,696,669]
78
79Benchmarking bundle: org.apache.felix.dependencymanager.benchmark.dependencymanager.parallel .....
80-> results in nanos: [9,660,520,501 | 9,810,057,488 | 9,870,295,166 | 10,014,334,906 | 10,628,193,815]
81
82 [Starting benchmarks with processing done in components start methods]
83
84Benchmarking bundle: org.apache.felix.dependencymanager.benchmark.dependencymanager .....
85-> results in nanos: [42,700,651,438 | 43,207,156,615 | 43,653,372,523 | 43,869,438,994 | 44,715,701,457]
86
87Benchmarking bundle: org.apache.felix.dependencymanager.benchmark.dependencymanager.parallel .....
88-> results in nanos: [15,021,876,153 | 15,091,340,552 | 15,202,305,936 | 15,248,728,826 | 15,398,221,492]
89-----------------------------------------------------------------------------------------------------------------
90
91and with index.bndrun (using DM filter indices):
92
93>> --------------------------------------------------------------------------------------------------------------
94g! Starting benchmarks (each tested bundle will add/remove 6300 components during bundle activation).
95
96 [Starting benchmarks with no processing done in components start methods]
97
98Benchmarking bundle: org.apache.felix.dependencymanager.benchmark.dependencymanager .....
99-> results in nanos: [3,142,869,517 | 3,564,970,695 | 4,023,603,870 | 6,206,640,362 | 6,918,113,818]
100
101Benchmarking bundle: org.apache.felix.dependencymanager.benchmark.dependencymanager.parallel .....
102-> results in nanos: [2,868,554,914 | 2,873,491,201 | 2,897,439,973 | 2,913,317,331 | 3,890,123,728]
103
104 [Starting benchmarks with processing done in components start methods]
105
106Benchmarking bundle: org.apache.felix.dependencymanager.benchmark.dependencymanager .....
107-> results in nanos: [28,515,623,505 | 28,558,774,886 | 28,661,315,061 | 28,808,682,302 | 28,915,519,208]
108
109Benchmarking bundle: org.apache.felix.dependencymanager.benchmark.dependencymanager.parallel .....
110-> results in nanos: [7,702,400,991 | 7,749,145,806 | 7,760,650,323 | 7,832,386,237 | 7,854,739,136]
111-----------------------------------------------------------------------------------------------------------------
112
113
114How to interpret results:
115========================
116
117for each tested bundle, the time spent is displayed in nanos.
118for example:
119
120 -> results in nanos: [85,158,366 | 103,439,337 | 122,633,515 | 157,082,407 | 284,332,202]
121
122Here is how to interpret the results: when testing a bundle, the benchmark controller starts/stops
123it many times, then the elapsed time used to start the bundle, activate/deactivate all services, and
124stop the bundle is recorded in a list. Then this list is sorted: the first entry is the fastest
125execution time, the last entry is the slowest. the middle one is the average. We display the first
126entry (fastest), the entry at 1/4 of the list, the middle of the list, the entry at 3/4 of the list,
127and the last entry (slowest time).
128
129We don't do an average, because usually, when running benchmark, some measurements don't reflect
130reality, especially, when there is a full GC or when the JVM is warming up. (we actually do the same
131as in Java Chronicle: https://github.com/peter-lawrey/Java-Chronicle).
132
133Stress test scenario description
134--------------------------------
135
136For sake of simplicity, a simple scenario domain is used (actually, this example domain has been
137inspired from the "Java8 Lambdas" book, O'reilly): We have the following services:
138
139"Artist" service: An Artist is an individual or group of musicians, who creates some "Albums". One
140Artist service depends on several Album services.
141
142"Album" service: is a single release of musics, comprising several music Tracks. One Album depends
143on several Track services.
144
145"Track" service: A piece of music.
146
147The scenario is implemented in the following bundles
148
149- org.apache.felix.dm.benchmark.scenario: defines the interfaces.
150- org.apache.felix.dm.benchmark.scenario.impl: defines the basic implementations for the services.
151
152The Scenario Controller (see
153org.apache.felix.dm.benchmark.scenario/org.apache.felix.dm.benchmark.scenario.impl) is in charge of
154starting/stopping many times some specific bundles (DM, Parallel DM).
155
156By default, when a tested bundle is started, it will create several Artists (see Artists.ARTISTS
157constant). each Artist depends on several Albums (see Artists.ALBUMS constant), and each Album
158depends on several music Tracks (see Artists.TRACKS constants).
159
160Test bundles (DM, parallel DM)
161==============================
162
163- org.apache.felix.dm.benchmark.dependencymanager:
164It contains a simple activator, which creates the various services using dependency manager API.
165
166- org.apache.felix.dm.benchmark.dependencymanager.parallel:
167same as above, but using parallel dependency manager where components dependency management and
168components activation processing is performed concurrently, using a fixed thread pool.
169
170The org.apache.felix.dm.benchmark.controller bundle, when started, first stops all tested bundles.
171Then for each one, it performs the following test (multiple times):
172
173- start the tested bundle
174- wait for all expected services to be registered (Artists/Albums/Tracks)
175- stop the tested bundle.
176- wait for all expected services to be unregistered
177
178All the elapsed time (nanoseconds) used to execute each iteration is then recorded in a list.
179When enough iterations are done, the list is sorted (that is : the first entry in the list
180corresponds to the fastest execution time, and the last entry corresponds to the slowest execution
181time). Then, we display some meaningful entries in the list (like the first entry, the entry in the
182middle of the list (average), and the last entry (slowest).