dependencymanager/org.apache.felix.dependencymanager.benchmark/README

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

Stress test for Dependency Manager
==================================

This module provides a little framework used to perform some stress tests on Dependency Manager.
(but other DI frameworks can easily be integrated).

What is doing this test ?
=========================

This loader measures the time nedded to create a graph of service components that have dependencies
between each other. For sake of simplicity, a simple scenario domain is used (actually, this example
domain has been inspired from the "Java8 Lambdas" book, O'reilly) and the following services are used:

- Artist service: An Artist is an individual or group of musicians, who creates some "Albums". One
  Artist service depends on several Album services.

- Album service: is a single release of musics, comprising several music Tracks. One Album depends
  on several Track services.

- Track service: A piece of music.

The scenario consists in starting/stopping many times a bundle that will synchronously create the
graph of Artist/Album/Track components (630 by default). A scenario controller monitors the number
of created components and when the number of expected components are created, then the controller
stops the bundle, which will then unregister all components. Finally, when the controller detects
that all components are unregistered, the elapsed time is recorded in a list of time duration (in
nano seconds).

The same is done by another bundle that does exactly the same, but using concurrent component
registration. 

At the end of the test (that is, when the bundle that creates the components has been
started/stopped many times), then the list of all time durations (start/stop) is sorted: the first
element of the list corresponds to the shortest elapsed time used by the bundle to create and destroy the
components; and the last element in the list corresponds to the slowest elapsed time. The
middle in the duration time list is the average. We display the first entry (fastest), the entry at 1/4
of the list, the middle of the list, the entry at 3/4 of the list, and the last entry (slowest time).
We don't do an average, because usually, when running benchmark, some measurements don't reflect reality,
especially, when there is a full GC or when the JVM is warming up. (we actually do the same as in Java
Chronicle: https://github.com/peter-lawrey/Java-Chronicle).

Bundle descriptions:
===================

- org.apache.felix.dm.benchmark.dependencymanager: a test bundle that creates the components synchronously
  when the bundle is started. And when it is stopped, then the components are unregistered.

- org.apache.felix.dm.benchmark.dependencymanager.parallel: same as before, but the components are
  created concurrently.

- org.apache.felix.dm.benchmark.scenario: this bundle contains the component classes that are
  part of the scenario: we have an Artist service that depends on some Albums services, each Album
  also depends on some music Track services. The components are bound using a special "id" service
  property.

- org.apache.felix.dm.benchmark.scenario.impl: the simple Artist/Albums/Track implementations.

- org.apache.felix.dm.benchmark.controller: provides a ScenarioController service that is
  injected in all Artist/Album/Track components. When an Artist, an Album, or a Track component is
  started, it notifies the ScenerioController. Then when the controller detects that all components
  are properly created, it then stops the bundle, which in turns unregisters all components.

- org.apache.felix.dm.benchmark.controller.impl: this is the ScenarioController implementation.

How to launch the stress test under bndtools:
============================================

The stress test performs two kinds of tests on DM and parallel DM:

-noindex.bndrun: the tests is performed without using optimized DM filter indices.
-index.bndrun: the test is performed with the DM filter indices.

To launch the stress test under BndTools, click on the noindex.bndrun file of the
"org.apache.felix.dm.benchmark" project, then click on "Run", then in "Run OSGi". 

You should see something like that in the eclipse console:

>> --------------------------------------------------------------------------------------------------------------
g! Starting benchmarks (each tested bundle will add/remove 630 components during bundle activation).

	[Starting benchmarks with no processing done in components start methods]

Benchmarking bundle: org.apache.felix.dependencymanager.benchmark.dependencymanager ....................
-> results in nanos: [189,130,687 | 205,730,144 | 312,092,102 | 357,470,857 | 871,419,487]

Benchmarking bundle: org.apache.felix.dependencymanager.benchmark.dependencymanager.parallel ....................
-> results in nanos: [85,158,366 | 103,439,337 | 122,633,515 | 157,082,407 | 284,332,202]

	[Starting benchmarks with processing done in components start methods]

Benchmarking bundle: org.apache.felix.dependencymanager.benchmark.dependencymanager .....
-> results in nanos: [2,748,431,149 | 2,750,475,610 | 2,756,254,193 | 2,772,447,115 | 2,774,345,245]

Benchmarking bundle: org.apache.felix.dependencymanager.benchmark.dependencymanager.parallel .....
-> results in nanos: [687,259,058 | 696,725,568 | 700,220,615 | 704,310,739 | 740,325,481]
-----------------------------------------------------------------------------------------------------------------

You can also run the same test using optimized DM filter indices.
To do so, run "index.bndrun". But using DM filter indices has a CPU cost and are useful if you have many service dependencies.
To test filter indices, first increase the number of components created/removed during bundle
startup. To do so, edit the Artist.java and change the "Artists" from 30 to 300.

You should then observe some significant performance improvements:

for example, with Artist.ARTISTS=300, you should observe the following:

noindex.bndrun (no filter indices used):

>> --------------------------------------------------------------------------------------------------------------
 g! Starting benchmarks (each tested bundle will add/remove 6300 components during bundle activation).

        [Starting benchmarks with no processing done in components start methods]

Benchmarking bundle: org.apache.felix.dependencymanager.benchmark.dependencymanager .....
-> results in nanos: [17,436,869,644 | 17,525,534,346 | 18,080,624,001 | 18,246,597,908 | 20,715,696,669]

Benchmarking bundle: org.apache.felix.dependencymanager.benchmark.dependencymanager.parallel .....
-> results in nanos: [9,660,520,501 | 9,810,057,488 | 9,870,295,166 | 10,014,334,906 | 10,628,193,815]

        [Starting benchmarks with processing done in components start methods]

Benchmarking bundle: org.apache.felix.dependencymanager.benchmark.dependencymanager .....
-> results in nanos: [42,700,651,438 | 43,207,156,615 | 43,653,372,523 | 43,869,438,994 | 44,715,701,457]

Benchmarking bundle: org.apache.felix.dependencymanager.benchmark.dependencymanager.parallel .....
-> results in nanos: [15,021,876,153 | 15,091,340,552 | 15,202,305,936 | 15,248,728,826 | 15,398,221,492]
-----------------------------------------------------------------------------------------------------------------
 
and with index.bndrun (using DM filter indices):

>> --------------------------------------------------------------------------------------------------------------
g! Starting benchmarks (each tested bundle will add/remove 6300 components during bundle activation).

        [Starting benchmarks with no processing done in components start methods]

Benchmarking bundle: org.apache.felix.dependencymanager.benchmark.dependencymanager .....
-> results in nanos: [3,142,869,517 | 3,564,970,695 | 4,023,603,870 | 6,206,640,362 | 6,918,113,818]

Benchmarking bundle: org.apache.felix.dependencymanager.benchmark.dependencymanager.parallel .....
-> results in nanos: [2,868,554,914 | 2,873,491,201 | 2,897,439,973 | 2,913,317,331 | 3,890,123,728]

        [Starting benchmarks with processing done in components start methods]

Benchmarking bundle: org.apache.felix.dependencymanager.benchmark.dependencymanager .....
-> results in nanos: [28,515,623,505 | 28,558,774,886 | 28,661,315,061 | 28,808,682,302 | 28,915,519,208]

Benchmarking bundle: org.apache.felix.dependencymanager.benchmark.dependencymanager.parallel .....
-> results in nanos: [7,702,400,991 | 7,749,145,806 | 7,760,650,323 | 7,832,386,237 | 7,854,739,136]
-----------------------------------------------------------------------------------------------------------------
 

How to interpret results:
========================

for each tested bundle, the time spent is displayed in nanos.
for example: 

	-> results in nanos: [85,158,366 | 103,439,337 | 122,633,515 | 157,082,407 | 284,332,202]

Here, the shortest time used to activate and bind the components with each other took around 85,158,366 nanos.
the value in the midle of the list represents the averate time (122,633,515 nanos.
the last entry is the slowest elapsed time (284,332,202).