[email protected]

ABSTRACT

Keywords

improvements are made no matter how much longer the EA is run. What has happened is called premature convergence, which is that the existing genetic material in the population has converged such that the variation operators cannot produce new individuals which will move the population into a better part of the fitness landscape [2, 5, 8, 17]. Many attempts at creating a more robust EA have been tried and this is still an ongoing area of research in the Evolutionary Computation community. In the Genetic Programming (GP) community, the AgeLayered Population Structure (ALPS) Evolutionary Algorithm (EA) was introduced as a way of addressing, and significantly reducing, premature convergence [11]. ALPS differs from other EAs in that it segregates the population into multiple layers using a novel measure of age, and it reduces premature convergence by introducing a new group of randomly generated individuals into the bottom layer at regular intervals. This approach can be thought of as combining multiple, independent search runs that are done sequentially into a single, multi-layered meta-run. One advantage of ALPS is that it does not use any details of the representation and, thus, can be used with any sort of encoding scheme: GP programs, bit-strings, real-valued vectors of parameters, sequences of integers (such as for solving a scheduling or TSP problem), and anything else. Having been demonstrated to be effective on GP problems with GP representations [15, 19, 20], the main objective of this paper is to introduce the ALPS paradigm to the GA community by combining ALPS with a basic GA and running it on the problems in the Black-Box Optimization Benchmarking (BBOB) 2009 workshop at GECCO-09.

Benchmarking, Black-box optimization, Evolutionary computation

2. ALGORITHM PRESENTATION

To reduce the problem of premature convergence we define a new method for measuring an individual’s age and propose the Age-Layered Population Structure (ALPS). This measure of age measures how long the genetic material has been evolving in the population: offspring start with an age of 1 plus the age of their oldest parent instead of starting with an age of 0 as with traditional measures of age. ALPS differs from a typical Evolutionary Algorithm (EA) by segregating individuals into different age-layers by their age and by regularly introducing new, randomly generated individuals in the youngest layer. The introduction of randomly generated individuals at regular intervals results in an EA that is never completely converged and is always exploring new parts of the fitness landscape and by using age to restrict competition and breeding, younger individuals are able to develop without being dominated by older ones. In effect, ALPS is a novel way to run multiple EAs simultaneously.

Categories and Subject Descriptors G.1.6 [Numerical Analysis]: Optimization—global optimization, unconstrained optimization; F.2.1 [Analysis of Algorithms and Problem Complexity]: Numerical Algorithms and Problems

General Terms Algorithms

1.

INTRODUCTION

A common problem experienced in running an evolutionary algorithm (EA) is that after some number of evaluations the population converges to a local optima and no further

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. GECCO’09, July 8–12, 2009, Montréal Québec, Canada. Copyright 2009 ACM 978-1-60558-505-5/09/07 ...$5.00.

In the field of optimization algorithms, one type of algorithmic improvement is that of increasing the speed at which problems of solvable difficulty can be solved. This is shown in papers in which the comparison is on which algorithm can find the global optima of a benchmark problem in the fewest number of evaluations. Another type of algorithmic improvement is increasing the robustness of the algorithm. That is, either increasing the reliability of finding the global optima or being able to find better results then other algorithms. This interest in improving robustness on hard problems, at the cost of slightly slower speed on easy problems, is becoming a more useful tradeoff with the continual increase in computing power available. One approach for improving the robustness of a particular search algorithm is to regularly restart it, using a different

ALPS:

L0 I.0,0 .. I 0,m−1

L1 ... L n−1 I.n−1,0 I.1,0 .. .. I n−1,m−1 I 1,m−1

Figure 1: The layout of an ALPS system with n layers (L0 to Ln−1 ) and m individuals in each layer (Ii,0 to Ii,m−1 ). random number seed for each restart. Restarting can be done at fixed intervals with a multi-run EA, which divides a total of n generations into m runs of n/m generations. While regularly restarting the EA can improve search performance [3, 18], the challenge becomes developing a good method for deciding when to restart. An alternative to restarting the entire EA is to run multiple EAs simultaneously and only restart one of them, and this is what is done with ALPS. ALPS was developed to be a more robust EA – especially on hard problems – although not necessarily the fastest one on easy problems [11]. This ability is most easily noticed when using a small population or when performing extremely long evolutionary runs. With ALPS, several instances of the search algorithm are run in parallel, each in its own age-layer (see Figure 1) and the age of solutions is kept track of using a novel age measure. The key properties of ALPS are: • Multiple instances of a search algorithm are run in parallel, with each instance in its own age layer and having its own population of one or more candidate solutions (individuals). • Each age-layer has a maximum age and it may not contain individuals older than that maximum age. • The age of individuals is based on when the original genetic material was created from random. • The search algorithm in a given age-layer can look at individuals in its own population and at the populations in younger age layers but it can only replace individuals in its own population. • At regular intervals, the search algorithm in the first age-layer is restarted. We first review the measure of age and then how ALPS manages individuals in its age layers.

2.1 Measuring Age Over the years age has been used in various EAs to try to improve performance [12, 13, 14, 16]. In these systems all individuals, whether they are created randomly or through mutation or recombination, start with an age value of 1. After each generation in which an individual is kept in the population (eg. it is not changed through mutation or recombination) its age is increased by one. Thus in these systems age is a measure of how long a particular individual with the same set of alleles has been in the population. With ALPS, age is a measure of how long an individual’s family of genotypic material has been in the population. Randomly generated individuals, such as those that

are created in the initial generation of a canonical EA run, start with an age of 1. Each generation that an individual stays in the population (such as through elitism) its age is increased by one. Individuals that are created through mutation or recombination take the age of their oldest parent and add one to it. For example, if individual IndA , age 23, and individual IndB , age 28, are selected as parents for recombination then their offspring, IndC , will be assigned an age of 29. At the end of the reproduction phase IndA will have its age increased to 24 and IndB will have its age increased to 29 and IndC will keep its age of 29. In contrast, other age-based EA systems would assign an age of 1 to IndC . Rather than counting generations, age is implemented by taking the number of evaluations in which an individual’s genetic material has been around and dividing it by the size of the population. Randomly generated individuals store the number of evaluations that have been performed so far, and individuals created through mutation and recombination store the smallest (which is equivalent to “oldest”) value of their parents. The equation for calculating the age of an individual is: age = 1 + (evalscurrent − evalscreated )/popsize

(1)

Where: evalscurrent is the number of evaluations that have been performed so far; evalscreated is the number of evaluations that had been performed when the individual’s genetic material was first created; and popsize is the total number of individuals in all layers. A constant of 1 is added so that the age of randomly generated individuals is 1 at creation time.

2.2 ALPS-EA With the ALPS paradigm, the population is segregated into multiple age layers, with each layer having an upper age limit. The EA acts on each age layer somewhat independently of the others, with an exception being that parents can be selected from both the current layer and the layers below. When an individual is too old for its current layer, Li , the algorithm can move it up to the next layer, Li+1 . Also, at regular intervals the bottom layer is replaced with a new sub-population of randomly generated individuals, each with an age of 1. Table 1: Different systems for setting the age-limits for each age-layer and the corresponding maximum age in each layer for an age-gap of 1. Aging-scheme Linear Fibonacci Polynomial (n2 ) Exponential (2n )

0 1 1 1 1

1 2 2 2 2

Max 2 3 3 4 4

age 3 4 5 9 8

in layer 4 5 5 6 8 13 16 25 16 32

6 7 21 49 64

In setting up an ALPS run, the number of age layers and the age limits for each layer are parameters that are set by the user. Different systems can be used for setting these values, such as by using linearly, polynomially or exponentially increasing limits (Table 1). To keep the size of the population and number of layers manageable, and since there

is generally little need to segregate individuals which are within a few “generations” of each other, these values are then multiplied by an age-gap parameter. Also, there is no maximum age for the last layer and a single-layer version of ALPS operates exactly as the standard EA. For example, in a system with five layers, a polynomial aging-scheme and an age gap of seven the maximum ages for the five layers are: 7, 14, 28, 56, and ∞. With an ALPS-EA, evolution occurs in each layer somewhat independently of the others. When selecting parents to create new individuals for a given layer, parents are selected from individuals in that layer as well as the previous one. Restricting the selection of parents in this way limits selection competition to those individuals of similar ages and prevents the older individuals from dominating the younger ones. Once individuals have been in the “bottom” layer of the population for as many generations as its age limit, all individuals in this layer are replaced with randomly created individuals. For example, with the example in the previous paragraph, the bottom layer is replaced with a new group of randomly created individuals every 7 generations.

2.3 Pseudo-Code An ALPS-EA works as follows. The algorithm starts by configuring the age layers and then creating, and evaluating, an initial, random population. Once the initial population is created and evaluated, ALPS-EA enters its main loop which consists of cycling through the layers, from Ln−1 to L0 , and then evolving the EA in that layer for one generation: 1: procedure AlpsEA( ) 2: gen ← 0 3: while not done do 4: for i = n − 1 to 0 do 5: if (i == 0) && (gen % age gap == 0) then 6: Restart L0 with a new random population. 7: end if 8: Evolve(Li ). ⊲ For one generation. 9: end for 10: end while 11: end procedure ALPS allows for different methods for selecting the parents (eg tournament selection or some form of roulette wheel selection) and different representations and variation operators. One way for genetic material to move up from one layer to the next is by implementing the selection method so to pick individuals from layers Li−1 and Li . In addition, elitism can be added either to just the top layer or to all layers.

2.4 Additional Comments The above algorithm allows for different methods of selecting the parents (eg tournament selection or some form of roulette wheel selection). In addition, elitism can be added either to just the top layer or to all layers. More generally, ALPS can be thought of as an approach for solving the restart problem when trying to perform multiple, sequential optimization runs. In thinking of ALPS as a method for combining multiple, independent optimization runs into a single run, there are two important criteria. Individuals in the population have an age, and this age value is based on when the genotypic material was randomly created. That is, randomly created individuals start with an age of 1 and individuals created through mutation and

offspring inherit the age of the parent. Age can be measured in generations (Generational ALPS) or in number of evaluations divided by population size (Steady-State ALPS) since the two are equivalent. By increasing an individual’s age each generation in which it is used as a parent, it will eventually age its way up the layers and, likely, out of the population. An individual is only guaranteed to stay in the population forever it if is at the global optima, otherwise it will eventually be replaced as better individuals are evolved. Likely any measure of age that is relatively similar to the one proposed will work just as well. The second important criteria is that selection and replacement are restricted to individuals of similar ages. The version of ALPS described above has explicit age-layers, and in this case the selection of parents is restricted to parents who are young enough that their offspring will not be too old for that layer. In this implementation parents are only selected from the current and previous layer, but it would also be acceptable to select from all previous layers. Explicit age layers may not be necessary as long as the ALPS implementation restricts breeding and competition between similarly aged individuals. An advantage of using explicit age layers is that any optimization algorithm could be used in each layer. In the experiments presented in this paper a fairly standard EA is used, but it would be perfectly acceptable to combine ALPS with CMA-ES [1], Differential Evolution [6], Particle Swarm Evolution [4], Hill-Climbing, Simulated Annealing, or anything else.

3. EXPERIMENTAL PROCEDURE The objective of these experiments is to compare the performance of an ALPS GA against other mainstream algorithms in the GA community on the BBOB functions. For this comparison we combine ALPS with a standard GA and configure it as follows: 1 • 12 age layers are used with a variation of the exponential aging scheme and an age-gap of 3. The maximum ages for each age layer are: 3, 5, 9, 17, 33, 65, 129, 257, 513, 1025, 2049 and ∞. • The GA in each age layer has 30 individuals. • An elitism of 4 is used by the GA in each age layer and the other individuals are created by using one of two recombination operators (chosen with equal likelihood). • To create a new individual tournament selection is used with a tournament size of 4. Individuals are chosen from the population in current layer (Li ) or, with 20% likelihood, from the GA’s population in the previous layer (Li−1 ). • The first recombination operator uses the winner of the tournament as a “center” point and uses another member of the tournament (chosen at random) as the second parent. The values for each gene are selected at random using: Ci = P1,i + k ∗ (P1,i − P2,i ), where k is a random number with a normal distribution. 1 Source code for ALPS http://idesign.ucsc.edu

is

available

online

at

• The second operator is the similar to first except instead of generating a new random number k for each gene, the same value of k is used for all genes. This results in the offspring, C, falling somewhere on the line through P1 and P2 .

[12]

Because ALPS is running several GAs simultaneously, and the GA in the first layer is being regularly restarted – in this case, every 3 generations – each trial consists of a single ALPS run for 5 million evaluations.

4.

RESULTS

[13]

Results from experiments according to [9] on the benchmark functions given in [7, 10] are presented in Figures 2 and 3 and in Table 2.

5.

REFERENCES

[1] A. Auger and N. Hansen. A restart CMA evolution strategy with increasing population size. In Proceedings of the IEEE Congress on Evolutionary Computation (CEC 2005), pages 1769–1776. IEEE Press, 2005. [2] J. E. Baker. Adaptive selection methods for genetic algorithms. In J. J. Grefenstette, editor, Proc. of the First Intl. Conf. on Genetic Algorithms, pages 101–111, 1985. ˜ z-Paz and D. E. Goldberg. Are multiple runs [3] E. CantAˇ of genetic algorithms better than one? In E. C.-P. et al., editor, Proc. of the Genetic and Evolutionary Computation Conference, LNCS 2724, pages 801–812, Berlin, 2003. Springer-Verlag. [4] M. Clerc and J. Kennedy. The particle swarm-explosion, stability, and convergence in a multidimensional complex space. IEEE Transactions on Evolutionary Computation, 6:58–73, 2002. [5] K. A. DeJong. Analysis of the Behavior of a Class of Genetic Adaptive Systems. Dept. Computer and Communication Sciences, University of Michigan, Ann Arbor, 1975. [6] V. Feoktistov. Differential Evolution: In Search of Solutions (Springer Optimization and Its Applications). Springer-Verlag New York, Inc., Secaucus, NJ, USA, 2006. [7] S. Finck, N. Hansen, R. Ros, and A. Auger. Real-parameter black-box optimization benchmarking 2009: Presentation of the noiseless functions. Technical Report 2009/20, Research Center PPE, 2009. [8] D. E. Goldberg and P. Segrest. Finite markov chain analysis of genetic algorithms. In J. J. Grefenstette, editor, Proc. of the Second Intl. Conf. on Genetic Algorithms, pages 1–8. Lawrence Erlbaum Associates, 1987. [9] N. Hansen, A. Auger, S. Finck, and R. Ros. Real-parameter black-box optimization benchmarking 2009: Experimental setup. Technical Report RR-6828, INRIA, 2009. [10] N. Hansen, S. Finck, R. Ros, and A. Auger. Real-parameter black-box optimization benchmarking 2009: Noiseless functions definitions. Technical Report RR-6829, INRIA, 2009. [11] G. S. Hornby. ALPS: The age-layered population structure for reducing the problem of premature

[14]

[15]

[16]

[17]

[18]

[19]

[20]

convergence. In M. K. et al., editor, Proc. of the Genetic and Evolutionary Computation Conference, GECCO-2006, pages 815–822, Seattle, WA, 2006. ACM Press. G. S. Hornby, M. Fujita, S. Takamura, T. Yamamoto, and O. Hanagata. Autonomous evolution of gaits with the sony quadruped robot. In Banzhaf, Daida, Eiben, Garzon, Honavar, Jakiel, and Smith, editors, Proc. of the Genetic and Evolutionary Computation Conference, pages 1297–1304. Morgan Kaufmann, 1999. A. Huber and D. A. Mlynski. An age-controlled evolutionary algorithm for optimization problems in physical layout. In International Symposium on Circuits and Systems, pages 262–265. IEEE Press, 1998. J.-H. Kim, J.-Y. Jeon, H.-K. Chae, and K. Koh. A novel evolutionary algorithm with fast convergence. In IEEE International Conference on Evolutionary Computation, pages 228–29. IEEE Press, 1995. M. F. Korns and L. Nunez. Profiling symbolic regression-classification. In R. L. Riolo, T. Soule, and B. Worzel, editors, Genetic Programming Theory and Practice VI, Genetic and Evolutionary Computation, chapter 14, pages 215–229. Springer, Ann Arbor, 15-17May 2008. N. Kubota, T. Fukuda, F. Arai, and K. Shimojima. Genetic algorithm with age structure and its application to self-organizing manufacturing system. In IEEE Symposium on Emerging Technologies and Factory Automation, pages 472–477. IEEE Press, 1994. S. J. Louis and G. J. E. Rawlins. Syntactic analysis of convergence in genetic algorithms. In L. D. Whitley, editor, Foundations of Genetic Algorithms 2, pages 141–151. Morgan Kaufmann, 1993. S. Luke. When short runs beat long runs. In L. Spector, E. D. Goodman, A. Wu, W. B. Langdon, H.-M. Voigt, M. Gen, S. Sen, M. Dorigo, S. Pezeshk, M. H. Garzon, and E. Burke, editors, Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2001), pages 74–80, San Francisco, California, USA, 7-11 July 2001. Morgan Kaufmann. T. McConaghy, P. Palmers, G. Gielen, and M. Steyaert. Genetic programming with reuse of known designs. In R. L. Riolo, T. Soule, and B. Worzel, editors, Genetic Programming Theory and Practice V, Genetic and Evolutionary Computation, chapter 10, pages 161–186. Springer, Ann Arbor, 17-19May 2007. A. Willis, S. Patel, and C. D. Clack. GP age-layer and crossover effects in bid-offer spread prediction. In Proceedings of the 10th annual conference on Genetic and Evolutionary Computation Conference, Atlanta, GA, July 12-16 2008.

1 Sphere

6

2 Ellipsoid separable

7 6

5

5

4 +1 3

-1 -2

-5 -8

0 2

5

3

10

20

2

3

2

1

0 2

5

3

10

0 2

3

20

40

9 Rosenbrock rotated

5

10

20

6

6

5

5

4

4

3

3

2

2

1

1

0

0

40

6 Attractive sector

8

4

9

1

40

5 Linear slope

5

7

2

3

5

10

20

40

7 Step-ellipsoid

8

2

7

7

7

6

6

5

5

5

4

4

4

3

3

3

2

2

2

1

1

1

0

0

0

2

3

5

8

7

7

6

6

10

20

40

10 Ellipsoid

9

2

3

5

10

20

40

11 Discus

9

5

5

4

4

3

3

3

3

2

2

2

2

1

1

1

1

0

0

0

0

8 7

10

20

40

21

12

6 5 4 3 2 1 0 2

3

5

10

20

40

17 Schaffer F7, condition 10

9

24

3

5

10

20

14 Sum of different powers

40

2

3

5

10

20

40

15 Rastrigin

9

6

2

3

5

10

20

40

18 Schaffer F7, condition 1000

8

8

7

7

6

6

5

5

4

4

3

3

2

2

8

7

7

6

6

5

5

5

4

4

4

3

3

3

2

2

2

1

1

1

40

5

3

10

20

40

20

40

16 Weierstrass 6

0 2

8

20

1

0 3

5

10

20

40

19 Griewank-Rosenbrock F8F2

9

39

2

9

1

9

8 7

2

9 8 7 6 5 4 3 2 1 0

10

12 Bent cigar

6

5

5

5

3

7

6

4

13 Sharp ridge

40

8 33

7

5

3

20

16

2

4

2

10

8 Rosenbrock original

9

8 27

5

3

8

6

8

9

4 Skew Rastrigin-Bueche separable

3

-3 1

8

7

4

+0

2

3 Rastrigin separable

8

2

5

3

14

7 30

10

20 Schwefel x*sin(x)

8

6

6

5 4

0

0 2

3

5

10

20

40

21 Gallagher 101 peaks

6

3

5

10

20

40

22 Gallagher 21 peaks 1

8

7

3

5

10

20

40

23 Katsuuras

2

3

5

3

10

20

8 7

6

6

5

5

4

4

3

3

2

2

1

1

1

0

0

0

42

+1

4 3

2

2 1

40

24 Lunacek bi-Rastrigin

9

7

5

3

1 0 2

9

6 4

2

0 2

8

5

3

+0 -1 -2 -3 -5 -8

0 2

3

5

10

20

40

2

3

5

10

20

40

2

3

5

10

20

40

2

3

5

10

20

40

Figure 2: Expected Running Time (ERT, •) to reach fopt + ∆f and median number of function evaluations of successful trials (+), shown for ∆f = 10, 1, 10−1 , 10−2 , 10−3 , 10−5 , 10−8 (the exponent is given in the legend of f1 and f24 ) versus dimension in log-log presentation. The ERT(∆f ) equals to #FEs(∆f ) divided by the number of successful trials, where a trial is successful if fopt + ∆f was surpassed during the trial. The #FEs(∆f ) are the total number of function evaluations while fopt + ∆f was not surpassed during the trial from all respective trials (successful and unsuccessful), and fopt denotes the optimal function value. Crosses (×) indicate the total number of function evaluations #FEs(−∞). Numbers above ERT-symbols indicate the number of successful trials. Annotated numbers on the ordinate are decimal logarithms. Additional grid lines show linear and quadratic scaling.

f 1 in 5-D, N=45, mFE=20996 f 1 in 20-D, N=15, mFE=100032 # ERT 10% 90% RTsucc # ERT 10% 90% RTsucc 45 1.0 e2 8.2 e1 1.2 e2 1.0 e2 15 6.4 e3 6.1 e3 6.6 e3 6.4 e3 45 1.7 e3 1.6 e3 1.8 e3 1.7 e3 15 1.4 e4 1.3 e4 1.4 e4 1.4 e4 45 3.6 e3 3.6 e3 3.7 e3 3.6 e3 15 2.2 e4 2.2 e4 2.3 e4 2.2 e4 45 8.3 e3 8.2 e3 8.4 e3 8.3 e3 15 3.9 e4 3.8 e4 4.0 e4 3.9 e4 45 1.3 e4 1.2 e4 1.3 e4 1.3 e4 15 5.8 e4 5.7 e4 5.9 e4 5.8 e4 45 2.0 e4 1.9 e4 2.0 e4 2.0 e4 15 9.2 e4 9.1 e4 9.3 e4 9.2 e4 f 3 in 5-D, N=45, mFE=120523 f 3 in 20-D, N=15, mFE=5.00 e6 ∆f # ERT 10% 90% RTsucc # ERT 10% 90% RTsucc 10 45 4.1 e3 3.8 e3 4.3 e3 4.1 e3 15 2.5 e5 2.1 e5 3.0 e5 2.5 e5 1 45 1.9 e4 1.8 e4 2.0 e4 1.9 e4 15 1.4 e6 1.3 e6 1.5 e6 1.4 e6 1e−1 45 3.7 e4 3.2 e4 4.1 e4 3.7 e4 15 1.8 e6 1.5 e6 2.0 e6 1.8 e6 1e−3 45 4.2 e4 3.8 e4 4.7 e4 4.2 e4 15 2.1 e6 1.8 e6 2.3 e6 2.1 e6 1e−5 45 4.8 e4 4.4 e4 5.3 e4 4.8 e4 15 3.0 e6 2.8 e6 3.3 e6 3.0 e6 1e−8 45 5.9 e4 5.5 e4 6.4 e4 5.9 e4 0 39e–8 81e–9 31e–7 4.5 e6 f 5 in 5-D, N=45, mFE=2786 f 5 in 20-D, N=15, mFE=18392 ∆f # ERT 10% 90% RTsucc # ERT 10% 90% RTsucc 10 45 6.4 e2 6.0 e2 6.9 e2 6.4 e2 15 6.4 e3 6.1 e3 6.8 e3 6.4 e3 1 45 1.3 e3 1.3 e3 1.4 e3 1.3 e3 15 8.5 e3 8.2 e3 8.8 e3 8.5 e3 1e−1 45 1.6 e3 1.5 e3 1.7 e3 1.6 e3 15 1.0 e4 9.6 e3 1.0 e4 1.0 e4 15 1.2 e4 1.1 e4 1.3 e4 1.2 e4 1e−3 45 1.6 e3 1.5 e3 1.7 e3 1.6 e3 1e−5 45 1.6 e3 1.5 e3 1.7 e3 1.6 e3 15 1.3 e4 1.3 e4 1.4 e4 1.3 e4 1e−8 45 1.6 e3 1.5 e3 1.7 e3 1.6 e3 15 1.3 e4 1.3 e4 1.4 e4 1.3 e4 f 7 in 5-D, N=45, mFE=23854 f 7 in 20-D, N=15, mFE=5.00 e6 ∆f # ERT 10% 90% RTsucc # ERT 10% 90% RTsucc 10 45 7.9 e2 7.3 e2 8.5 e2 7.9 e2 15 4.0 e4 2.2 e4 6.0 e4 4.0 e4 1 45 3.3 e3 3.1 e3 3.5 e3 3.3 e3 10 3.6 e6 2.5 e6 5.3 e6 2.4 e6 6.7 e3 6 8.9 e6 6.0 e6 1.5 e7 3.6 e6 1e−1 45 6.7 e3 6.4 e3 6.9 e3 1e−3 45 1.4 e4 1.3 e4 1.5 e4 1.4 e4 0 48e–2 31e–3 19e–1 4.5 e6 1e−5 45 1.4 e4 1.3 e4 1.5 e4 1.4 e4 . . . . . 1e−8 45 1.7 e4 1.6 e4 1.8 e4 1.7 e4 . . . . . f 9 in 5-D, N=45, mFE=1.02 e6 f 9 in 20-D, N=15, mFE=5.00 e6 ∆f # ERT 10% 90% RTsucc # ERT 10% 90% RTsucc 10 45 3.1 e3 3.0 e3 3.2 e3 3.1 e3 15 6.0 e5 4.9 e5 7.1 e5 6.0 e5 1 45 1.2 e4 1.1 e4 1.3 e4 1.2 e4 0 48e–1 25e–1 60e–1 4.5 e6 2.9 e4 . . . . . 1e−1 45 2.9 e4 2.6 e4 3.1 e4 1e−3 45 7.8 e4 7.3 e4 8.3 e4 7.8 e4 . . . . . 1e−5 45 1.3 e5 1.2 e5 1.3 e5 1.3 e5 . . . . . 1e−8 45 2.7 e5 2.3 e5 3.2 e5 2.7 e5 . . . . . f 11 in 5-D, N=45, mFE=5.00 e6 f 11 in 20-D, N=15, mFE=5.00 e6 ∆f # ERT 10% 90% RTsucc # ERT 10% 90% RTsucc 10 45 7.7 e3 6.6 e3 8.7 e3 7.7 e3 15 5.0 e5 4.5 e5 5.4 e5 5.0 e5 1 45 6.9 e4 6.2 e4 7.5 e4 6.9 e4 15 2.2 e6 2.0 e6 2.4 e6 2.2 e6 2.3 e5 5 1.4 e7 9.5 e6 2.4 e7 4.6 e6 1e−1 45 2.3 e5 2.1 e5 2.5 e5 1e−3 45 9.5 e5 8.8 e5 1.0 e6 9.5 e5 0 12e–2 47e–3 30e–2 4.5 e6 1e−5 42 2.7 e6 2.4 e6 2.9 e6 2.5 e6 . . . . . 1e−8 0 72e–8 93e–9 79e–7 4.5 e6 . . . . . f 13 in 5-D, N=45, mFE=5.00 e6 f 13 in 20-D, N=15, mFE=5.00 e6 ∆f # ERT 10% 90% RTsucc # ERT 10% 90% RTsucc 10 45 6.6 e3 6.4 e3 6.8 e3 6.6 e3 15 6.3 e4 5.5 e4 7.3 e4 6.3 e4 1 45 1.6 e4 1.5 e4 1.7 e4 1.6 e4 15 2.6 e5 2.0 e5 3.2 e5 2.6 e5 1e−1 45 7.0 e4 6.2 e4 7.8 e4 7.0 e4 15 1.2 e6 8.3 e5 1.5 e6 1.2 e6 1e−3 45 7.4 e5 5.5 e5 9.5 e5 7.4 e5 0 14e–3 31e–4 47e–3 4.5 e6 1e−5 24 6.8 e6 5.7 e6 8.4 e6 3.7 e6 . . . . . 1e−8 0 76e–7 38e–8 16e–5 4.5 e6 . . . . . f 15 in 5-D, N=45, mFE=1.40 e6 f 15 in 20-D, N=15, mFE=5.00 e6 ∆f # ERT 10% 90% RTsucc # ERT 10% 90% RTsucc 10 45 4.7 e3 4.5 e3 4.9 e3 4.7 e3 0 20e+0 16e+0 24e+0 4.5 e6 45 8.4 e4 7.2 e4 9.6 e4 8.4 e4 . . . . . 1 1e−1 45 5.1 e5 4.4 e5 5.8 e5 5.1 e5 . . . . . 1e−3 45 5.1 e5 4.4 e5 5.9 e5 5.1 e5 . . . . . 1e−5 45 5.2 e5 4.5 e5 6.0 e5 5.2 e5 . . . . . 1e−8 45 5.5 e5 4.8 e5 6.3 e5 5.5 e5 . . . . . f 17 in 5-D, N=45, mFE=5.00 e6 f 17 in 20-D, N=15, mFE=5.00 e6 ∆f # ERT 10% 90% RTsucc # ERT 10% 90% RTsucc 10 45 1.3 e1 1.1 e1 1.4 e1 1.3 e1 15 1.2 e3 1.0 e3 1.4 e3 1.2 e3 45 2.6 e3 2.5 e3 2.8 e3 2.6 e3 15 2.9 e4 2.4 e4 3.4 e4 2.9 e4 1 1e−1 45 7.2 e3 6.9 e3 7.4 e3 7.2 e3 3 2.0 e7 1.2 e7 5.7 e7 5.0 e6 1e−3 45 2.6 e4 2.3 e4 2.9 e4 2.6 e4 0 16e–2 79e–3 27e–2 3.2 e6 1e−5 45 1.3 e5 1.1 e5 1.5 e5 1.3 e5 . . . . . 1e−8 24 6.1 e6 5.0 e6 7.6 e6 3.3 e6 . . . . . f 19 in 5-D, N=45, mFE=5.00 e6 f 19 in 20-D, N=15, mFE=5.00 e6 ∆f # ERT 10% 90% RTsucc # ERT 10% 90% RTsucc 10 45 5.8 e1 4.8 e1 6.7 e1 5.8 e1 15 1.2 e3 1.1 e3 1.4 e3 1.2 e3 45 5.1 e3 4.6 e3 5.6 e3 5.1 e3 15 6.2 e5 4.1 e5 8.4 e5 6.2 e5 1 1e−1 45 1.2 e5 1.0 e5 1.4 e5 1.2 e5 0 31e–2 18e–2 48e–2 4.5 e6 1e−3 30 4.7 e6 4.0 e6 5.7 e6 3.2 e6 . . . . . 1e−5 30 4.9 e6 4.2 e6 5.8 e6 3.3 e6 . . . . . 1e−8 30 5.1 e6 4.4 e6 6.0 e6 3.5 e6 . . . . . f 21 in 5-D, N=45, mFE=23469 f 21 in 20-D, N=15, mFE=611471 ∆f # ERT 10% 90% RTsucc # ERT 10% 90% RTsucc 10 45 1.4 e2 1.2 e2 1.6 e2 1.4 e2 15 8.1 e3 7.4 e3 8.8 e3 8.1 e3 45 2.6 e3 2.4 e3 2.8 e3 2.6 e3 15 8.2 e4 4.2 e4 1.2 e5 8.2 e4 1 1e−1 45 4.8 e3 4.5 e3 5.2 e3 4.8 e3 15 9.8 e4 6.0 e4 1.4 e5 9.8 e4 1e−3 45 8.0 e3 7.6 e3 8.4 e3 8.0 e3 15 1.2 e5 8.4 e4 1.7 e5 1.2 e5 1e−5 45 1.1 e4 1.1 e4 1.2 e4 1.1 e4 15 1.5 e5 1.1 e5 2.0 e5 1.5 e5 1e−8 45 1.6 e4 1.5 e4 1.6 e4 1.6 e4 15 2.0 e5 1.6 e5 2.5 e5 2.0 e5 f 23 in 5-D, N=40, mFE=5.00 e6 f 23 in 20-D, N=15, mFE=5.00 e6 ∆f # ERT 10% 90% RTsucc # ERT 10% 90% RTsucc 10 40 6.5 e0 5.5 e0 7.5 e0 6.5 e0 15 5.9 e0 4.8 e0 7.0 e0 5.9 e0 40 1.1 e4 9.8 e3 1.2 e4 1.1 e4 15 1.3 e5 1.1 e5 1.5 e5 1.3 e5 1 1e−1 40 4.1 e5 3.3 e5 4.8 e5 4.1 e5 0 29e–2 17e–2 45e–2 4.5 e6 1e−3 2 9.8 e7 4.9 e7 >2 e8 5.0 e6 . . . . . 1e−5 0 54e–4 13e–4 24e–3 3.2 e6 . . . . . 1e−8 . . . . . . . . . . ∆f 10 1 1e−1 1e−3 1e−5 1e−8

f 2 in 5-D, N=45, mFE=29358 f 2 in 20-D, N=15, mFE=225513 # ERT 10% 90% RTsucc # ERT 10% 90% RTsucc 45 6.1 e3 6.0 e3 6.2 e3 6.1 e3 15 3.4 e4 3.3 e4 3.4 e4 3.4 e4 45 8.3 e3 8.1 e3 8.4 e3 8.3 e3 15 4.3 e4 4.3 e4 4.4 e4 4.3 e4 45 1.1 e4 1.0 e4 1.1 e4 1.1 e4 15 5.3 e4 5.2 e4 5.4 e4 5.3 e4 45 1.5 e4 1.5 e4 1.5 e4 1.5 e4 15 7.4 e4 7.3 e4 7.5 e4 7.4 e4 45 2.0 e4 1.9 e4 2.0 e4 2.0 e4 15 1.0 e5 9.8 e4 1.0 e5 1.0 e5 45 2.7 e4 2.6 e4 2.7 e4 2.7 e4 15 2.0 e5 2.0 e5 2.1 e5 2.0 e5 f 4 in 5-D, N=45, mFE=222315 f 4 in 20-D, N=15, mFE=5.00 e6 ∆f # ERT 10% 90% RTsucc # ERT 10% 90% RTsucc 10 45 5.8 e3 5.6 e3 6.0 e3 5.8 e3 15 6.9 e5 6.0 e5 7.8 e5 6.9 e5 1 45 4.6 e4 4.0 e4 5.1 e4 4.6 e4 1 7.5 e7 3.7 e7 >7 e7 5.0 e6 1e−1 45 1.0 e5 9.0 e4 1.1 e5 1.0 e5 0 30e–1 20e–1 40e–1 4.5 e6 1e−3 45 1.0 e5 9.4 e4 1.2 e5 1.0 e5 . . . . . 1e−5 45 1.1 e5 1.0 e5 1.2 e5 1.1 e5 . . . . . 1e−8 45 1.2 e5 1.1 e5 1.4 e5 1.2 e5 . . . . . f 6 in 5-D, N=45, mFE=70473 f 6 in 20-D, N=15, mFE=5.00 e6 ∆f # ERT 10% 90% RTsucc # ERT 10% 90% RTsucc 10 45 2.3 e3 2.1 e3 2.4 e3 2.3 e3 15 4.4 e4 4.1 e4 4.7 e4 4.4 e4 1 45 6.1 e3 5.8 e3 6.3 e3 6.1 e3 15 9.4 e4 8.7 e4 1.0 e5 9.4 e4 1e−1 45 1.1 e4 1.1 e4 1.2 e4 1.1 e4 15 1.6 e5 1.5 e5 1.7 e5 1.6 e5 2.3 e4 15 3.8 e5 3.4 e5 4.1 e5 3.8 e5 1e−3 45 2.3 e4 2.2 e4 2.3 e4 1e−5 45 3.5 e4 3.4 e4 3.6 e4 3.5 e4 15 1.8 e6 1.6 e6 2.1 e6 1.8 e6 1e−8 45 5.7 e4 5.5 e4 5.8 e4 5.7 e4 0 14e–7 46e–8 56e–7 4.5 e6 f 8 in 5-D, N=45, mFE=570079 f 8 in 20-D, N=15, mFE=5.00 e6 ∆f # ERT 10% 90% RTsucc # ERT 10% 90% RTsucc 10 45 3.8 e3 3.6 e3 3.9 e3 3.8 e3 15 2.0 e5 1.6 e5 2.4 e5 2.0 e5 1 45 1.3 e4 1.2 e4 1.5 e4 1.3 e4 12 1.7 e6 9.3 e5 2.7 e6 1.2 e6 3.5 e4 12 2.4 e6 1.7 e6 3.4 e6 1.8 e6 1e−1 45 3.5 e4 3.2 e4 3.8 e4 1e−3 45 6.9 e4 6.6 e4 7.3 e4 6.9 e4 11 4.3 e6 3.3 e6 5.8 e6 3.0 e6 1e−5 45 1.0 e5 9.8 e4 1.1 e5 1.0 e5 5 1.4 e7 9.2 e6 2.4 e7 4.4 e6 1e−8 45 2.1 e5 1.8 e5 2.3 e5 2.1 e5 0 26e–6 33e–7 23e–1 4.5 e6 f 10 in 5-D, N=45, mFE=5.00 e6 f 10 in 20-D, N=15, mFE=5.00 e6 ∆f # ERT 10% 90% RTsucc # ERT 10% 90% RTsucc 10 45 3.1 e4 2.8 e4 3.4 e4 3.1 e4 0 16e+1 36e+0 29e+1 4.5 e6 1 45 1.6 e5 1.4 e5 1.8 e5 1.6 e5 . . . . . 5.1 e5 . . . . . 1e−1 45 5.1 e5 4.6 e5 5.6 e5 1e−3 39 3.3 e6 2.9 e6 3.7 e6 2.9 e6 . . . . . 1e−5 0 21e–5 29e–6 11e–4 4.5 e6 . . . . . 1e−8 . . . . . . . . . . f 12 in 5-D, N=45, mFE=5.00 e6 f 12 in 20-D, N=15, mFE=5.00 e6 ∆f # ERT 10% 90% RTsucc # ERT 10% 90% RTsucc 10 45 1.4 e4 1.3 e4 1.4 e4 1.4 e4 15 9.1 e4 6.5 e4 1.2 e5 9.1 e4 1 45 4.5 e4 3.8 e4 5.3 e4 4.5 e4 15 4.7 e5 3.6 e5 5.9 e5 4.7 e5 2.0 e5 13 2.3 e6 1.6 e6 3.2 e6 1.9 e6 1e−1 45 2.0 e5 1.7 e5 2.3 e5 1e−3 42 1.9 e6 1.7 e6 2.2 e6 1.8 e6 0 33e–3 11e–4 29e–2 4.5 e6 1e−5 6 3.5 e7 2.3 e7 6.9 e7 5.0 e6 . . . . . 1e−8 0 24e–5 55e–7 90e–5 4.0 e6 . . . . . f 14 in 5-D, N=45, mFE=5.00 e6 f 14 in 20-D, N=15, mFE=5.00 e6 ∆f # ERT 10% 90% RTsucc # ERT 10% 90% RTsucc 10 45 2.4 e1 1.9 e1 2.9 e1 2.4 e1 15 3.1 e3 2.8 e3 3.3 e3 3.1 e3 1 45 1.3 e3 1.2 e3 1.4 e3 1.3 e3 15 1.3 e4 1.2 e4 1.4 e4 1.3 e4 1e−1 45 3.8 e3 3.7 e3 3.9 e3 3.8 e3 15 2.4 e4 2.3 e4 2.5 e4 2.4 e4 1e−3 45 1.0 e4 1.0 e4 1.1 e4 1.0 e4 15 1.8 e5 1.7 e5 1.9 e5 1.8 e5 1e−5 45 9.9 e4 9.1 e4 1.1 e5 9.9 e4 0 34e–6 19e–6 57e–6 4.5 e6 1e−8 6 3.7 e7 2.5 e7 7.3 e7 5.0 e6 . . . . . f 16 in 5-D, N=45, mFE=5.00 e6 f 16 in 20-D, N=15, mFE=5.00 e6 ∆f # ERT 10% 90% RTsucc # ERT 10% 90% RTsucc 10 45 3.1 e2 2.5 e2 3.6 e2 3.1 e2 15 1.3 e4 1.2 e4 1.4 e4 1.3 e4 45 5.2 e3 4.9 e3 5.6 e3 5.2 e3 11 3.1 e6 2.3 e6 4.0 e6 2.6 e6 1 1e−1 45 1.7 e4 1.5 e4 1.8 e4 1.7 e4 0 78e–2 48e–2 11e–1 4.0 e6 1e−3 45 1.4 e5 1.2 e5 1.6 e5 1.4 e5 . . . . . 1e−5 42 1.8 e6 1.5 e6 2.2 e6 1.7 e6 . . . . . 1e−8 6 3.5 e7 2.3 e7 7.0 e7 4.2 e6 . . . . . f 18 in 5-D, N=45, mFE=5.00 e6 f 18 in 20-D, N=15, mFE=5.00 e6 ∆f # ERT 10% 90% RTsucc # ERT 10% 90% RTsucc 10 45 8.4 e2 7.8 e2 9.1 e2 8.4 e2 15 9.2 e3 8.5 e3 9.7 e3 9.2 e3 45 5.3 e3 5.1 e3 5.6 e3 5.3 e3 14 1.7 e6 1.1 e6 2.4 e6 1.6 e6 1 1e−1 45 1.3 e4 1.3 e4 1.4 e4 1.3 e4 0 69e–2 18e–2 96e–2 4.0 e6 1e−3 45 4.6 e5 4.0 e5 5.2 e5 4.6 e5 . . . . . 1e−5 30 4.6 e6 3.8 e6 5.5 e6 3.0 e6 . . . . . 1e−8 0 53e–7 46e–8 15e–5 4.5 e6 . . . . . f 20 in 5-D, N=45, mFE=262762 f 20 in 20-D, N=15, mFE=5.00 e6 ∆f # ERT 10% 90% RTsucc # ERT 10% 90% RTsucc 10 45 2.9 e2 2.6 e2 3.2 e2 2.9 e2 15 4.4 e3 4.0 e3 4.7 e3 4.4 e3 45 6.0 e3 5.7 e3 6.3 e3 6.0 e3 15 4.3 e5 3.2 e5 5.4 e5 4.3 e5 1 1e−1 45 4.8 e4 3.8 e4 5.9 e4 4.8 e4 0 47e–2 39e–2 66e–2 4.5 e6 1e−3 45 5.5 e4 4.5 e4 6.6 e4 5.5 e4 . . . . . 1e−5 45 6.2 e4 5.2 e4 7.3 e4 6.2 e4 . . . . . 1e−8 45 7.5 e4 6.6 e4 8.7 e4 7.5 e4 . . . . . f 22 in 5-D, N=45, mFE=69317 f 22 in 20-D, N=15, mFE=5.00 e6 ∆f # ERT 10% 90% RTsucc # ERT 10% 90% RTsucc 10 45 4.7 e2 4.0 e2 5.4 e2 4.7 e2 15 8.9 e3 8.1 e3 9.6 e3 8.9 e3 45 3.6 e3 3.4 e3 3.8 e3 3.6 e3 15 5.4 e4 3.7 e4 7.4 e4 5.4 e4 1 1e−1 45 8.3 e3 7.6 e3 9.0 e3 8.3 e3 12 1.9 e6 1.0 e6 2.9 e6 1.4 e6 1e−3 45 1.7 e4 1.6 e4 1.8 e4 1.7 e4 12 2.1 e6 1.3 e6 3.2 e6 1.6 e6 1e−5 45 2.5 e4 2.3 e4 2.7 e4 2.5 e4 10 4.9 e6 3.9 e6 6.5 e6 3.4 e6 1e−8 45 4.2 e4 3.9 e4 4.5 e4 4.2 e4 1 7.5 e7 3.7 e7 >7 e7 5.0 e6 f 24 in 5-D, N=30, mFE=5.00 e6 f 24 in 20-D, N=15, mFE=5.00 e6 ∆f # ERT 10% 90% RTsucc # ERT 10% 90% RTsucc 10 30 1.3 e4 1.2 e4 1.4 e4 1.3 e4 0 22e+0 16e+0 31e+0 4.5 e6 28 2.1 e6 1.7 e6 2.4 e6 1.9 e6 . . . . . 1 1e−1 0 75e–2 18e–2 94e–2 4.5 e6 . . . . . 1e−3 . . . . . . . . . . 1e−5 . . . . . . . . . . 1e−8 . . . . . . . . . . ∆f 10 1 1e−1 1e−3 1e−5 1e−8

Table 2: Shown are, for a given target difference to the optimal function value ∆f : the number of successful trials (#); the expected running time to surpass fopt + ∆f (ERT, see Figure 2); the 10%-tile and 90%-tile of the bootstrap distribution of ERT; the average number of function evaluations in successful trials or, if none was successful, as last entry the median number of function evaluations to reach the best function value (RTsucc ). If fopt + ∆f was never reached, figures in italics denote the best achieved ∆f -value of the median trial and the 10% and 90%-tile trial. Furthermore, N denotes the number of trials, and mFE denotes the maximum of number of function evaluations executed in one trial. See Figure 2 for the names of functions.

D=5

D = 20

1.0

1.0 f1-24

+1:21/24

-1:23/24 proportion of trials

proportion of trials

all functions

+1:24/24 0.8

-4:22/24 -8:17/24

0.6

0.4

0.2

0.0 0

1

2

3

4

5

6

0

2

4

6

8

10

12

14

-4:5/5

0.2

1

2

3

4

5

6

0

2

4

6

8

10

12

14

proportion of trials

proportion of trials

moderate fcts

-4:4/4 -8:4/4

0.4

6

8

10

12

14

16

14

16

14

16

18

log10 of Df / Dftarget

f1-5

-4:4/5

0.4

f1-5

1

2

3

4

5

0

2

4

6

8

10

12

18

log10 of Df / Dftarget

0.8

f6-9

-1:3/4 -4:2/4 -8:0/4

0.6

0.4

0.2

f6-9

1

2

3

4

5

6

0

2

4

6

8

10

12

14

0.0 0

16

f6-9

1

2

3

4

5

0

2

4

log10 of FEvals / DIM

log10 of Df / Dftarget

6

8

10

12

18

log10 of Df / Dftarget

1.0 +1:5/5

f10-14

f10-14

+1:4/5

-1:5/5 proportion of trials

proportion of trials

ill-conditioned fcts

4

-1:4/5

+1:4/4

1.0

-4:5/5 -8:1/5

0.6

0.4

0.2

0.8

-1:4/5 -4:1/5 -8:0/5

0.6

0.4

0.2

f10-14

1

2

3

4

5

6

0

2

4

log10 of FEvals / DIM

6

8

10

12

14

0.0 0

16

f10-14

1

2

3

4

5

0

2

4

log10 of FEvals / DIM

log10 of Df / Dftarget

1.0

6

8

10

12

14

16

18

log10 of Df / Dftarget

1.0 +1:5/5

f15-19

f15-19

+1:4/5

-1:5/5 proportion of trials

proportion of trials

2

-8:3/5

f6-9

-1:4/4

log10 of FEvals / DIM

multi-modal fcts

0

log10 of FEvals / DIM

0.2

-4:5/5 -8:4/5

0.6

0.4

0.2

0.8

-1:1/5 -4:0/5 -8:0/5

0.6

0.4

0.2

f15-19

1

2

3

4

5

6

0

2

4

log10 of FEvals / DIM

6

8

10

12

14

0.0 0

16

f15-19

1

2

3

4

5

0

2

4

log10 of FEvals / DIM

log10 of Df / Dftarget

1.0

6

8

10

12

14

16

18

log10 of Df / Dftarget

1.0 f20-24

0.6

0.4 +1:5/5 -1:4/5 0.2

-8:3/5 2

3

4

log10 of FEvals / DIM

5

6

0.8

-1:2/5 -4:2/5 -8:2/5

0.6

0.4

0.2

-4:3/5

1

f20-24

+1:4/5

0.8

proportion of trials

proportion of trials

5

0.6

0.0 0

16

0.6

0.0 0

4

1.0 +1:4/4

weak structure fcts

0.8

log10 of Df / Dftarget

1.0

0.0 0

3

0.2

f1-5

log10 of FEvals / DIM

0.8

2

+1:5/5

proportion of trials

proportion of trials

separable fcts

f1-5

-1:5/5

0.4

0.0 0

f1-24

1

log10 of FEvals / DIM

-8:5/5

0.8

0.4

0.0 0

16

0.6

0.0 0

-8:5/24

1.0 +1:5/5

0.8

-4:9/24

0.6

log10 of Df / Dftarget

1.0

0.0 0

f1-24

-1:14/24

0.2

f1-24

log10 of FEvals / DIM

0.8

0.8

f20-24 0

2

4

6

8

10

12

log10 of Df / Dftarget

14

16

0.0 0

f20-24

1

2

3

4

log10 of FEvals / DIM

5

0

2

4

6

8

10

12

14

16

18

log10 of Df / Dftarget

Figure 3: Empirical cumulative distribution functions (ECDFs), plotting the fraction of trials versus running time (left subplots) or versus ∆f (right subplots). The thick red line represents the best achieved results. Left subplots: ECDF of the running time (number of function evaluations), divided by search space dimension D, to fall below fopt + ∆f with ∆f = 10k , where k is the first value in the legend. Right subplots: ECDF of the best achieved ∆f divided by 10k (upper left lines in continuation of the left subplot), and best achieved ∆f divided by 10−8 for running times of D, 10 D, 100 D . . . function evaluations (from right to left cycling blackcyan-magenta). Top row: all results from all functions; second row: separable functions; third row: misc. moderate functions; fourth row: ill-conditioned functions; fifth row: multi-modal functions with adequate structure; last row: multi-modal functions with weak structure. The legends indicate the number of functions that were solved in at least one trial. FEvals denotes number of function evaluations, D and DIM denote search space dimension, and ∆f and Df denote the difference to the optimal function value.