Group ranking r&d projects by discordant multiple criteria estimates

Размещено на http://www.allbest.ru/

GROUP RANKING R&D PROJECTS BY DISCORDANT MULTIPLE CRITERIA ESTIMATES This work is partially supported by the Russian Academy of Sciences, Research Programs “Intellectual Information Technologies, Mathematical Modeling, Systems Analysis, and Automation”, “Information Technologies and Methods for Complex Systems Analysis”, the Russian Foundation for Basic Research (projects 08-01-00247, 09-07-00009, 09-07-12111).

Alexey B. Petrovsky¹, Gregory V. Royzenson¹, Igor P. Tikhonov², Alexander V. Balyshev²

¹Institute for Systems Analysis, Russian Academy of Sciences,

Prospect 60 Let Octyabrya, 9, Moscow 117312, Russia, pab@isa.ru, rgv@isa.ru

²Russian Foundation for Basic Research, Moscow 119991, Russia

Abstract: There are many real-life decision problems, where objects are compared and arranged by their properties. Among these problems, there is evaluating the efficiency of R&D projects, which are estimated with many quantitative and/or qualitative criteria and may exist in several copies. These projects have been ranked by efficiency using the technique for reducing the multi-attribute space dimension, and ARAMIS method for group ordering multi-attribute objects.

Keywords: project efficiency, verbal decision analysis, group ordering, multi-attribute objects

project multi space dimension

1. Introduction

Traditionally business activity and position of the enterprise in any market sector are evaluated with various economical, financial, organizational, operational, and other indicators. Nature of indicators depends on the aim of analysis. Some indicators are measured or calculated, whereas the others have to be estimated. Ranking organizations by activity indicators is one of the famous decision aiding problems. The situation becomes more complicated when indicators of business activity are estimated differently by several independent experts under multiple criteria. In these cases, the same object can exist in multiple versions or copies with diverse, inconsistent or contradictory descriptions, and convolution of numerical and/or verbal estimates is either mathematically incorrect or impossible.

Various methods for ranking multi-attribute objects have been developed (Hwang and Lin, 1987; Larichev and Olson, 2001; Pawlak and Slowinski, 1994; Roy, 1996; Saaty, 1990; Zimmerman, Zadeh and Gaines, 1984). The most of these methods operate with numerical attributes. In the presence of many criteria and several decision makers, it's necessary to aggregate individual rankings on separate criteria and separate persons, which, as a rule, do not coincide. To construct the generalized rankings, one can use any voting procedures, as Bordeaux scores or Kemeny median (Hwang and Yoon, 1987; Kemeny and Snell, 1972; Mirkin, 1980; Petrovsky, 2009). But in such cases, a large number of individual different rankings are to be aggregated. So, we need in special techniques for ranking such multi-attribute objects.

In this paper, we propose the approach to compare objects by the complex indicator that integrates multiple criteria estimates. This approach combines methods of verbal decision analysis and the technique for reducing the multi-attribute space dimension, and deals with many numerical, symbolic and verbal attributes. We also use ARAMIS method for group ordering multi-attribute objects that does not require pre-construction of individual rankings. Multi-attribute objects are represented as multisets and arranged by their closeness with regard to any `ideal' object in a multiset metric space. The suggested approach is applied to compare efficiency of R&D projects, which are subsidized by the Russian Foundation for Basic Research.

2. Multiple criteria expertise of R&D projects

The Russian Foundation for Basic Research (RFBR) is the Federal agency that organizes and funds basic research, and exams their practical application. One of the important RFBR activities is evaluation of efficiency of the goal-oriented projects, which have been performed in the interest of Federal agencies and Departments of Russia. In RFBR, there is the special original expertise for selection of the applications and assessment of the completed projects - the peer review system, similar to that found nowhere else in the world.

Each project is estimated independently by several experts without the consent of their views. The experts are well-known specialists working in research institutes, universities, industrial organizations. To assess the content of application and the results obtained experts use specific qualitative criteria with detailed verbal rating scales. In addition, the expert gives his opinion on whether to support the project (at the competition stage), whether to continue the project (at the intermediate stage), and what is scientific and practical value of the obtained result (at the end of the project). Expert opinions, of course, may be the same or different. On the basis of expert recommendations, the Expert Board of RFBR decides to approve or reject the new project, to continue the project implementation, and evaluates the project efficiency.

Numerous domestic and foreign technologies for expertise of programs and projects of various kinds are known and widely used in practice. Mention such tools as `Peer review', `Cost-Effectiveness', `Programming-Planning-Budgeting', `Balanced Score Card' and others. The majority of methodologies, which are applied for expert estimation of different objects, uses quantitative approach that is based on a numerical measurement of object characteristics.

However, such quantitative approaches are not suitable for the expertise used in RFBR, where projects are evaluated by several experts on many qualitative criteria with verbal scales. For instance, the obtained results of goal-oriented projects are estimated by the following criteria: K₁“Degree of the problem solution”, K₂“Scientific level of results”, K₃“Appropriateness of patenting results”, K₄“Prospective application of results”, K₅“Result correspondence to the project goal”, K₆“Achievement of the project goal”, K₇“Difficulties of the project performance”, K₈“Interaction with potential users of results”. Each criterion has 2- or 3-point scale of ordered verbal grades. For example, the scale of the criterion “Degree of the problem solution” looks as follows: q₁¹ - the problem is solved completely, q₁² - the problem is solved partially, q₁³ - the problem is not solved. The “Achievement of the project goal” is estimated as q₆¹ - really, q₆² - non-really.

For multi-aspect assessment of the content and results of goal-oriented research we used methods of the verbal decision analysis and the original interactive procedure for reducing the dimension of attribute space.

3. Multiple criteria evaluation of research efficiency

In the previous papers, we suggested the special hierarchical aggregation approach to building the integrated criterion of R&D project efficiency (Petrovsky and Royzenson, 2008; Petrovsky, Royzenson and Tikhonov, 2009). Construction of this indicator is considered as the problem of ordinal classification with reducing the multi-attribute space dimension. The classified objects are combinations of multiple criteria estimates of projects in the attribute space K₁…K₈. The ordered classes are rates of the project efficiency, which correspond to grades on a scale of the top complex criterion D “Project efficiency” as d¹ - superior, d² - high, d³ - average, d⁴ - low, d⁵ - unsatisfactory.

The grades of the complex criterion scale was formed with the following different methods of verbal decision analysis: M₁ - ORCLASS method on all levels of the criteria hierarchy (OC); M₂ - stratification of tuples on all levels of the criteria hierarchy (ST), M₃ - stratification of tuples on the lower level of the criteria hierarchy, and ORCLASS method on the upper level of the criteria hierarchy (ST+OC); M₄ - ORCLASS method on the lower level of the criteria hierarchy, and stratification of tuples on the upper level of the criteria hierarchy (OC+ST). Examples of the complex indicators of project efficiency estimated by two experts are shown on Fig.1.

Fig.1. Examples of the integrated indicators of project efficiency.

To select the best projects we applied ARAMIS (Aggregation and Ranking Alternatives nearby the Multi-attribute Ideal Situations) method that allow us to order multi-attribute objects on the bases of discordant preferences of several decision makers (Petrovsky, 2008; Petrovsky, this volume). In this method, objects described by many repeated quantitative and/or qualitative attributes Q₁,…,Q_m are represented as multisets. Considering multi-attribute objects A₁,...,A_n as points of multiset metric space (A,d) with any distance d (Petrovsky, 2003), we can compare and arrange objects with respect to closeness to the best (ideal) object A_max or worst (anti-ideal) object A_min in this space. The best and worst objects (may be hypothetical) have the highest and lowest estimates by all criteria Q_s. All objects are ordered in accordance with the index of relative closeness to the best object l(A_i)=d(A_max,A_i)/[d(A_max,A_i)+d(A_min,A_i)], where d(A_max,A_i) is the distance from the best object A_max, and d(A_min,A_i) is the distance from the worst object A_min.

Let us consider methods M₁, M₂, M₃, M₄, which are used by experts to assess the project efficiency, as the new attributes that characterize the projects. Every attribute M_j takes values

m_j¹, m_j², m_j³, m_j⁴, m_j⁵, corresponding the grades d¹, d², d³, d⁴, d⁵ of the scale of the integrated criterion D “Project efficiency”. We now represent each project A_i as the following multiset A_i over the set of methods, the hyperscale X=M₁M₂M₃M₄, or as set of repeating attributes:

A_i={k_Ai(m₁¹)?m₁¹,…,k_Ai(m₁⁵)?m₁⁵;…; k_Ai(m₄¹)?m₄¹,…,k_Ai(m₄⁵)?m₄⁵}

Here multiplicity k_Ai(m_j^hj), h_j=1,…,5, j=1,…,4 of each attribute value in the multiset A_i indicates how many times the method m_j^hj is used by all experts during formation of the appropriate grades of efficiency; the sign ? denotes that there are k_Ai(m_j^hj) copies of attribute m_j^hj within the description of object A_i.

For instance, the projects A₁ and A₂ shown on Fig.1 are represented as the following multisets

A₁={1?m₁¹, 0?m₁², 1?m₁³, 0?m₁⁴, 0?m₁⁵; 1?m₂¹, 1?m₂², 0?m₂³, 0?m₂⁴, 0?m₂⁵;

1?m₃¹, 0?m₃², 1?m₃³, 0?m₁⁴, 0?m₁⁵; 1?m₄¹, 1?m₄², 0?m₄³, 0?m₄⁴, 0?m₄⁵},

A₂={0?m₁¹, 1?m₁², 1?m₁³, 0?m₁⁴, 0?m₁⁵; 0?m₂¹, 2?m₂², 0?m₂³, 0?m₂⁴, 0?m₂⁵;

0?m₃¹, 1?m₃², 1?m₃³, 0?m₁⁴, 0?m₁⁵; 0?m₄¹, 2?m₄², 0?m₄³, 0?m₄⁴, 0?m₄⁵},

the best project A_max and worst project A_min are represented as the multisets

A_max={2?m₁¹,0,…,0; 2?m₂¹,0,…,0; 2?m₃¹,0,…,0; 2?m₄¹,0,…,0},

A_min={0,…,0, 2?m₁⁵; 0,…,0, 2?m₂⁵; 0,…,0, 2?m₃⁵; 0,…,0, 2?m₄⁵}.

The model database of the results of goal-oriented research includes expert assessments of projects on Mathematics, Mechanics and Computer Science (total 48 projects), Chemistry (total 54 projects), Information and telecommunication resources (total 21 projects), which have been completed in 2007. So, the final ranking projects on Mathematics, Mechanics and Computer Science in accordance with the index l(A_i) looks as follows:

rank 1 (l=0,333) - projects A₁, A₅, A₇, A₈, A₁₀, A₁₃, A₁₄, A₁₇, A₂₂, A₂₃, A₂₅, A₂₆, A₂₈, A₃₀,

A₃₁, A₃₂, A₃₃, A₃₄, A₃₆, A₃₉, A₄₁, A₄₃, A₄₅;

rank 2 (l=0,429) - project A₁₉;

rank 3 (l=0,500) - projects A₂, A₃, A₄, A₆, A₉, A₁₁, A₁₂, A₁₅, A₁₆, A₁₈, A₂₀, A₂₁, A₂₄, A₂₇,

A₂₉, A₃₅, A₃₇, A₃₈, A₄₀, A₄₂, A₄₄, A₄₆, A₄₇, A₄₈.

Projects of rank 1 have the superior complex efficiency, of rank 2 - the more than high complex efficiency, of rank 3 - the high complex efficiency.

Conclusion

In this paper, we suggested the transparent approach to evaluate efficiency of R&D project estimated by several experts upon many numerical, symbolic and verbal criteria. Using the technique for reducing the multi-attribute space dimension we constructed the complex indicator of project efficiency and determined the project efficiency in various ways. Using ARAMIS method for group ordering multi-attribute objects, that is based on the theory of multiset metric spaces, we ranked the projects and found the most effective projects. The multiset approach allows us to discover, present and utilize the available information, to analyze the obtained results and their peculiarities, especially for inconsistent multiple criteria estimatess of projects and contradictory preferences of decision makers. The proposed tools have been testified on goal-oriented projects, which are supported by the Russian Foundation for Basic Research, and demonstrated the high ability and simplicity of application.

References

Hwang, C.L. and Lin, M.J. (1987); Group Decision Making under Multiple Criteria; Springer-Verlag, Berlin.

Hwang C.L. and Yoon K. (1981); Multiple Attribute Decision Making - Methods and Applications: A State of the Art Survey; Springer-Verlag, New York.

Kemeny J.G. and Snell J.L. (1972); Mathematical Models in the Social Sciences; The MIT Press, Cambridge.

Larichev, O.I. and D.L.Olson (2001); Multiple Criteria Analysis in Strategic Siting Problems; Kluwer Academic Publishers, Boston.

Mirkin B.G. (1980); Analysis of Qualitative Attributes and Structures; Statistika, Moscow (in Russian).

Pawlak, Z. and Slowinski, R. (1994); Rough Set Approach to Multi-Attribute Decision Analysis; Europ. J. Op. Res. 72 (pp. 443-459).

Petrovsky, A.B. (2003); Spaces of Sets and Multisets; Editorial URSS, Moscow (in Russian).

Petrovsky, A. (2008); Group Verbal Decision Analysis; Encyclopedia of Decision Making and Decision Support Technologies (ed. by F.Adam, P.Humphreys); Hershey, IGI Global, (pp. 418-425).

Petrovsky, A.B. and G.V.Royzenson (2008); Sorting multi-attribute objects with a reduction of space dimension; Advances in Decision Technology and Intelligent Information Systems, Vol.IX (ed. by K.J. Engemann, G.E.Lasker); The International Institute for Advanced Studies in Systems Research and Cybernetics, Tecumseh, Canada (pp.46-50).

Petrovsky A.B., Royzenson G.V. and Tikhonov I.P. (2009); Hierarchical Aggregation Approach to Building the Integrated Criterion of R&D Project Efficiency; Advances in Decision Technology and Intelligent Information Systems, Vol.X (ed. by K.J. Engemann, G.E.Lasker); The International Institute for Advanced Studies in Systems Research and Cybernetics, Tecumseh, Canada (pp.26-30).

Petrovsky, A.B. (2009); Theory of Decision Making; Academia, Moscow (in Russian).

Roy, B. (1996); Multicriteria Methodology for Decision Aiding; Kluwer Academic Publishers, Dordrecht.

Saaty, T. (1990); Multicriteria Decision Making: The Analytic Hierarchy Process; RWS Publications, Pittsburgh.

Zimmerman H.J., Zadeh L.A. and Gaines B.R. (1984); Fuzzy Sets and Decision Analysis; North-Holland, Amsterdam.

Размещено на Allbest.ru