A HYBRID APPROACH TO APPLY DEMATEL IN A MULTI-CRITERIA SETTING

Frederico Silva Valentim Sallum

SENSE Company, Brazil

E-mail: frederico.sallum@wearesense.company

Luiz Flavio Autran Monteiro Gomes

Ibmec/RJ, Brazil

E-mail: luiz.gomes@ibmec.edu.br

Maria Augusta Soares Machado

Ibmec/RJ, Brazil

E-mail: maria.machado@ibmec.edu.br

Leonardo Silva Valentim Sallum

IBMR University Center, Brazil

E-mail: leonardosvsallum@gmail.com

Submission: 4/8/2019

Accept: 5/2/2019

ABSTRACT

The DEMATEL method has been applied in the Decision Sciences in several studies. However, one has not been able to apply DEMATEL directly to a multi-criteria matrix formed by a set of alternatives and a set of criteria yet. In order to approach this, we propose a novel way to apply DEMATEL to a multi-criteria matrix for ranking a set of alternatives according to their performances in a set of criteria. For accomplishing this, we consider the set of alternatives in a classical multi-criteria problem as the set of components used in a usual DEMATEL application. To set up the influence degree among studied components, we used the preference index of PROMÉTHÉE II. Such preference index takes into consideration the performances of alternatives on all criteria to establish each influence degree. Thereby, we denote the influence degree by preference degree. This new approach is applied to a case study and results are compared against those of three multi-criteria methods. It is then possible to note small, understandable differences among the rankings. This hybrid approach has therefore shown to be theoretically sound and feasible to be used in the practice of Multi-Criteria Decision Analysis.

Keywords: DEMATEL method; preference degree; Multiple Criteria Decision Making; PROMÉTHÉE II method

1. INTRODUCTION

The DEMATEL (Decision Making Trial and Evaluation Laboratory) method was developed by Gabus and Fontela (1972) for the structuring of complex problems (GABUS; FONTELA, 1973; FONTELA; GABUS, 1976). The complex problems studied by DEMATEL are based on a set of components in which each component can exert an influence degree over each other (LI et al., 2014; Gölcük; Baykasoğlu, 2016). That is, this method can be used whenever there is a cause-effect relation among the studied components. DEMATEL can therefore confirm the interdependence between variables in a decision system (TSENG, 2011).

In the cases studied by DEMATEL, one seeks to identify the received and exerted impacts from each component. After this, this method computes the full involvement and the net effect of each component in the decision system (Altuntas; Dereli, 2015). Through these attributes, it is possible to develop the impact-relationship map, a two-dimensional chart used to illustrate the causal relations among components in the studied system (WANG, 2012). The DEMATEL method has thus been applied in several fields of study for selection, industrial planning, competence evaluation, etc. (SALLUM; GOMES; MACHADO, 2018; SHIEH; WU; HUANG, 2010).

Among the various cases that have been approached by DEMATEL, one should highlight problems of a multiple criteria nature as some of the most important ones. It is important to emphasize that DEMATEL is not a multiple criteria method in itself. However, it has been applied in combination with Multiple Criteria Decision Making (MCDM) methods in many situations (QUEZADA et al., 2018; Baykasoğlu; Gölcük, 2017). DEMATEL has been used in MCDM field mainly to identify interrelations in a set of criteria (HSU, 2012). Besides, DEMATEL results can also be used in many other steps of a hybrid MCDM template.

Considering the potential for compatibility between DEMATEL and the MCDM field, we propose in this paper an approach to use directly the DEMATEL method in a classical MCDM problem. In order to accomplish this, the influence degree between the studied components (alternatives) must be determined through the behavior of each alternative on all criteria of a multiple criteria matrix. We go to reinterpret some DEMATEL issues in order to take its methodology for obtain a ranking of alternatives based in the mutual influence between alternatives, according to their behaviors in the criteria.

A classical MCDM problem is given in a multi-criteria matrix, a matrix formed by a set of alternatives and a set of criteria (VELASQUEZ; HESTER, 2013; BELTON; STEWART, 2002). Then, to identify the influence degree between studied alternatives we use a MCDM method. For this, we use the PROMÉTHÉE II (Preference Ranking Organization Method for Enrichment of Evaluations II) method’s preference index in order to obtain the influence degree of each alternative over the others. For this reason, we call the influence degree as preference degree. This happen because the preference degree is performed without there is a formal cause-effect relation between alternatives. The preference degree measures how much an alternative is preferable to the other according to its behaviors on all criteria.

As the PROMÉTHÉE II method performs a pairwise comparison between alternatives criterion by criterion, it is possible to know on a 0 to 1 level the preference of each alternative over the others through its preference index (BRANS; VINCKE, 1985). Thus, with this methodology, we can perform the preference degree between studied alternatives and, after that, to apply DEMATEL. For test this approach, we use a part of the Jati and Dominic’s article (2017) database as a case study and to apply it in order to obtain a ranking of alternatives through DEMATEL methodology from a multi-criteria matrix.

2. DEMATEL

The basic element to apply the DEMATEL method is a set with 2 or more components (alternatives or criteria) forming a decision system. In this set, the decision maker should establish the influence degree that each component exerts over the each other. For this, it should be used a verbal scale from 0 to 4 in which: 0 is no influence; 1 is low influence; 2 is moderate influence; 3 is high influence; and 4 is very high influence. This scale can be extended according to the decision maker’s preference. Through the influence degree values the direct-influence matrix A should be built. The matrix A is a square matrix where the numbers of rows and columns are equal to the components number n. Thus, a_ij is the influence that element i in the matrix’s row exerts over element j in the matrix’s column. The diagonal of the matrix values must be 0, because one component cannot exert influence over itself.

After to build the matrix A, it should be calibrate according to the Equations 1 and 2 generating the direct-relation matrix D.

(1)

Where

(2)

Next, the total-influence matrix T should be calculated through the Equation 3:

(3)

In Equation 3, I is an identity matrix and (I-D)^-1 is an inverse matrix. Then, the sum of each row r_i of matrix T (Equation 4) should be calculated as well as the sum of each column ci of matrix T (Equation 5).

	(4)
	(5)

The sum of rows of each component (r_i) represents the impact of one component over the others in the system. The sum of each column (c_i) represents the received impact of each component by the others in the system. The addition r_i+c_i should also be made for each component, representing the full involvement of a component in the system. Similarly, the subtraction r_i-c_i represents the net effect of a component in the system.

In addition, an impact-relations graph can be built. This graph is created to visualize the causal relationships between components in the system, where r_i+c_i is the horizontal axis and r_i-c_i the vertical axis. The result of r_i-c_i, if positive, classifies a component as an influencer inside the system. That is because its exerted impact is higher than its received impact. Moreover, the result of r_i-c_i, if negative, classifies a component as influenced inside the system, that is, its exerted impact is lower than its received impact.

3. THE PROMÉTHÉE II-DEMATEL APPROACH

The approach here presented aims to apply DEMATEL directly in a MCDM problem. Although DEMTAEL works with cause-effect relations between its studied components, we believe that the classical MCDM problems are questions that can be solved by DEMATEL. This occurs because in a set of alternatives studied in a multi-criteria matrix some alternatives can be obtain a good performance just in some criteria and other alternatives can be obtain a good performance on other criteria, framing the decision-making difficult (Pomerol; Barba-Romero, 2000; Belton; Stewart, 2002). This kind of conflict observed in those cases remits to the problems of interrelations between components that can be solved by DEMATEL.

By doing this, for apply DEMATEL, firstly, we should establish the influence degree that each component exerts over the others. As we work with a MCDM problem, the studied components are a set of alternatives and the influence degree of each alternative over the others should be extracted from the behaviors of studied alternatives on designed criteria for the studying.

As in a MCDM problem there is not, necessarily, a formal cause-effect relation between alternatives, considering the interdependence between criteria, we call the influence degree as preference degree. The goal of preference degree is identifying how much an alternative is better the other in a multi-criteria context. When we have the preference degree of each alternative over the others in a decision system, this problem can be solved by the DEMATEL methodology.

Then, for measure the preference degree of each alternative over the others through their behaviors on the studied criteria; we use the PROMÉTHÉE II method’s preference index. This index is used because measures the value of all differences between pair of alternatives on all criteria. Besides, positives and negatives values are measure reciprocally.

Not all MCDM methods measure the values of all performance differences between alternative pairs, criterion by criterion, for example the ELECTRE (ÉLimination Et Choix Traduissant la REalité) methods (POMEROL; BARBA-ROMERO, 2000). Besides, other multi-criteria methods do not make all possible comparisons between alternatives. That is, after to designee the first distance or preference between two alternatives on a certain criterion, the second of the same pair is an inverse value from the first. As examples this, we can notice the AHP (Analytic Hierarchy Process), ANP (Analytic Network Process) and MACBETH (Measuring Attractiveness by a Categorical Based Evaluation Technique) methods (BELTON; STEWART, 2002; SAATY, 1980; SHARMA; GARG, 2015; AKYÜZ; TOSUN; AKA, 2018).

The TODIM (TOmada de Decisão Interativa e Multicritério) method makes a comparison between pairs of alternatives, criterion by criterion, but it measures in a dissimilarity way positive and negative differences (GOMES; LIMA, 1991). Some methods, such as TOPSIS (Technique for Order of Preference by Similarity to Ideal Solution) and VIKOR (VlseKriterijuska Optimizacija I Komoromisno Resenje), do not make a pairwise comparison between pair of alternatives, because they are based on a compromise solution (OPRICOVIC; TZENG, 2004). For those reasons, we understand the PROMÉTHÉE II methodology as the best representative way to measure the preference degree.

3.1. Setting the preference degree through PROMÉTHÉE II method’s preference index

For the PROMÉTHÉE II method’s preference index calculating is necessary a set of alternatives A = {a_k |k=1, …, n}, where k is a generic alternative’s ordinal number and n is the maximum number of alternatives in the multi-criteria matrix; a set of criteria C = {c_j |j=1, …, m}, where j is each criterion c_j’s ordinal number and m is the maximum number of criteria in the same multi-criteria matrix; X = {x_kj |k=1, …, n; j=1, …, m}, where x_kj is the performance evaluation of each alternative k according to each criterion j; and W = {w_j |j=1, …, m} is the set of weights w_j assigned to each criterion according to the decision maker's preferences.

After that, the decision maker must choose a preference function for each criterion. The preference function is the way to measure the difference of performances between alternatives on each criterion. There are 6 preference function types. We suggest in this approach the use of the V-Shape (Type 3) preference function on all criteria. This preference function measures the difference of performances between alternatives on each criterion using all values from 0 to 1 with only one threshold, the preference threshold p.

Some preference functions use just some specific values to measure those differences. Thus, the V-Shape preference function measures the differences between alternatives in a more sensitive way than those. The Linear (Type 5) preference function also uses all values from 0 to 1 but using 2 thresholds. It is important emphasize here that we do not express decision maker’s preferences. That is, we use a part of PROMÉTHÉE II method in order to calculate the preference degree that each alternative exerts over the others according to their behaviors in the studied criteria.

For compute the threshold p for each criterion using V-Shape preference function, the threshold p should be computed by Equation 6.

(6)

In the Equation 6, p_j is the preference threshold of the criterion j; a_j⁺ is the value of the best performance in criterion j; and the a_j^- is the worst performance in criterion j. In this way, the larger difference between 2 performances is measured as 1. Differences between performances that results 0 are measured as 0. Any other value from those differences is measured proportionally between the larger difference value and the difference equals to 0.

Next, the value of the V-Shape preference function for each pair of alternatives must be computed according to Equation 7.

(7)

In Equation 7, P_j (a₁,a₂) is the value of the preference index of a₁ over a₂ in criterion j; d_j (a₁,a₂) is the performance differences between the alternatives a₁ and a₂ in criterion j; p is the preference threshold of the criterion j.

Finally, we can perform the preference degree for each pair of alternatives through the preference index following Equation 8.

(8)

In Equation 8, π (a₁,a₂) is the preference index of a₁ over a₂; w_j is the weight of the criterion j. By doing this, π (a₁,a₂) is the value of the preference degree of a₁ over a₂, which is a number from 0 to 1.

After calculating the preference degree of each alternative over the others, the direct-influence matrix A of the DEMATEL method can be build. Now, to provide continuity to this approach, it is enough to apply DEMATEL following the steps outlined in the Section 2.

4. CASE SUDY

Jati and Dominic (2017) approached in their article the problem of to rank 27 Indonesian universities’ websites considering 4 criteria. For this, they applied the PROMÉTHÉE II method. The criteria used by those authors with their weights computed by Entropy method and their sense (maximization or minimization) were:

· C₁: Presence: number of pages of the main web domain of the institution, it includes all sub domains that share the web domain and all types of files including PDF. Its weight is 0.17. This criterion should be maximized;

· C₂: Visibility: number of external networks that have backlinks to the institutions' web pages. Its weight is 0.33. This criterion should be maximized;

· C₃: Openness: number of citations from the main authors by Google Scholar citations. Its weight is 0.23. This criterion should be maximizing; and

· C₄: Excellence: number of academic articles published in international journals of high impact among the 10% most cited in their respective scientific disciplines which data is provided by Grup Scimago (2010-2014). Its weight is 0.27. This criterion should be maximized.

For this case study, we use the 10 best universities ranked by Jati and Dominic (2017) in order to form the set of alternatives. We also use the same criteria and their respective weights. Then, our goal is to rank those 10 universities through their performances on the criteria explained above by DEMATEL. The name of each university and their performances on each criterion are shown by Table 1.

Table 1: University’s performances on each criterion

University	Presence	Visibility	Openness	Excellence
UI	2,560,000	182,006	10,166	2,353
ITB	1,227,143	64,899	18,210	1,303
UGM	2,583,571	81,068	28,646	1,322
UNDIP	1,385,714	32,556	10,145	475
UB	1,642,857	7,530	7,302	611
IPB	2,782,857	41,688	5,997	247
UNPAD	453,071	129,457	5,849	360
UNHAS	912,214	20,894	1,752	392
UNAIR	985,357	12,977	3,032	333
ITS	944,929	40,727	6,009	299

Source: Jati and Dominic (2017)

4.1. Preference Degree Setting

Now, before applying DEMATEL, we should establish the preference degree of each alternative over each other one. For this, we must calculate the preference index of the PROMÉTHÉE II method. The first step for this is to calculate the threshold p for each one of 4 criteria that is necessary to use the V-Shape preference function. Table 2 shows the threshold values for each criterion calculated according Equation 6.

Table 2: Criteria thresholds

Criteria	a^+	a^-	p
Presence	2,782,857	453,071	2,329,786
Visibility	182,006	7,530	174,476
Openness	28,646	1,752	26,894
Excellence	2,353	247	2,106

After to calculate the threshold of each criterion, we should perform the value of preference function following Equation 7. Next, we can achieve the preference index value that is the preference degree value accordingly with Equation 8. Table 3 presents the value which each alternative exerts over each other alternative.

Table 3: Direct-influence matrix A with the preference degrees values

University	UI	ITB	UGM	UNDIP	UB	IPB	UNPAD	UNHAS	UNAIR	ITS
UI	0.0000	0.4534	0.3231	0.6093	0.6447	0.5710	0.5456	0.7483	0.7546	0.6839
ITB	0.0688	0.0000	0.0000	0.2363	0.2905	0.2837	0.2831	0.3638	0.3700	0.2994
UGM	0.1598	0.2212	0.0000	0.4460	0.4814	0.4060	0.4738	0.5850	0.5913	0.5206
UNDIP	0.0000	0.0116	0.0000	0.0000	0.0716	0.0647	0.1195	0.1390	0.1453	0.0901
UB	0.0000	0.0303	0.0000	0.0362	0.0000	0.0578	0.1314	0.1289	0.1201	0.1020
IPB	0.0163	0.1135	0.0145	0.1192	0.1478	0.0000	0.1713	0.2121	0.2108	0.1359
UNPAD	0.0000	0.1221	0.0915	0.1833	0.2306	0.1805	0.0000	0.2404	0.2479	0.1756
UNHAS	0.0000	0.0000	0.0000	0.0000	0.0253	0.0186	0.0376	0.0000	0.0225	0.0119
UNAIR	0.0000	0.0000	0.0000	0.0000	0.0103	0.0110	0.0388	0.0163	0.0000	0.0073
ITS	0.0000	0.0000	0.0000	0.0155	0.0628	0.0068	0.0373	0.0763	0.0779	0.0000

In Table 3, the diagonals are equal 0 because one alternative does not exert influence on itself. In addition, when we see the first row with second column, we see the number 0.4534 that is the preference degree from UI over ITB and as so on for the other cases.

Then, through the Table 3 data, we can apply DEMATEL following the Section 2 steps. Table 3 is the direct-influence matrix A. The total-influence matrix T is presented by Table 4.

Table 4: Total-influence matrix T

University	UI	ITB	UGM	UNDIP	UB	IPB	UNPAD	UNHAS	UNAIR	ITS
UI	0.0035	0.0946	0.0633	0.1327	0.1459	0.1260	0.1275	0.1747	0.1763	0.1532
ITB	0.0132	0.0043	0.0020	0.0502	0.0627	0.0589	0.0610	0.0796	0.0810	0.0641
UGM	0.0310	0.0494	0.0039	0.0963	0.1076	0.0891	0.1054	0.1333	0.1348	0.1146
UNDIP	0.0001	0.0031	0.0005	0.0015	0.0155	0.0135	0.0240	0.0287	0.0300	0.0187
UB	0.0001	0.0066	0.0005	0.0084	0.0024	0.0124	0.0262	0.0270	0.0255	0.0210
IPB	0.0035	0.0229	0.0036	0.0257	0.0324	0.0040	0.0364	0.0460	0.0459	0.0301
UNPAD	0.0010	0.0250	0.0174	0.0386	0.0490	0.0382	0.0073	0.0536	0.0551	0.0392
UNHAS	0.0000	0.0003	0.0001	0.0004	0.0052	0.0038	0.0074	0.0007	0.0049	0.0027
UNAIR	0.0000	0.0002	0.0001	0.0004	0.0024	0.0024	0.0075	0.0036	0.0006	0.0018
ITS	0.0000	0.0003	0.0001	0.0033	0.0123	0.0018	0.0077	0.0152	0.0155	0.0007

Now, we can to calculate the DEMATEL results. Table 5 presents DEMATEL results.

Table 5: DEMATEL method results

University	r	c	r+c	r-c
UI	1.1978	0.0524	1.2502	1.1454
ITB	0.4770	0.2069	0.6838	0.2701
UGM	0.8655	0.0916	0.9571	0.7738
UNDIP	0.1356	0.3575	0.4931	-0.2219
UB	0.1302	0.4355	0.5658	-0.3053
IPB	0.2504	0.3502	0.6007	-0.0998
UNPAD	0.3244	0.4103	0.7347	-0.0859
UNHAS	0.0257	0.5625	0.5882	-0.5367
UNAIR	0.0190	0.5695	0.5885	-0.5505
ITS	0.0570	0.4461	0.5031	-0.3891

As our goal is to establish a ranking of alternatives, we use the net effect value r-c in order to obtain this ranking. The net effect represents the difference between the exerted and received impacts of each alternative. For this reason, we choose it to rank the set of alternatives. Table 6 presents the ranking of alternatives performed by DEMATEL.

Table 6: Ranking of alternatives

Rank	University	r-c
1^st	UI	1.1454
2^nd	UGM	0.7738
3^rd	ITB	0.2701
4^th	UNPAD	-0.0859
5^th	IPB	-0.0998
6^th	UNDIP	-0.2219
7^th	UB	-0.3053
8^th	ITS	-0.3891
9^th	UNHAS	-0.5367
10^th	UNAIR	-0.5505

Observing the ranking in Table 6, we can notice that UI, UGM, and ITB are better alternatives than the others. This is happening because those universities are classified by DEMATEL as influencers. In the influencers group, UI is the best university.

4.2. Comparing PROMÉTHÉE II-DEMATEL Approach with other MCDM Methods

In this Section, we compare the PROMÉTHÉE II-DEMATEL approach results against the results reached by other MCDM methods for the same database presented in Table 1 using the same weights of criteria. Next, we use the PROMÉTHÉE II method using the Usual Preference Function (Type I) for all criteria. The Usual Preference Function will be used because the authors Jati and Dominic (2017) also use this preference function for all criteria in their original database.

We then use the TOPSIS method (Hwang; Yoon, 1981; SREENIVASULU; SRINIVASARAO, 2016) and the TODIM method (Gomes; RANGEL, 2009). In the TODIM application we use θ equal 1.0 for all criteria. Table 7 presents the rankings reached by PROMÉTHÉE II-DEMATEL approach, PROMÉTHÉE II method, TOPSIS method and TODIM method.

Table 7: Rankings by PROMÉTHÉE II-DEMATEL approach, PROMÉTHÉE II method, TOPSIS method and TODIM method

Rank	PROMÉTHÉE II- DEMATEL	PROMÉTHÉE II	TOPSIS	TODIM
1^st	UI	UI	UI	UI
2^nd	UGM	UGM	UGM	UGM
3^rd	ITB	ITB	ITB	ITB
4^th	UNPAD	UNDIP	UNPAD	IPB
5^th	IPB	UNPAD	IPB	UNDIP
6^th	UNDIP	IPB	UNDIP	UB
7^th	UB	UB	UB	UNPAD
8^th	ITS	ITS	ITS	ITS
9^t^h	UNHAS	UNHAS	UNHAS	UNAIR
10^th	UNAIR	UNAIR	UNAIR	UNHAS

Observing the results from Table 7, PROMÉTHÉE II, TOPSIS and TODIM are unanimous in ranking UI, UGM and ITB as the three top universities just like the PROMÉTHÉE II-DEMATEL approach did. This fact is more telling in the PROMÉTHÉE II-DEMATEL approach, because this approach classifies those universities as influencers inside the decision system. There are some divergences among the rankings in other positions, except in eighth position that is always occupied by ITS. The TODIM method is the single method that classifies UNAIR and UNHAS, respectively, in last positions. Those positions are opposite in the other methods. TODIM also diverges from the other methods about the seventh place, ranking UNPAD in a position below UB.

5. CONCLUSION

We proposed a new approach to apply DEMATEL in order to solve MCDM problems called PROMÉTHÉE II-DEMATEL. We present that in a MCDM problem an alternative can exert an influence degree over another alternative without there is a formal cause-effect relation between them. For this reason, we called the influence degree as preference degree. The PROMÉTHÉE II method’s preference index is used for setting the preference degrees between components in the decision system. This occurs because the preference degree is setting from the behaviors of alternatives on each criterion in a multi-criteria-matrix.

We applied the PROMÉTHÉE II-DEMATEL approach in a MCDM problem to rank a set of 10 universities according to their performances on a set of 4 criteria. The resulting methodology is theoretically sound and provides meaningful results when applied in a MCDM problem. Besides, we applied PROMÉTHÉE II, TOPSIS and TODIM multi-criteria methods to the same problem and the results ratify the PROMÉTHÉE II-DEMATEL approach consistency. Thus, this approach opens a new way to study MCDM problems using the DEMATEL method.

REFERENCES

AKYÜS, G.; TOSUN, Ö.; AKA, S. (2018) Multi-criteria decision-making approach for evaluation of supplier performance with MACBETH method. International Journal of Information and Decision Sciences, v. 10, n. 3, p. 249-262.

Altuntas, s.; Dereli, T. (2015) A novel approach based on DEMATEL method and patent citation analysis for prioritizing a portfolio of investment projects. Expert Systems with Applications, v. 42, n. 3, p. 1003-1012.

Baykasoğlu, A.; Gölcük, I. (2017) Development of an interval type-2 fuzzy sets based hierarchical MADM model by combining DEMATEL and TOPSIS. Expert Systems with Applications, v. 70, p. 37-51.

Belton, V.; Stewart, J. T. (2002) Multiple criteria decision analysis an integrated approach. Boston: Kluwer Academic Publishers.

Brans, J. P.; Vincke, P. (1985) A preference ranking organization method (the PROMETHEE method for multiple criteria decision-making). Management Science, v. 31, p. 647-656.

FONTELA, E.; GABUS, A. (1976) The DEMATEL observer (DEMATEL 1976 report). Battelle Geneva Research Centre.

GABUS, A.; FONTELA, E. (1972) World problems, an invitation to further thought within the framework of DEMATEL. Battelle Geneva Research Centre.

GABUS, A.; FONTELA, E. (1973) Perceptions of the world problematique: Communication procedure, communication with those bearing collective responsibility (DEMATEL Report 1). Battelle Geneva Research Centre.

GÖLCÜK, I.; BAYKASOĞLU, A. (2016) An analysis of DEMATEL approaches for criteria interaction handling within ANP. Expert Systems with Applications, v. 46, p. 346-366.

Gomes, L. F. A. M.; Lima, M. M. P. P. (1991) TODIM: Basics and application to multicriteria ranking of projects with environmental impacts. Foundations of Computing and Decision Sciences, v. 16, n. 3-4, p. 113-127.

Gomes, L. F. A. M.; Rangel, L. A. D. (2009) An application of the TODIM method to multicriteria rental evaluation of residential properties. European Journal of Operational Research, v. 193, n. 1, p. 204-211.

HSU, C.-C. (2012) Evaluation criteria for blog design and analysis of causal relationship using factors analysis and DEMATEL. Expert Systems with Applications, v. 39, n. 1, p. 187-193.

HWANG, C. L.; YOON, K. (1981) Multiple attribute decision making: methods and applications. Berlin: Springer-Verlag.

JATI, H.; DOMINIC, D. D. (2017) A new approach of Indonesian university webometrics ranking using Entropy and PROMÉTHÉE II. Procedia Computer Science, v. 124, p. 444-451.

LI, Y.; HU, Y.; ZHANG, X.; DENG, Y.; MAHADEVAN, S. (2014) An evidential DEMATEL method to identify critical success factors in emergency management. Applied Soft Computing, v. 22, p. 504-510.

OPRICOVIC, S.; TZENG, G.-H. (2004) Compromise solution by MCDM methods: A comparative analysis of VIKOR and TOPSIS. European Journal of Operational Research, v. 156, n. 2, p. 445-455.

Pomerol, J.-C.; Barba-Romero, S. (2000) Multicriterion Decision in Management: principles and Practice. New York: Kluwer Academic Publisher.

QUEZADA, L. E.; LÓPEZ-OPINA, H. A.; PALOMINOS, P. I.; ODDERSHEDE, A. M. (2018) Identifying causal relationships in strategy maps using ANP and DEMATEL. Computers & Industrial Engineering, v. 118, p.170-179.

Saaty, T. L. (1980) The analytic hierarchy process. New York: Mc Graw Hill.

SALLUM, F. S. V.; GOMES, L. F. A. M.; MACHADO, M. A. S. (2018) A DEMATEL-TOPSIS-WINGS approach to the classification of multimarket investment funds. Independent Journal of Management & Production, v. 9, n. 4, p. 1203-1234.

SHARMA, R.; GARG, S. (2015) Selecting the best operational strategy for job shop system: and ANP approach. International Journal of Industrial and Systems Engineering, v. 20, n. 5, p. 231-262.

SHIEH, J.-I.; WU, H.-H.; HUANG, K.-K. (2010) A DEMATEL method in identifying key success factors of hospital service quality. Knowledge-Based Systems, v. 23, n. 3, p. 277-282.

SREENIVASULU, R.; SRINIVASARAO, C. (2016) Optimization od surface roughness, circularity deviation and selection of different alluminum alloys during drilling for automotive and aerospace industry. Independent Journal of Management & Production, v. 7, n. 2, p. 413-430.

TSENG, M.-L. (2011) Using a hybrid MCDM model to evaluate firm environmental knowledge management in uncertainty. Applied Soft Computing, v. 11, n. 1, p. 1340-1352.

Velasquez, M.; Hester, P. T. (2013) An analysis of multi-criteria decision making methods. International Journal of Operations Research, v. 10, n. 2, p. 56-66.

Wang, W.-C.; Lin, Y.-H.; Lin, C.-L.; Chung, C.-H.; Lee, M.-T. (2012) DEMATEL-based model to improve the performance in a matrix organization. Expert Systems with Applications, v. 39, n. 5, p. 4978-4986.