Journal Search Engine
Search Advanced Search Adode Reader(link)
Download PDF Export Citaion korean bibliography PMC previewer
ISSN : 1598-7248 (Print)
ISSN : 2234-6473 (Online)
Industrial Engineering & Management Systems Vol.18 No.2 pp.260-273

A DEA Model for Using Qualitative Data to Rank Options for Adapting to Climate Change

Daiki Min*, Ho-Sik Chon, Huncheol Im
School of Business, Ewha Womans University, Seoul, Korea
Green Technology Center, Seoul, Republic of Korea
Korea Environmental Industry & Technology Institute, Seoul, Republic of Korea
Corresponding Author, E-mail:
October 8, 2018 March 11, 2019 June 7, 2019


This paper is motivated by observations on the decision process for prioritizing adaptation options using Multi-criteria decision analysis (MCDA) in Korea. To overcome the drawbacks of MCDA such as weighting and handling uncertainty, this paper formulates the problem of prioritizing climate change adaptation options as a DEA-like model. The decisions on how to assess such adaptation options involve qualitative evaluations that rely on subjective judgements, which results in data uncertainty. We propose a Monte-Carlo simulation-based method to provide robust analysis against the data uncertainty. The proposed method is applied to a case study in which 11 adaptation options in water sector are evaluated. Whereas a conventional deterministic approach that ignores data uncertainty may mislead decision-makers, a stochastic approach provides more reliable information because of its statistical evidence.



    Growing efforts have made to address global warming and climate change around the world. Despite the need for adaptation to have an important role in the efforts to address climate change (Pielke, 1998), the efforts have given more on mitigation responses than on adaptation actions (Liu et al., 2008). Ford et al. (2011) concluded that actual evidence on how developed nations are adapting to climate change is limited. On the contrary, many developing countries, unlike developed countries, have reported their demand for implementing adaptation interventions. For example, TNA(Technology Needs Assessments) shows that 79 developing countries have reported their demand for climate change adaptation options to the UNFCCC (United Nations Framework Convention on Climate Change) in over 700 applications by April 20161.

    The Korean government has cooperated with developing countries to plan, develop, and implement adaptation interventions. In the cooperative route to climate change adaptation, the Korean government needs to prioritize all possible adaptation options based on proper assessment, because this prioritization is critical for achieving anticipated outcomes with limited resources in an effective way. Such decisions on adaptation options involve several challenges; diverse stakeholders with dif-ferent views and priorities, much uncertainty inherent in adaptation options, and a broad spectrum of technological options. Moreover, these challenges often contain potential conflicts with each other. Therefore, it is critical for the Korean government to reconcile theses conflicts when prioritizing adaptation options.

    Multi-criteria decision analysis (MCDA) has been used as the most appropriate method to prioritize climate change adaptation options (Dixit and McGray, 2013;Van Ierland et al., 2013). The purpose of MCDA is to support the decision-making process for ranking a set of alternatives given multiple criteria. The major strength of MCDA is that it allows a variety of criteria to be incorporated and takes their priorities into account. Despite its wide use, MCDA has several inherent drawbacks, such as weighting and aggregating sub-criteria and handling uncertainty. MCDA requires expert judgement on weights corresponding to criteria. Although supporting methods such as AHP (Analytic Hierarchy Process), Delphi method, and PCA (Principal Component Analysis) are used in this process, the heavy reliance on expert judgement could raise an argument over the feasibility and validity of MCDA methods.

    This paper is motivated by observations on how decisions are made in prioritizing adaptation options using MCDA in Korea. As part of the cooperation with the Korean government, we aimed to develop a systematic method for assessing and prioritizing adaptation options in response to the demand from developing countries. In particular, this paper proposes a DEA (Data Envelopment Analysis)-based method that is robust against the uncertainty originating from subjective judgement (See Section 2 for detailed descriptions of the research background). We use DEA as an MCDA tool to produce a substantial evaluation of adaptation options without any of the extensive subjective judgement and value measurements that are carried out by human experts.

    Here, we first review DEA as an underlying theoretical background of the method proposed in this study. DEA is a viable alternative that can overcome the drawbacks of MCDA. DEA aims to measure the relative efficiency of each Decision Making Unit (DMU), given various inputs and outputs. In this sense, MCDA and DEA share conceptual similarities and formulations coincide (Stewart, 1996). Beyond the conceptual similarities, DEA has a structural strength over MCDA. DEA does not require any prior information or subjective judgement in order to determine weights on criteria, which is more advantageous than other aforementioned MCDA methods. In addition, DEA is a non-parametric approach that does not need any assumption on the functional relationships among criteria in advance. Because of this strength, much literature has adopted DEA in empirical MCDA studies (Wang, 2015;Hatefi and Torabi, 2010;Bernini et al., 2013;Taghavifar et al., 2014).

    The literature on DEA for prioritizing DMUs requires known, deterministic, and quantifiable data. However, decisions on assessing and prioritizing adaptation options often do not satisfy this requirement. The decisions involve many qualitative evaluations because quantifying the effects of climate change adaptation options is very difficult; no quantitative indicator is available, the effect tends to be realized over a long-time horizon, and climate change adaptation options cover a wide range. The subjective judgements are highly likely to lead to large variations in the data when experts fail to reach a consensus, a situation generally named data uncertainty. Therefore, we should carefully investigate how to address uncertain data caused by the subjective evaluations when using DEA for prioritizing options for climate change adaptation.

    A basic form of a DEA model for prioritizing DMUs can be found in Torabi et al. (2012) and Amin and Emrouznejad (2013), which modified common-weight DEA models by assuming the unity of inputs. Their DEA models accommodate not only quantitative data but also qualitative data that result from subjective evaluations. A few DEA models recently have been proposed to improve their discriminating power when ranking alternatives. For example, Torabi et al. (2012) prevented their DEA model from having many efficient DMUs by forcing efficient DMUs to be inefficient. Amin and Emrouznejad (2013) added a constraint that guarantees having a single efficient DMU. Although such variants of DEA models are available, uncertain data has not been properly considered in the context of multi-criteria decision analysis.

    Variants of DEA models have been developed to provide reliable results given uncertain data, not in the area of multi-criteria decision analysis, but in line with conventional methods. These variants are roughly classified into a stochastic model with probabilistic chance constraints (Charnes and Cooper, 1963;Sengupta, 1987;Cooper et al., 1998;Wu and Lee, 2010), an imprecise DEA (IDEA) model to handle interval data (Despotis and Smirlis, 2002;Wang et al., 2005), and a simulation-based approach (Kao and Liu, 2009).

    The chance-constrained stochastic model constrains the probability of having an efficiency score that is less than or equal to a certain level of expected efficiency. The stochastic DEA model is transformed into a deterministic equivalent model by taking an inverse function of the joint probability distribution associated with uncertain data. Because of this property, this approach is only applicable when the inverse function is known and attainable. This paper considers a multinomial distribution of which the inverse function is unfortunately not known; so it is hardly difficult to apply the chance-constrained programming approach.

    When the upper and lower bounds of data are given without any information on the exact distribution of stochastic data, IDEA has been used to obtain an efficiency interval that defines the upper and lower bounds of efficiency scores (Despotis and Smirlis, 2002;Wang et al., 2005). Although this approach provides meaningful bounds on the efficiency scores, it fails to specify how efficiency scores are distributed within the bounds. Furthermore, the interval data imply that the uncertainty is assumed to be uniformly distributed. However, many practical applications, including the problem observed in this paper, are unfit for this uniform assumption. A few papers have recently extended the interval efficiency into a robust DEA model in which input/output parameters are constrained to be within a pre-defined worst-case efficiency (Shokouhi et al., 2010, 2014). In addition, a ranking method follows to prioritize DMUs (Despotis and Smirlis, 2002;Wang et al., 2005;Saen, 2008) after obtaining an efficiency interval. For example, Wang et al. (2005) and Saen (2008) determined the ranks of DMUs with respect to the minimax regret criterion.

    Banker (1993) established the statistical foundation for DEA by employing a simulation-based approach. The simulation-based approach has strengths over the others because of its simple computation and statistical properties of efficiency scores (Atkinson and Wilson, 1995;Premachandra et al., 2000;Ruggiero, 2004;Wong, 2009;Kuah et al., 2012;Sheikhalishahi, 2014;Kao and Liu, 2014;Dotoli et al., 2016). The simulation-based approach first samples inputs and outputs from a probability distribution and repeats measuring efficiency scores for the generated samples. For ranking DMUs based on the sample efficiency scores, two approaches have been found in the literature; assessing average ranks (Sheikhalishahi, 2014;Dotoli et al., 2016) and establishing ranks based on average efficiency scores (Ruggiero, 2004;Kao and Liu, 2014).

    The purposes of this research are threefold; First, it is to provide a systematic method for assessing and prioritizing options for adapting to climate change by using DEA. From this, we could validate the usefulness of DEA in making decisions about evaluating climate change adaptation options. Second, we attempt to contribute to the literature on DEA for prioritizing DMUs with uncertain data. We modify the existing models to use stochastic data and propose a Monte Carlo simulation-based method to provide robust analysis. The proposed method provides an efficiency distribution for making a robust decision unlike the literature that calculates only a single efficiency value. Finally, for a validation purpose, we provide a numerical illustration that helps a decision-maker to understand which options are the most effective for adapting to climate change. The decision should reconcile various aspects such as the demand of developing countries, the effectiveness of adaptation options, and the capability of transferring adaptation technologies in Korea.

    The rest of this paper is organized as follows: Section 2 introduces the practical experience as a research background. Section 3 explains DEA models and Monte- Carlo simulation-based approach to obtain an efficiency distribution. Section 4 provides a numerical example of applying the DEA model to rank climate change adaptation options. Finally, Section 5 concludes.


    This paper is based on a project2, which aimed to provide the Korean government with practical implications for transferring climate change adaptation technologies to developing countries by identifying promising adaptation options. Therefore, an important part of the project purpose is to evaluate and prioritize climate change adaptation options.

    The project team reviewed UNFCCC TNA reports published by 79 developing countries during 1998-2013 and classified 37 adaptation demands in water sector addressed in the reports into 11 options. The project team initially classified adaptation demands based on their technological similarities and by referring to the literature (Bertule et al., 2018) that provides water adaptation technology taxonomy. Then, we validated the classification by employing expert opinions. The project identified that 59 countries particularly had technical requests in water sector. Thus, without loss of generality, we decided to focus on water sector options to reduce any burden caused by a wide range of climate change adaptation options. Table 1 describes 11 options for climate change adaptation in water sector.

    The project extensively reviewed literature on evaluating climate change adaptation options to design the evaluation criteria. For example, Chae and Jo (2013) proposed an evaluation scheme for prioritizing climate change adaptation options in Korea, and Dixit and McGray (2013) used MCDA approach for analyzing climate change adaptation options. Table 2 indicates a multi-faceted evaluation that consists of seven quantitative and qualitative criteria in three categories: ‘Demand of developing countries’, ‘Internal capability’, and ‘Effectiveness’ of each adaptation option.

    Demand of developing countries includes two criteria, one quantitative, the other qualitative. The developing countries’ demand for each adaptation option is quantified by counting the number of countries that declare their demand for the option in UNFCCC TNA reports. Although it is an incomplete measure of exact market size, for a relative comparison, the number of countries is sufficient to represent how attractive each adaptation option is for developing countries. As an alternative measure, we consi-dered a GDP-weighted sum of the developing countries. In addition to the quantitative measure of demand, a qualitative measure of market growth potential is considered.

    The second category, Internal Capability, represents how well Korea can respond to the demand for climate change adaptation from developing countries. There are three quantitative criteria. First, supply capacity is measured by counting the number of domestic SMEs (Small and Medium Enterprises) that have the Korean green technology certification (This certification is approved by Korean government, and a list of SMEs are available at or have KSIC (Korea Standard Industry Classification) codes corresponding to the adaptation options and their sales ranks are in the top 10%. We used the Korea Investors’ Service (KIS-VALUE) database ( to investigate company profiles.

    The achievements in paper and patents are employed to measure the competitive advantage of Korea in R&D. The paper or patent share is a well-known indicator of the global impact of technologies in a given sector, which is measured by the ratio of the shares of a particular technology in a given country to the total patents in the same sector, namely

    P S i j = P i j j P i j

    where PSij denotes the ratio of paper or patent share in sector i of country j to total patents in the same sector across all countries. For each adaptation option, we compiled papers containing the option in the title or keywords from the SCOPUS database during 2012-2016. Information on patents was collected from the United States patent database (

    The last category (Effectiveness) consists of two criteria, Applicability and Economic feasibility. The last category is mainly measured by experts judgements.

    The aforementioned evaluation criteria and their corresponding weights were finally chosen by a group of 27 experts. To collect data on the qualitative criteria (i.e., Market growth, Applicability and Economic feasibility), the same expert group was interviewed by email in late 2016: 11 from academia, 10 from industry, and 6 from research institutions. The number of respondents varied by adaptation option from 18 to 26 depending on their confidence in their knowledge of a certain adaptation option. The responses are on a 5-point Likert scale ranging from one (indicating the least preferred) to five (indicating the most preferred).

    To support evaluation and prioritization of adaptation options, this paper attempts to focus on providing a DEA-based method as a part of the project. The DEAbased method is more applicable than a conventional weighted sum method, which was initially used in the project, for the problem of prioritizing adaptation options because of its robustness against the uncertainty originating from subjective judgements on weights and measurements. For the rest of this paper, we reused the part of the project results such as classifications of adaptation options and the evaluation criteria. The water sector options shown in Table 1 are evaluated by means of the multi-criteria described in Table 2, which was designed in consultation with the expert advisory group of the project.


    3.1 Basic Model with Ordinal Data

    This paper mainly refers to the model proposed by Amin and Emrouznejad (2013), Cook and Zhu (2006), Torabi et al. (2012), which modifies a conventional common-weight DEA model. We present Model (1) for assessing and prioritizing DMUs with notations summarized in Table 3.

    min M s . t . M d j 0 ,   j J i O u i y i j + i S r w i , r y i j r + d j = 1 ,     j J j J b j = 1 d j 1 b j ,     j J N d j 1 b j ,   j J w i , r Ψ = { w i , r :   w i , r + 1 w i , r ε ,   r = 1 , 2 , , R 1 ,     i O } u i , w i , r ε , i , d j 0 , b j 0 , 1 , j J

    Model (1) does not fix the weights on the criteria and defines them as decision variables; the only restriction is that they should be non-negative. The weights are decided within the formulation above, which mainly differs from MCDA in which it exogenously decides weights by subjective judgements.

    The objective is to minimize the maximum inefficiency among all DMUs. The inefficiency di is measured by the efficiency deviation from unity and its maximum value is assured by the constraints M d j 0 ,   j J . Let θj denote the efficiency score of DMU j. Then, θj becomes 1−dj by the definition of di or θ j = i O u i * y i j + i S r w i , r * y i j r , where u i * and w i , r * are an optimal solution of Model (1). There are constraints associated with binary decision variables bj, which allow only one DMU to be efficient.

    Model (1) shows that both quantitative and qualitative data are taken into account. We consider a situation where a group of decision-makers evaluate qualitative criteria on an R-point Likert scale, where the preference is arranged in an increasing order (i.e., R is the most preferred score). For a given set of decision-makers’ evaluations, we formulate them as ordinal data by referring to the literature (Cook et al., 1996;Karsak and Ahiska, 2005;Cook and Zhu, 2006;Amin and Emrouznejad, 2013). For each DMU j and criterion i, define the R-dimensional unit vector Yij = (yijr), where yijr = 1 if a decision-maker ranks DMU j in the r-th position on the criterion i, otherwise yijr = 0. Thus, r y i j r = 1 .

    The weights associated with ordinal rank positions on criterion i (i.e., wi,r) should specify that a weight being ranked in (r + 1)st position is higher than that in r-th position for any DMU j because it is more preferred to be ranked in the higher position. Ψ denotes the set of permissible weights vectors, which represent the relations between weights for the two consecutive rank positions. A linear condition w i , r + 1 w i , r ε ensures that a strictly higher value is given to the weight being rated in the higher place, where ε is the non-Archimedean epsilon and assumed to be independent of rank position r and criterion i.

    3.2 Interval Data

    Model (1) effectively measures the efficiency of DMUs when both quantitative and qualitative data are known and deterministic. Hence, Model (1) is not applicable for a problem in which data are uncertain and stochastic. As described in Section 2, we observed there are completely diverse evaluations on qualitative criteria across a group of decision-makers. Obviously, we should consider the data Yij as being not deterministic but uncertain.

    Qualitative data are uncertain but known to lie within the interval of [1, R]. In this sense, Yij is similar to interval data whose lower and upper bounds are known but whose exact value is uncertain. By considering Yij as an interval data defined in [1, R], we find the lower and upper bounds of efficiency. For this purpose, we employed the idea of the IDEA model, which provides the lower and upper bounds of efficiency when there are interval data (Despotis and Smirlis, 2002;Wang et al., 2005).

    The IDEA models in the literature are extended from a conventional fractional BCC (Banker, Charnes and Cooper) or CCR (Charnes, Cooper and Rhodes) model to accommodate interval data. The IDEA model calculates the upper (or lower) bound of efficiency by applying the most favorable (or unfavorable) situation for the DMU under consideration and the most unfavorable (or favorable) situation for other DMUs. In a similar way, we can find the maximum (or minimum) efficiency score of DMU o under consideration by setting y i o R = 1 or y i o 1 = 1 ; i S and yij1 =1 (or yioR =1 ); ∀i∈S, where ∀j∈J / o. This idea is concretely described in the following two propositions, which calculate and prove the existence of the efficiency interval for Model (1). Here, it is noteworthy that no research has investigated efficiency intervals for a common-weight DEA model such as given in Model (1)

    Proposition 1Let ( u * , w * , d * ) be an optimal solution of Model (2). Then, ( u * ,   w * ) is a feasible solution of Model (1).

    min M s . t . M d j 0 , j J i O u i y i j + i S w i , 1 + d j = 1 ,     f o r   j i O u i y i j + i S w i , R + d k = 1 ,     f o r   k j j J b j = 1 d j 1 b j ,     j J N d j 1 b j ,   j J w i , r Ψ = { w i , r : w i , r + 1 w i , r ε , r = 1 , 2 , , R 1 ,     i O } u i , w i , r ε , i , d j 0 , b j 0 , 1 , j J

    Proof See Appendix.

    Proposition 2Let ( u * , w * , d * ) be an optimal solution of Model (2). Then, θ j * = 1 d j * is the lower bound of efficiency score for DMUj.

    Proof In Proposition 1, for DMU j, there exits dj satisfying

    d j = { d j * , i f y i j 1 = 1 d j * + i S [ w i , 1 * w i , r * ] , i f y i j r = 1 , w h e r e r { 2 , 3 , , R }

    Thus, by applying weights relation w i , r + 1 w i , r ε , we have

    d j max { d j * , d j * i S w i , r * w i , 1 * } max { d j * , d j * ( r 1 ) ε × | S | }

    It means that d j * is the upper bound of inefficiency of DMU j. The efficiency is given as θj = 1 − dj, and thus θ j * is the lower bound of efficiency.

    Proposition 2 explains that the lower bound of the jth DMU’s efficiency is determined when the ordinal data of DMU j are ranked in the lowest position (i.e., yijr =1). As in Proposition 2, we can find the upper bound of the efficiency score by setting yijr = R.

    The efficiency interval is meaningful information for ranking DMUs if an exact value of data is not known. However, qualitative evaluations observed in the motivating decision process provide not only the bounds of data but also the probability of observing each value within the bounds (see Section 2). That is, a multinomial probability distribution is available for Yij . Consequently, the efficiency associated with a certain value of Yij becomes a random variable, and the probability of observing the minimum efficiency could be different from that of the maximum efficiency. For example, the upper bound of the j-th DMU’s efficiency is observable with a probability of i p i j R , where pijr is the probability of rank position r of DMU j on criterion i. The next section investigates how to use the probability distribution in measuring efficiency in detail.

    3.3 Stochastic Data

    Existing IDEA models that assume a uniform efficiency interval are not applicable for the problem considered in this paper, which attempts to derive an efficiency distribution rather than simple bounds of efficiency by fully utilizing the distribution of uncertain data. For this purpose, we first explain an approach based on Monte- Carlo sampling, which is named DEA-S. In addition, for comparison, we consider two variants of Model (1) that transform the stochastic factors into deterministic ones by taking an expected value or mode of the data distribution.

    3.3.1 DEA-S: Stochastic DEA Model and Monte- Carlo Sampling Approach

    We consider Yij as a random variable following a multinomial probability distribution. For example, let us consider a situation where a group of 20 decision-makers evaluate DMU j on criterion i, and the number of decision- makers for each rank position is assumed to be given as (1, 4, 10, 3, 2). Then, the multinomial probability distribution associated with a random variable Yij becomes Pij = (1/20, 4/20, 10/20, 3/20, 2/20) by taking its relative frequency.

    Let θj denote an expected value of j-th DMU’s ef ficiency. It becomes θ ¯ j = k = 1 K p k θ j k , where pk is the probability of k-th sample and θk is the corresponding efficiency of DMU j. Furthermore, p k = i p i j r , where pijr is the probability of observing rank position r of DMU j with respect to criterion i.

    An exact value of θj is obtainable by fully enumerating all possible instance of Yir. If there are |J| DMUs and |O| number of qualitative evaluation criteria are measured on an R-point Likert scale, then we should repeatedly solve Model (1) R | O | × | J | times. Hence, computational efforts on obtaining θj increase exponentially with respect to |J|, |O| and |R|. For example, when considering three qualitative evaluation criteria (|O| = 3 ) on a 5-point Likert scale (R = 5) and 11 DMUs (|J| = 11), the problem size becomes 53×11, which is not computationally tractable.

    To overcome this computational burden, we use a sampling-based approach and approximate θj by taking the sample average. Let define θ ^ j be the sample average. Then, θ ^ j = 1 L l = 1 L θ j l , where θ j l is the l-th average efficiency of DMU j with a sample size of H, and L is the number of replications. Here, θ j l = 1 H h = 1 H θ j l ( h ) . Based on the idea of the batch-mean approach (Law and Kelton, 2000), DEA-S consists of three steps; it first generates H samples of Yij from corresponding probability distribution Pij. Second, we apply the sample for Model (1) and obtain efficiency. Then, the average efficiency θ j l is calculated with H number of efficiency scores. We repeat these first two steps L times, and the expected value, variance and efficiency distribution are taken at the end of the three-step procedure.

    3.3.2 DEA-A: Average of Qualitative Evaluations

    DEA-A replaces random variables Yij with their expected values. For example, if Pij = (1/20, 4/20, 10/20, 3/20, 2/20), then the corresponding expected value of 3.05 is used for criterion i of DMU j. DEA-A is conceptually consistent with the conventional weighted-sum method.

    We describe DEA-A in Model (3), which is similar to the common-weight DEA-MCDA model proposed by Hatefi and Torabi (2010). Notice that Model (3) transforms qualitative data into quantitative data and has no constraint associated with ordinal relations.

    min M s . t . M d j 0 , j J i S u i y i j + d j = 1 ,   j j J b j = 1 d j 1 b j ,     j J N d j 1 b j ,   j J u i ε ,   i , d j 0 , b j 0 , 1 , j J

    3.3.3 DEA-M: Mode of Qualitative Evaluations

    DEA-M is not different from DEA-A shown in Model (3) except for using the mode of the multinomial probability distribution Pij . For example, if Pij = (1/20, 4/20, 10/20, 3/20, 2/20), then the mode of 3 is used for criterion i of DMU j. This is equivalent to setting yij3 =1 and yijr = 0 ,∀r ≠ 3 in Model (1). DEA-M conceptually resembles a Delphi method in which the most frequently chosen value represents the experts’ judgements.


    In this section, we give a numerical example in which the proposed models are applied for prioritizing the 11 adaptation options shown in Table 1 with the criteria described in Table 2. For the numerical analysis, we use the data obtained from the project described in Section 2. Table 4 summarizes the data used for the further numerical analysis.

    4.1 Measuring Efficiency by Using DEA-S

    The Monte-Carlo sampling-based approach used for DEA-S requires large enough samples to obtain a stable solution. In order to find the sample size, we conduct a series of preliminary tests. The average efficiency θ ^ j is measured by varying sample size until it becomes stable.

    Figure 1 shows the convergence of efficiency as the sample size H increases. The efficiency stabilizes when sample size H is approximately over 1,000. From the test results, we set the sample size H and the replications L to be 1,000 and 100, respectively.

    Figure 2 illustrates parts of the efficiency distributions obtained from 100,000 sample efficiency scores (H = 1,000 and L = 100). Although the efficiency distribution does not fit into any known distribution, we roughly estimate that the efficiency will appear in a certain range from the empirical distribution. For example, the average efficiency of Option 11 and Option 1 are close to each other but seem to be larger than that of Option 4. Based on the average efficiency, Option 4 is ranked lower than Option 1 and Option 11.

    The most valuable advantage of the efficiency distribution is that statistical validation is allowed. We not only rank the adaptation options with respect to the average efficiency but also statistically validate them by using their confidence intervals. For a statistical validation purpose, 95% confidence intervals (CIs) of efficiency are calculated. The 95% CI of efficiency for Option 1 and Option 11 are [0.6172, 0.6196] and [0.5991, 0.6016], respectively. From the comparison between two 95% CIs, Option 1 is statistically ranked higher than Option 11. An additional t-test for the mean efficiency results in a pvalue < 0.0001 and supports the same conclusion.

    Table 5 summarizes the statistical analysis results including mean, standard deviation and 95% CIs. We conducted an ANOVA test to analyze the difference among mean efficiency scores, and it returned a statistically significant result (p-value < 0.000). In addition, we used Tukey’s multiple comparison test, which compares the difference between each pair of 11 efficiency means. The multiple comparison test result shows that any pair of efficiency means is statistically different. From these statistical analyses, 11 options are prioritized.

    4.2 Ranking Adaptation Options

    This section decides the ranks of 11 climate change adaptation options based on efficiency scores. The three DEA models (i.e., DEA-S, DEA-A, DEA-M) described in Section 3.3 are used to rank the options. For DEA-S, we rank options based on estimator θ ^ j and the results are reported in Table 5. Additionally, DEA models are compared with weighted sum, which is the most common method for MCDA. In the use of weighted sum method, input data are normalized and then, the weighted vector shown in Table 2 is applied to come up with a total score for each option. Table 6 summarizes the ranks of 11 adaptation options for each of four different methods.

    Based on the results from DEA-S, the top-ranked adaptation option within 11 options for adapting to climate change is Option 2 (Seawater desalination), followed by Option 5 (Surface water storage & supply) and Option 10 (Water reuse). Option 6 (Artificial rainfall) is ranked in the lowest position. DEA-M results in the same top ranks as DEA-S, except that Option 3 and Option 10 switch their positions with each other. However, the results obtained from DEA-A, DEA-M and weighted sum are somewhat different. For example, DEA-A ranks Option 10 (Water reuse) in the first position, and Option 5 (Surface water storage & supply) is top ranked when using weighted sum.

    The different results by the four different methods are mainly due to how to accommodate expert judgements into the model. If the probability distributions of Yij over r (i.e., Pij ) is closed to symmetric, we expect little difference in the inputs for the models and the resulting ranks. On the other hand, the skewed distribution of Pij would make the results dependent on the models. In particular, DEA-A relying on the average is highly likely to be unreliable if the distribution contains any extremely small or large value.

    For some options such as 3, 8 and 11, the distributions used in the numerical analysis are overall skewed, which means the mode and average are not the representative measure of subjective judgements. Therefore, DEAA and DEA-M fail to precisely represent expert judgements, and we believe that the proposed DEA-S provides more reliable results than DEA-A and DEA-M when observing divergence of subjective judgements.

    In overall, Option 2, 5, and 10 are top ranked, but Option 6 (Artificial rainfall) is found to be the least preferred option across all four methods. The top-ranked options are validated by experts survey, which shows that Option 2 (Seawater desalination) and Option 10 (Water reuse) provide more technical and economic benefits than other options. In addition, Option 5 (Surface water storage & storage) and Option 10 have been considered as promising water sector options because of their extensive demands from developing countries and competitive supply capacity in Korea (Chon et al., 2017), respectively.

    4.3 Efficiency Decomposition for DEA-S

    The preceding analysis identifies which adaptation option is more attractive or less. To clarify what makes the difference in the mean efficiency of low-ranked op- tions and top-ranked options, we conduct an efficiency decomposition analysis, which decomposes the mean efficiency into sub-efficiencies according to the evaluation criteria described in Table 2. The sub-efficiency is measured by the portion of efficiency corresponding to a certain criterion to the total efficiency. Mathematically, the sub-efficiency of DMU i on criterion k is given as ( u k * y k j ) / ( i S u i * y i j + i O r w i , r * y i j r , where u i * , w i , r * is an optimal solution of Model (1). Therefore, the subefficiency explains the contribution of k-th criterion to the mean efficiency.

    The decomposition analysis is conducted for the DEA-S model, and its result is available in Table 7, which shows the ratio of the sub-efficiency corresponding to each criterion to the efficiency for each adaptation option. For example, the sub-efficiency corresponding to ‘Economic feasibility’ accounts for 12.92% of the efficiency for Option 1. The last row averages the sub-efficiency of 11 adaptation options for each evaluation criterion.

    Overall, the importance of ‘Internal capability’ dominates that of the other two categories, ‘Demand of developing countries’ and ‘Effectiveness’. About 65% of efficiency arises mainly from ‘Internal capability’ while the sub-efficiency of ‘Demand of developing countries’ and ‘Effectiveness’ remain approximately 18% and 17% of efficiency, respectively. Further analysis made at each criterion level shows that the contribution of ‘Supply capacity’ is the most significant and ‘Market growth’, ‘Paper’, ‘Economic feasibility’ follow. ‘Supply capacity’, which is measured by the number of domestic companies that own technical capability, accounts for about half of efficiency. This result explains that top-ranked adaptation options are likely to outperform others in terms of ‘Internal capability’ or, in particular, ‘Supply capacity’.

    Finally, we calculate the sub-score corresponding to a criterion when applying the weighted-sum method for ranking adaptation options. The average ratios of subscores are 9.44%, 21.73%, 13.84%, 8.46%, 7.97%, 19.27%, and 19.30%. When using a weighted-sum method, ranking adaptation options is not mainly affected by a single category but is decided by accommodating three categories evenly. It reveals that ‘Market growth’, ‘Applicability’ and ‘Economic feasibility’ are the most significant factors contributing to the total weighted-sum score.

    Comparing this result with that shown in Table 7 indicates a difference in the contribution of each criterion made by DEA-S and the weighted-sum method. We understand that the difference results mainly from two facts. First, optimal weights obtained from DEA-S are different from the weights used in the weighted-sum method. DEA-S systematically determines optimal weights within the formulation without any exterior information, but the weighted-sum method uses weights that are qualitatively chosen and given by decision-makers. Second, the weighted-sum method normalizes data, but DEA-S does not. Such differences in the inherent characteristics of DEA and the weighted-sum method make direct comparison of the results inappropriate.

    If we set the weights on the seven criteria to be approximately W = (1%, 12%, 47%, 26%, 1%, 4%, 9%) and apply them for the weighted-sum method, the resulting contribution of sub-scores becomes close to that of DEAS shown in Table 7. Also, the ranks of the 11 adaptation options are the same as those from DEA-S. The weights on the 3rd and 4th criteria (‘Supply capacity’ and ‘Paper’) dramatically increase from 14% and 13% (see Table 2) to 47% and 26%, respectively. On the contrary, the weights on the 1st and 5th criteria (‘Demand’ and ‘Patent’) drop from 15% and 13% to 1%. This analysis implies that DEA-S gives more weight on ‘Supply capacity’ and ‘Paper’ than the weighted-sum method.

    This analysis shows that the evaluation result is very sensitive to the weights. Therefore, it is worth considering how to include the approaches for imposing restrictions on weights in DEA formulations if a priori information on weights is available; see Thanassoulis et al. (2004) for a survey. For example, the DEA-AR (Assurance Region) model, which was initially proposed by Thompson et al. (1986), can further restrict the endogenously selected weights by imposing upper and/or lower bounds of relative ratio of weights (Lai et al., 2015) or sub-efficiency (Cherchye et al., 2008). We leave this issue for future research.


    This study considers a problem that evaluates and prioritizes several options of climate change adaptation. For the problem of prioritizing alternatives, MCDA methods have been widely used, but the conventional MCDA methods have drawbacks. In particular, to address the issue of relying on subjective judgements on weights decision and evaluation, we suggest the use of DEA because of its several appealing advantages over MCDA; DEA avoids any subjective judgements on weights and evaluates all DMUs on the same scale. It makes three novel contributions to the literatures.

    First, we modify common weights DEA to address a situation where qualitative evaluations relying on experts’ judgements are not of one accord so that qualitative criteria are not quantifiable in a single value. A Monte-Carlo simulation-based approach is proposed to make robust decisions given data uncertainty. The proposed method is applied to a case study in which 11 adaptation options in water sector are evaluated. It demonstrates the usefulness of the proposed DEA for MCDA in the field of evaluating climate change adaptation options.

    Second, this paper provides structural investigation of the efficiency intervals for a common- weight DEA model. The efficiency interval is meaningful for ranking DMUs under a situation where lower and upper bounds of uncertain data are only known.

    Lastly, the empirical analysis provides the following implications for ranking climate change adaptation options. First, the most attractive option of adaptation to climate change is ‘Seawater desalination’, and the least preferred option is ‘Artificial rainfall’. Second, ‘Internal capability’ is the most significant factor in deciding on final ranks. Third, empirical weights, which are chosen by expert judgements, underestimate the importance of ‘Supply capacity’. Unlike the existing method, it is noteworthy that the proposed approach provides statistical evidence on the results.

    The results give us some interesting directions for future research. It is of interest to apply the proposed model not only to water sector but also to other sectors such as energy or agriculture. Second, we can extend our proposed model to include probabilistic chance-constraints, which re- quire an inverse probability function of random variables. Therefore, we should investigate how to linearize the chance constraints in the case of a discrete multinomial distribution. Third, a tie- breaking rule should be included in DEA-S for prioritizing options. We rank the options based on the average efficiency scores and conduct statistical validation. However, the rank becomes unclear if the mean difference of efficiency scores is not statistically significant. Thus, we should study further how to determine the ranks when statistical significance is not attained.


    Proposition 1Let ( u * , w * , d * ) be an optimal solution of Model (2). Then, ( u * , w * ) is a feasible solution of Model (1).

    Proof The proof is very trivial because Model (2) is simply an instance of Model (1).

    i O u i * y i j + i S w i , R * y i j R + d k * = 1 for DMU k j

    (a) For DMU j

    If we apply u * ,   w * for Model (1), then it should satisfy i O u i * y i j + i S r w i , r * y i j r + d j * = 1 . From Model (2), u * ,   w * also satisfies i O u i * y i j + i S w i , 1 * + d j * = 1 . From these two equalities, we have i O u i * y i j + i S w i , 1 * y i j 1 + i S r = 2 R w i , r * y i j r + d j * = i O u i * y i j + i S w i , 1 * + d j * . Reorganizing this equality yields d j = d j * i S [ w i , 1 * y i j 1 1 + r = 2 R w i , r * y i j r ] .

    Therefore, u * ,   w * is a feasible solution of Model (1) with ∃d j satisfying

    d j = d j * , i f   y i j 1 = 1 d j * + i S w i , 1 * w i , r * , i f y i j r = 1 , w h e r e r 2 , 3 , , R .

    (ii) For DMU k j

    If we apply u * ,   w * for Model (1), then it should satisfy i O u i * y i k + i S w i , r * y i k r + d k = 1 . From Model (2), u * ,   w * also satisfies i O u i * y i k + i S w i , R * + d k * = 1 . From these two equalities, we have i O u i * y i k + i S w i , R * y i k R + i S r = 1 R 1 w i , r * y i k r + d k = i O u i * y i k + i S w i , R * + d k * . Reorganizing this equality yields d k = d k * i S [ w i , R * y i k R 1 + r = 1 R 1 w i , r * y i k r ] .

    Therefore, u * ,   w * is a feasible solution of Model (1) with ∃dk satisfying

    d k = d k * , i f   y i k R = 1 d k * + i S w i , R * w i , r * , i f y i k r = 1 , w h e r e r 1 , 2 , , R 1

    According to (i) and (ii), we prove the proposition.


    This paper partially used the data from the research project, entitled “Promising Adaptation Technologies and Evaluation Framework in Response to Demand from Developing Countries (No. 2016-011)”, which was supported by the Green Technology Center (GTC) and Ministry of Science and ICT of South Korea (MSIT).

    This work was supported by the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea (NRF-2017S1A5A2A03067552)



    Convergence of efficiency scores.


    Efficiency distribution: Option 1, 4 and 11.


    Climate change adaptation options in water sector

    Evaluation criteria

    *<i>Q<sub>t</sub></i>: Quantitative, <i>Q<sub>l</sub></i>: Qualitative.

    Notation summary

    Data summary for numerical evaluations

    *Mode of subjective judgements on a 5-point Likert scale.

    Summary of statistical analysis

    Evaluation results for 11 options

    *<i>H</i> = 1,000 and <i>L</i> = 100.

    Efficiency decomposition (%): DEA-S


    1. Amin, G. R. and Emrouznejad, A. (2013), A new DEA model for technology selection in the presence of ordinal data, International Journal of Advanced Manufacturing Technology, 65(9-12), 1567-1572.
    2. Atkinson, S. and Wilson, P. W. (1995), Comparing mean efficiency and productivity scores from small samples: A bootstrap methodology, Journal of Productivity Analysis, 6(2), 137-152.
    3. Banker, R. D. (1993), Maximum likelihood, consistency and data envelopment analysis: A statistical foundation, Management Science, 39(10), 1265-1273.
    4. Bernini, C. ,Guizzardi, A. , and Angelini, G. (2013), DEAlike model and common weights approach for the construction of a subjective community well-being indicator, Social Indicators Research, 114(2), 405-424.
    5. Bertule, M. ,Appelquist, L. R. ,Spensley, J. ,Trærup, S. , and Naswa, P. (2018), Climate change adaptation technologies for water: A practitioner’s guide to adaptation technologies for increased water sector resilience. CTCN publications, Copenhagen, Denmark.
    6. Chae, Y. R. and Jo, H. J. (2013), Analysis of methodologies for prioritizing climate change adaptation measures, Journal of Environmental Policy, 12(4), 23-44.
    7. Charnes, A. and Cooper, W. W. (1963), Deterministic equivalents for optimizing and satisficing under chance constraints, Management Science, 11(1), 18-39.
    8. Cherchye, L. ,Moesen, W. ,Rogge, N. ,Van Puyenbroeck, T. ,Saisana, M. ,Saltelli, A. ,Liska, R. , and S. Tarantola (2008), Creating composite indicators with DEA and robustness analysis: The case of the technology achievement index, Journal of the Operational Research Society, 59(2), 239-251.
    9. Chon, H. ,Im, H. , and An, S. (2017), Adaptation technologies in water sector demanded by developing countries and the potential of technology transfer of SMEs in South Korea, In: Alves F., Leal Filho W., Azeiteiro U. (eds) Theory and Practice of Climate Adaptation. Climate Change Management, Springer, Cham.
    10. Cook, W. D. and Zhu, J. (2006), Rank order data in DEA: A general framework, European Journal of Operational Research, 174(2), 1021-1038.
    11. Cook, W. D. ,Kress, M. , and Seiford, L. M. (1996), Data envelopment analysis in the presence of both quantitative and qualitative factors, Journal of the Operational Research Society, 47(7), 945-953.
    12. Cooper, W. W. ,Huang, Z. ,Lelas, V. ,Li, S. X. , and Olesen, O. B. (1998), Chance constrained programming formulations for stochastic characterizations of efficiency and dominance in DEA, Journal of Productivity Analysis, 9(1), 53-79.
    13. Despotis, D. K. and Smirlis, Y. G. (2002), Data envelopment analysis with imprecise data, European Journal of Operational Research, 140(1), 24-36.
    14. Dixit, A. and McGray, H. (2013), Analyzing climate change adaptation options using multi-criteria analysis, Technical report, USAID, Washington, DC.
    15. Dotoli, M. ,Epicoco, N. ,Falagario, M. , and Sciancalepore, F. (2016), A stochastic cross-efficiency data envelopment analysis approach for supplier selection under uncertainty, International Transactions in Operational Research, 23(4), 725-748.
    16. Ford, F. D. ,Berrang-Ford, L. , and Paterson, J. (2011), A systematic review of observed climate change adaptation in developed nations, Climate Change, 106(2), 327-336.
    17. Hatefi, S. M. and Torabi, S. A. (2010), A common weight MCDA-DEA approach to construct composite indicator, Ecological Economics, 70(1), 114-120.
    18. Kao, C. and Liu, S. T. (2009), Stochastic data envelopment analysis in measuring the efficiency of Taiwan commercial banks, European Journal of Operational Research, 196(1), 312-322.
    19. Kao, C. and Liu, S. T. (2014), Measuring performance improvement of Taiwanese commercial banks under uncertainty, European Journal of Operational Research, 235(3), 755-764.
    20. Karsak, E. E. and Ahiska, S. S. (2005), Practical common weight multi-criteria decision-making approach with an improved discriminating power for technology selection, International Journal of Production Research, 43(8), 1537-1554.
    21. Kuah, C. T. ,Wong, K. Y. , and Wong, W. P. (2012), Monte Carlo data envelopment analysis with genetic algorithm for knowledge management performance measurement, Expert Systems with Applications, 39(10), 9348-9358.
    22. Lai, P. ,Potter, A. ,Beynon, M. , and Beresford, A. (2015), Evaluating the efficiency performance of airports using an integrated AHP/DEA-AR technique, Transport Policy, 42, 75-85.
    23. Law, A. M. and Kelton, W. D. (2000), Simulation modelling and analysis (3 ed.), McGraw-Hll, New York:
    24. Liu, X. ,Vedlitza, A. , and Alstona, L. (2008), Regional news portrayals of global warming climate change, Environmental Science & Policy, 11(5), 379-393.
    25. Pielke, R. A. (1998), Rethinking the role of adaptation in climate policy, Global Environmental Change, 8(2), 159-170.
    26. Premachandra, I. M. ,Powell, J. G. , and Watson, J. , (2000), A simulation approach for stochastic data envelopment analysis, Information and Management Science, 11(1), 11-31.
    27. Ruggiero, J. (2004), Data envelopment analysis with stochastic data, Journal of the Operational Research Society, 55(9), 1008-1012.
    28. Saen, R. F. (2008), Supplier selection by the new ARIDEA model, International Journal of Advanced Manufacturing Technology, 39(11-12), 1061-1070.
    29. Sengupta, J. K. (1987), Data envelopment analysis for efficiency measurement in the stochastic case, Computers & Operations Research, 14(2), 117-129.
    30. Sheikhalishahi, M. (2014), An integrated simulation-data envelopment analysis approach for maintenance activities planning, International Journal of Computer Integrated Manufacturing, 27(9), 858-868.
    31. Shokouhi, A. H. ,Hatami-Marbini, A. ,Tavana, M. , and Saati, S. (2010), A robust optimization approach for imprecise data envelopment analysis, Computers & Industrial Engineering, 59(3), 387-397.
    32. Shokouhi, A. H. ,Shahriari, H. ,Agrell, P. J. , and Hatami- Marbini, A. (2014), Consistent and robust ranking in imprecise data envelopment analysis under perturbations of random subsets of data, OR Spectrum, 36(1), 133-160.
    33. Stewart, T. J. (1996), Relationships between data envelopment analysis and multi-criteria decision analysis, Journal of the Operational Research Society, 47(5), 654-665.
    34. Taghavifar, H. ,Mardani, A. , and Karim-Maslak, H. (2014), Multi-criteria optimization model to investigate the energy waste of off-road vehicles utilizing soil bin facility, Energy, 73, 762-770.
    35. Thanassoulis, E. ,Portela, M. C. , and Allen, R. (2004), Incorporating value judgments in DEA, Handbook on Data Envelopment Analysis, Springer US, Boston, MA, 99-138.
    36. Thompson, R. G. ,Singleton, F. D. ,Thrall, R. M. , and Smith, B. A. (1986), Comparative site evaluations for locating a high-energy physics lab in Texas, Informs Journal on Applied Analytics, 16(6), 35-49.
    37. Torabi, S. A. ,Hatefi, S. M. , and Pay, B. S. (2012), ABC inventory classification in the presence of both quantitative and qualitative criteria, Computers & Industrial Engineering, 63(2), 530-537.
    38. Van, E. and C. ,de Bruin,K. ,Watkiss, and (2013), Multi-criteria analysis: Decision support methods for adaptation. Technical report 452821, Environmental Economics and National Resource Group, Wageningen University & Research, Wageningen.
    39. Wang, H. (2015), A generalized MCDA-DEA (multicriterion decision analysis-data envelopment analysis) approach to construct slacks-based composite indicator, Energy, 80, 114-122.
    40. Wang, Y. M. ,Greatbanks, R. , and Yang, J. B. (2005), Interval efficiency assessment using data envelopment analysis, Fuzzy Sets and Systems, 153(3), 347-370.
    41. Wong, W. P. (2009), Performance evaluation of supply chain in stochastic environment: Using a simulation based DEA framework, International Journal of Business Performance and Supply Chain Modelling, 1(2-3), 203-228.
    42. Wu, D. and Lee, C. G. (2010), Stochastic DEA with ordinal data applied to a multi-attribute pricing problem, European Journal of Operational Research, 207(3), 1679-1688.