## 1. INTRODUCTION

Growing efforts have made to address global warming and climate change around the world. Despite the need for adaptation to have an important role in the efforts to address climate change (Pielke, 1998), the efforts have given more on mitigation responses than on adaptation actions (Liu *et al*., 2008). Ford *et al*. (2011) concluded that actual evidence on how developed nations are adapting to climate change is limited. On the contrary, many developing countries, unlike developed countries, have reported their demand for implementing adaptation interventions. For example, TNA(Technology Needs Assessments) shows that 79 developing countries have reported their demand for climate change adaptation options to the UNFCCC (United Nations Framework Convention on Climate Change) in over 700 applications by April 20161.

The Korean government has cooperated with developing countries to plan, develop, and implement adaptation interventions. In the cooperative route to climate change adaptation, the Korean government needs to prioritize all possible adaptation options based on proper assessment, because this prioritization is critical for achieving anticipated outcomes with limited resources in an effective way. Such decisions on adaptation options involve several challenges; diverse stakeholders with dif-ferent views and priorities, much uncertainty inherent in adaptation options, and a broad spectrum of technological options. Moreover, these challenges often contain potential conflicts with each other. Therefore, it is critical for the Korean government to reconcile theses conflicts when prioritizing adaptation options.

Multi-criteria decision analysis (MCDA) has been used as the most appropriate method to prioritize climate change adaptation options (Dixit and McGray, 2013;Van Ierland *et al*., 2013). The purpose of MCDA is to support the decision-making process for ranking a set of alternatives given multiple criteria. The major strength of MCDA is that it allows a variety of criteria to be incorporated and takes their priorities into account. Despite its wide use, MCDA has several inherent drawbacks, such as weighting and aggregating sub-criteria and handling uncertainty. MCDA requires expert judgement on weights corresponding to criteria. Although supporting methods such as AHP (Analytic Hierarchy Process), Delphi method, and PCA (Principal Component Analysis) are used in this process, the heavy reliance on expert judgement could raise an argument over the feasibility and validity of MCDA methods.

This paper is motivated by observations on how decisions are made in prioritizing adaptation options using MCDA in Korea. As part of the cooperation with the Korean government, we aimed to develop a systematic method for assessing and prioritizing adaptation options in response to the demand from developing countries. In particular, this paper proposes a DEA (Data Envelopment Analysis)-based method that is robust against the uncertainty originating from subjective judgement (See Section 2 for detailed descriptions of the research background). We use DEA as an MCDA tool to produce a substantial evaluation of adaptation options without any of the extensive subjective judgement and value measurements that are carried out by human experts.

Here, we first review DEA as an underlying theoretical background of the method proposed in this study. DEA is a viable alternative that can overcome the drawbacks of MCDA. DEA aims to measure the relative efficiency of each Decision Making Unit (DMU), given various inputs and outputs. In this sense, MCDA and DEA share conceptual similarities and formulations coincide (Stewart, 1996). Beyond the conceptual similarities, DEA has a structural strength over MCDA. DEA does not require any prior information or subjective judgement in order to determine weights on criteria, which is more advantageous than other aforementioned MCDA methods. In addition, DEA is a non-parametric approach that does not need any assumption on the functional relationships among criteria in advance. Because of this strength, much literature has adopted DEA in empirical MCDA studies (Wang, 2015;Hatefi and Torabi, 2010;Bernini *et al*., 2013;Taghavifar *et al*., 2014).

The literature on DEA for prioritizing DMUs requires known, deterministic, and quantifiable data. However, decisions on assessing and prioritizing adaptation options often do not satisfy this requirement. The decisions involve many qualitative evaluations because quantifying the effects of climate change adaptation options is very difficult; no quantitative indicator is available, the effect tends to be realized over a long-time horizon, and climate change adaptation options cover a wide range. The subjective judgements are highly likely to lead to large variations in the data when experts fail to reach a consensus, a situation generally named data uncertainty. Therefore, we should carefully investigate how to address uncertain data caused by the subjective evaluations when using DEA for prioritizing options for climate change adaptation.

A basic form of a DEA model for prioritizing DMUs can be found in Torabi *et al*. (2012) and Amin and Emrouznejad (2013), which modified common-weight DEA models by assuming the unity of inputs. Their DEA models accommodate not only quantitative data but also qualitative data that result from subjective evaluations. A few DEA models recently have been proposed to improve their discriminating power when ranking alternatives. For example, Torabi *et al*. (2012) prevented their DEA model from having many efficient DMUs by forcing efficient DMUs to be inefficient. Amin and Emrouznejad (2013) added a constraint that guarantees having a single efficient DMU. Although such variants of DEA models are available, uncertain data has not been properly considered in the context of multi-criteria decision analysis.

Variants of DEA models have been developed to provide reliable results given uncertain data, not in the area of multi-criteria decision analysis, but in line with conventional methods. These variants are roughly classified into a stochastic model with probabilistic chance constraints (Charnes and Cooper, 1963;Sengupta, 1987;Cooper *et al*., 1998;Wu and Lee, 2010), an imprecise DEA (IDEA) model to handle interval data (Despotis and Smirlis, 2002;Wang *et al*., 2005), and a simulation-based approach (Kao and Liu, 2009).

The chance-constrained stochastic model constrains the probability of having an efficiency score that is less than or equal to a certain level of expected efficiency. The stochastic DEA model is transformed into a deterministic equivalent model by taking an inverse function of the joint probability distribution associated with uncertain data. Because of this property, this approach is only applicable when the inverse function is known and attainable. This paper considers a multinomial distribution of which the inverse function is unfortunately not known; so it is hardly difficult to apply the chance-constrained programming approach.

When the upper and lower bounds of data are given without any information on the exact distribution of stochastic data, IDEA has been used to obtain an efficiency interval that defines the upper and lower bounds of efficiency scores (Despotis and Smirlis, 2002;Wang *et al*., 2005). Although this approach provides meaningful bounds on the efficiency scores, it fails to specify how efficiency scores are distributed within the bounds. Furthermore, the interval data imply that the uncertainty is assumed to be uniformly distributed. However, many practical applications, including the problem observed in this paper, are unfit for this uniform assumption. A few papers have recently extended the interval efficiency into a robust DEA model in which input/output parameters are constrained to be within a pre-defined worst-case efficiency (Shokouhi *et al*., 2010, 2014). In addition, a ranking method follows to prioritize DMUs (Despotis and Smirlis, 2002;Wang *et al*., 2005;Saen, 2008) after obtaining an efficiency interval. For example, Wang *et al*. (2005) and Saen (2008) determined the ranks of DMUs with respect to the minimax regret criterion.

Banker (1993) established the statistical foundation for DEA by employing a simulation-based approach. The simulation-based approach has strengths over the others because of its simple computation and statistical properties of efficiency scores (Atkinson and Wilson, 1995;Premachandra *et al*., 2000;Ruggiero, 2004;Wong, 2009;Kuah *et al*., 2012;Sheikhalishahi, 2014;Kao and Liu, 2014;Dotoli *et al*., 2016). The simulation-based approach first samples inputs and outputs from a probability distribution and repeats measuring efficiency scores for the generated samples. For ranking DMUs based on the sample efficiency scores, two approaches have been found in the literature; assessing average ranks (Sheikhalishahi, 2014;Dotoli *et al*., 2016) and establishing ranks based on average efficiency scores (Ruggiero, 2004;Kao and Liu, 2014).

The purposes of this research are threefold; First, it is to provide a systematic method for assessing and prioritizing options for adapting to climate change by using DEA. From this, we could validate the usefulness of DEA in making decisions about evaluating climate change adaptation options. Second, we attempt to contribute to the literature on DEA for prioritizing DMUs with uncertain data. We modify the existing models to use stochastic data and propose a Monte Carlo simulation-based method to provide robust analysis. The proposed method provides an efficiency distribution for making a robust decision unlike the literature that calculates only a single efficiency value. Finally, for a validation purpose, we provide a numerical illustration that helps a decision-maker to understand which options are the most effective for adapting to climate change. The decision should reconcile various aspects such as the demand of developing countries, the effectiveness of adaptation options, and the capability of transferring adaptation technologies in Korea.

The rest of this paper is organized as follows: Section 2 introduces the practical experience as a research background. Section 3 explains DEA models and Monte- Carlo simulation-based approach to obtain an efficiency distribution. Section 4 provides a numerical example of applying the DEA model to rank climate change adaptation options. Finally, Section 5 concludes.

## 2. BACKGROUND

This paper is based on a project2, which aimed to provide the Korean government with practical implications for transferring climate change adaptation technologies to developing countries by identifying promising adaptation options. Therefore, an important part of the project purpose is to evaluate and prioritize climate change adaptation options.

The project team reviewed UNFCCC TNA reports published by 79 developing countries during 1998-2013 and classified 37 adaptation demands in water sector addressed in the reports into 11 options. The project team initially classified adaptation demands based on their technological similarities and by referring to the literature (Bertule *et al*., 2018) that provides water adaptation technology taxonomy. Then, we validated the classification by employing expert opinions. The project identified that 59 countries particularly had technical requests in water sector. Thus, without loss of generality, we decided to focus on water sector options to reduce any burden caused by a wide range of climate change adaptation options. Table 1 describes 11 options for climate change adaptation in water sector.

The project extensively reviewed literature on evaluating climate change adaptation options to design the evaluation criteria. For example, Chae and Jo (2013) proposed an evaluation scheme for prioritizing climate change adaptation options in Korea, and Dixit and McGray (2013) used MCDA approach for analyzing climate change adaptation options. Table 2 indicates a multi-faceted evaluation that consists of seven quantitative and qualitative criteria in three categories: ‘Demand of developing countries’, ‘Internal capability’, and ‘Effectiveness’ of each adaptation option.

*Demand of developing countries* includes two criteria, one quantitative, the other qualitative. The developing countries’ demand for each adaptation option is quantified by counting the number of countries that declare their demand for the option in UNFCCC TNA reports. Although it is an incomplete measure of exact market size, for a relative comparison, the number of countries is sufficient to represent how attractive each adaptation option is for developing countries. As an alternative measure, we consi-dered a GDP-weighted sum of the developing countries. In addition to the quantitative measure of demand, a qualitative measure of market growth potential is considered.

The second category, *Internal Capability*, represents how well Korea can respond to the demand for climate change adaptation from developing countries. There are three quantitative criteria. First, supply capacity is measured by counting the number of domestic SMEs (Small and Medium Enterprises) that have the Korean green technology certification (This certification is approved by Korean government, and a list of SMEs are available at http://www.greencertif.or.kr) or have KSIC (Korea Standard Industry Classification) codes corresponding to the adaptation options and their sales ranks are in the top 10%. We used the Korea Investors’ Service (KIS-VALUE) database (http://www.kisvalue.com) to investigate company profiles.

The achievements in paper and patents are employed to measure the competitive advantage of Korea in R&D. The paper or patent share is a well-known indicator of the global impact of technologies in a given sector, which is measured by the ratio of the shares of a particular technology in a given country to the total patents in the same sector, namely

where *PS _{ij}* denotes the ratio of paper or patent share in sector

*i*of country

*j*to total patents in the same sector across all countries. For each adaptation option, we compiled papers containing the option in the title or keywords from the SCOPUS database during 2012-2016. Information on patents was collected from the United States patent database (http://patft.uspto.gov/).

The last category (Effectiveness) consists of two criteria, Applicability and Economic feasibility. The last category is mainly measured by experts judgements.

The aforementioned evaluation criteria and their corresponding weights were finally chosen by a group of 27 experts. To collect data on the qualitative criteria (i.e., Market growth, Applicability and Economic feasibility), the same expert group was interviewed by email in late 2016: 11 from academia, 10 from industry, and 6 from research institutions. The number of respondents varied by adaptation option from 18 to 26 depending on their confidence in their knowledge of a certain adaptation option. The responses are on a 5-point Likert scale ranging from one (indicating the least preferred) to five (indicating the most preferred).

To support evaluation and prioritization of adaptation options, this paper attempts to focus on providing a DEA-based method as a part of the project. The DEAbased method is more applicable than a conventional weighted sum method, which was initially used in the project, for the problem of prioritizing adaptation options because of its robustness against the uncertainty originating from subjective judgements on weights and measurements. For the rest of this paper, we reused the part of the project results such as classifications of adaptation options and the evaluation criteria. The water sector options shown in Table 1 are evaluated by means of the multi-criteria described in Table 2, which was designed in consultation with the expert advisory group of the project.

## 3. DEA MODELS

### 3.1 Basic Model with Ordinal Data

This paper mainly refers to the model proposed by Amin and Emrouznejad (2013), Cook and Zhu (2006), Torabi *et al*. (2012), which modifies a conventional common-weight DEA model. We present Model (1) for assessing and prioritizing DMUs with notations summarized in Table 3.

Model (1) does not fix the weights on the criteria and defines them as decision variables; the only restriction is that they should be non-negative. The weights are decided within the formulation above, which mainly differs from MCDA in which it exogenously decides weights by subjective judgements.

The objective is to minimize the maximum inefficiency among all DMUs. The inefficiency *d _{i}* is measured by the efficiency deviation from unity and its maximum value is assured by the constraints $M-{d}_{j}\ge 0,\text{}\forall j\in J$. Let

*θ*denote the efficiency score of DMU

_{j}*j*. Then,

*θ*becomes 1−

_{j}*d*by the definition of

_{j}*d*or ${\theta}_{j}={\displaystyle {\sum}_{i\in O}{u}_{i}^{\text{*}}{y}_{ij}}+{\displaystyle {\sum}_{i\in S}}{\displaystyle {\sum}_{r}{w}_{i,r}^{\text{*}}{y}_{ijr}}$, where ${u}_{i}^{\text{*}}$ and ${w}_{i,r}^{\text{*}}$ are an optimal solution of Model (1). There are constraints associated with binary decision variables

_{i}*b*, which allow only one DMU to be efficient.

_{j}Model (1) shows that both quantitative and qualitative data are taken into account. We consider a situation where a group of decision-makers evaluate qualitative criteria on an *R*-point Likert scale, where the preference is arranged in an increasing order (i.e., *R* is the most preferred score). For a given set of decision-makers’ evaluations, we formulate them as ordinal data by referring to the literature (Cook *et al*., 1996;Karsak and Ahiska, 2005;Cook and Zhu, 2006;Amin and Emrouznejad, 2013). For each DMU *j* and criterion *i*, define the *R*-dimensional unit vector *Y _{ij}* = (

*y*), where

_{ijr}*y*= 1 if a decision-maker ranks DMU

_{ijr}*j*in the

*r*-th position on the criterion

*i*, otherwise

*y*= 0. Thus, ${\sum}_{r}{y}_{ijr}=1$.

_{ijr}The weights associated with ordinal rank positions on criterion *i* (i.e., *w _{i,r}*) should specify that a weight being ranked in (

*r*+ 1)st position is higher than that in

*r*-th position for any DMU

*j*because it is more preferred to be ranked in the higher position. Ψ denotes the set of permissible weights vectors, which represent the relations between weights for the two consecutive rank positions. A linear condition ${w}_{i,r+1}-{w}_{i,r}\ge \epsilon $ ensures that a strictly higher value is given to the weight being rated in the higher place, where

*ε*is the non-Archimedean epsilon and assumed to be independent of rank position

*r*and criterion

*i*.

### 3.2 Interval Data

Model (1) effectively measures the efficiency of DMUs when both quantitative and qualitative data are known and deterministic. Hence, Model (1) is not applicable for a problem in which data are uncertain and stochastic. As described in Section 2, we observed there are completely diverse evaluations on qualitative criteria across a group of decision-makers. Obviously, we should consider the data *Y _{ij}* as being not deterministic but uncertain.

Qualitative data are uncertain but known to lie within the interval of [1, R]. In this sense, *Y _{ij}* is similar to interval data whose lower and upper bounds are known but whose exact value is uncertain. By considering

*Y*as an interval data defined in [1, R], we find the lower and upper bounds of efficiency. For this purpose, we employed the idea of the IDEA model, which provides the lower and upper bounds of efficiency when there are interval data (Despotis and Smirlis, 2002;Wang

_{ij}*et al*., 2005).

The IDEA models in the literature are extended from a conventional fractional BCC (Banker, Charnes and Cooper) or CCR (Charnes, Cooper and Rhodes) model to accommodate interval data. The IDEA model calculates the upper (or lower) bound of efficiency by applying the most favorable (or unfavorable) situation for the DMU under consideration and the most unfavorable (or favorable) situation for other DMUs. In a similar way, we can find the maximum (or minimum) efficiency score of DMU o under consideration by setting ${y}_{ioR}=1\hspace{0.33em}\left(\text{or}\hspace{0.33em}{y}_{io1}=1\right);\hspace{0.33em}\forall \text{i}\in \hspace{0.33em}\text{S}$ and *y _{ij}*

_{1}=1 (or

*y*=1 ); ∀i∈S, where ∀j∈J / o. This idea is concretely described in the following two propositions, which calculate and prove the existence of the efficiency interval for Model (1). Here, it is noteworthy that no research has investigated efficiency intervals for a common-weight DEA model such as given in Model (1)

_{ioR}**Proposition 1***Let*$({u}^{*},\hspace{0.33em}{w}^{\text{*}},\hspace{0.33em}{d}^{\text{*}})$*be an optimal solution of Model (2). Then,*$({u}^{\text{*}},\text{}{w}^{\text{*}})$*is a feasible solution of Model (1)*.

**Proof** See Appendix.

**Proposition 2***Let*$({u}^{\text{*}},\hspace{0.33em}\hspace{0.33em}{w}^{\text{*}},\hspace{0.33em}\hspace{0.33em}{d}^{\text{*}})$*be an optimal solution of Model (2). Then,*${\theta}_{j}^{\text{*}}=1-{d}_{j}^{\text{*}}$*is the lower bound of efficiency score for DMU _{j}*.

**Proof** In Proposition 1, for DMU *j*, there exits *d _{j}* satisfying

Thus, by applying weights relation ${w}_{i,r+1}-{w}_{i,r}\ge \epsilon $, we have

It means that ${d}_{j}^{\text{*}}$ is the upper bound of inefficiency of DMU *j*. The efficiency is given as *θ _{j}* = 1 −

*d*, and thus ${\theta}_{j}^{\text{*}}$ is the lower bound of efficiency.

_{j}Proposition 2 explains that the lower bound of the *j*th DMU’s efficiency is determined when the ordinal data of DMU *j* are ranked in the lowest position (i.e., *y _{ijr}* =1). As in Proposition 2, we can find the upper bound of the efficiency score by setting

*y*=

_{ijr}*R*.

The efficiency interval is meaningful information for ranking DMUs if an exact value of data is not known. However, qualitative evaluations observed in the motivating decision process provide not only the bounds of data but also the probability of observing each value within the bounds (see Section 2). That is, a multinomial probability distribution is available for *Y _{ij}* . Consequently, the efficiency associated with a certain value of

*Y*becomes a random variable, and the probability of observing the minimum efficiency could be different from that of the maximum efficiency. For example, the upper bound of the

_{ij}*j*-th DMU’s efficiency is observable with a probability of $\prod}_{i}{p}_{ijR$, where

*p*is the probability of rank position

_{ijr}*r*of DMU

*j*on criterion

*i*. The next section investigates how to use the probability distribution in measuring efficiency in detail.

### 3.3 Stochastic Data

Existing IDEA models that assume a uniform efficiency interval are not applicable for the problem considered in this paper, which attempts to derive an efficiency distribution rather than simple bounds of efficiency by fully utilizing the distribution of uncertain data. For this purpose, we first explain an approach based on Monte- Carlo sampling, which is named DEA-S. In addition, for comparison, we consider two variants of Model (1) that transform the stochastic factors into deterministic ones by taking an expected value or mode of the data distribution.

#### 3.3.1 DEA-S: Stochastic DEA Model and Monte- Carlo Sampling Approach

We consider *Y _{ij}* as a random variable following a multinomial probability distribution. For example, let us consider a situation where a group of 20 decision-makers evaluate DMU

*j*on criterion

*i*, and the number of decision- makers for each rank position is assumed to be given as (1, 4, 10, 3, 2). Then, the multinomial probability distribution associated with a random variable

*Y*becomes

_{ij}*P*= (1/20, 4/20, 10/20, 3/20, 2/20) by taking its relative frequency.

_{ij}Let *θ _{j}* denote an expected value of

*j*-th DMU’s ef ficiency. It becomes ${\overline{\theta}}_{j}={\displaystyle {\sum}_{k=1}^{K}{p}_{k}{\theta}_{j}^{k}}$, where

*p*is the probability of

_{k}*k*-th sample and

*θ*is the corresponding efficiency of DMU

_{k}*j*. Furthermore, ${p}_{k}={\displaystyle {\prod}_{i}{p}_{ijr}}$, where

*p*is the probability of observing rank position

_{ijr}*r*of DMU

*j*with respect to criterion

*i*.

An exact value of *θ _{j}* is obtainable by fully enumerating all possible instance of

*Y*. If there are |J| DMUs and |O| number of qualitative evaluation criteria are measured on an

_{ir}*R*-point Likert scale, then we should repeatedly solve Model (1) ${R}^{\left|O\right|\times \left|J\right|}$ times. Hence, computational efforts on obtaining

*θ*increase exponentially with respect to |J|, |O| and |R|. For example, when considering three qualitative evaluation criteria (|O| = 3 ) on a 5-point Likert scale (

_{j}*R*= 5) and 11 DMUs (|J| = 11), the problem size becomes 5

^{3×11}, which is not computationally tractable.

To overcome this computational burden, we use a sampling-based approach and approximate *θ _{j}* by taking the sample average. Let define ${\widehat{\theta}}_{j}$ be the sample average. Then, ${\widehat{\theta}}_{j}=\frac{1}{L}{\displaystyle {\sum}_{l=1}^{L}{\theta}_{j}^{l}}$, where ${\theta}_{j}^{l}$ is the

*l*-th average efficiency of DMU

*j*with a sample size of

*H*, and

*L*is the number of replications. Here, ${\theta}_{j}^{l}=\frac{1}{H}{\displaystyle {\sum}_{h=1}^{H}{\theta}_{j}^{l}(h)}$. Based on the idea of the batch-mean approach (Law and Kelton, 2000), DEA-S consists of three steps; it first generates H samples of

*Y*from corresponding probability distribution

_{ij}*P*. Second, we apply the sample for Model (1) and obtain efficiency. Then, the average efficiency ${\theta}_{j}^{l}$ is calculated with H number of efficiency scores. We repeat these first two steps L times, and the expected value, variance and efficiency distribution are taken at the end of the three-step procedure.

_{ij}#### 3.3.2 DEA-A: Average of Qualitative Evaluations

DEA-A replaces random variables *Y _{ij}* with their expected values. For example, if

*P*= (1/20, 4/20, 10/20, 3/20, 2/20), then the corresponding expected value of 3.05 is used for criterion

_{ij}*i*of DMU

*j*. DEA-A is conceptually consistent with the conventional weighted-sum method.

We describe DEA-A in Model (3), which is similar to the common-weight DEA-MCDA model proposed by Hatefi and Torabi (2010). Notice that Model (3) transforms qualitative data into quantitative data and has no constraint associated with ordinal relations.

#### 3.3.3 DEA-M: Mode of Qualitative Evaluations

DEA-M is not different from DEA-A shown in Model (3) except for using the mode of the multinomial probability distribution *P _{ij}* . For example, if

*P*= (1/20, 4/20, 10/20, 3/20, 2/20), then the mode of 3 is used for criterion

_{ij}*i*of DMU

*j*. This is equivalent to setting

*y*

_{ij}_{3}=1 and

*y*= 0 ,∀r ≠ 3 in Model (1). DEA-M conceptually resembles a Delphi method in which the most frequently chosen value represents the experts’ judgements.

_{ijr}## 4. NUMERICAL ILLUSTRATION

In this section, we give a numerical example in which the proposed models are applied for prioritizing the 11 adaptation options shown in Table 1 with the criteria described in Table 2. For the numerical analysis, we use the data obtained from the project described in Section 2. Table 4 summarizes the data used for the further numerical analysis.

### 4.1 Measuring Efficiency by Using DEA-S

The Monte-Carlo sampling-based approach used for DEA-S requires large enough samples to obtain a stable solution. In order to find the sample size, we conduct a series of preliminary tests. The average efficiency ${\widehat{\theta}}_{j}$ is measured by varying sample size until it becomes stable.

Figure 1 shows the convergence of efficiency as the sample size *H* increases. The efficiency stabilizes when sample size *H* is approximately over 1,000. From the test results, we set the sample size *H* and the replications L to be 1,000 and 100, respectively.

Figure 2 illustrates parts of the efficiency distributions obtained from 100,000 sample efficiency scores (*H* = 1,000 and *L* = 100). Although the efficiency distribution does not fit into any known distribution, we roughly estimate that the efficiency will appear in a certain range from the empirical distribution. For example, the average efficiency of Option 11 and Option 1 are close to each other but seem to be larger than that of Option 4. Based on the average efficiency, Option 4 is ranked lower than Option 1 and Option 11.

The most valuable advantage of the efficiency distribution is that statistical validation is allowed. We not only rank the adaptation options with respect to the average efficiency but also statistically validate them by using their confidence intervals. For a statistical validation purpose, 95% confidence intervals (CIs) of efficiency are calculated. The 95% CI of efficiency for Option 1 and Option 11 are [0.6172, 0.6196] and [0.5991, 0.6016], respectively. From the comparison between two 95% CIs, Option 1 is statistically ranked higher than Option 11. An additional t-test for the mean efficiency results in a pvalue < 0.0001 and supports the same conclusion.

Table 5 summarizes the statistical analysis results including mean, standard deviation and 95% CIs. We conducted an ANOVA test to analyze the difference among mean efficiency scores, and it returned a statistically significant result (p-value < 0.000). In addition, we used Tukey’s multiple comparison test, which compares the difference between each pair of 11 efficiency means. The multiple comparison test result shows that any pair of efficiency means is statistically different. From these statistical analyses, 11 options are prioritized.

### 4.2 Ranking Adaptation Options

This section decides the ranks of 11 climate change adaptation options based on efficiency scores. The three DEA models (i.e., DEA-S, DEA-A, DEA-M) described in Section 3.3 are used to rank the options. For DEA-S, we rank options based on estimator ${\widehat{\theta}}_{j}$ and the results are reported in Table 5. Additionally, DEA models are compared with weighted sum, which is the most common method for MCDA. In the use of weighted sum method, input data are normalized and then, the weighted vector shown in Table 2 is applied to come up with a total score for each option. Table 6 summarizes the ranks of 11 adaptation options for each of four different methods.

Based on the results from DEA-S, the top-ranked adaptation option within 11 options for adapting to climate change is Option 2 (Seawater desalination), followed by Option 5 (Surface water storage & supply) and Option 10 (Water reuse). Option 6 (Artificial rainfall) is ranked in the lowest position. DEA-M results in the same top ranks as DEA-S, except that Option 3 and Option 10 switch their positions with each other. However, the results obtained from DEA-A, DEA-M and weighted sum are somewhat different. For example, DEA-A ranks Option 10 (Water reuse) in the first position, and Option 5 (Surface water storage & supply) is top ranked when using weighted sum.

The different results by the four different methods are mainly due to how to accommodate expert judgements into the model. If the probability distributions of *Y _{ij}* over r (i.e.,

*P*) is closed to symmetric, we expect little difference in the inputs for the models and the resulting ranks. On the other hand, the skewed distribution of

_{ij}*P*would make the results dependent on the models. In particular, DEA-A relying on the average is highly likely to be unreliable if the distribution contains any extremely small or large value.

_{ij}For some options such as 3, 8 and 11, the distributions used in the numerical analysis are overall skewed, which means the mode and average are not the representative measure of subjective judgements. Therefore, DEAA and DEA-M fail to precisely represent expert judgements, and we believe that the proposed DEA-S provides more reliable results than DEA-A and DEA-M when observing divergence of subjective judgements.

In overall, Option 2, 5, and 10 are top ranked, but Option 6 (Artificial rainfall) is found to be the least preferred option across all four methods. The top-ranked options are validated by experts survey, which shows that Option 2 (Seawater desalination) and Option 10 (Water reuse) provide more technical and economic benefits than other options. In addition, Option 5 (Surface water storage & storage) and Option 10 have been considered as promising water sector options because of their extensive demands from developing countries and competitive supply capacity in Korea (Chon *et al*., 2017), respectively.

### 4.3 Efficiency Decomposition for DEA-S

The preceding analysis identifies which adaptation option is more attractive or less. To clarify what makes the difference in the mean efficiency of low-ranked op- tions and top-ranked options, we conduct an efficiency decomposition analysis, which decomposes the mean efficiency into sub-efficiencies according to the evaluation criteria described in Table 2. The sub-efficiency is measured by the portion of efficiency corresponding to a certain criterion to the total efficiency. Mathematically, the sub-efficiency of DMU *i* on criterion *k* is given as $({u}_{k}^{*}{y}_{kj})/({\displaystyle {\sum}_{i\in S}{u}_{i}^{*}{y}_{ij}}+{\displaystyle {\sum}_{i\in O}{\displaystyle {\sum}_{r}{w}_{i,r}^{*}{y}_{ijr}}}$, where $\left({u}_{i}^{\text{*}},\hspace{0.33em}{w}_{i,r}^{*}\right)$ is an optimal solution of Model (1). Therefore, the subefficiency explains the contribution of *k*-th criterion to the mean efficiency.

The decomposition analysis is conducted for the DEA-S model, and its result is available in Table 7, which shows the ratio of the sub-efficiency corresponding to each criterion to the efficiency for each adaptation option. For example, the sub-efficiency corresponding to ‘Economic feasibility’ accounts for 12.92% of the efficiency for Option 1. The last row averages the sub-efficiency of 11 adaptation options for each evaluation criterion.

Overall, the importance of ‘Internal capability’ dominates that of the other two categories, ‘Demand of developing countries’ and ‘Effectiveness’. About 65% of efficiency arises mainly from ‘Internal capability’ while the sub-efficiency of ‘Demand of developing countries’ and ‘Effectiveness’ remain approximately 18% and 17% of efficiency, respectively. Further analysis made at each criterion level shows that the contribution of ‘Supply capacity’ is the most significant and ‘Market growth’, ‘Paper’, ‘Economic feasibility’ follow. ‘Supply capacity’, which is measured by the number of domestic companies that own technical capability, accounts for about half of efficiency. This result explains that top-ranked adaptation options are likely to outperform others in terms of ‘Internal capability’ or, in particular, ‘Supply capacity’.

Finally, we calculate the sub-score corresponding to a criterion when applying the weighted-sum method for ranking adaptation options. The average ratios of subscores are 9.44%, 21.73%, 13.84%, 8.46%, 7.97%, 19.27%, and 19.30%. When using a weighted-sum method, ranking adaptation options is not mainly affected by a single category but is decided by accommodating three categories evenly. It reveals that ‘Market growth’, ‘Applicability’ and ‘Economic feasibility’ are the most significant factors contributing to the total weighted-sum score.

Comparing this result with that shown in Table 7 indicates a difference in the contribution of each criterion made by DEA-S and the weighted-sum method. We understand that the difference results mainly from two facts. First, optimal weights obtained from DEA-S are different from the weights used in the weighted-sum method. DEA-S systematically determines optimal weights within the formulation without any exterior information, but the weighted-sum method uses weights that are qualitatively chosen and given by decision-makers. Second, the weighted-sum method normalizes data, but DEA-S does not. Such differences in the inherent characteristics of DEA and the weighted-sum method make direct comparison of the results inappropriate.

If we set the weights on the seven criteria to be approximately *W* = (1%, 12%, 47%, 26%, 1%, 4%, 9%) and apply them for the weighted-sum method, the resulting contribution of sub-scores becomes close to that of DEAS shown in Table 7. Also, the ranks of the 11 adaptation options are the same as those from DEA-S. The weights on the 3rd and 4th criteria (‘Supply capacity’ and ‘Paper’) dramatically increase from 14% and 13% (see Table 2) to 47% and 26%, respectively. On the contrary, the weights on the 1st and 5th criteria (‘Demand’ and ‘Patent’) drop from 15% and 13% to 1%. This analysis implies that DEA-S gives more weight on ‘Supply capacity’ and ‘Paper’ than the weighted-sum method.

This analysis shows that the evaluation result is very sensitive to the weights. Therefore, it is worth considering how to include the approaches for imposing restrictions on weights in DEA formulations if a priori information on weights is available; see Thanassoulis *et al*. (2004) for a survey. For example, the DEA-AR (Assurance Region) model, which was initially proposed by Thompson *et al*. (1986), can further restrict the endogenously selected weights by imposing upper and/or lower bounds of relative ratio of weights (Lai *et al*., 2015) or sub-efficiency (Cherchye *et al*., 2008). We leave this issue for future research.

## 5. CONCLUSION

This study considers a problem that evaluates and prioritizes several options of climate change adaptation. For the problem of prioritizing alternatives, MCDA methods have been widely used, but the conventional MCDA methods have drawbacks. In particular, to address the issue of relying on subjective judgements on weights decision and evaluation, we suggest the use of DEA because of its several appealing advantages over MCDA; DEA avoids any subjective judgements on weights and evaluates all DMUs on the same scale. It makes three novel contributions to the literatures.

First, we modify common weights DEA to address a situation where qualitative evaluations relying on experts’ judgements are not of one accord so that qualitative criteria are not quantifiable in a single value. A Monte-Carlo simulation-based approach is proposed to make robust decisions given data uncertainty. The proposed method is applied to a case study in which 11 adaptation options in water sector are evaluated. It demonstrates the usefulness of the proposed DEA for MCDA in the field of evaluating climate change adaptation options.

Second, this paper provides structural investigation of the efficiency intervals for a common- weight DEA model. The efficiency interval is meaningful for ranking DMUs under a situation where lower and upper bounds of uncertain data are only known.

Lastly, the empirical analysis provides the following implications for ranking climate change adaptation options. First, the most attractive option of adaptation to climate change is ‘Seawater desalination’, and the least preferred option is ‘Artificial rainfall’. Second, ‘Internal capability’ is the most significant factor in deciding on final ranks. Third, empirical weights, which are chosen by expert judgements, underestimate the importance of ‘Supply capacity’. Unlike the existing method, it is noteworthy that the proposed approach provides statistical evidence on the results.

The results give us some interesting directions for future research. It is of interest to apply the proposed model not only to water sector but also to other sectors such as energy or agriculture. Second, we can extend our proposed model to include probabilistic chance-constraints, which re- quire an inverse probability function of random variables. Therefore, we should investigate how to linearize the chance constraints in the case of a discrete multinomial distribution. Third, a tie- breaking rule should be included in DEA-S for prioritizing options. We rank the options based on the average efficiency scores and conduct statistical validation. However, the rank becomes unclear if the mean difference of efficiency scores is not statistically significant. Thus, we should study further how to determine the ranks when statistical significance is not attained.

## APPENDIX

**Proposition 1***Let*$({u}^{*},\hspace{0.33em}\hspace{0.33em}{w}^{*},\hspace{0.33em}\hspace{0.33em}{d}^{*})$*be an optimal solution of Model (2). Then, *$({u}^{*},\hspace{0.33em}\hspace{0.33em}{w}^{*})$*is a feasible solution of Model (1)*.

Proof The proof is very trivial because Model (2) is simply an instance of Model (1).

(a) For DMU *j*

If we apply ${u}^{\text{*}},\text{}{w}^{\text{*}}$ for Model (1), then it should satisfy $\sum}_{i\in O}{u}_{i}^{*}{y}_{ij}}+{\displaystyle {\sum}_{i\in S}{\displaystyle {\sum}_{r}{w}_{i,r}^{*}{y}_{ijr}+{d}_{j}^{*}=1$. From Model (2), $\left({u}^{\text{*}},\text{}{w}^{\text{*}}\right)$ also satisfies ${\sum}_{i\in O}{u}_{i}^{*}{y}_{ij}}+{\displaystyle {\sum}_{i\in S}{w}_{i,1}^{*}+{d}_{j}^{*}=1$. From these two equalities, we have $\sum}_{i\in O}{u}_{i}^{*}{y}_{ij}}+{\displaystyle {\sum}_{i\in S}{w}_{i,1}^{*}{y}_{ij1}}+{\displaystyle {\sum}_{i\in S}{\displaystyle {\sum}_{r=2}^{R}{w}_{i,r}^{*}{y}_{ijr}}}+{d}_{j}^{*}={\displaystyle {\sum}_{i\in O}{u}_{i}^{*}{y}_{ij}}+{\displaystyle {\sum}_{i\in S}{w}_{i,1}^{*}+{d}_{j}^{*$. Reorganizing this equality yields ${d}_{j}={d}_{j}^{*}-{\displaystyle {\sum}_{i\in S}}[{w}_{i,1}^{*}\left({y}_{ij1}-1\right)+{\displaystyle {\sum}_{r=2}^{R}{w}_{i,r}^{*}{y}_{ijr}}]$.

Therefore, $\left({u}^{\text{*}},\text{}{w}^{\text{*}}\right)$ is a feasible solution of Model (1) with ∃d j satisfying

**(ii) For DMU**$\forall k\ne j$

If we apply $\left({u}^{\text{*}},\text{}{w}^{\text{*}}\right)$ for Model (1), then it should satisfy ${\sum}_{i\in O}{u}_{i}^{*}{y}_{ik}}+{\displaystyle {\sum}_{i\in S}{w}_{i,r}^{*}{y}_{ikr}+{d}_{k}=1$. From Model (2), $\left({u}^{\text{*}},\text{}{w}^{\text{*}}\right)$ also satisfies ${\sum}_{i\in O}{u}_{i}^{*}{y}_{ik}}+{\displaystyle {\sum}_{i\in S}{w}_{i,R}^{*}+{d}_{k}^{*}=1$. From these two equalities, we have ${\sum}_{i\in O}{u}_{i}^{*}{y}_{ik}}+{\displaystyle {\sum}_{i\in S}{w}_{i,R}^{*}{y}_{ikR}}+{\displaystyle {\sum}_{i\in S}{\displaystyle {\sum}_{r=1}^{R-1}{w}_{i,r}^{*}{y}_{ikr}}}+{d}_{k}={\displaystyle {\sum}_{i\in O}{u}_{i}^{*}{y}_{ik}}+{\displaystyle {\sum}_{i\in S}{w}_{i,R}^{*}}+{d}_{k}^{*$. Reorganizing this equality yields ${d}_{k}={d}_{k}^{*}-{\displaystyle {\sum}_{i\in S}[{w}_{i,R}^{*}\left({y}_{ikR}-1\right)}+{\displaystyle {\sum}_{r=1}^{R-1}{w}_{i,r}^{*}{y}_{ikr}]}$.

Therefore, $\left({u}^{\text{*}},\text{}{w}^{\text{*}}\right)$ is a feasible solution of Model (1) with ∃*d _{k}* satisfying

According to (i) and (ii), we prove the proposition.