Journal Search Engine
Search Advanced Search Adode Reader(link)
Download PDF Export Citaion korean bibliography PMC previewer
ISSN : 1598-7248 (Print)
ISSN : 2234-6473 (Online)
Industrial Engineering & Management Systems Vol.19 No.4 pp.901-907
DOI : https://doi.org/10.7232/iems.2020.19.4.901

Automatic Classification Models and Algorithms Based on the Minimum Sum-of-Squared Errors Model

Guzel Shkaberina, Lev Kazakovtsev*
Reshetnev Siberian State University of Science and Technology, Russia,
*Corresponding Author, E-mail: levk@bk.ru
October 9, 2020 October 21, 2020 October 26, 2020

ABSTRACT


We propose new models and algorithms for automatic classification of objects (clustering) based on the minimum sum-of-squared errors clustering (MSSC) model. Our approach was aimed at improving the accuracy and stability of the result in solving practical problems, such as identifying homogeneous batches of industrial products. We examined the application of the MSSC model and k-means algorithm with various distance measures: Euclidean, Manhattan, Mahalanobis for the problem of automatic classification of objects in a multi-dimensional space of measured parameters (features). For such problems, we present a new model (Mahalanobis Minimum Sum-of-Squared Error Clustering, MMSSC) for solving problems of automatic classification based on the MSSC model with Mahalanobis distance. In addition, we present a new algorithm for automatic classification of objects based on the MMSSC optimization model with the Mahalanobis distance measure and the weighted average covariance matrix calculated from the training sample (pre-labeled data). This algorithm allows us to reduce the number of errors (increasing the Rand index) when identifying homogeneous production batches based on the results of quality control tests. A new approach in the development of evolutionary algorithms for the MSSC problem is presented using a greedy agglomerative heuristic procedure contained in several genetic operators. The use of this approach enables a statistically significant increase in the accuracy of the result (the achieved value of the objective function within the chosen MMSSC mathematical model), as well as its stability, in a fixed time, in comparison with the known algorithms. Thus, in this work, an increase in the accuracy of solving the problem of automatic classification is achieved both by increasing the adequacy of the model (according to the Rand index) and by improving the algorithm that allows us to achieve the best objective function values of within the framework of the chosen model.



초록


    1. INTRODUCTION

    Automatic classification (clustering) involves dividing a set of objects into subsets (groups) so that objects from the same subset are more similar to each other than objects from different subsets by any criterion. The process of classifying objects of the set into certain groups (subsets) depends on the general characteristics of objects and the methods by which they are divided.

    Let us consider a practical example. To exclude the emergence of unreliable products intended for installation in highly reliable equipment (for example, onboard equipment of a spacecraft with a long period of active existence), the entire component base passes through specialized test cen- ters (Kazakovtsev et al., 2017;Kazakovtsev et al., 2020). These centers perform operations of full incoming inspection of all products, diagnostic non-destructive testing and selective destructive physical analysis. The detection of the homogeneous production batches of products as part of shipped prefabricated batches is one of the most important stages of such testing (Kazakovtsev et al., 2017). Without identifying homogeneous batches made in identical production conditions, we cannot extend the results of the destructive physical analysis to the entire shipped production batch.

    One of the simplest and most popular automatic classification models is the Minimum Sum-of-Squared Errors Clustering (MSSC) model also called k-means (Gribel and Vidal, 2019). The goal is to find k points (centers or centroids) X1, … Xk in a d-dimensional space, so that the sum of the squared distances from known points (data vectors) A1,…, AN to the nearest of the resulting points (centers) reaches its minimum (MacQueen, 1967):

    arg min F ( X 1 , ... , X k ) = i = 1 N min j { 1 , k } ¯ X j A i 2 .
    (1)

    Here, k is the number of groups to be found, N is the number of objects. The k-means algorithm requires k to be pre-specified. Moreover, this algorithm requires the initial centers to be specified, and its result depends on this choice. In addition, the distance function (metric) highly influences the obtained result (division into subsets). The group (cluster) is a subset of data vectors of which a specific center is the nearest.

    The first genetic algorithm for solving a similar problem (the discrete p-median problem) was proposed by Hosage and Goodchild (1986). Its improved version, algorithm (Bozkaya et al., 2002) gives fairly accurate results, however, its convergence rate is not high. In Alp et al. (2003), Alp, Erkut and Drezner presented a faster simple genetic algorithm with a special “greedy” recombination (crossover) procedure, which also gives accurate results. These algorithms solve discrete problems. Subsequently, algorithms with a greedy agglomerative procedure were adapted to solve the continuous MSSC problem (Neema et al., 2011). In such algorithms (Maulik and Bandyopadhyay, 2000), the authors encode solutions (chromosomes) in their genetic algorithms as sets of centers represented by their coordinates (vectors of real numbers) in a multidimensional space. A very limited set of genetic operators has been proposed for such algorithms with real coding.

    Analysis of the modern literature (Rand, 1971;De Maesschalck et al., 2000;McLachlan, 1999;Xing et al., 2003) shows that the existing solutions in the field of automatic classification of objects in a multidimensional space of features have either high accuracy, or ensure the stability of the result with multiple runs of the randomized algorithm, or high convergence speed, but not all these qualities at the same time. Algorithms have been developed for solving the MSSC and k-median problems only for the most common distance measures (Euclidean, Manhattan). However, taking into account the space of features for a specific practical problem, a correct choice of the distance measure can lead to an increase in the accuracy of the automatic classification problem. In this research, we use the Rand Index (RI) (1971) as a measure of accuracy related to the model adequacy as well as the achieved objective function value (1) as a measure of the clustering accuracy related to the algorithm efficiency.

    It is extremely difficult to improve the automatic classification result with increased requirements for the accuracy and stability of the result using known algorithms without a significant increase in time costs. When solving practical automatic classification problems with multidimensional data, for example, when solving the problems of identifying homogeneous batches of industrial products and the adequacy of the models and the accuracy of the automatic classification result highlights many questions. However, it is still possible to develop algorithms that provide further improvement of the result based on the selected model such as the MSSC model.

    In a multidimensional space of features, there is often a correlation between individual features and groups of features. The use of correlation dependences can be used by moving from a search in space with a Euclidean or rectangular metric to a search in a space with the Mahalanobis metric (De Maesschalck et al., 2000;McLachlan, 1999;Xing et al., 2003). The squared Mahalanobis distance DM is defined as follows:

    D M ( X ) = ( X μ ) T C 1 ( X μ ) ,
    (2)

    where X is a vector of measured feature values, μ is a vector of averaged values (cluster center), С is the covariance matrix.

    The purpose of this research is to improve the accuracy and stability of the result in solving problems of automatic classification. The idea is to use the Mahalanobis distance measure with the averaged estimate of the covariance matrix in the MSSC clustering problem to reduce the number of the automatic classification errors in comparison with the other known clustering optimization models, as well as to improve the accuracy and the stability of the solution in terms of the achieved objective function value (1) for a fixed execution time.

    Data. In this study, we used data of quality control tests of the shipped batches of electronic components (Orlov and Fedosov, 2016) intended for installation in highly reliable equipment. The data was obtained from a specialized test center. Each data vector (data point) is a set of measured product parameters. The original batch of products consists of several homogeneous batches, in accordance with the manufacturer’s markings, which are often absent in practical tasks. The total number of products is up to several thousands. In each batch, the product is described by 205 measured parameters.

    Model and algorithm with the Mahalanobis distance. The results of computational experiments on automatic classification of industrial products with the kmedoid and MSSC models, in which the Mahalanobis metric is applied, show an increase in clustering accuracy with automatic classification into 2 to 6 clusters and a small number of objects and informative features.

    We propose to calculate the average estimate of the covariance matrix for homogeneous batches of products (based on pre-labeled data) and use it instead of the covariance matrix in equation (2). Such an averaged matrix can be calculated with the use of a training sample (prelabeled data):

    C = 1 n j = 1 k C j n j
    (3)

    where nj is the number of objects in the jth batch, n is the total number of the objects in the training set, Cj are covariance matrices of the separate batches in the training sets.

    Let us denote model (1) with distances (2) and covariance matrix C calculated according to (3), as Mahalanobis Minimum Sum-of-Squared Errors Clustering (MMSSC) model.

    We propose an algorithm for automatic classification of objects based on the MMSSC model with the adjustment of the Mahalanobis distance measure parameter (covariance matrix) based on the training sample:

    Algorithm 1. MMSSC clustering problem solving with covariance matrix training

    • Step 1. Solve a MSSC problem with Euclidean distances and obtain k clusters (here, k is an estimation of the number of homogeneous groups, given in advance);

    • Step 2. For each cluster, calculate its center μi. In the cases of both Mahalanobis and Euclidean distances, center coordinates are calculated as the averaged object coordinates are calculated as the averaged object coordinates in the Cith cluster, i =1,…, k:

    • μ i = 1 m j = 1 m X j i
      (4)

    • where m is the number of objects in the cluster, Xij are values of the ith feature (j=1, ., m);

    • Step 3. Calculate the averaged covariance matrix (3). If the average estimate of the covariance matrix is degenerate, go to Step 4, otherwise go to Step 5.

    • Step 4. Increase the number of clusters (kk+1) and repeat Steps 1 and 2. Form new clusters with the use of Euclidean distances:

    • D ( X j , μ i ) = i = 1 n ( X j i μ i ) 2
      (5)

    • where n is the number of features. Go to Step 3.

    • Step 5. Match each point to the nearest centroid using the square of the Mahalanobis distance (2) with the averaged estimate of the covariance matrix C (3) to form new clusters.

    • Step 6. Repeat from Step 2 until all the clusters stay unchanged.

    In our research, we performed three series of experiments with industrial product test data.

    1stseries. The training sample corresponds to the working sample for which the clustering was carried out.

    2ndseries. The training and work samples are not the same. In practice, a test center can use retrospective data on deliveries and testing of products of the same type as a training sample.

    3rdseries. The training and working samples also do not match, and the results of the automatic classification of products (results of the k-means algorithm in the multistart mode with Euclidean distances) were used as the training sample.

    In each of the experiment series, for each working sample, we ran the k-means algorithm 30 times with each of the five clustering models.

    M1 model is the MSSC model with Mahalanobis distances, the covariance matrix is calculated for the whole training sample.

    MС model is the MSSC model with distances (2) where C is the correlation matrix.

    M2 model is the MSSC model with Mahalanobis distances and averaged covariance matrix (i.e. new MMSSC model).

    MR model is the MSSC model with Manhattan distances.

    ME is the MSSC model with Euclidean distances.

    For each model, the minimum (Min), maximum (Max), mean values (Average), standard deviation (Std.Dev) of the Rand index (RI) and the objective function, as well as the values of the coefficients of variation (V) and the range (R) of the target functions are shown in Table 1. In this work, the following data were used for computational experiments. We composed a mixed batch from 7 homogeneous production batches (pre-labeled data). Batch 1 contains 71 objects, Batch 2 contains 116 objects, 1867 objects in Batch 3, 1250 objects in Batch 4, 146 objects in Batch 5, 113 objects in Batch 6, and 424 objects in Batch 7.

    It was found that the new model M2 with an average estimate of the covariance matrix shows the best accuracy among the presented models in almost all series of experiments using the Rand index (RI) and in all cases it exceeds the ME model, where the Euclidean distance is used. It was also shown in experiments that in most cases the coefficient of variation of the objective function values is higher for the ME model, where the Euclidean measure of distance is used, and also that the coefficient of the range of the objective function values has the highest values for the M2 model, where the Mahalanobis distances are used with an average covariance matrix.. Therefore, to obtain consistently good values of the objective function (1), multiple runs of the k-means algorithm or the use of other algorithms based on the MSSC model (for example, j-means (Shkaberina et al., 2020) or greedy heuristic algorithms (Kazakovtsev et al., 2020) are required.

    New genetic algorithm for the MMSSC problem. With the known parameters of the Mahalanobis distance, we can use various optimization algorithms to solve the MMSSC problem without a training sample, which differ significantly in the achieved value of the objective function (1). Our new algorithm improves the accuracy of solving the MMSSC clustering problem and the stability of the result within a limited execution time. In this section, by the accuracy of the algorithm we mean exclusively the achieved value of the objective function, without taking into account the indicators of the adequacy of the model and the correspondence of results of the algorithm to the real separation of objects, if they are known.

    For genetic algorithms (GA) solving the MSSC clustering (k-means) problem with real coding of solutions, a very limited set of possible genetic operators is known, including mutation operators. For example, in (Maulik and Bandyopadhyay, 2000), the authors encode solutions (chromosomes) in their GA as sets of centroids represented by their coordinates (vectors of real numbers) in a multidimensional space. Every chromosome undergoes mutation at a fixed probability ω. The procedure (operator) of mutation is as follows.

    In our work, we replaced this mutation procedure for the k-means genetic algorithm with a single-point crossover with the following mutation procedure.

    This procedure is used with probability equal to 1, after each crossover operator.

    Results of running Algorithm 2 with original mutation probability 0.01 and its version with Algorithm 3 as the mutation operator are shown on Figure 1. The new mutation procedure is fast and effective in comparison with uniform random mutation procedure GA, and shows high convergence rate of the objective function.

    Algorithm 2. Uniform random mutation procedure GA for MMSSC clustering problem

    • STEP 1. Generating a random number b∈(0,1] with the uniform distribution;

    • STEP 2. IFb< ωTHEN the chromosome will mutate. If the position of the current centroid is ʋ, after mutation it becomes:

    • υ { υ ± 2 × b × υ , υ 0 , υ ± 2 × b , υ = 0.

    • Sings «+» and «-» have the same probability. The coordinates of the centroid are shifted randomly.

    Algorithm 3. New mutation procedure in the GA for the MMSSC problem

    • STEP 1. Generating a random initial solution S = {X1 ... Xk};

    • STEP 2. Application of the k-means algorithm to S for obtaining local optimum S’;

    • STEP 3. Applying the simple one-point crossover procedure to the mutated solution S’ from the population and S for obtaining the new solution S’’;

    • STEP 4. Application of the k-means algorithm to S’’for obtaining local optimum S’’;

    • STEP 5. IFF(S’’)<F(S’) THENS’← S’’.

    Greedy genetic algorithms and many other evolutionary algorithms for the MSSC clustering problem dispense with the mutation operator. The idea of the greedy agglomerative heuristic procedure is to combine two known solutions into one invalid solution with an excess number of centroids, and then the number of centroids is successively reduced. At each iteration, the centroid is removed, which shows the smallest increase in the objective function value (1).

    Algorithm 4. Basic Greedy Agglomerative Heuristic Procedure

    • Required: initial number of clusters K, needed number of clusters k, k<K, initial solution S = { X 1 , ... , X K } , re | S | = K .

    • STEP 1. Improve initial solution with k-means algorithm

    •   WHILEK>k

    •   FOR each i ' { 1 , K ¯ } do:

      •   STEP 2. S ' S { X i ' } . Improve solution S ’with the k-means algorithm and store the corresponding obtained values of the objective function MSSC (1) to variables F i ' F { X ' } .

    •   END WHILE

    •   STEP 3. Select a subset Selim of centers nelim, SelimS , | S e l i m | = n e l i m with the minimal values of corresponding variables F i ' ' , n e l i m = max { 1 , 0.2 ( | S | k ) } .

    •   STEP 4. Obtain the new solution S S / S e l i m , K=K-1. Improve solution with k-means algorithm.

    • END WHILE

    In addition, the initial solution S can be obtained by combining two known solutions. Algorithms 5 and 6 modify the initial solution with the second known solution. Greedy Procedure 1 supplements the first set in turn with each element from the second set. Greedy Procedure 2 combines both sets.

    Algorithm 5. Greedy Procedure 1 with partial merge

    • Required: Two sets of cluster centers: S ' = { X 1 ' , ... , X K ' } and S ' ' = { X 1 ' ' , ... , X K ' ' }

    • FOR: each i' ∈{1,K}

    •   STEP 1. Merge S’ and a single item of the set S ' ' : S S { X 1 ' ' , ... , X K ' ' } .

    •   STEP2. Run Algorithm 4 with initial solution S and save the obtained results.

    • END FOR

    • STEP 3. Return the best of the solutions saved in STEP 2.

    Algorithm 6. Greedy Procedure 2 with full merge of sets

    • Required: Two sets of cluster centers S ' = { X 1 ' , ... , X K ' } and S ' ' = { X 1 ' ' , ... , X K ' ' }

    • Step 1. Combine two sets of cluster centers S S ' S ' ' .

    • Step 2. Run the Algorithm 4 with initial solution S.

    The basic genetic algorithm for k-means problems is described as follows:

    Algorithm 7. GA with real-number alphabet for the MSSC problem

    • Required: Initial population size NPOP

    • STEP 1. Generate NPOP initial solutions S 1 , ... , S N P O P , where | S i | = k , and { S 1 , ... , S N P O P } is a randomly selected subset of a set of data vectors. Improve each initial solution with the kmeans algorithm and store the corresponding obtained MSSC values (1) to variables f k F ( S k ) , k = 1 , N P O P ¯ .

    • LOOP

    •   STEP 2.IF stop condition is satisfied, THENSTOP. Return solution S i * , i * { 1 , N P O P ¯ } with minimal value f i * .

    •   STEP 3. Randomly choose 2 indexes k 1 , k 2 { 1 , N P O P ¯ } k 1 k 2 .

    •   STEP4.Run crossover procedure:

    •   S ←Crossingover .

    •   STEP5. Run mutation procedure: ( ) c c S ← Mutation S .

    •   STEP6.Runchosen selection procedure to change population set.

    • END LOOP

    At STEP 6, the following algorithm is proposed.

    Algorithm 8. Tournament selection

    • STEP 1. Randomly select 2 indices k 4 , k 5 { 1 , N P O P ¯ } , k 4 k 5 .

    • STEP 2. IF f k 4 > f k 5 THEN S k 4 S c , f k 4 F ( S c ) , ELSE S k 5 S c , f k 5 F ( S c ) .

    A GA with a greedy heuristic for p-median problems and MSSC clustering problems can be described as follows.

    Algorithm 9. GA with greedy heuristic for p-median problems and MSSC clustering problems (modifications GMSSC-FULL, GMSSC-ONE and GMSSC-MIX)*

    • Required: Population size NPOP.

    • STEP 1. Set Niter ← 0. Choose a set of initial solutions { S 1 , ... , S N P O P } , where |Si|=k. Improve each initial solution with the k-means algorithm and save the corresponding obtained values of the objective function(1) to variables f k F ( S k ) , k = 1 , N P O P ¯ . In this work, the initial value of the population NPOP=5.

    • LOOP

    •   STEP 2. IF stop condition is satisfied, THEN STOP. Return solution S i * , i * { 1 , N P O P ¯ } with minimal value f i * , ELSE set the population size as follows: N i t e r N i t e r + 1 ;

    •    N P O P max { N P O P , [ 1 + N i t e r ] } ; IFNPOP was changed,

    •   THEN generate a new NPOP was described in STEP 1.

    •   STEP 3. Randomly select 2 indices k 1 , k 2 { 1 , N P O P ¯ } .

    •   STEP 4. Run Algorithm 5 (for GMSSC-ONE) or Algorithm6 (for GMSSC-FULL) with solutions S k 1 и S k 2 . For GMSSC-MIX, Algorithm5 or Algorithm 6 are chosen at random with equal probability. Get a new solution. Sc.

    •   STEP 5. ScMutation (Sc) by default, the mutation procedure is not used.

    •   STEP 6. RunAlgorithm 5

    • ENDLOOP

    • *GMSSC-ONE – GA with greedy partial join heuristics, GMSSC-FULL–GA with greedy complete union heuristic, GMSSC-MIX–random choice of Algorithms 5 or 6.

    This algorithm uses a dynamically growing population. In our new version of step 5, the crossover-like mutation operator is proposed.

    Algorithm 10. New mutation operator for STEP 5 of Algorithm 9 (modifications GMSSC-FULL-MUT, GMSSC-ONE-MUT and GMSSC-MIX-MUT)

    • STEP 1. Run the k-means algorithm on a randomly selected initial solution to obtain a solution S’.

    • STEP 2. Run Algorithm 5 (for GMSSC-ONE) or Algorithm 6 (for GMSSC-FULL) with solution ScиS″. Get new solution S c ' .

    • STEP 3. IF F ( S c ' ) < F ( S c ) , THEN S c S c ' .

    In our experiments (Table 2),new modifications of three genetic algorithms (GMSSC-FULL-MUT, GMSSCONE- MUT and GMSSC-MIX-MUT) were compared to other known j-means and k-means algorithms (in multistart mode), genetic algorithms without mutation (GMSSC-FULL, GMSSC-ONE and GMSSC-MIX), automatic classification algorithms for the MMSSC clustering problem with a combined application of search algorithms with alternating randomized neighborhoods formed by the use of greedy agglomerative heuristic procedures (k-GH-VNS1, k-GH-VNS2, k-GH-VNS3), and also for the j-means problem (j-means GH-VNS1, jmeans GH-VNS2).For all data sets, 30 attempts were made to run each of the algorithms. For each algorithm it was calculated the minimum value (Min), maximum value (Max), mean value (Average) and standard deviation (Std.Dev.) of the objective function.

    The best values of new algorithm (*) are given in a bold font, the best values of the known algorithms are given in italic, the best-achieved values of the objective function are underlined (Table 2). To confirm the statistical significance of the advantages ( ↑⇑ ) and disadvantages ( ↓⇓ ) of new algorithms over known algorithms, the Mann-Whitney U-test ( ↑↓↕ ) and t-test ( ⇑⇓⇕ ) were used.

    Our experiments show that the genetic algorithms with greedy agglomerative crossover operator built in accordance with new idea of mutation procedure outperform the genetic algorithms without mutation by the obtained objective function value for the MMSSC clustering problem.

    2. CONCLUSION

    The proposed new MMSSC model for automatic classification of samples of industrial products and new algorithm based on the MMSSC optimization model with Mahalanobis distances and a trained covariance matrix enables us to reduce the proportion of errors (increase the Rand index) when identifying homogeneous production batches of industrial products. The presented genetic algorithm for the MMSSC clustering problem with an idea of using a greedy heuristic procedure in several genetic operators demonstrates more accurate and stable results (achieved values of the MMSSC objective function) in a fixed execution time in comparison with the other algorithms.

    ACKNOWLEDGMENT

    This work was supported by the Ministry of Science and Higher Education of the Russian Federation (project No. FEFE-2020-0013).

    Figure

    IEMS-19-4-901_F1.gif

    Results for data set Mopsi-Joensuu (6014 data vectors of dimension 2), 300 clusters, time limitation 3 minutes.

    Table

    Results of computational experiments on the data of semiconductor devices 1526IE10_002 (3987 data vectors), the training sample consists of 10 batches (the 3rd series of experiments), the working sample was composed of 7 batches of products

    Computational experiment results for various data set of semiconductors (168309 data vectors of dimensionality 2), 30 clusters, 20 minutes

    REFERENCES

    1. Alp, O. , Erkut, E. , and Drezner, Z. (2003), An efficient genetic algorithm for the p-median problem, Annals of Operations Research, 122, 21-42,
    2. Bozkaya, B. , Zhang, J. , and Erkut, E. (2002), An efficient genetic algorithm for the p-median problem, Facality Location: Applications and Theory, 29, 179-205.
    3. De Maesschalck, R. , Jouan-Rimbaud, D. , and Massart, D. L. (2000), The mahalanobis distance, Chemometrics and Intelligent Laboratory Systems, 50(1), 1-18,
    4. Gribel, G. and Vidal, T. (2019), HG-means: A scalable hybrid genetic algorithm for minimum sum-of-squares clustering, Pattern Recognition, 88, 569-583,
    5. Hosage, C. M. and Goodchild, M. F. (1986), Discrete space location-allocation solutions from genetic algorithms, Annals of Operations Research, 6, 35-46,
    6. Kazakovtsev, L. A. , Orlov, V. I. , Stashkov, D. V. , Antamoshkin, A. N. , and Masich, I. S. (2017), Improved model for detection of homogeneous production batches of electronic components, IOP Conference Series: Materials Science and Engineering, 255(1), Article ID 012004,
    7. Kazakovtsev, L. , Shkaberina, G. , Rozhnov, I. , Li, R. , and Kazakovtsev, V. (2020), Genetic algorithms with the crossover-like mutation operator for the k-means problem, Communications in Computer and Information Science, 1275, 350-362,
    8. MacQueen, J. (1967), Some methods for classification and analysis of multivariate observations, Proc. Fifth Berkeley Symp. Math. Stat. Probab., 1, 281-297.
    9. Maulik, U. and Bandyopadhyay, S. (2000), Genetic algorithm-based clustering technique, Pattern Recognition, 33(9), 1455-1465,
    10. McLachlan, G. J. (1999), Mahalanobis distance, Resonance, 4, 20-26,
    11. Neema, M. N. , Maniruzzaman, K. M. , and Ohgai, A. (2011), New genetic algorithms based approaches to continuous p-median problem, Networks and Spatial Economics, 11, 83-99,
    12. Rand, W. M. (1971), Objective criteria for the evaluation of clustering methods, Journal of the American Statistical Association, 66(336), 846-850.
    13. Shkaberina, G. S. , Orlov, V. I. , Tovbis, E. M. , and Kazakovtsev, L. A. (2020), On the optimization models for automatic grouping of industrial products by homogeneous production batches, Communications in Computer and Information Science, 1275, 421-436,
    14. Xing, E. P. , Ng, A. Y. , Jordan, and M. I., Russell, S. (2003), Distance metric learning with application to clustering with side-information, Proceedings of the 15th International Conference on Neural Information Processing Systems, 505-512.