Kernel-based framework for spectral dimensionality reduction and clustering formulation: A theoretical study

Xiomara Patricia BLANCO VALENCIA, M. A. BECERRA, A. E. CASTRO OSPINA, M. ORTEGA ADARME, D. VIVEROS MELO, D. H. PELUFFO ORDÓÑEZ

Abstract


This work outlines a unified formulation to represent spectral approaches for both dimensionality reduction and clustering. Proposed formulation starts with a generic latent variable model in terms of the projected input data matrix.
Particularly, such a projection maps data onto a unknown high-dimensional space. Regarding this model, a generalized optimization problem is stated using quadratic formulations and a least-squares support vector machine.
The solution of the optimization is addressed through a primal-dual scheme.
Once latent variables and parameters are determined, the resultant model outputs a versatile projected matrix able to represent data in a low-dimensional space, as well as to provide information about clusters. Particularly, proposed
formulation yields solutions for kernel spectral clustering and weighted-kernel principal component analysis.


Keywords


cluster; dimension reduction; support vector machine; low-dimensional space

Full Text:

PDF

References


Aldrich, C. and Auret, L., 2013. Statistical Learning Theory and Kernel-Based Methods. In Unsupervised Process Monitoring and Fault Diagnosis with Machine Learning Methods, pages 117–181. Springer. https://doi.org/10.1007/978-1-4471-5185-2_4

Alvarado-Pérez, J. C. and Peluffo-Ordó’nez, D. H., 2015. Artificial and Natural Intelligence Integration. In 12th International Conference on Distributed Computing and Artificial Intelligence (DCAI 2016), pages 167–173. Springer. https://doi.org/10.1007/978-3-319-19638-1_19

Alvarado-Pérez, J. C., Peluffo-Ordó-ez, D. H., and Therón, R., 2015. Bridging the gap between human knowledge and machine learning. ADCAIJ: Advances in Distributed Computing and Artificial Intelligence Journal, 4(1):54–64. https://doi.org/10.14201/ADCAIJ2015415464

Alzate, C. and Suykens, J. A. K., 2010. Multiway spectral clustering with out-of-sample extensions through weighted kernel PCA. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 32(2):335–347.

Belanche Mu-oz, L. A., 2013. Developments in kernel design. In European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN 2013), Bruges, Belgium, pages 369–378.

Domeniconi, C., Peng, J., and Yan, B., 2011. Composite kernels for semi-supervised clustering. Knowledge and information systems, 28(1):99–116. https://doi.org/10.1007/s10115-010-0318-8

Filippone, M., Camastra, F., Masulli, F., and Rovetta, S., 2008. A survey of kernel and spectral methods for clustering. Pattern recognition, 41(1):176–190. https://doi.org/10.1016/j.patcog.2007.05.018

González, F., Bermeo, D., Ramos, L., and Nasraoui, O., 2012. On the Robustness of Kernel-Based Clustering. Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications, pages 122–129. https://doi.org/10.1007/978-3-642-33275-3_15

Ham, J., Lee, D. D., Mika, S., and Schölkopf, B., 2004. A kernel view of the dimensionality reduction of manifolds. In Proceedings of the twenty-first international conference on Machine learning, page 47. ACM. https://doi.org/10.1145/1015330.1015417

Harchaoui, Z., Bach, F., Cappé, O., and Moulines, E., 2013. Kernel-based methods for hypothesis testing: a unified view. IEEE Signal Processing Magazine, 30(4):87–97. https://doi.org/10.1109/MSP.2013.2253631

Huang, H., Chuang, Y., and Chen, C., 2012. Multiple Kernel Fuzzy Clustering. Fuzzy Systems, IEEE Transactions on, 20(1):120–134.

Langone, R., Alzate, C., and Suykens, J. A., 2013. Kernel spectral clustering with memory effect. Physica A: Statistical Mechanics and its Applications. https://doi.org/10.1016/j.physa.2013.01.058

Molina-Giraldo, S., Álvarez-Meza, A., Peluffo-Ordo-ez, D., and Castellanos-Domínguez, G., 2012. Image Segmentation Based on Multi-Kernel Learning and Feature Relevance Analysis. Advances in Artificial Intelligence–IBERAMIA 2012, pages 501–510. https://doi.org/10.1007/978-3-642-34654-5_51

Peluffo-Ordó-ez, D. H., Lee, J. A., Verleysen, M., Rodr?guez, J. L., and Castellanos-Dom?nguez, G., 2014. Unsupervised relevance analysis for feature extraction and selection. In ICPRAM 2014, pages 310–315.

Peluffo-Ordónez,D., Garcia-Vega, S., Langone, R., Suykens, J., Castellanos-Dominguez,G. et al., 2013. Kernel spectral clustering for dynamic data using multiple kernel learning. In Neural Networks (IJCNN), The 2013 International Joint Conference on, pages 1–6. IEEE. https://doi.org/10.1109/ijcnn.2013.6706858

Peluffo-Ordonez,D. H., Aldo Lee, J., and Verleysen,M., 2014. Generalized kernel framework for unsupervised spectral methods of dimensionality reduction. In 2014 IEEE Symposium on Computational Intelligence and Data Mining (CIDM), pages 171–177. IEEE. https://doi.org/10.1109/cidm.2014.7008664

Peluffo-Ordónez, D. H., Alvarado-Pérez, J. C., Lee, J. A., and Verleysen, M., 2015. Geometrical homotopy for data visualization. In European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN 2015).

Peluffo-Ordó-ez, D. H., Alzate, C., Suykens, J. A., and Castellanos-Domínguez, G., 2014a. Optimal Data Projection for Kernel Spectral Clustering. In European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, pages 553–558.

Peluffo-Ordó-ez, D. H., Lee, J. A., and Verleysen,M., 2014b. Recent methods for dimensionality reduction: A brief comparative analysis. In European Symposium on Artificial Neural Networks (ESANN). Citeseer.

Peluffo Ordo-ez, D. H., Lee, J. A., Verleysen, M., Rodriguez, J. L., Castellanos-Dominguez, G. et al., 2015. Unsupervised relevance analysis for feature extraction and selection. A distance-based approach for feature relevance. In 3rd International Conference on Pattern Recognition Applications and Methods (ICPRAM2014).

Schölkopf, B. and Smola, A. J., 2002. Learning with Kernels.

Seeland, M., Karwath, A., and Kramer, S., 2012. A structural cluster kernel for learning on graphs. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 516–524. ACM. https://doi.org/10.1145/2339530.2339614

Wolf, L. and Bileschi, S., 2005. Combining variable selection with dimensionality reduction. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), volume 2, pages 801–806. IEEE. https://doi.org/10.1109/cvpr.2005.103

Wolf, L. and Shashua, A., 2005. Feature selection for unsupervised and supervised inference: The emergence of sparsity in a weight-based approach. Journal of machine learning, 6:1855 – 1887.

Wu, Y., Ma, W., Gong, M., Li, H., and Jiao, L., 2015. Novel Fuzzy Active Contour Model With Kernel Metric for Image Segmentation. Applied Soft Computing. https://doi.org/10.1016/j.asoc.2015.04.058

Zelnik-manor, L. and Perona, P., 2004. Self-tuning spectral clustering. In Advances in Neural Information Processing Systems 17, pages 1601–1608.MIT Press.




DOI: http://dx.doi.org/10.14201/ADCAIJ2017613140





Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.

Clarivate Analytics