A Novel Approach to Outliers Removal in a Noisy Numeric Dataset for Efficient Mining

  • Abdulraheem Ajiboye Department of Computer Science, University of Ilorin, Ilorin
  • Kayode Zakariyah Adewole
  • R.S. Babatunde
  • Idowu Dauda Oladipo
Keywords: Algorithm, Data mining, Data pre-processing, Outliers, Prediction

Abstract

Data pre-processing is a key task in the data mining process. The task generally consumes the largest portion of the total data engineering effort while unveiling useful patterns from datasets. Basically, data mining is about fitting descriptive or predictive models from data. However, the presence of outlier sometimes reduces the reliability of

the models created. It is, therefore, essential to have raw data properly pre-processed before exploring them for mining. In this paper, an algorithm that detects and removes outliers in a numeric dataset is proposed. In order to establish the effectiveness of the proposed algorithm, the clean data obtained through the implementation of the proposed approach is used to create a prediction model. Similarly, the clean data obtained through the use of one of the existing techniques is also used to create a prediction model. Each of the models created is simulated using a set of untrained data and the error associated with each model is measured. The resulting outputs from the two approaches reveal that, the prediction model created using the output from the proposed algorithm has an error of 0.38, while the prediction model created using the cleaned data from the clustering method gives an error of 0.61. Comparison of the errors associated with the models created using the two approaches shows that, the proposed algorithm is suitable for

cleaning numeric dataset. The results of the experiment also unveils that, the proposed approach is efficient and can be used as an alternative technique to other existing cleaning methods.

References

Ahmed, I., & Aziz, A. (2010). Dynamic approach for data scrubbing
process. International Journal on Computer Science and Engineering,
2(02), 416-423.
Databank. (2012). World Bank Development Indicator, Annual GDP
Growth. from
http://databank.worldbank.org/data/reports.aspx?Code=NY.GD
P.MKTP.KD.ZG&id=af3ce82b&report_name=Popular_indicators&
populartype=series&ispopular=y; accessed on December 15, 2015.
Ebaid, A., Elmagarmid, A., Ilyas, I. F., Ouzzani, M., Quiane-Ruiz, J.-A.,
Tang, N., & Yin, S. (2013). NADEEF: A generalized data cleaning
system. Proceedings of the VLDB Endowment, 6(12), 1218-1221.
Han, J., Kamber, M., & Pei, J. (2012). Data mining, southeast asia edition:
Concepts and techniques: Morgan kaufmann.
Hodge, V. J., & Austin, J. (2004). A survey of outlier detection
methodologies. Artificial Intelligence Review, 22(2), 85-126.
Jiang, M.-F., Tseng, S.-S., & Su, C.-M. (2001). Two-phase clustering process
for outliers detection. Pattern recognition letters, 22(6), 691-700.
Kaufman, L. R., & Rousseeuw, P. (1990). Finding groups in data: An
introduction to cluster analysis: John Wiley & Sons Inc.
Last, M., & Kandel, A. (2001). Automated detection of outliers in real-world
data. Paper presented at the Proceedings of the second international
conference on intelligent technologies.
Liu, H., Shah, S., & Jiang, W. (2004). On-line outlier detection and data
cleaning. Computers and Chemical Engineering.
Loureiro, A., Torgo, L., & Soares, C. (2004). Outlier detection using clustering
methods: a data cleaning application. Paper presented at the
Proceedings of KDNet Symposium on Knowledge-based systems
for the Public Sector.
Low, W. L., Lee, M. L., & Ling, T. W. (2001). A knowledge-based approach
for duplicate elimination in data cleaning. Information Systems,
26(8), 585-606.
Mohamed, H. H., Kheng, T. L., Collin, C., & Lee, O. S. (2011). E-Clean: A
Data Cleaning Framework for Patient Data. Paper presented at the
Informatics and Computational Intelligence , ICI Conference, 2011.
Paul, A., Ganesan, V., Challa, J. S., & Sharma, Y. (2012). HADCLEAN: A
hybrid approach to data cleaning in data warehouses. Paper presented at
the Information Retrieval & Knowledge Management (CAMP),
2012.
Raman, V., & Hellerstein, J. M. (2001). Potter's wheel: An interactive data
cleaning system. Paper presented at the VLDB.
Sim, K., & Hartley, R. (2006). Removing outliers using the Linfty norm. Paper
presented at the Computer Vision and Pattern Recognition, IEEE
Conference, 2006.
Singh, K., & Upadhyaya, S. (2012). Outlier detection: applications and
techniques. International Journal of Computer Science Issues, 9(1), 307-
323.
Witten, I. H., Frank, E., & Hall, M. A. (2011). DATA MINING Practical
Machine Learning Tools and Techniques (3rd Edition ed.): Morgan
Kaufmann.
Zhang, S., Zhang, C., & Yang, Q. (2003). Data preparation for data mining.
Applied Artificial Intelligence, 17(5-6), 375-381.
Published
2020-05-03
How to Cite
Ajiboye, A., Adewole, K. Z., Babatunde, R., & Oladipo, I. D. (2020). A Novel Approach to Outliers Removal in a Noisy Numeric Dataset for Efficient Mining. Ilorin Journal of Computer Science and Information Technology, 1(1), 39 - 53. Retrieved from https://iljcsit.com.ng/index.php/ILJCSIT/article/view/11