This project implements iterative Principal Component Analysis on the MNIST dataset. We focused only on digit zero to helpe reduce the workload. We implemented two learning methods, the first was one-step learning using Oja’s rule, the second was two-step learning with explicit normalization. The online version of the algorithm gave us NaN errors, which we switched to a batch update version. That fixed the numerical instability. The dot product between the two methods was 1.000000, which means they converged to identical principal components. The reconstruction error went down, with one PC it was 3416.86, and with two PCs it dropped to 2896.58. That is a reduction of 15.23%. The second PC was orthogonal to the first, with dot product was 0.000000.
MNIST stands for Modified National Institute of Standards and Technology. MNIST is a database of handwritten digits used for training and testing many image processing systems and machine learning research [6]. It has 60,000 training images and 10,000 testing images of handwritten digits [6]. Later in 2017, an extended database has been published that has 240,000 training images and 40,000 testing images. Since MNIST is a standard training dataset for digits in English, in recent times, others have also provided similar databases for training digit datasets in other languages. The dataset is considered as a benchmark for neural networks worldwide. The MNIST database is good for people who want to try machine learning techniques and neural network methods on real-world data while spending minimal efforts on preprocessing and formatting [6]. It reduces the time and effort spent on preprocessing and formatting of data. The MNIST dataset was used, then neural layers with fully connected (dense) architectures implemented. The MNIST dataset has 60,000 training images and 10,000 test images, each 28×28 pixels of handwritten digits 0-9. It is the standard benchmark for handwritten digit recognition. The principal component analysis, is a multivariate technique that analyzes a data table in which observations are described by several inter-correlated quantitative dependent variables [2]. PCA extracts the important information from the statistical data to represent it as a set of new orthogonal variables called principal components, and to display the pattern of similarity between the observations and of the variables as points in spot maps [2]. In this project, we implemented iterative Principal Component Analysis (PCA) on the MNIST dataset. . To reduce workload, we kept only images of digit zero. We used two approaches for learning the principal components. The first approach is one-step learning using Oja’s rule [7]. The second approach is two-step learning with explicit normalization. We reconstructed the original data and computed the error in the first principal com- ponent. In the second principal component, we removed the first PC from the data. We reconstructed using the first two PCs, and computed the reconstruction error again.
Publication Date: 2021-04-18