A Preprocessing Manifold Learning Strategy Based on t-Distributed Stochastic Neighbor Embedding

Abstract

In machine learning and data analysis, dimensionality reduction and high-dimensional data visualization can be accomplished by manifold learning using a t-Distributed Stochastic Neighbor Embedding (t-SNE) algorithm. We significantly improve this manifold learning scheme by introducing a preprocessing strategy for the t-SNE algorithm. In our preprocessing, we exploit Laplacian eigenmaps to reduce the high-dimensional data first, which can aggregate each data cluster and reduce the Kullback-Leibler divergence (KLD) remarkably. Moreover, the k-nearest-neighbor (KNN) algorithm is also involved in our preprocessing to enhance the visualization performance and reduce the computation and space complexity. We compare the performance of our strategy with that of the standard t-SNE on the MNIST dataset. The experiment results show that our strategy exhibits a stronger ability to separate different clusters as well as keep data of the same kind much closer to each other. Moreover, the KLD can be reduced by about 30% at the cost of increasing the complexity in terms of runtime by only 1-2%.

Keywords: dimensionality reducing; k-nearest neighbor; manifold learning; t-SNE.

Figure 1

Visualization of 5000 data randomly selected from MNIST by using t-SNE, the iteration number is 500.

Figure 2

Visualization of 5000 data randomly selected from MNIST by using t-SNE with LE as the preprocessing strategy. Here we take $k_0=40$ and $N_0=80$. The iteration number of t-SNE is also 500.

Figure 3

Visualization of 5000 data selected from MNIST randomly with our preprocessing strategy as shown in Algorithm 1. Here, we take $k_0=k_1=40$, $N_0=80$ and the iteration number is 500. During the first 100 iteration of gradient descent, early exaggeration is exploited.

Figure 4

Visualization of 720 data from 10 objects randomly selected from Coil−100 by using t−SNE.

Figure 5

Visualization of 720 data from 10 objects randomly selected from Coil−100 with our preprocessing strategy.

Figure 6

Visualization of 5000 data randomly selected from Fashion−MNIST by using t−SNE.

Figure 7

Visualization of 5000 data randomly selected from Fashion−MNIST with our preprocessing strategy.

Figure 8

The gradient descent process of the SNE, the standard t-SNE and our preprocessing strategy.

Figure 9

The gradient descent process of our preprocessing strategy when using different parameters $\alpha$.

Figure 10

The gradient descent process of our preprocessing strategy when using different parameters k, where we let k = $k_0$ = $k_1$.