Autoencoder

Autoencoders are a type of artificial neural network based machine learning method that learns by compressing input data into a lower-dimensional meaningful representation and reconstructing the input from this representation. First introduced in the 1980s, autoencoders operate within the framework of unsupervised learning and aim to learn important features within the data to construct a low-dimensional representation. In this architecture where input and output data are identical, the training process seeks to minimize data loss. Autoencoders are used in many areas such as data compression, feature extraction, dimensionality reduction, denoising, and generative models.

Autoencoders consist primarily of two main components: a encoder and a decoder. The encoder transforms the input into a low-dimensional code while the decoder attempts to reconstruct the original data from this code.

^{The basic structure of an Autoencoder. (Created with YZ.)}

The encoder converts high-dimensional input data into a lower-dimensional hidden layer representation. The goal in this process is to represent the data without losing its most important features. The low-dimensional representation produced by the encoder is called the code or latent representation.

The decoder takes this code and generates an output that is an approximation of the original input. Autoencoders are optimized so that the output closely matches the input data.

Autoencoders are defined by the learning of two functions:

• A : Rⁿ→R^p (Encoder)
• B : R^p→Rⁿ (Decoder)

The goal is to minimize the following optimization problem:

argmin_A,B E[Δ(x,B(A(x)))]

Here, the function Δ (Delta) measures the difference between the output and the original input, and the ℓ2-norm (mean squared error) is typically used.

Different variations of autoencoders have been developed to serve specific purposes. These variations maintain the basic structure but employ different regularization strategies and loss functions.

• Undercomplete Autoencoder: The hidden layer dimension is smaller than the input layer dimension. This forces the network to learn only the most important features.
• Overcomplete Autoencoder: The hidden layer dimension is equal to or larger than the input dimension. In this case, additional regularization is required to prevent overfitting.

In sparse autoencoders, a large portion of the hidden layer activations are encouraged to be close to zero. This ensures that only the most important features of the data are extracted. Sparsity is typically achieved using methods such as L1 regularization or Kullback-Leibler divergence.

Denoising autoencoders intentionally add noise to the input data and train the network to reconstruct the original clean data. This approach helps the model learn more robust and generalizable features.

Variational autoencoders differ from classical autoencoders in that they are generative models. They generate data by sampling from a probability distribution over the latent representation. In this structure, the encoder learns the parameters of a probability distribution (mean and variance), while the decoder generates data using samples drawn from these parameters.

Autoencoders have a wide range of applications. Different configurations can be used for various types of problems.

^{Autoencoderların günlük hayatta kullanımları hakkında. (YZ ile oluşturulmuştur.)}

Autoencoders can be used to reduce data to a lower-dimensional representation. Although this use is similar to traditional dimensionality reduction methods such as Principal Component Analysis - PCA, autoencoders can also learn nonlinear relationships.

Denoising autoencoders are particularly used in image processing to remove noise from data. The network learns to predict the original clean data from corrupted input.

After being trained on normal data samples, autoencoders can be used to detect anomalies. Anomalous data will exhibit high reconstruction error because the network cannot accurately reconstruct it.

Variational autoencoders (VAEs) in particular can be used to generate new data samples. New data can be created by randomly sampling from the distribution learned during training.

Autoencoders are also used for feature extraction from data. In particular, for other machine learning tasks such as classification or clustering, the encoder part of the autoencoder can serve as a feature extractor.