L1 and L2 regularization

L1 and L2 penalties are forms of regularization used to prevent overfitting in machine learning models by adding constraints to the model’s complexity.

L1 Penalty (Lasso Regularization)

The L1 penalty adds the absolute values of the coefficients (weights) to the loss function.

$$L1=\lambda \sum |w_i|$$

Where: $w_i$ are the model weights (coefficients), and
$\lambda$ is the regularization parameter that controls the strength of the penalty.

It encourages sparsity by shrinking some coefficients to exactly zero, effectively selecting only the most relevant features. It’s ideal when you suspect many features are irrelevant and want to perform feature selection.

L2 Penalty (Ridge Regularization)

The L2 penalty adds the squared values of the coefficients (weights) to the loss function.

$$L2=\lambda \sum w_i^2$$

Where: $w_i$ are the model weights (coefficients), and
$\lambda$ is the regularization parameter that controls the strength of the penalty.

It penalizes large coefficients, leading to smaller weights but not exactly zero. This helps reduce model sensitivity to individual data points: it is suitable when you want to reduce overfitting while retaining all features in the model.

Combined Use: Elastic Net

In Elastic Net regularization, both L1 and L2 penalties are combined:

$$\text{Elastic Net Loss}=\text{Loss Function}+{\lambda }_1\sum |w_i|+{\lambda }_2\sum w_i^2$$

Where:
${\lambda }_1$ controls the strength of the L1 penalty, and
${\lambda }_2$ controls the strength of the L2 penalty.