136.136. Weight Decay

Weight Decay

Implement weight decay, a regularization technique that shrinks the model weights towards zero at each training step. Weight decay penalizes large weights by subtracting a fraction of the current weight value during each update, preventing the model from becoming overly complex and overfitting.

Formula:

w_new = w * (1 - decay_rate)

Equivalently (during gradient descent):

w_new = w - lr * gradient - decay_rate * w

Where:

w = current weight value

decay_rate = weight decay rate (lambda), typically 0.0001 to 0.01

w_new = updated weight value (closer to zero)

Example:

Input: weights = [1, 2, 3], weight_decay_rate = 0.1

Updated weights:

w[0] = 1 * (1 - 0.1) = 1 * 0.9 = 0.9

w[1] = 2 * (1 - 0.1) = 2 * 0.9 = 1.8

w[2] = 3 * (1 - 0.1) = 3 * 0.9 = 2.7

Output: [0.9, 1.8, 2.7]

**Explanation:** Weight decay acts as a form of L2 regularization. By continuously shrinking weights toward zero, it discourages the model from relying too heavily on any single feature or developing very large weights that can lead to overfitting. Larger weights are penalized more heavily (since the decay is proportional to the weight magnitude), encouraging the model to distribute its capacity more evenly across features. Weight decay is built into most modern optimizers (e.g., AdamW separates weight decay from the gradient update for better training dynamics).

Constraints: