149.149. Pruning Neural Networks

Pruning Neural Networks

Implement magnitude-based weight pruning for neural networks. Pruning removes weights whose absolute values fall below a given threshold by setting them to zero, creating a sparse network. This reduces model size and computational cost while aiming to preserve accuracy.

Algorithm (Magnitude Pruning):

1. For each weight w in the weight matrix:

- If |w| < threshold: set w = 0 (prune)

- Otherwise: keep w unchanged

2. Equivalently: mask = |weights| >= threshold

pruned_weights = weights * mask

Using NumPy:

pruned = np.where(np.abs(weights) < threshold, 0, weights)

Example:

Input: weights = [[0.1, 0.2],

[0.3, 0.4]], threshold = 0.15

|0.1| = 0.1 < 0.15 -> pruned to 0

|0.2| = 0.2 >= 0.15 -> kept

|0.3| = 0.3 >= 0.15 -> kept

|0.4| = 0.4 >= 0.15 -> kept

Output: [[0, 0.2],

[0.3, 0.4]]

Small-magnitude weights contribute less to the network output and can often be removed without significantly hurting performance. After pruning, the network can be fine-tuned to recover any lost accuracy. The resulting sparse weight matrices can be stored more efficiently and enable faster inference.

Constraints: