Summary Points
- Effective dimension measures usable capacity—how many parameter directions the data can actually distinguish, not simply how many weights a model has.
- Estimate it with empirical Fisher matrices: use autograd to compute probability gradients across data samples, skipping near-zero probabilities to avoid numerical issues.
- Use Monte Carlo sampling for parameter-space averages: sample globally across the allowed range, or locally around trained weights; for a uniform ball, scale the radius by (q^Expand Your Tech Knowledge) to avoid over-sampling its center.
- Keep calculations stable—and watch for diminishing returns: log-determinants and log-sum-exp prevent overflow; experiments show effective dimension eventually plateaus as deeper models add redundant parameters.
Build the Fisher Matrix With Autograd
Effective dimension estimates how many parameter directions a model’s data can distinguish. Start by computing an empirical Fisher information matrix at each parameter setting.
For a model that returns outcome probabilities, differentiate each meaningful probability with respect to the weights. Then add the gradient’s outer product, divided by that probability:
python
fisher = torch.zeros(num_params, num_params, device=weights.device)
for p in probabilities:
if p.detach() > 1e-12:
grad = torch.autograd.grad(
p, weights, retain_graph=True
)[0].reshape(-1)
fisher += torch.outer(grad, grad) / p
Average these matrices across input examples. The probability threshold helps avoid unstable divisions when an outcome is nearly impossible.
Sample Parameters, Then Compute the Dimension
The measure also averages across parameter settings. Use Monte Carlo sampling: draw settings, compute a Fisher matrix for each, and average the results. For global effective dimension, sample across the model’s allowed parameter range. For local effective dimension, sample inside a ball around trained weights.
For a uniform sample inside a ball with (d) parameters, draw a random direction and set the radius to epsilon * q**(1/d), where q comes from a uniform distribution between 0 and 1. Sampling the radius uniformly would overfill the center.
Normalize each Fisher matrix by the trace of their average, then calculate the log-volume term:
python
sign, logdet = torch.linalg.slogdet(
identity + kappa normalized_fishers
)
terms = 0.5 logdet
effective_dimension = (
2 * (torch.logsumexp(terms, dim=0) – math.log(len(terms)))
/ math.log(kappa)
)
This uses slogdet and logsumexp to reduce overflow risk. Choose a dataset scale that makes kappa greater than 1.
Read the Result With Care
A rising effective dimension can show that added parameters create useful, distinguishable directions. If the measure levels off while the model grows, some added parameters may contribute little. Comparing local and global estimates can also show how training changes the model’s useful directions.
However, the calculation costs time. Runtime grows with the number of parameter samples and input examples, and small samples can produce noisy estimates. Start with modest sample counts, then check whether larger runs change the result. Effective dimension complements test performance; it does not replace it.
Learn how the Internet of Things (IoT) is transforming everyday life.
Explore past and present digital transformations on the Internet Archive.
Access comprehensive resources on technology by visiting Wikipedia.
AITechV1
