Consider a linear regression problem with elastic net regularization. The optimization objective is: \[\arg\min_{\beta}||Y-X\beta||_2^2+\lambda\Big(\alpha||\beta||_1+(1-\alpha)||\beta||_2^2\Big)\]
In many practical applications, it is desirable to impose additional structure on the model coefficients. One might want to enforce monotonicity, apply budget constraints, or—our focus here—limit the contribution of each covariate to the final prediction. This is especially relevant when:
We define the contribution of covariate \(j\) as the absolute value of the product \(\beta_j Z_j\), where \(Z\) is a matrix used to evaluate feature contributions (often the training data or a held-out evaluation set), and we penalize contributions that exceed a per-feature ceiling \(L_j\). This leads to a soft-constrained formulation: \[\min_{\beta_0,\beta}\frac{1}{2N}\sum_{i=1}^N\Big(y_i-(\beta_0+x_i^T\beta)\Big)^2+\lambda\bigg[\frac{(1-\alpha)}{2}||\beta||_2^2+\alpha||\beta||_1\bigg]+\gamma\sum_{j=1}^p\max\big(0,|\beta_jZ_j|-L_j\big)^2\]
where \(p\) is the number of covariates. When \(\gamma=0\) the model reduces to standard elastic net. The cap penalty uses squared violations (\(p=2\)), which is differentiable almost everywhere and facilitates convergence with quasi-Newton methods.
This formulation resembles generalized LASSO-type problems (Tibshirani and Taylor 2011) and constrained penalization frameworks (James et al. 2020), but the penalty is applied to feature contributions rather than coefficients themselves. Because the capped contributions involve data-dependent terms (\(\beta_j X_j\)), there is no simple matrix \(D\) such that the penalty reduces to a standard form \(\|D\beta\|_1\).
The contribution cap penalty renders off-the-shelf solvers like
glmnet unsuitable. Instead, capnet uses:
Both solvers are provided by the lbfgs package. The
internal optimizer selection is automatic:
orthantwise_c = lambda * alpha is passed to
lbfgs(), which switches to OWL-QN whenever this is
positive.
set.seed(123)
example <- simulate_capnet_data()
X <- example$X
y <- example$y
n <- nrow(X)
p <- ncol(X)
L <- example$L
guide_lambda <- exp(seq(1, -5, length.out = 25))
guide_alpha <- seq(0, 1, length.out = 5)
maxx <- t(as.matrix(apply(X, 2, function(x) {
mx <- max(x)
mn <- min(x)
if (abs(mx) > abs(mn)) mx else mn
})))
beta_init <- rep(0, p + 1)The simulator returns a 100$$20 input matrix X, response
vector y, and a contribution cap vector L.
maxx collects the maximum-magnitude value in each
column—used later to evaluate caps against the worst-case
observation.
Use cv_capnet to search over \(\alpha\) and \(\lambda\) via K-fold cross-validation, then
pass the selected hyperparameters to capnet:
set.seed(1)
capnet.cv1 <- cv_capnet(
X, y, 0, L, lambda = guide_lambda, alpha = guide_alpha,
standardize = FALSE
)
alpha1 <- capnet.cv1$best_alpha
lambda1 <- capnet.cv1$best_lambda
capnet.mod1 <- capnet(X, y,
alpha = alpha1, lambda = lambda1,
gamma = 0, L = L, standardize = FALSE
)Best alpha value: 1.000 | Best lambda value 0.0498
Adding a contribution cap penalty (\(\gamma > 0\)) may shift the selected \(\lambda\):
set.seed(2)
capnet.cv2 <- cv_capnet(
X, y, 1, L, lambda = guide_lambda, alpha = guide_alpha,
standardize = FALSE
)
alpha2 <- capnet.cv2$best_alpha
lambda2 <- capnet.cv2$best_lambda
capnet.mod2 <- capnet(X, y,
alpha = alpha2, lambda = lambda2,
gamma = 0.5, L = L, standardize = FALSE
)Best alpha value: 0.250 | Best lambda value 0.0067
plot_redistribution compares two fitted models side by
side, showing both raw coefficient values and the implied per-feature
contributions:
capnet is slower than glmnet because it
relies on a general-purpose quasi-Newton optimizer rather than the
coordinate descent path algorithm. The following timings give a rough
sense of overhead on the simulated data:
## [1] "System runtime for unconstrained 5-fold cross-validation"
## user system elapsed
## 4.482 0.003 4.486
## [1] "System runtime for constrained 5-fold cross-validation"
## user system elapsed
## 4.394 0.009 4.403
## [1] "System runtime for constrained model fit"
## user system elapsed
## 0.015 0.000 0.016
Like glmnet, capnet accepts a
standardize argument (default TRUE) that
scales the columns of X before fitting. The cap penalty is
always evaluated at the original (unstandardized)
scale: internally, coefficients are converted back to raw scale
before computing cap violations, and the resulting gradient is
chain-rule-corrected back into standardized space. This means
z (the evaluation matrix) and L should always
be supplied in the same scale as X, regardless of
standardize.
The multiplier argument scales feature contributions
row-wise during cap enforcement only; it does not affect the likelihood.
When contributions must be evaluated at multiple prediction points with
different per-observation scaling factors, pass multiplier
as an \(n \times 1\) vector. This
avoids the need to reshape L into a matrix aligned with
each observation.
upper.limits and lower.limits enforce hard
bounds on individual coefficients via gradient masking: if a parameter
sits within tol of its bound and the gradient would push it
further out of bounds, that gradient entry is zeroed. Increase
tol if the default fails to enforce the constraint;
decrease it if updates are suppressed prematurely. Because masking is
applied to gradients (not the iterates themselves), initial parameter
values must lie within the specified bounds—the default zero
initialization satisfies symmetric bounds.
capnet does not stop or raise an error when the
optimizer fails to converge. Inspect fit$convergence and
fit$message after fitting: a value other than
0 (or the acceptable rounding-error code
-1001) indicates the optimizer stopped early. A convergence
code of \(-998\) typically means the
line search failed because the gradient became too large or numerically
unstable. Reducing gamma, reducing lambda, or
rescaling ill-conditioned features can help.
The contribution cap optimizer introduces a new hyperparameter \(\gamma\) in addition to the \(\alpha\) and \(\lambda\) parameters of the elastic net
fitter. A large or ill-conceived \(\gamma\) value can force the line search
algorithm in the underlying OWL-QN method to fail. To prevent scaling
mismatch, capnet normalizes the cap loss and gradient so
that they are in the same units as the rest of the objective.
Nevertheless, the algorithm can fail at large \(\gamma\), and the failure threshold depends
heavily on the data and specification. The following table shows
convergence results for eight specifications across a wide range of
\(\gamma\) values.
for (i in seq(-5, 10)) {
# Regular model
model1 <- capnet(X, y,
alpha = 0.5, lambda = 0.01,
gamma = 10**i, L = L, z = X, standardize = TRUE, multiplier = 1
)
convergence1 <- model1$convergence
# Larger lambda model
model2 <- capnet(X, y,
alpha = 0.5, lambda = 0.5,
gamma = 10**i, L = L, z = X, standardize = TRUE, multiplier = 1
)
convergence2 <- model2$convergence
# No elastic net penalty model
model3 <- capnet(X, y,
alpha = 0.5, lambda = 0,
gamma = 10**i, L = L, z = X, standardize = TRUE, multiplier = 1
)
convergence3 <- model3$convergence
# Ridge regression model
model4 <- capnet(X, y,
alpha = 0, lambda = 0.01,
gamma = 10**i, L = L, z = X, standardize = TRUE, multiplier = 1
)
convergence4 <- model4$convergence
# LASSO regression model
model5 <- capnet(X, y,
alpha = 1, lambda = 0.01,
gamma = 10**i, L = L, z = X, standardize = TRUE, multiplier = 1
)
convergence5 <- model5$convergence
# Large multiplier model
model6 <- capnet(X, y,
alpha = 0.5, lambda = 0.01,
gamma = 10**i, L = L, z = X, standardize = TRUE, multiplier = 100
)
convergence6 <- model6$convergence
# Small L model
model7 <- capnet(X, y,
alpha = 0.5, lambda = 0.01,
gamma = 10**i, L = L / 2, z = X, standardize = TRUE, multiplier = 1
)
convergence7 <- model7$convergence
# Single max-row model
model8 <- capnet(X, y,
alpha = 0.5, lambda = 0.01,
gamma = 10**i, L = L, z = maxx, standardize = TRUE, multiplier = 1
)
convergence8 <- model8$convergence
convergence.table <- rbind(
convergence.table,
c(
convergence1, convergence2,
convergence3, convergence4,
convergence5, convergence6,
convergence7, convergence8
)
)
}| Gamma | Base | Large Lambda | No Elastic Net | Ridge | LASSO | Large Multiplier | Small L | Single Max Row |
|---|---|---|---|---|---|---|---|---|
| 1e-05 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1e-04 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1e-03 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1e-02 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1e-01 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1e+00 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1e+01 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1e+02 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1e+03 | 0 | 0 | -1001 | 0 | 0 | 0 | 0 | 0 |
| 1e+04 | 0 | 0 | -1001 | -1001 | 0 | 0 | 0 | 0 |
| 1e+05 | 0 | 0 | -1001 | -1001 | 0 | 0 | 0 | -998 |
| 1e+06 | 0 | 0 | -1001 | -1001 | 0 | 0 | 0 | -998 |
| 1e+07 | -998 | -998 | -1001 | -1001 | -998 | 0 | -998 | -998 |
| 1e+08 | -998 | -998 | -1001 | -1001 | -998 | -998 | -998 | -998 |
| 1e+09 | -998 | -998 | -1001 | -1001 | -998 | -998 | -998 | -998 |
| 1e+10 | -998 | -998 | -1001 | -1001 | -998 | -998 | -998 | -998 |
Most configurations tolerate a wide range of \(\gamma\) values before failing. The failure threshold varies systematically across specifications for reasons rooted in the optimizer’s sensitivity to sudden large gradient changes. The cap penalty gradient has the form
\[\frac{2\gamma}{m} \sum_{i} e_{ij} \cdot \operatorname{sgn}(x_{ij}\hat\beta_j) \cdot \frac{x_{ij}}{\sigma_j}\]
where \(e_{ij} = \max(0,\,|x_{ij}\hat\beta_j| - L_j)\) is the per-observation cap excess and \(m = \text{nrow}(\mathbf{z})\). When this gradient becomes very large relative to the likelihood gradient, L-BFGS and OWL-QN accumulate a corrupted inverse-Hessian approximation over successive iterations, causing the line search to fail.
Optimizer type (Ridge and No Elastic Net fail
earliest). The underlying optimizer is selected via
orthantwise_c \(=
\lambda\alpha\) passed to lbfgs(). When \(\alpha = 0\) (Ridge) or \(\lambda = 0\) (No Elastic Net),
orthantwise_c \(= 0\) and
the solver falls back to standard L-BFGS. All other configurations with
\(\alpha > 0\) and \(\lambda > 0\) use OWL-QN, which enforces
an orthant constraint at each step: coefficients cannot cross zero
within a single iteration. This constraint bounds step sizes and
prevents the large overshoots that corrupt the curvature history when
\(\gamma\) is large. L-BFGS has no such
bound and therefore degrades faster, explaining why Ridge and No Elastic
Net fail at lower \(\gamma\) than the
OWL-QN models.
Number of evaluation rows \(m\) (Single Max Row fails earliest overall). The cap gradient is divided by \(m\), the number of rows in \(\mathbf{z}\). With \(m = n\) (models 1–7, \(\mathbf{z} = X\)), this average moderates the gradient magnitude: each observation contributes only \(1/n\) of its excess to the gradient. With \(m = 1\) (model 8, \(\mathbf{z} = \mathbf{x}_{\max}\)), the full excess of the single worst-case row enters the gradient undiluted, making the cap gradient \(n\) times larger for the same \(\gamma\). This is why model 8 fails at a \(\gamma\) roughly \(n\) times smaller than the baseline.
Effective cap threshold (Large Multiplier and Small L fail
earlier). A multiplier \(> 1\) tightens the effective cap: the
excess term becomes \(\max(0,\,|\text{multiplier}_i \cdot
x_{ij}\hat\beta_j| - L_j) / \text{multiplier}_i =
\max(0,\,|x_{ij}\hat\beta_j| - L_j/\text{multiplier}_i)\), so
multiplier = 100 reduces the effective ceiling from \(L_j\) to \(L_j/100\). Similarly, halving \(L\) (Small L) directly tightens the cap.
Both changes produce more cap violations and larger excess values during
optimization, increasing the cap gradient magnitude and lowering the
failure threshold.
Regularization strength (Larger \(\lambda\) tolerates higher \(\gamma\)). Stronger elastic net regularization keeps \(\hat\beta\) smaller throughout the optimization path, which reduces cap violations and therefore the magnitude of the cap gradient. A model with \(\lambda = 0.5\) reaches the cap boundary later (at higher \(\gamma\)) than the baseline \(\lambda = 0.01\), while \(\lambda = 0\) (No Elastic Net) eliminates all coefficient-shrinkage pressure and increases violations, shifting the failure threshold downward.
The elastic net regularization and contribution cap constraints are correlated with each other despite serving different roles: both shrink coefficients and reduce realized contributions. This interaction can destabilize cross-validation. The following experiment runs 10 independent CV rounds at three \(\gamma\) values to illustrate the effect. Each run uses five folds and six \(\lambda\) values, with the maximum-magnitude row of \(X\) as the evaluation matrix so cap violations are activated across folds.
K <- 5L
n_runs <- 10L
lambda_grid <- 10^seq(-11, -1, length.out = 6)
run_one <- function(i) {
set.seed(i)
cv0 <- cv_capnet(X, y,
gamma = 0, L = L,
alpha = 0.5, lambda = lambda_grid,
z = maxx, K = K, standardize = TRUE, multiplier = 1
)
cv01 <- cv_capnet(X, y,
gamma = 0.1, L = L,
alpha = 0.5, lambda = lambda_grid,
z = maxx, K = K, standardize = TRUE, multiplier = 1
)
cv100 <- cv_capnet(X, y,
gamma = 100, L = L,
alpha = 0.5, lambda = lambda_grid,
z = maxx, K = K, standardize = TRUE, multiplier = 1
)
fit0 <- capnet(X, y,
alpha = 0.5, lambda = cv0$best_lambda,
gamma = 0, L = L, z = maxx, standardize = TRUE, multiplier = 1
)
fit01 <- capnet(X, y,
alpha = 0.5, lambda = cv01$best_lambda,
gamma = 0.1, L = L, z = maxx, standardize = TRUE, multiplier = 1
)
fit100 <- capnet(X, y,
alpha = 0.5, lambda = cv100$best_lambda,
gamma = 100, L = L, z = maxx, standardize = TRUE, multiplier = 1
)
list(
lambda0 = cv0$best_lambda, beta0 = as.numeric(fit0$beta),
lambda01 = cv01$best_lambda, beta01 = as.numeric(fit01$beta),
lambda100 = cv100$best_lambda, beta100 = as.numeric(fit100$beta)
)
}
results <- lapply(seq_len(n_runs), run_one)
lambdas0 <- sapply(results, `[[`, "lambda0")
lambdas01 <- sapply(results, `[[`, "lambda01")
lambdas100 <- sapply(results, `[[`, "lambda100")
betas0 <- do.call(rbind, lapply(results, `[[`, "beta0"))
betas01 <- do.call(rbind, lapply(results, `[[`, "beta01"))
betas100 <- do.call(rbind, lapply(results, `[[`, "beta100"))When \(\gamma = 0\), cross-validation reliably identifies a concentrated range of optimal \(\lambda\) values. When a small \(\gamma = 0.1\) is introduced, the selected \(\lambda\) values spread and generally shift lower—the model learns to rely on contribution caps rather than \(\ell_1\)/\(\ell_2\) norms for shrinkage. At \(\gamma = 100\), the distribution becomes bimodal: some runs pick a very small \(\lambda\) (caps dominate regularization) while others pick a very large one (elastic net shrinks coefficients so aggressively that cap violations rarely occur). This bimodality arises because the two penalties are redundant—both shrink coefficients and reduce excess contributions—so the CV error surface flattens along the \(\lambda\) axis. Under random fold splits, the argmin can jump between the two grid extremes. Because this instability stems from correlated regularization, running cross-validation on \(\lambda\) while holding \(\gamma\) fixed can produce unreliable results; users should fix \(\gamma\) based on domain knowledge and treat it as a model-design choice rather than a tuning parameter.
Although different CV rounds may select very different \(\lambda\) values when caps are active, the downstream coefficient estimates remain stable. The following plot shows the mean coefficient (solid line) and the full range (min-to-max band) across the 50 runs for each \(\gamma\) level.
The narrow confidence bands—barely visible even at \(\gamma = 100\)—confirm that while cross-validation selects very different \(\lambda\) values across runs, the model coefficients are nearly identical. This is consistent with the correlated-regularization interpretation: the elastic net and cap penalties trade off against each other along a near-flat objective surface, so the optimizer reaches essentially the same point regardless of which end of the \(\lambda\) grid is chosen.
The first experiment uses the QuickStartExample data
from glmnet. All features are well-behaved with stable
variance, so cap violations should be rare and the three models should
perform similarly.
ds_qs <- prepare_data("quickstart")
set.seed(123)
fits_qs <- fit_models(
ds_qs$train_X, ds_qs$train_y, ds_qs$test_X, ds_qs$L,
ds_qs$gamma
)
qs_enet_is <- mse_from_beta(
ds_qs$train_X, ds_qs$train_y, fits_qs$enet$beta0,
fits_qs$enet$beta
)
qs_cap1_is <- mse_from_beta(
ds_qs$train_X, ds_qs$train_y, fits_qs$cap1$beta0,
fits_qs$cap1$beta
)
qs_enet_oos <- mse_from_beta(
ds_qs$test_X, ds_qs$test_y, fits_qs$enet$beta0,
fits_qs$enet$beta
)
qs_cap1_oos <- mse_from_beta(
ds_qs$test_X, ds_qs$test_y, fits_qs$cap1$beta0,
fits_qs$cap1$beta
)
qs_cap2_oos <- mse_from_pred(fits_qs$cap2$predictions, ds_qs$test_y)
report_qs <- data.frame(
Model = c("Elastic Net (post-fit cap)", "Optimized Cap", "Walk Cap"),
IS = c(qs_enet_is, qs_cap1_is, NA),
OOS = c(qs_enet_oos, qs_cap1_oos, qs_cap2_oos)
)
knitr::kable(report_qs, digits = 6)| Model | IS | OOS |
|---|---|---|
| Elastic Net (post-fit cap) | 0.501179 | 0.591483 |
| Optimized Cap | 0.495869 | 0.580050 |
| Walk Cap | NA | 0.538647 |
The in-sample (IS) and out-of-sample (OOS) MSEs show that the elastic net and optimized cap models perform comparably. The walk-forward model achieves slightly better OOS performance, attributable to its per-observation refitting—but at higher computational cost and without a single stable coefficient vector.
To illustrate the strength of capnet, the second
experiment uses a synthetic latent data-generating process \(Z \rightarrow Y\), where the true
predictors \(Z\) drive the outcome but
only a noisy proxy \(X\) is observed.
Some columns of \(X\) have unstable
variance—occasionally taking extreme values with probability
high_prob. Under these conditions, a few extreme
observations can dominate an elastic net fit, inflating OOS error. The
contribution cap explicitly limits how much any single feature can
contribute, regardless of the magnitude of its realized value.
ds_px <- prepare_data("proxy")
set.seed(123)
fits_px <- fit_models(
ds_px$train_X, ds_px$train_y, ds_px$test_X, ds_px$L,
ds_px$gamma
)
px_enet_is <- mse_from_beta(
ds_px$train_X, ds_px$train_y, fits_px$enet$beta0,
fits_px$enet$beta
)
px_cap1_is <- mse_from_beta(
ds_px$train_X, ds_px$train_y, fits_px$cap1$beta0,
fits_px$cap1$beta
)
px_enet_oos <- mse_from_beta(
ds_px$test_X, ds_px$test_y, fits_px$enet$beta0,
fits_px$enet$beta
)
px_cap1_oos <- mse_from_beta(
ds_px$test_X, ds_px$test_y, fits_px$cap1$beta0,
fits_px$cap1$beta
)
px_cap2_oos <- mse_from_pred(fits_px$cap2$predictions, ds_px$test_y)
report_px <- data.frame(
Model = c("Elastic Net (post-fit cap)", "Optimized Cap", "Walk Cap"),
IS = c(px_enet_is, px_cap1_is, NA),
OOS = c(px_enet_oos, px_cap1_oos, px_cap2_oos)
)
knitr::kable(report_px, digits = 6)| Model | IS | OOS |
|---|---|---|
| Elastic Net (post-fit cap) | 0.860357 | 0.795325 |
| Optimized Cap | 0.803383 | 0.745417 |
| Walk Cap | NA | 0.686337 |
Both cap models outperform the elastic net baseline on OOS MSE. The walk-forward implementation achieves the greatest improvement because it refits on the most recent window before each prediction, adapting to any shift in the spike pattern seen out of sample.
Social Science