Regularization and the small-ball method I: Sparse recovery

We obtain bounds on estimation error rates for regularization procedures of the form f ∈ argmin _f∈_{F N}¹_i=^N₁ Y_i − f(X_i)² + λ(f) when is a norm and F is convex. Our approach gives a common framework that may be used in the analysis of learning problems and regularization problems alike. In particular, it sheds some light on the role various notions of sparsity have in regularization and on their connection with the size of subdifferentials of in a neighborhood of the true minimizer. As “proof of concept” we extend the known estimates for the LASSO, SLOPE and trace norm regularization.