"We address the choice of the tuning parameter $lambda$ in $ell_1$-penalized M-estimation. Our main concern is models which are highly nonlinear, such as the Gaussian mixture model. The number of parameters $p$ is moreover large, possibly larger than the number of observations $n$. The generic chaining technique of Talagrand[2005] is tailored for this problem. It leads to the choice $lambda asymp sqrt {log p / n}$, as in the standard Lasso procedure (which concerns the linear model and least squares loss)."

"Given an i.i.d. sample from a distribution $F$ on $mathbb{R}$ with uniformly continuous density $p_0$, purely data-driven estimators are constructed that efficiently estimate $F$ in sup-norm loss and simultaneously estimate $p_0$ at the best possible rate of convergence over H"older balls, also in sup-norm loss. The estimators are obtained by applying a model selection procedure close to Lepski's method with random thresholds to projections of the empirical measure onto spaces spanned by wavelets or $B$-splines. The random thresholds are based on suprema of Rademacher processes indexed by wavelet or spline projection kernels. This requires Bernstein-type analogs of the inequalities in Koltchinskii [Ann. Statist. 34 (2006) 2593-2656] for the deviation of suprema of empirical processes from their Rademacher symmetrizations."

"About forty years ago it was realized by several researchers that the essential features of certain objects of Probability theory, notably Gaussian processes and limit theorems, may be better understood if they are considered in settings that do not impose structures extraneous to the problems at hand. For instance, in the case of sample continuity and boundedness of Gaussian processes, the essential feature is the metric or pseudometric structure induced on the index set by the covariance structure of the process, regardless of what the index set may be. This point of view ultimately led to the Fernique-Talagrand majorizing measure characterization of sample boundedness and continuity of Gaussian processes, thus solving an important problem posed by Kolmogorov. Similarly, separable Banach spaces provided a minimal setting for the law of large numbers, the central limit theorem and the law of the iterated logarithm, and this led to the elucidation of the minimal (necessary and/or sufficient) geometric properties of the space under which different forms of these theorems hold. However, in light of renewed interest in Empirical processes, a subject that has considerably influenced modern Statistics, one had to deal with a non-separable Banach space, namely $mathcal{L}_{infty}$. With separability discarded, the techniques developed for Gaussian processes and for limit theorems and inequalities in separable Banach spaces, together with combinatorial techniques, led to powerful inequalities and limit theorems for sums of independent bounded processes over general index sets, or, in other words, for general empirical processes."

"We derive a new representation for U- and V-statistics. Using this representation the asymptotic distribution of U- and V-statistics can be derived by a direct application of the Continuous Mapping Theorem. That novel approach not only encompasses most of the results on the asymptotic distribution known in literature, but also allows for the first time a unifying treatment of non-degenerate and degenerate U- and V-statistics. Moreover, it yields a new and powerful tool to derive the asymptotic distribution of very general U- and V-statistics based on long-memory sequences. This will be exemplified by several astonishing examples. In particular, we shall present an example where weak convergence of the U or V-statistic occurs at the rate a_n^3 when a_n is the rate of weak convergence of the empirical process. We also introduce the notion of asymptotic (non-) degeneracy which often appears in the presence of long-memory sequences."

"In this paper, we obtain some uniform laws of large numbers and functional central limit theorems for sequential empirical measure processes indexed by classes of product functions satisfying appropriate Vapnik-Chervonenkis properties."

"Using the renewal approach we prove exponential inequalities for additive functionals and empirical processes of ergodic Markov chains, thus obtaining counterparts of inequalities for sums of independent random variables. The inequalities do not require functions of the chain to be bounded and moreover have all the constants accessible whenever the usual drift condition holds, which is crucial for practical applications e.g. in MCMC algorithms."

"We study weak convergence of empirical processes of dependent data, indexed by classes of functions. We obtain results that are especially suitable for data arising from dynamical systems and Markov chains, where the Central Limit Theorem for partial sums is commonly derived via the spectral gap technique. Our results apply, e.g. to the empirical process of ergodic torus automorphisms."

"While effective concentration inequalities for suprema of empirical processes exist under boundedness or strict tail assumptions, no comparable results have been available under considerably weaker assumptions. In this paper, we derive concentration inequalities assuming only low moments for an envelope of the empirical process. These concentration inequalities are beneficial even when the envelope is much larger than the single functions under consideration."

"We introduce two new concepts designed for the study of empirical processes. First, we introduce a new Orlicz norm which we call the Bernstein-Orlicz norm. This new norm interpolates sub-Gaussian and sub-exponential tail behavior. In particular, we show how this norm can be used to simplify the derivation of deviation inequalities for suprema of collections of random variables. Secondly, we introduce chaining and generic chaining along a tree. These simplify the well-known concepts of chaining and generic chaining. The supremum of the empirical process is then studied as a special case. We show that chaining along a tree can be done using entropy with bracketing. Finally, we establish a deviation inequality for the empirical process for the unbounded case."

"In this paper, we study the generalization bound for an empirical process of samples independently drawn from an infinitely divisible (ID) distribution, which is termed as the ID empirical process. In particular, based on a martingale method, we develop deviation inequalities for the sequence of random variables of an ID distribution. By applying the obtained deviation inequalities, we then show the generalization bound for ID empirical process based on the annealed Vapnik- Chervonenkis (VC) entropy. Afterward, according to Sauer’s lemma, we get the generalization bound for ID empirical process based on the VC dimension. Finally, by using a resulted result bound, we analyze the asymptotic convergence of ID empirical process and show that the convergence rate of ID empirical process can reach O((frac{Lambda_mathcal{F}(2N)}{N})^{frac{1}{1.3}}) and it is faster than the results of the generic i.i.d. empirical process (Vapnik, 1999) "

"Let $(U_n(t))_{tinR^d}$ be the empirical process associated to an $R^d$-valued stationary process $(X_i)_{ige 0}$. We give general conditions, which only involve processes $(f(X_i))_{ige 0}$ for a restricted class of functions $f$, under which weak convergence of $(U_n(t))_{tinR^d}$ can be proved. This is particularly useful when dealing with data arising from dynamical systems or functional of Markov chains. This result improves those of [DDV09] and [DD11], where the technique was first introduced, and provides new applications."

"We present a very general chaining method which allows one to control the supremum of the empirical process $\sup_{h \in H} |N^{-1}\sum_{i=1}^N h^2(X_i)-\E h^2|$ in rather general situations. We use this method to establish two main results. First, a quantitative (non asymptotic) version of the classical Bai-Yin Theorem on the singular values of a random matrix with i.i.d entries that have heavy tails, and second, a sharp estimate on the quadratic empirical process when $H=\{\inr{t,\cdot} : t \in T\}$, $T \subset \R^n$ and $\mu$ is an isotropic, unconditional, log-concave measure."

I think this is the only time I have seen the LIL actually _used_ for anything.

From 1993: "By means of two simple convexity arguments we are able to develop a general method for proving consistency and asymptotic normality of estimators that are defined by minimisation of convex criterion functions. This method is then applied to a fair range of different statistical estimation problems, including Cox regression, logistic and Poisson regression, least absolute deviation regression outside model conditions, and pseudo-likelihood estimation for Markov chains. Our paper has two aims. The first is to exposit the method itself, which in many cases, under reasonable regularity conditions, leads to new proofs that are simpler than the traditional proofs. Our second aim is to exploit the method to its limits for logistic regression and Cox regression, where we seek asymptotic results under as weak regularity conditions as possible. For Cox regression in particular we are able to weaken previously published regularity conditions substantially."

"This paper introduces a version of the argmax continuous mapping theorem that applies to M-estimation problems in which the objective functions converge to a limiting process with multiple maximizers. The concept of the smallest maximizer of a function in the d-dimensional Skorohod space is introduced and its main properties are studied. The resulting continuous mapping theorem is applied to three problems arising in change-point regression analysis. Some of the results proved in connection to the d-dimensional Skorohod space are also of independent interest."

"Given an i.i.d. sample from a distribution F on ℝ with uniformly continuous density p0, purely data-driven estimators are constructed that efficiently estimate F in sup-norm loss and simultaneously estimate p0 at the best possible rate of convergence over Hölder balls, also in sup-norm loss. The estimators are obtained by applying a model selection procedure close to Lepski’s method with random thresholds to projections of the empirical measure onto spaces spanned by wavelets or B-splines. The random thresholds are based on suprema of Rademacher processes indexed by wavelet or spline projection kernels. This requires Bernstein-type analogs of the inequalities in Koltchinskii [Ann. Statist. 34 (2006) 2593–2656] for the deviation of suprema of empirical processes from their Rademacher symmetrizations."

"The concentration of empirical measures is studied for dependent data, whose joint distribution satisfies Poincaré-type or logarithmic Sobolev inequalities. The general concentration results are then applied to spectral empirical distribution functions associated with high-dimensional random matrices."

"In nonparametric classification and regression problems, support vector machines (SVMs) attract much attention in theoretical and in applied statistics. In an abstract sense, SVMs can be seen as regularized M-estimators for a parameter in a (typically infinite dimensional) reproducing kernel Hilbert space. In the article, it is shown that the difference between the empirical SVM and the theoretical SVM is asymptotically normal with rate $\sqrt{n}$. That is, the standardized difference converges weakly to a Gaussian process in the reproducing kernel Hilbert space. This is done by an application of the functional delta-method and by showing that the SVM-functional is suitably Hadamard-differentiable."

"This paper proposes a new notion of typical sequences on a wide class of abstract alphabets (so-called standard Borel spaces), which is based on approximations of memoryless sources by empirical distributions uniformly over a class of measurable "test functions." In the finite-alphabet case, we can take all uniformly bounded functions and recover the usual notion of strong typicality (or typicality under the total variation distance). For a general alphabet, however, this function class turns out to be too large, and must be restricted. With this in mind, we define typicality with respect to any Glivenko-Cantelli function class (i.e., a function class that admits a Uniform Law of Large Numbers)..."

Sequel to their recent Annals of Probability paper, where they use the same trick to get convergence for function classes in terms of the gap (a.k.a. "fat shattering") dimension.

Exchangeable sequences, rather than anything interesting, but worth looking at for ideas?  ETA: Eh.

"A time-varying empirical spectral process indexed by classes of functions is defined for locally stationary time series. We derive weak convergence in a function space, and prove a maximal exponential inequality and a Glivenko--Cantelli-type convergence result. The results use conditions based on the metric entropy of the index class. In contrast to related earlier work, no Gaussian assumption is made. As applications, quasi-likelihood estimation, goodness-of-fit testing and inference under model misspecification are discussed. In an extended application, uniform rates of convergence are derived for local Whittle estimates of the parameter curves of locally stationary time series models."

Cute test for nonlinear autoregressions.

I very much like the approach of treating the likelihood ratio as an empirical process; why haven't I seen it before? (Also, the state-of-the-art in simulating Gaussian processes must be much better now than what Hansen was doing in '92, which would make this even more practical.)

Michael Jordan's theoretical statistics class. Needless to say, I approve.

1994; presumably much development since.