Today’s data pose unprecedented challenges to statisticians. It may be incomplete, corrupted or exposed to some unknown source of contamination or adversarial attack. We need new methods and theories to grapple with these challenges. Robustness is one of the revived concepts in statistics and machine learning that can accommodate such complexity and glean useful information from modern datasets. This workshop will address several aspects of robustness such as statistical estimation and computational efficiency in the context of modern high-dimensional data analysis.

The workshop is part of the Fall 2021 Special Quarter on Robustness in High-dimensional Statistics and Machine Learning; co-organized by Professors Yu Cheng (University of Illinois at Chicago), Chao Gao (University of Chicago), and Aravindan Vijayaraghavan (Northwestern University).

• Date: Tuesday, October 19, 2021 11:00am-2:45pm CDT (Chicago Time)
• Location: Virtual IDEAL (on gather.town)- watch the full event here
• Free Registration: Attendees must register to receive information for joining. Login information for Gather.Town will be provided via email. 

Schedule 

• 11:00-11:05: Introduction
• 11:05-11:35: Po-Ling Loh (Univ. of Cambridge) watch the talk here
• 11:35-12:05: Ilias Diakonikolas (UW Madison) watch the talk here
• 12:05-1:00 CT: Lunch & Q&A on gather.town
• 1:00-1:30: Ankur Moitra (MIT) watch the talk here
  
• 1:30-2:00: Jacob Steinhardt (UC Berkeley) watch the talk here
• 2:00-2:45: Q&A + Socializer in gather.town

Titles and Abstracts

Speaker: Po-Ling Loh
Title: Robust regression with covariate filtering
Abstract: We study the problem of linear regression where both covariates and responses are potentially (i) heavy-tailed and (ii) adversarially contaminated. Several computationally efficient estimators have been proposed for the simpler setting where the covariates are sub-Gaussian and uncontaminated; however, these estimators may fail when the covariates are either heavy-tailed or contain outliers. In this work, we show how to modify the Huber regression, least trimmed squares, and least absolute deviation estimators to obtain estimators which are simultaneously computationally and statistically efficient in the stronger contamination model. Our approach is quite simple, and consists of applying a filtering algorithm to the covariates, and then applying the classical robust regression estimators to the remaining data. We show that the Huber regression estimator achieves near-optimal error rates in this setting, whereas the least trimmed squares and least absolute deviation estimators can be made to achieve near-optimal error after applying a postprocessing step. 

This is joint work with Ankit Pensia (UW-Madison) and Varun Jog (Cambridge).

Speaker: Ilias Diakonikolas
Title: Distribution-Free Learning with Massart Noise
Abstract: In recent years, there has been an explosion of research in algorithmic high-dimensional robust statistics– that is, high-dimensional learning and inference in the presence of adversarial corruptions. A limitation of these algorithmic results is the assumption that the clean data comes from a “well-behaved” distribution. This suggests the following question: Are there realistic corruption models that allow for efficient learning
in the distribution-free setting? In this talk, we will survey recent work that shed new light on this broad question. We focus on the classical problem of PAC learning with Massart noise. In the Massart model, an adversary can flip the label of each example independently with probability at most $eta<1/2$. Our findings highlight the algorithmic possibilities and limitations of distribution-free robustness with respect to natural semi-random noise models and suggest a number of research directions.

Speaker: Ankur Moitra
Title: Regression and Contextual Bandits with Huber Contamination
Abstract: In this talk we will revisit two classic online learning problems, linear regression and contextual bandits, from the perspective of adversarial robustness. Existing works in algorithmic robust statistics make strong distributional assumptions that ensure that the input data is evenly spread out or comes from a nice generative model. Is it possible to achieve strong robustness guarantees even without distributional assumptions altogether, where the sequence of tasks we are asked to solve is adaptively and adversarially chosen?

We answer this question in the affirmative and design simple new algorithms that succeed where conventional methods, such as those based on convex M-estimation, fail. Moreover our algorithms obtain essentially optimal dependence on the contamination level, reach the optimal breakdown point, and naturally apply to infinite dimensional settings where the feature vectors are represented implicitly via a kernel map.

This is based on joint work with Sitan Chen, Frederic Koehler and Morris Yau.

Speaker: Jacob Steinhardt
Title: Regard Learning as Doubly Nonparametric Bandits: Optimal Design and Scaling Laws
Abstract: Specifying reward functions for complex tasks like object manipulation or driving is challenging to do by hand. Reward learning addresses this by learning a reward model from human feedback on selected query policies, which shifts the burden to optimal design of the queries. We present a theoretical framework for studying reward learning, modeling rewards and policies as non-parametric functions belonging to subsets of Reproducing Kernel Hilbert Spaces (RKHSs). For this setting, we first derive non-asymptotic excess risk bounds for a simple plug-in estimator based on ridge regression. We then solve the query design problem by optimizing these risk bounds with respect to the choice of query set. Despite the generality of our setting, our bounds are stronger than previous bounds developed for more specialized problems. We specifically show that the well-studied problem of Gaussian process (GP) bandit optimization is a special case of our framework, and that our bounds either improve or are competitive with known regret guarantees for the Matern kernel.