Schedule at a Glance

Registration is open 8:00 AM - 4:00 PM each day.

Online controlled experiments (i.e., A/B tests) have seen a meteoric rise in prominence in the last decade. Generally speaking, such experiments seek to test and improve internet-based products and services using user-generated data to determine what works and what doesn’t. A/B tests have become an indispensable tool for major tech companies when it comes to maximizing revenue and optimizing the user experience, with some companies running hundreds of experiments engaging millions of users each day.

In this online setting, and with a culture of testing as many ideas as possible, as quickly as possible, novel practical issues and modern challenges abound.

This session brings together researchers and practitioners from both industry and academia to discuss such modern challenges and their work on them.

Session Chair: Nathaniel Stevens

Session Sponsor: University of San Francisco Master of Science in Applied Economics

Speakers & Abstracts:
Lo-Hua Yuan, Airbnb: How Airbnb Learns from Product Releases When We Can't A/B Test
Airbnb runs thousands of A/B tests (i.e., randomized controlled experiments) each year, but one notable exception is its biannual major product release. In these major releases, many new features are bundled together and launched globally at the same time for publicity purposes, making standard A/B testing infeasible. Yet, we still need to understand the impacts and causal dynamics behind these global launches, so that we can decide how to prioritize feature iterations post-launch. In this talk, I discuss how we built a measurement framework for Airbnb’s global product releases by casting the problem as an interrupted time series (ITS) design, where we have pre- and post-intervention time series data for only a single treated unit and no comparable controls. The cornerstone of the ITS design is counterfactual time series forecasting: i) Build a model using pre-intervention time series to forecast counterfactual post-intervention trajectories of what we would expect to see in the absence of the intervention; ii) Compare these estimated counterfactual trends to the trends we actually observe post-intervention to estimate time-varying impacts and their uncertainty. I describe several Bayesian regression ITS techniques that we implemented, the tooling we built to scale accessibility and reusability of the ITS workflow, and practical learnings from challenges that we tackled.

Cindy Zhang, Pinterest: Long-term Holdouts as Ground Truth Measurements
At Pinterest we run thousands of online experiments to make decisions on features. Since 2022, we have been running team-level holdout experiments, where we hold back all new features developed by a team in a quarter from certain users. With individual experiments for each of the features and holdout experiments side by side, we observed numerous examples of an effect in the opposite direction to that expected from the winner's curse. The aggregated result from individual experiments showed an underestimate rather than an overestimate as the winner’s curse states, which has led to some changes in how we think about setting goals as a team and demonstrated the importance of holdouts. In this talk, we’ll discuss the potential causes of this phenomenon, the best practices for setting up team-level holdout experiments and our other learnings from the process.

Jiannan Lu, formerly Microsoft ExP: Towards Trustworthy, Efficient and Private A/B tests: Learnings and Insights
Modern A/B testing on-device poses various statistical challenges, from treatment deployment, to data collection, processing and analytics. In this talk, we share our learnings in addressing these challenges, in particular on data-privacy friendly aggregations, and de-biasing methods for treatment non-compliance. Our goal is to not only discuss exploratory solutions, but more importantly, highlight open questions that might be of interest to both researchers and practitioners.

Nicholas Larsen, NCSU: HODOR: A Two-Stage Hold-Out Design for Online Controlled Experiments on Networks
A/B tests are standard tools for estimating the average treatment effect in online controlled experiments (OCEs). The majority of OCE theory relies on the Stable Unit Treatment Value Assumption, which presumes the response of individual users depends only on the assigned treatment, not the treatments of others. Violations of this assumption occur when users are subjected to network interference, a common phenomenon in social media platforms. Standard methods for estimating the average treatment effect typically ignore network effects and produce heavily biased results. Additionally, unobserved user covariates, such as offline information or variables hidden due to privacy restrictions, that influence user response and network structure also bias current estimators of the average treatment effect. In this paper, we demonstrate that network-influential lurking variables can heavily bias popular network clustering-based methods, thereby making them unreliable. To address this problem, we propose a two-stage design and estimation technique called HODOR (Hold-Out Design for Online Randomized experiments). We show that HODOR is unbiased for the average treatment effect, has minimizable variance, and provides reliable estimation even when the underlying network is partially-unknown or uncertain.

Session Chair: Steve Devlin, University of San Francisco

Speakers:

Justin Jacobs, Squared2020, NBA: A Brief Tour into Computer Vision in Sports

With the advances in deep learning and computer processing, difficult tasks such as classifying objects in video have become much more achievable for the sports analyst. In this talk, we will take a look at ways to perform object detection in video and discuss the pros and cons of various types of methods such as optical flow, Yolo, and part affinity fields. Specifically, we will break down some known packages available to the public and identify how to extract data to begin asking more challenging questions.

Jake Toffler, New York Mets: Model Benchmarking in a Competitive Industry Abstract

How do you know how your models compare to the competition when everyone wants to keep their edge a secret?

Thomas Treloar, Hillsdale College: Global and Iterative Ranking Models Based on Network Diffusion

In this talk we discuss families of ranking models based on network diffusion processes. The framework of a diffusion process unites these various methods and provides a strong intuition into how and why they differ. Included in this framework are global methods (i.e. Markov, Colley, Massey, Bradley-Terry rankings) and iterative methods (i.e. Elo and Local Markov rankings).

David Uminsky, University of Chicago: NBA Lineup Attribution: A Fourier Transform Approach

Basketball analytics has seen real growth in methodology and practice in line with the explosive increase in the use of data in professional sports over the past decade. Nonetheless, basketball data has unique challenges and hard questions that remain open for rigorous inquiry. These include: Do winning teams need a "big 3"?Am I a good player, or am I only a good player because I'm on the court with Lebron James? Generally, understanding a player’s contribution to winning basketball individually as well as a member of subgroups on the floor is difficult. In this talk we will present a new, Fourier transform based mathematical tool that was designed to orthogonally break down the player attribution problem. We will demonstrate the tool and findings using lineup level play-by-play data from the 2015-2016 NBA season.

Session Chair: Jeff Hamrick

Speakers:

OLIVER ZEIGERMANN, Open Knowledge: Resilient Machine Learning

Resilience is known as the ability to adapt to difficult or unexpected situations. Such a phenomenon also exists in the field of machine learning, where we have to deal with adversarial attacks, out-of distribution robustness, drift, and unwanted bias. Additionally, model stability must be ensured during retraining and post-training. Measures include monitoring machine learning models, detecting outliers, and running with fallbacks and/or multiple models as an ensemble. Clever model selection for production can also go a long way. In this talk, I explain the phenomena mentioned and address the appropriate measures based on an example in the field of computer vision, which poses additional challenges.

PHIL MUI, Salesforce: An Embedding Analysis on the Impact of Diversity Awareness on Diversity of Ideas

Diversity in its various forms is often touted as vital for progress and social justice. In particular, many colleges are expected to foster "diversity" in admission, student body, and faculty hiring. etc. Has the recent emphasis on diversity across colleges led to a corresponding increase in the diversity of ideas being discussed?

We analyze student opinion writings in college newspapers from the past decade, using word vectors to measure the range of ideas discussed. We then compare this data with the trend in the number of times diversity-related words are mentioned in the same newspapers.

Our main finding is that explicit mentions of diversity have definitely increased across colleges in the past decade. However, the diversity of ideas has often decreased in most colleges during this time. This seems to indicate that while colleges are encouraging more discourse about diversity, the actual space of ideas being discussed in school newspaper opinion sections has unfortunately been shrinking.

While colleges proclaim their desire to promote diversity on their campuses, our research suggests that this desire may not always be reflected in the diversity of ideas present in college newspapers. To ensure that diverse voices are not drowned out, it is important that we continue to examine and address this issue.

KICHO YU, University of Southern California: Flexible and Robust Real-Time Intrusion Detection Systems to Network Dynamics

Deep learning-based intrusion detection systems have advanced due to their technological innovations such as high accuracy, automation, and scalability to develop an effective network intrusion detection system (NIDS). However, most of the previous research has focused on model generation through intensive analysis of feature engineering instead of considering real environments. They have limitations to applying the previous methods for a real network environment to detect real-time network attacks. In this paper, we propose a new flexible and robust NIDS based on Recurrent Neural Network (RNN) with a multi-classifier to generate a detection model in real-time. The proposed system adaptively and intelligently adjusts the generated model with given system parameters that can be used as security parameters to defend against the attacker’s obfuscation techniques in real-time. In the experimental results, the proposed system detects network attacks with a high accuracy and high-speed model upgrade in real-time while showing robustness under an attack.

CONNOR GIBBS, Colorado State University: ECoHeN: A hypothesis testing framework to extract communities from heterogeneous networks

Community discovery is a process of identifying assortative communities in a network: collections of nodes which are densely connected within but sparsely connected to the rest of the network. While community discovery has been extensively studied, there are few techniques available for heterogeneous networks that contain different types of nodes and possibly different connectivity patterns between the node types. In this talk, we introduce a framework called ECoHeN to extract communities from a heterogeneous network in a statistically meaningful way. ECoHeN uses a heterogeneous configuration model as a reference distribution to identify communities that are significantly more densely connected than expected given the node types and connectivity of its membership. The ECoHeN algorithm extracts communities one at a time using a dynamic set of iterative updating rules, is guaranteed to converge, and imposes no constraints on the type composition of extracted communities. To our knowledge, ECoHeN is the first method that can distinguish and identify both homogeneous and heterogeneous, possibly overlapping, community structure in a network. We demonstrate the utility of ECoHeN in the context of a popular political blogs network to identify collections of blogs that reference one another more than expected considering the ideology of its members.

Join mentors and mentees for a sponsored lunch where you can enjoy good food and meaningful conversation.

DJ Patil will sit down with Jeff Hamrick (Data Institute) to talk about his time as our country’s first Chief Data Scientist and much more.

The proliferation of AI has exposed a new area of risk to data privacy and security. Although data may be secured when at rest, ML models may divulge private traces of the data when included in a training set. Data in transit may be secured cryptographically, yet may unintentionally convey sensitive information about individuals to unintended parties through dissemination by the recipient.

Zero trust for hosted ML services presents a challenge of sharing data that may be insecure “at inference”. Additionally, efforts to improve model privacy may be at odds with efforts to track model provenance of large language models that might be used in disinformation campaigns.

In this session, invited speakers confront challenges to keep data secure and private when used to train ML models.

Session Chair:

Hyrum Anderson

Confirmed Speakers:

DANIEL ZIELASKI, Salesforce: The Importance of AI to Business, Marketing, and Digital Products
Artificial intelligence (AI) is one of the most rapidly advancing technologies of our time, with the potential to revolutionize the way businesses operate. As a subset of data science, AI can provide companies with valuable insights into customer behavior, preferences, and trends, enabling them to make informed decisions that drive growth and success. In this presentation, we will explore the importance of AI in marketing, business, and digital product use cases. In addition, we will examine how AI can be used in the development of digital products, such as mobile apps, websites, and e-commerce platforms. We will discuss the benefits of using AI in product design, development, and testing, and provide examples of companies that have leveraged AI to create innovative and successful products. While the benefits of AI are significant, we will also address the risks that businesses must consider as they implement AI. We will provide a link to resources that businesses can use to understand and mitigate these risks, including ethical considerations, data privacy, and security.

NAVEEN JAIN: Privacy by Design: Building Privacy into Your Data Systems from the Ground Up

As data scientists, it's our responsibility to build data systems that protect user privacy by design. In this talk, we'll explore the principles of "privacy by design" and discuss best practices for integrating privacy considerations into your data workflows from the ground up.

We'll start by discussing key concepts such as anonymization and pseudonymization, and explore technical methods, tools, syntax, and data systems that data scientists can use to ensure that sensitive user information is protected throughout the data pipeline. We'll discuss techniques such as differential privacy and k-anonymity, and explore how to use tools such as Python's "pandas" library and SQL to implement privacy-enhancing features.

By the end of this talk, attendees will have a clear understanding of how to implement privacy by design in their own data systems, from data collection and storage through analysis and reporting. They'll gain practical tips for anonymizing and pseudonymizing data, and leave with a toolkit of privacy-preserving techniques that they can use to build more responsible and ethical data systems.

HYRUM ANDERSON, Robust Intelligence: Data and Model Supply Chain Risk in the AI Development Lifecycle

With advancements in generative AI and other large language models, many organizations are forgoing in-house data collection and model development in favor of open source alternatives. While this makes AI accessible to more companies, it also opens the door to substantial unmanaged risk. In this talk, I review practical risks in poisoning webscale datasets, model repositories and models and review solutions for AI Supply Chain Risk Assurance.

ERWIN QUIRING, International Computer Science Institute (ICSI) and Ruhr University Bochum: Dos and Don’ts of Machine Learning in Computer Security

Machine learning algorithms enabled major breakthroughs in many different areas. This development has influenced computer security, spawning a series of work on learning-based security systems, such as for malware detection and vulnerability discovery. Despite great potential, machine learning in security is prone to subtle pitfalls that undermine its performance and render learning-based systems potentially unsuitable for security tasks and practical deployment.

This talk gives an overview of this problem. Generic pitfalls in the design, implementation, and evaluation of learning-based security systems are presented. The talk discusses their prevalence in top-tier security research, and finally shows with case studies how individual pitfalls can lead to unrealistic performance and interpretations, obstructing the understanding of the security problem at hand.

Session Chair:

Jennifer Zhu & Vidya Sagar Ravipati

Confirmed Speakers:

SEAN McCURDY, Pinterest: Intro to Pinterest Ads and ML Systems

Pinterest is one of the world's largest visual discovery and bookmarking platforms, with over 450 million monthly active users and nearly 3b in yearly revenue. Behind this massive platform is a sophisticated machine learning (ML) system that helps to deliver relevant and personalized content to its users. In this talk, we will focus on Pinterest’s Ads Serving Funnel and explain how different components, such as targeting, retrieval, relevance, ranking, and marketplace work together to deliver relevant ads to Pinners. Finally, we will discuss how Pinterest leverages advancements in ML, most notably AutoML to unlock massive gains in revenue and Pinner ad engagement. Overall, attendees will gain a deeper understanding of how recommender systems are monetized and what considerations we made to balance outcomes for Pinners, Partners, and Pinterest.

SAMAN SARRAF, Johnson & Johnson: Computer Vision Applications Across Industries: Wins and Challenges

Numerous industries have shown interests in applying computer vision (CV) techniques to solve real-world use cases. Sports, media entertainment, autonomous driving, manufacturing, and healthcare are among top sectors that utilize such solutions for enhancing their audience engagement, security, and automation purposes. Successful computer vision end-to-end pipelines often include image classification, segmentation, object detection and tracking as well as pose estimation. However, using CV techniques usually face obstacles to meet business success criteria from both model performance and infrastructure perspective. In this presentation, we will review wins and challenges in this domain.

RUIXUAN ZHANG, Airbnb: Machine Learning Driven User Growth and Lifecycle Engagement Marketing

User growth has been a main focus for Airbnb as a two-sided marketplace. The ability of using inhouse targeting mechanisms to precisely identify the right users and proactively target them with relevant messages at the right time, largely enables us to improve marketing efficiency and drive user growth. In this talk, I will describe a machine learning system built at Airbnb that solves user growth problems at a large scale, which focuses on converting existing guests to hosts. In addition, I will also discuss how we used this ML framework in user lifecycle management and engagement marketing, to help increase campaign reach, user engagement and achieve high efficiency.

APRIL LIU, Intuit: AI Application in Fighting Synthetic Account Fraud

Fighting Synthetic Account Fraud with AI solution. Synthetic account fraud can have serious consequences for victims, such as negative credit score impact, financial loss and reputational harm. In recent years, a common ploy by fraudsters is to open up a large number of synthetic accounts that they can quickly recycle and use to cause damage (e.g., file fraudulent tax returns, send spam or phishing emails). At Intuit, we focus on developing AI-powered solutions for live risk assessment to effectively block fraudulent account creation at account sign-up time. In this talk, we’ll describe how we utilize short-term/long-term fraud models by leveraging AI auto-retrain capabilities to protect consumers and small businesses from harm.

Session Chair: Daniel Jerison
Speakers:

Moinak Bhaduri, Bentley University: Distribution-free, Online Change Detection in Multidimensional Point Processes Through Repeated Testing

As we surface, probably momentarily, from the pandemic, other crises thwart normalcy: appalling inequality, climate calamity, distressed refugees, upped possibilities of a fresh Cold War. The enduring motif of our time is incessant chaos. Frequently, that chaos results when one type of a stationary system gives way to another. Change detection is mainly about estimating these points of deviation. In case a Poisson-type point process carries the system forward, I’ll offer a brand of online detection algorithms, engineered through permutations of trend switched statistics and a judicious application of false discovery rate control. Certain members of this family that remain asymptotically consistent and close to the ground truth (evidenced through some Hausdorff-similarity) are isolated to pinpoint estimated change locations. Efficient forecasting proves to be a natural corollary. Change point-based clustering tools will also be examined. I’ll describe how such analyses offer concrete definitions to vague objects like COVID “waves” and measure their enormity.


WIRANTHE BANDARA HERATH, Drake University: Dimension Reduction for Vector Autoregressive Models

The classical vector autoregressive (VAR) models have been widely used to model multivariate time series data, because of their flexibility and ease of use. However, the VAR model suffers from overparameterization particularly when the number of lags and the number of time series get large. There are several statistical methods of achieving dimension reduction of the parameter space in VAR models. In this talk, we introduce the reduced-rank VAR model (Velu et al., 1986; Reinsel and Velu, 2013) which restricts the rank of the parameter matrix in one direction, and the envelope VAR model (Wang and Ding, 2018) which is another solution to overcome the overparameterization problem. Then, we propose a new parsimonious VAR model by incorporating the idea of envelope models into the reduced-rank VAR. We show the strength and efficacy of the proposed model through some simulation studies and an economic data set.

MUSTAFA HAJIJ, University of San Francisco: What is Topological Deep Learning?

Over the past decade, deep learning has been remarkably successful at solving a massive set of problems on data types including images and sequential data. This success drove the extension of deep learning to other discrete domains such as sets, point clouds, graphs, 3D shapes, and discrete manifolds. While many of the extended schemes have successfully tackled notable challenges in each particular domain, the plethora of fragmented frameworks have created or resurfaced many long-standing problems in deep learning such as explainability, expressiveness and generalizability. Moreover, theoretical development proven over one discrete domain does not naturally apply to the other domains. Finally, the lack of a cohesive mathematical framework has created many ad hoc and inorganic implementations and ultimately limited the set of practitioners that can potentially benefit from deep learning technologies. This talk introduces the foundation of topological deep learning, a general mathematical framework for deep learning on topology spaces that expands the domains upon which deep learning protocols can be applied, all while maintaining intuitive conceptualization, implementation and relevance to a wide range of practical applications. We demonstrate the practical relevance of this framework with practical applications ranging from graph classification to mesh and segmentation.

GREESHMA BALABHADRA, Stony Brook University: High-Frequency Risk Estimators Using Change Point Detection Methods

We introduce new high-frequency volatility estimators that account for possible breakpoints in the spot volatility process. We further extend our method to the multivariate case of estimating covariance matrix in both high-frequency and high dimensional low-frequency space using the group fused LASSO method. In the univariate case, the resulting estimators are ℓ1-penalized versions of two classical high-frequency volatility estimators: quadratic variation and a jump-robust version of it, the bipower variation. We show that our estimators are consistent for the actual unobserved volatility. Numerically, the proposed estimators are fast in computations, and they accurately identify breakpoints close to the sample's end - both properties are desirable in the modern electronic trading environment. In terms of out-of-sample volatility prediction, qualitatively, the new estimators provide more smooth and realistic volatility forecasts; quantitatively, they outperform the aforementioned competitors and their extensions at various frequencies and forecasting horizons. In the multivariate case, the proposed estimator is used in minimum variance portfolio optimization and outperforms the benchmarks - realized covariance and multivariate heterogeneous autoregressive estimators in high-frequency and linear shrinkage and quadratic inverse shrinkage in low-frequency space.

Data Science as an Equalizer
With the daily increase in data creation, improvements in technology, and the ever-growing need for informed decision-making, the growth of Data Science and Analytics is only expected to accelerate. As public and private organizations scramble to build their data teams, we focus on two timely issues in this panel discussion.

One, data science, like most STEM fields, has dismaying diversity statistics. Does data science have an image problem? Is it considered impact-driven or vague and nerdy? Are there any unique strengths of the field that can improve diversity in the tech industry?

Two, how can data science inform a more fair and equitable society? What are some recent developments in the public and private sectors that are using data to improve equity? Finally, what is hype vs. reality when we look toward the future of data science and its societal impact?

Join us as we cover a broad range of opinions and perspectives on the role of data science as an equalizer.

Session Chair
Dr. Bushra Anum
Director of Data Science & Analytics, Doximity

Panelists:

Elaine Zhou
CTO, Change.org

Carly Villareal
Head of Data Science & Analytics Engineering, Nextdoor

Megan Yahya, Senior Product Manager, Google Cloud

Mark Freeman, Founder, On the Mark Data

Disclaimer: The views and opinions expressed in this panel discussion are those of the speakers and do not necessarily reflect the views or positions of any entities they represent.

Join us for a reception where you can catch up with industry colleagues, make new friends, and create valuable connections. Music, refreshments, and hors d'oeuvres will be provided.

Make sure to add this valuable networking experience to your conference schedule from 5:30 - 6:30 pm.