Draft:Dos and Don'ts of Machine Learning in Computer Security

Pitfalls of machine learning in computer security describes a set of common errors and methodological deficiencies identified in the application of machine learning (ML) to computer security problems. According to academic sources, these pitfalls can lead to invalid conclusions, over-optimistic performance estimates, and systems that are ineffective or insecure in practice.

The topic has been the subject of significant academic study, as the complex and adversarial nature of computer security creates unique challenges for standard ML workflows. Researchers have categorized these pitfalls across the typical stages of an ML pipeline, from data collection to real-world deployment.

Categorization of Pitfalls

A 2022 study by Daniel Arp, et al., published at the USENIX Security Symposium, identified and analyzed ten distinct pitfalls by reviewing 30 papers from top-tier security conferences. The study reported that these issues were widespread, with the most common being sampling bias, data snooping, and lab-only evaluations.

Ten Pitfalls in Machine Learning for Computer Security

ML Workflow Stage	Pitfall (P)	Description	Prevalence in Study
Data Collection and Labeling	P1: Sampling Bias	The collected data does not sufficiently represent the true data distribution.	60%
Data Collection and Labeling	P2: Label Inaccuracy	Ground-truth labels are inaccurate, unstable, or erroneous.	10%
System Design and Learning	P3: Data Snooping	The learning model is trained with information typically unavailable in practice.	57%
System Design and Learning	P4: Spurious Correlations	Artifacts unrelated to the security problem create shortcut patterns for separating classes.	20%
System Design and Learning	P5: Biased Parameter Selection	Final parameters indirectly depend on the test set, as they were not entirely fixed at training time.	10%
Performance Evaluation	P6: Inappropriate Baseline	Evaluation is conducted without, or with limited, baseline methods.	20%
Performance Evaluation	P7: Inappropriate Performance Measures	Chosen measures do not account for application constraints, such as imbalanced data.	33%
Performance Evaluation	P8: Base Rate Fallacy	Large class imbalance is ignored when interpreting performance measures.	10%
Deployment and Operation	P9: Lab-Only Evaluation	System is solely evaluated in a laboratory setting, without discussing practical limitations.	47%
Deployment and Operation	P10: Inappropriate Threat Model	The security of machine learning itself is not considered, exposing the system to attacks.	17%

Data Collection and Labeling

This stage involves acquiring and preparing data, which sources identify as a potential origin of subtle bias in security applications.

Sampling bias (P1) occurs when the collected data does not reflect the real-world distribution of data. In security, this is described as potentially happening when relying on limited public malware sources or mixing data from incompatible sources.

Label inaccuracy (P2) arises when ground-truth labels are incorrect or unstable. For example, malware labels from sources like VirusTotal can be inconsistent, and adversary behavior can shift over time, causing "label shift."

System Design and Learning

This stage includes feature engineering and model training, where models may be exposed to information not available in a real-world scenario.

Data snooping (P3) is a common pitfall where a model is trained using information that would not be available in a real-world scenario. This can happen by ignoring time dependencies (temporal snooping) or by cleansing the test set based on global knowledge (selective snooping).

Spurious correlations (P4) result when a model learns to associate artifacts with a label, rather than the underlying security-relevant pattern. For example, a malware classifier might learn to identify a specific compiler artifact instead of malicious behavior itself.

Biased parameter selection (P5) is a form of data snooping where model hyperparameters (e.g., decision thresholds) are tuned using the test set, which can lead to over-optimistic results.

Performance Evaluation

This stage measures a model's performance, where the choice of metrics can impact the perceived validity of the results.

Inappropriate baseline (P6) involves failing to compare a new model against simpler, well-established baselines. Researchers note that a complex deep learning model may not justify its overhead if it does not significantly outperform a simple logistic regression or non-ML heuristic.

Inappropriate performance measures (P7) means using metrics that do not align with the practical goals of the system. For instance, reporting only "accuracy" is often described as insufficient for an intrusion detection system, where false-positive rates are considered critically important.

Base rate fallacy (P8) is a failure to correctly interpret performance in the context of large class imbalances. In tasks like intrusion detection, a 0.1% false-positive rate, while appearing low, could result in an unmanageably high number of false alerts in practice.

Deployment and Operation

This final stage concerns the model's performance and security in a live environment.

Lab-only evaluation (P9) is the practice of evaluating a system only in a controlled, static laboratory setting, which does not account for real-world challenges like concept drift (where data distributions change over time) and performance overhead.

Inappropriate threat model (P10) refers to failing to consider the ML system itself as an attack surface. This includes vulnerability to adversarial attacks (e.g., evasion attacks) that are specifically designed to fool the model.