Dissecting malware dynamics: gleaning behavioral and semantic insights for modern threats

Malware remains a pervasive and rapidly evolving threat to modern computing systems, with adversaries continuously adapting their techniques to evade defenses. This sustained arms race exposes fundamental limitations in prevailing detection paradigms, particularly those relying on low-level artifacts, isolated execution traces, or narrowly scoped behavioral heuristics. Such approaches lack principled abstraction mechanisms capable of capturing malicious intent across implementation variants, concealment strategies, and execution-dependent behavior. Advancing dynamic malware detection therefore requires abstractions that remain stable under variability while preserving discriminative power. This thesis advances dynamic malware detection along two complementary dimensions: abstraction over technique variability and over execution variability. First, we investigate process injection, a widely adopted evasion technique in which malicious payloads execute within the address space of legitimate processes to bypass process-level monitoring. We introduce a behavior-centric detection methodology grounded in the observation that injection variants—despite surface diversity—rely on three fundamental operational primitives: memory allocation, payload writing, and execution triggering. By detecting the correlated co-occurrence of these operations rather than technique-specific signatures, the approach generalizes across variants and raises the evasion bar. An empirical study of malware collected in the wild characterizes the prevalence and evolution of injection techniques and reveals systematic blind spots in existing analysis frameworks. Second, we address execution-induced behavioral variability, a defining characteristic of modern malware arising from environmental adaptation, anti-analysis mechanisms, and conditional logic. Single-execution analysis therefore yields an incomplete and potentially misleading view of program intent. We introduce a multi-trace analysis framework that consolidates behavioral evidence across executions using an augmented API call graph abstraction and structure-aware exploration. We examine how aggregation, representation, and exploration interact under statistical constraints. While multi-trace aggregation improves detection sensitivity, it introduces a reliability–stability trade-off in which repeated evaluation increases exposure to false positives. Structural abstraction provides inductive bias toward connectivity and dependency patterns, while guided exploration mitigates redundant execution and limits error accumulation. Taken together, the results demonstrate that robust malware detection cannot rely solely on incremental accuracy improvements. Instead, it requires principled abstraction mechanisms that connect low-level behavioral evidence to higher-level structural reasoning under statistical control. By combining behavior-centric detection of stealth techniques with structured multi-trace modeling, this thesis contributes to more resilient, interpretable, and statistically robust dynamic malware analysis.