~/projects/netsec
All projectsshippedSecurity · Big Data2024
NetSec Analyzer
DNS-tunneling detection over large-scale packet data using Apache Spark, with an automated analytical report.
problem
The problem
DNS tunneling smuggles data out of networks inside ordinary-looking DNS queries, invisible to eyeballing, and the traffic volume is far too large to inspect by hand. Detection has to be statistical and it has to scale.
approach
How I approached it
- Built a scalable ML pipeline on Apache Spark to sift anomaly and tunneling patterns from large packet datasets.
- Engineered features from DNS query characteristics, entropy, length distributions, request cadence, that separate tunneling from benign lookups.
- Generated a custom analytical DNS report and synthetic data to exercise the detector.
architecture
Architecture
1Ingest
Packet capture → parsed DNS records at scale
2Feature extraction
Entropy, length, cadence, subdomain stats
3Spark pipeline
Distributed scoring across the dataset
4Report
Automated analytical report + visualizations
decisions
Engineering decisions & tradeoffs
$
Spark over pandas
// Packet volumes blow past single-machine memory; distributed processing was the point, not an optimization.
$
Feature engineering over deep learning
// Interpretable features make a security report defensible, an analyst can see why a domain was flagged.
stack
PythonApache SparkJupyterPandasMatplotlib
what's next
- › Stream mode for near-real-time detection instead of batch.
- › Labeled ground-truth eval set to report precision/recall properly.