MTDecipher: robust encrypted malicious traffic detection via multi-task graph neural networks

Abstract The widespread adoption of encrypted traffic protocols has significantly increased the challenge of detecting malicious traffic. Existing detection methods based on deep learning typically rely on fine-grained features of data packets, such as length sequences and intra-flow interaction graphs. However, these features are highly susceptible to disruption by diverse network environments and traffic obfuscation. This paper proposes MTDecipher, a robust method for detecting encrypted malicious traffic based on multi-task Graph Neural Network (GNN). MTDecipher employs a bidirectional attentive sequence encoder to mitigate the impact of diverse network environments and traffic obfuscation on packet length sequences, along with an edge-block dual sampling method and a multi-task GNN model to mitigate the training bias introduced by the unbalanced distribution of traffic. In the bidirectional attentive sequence encoder, a combination of a Bi-GRU layer and an attention pooling layer is utilized to enhance the bidirectional encoding by generating weights for each element in the sequence, thereby obtaining robust encrypted traffic sequence features. In the edge-blo