Alert Logic Container IDS Solution Detects Cryptomining Attack on AWS

Executive Summary

On the April 4, 2018, an incident was raised by an analyst in our SOC (security operations center) for one of our newly-provisioned customers, indicating they were hosting a cryptomining threat that targets the Monero cryptocurrency. The attack was identified via the threat detection capabilities of the beta version of our Alert Logic container intrusion detection system solution for containerized environments, which had detected strings indicative of Monero cryptocurrency passing across the network. A container within the customer environment had been breached before we provisioned the container solution, and was hosting this cryptominer. 

Our investigation determined that the exploit was achieved via a wget command, executed by an attacker in China (IP: 116.211.143.90), via an open AWS security Group (0.0.0.0), which had been applied to the Kubernetes nodes in question. Kubernetes nodes exposed to the open internet in this way are known to have an unauthenticated RCE (remote code execution) capability, which was disclosed by external security researchers on March 13, 2018, in a blog post on Medium

The AWS security group had been applied to the Kubernetes nodes as of February 22, 2018. Upon subsequent inspection of AWS CloudTrail logs, IDS events and kubelet (internal Kubernetes logs) supplied by the customer to Alert Logic, it was identified that at least 4 different nodes had been breached in the same way since the security group was applied:

  • Node 1 – April 3
  • Node 2 – March 8
  • Node 3 – March 8
  • Node 4 – April 4

Only the Node 4 and Node 1 security breaches in this chain were caught, as these were the only nodes breached after our container IDS solution was provisioned. Nodes 2 and 3 were identified as having been breached previously using the historical logs supplied by the customer. 

Using logs available, since February 22, for each node we see repeated attempts to execute wget and curl commands, but these fail in all cases since the containers being targeted do not contain these commands. The nodes were breached in each case by exploiting an intermittently-deployed container, which had wget installed. Each time, the node was breached by the same Chinese IP (which posts back to a .ru address).

At present we do not believe there to be any reason to expect that the cyber attacker was able to gain more permanent or persistent access to the AWS account and the remediation of closing the security group and deleting all Kubernetes nodes (as of late April 4) is likely to have been sufficient to remove the detected threat.

Timeline of the Cyber Attack

Date

Nodes Affected

Activity

Dec-20-2017

Nodes 2, 3 and 4

Created

Feb-13-2018

n/a

Monero miner incident detection released by Alert Logic to customers

Feb-22-2018

Nodes 2, 3 and 4

Open SG applied to nodes in cluster

Mar-6-2018

Node 3

First see scanning for open nodes; activity expected from medium blog

Mar-7-2018

Node 2

First see scanning for open nodes; activity expected from medium blog

Mar-8-2018 (16:12:56)

Node 3

Vulnerable pod started

Mar-8-2018 (16:12:57)

Node 2

Vulnerable pod started

Mar-8-2018 (16:40:52)

Node 2

Breached with miner; activity expected from medium blog

Mar-8-2018 (16:40:55)

Node 3

Breached with miner; activity expected from medium blog

Mar-13-2018

n/a

Medium blog released; in response to observed kubernetes breach

Mar-29-2018

Node 1

Created (inherits open SG)

Apr-3-2018 (15:42:07)

Node 1

Vulnerable pod started

Apr-3-2018 (15:42:08)

Node 4

Vulnerable pod started

Apr-3-2018 (19:37:10)

Node 1

Breached with miner; activity expected from medium blog

Apr-3-2018 (19:37:16)

Node 1???

Incident raised to SOC. Linked to Node 1 due to timestamp in event

Apr-4-2018 (08:24:31)

Node 4

Breached with miner; activity expected from medium blog

Apr-4-2018 (08:24:39)

Node 4???

Incident raised to SOC. Linked to Node 4 due to timestamp in event

Evidence of the Security Breach

Recon

Upon opening up the security group for the affected nodes, we were able to establish routine scanning activity in the kubelet logs as of March 6.

  • Mar  6 07:44:53 ip-172-20-59-158 kubelet[1223]: I0306 07:44:53.317031    1223 server.go:779] GET /runningpods: (21.334µs) 301 >