Log4Shell and Your Containers

On 9 and 10 December 2021 a critical remote-code-execution vulnerability in Apache Log4j 2 was disclosed and named Log4Shell (CVE-2021-44228). Log4j is a Java logging library embedded in an enormous amount of software, often inside “fat” JAR files, so it hides in container images that never list it as a dependency. This post is an operational checklist as of 20 December; the situation is still developing, so re-check the Apache Log4j security page.

What the bug is, in one paragraph

Affected Log4j versions (log4j-core 2.0-beta9 through 2.14.1) evaluate special ${...} lookups in logged text. If an attacker can get a string such as a ${jndi:...} lookup into anything the application logs (an HTTP header, a username, a search field), Log4j may fetch and execute code from an attacker-controlled server. The exposure is any Java application that logs attacker-controlled input with a vulnerable Log4j, including third-party products. Log4j 1.x is a different library without this specific flaw, but it is end of life and has its own issues.

Fix timeline (what “patched” means)

Date	Release	Notes
10 Dec	2.15.0	First fix; found to be incomplete in some configurations
13 Dec	2.16.0	Disables JNDI by default and removes message lookups (CVE-2021-45046 addressed)
17 to 18 Dec	2.17.0	Fixes a denial-of-service through uncontrolled recursion (CVE-2021-45105)
17 Dec	2.12.2 (Java 7) and 2.3.1 (Java 6)	Backports for old Java runtimes

Target 2.17.0 or later (Java 8 and newer). Earlier “fixes” and the flag -Dlog4j2.formatMsgNoLookups=true are not sufficient on their own for all versions and configurations; use them only as a stop-gap.

Response flow

flowchart TB
  A[Inventory all images and deployed workloads] --> B[Scan images and JARs]
  B --> C{log4j-core below 2.17.0 present?}
  C -->|no| OK[Record as not affected]
  C -->|yes| D[Reduce exposure now]
  D --> D1[WAF rules and egress deny]
  D --> D2[Remove JndiLookup class or set mitigations]
  D --> E[Upgrade dependency and rebuild image]
  E --> F[Redeploy and rescan]
  F --> G[Hunt in logs for exploit attempts]
  G --> H[Rotate secrets if compromise suspected]

1. Inventory what runs

# every image in the cluster
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{range .spec.containers[*]}{.image}{"\n"}{end}{end}' | sort -u > images.txt
wc -l images.txt

Also list images in registries, CI caches, VMs and vendor appliances. Ask vendors about their products (Elasticsearch, Kafka, Logstash, Solr, Jenkins plugins, Spring applications and many others bundle Log4j) and check their advisories.

2. Scan

# update the vulnerability database first; scanners shipped fixes within days
trivy image --severity CRITICAL,HIGH --vuln-type library registry.example.com/shop/api:1.7.3 | grep -i -E 'log4j|CVE-2021-44228|CVE-2021-45046|CVE-2021-45105'
while read img; do trivy image -q --vuln-type library "$img" | grep -q 'CVE-2021-44228' && echo "AFFECTED $img"; done < <(awk '{print $2}' images.txt)

Scanners can miss log4j hidden in shaded or nested JARs, so also search inside the filesystem:

docker run --rm --entrypoint sh registry.example.com/shop/api:1.7.3 -c "find / -name '*.jar' 2>/dev/null | xargs -I{} sh -c 'unzip -l {} 2>/dev/null | grep -q org/apache/logging/log4j/core/lookup/JndiLookup.class && echo {}'"

That path in any JAR means log4j-core classes are present. Check its version in META-INF/maven/org.apache.logging.log4j/log4j-core/pom.properties.

3. Reduce exposure immediately

• Upgrade log4j-core to 2.17.0 or later (Maven/Gradle: pin the version, check the dependency tree with mvn dependency:tree | grep log4j). Rebuild images with docker build --pull --no-cache.
• If you cannot upgrade yet: remove the vulnerable class from the JAR (zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class), and set LOG4J_FORMAT_MSG_NO_LOOKUPS=true as an interim measure on versions 2.10 and later.
• Block outbound access that the exploit needs. A default-deny egress NetworkPolicy (see the network policies post) or security group rules stop pods from reaching arbitrary LDAP (389, 636), RMI (1099) and HTTP servers on the internet.
• Add WAF rules that match ${jndi: patterns in headers and bodies (AWS WAF managed rule group Known bad inputs was updated for this). It reduces noise but attackers obfuscate payloads, so it does not replace patching.

4. Hunt for signs of exploitation

grep -r --include='*.log' -E '\$\{(jndi|lower|upper|env|::-)' /var/log 2>/dev/null | head
kubectl logs -n shop deploy/api --since=240h | grep -i -E 'jndi|ldap://|rmi://' | head

Look in load balancer and proxy logs for ${jndi: and obfuscated variants, in egress/flow logs for connections to unusual LDAP or RMI destinations, and for new processes or files on affected hosts. If you find evidence of successful exploitation, treat the host or pod as compromised: isolate, preserve evidence, rotate credentials the application could read, and rebuild from clean images.

5. Prevent a repeat

1. Maintain an SBOM (software bill of materials) per image so you can answer “where is library X?” in minutes (tools such as Syft generate one).
2. Scan in CI and rescan deployed images on a schedule.
3. Run containers as non-root with read-only filesystems and minimal egress; it limits what a code-execution bug can do.
4. Keep a named owner and a 24-hour patch process for critical vulnerabilities, and a list of your Java applications.

Expect further updates: additional Log4j releases and vendor patches will follow. Check the Apache advisory and your vendors again before closing the incident.