Advanced Remediation Guidance for Elastic Search in Chef Automate 2.0 and Backend

Brian Loomis | Posted on February 28, 2022 | Chef Automate | Product Announcements

Guidance for users who completed the Chef Backend 2.2.0 migration and are still being alerted by scanning tools that Elastic Search may have a vulnerability.

We’ve recently been notified by customers who took the actions outlined in the blog “Is Chef vulnerable to CVE-2021-44228 (Log4j)?” to remediate the Log4J vulnerabilities reported in December, that their security scanning tools are picking up that Elastic Search may have vulnerabilities associated with it. What the scanning tools may be identifying is that – even though the JNDI is disallowed in security file – the Java class file is still present and that there is still a risk that any system could override the configuration and call into those methods.

This blog post shows how to identify and remove the class file and remediate Log4J the way that Elastic Search has patched their product since the vulnerability discovery.

Chef’s use of Elastic Search does not use any of these macros in our logging, nor do we make any lookups to LDAP: in effect, we do not require this Log4J file and removing it does not impact any Chef functionality.

The additional steps below are being supplemental to the initial guidance:

The basic fix is to disable LDAP lookups by adding to: /etc/elasticsearch/jvm.options
File the following flag:-Dlog4j2.formatMsgNoLookups=true as described in https://gist.github.com/teknofire/a62ce04fa9201f72e691db5e2fee3a05 ;
In Chef Backend’s Elastic Search this configuration should go in /etc/chef-backend/chef-backend.rb and you will have to do chef-backend-ctl reconfigure to reapply the configuration.

The steps in this article are not required, however they may give additional assurance in cases where a customer cannot upgrade their Elastic Search instance or cluster immediately, and wants to apply the same steps ES performed to remediate and patch their versions for this one specific vulnerability. Note as well, that many enterprise firewalls already block this exploit, even in a completely unpatched state.

As always, you should practice the steps in your own QA environment before modifying a production system. The paths below are representative of the default installation: your system may have a different directory for Elastic Search. If you have multiple ES nodes, you should apply any changes to all nodes.

Method 1 – use the Google toolset to find the JAR file and remove the JndiLookup.class file automatically

One of the Googlers wrote a tool that searches for compromised class files and can delete the JndiLookup.class file inside the JAR. The following set of steps were taken on a Chef installation:

Download Go onto the Elastic Search server, if it is not already there.
Download the log4jscanner from https://github.com/google/log4jscanner

go install github.com/google/log4jscanner@latest
Locate the directory where Elastic Search JAR files are located and rewrite log4j-core-*.jar Typically, this is in /usr/share/elasticsearch.

log4jscanner --rewrite /myfolder-with-jar-files
zipinfo /myfolder-with-jar-files/log4j-core-2.12.1.jar
| grep Jndi

This should show no remaining JndiLookup.class files in the JAR. Since JAR files are effectively ZIP files, zipinfo comes in handy here.

Method 2 – manually find the JAR file and remove the JndiLookup.class file

If you do step #3 above and locate the Elastic Search JAR files, including the log4j core ones, you can use ZIP to delete a specific file out of the JAR:

zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

Rerun zipinfo, and again you should see no JndiLookup.class file:

zipinfo log4j-core-2.12.1.jar | grep Jndi

Method 3 – try out a cookbook and InSpec profile to identify and remediate Elastic Search

Our own Richard Nixon developed a Chef-native way to do all this without loading extra tools – both an InSpec profile and a cookbook to remediate any places where this Log4J class is found. In the InSpec profile shown below, we simply check for JAR files and inspect the classes to see if we find JndiLookup.class:

This style of search has broader applicability to finding errant Java class files in other applications and uses the power of the Chef domain-specific language embedded in the cookbooks.

Mr. Nixon also accompanies this with a Chef patch cookbook, able to remediate and remove the offending file. Simply pull up Chef Workstation, download the cookbook (or copy it over if you do not have a direct connection to the Internet or Chef Supermarket), and upload it for the Chef clients on each of those Elastic Search machines to pick up.

cd <your cookbooks directory> git clone https://github.com/trickyearlobe/patch-log4j knife cookbook upload patch-log4j

The cookbook is straightforward and shown below:

Next updates

We have a couple of updates on deck for Chef Automate and other Chef offerings which will bring our PostgreSQL up to version 13.4 and provide both an upgrade for Elastic Search and an option to convert over to Open Search. These updates will feature upgrades to Elastic Search versions 6.8.23 and 7.16.1 as well as to Open Search version 1.2, all of which contain the latest Log4j 2.17.1 and are fully remediated.

Finally, please reach out to your customer service team for complex situations or for any help on these steps.

References:

Elastic Search Post- https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476
Elastic Search new versions - https://www.elastic.co/blog/new-elasticsearch-and-logstash-releases-upgrade-apache-log4j2
Log4J vulnerability page - https://logging.apache.org/log4j/2.x/security.html
CVE-2021-44228 - https://nvd.nist.gov/vuln/detail/CVE-2021-44228
CVE-2021-45046 - https://nvd.nist.gov/vuln/detail/CVE-2021-45046
CVE-2021-44832 - https://nvd.nist.gov/vuln/detail/CVE-2021-44832
CVE-2021-45105 (denial-of-service) - https://nvd.nist.gov/vuln/detail/CVE-2021-45105
Associated Elastic Search update for 45105 https://www.elastic.co/guide/en/elasticsearch/reference/6.8/release-notes-6.8.23.html
A developer-focused post on ES specifically - https://xeraa.net/blog/2021_mitigate-log4j2-log4shell-elasticsearch/
SEI note on same - https://kb.cert.org/vuls/id/930724
Microsoft response - https://msrc-blog.microsoft.com/2021/12/11/microsofts-response-to-cve-2021-44228-apache-log4j2/