Chef Automate previously had a restriction on the size of Compliance reports it could ingest. Any report beyond 4MB in size was automatically rejected. However, with elaborate profiles and controls, Compliance reports often extend beyond the 4MB size limitation. Hence, we enhanced Chef Automate in our latest release to support Compliance reports of bigger sizes.
The new Chef Automate version 4.2.22 can now support the ingestion, display, and export of large Compliance reports.
Challenges
The challenge for Chef Automate was not only in data ingestion but also in displaying and exporting this data. Since not all users might have large Compliance reports to be fed to Chef Automate, it was necessary to find a way that supports existing data sets as well as large Compliance data sets.
Architecture Change
The latest enhancement in Chef Automate that enables the ingestion and management of large Compliance reports can be explained in the following diagram.
Firstly, Chef Automate would require a Minio Object Storage Bucket or AWS S3 bucket that is available out of the Automate ecosystem and is reachable from the machine running Chef Automate.
Next, the data flowing into Chef Automate will be pushed to this Minio Bucket/AWS S3 without any modifications while Chef Automate stores a copy in OpenSearch with some modifications. These modifications are needed for keyword search and display in the User Interface.
Enabling Ingestion of Large Compliance Reports
Since not every user might work with large Compliance reports, this enhancement is not available by default in Chef Automate. In order to enable ingestion of large data sets, the following changes need to patch into a running Chef Automate instance:
[global.v1.external.minio]
endpoint = "<minio server end point>:<port>"
root_user = "<username>"
root_password = "<password>"
[global.v1.large_reporting]
enable_large_reporting = true
Please refer to the following document for detailed instructions.
Impact of the Change
This change will allow ingestion of large Compliance reports but will have some impact on the ingestion performance. This is due to the introduction of a new component in data processing which needs to be communicated over a network. We recommend considering this benchmark test summary report for use.
Exporting Compliance reports from the Compliance tab in Chef Automate will now be a two-step process:
- Generation of Report
- Download of Report
Detailed steps to download report results (with large compliance report configuration).