Automated Compliance Assessment
Introduction
This blog article is a review of using applied AI tools to automate cybersecurity compliance assessments. It is the fourth article in a series on applied AI to cybersecurity compliance tasks. The first blog article located here provides an overview of the series. Please refer to that article for important contextual information about this blog series.
Cybersecurity compliance assessments are used to determine if an organization complies to a set of cybersecurity standards (the US Department of War (DoW) Cybersecurity Maturity Model Certification (CMMC) standards as used in these scenarios). Compliance is compulsory and enforced by government agencies. Failure to comply can result in penalties such as fines and in the case of the CMMC standard, prevents awards of large government contracts.
Disclaimer: All materials created in this blog series were developed solely by, are the opinions of the author and are provided with no express or implied warranty. Other than outputs generated by the AI models used in this exercise, no AI was used to prepare this blog series (in case you were wondering). The author accepts no liability for the use of any of these materials by the reader. We strongly encourage engaging appropriate professionals for developing materials for use within your own organization.
A cybersecurity compliance assessment is basically an audit. They can be internal, typically to assess preparedness, or external by an outside entity (a third-party assessment organization (3PAO), or a regulatory agency). In any case, a complete assessment to a given standard is a lengthy, labor intensive and expensive process. Key resources inside the organization (subject matter experts) are called upon to provide evidence artifacts and management is heavily involved to manage the process.
The idea of automating compliance assessments is relatively new driven by the emergence of AI tools generally, and experimental in this case. In this blog, we use ChatGPT version 5 as the AI model to attempt to automate a compliance assessment. Full automated compliance assessment by AI is not currently feasible mainly due to the wide variability of the formats of evidence and the environments to be assessed, however results from this initial use shows great promise.
Approach
The approach taken with this exercise is to use a set of control objectives from a cybersecurity standard and compare them to evidence that is intended to show compliance to the requirements. The standard used here is the US Department of War (DoW) Cybersecurity Maturity Model Certification (CMMC) standards, version 2, Level 2. These standards are comprised of 14 Security Requirements Families consisting of 110 individual cybersecurity Controls (called requirements in the CMMC standards).
A fictitious company, Alpha Nine Fabrication (ANF), Inc. (as described in a company description document, link here) is used as an example company seeking assessment (Organization Seeking Assessment (OSA) in CMMC speak). Assessment evidence consisting of a cybersecurity System Security Plan (SSP), a policy document, procedure and standards documents from ANF are used as input to the assessment.
In the CMMC Assessment Guide (Level 2 – Version 2.13, September), requirement families consist of a set of controls that are further broken down into objectives. In total there are 375 individual objectives across all of the controls.
Use of control objectives as the basis for conducting an assessment is consistent with the CMMC Assessment Guide (ref. CMMC Assessment Guide Level 2, US Department of War
https://dodcio.defense.gov/Portals/0/Documents/CMMC/AssessmentGuideL2v2.pdf).
Per the Methodology section of the CMMC guide:
“To verify and validate that an OSA (Organization Seeking Assessment) is meeting CMMC requirements, evidence needs to exist demonstrating that the OSA has fulfilled the objectives of the Level 2 requirements.” (ref., page 8, CMMC Assessment Guide – Level 2, Version 2.123)
The guide goes on to point out that objectives can be met in different ways and that a variety of techniques may be used to determine if the OSA meets the Level 2 requirements. The guide references the assessment methods from NIST SP800-171A (the NIST assessment guide) which include interviews, observations and tests.
The above suggests that an assessor collects evidence of compliance in potentially multiple forms and then somehow compares that evidence to the objective statements to determine if they are met. From a human perspective, this seems intuitive. However, when using a Machine Learning tool this present challenges. For example, what if there is partial evidence in one part of the evidence and the remaining part to meet the objective is elsewhere (e.g., a policy exists in a policy document for access control, but the setting of a lockout parameter exists in a configuration file)? Or, what if the evidence is in an image file (e.g., a network map) that is displayed visually, but not readily available in machine readable text?
The question becomes how do we go about using an AI tool such as ChatGPT to do an assessment? As a start point, the approach taken here is a simple, straight forward one.
We read as input the evidence presented (limited to PDF documents and text files for now), we instruct the AI model to compare the objective statement to the contents of the file and attempt to match the contents to the objective statement. Even though this seems as a more basic approach, we’re making multiple assumptions here. Three (at least) critical assumptions are:
- The AI model has the ability to understand what file format (e.g., .pdf, .txt, etc.) it is using as input and can read, parse it and use it as input. (In this case, it does to a certain extent – mostly text-based inputs. Image files and other non-text formats will require special processing).
- The AI model has the ability to “read” not only individual words in an input file, but understand phrases (perhaps full sentences and even paragraphs) to derive artifacts that correspond to an objective statement. (to a limited extent (mostly words and phrases, it can)).
- The AI model can reason over multiple inputs (documents in this case) to determine if there are related artifacts across them that potentially refer to a given objective. This requires a more advanced approach such as Retrieval Augmented Generation, or AI workflow – a topic for a future blog article) and not done in these examples.
We are getting pretty deep here on purpose because it’s worth noting the potential challenges that a cybersecurity analyst faces when attempting to use AI tools to create a solution. Nevertheless, interesting results were achieved from this exercise.
ChatGPT Dialog Prompt
I started with a simple dialog prompt to ChatGPT. Prior to this prompt, I had developed the SSP and test case generation prompts and ChatGPT had these as prior context for this prompt.
This included instructing ChatGPT that it was to function as a cybersecurity analyst, that the requirements involved with this prompt are the NIST SP800-171 requirements, and the background documents related to the fictitious company, Alpha Nine Fabrication, Inc. were provided.
Initial Prompt
The initial prompt for the compliance assessment task was:
Please generate a python script that reads a csv file of control objectives, iterates through them and determines if the objectives are met in the given set of compliance evidence documents.
ChatGPT responded with:
Here’s a python program you can run to get the desired result. This program:
- Loads your objectives CSV (e.g., /mydir/objectives.csv)
- Scans a directory of evidence docs (PDF, DOCX, TXT, MD, HTML) like /mydir/docs
- For each objective, finds the best-matching document snippet
- Rates status as Met / Partial / Not Found, and writes a CSV report with evidence
Adjust the two paths at the top (OBJECTIVES_CSV, DOCS_DIR) before running.
What this does:
- Uses fuzzy matching + keyword coverage to locate likely evidence passages.
- Gives an objective-level status and a snippet you can copy into audit notes.
- Produces a CSV you can hand to auditors or use for gap analysis.
Approach 1 – Fuzzy Matching
This first assessment approach uses a list of requirement control objectives and determines if they are met by the supplied evidence. The list of objectives in this example was contained in a comma separated variable file (.csv) constrained to the Access Control (AC) family of the NIST SP800-171 Ver. 2 cybersecurity requirements only.
The input files consist of a set of documents that include the Alpha Nine Fabrication SSP document, an access control policy, a small set of access control related procedures, a small set of standards documents and one evidence file (a file that contains a Windows 11 lockout control registry setting that was created by a script generated from the test case generation blog of this article series).
Approach 1 – Result
The following table is a subset of the full table resulting from running the generated python program by ChatGPT. The full table is 281 rows long. This subset is for the first two NIST SP800-171 Access Control requirements 3.1.1[a] and 3.1.1[b]).
assessmentfuzzytable1
| Requirement Number | Requirement Name | Objective Number | Determination Statement | Status | Score | Best Evidence Doc | Evidence Snippet |
| 3.1.1 | Authorized Access Control (CUI Data) | 3.1.1[a] | Determine if authorized users are identified | Partial | 72.9 | /Users /AIDev/Access Control Procedures.docx | identify [assignment: organization-defined controls asserted] ] , consistent with the trust relationships established with other organizations owning, operating, and/or maintaining external systems, allowing authorized individuals to: access the system from external systems; and process |
| 3.1.1 | Authorized Access Control (CUI Data) | 3.1.1[b] | Determine if processes acting on behalf of users are identified | Partial | 72.2 | /Users/AIDev/Access Control Procedures.docx | information flow control policies]. ac-5: separation of duties baseline(s): moderate high identify and document [assignment: organization-defined duties of individuals] ; and define system access authorizations to support separation of duties. ac-6: least privilege baseline(s): moderate high |
| 3.1.1 | Authorized Access Control (CUI Data) | 3.1.1[c] | Determine if devices (and other systems) authorized to connect to the system are identified | Partial | 60.5 | /Users /AIDev/2022-ac-access-control-standard-signedpdf-45564.pdf | is not possible to protect the confidentiality and integrity of information on ed-approved and authorized mobile devices and services. 2.16 ac-20 use of external systems (l, m, h) a. establish, maintain and monitor interconnection security agreements (isas), inter agency agreements (iaa) and other ag... |
| 3.1.1 | Authorized Access Control (CUI Data) | 3.1.1[d] | Determine if system access is limited to authorized users | Partial | 68.7 | /Users/AIDev/2022-ac-access-control-standard-signedpdf-45564.pdf | services. 2.17 ac-21 information sharing (m, h) enable authorized users to determine whether access authorizations assigned to a sharing partner match the information's access and use restrictions for information that may be restricted in some manner based on some formal or administrative |
| 3.1.1 | Authorized Access Control (CUI Data) | 3.1.1[e] | Determine if system access is limited to processes acting on behalf of authorized users | Partial | 69.9 | /Users /AIDev/Access Control Procedures.docx | information flow control policies]. ac-5: separation of duties baseline(s): moderate high identify and document [assignment: organization-defined duties of individuals] ; and define system access authorizations to support separation of duties. ac-6: least privilege baseline(s): moderate high |
| 3.1.1 | Authorized Access Control (CUI Data) | 3.1.1[f] | Determine if system access is limited to authorized devices (including other systems) | Partial | 61.5 | /Users /AIDev/2022-ac-access-control-standard-signedpdf-45564.pdf | 101/acsd-ofo-051, interagency agreements and ocio: 3-113/acsd-ocio-002, controlled unclassified information program; and establish, maintain and monitor other binding agreements with employees and contractors which include ed-defined controls to be implemented on personally owned systems, components... |
| 3.1.2 | Transaction & Function Flow | 3.1.2[a] | Determine if types of transactions and functions that authorized users are permitted to execute are defined | Partial | 50.3 | /Users /AIDev/Access Control Procedures.docx | identify [assignment: organization-defined controls asserted] ] , consistent with the trust relationships established with other organizations owning, operating, and/or maintaining external systems, allowing authorized individuals to: access the system from external systems; and process |
| 3.1.2 | Transaction & Function Flow | 3.1.2[b] | Determine if system access is limited to the defined types of transactions and functions for authorized users | Partial | 56 | /Users /AIDev/Access Control Procedures.docx | identify [assignment: organization-defined controls asserted] ] , consistent with the trust relationships established with other organizations owning, operating, and/or maintaining external systems, allowing authorized individuals to: access the system from external systems; and process |
Although interesting, as a tool for assessing compliance, this initial result is not all that useful. What’s interesting is that ChatGPT chose to use a fuzzy method as the algorithm that uses tokenized phrases to compare the objective (Determination Statement) to the contents of the document. The script also parses the objectives into tokenized phrases with overlaps in an attempt to create more meaningful matches. In essence, this approach is not unlike using a super powered grep string matching technique. In many cases, the matches do not directly reflect the meaning of the given objective. Fuzzy matching although powerful does not use an AI model directly for processing.
At best, this script could be used as a means to automatically identify which documents contain references to individual objectives with some insight to the level of interest (from the “Evidence Support” column). This provides a means of prescreening documentation and potentially providing a checklist for completeness of assessment. Further tuning of the fuzzy method algorithm can improve results, however incremental improvements are about all that can be expected with this approach.
Approach 2 – AI Model-based
Chat Dialog
Recognizing the limitations of a fuzzy-based method, I proceeded to instruct ChatGPT to use the ChatGPT model for evaluating documents against the set of control objectives with this prompt:
do the same as above, but instead of using fuzzy matching + keyword coverage, use a chatGPT v5 model to evaluate whether the objectives are met or not.
Chat Dialog Response
ChatGPT responded with:
Here’s a python script that performs the same compliance-checking workflow as before, but instead of fuzzy matching, it uses the ChatGPT (GPT-5) API to reason about whether each NIST SP 800-171A objective is met based on the text of your evidence documents.
What This Script Does
Step | Description |
Loads CSV | Reads all NIST objectives. |
Reads documents | Extracts text from .pdf, .docx, .txt, .md, .html. |
Calls GPT-5 | For each objective, submits the objective and document text to the model. |
Receives judgment | GPT-5 returns Met / Partially Met / Not Met, a short snippet, and confidence. |
Saves CSV report | Results are written to /mydir/objectives_gpt5_report.csv. |
Notes
- This will incur one API call per objective, so you may want to subset your CSV first if it’s large.
- You can reduce cost by:
- Limiting MAX_DOC_CHARS
- Combining related objectives before sending
- Using a smaller model like gpt-4o-mini for initial triage
- To make it auditable, store the raw GPT output or chat logs for each objective.
Would you like me to extend this notebook to analyze one objective per document separately (instead of combining all docs into one input) so you can see which specific file satisfied each control?
yes please
Perfect — here’s the enhanced python script that evaluates each objective against each individual document using ChatGPT v5 reasoning.
This way, your output report shows which specific document satisfied (or failed to satisfy) each objective, along with the model’s reasoning and confidence.
Output Files
- csv
assessmentfuzzyoutputfiles
| Requirement Number | Objective Number | Document | Status | Confidence | Reasoning |
| 3.1.1 | A.03.01.01.c.01 | Access Policy.docx | Met | 94 | Defines authorized user roles |
| 3.1.1 | A.03.01.01.f.02 | System Config.pdf | Partially Met | 71 | Mentions inactivity but no time limit |
- objectives+gpt5_summary.csv
Requirement Number | Requirement Name | Overall Status |
3.1.1 | Account Management | Fully Met |
3.1.2 | Access Enforcement | Partially Met |
3.1.5 | Least Privilege | Not Met |
⚙️ How the Roll-Up Works
- Fully Met → all objectives for the requirement are “Met”
- Partially Met → mixed “Met/Partial/Not Met”
- Not Met → none of the objectives show evidence
AI Model-based Results
In contrast to the initial fuzzy method result, the output from running the generated python script was very interesting and potentially highly usable. It is evident that not only did the ChatGPT model perform a matching of the objective statements to the provided input documents, it performed a reasoning as to why a given objective was met (or not). In some respects, this mimics a human-based approach.
Inputs
The NIST SP800-171Account Control (AC) requirement objectives only were used to in this example. Eight documents were processed as evidence consisting of:
- Alpha Nine Fabrication System Security Plan (SSP)
- Alpha Nine Fabrication Access Control Policy
- Alpha Nine Fabrication Access Control Procedures
- Alpha Nine Fabrication Account Provisioning Procedure
- Alpha Nine Fabrication DevSecOps CI/CD Pipeline Procedure
- Access Control standard
- Alpha Nine Fabrication Remote Access Standard
- ACL2-3.1.8 evidence file
All documents were in Microsoft Word format (.docx) except the ACL2-3.1.8 evidence file which was in ASCII text (.txt) format. A total of 169 spreadsheet rows were produced sorted by requirement control objective.
The following table shows the results for Objective number 3.1.1[a] – Determine of authorized users are identified. This is the first objective of the 3.1.1 Account Control requirement.
assessmentgpttable1
| Reqt. | Reqt. Name | Objective Number | Objective | Document | Status | Confidence | Reasoning |
| 3.1.1 | Authorized Access Control (CUI Data) | 3.1.1[a] | Determine if authorized users are identified | AC.L2-3.1.8 evidence file.txt | Not Met | 92 | Evidence shows lockout settings for unsuccessful login attempts (AC.L2-3.1.8) on Windows/Linux/macOS. It does not identify authorized users or provide user inventories/authorization records. Therefore it does not satisfy 3.1.1[a] (identifying authorized users)." } |
| 3.1.1 | Authorized Access Control (CUI Data) | 3.1.1[a] | Determine if authorized users are identified | Alpha Nine abrication SSP.docx | Not Met | 78 | The SSP references policies and a Role Definition Matrix but provides no concrete description or evidence that authorized users are identified |
| 3.1.1 | Authorized Access Control (CUI Data) | 3.1.1[a] | Determine if authorized users are identified | POL-AC-001_Access ControlPolicy.docx | Partially Met | 68 | Policy mandates unique identification and authorization of users |
| 3.1.1 | Authorized Access Control (CUI Data) | 3.1.1[a] | Determine if authorized users are identified | PROC-AC-P01_ Account Provisioning Procedure.docx | Met | 82 | Procedure defines how authorized users are identified before access: Dept Managers submit/authorize requests with role/system |
| 3.1.1 | Authorized Access Control (CUI Data) | 3.1.1[a] | Determine if authorized users are identified | SOP-DEVSECOPS- CICD-001_Secure_ CICD_ Procedure.docx | Partially Met | 62 | The SOP defines authorized roles (Developers, Administrators, etc) |
| 3.1.1 | Authorized Access Control (CUI Data) | 3.1.1[a] | Determine if authorized users are identified | STD-AC-002_RemoteAccess Standard.docx | Partially Met | 60 | Document implies identification via AD credentials |
A few observations about the produced results.
- The analysis by ChatGPT was documented-based (meaning that each document was analyzed for each objective). Although thorough, this method results in evaluations that were not relevant. For example, the first row in the above table evaluates the AC.L2-3.1.8 evidence file that contains evidence related to requirement 3.1.18 – Account Lockout. ChatGPT correctly determines that the objective was “Not Met” with a high confidence rating of 92), however the evidence file is not intended to meet the 3.1.1[a] objective. Had an evidence file been provided that contains a list of authorized users, presumably that file would be marked as meeting the objective. A modification to the assessment evaluation script to have ChatGPT identify relevant documents only (perhaps through document metadata) could reduce or eliminate this type of behavior.
- A more meaningful result can be seen on row 4 for the account provisioning procedure (PROC-AC-P01_AccountProvisioningPRocedure.docx). The Reasoning provided stated “Procedure defines how authorized users are identified before access: Dept Managers submit/authorize requests with role/system”. This row was correctly marked as “Met” with an 82 confidence factor. In the context of a procedure this analysis is correct meaning that a procedure to implement the presumed policy exists and fulfills that part of the requirement. However, to fully meet this objective requires that a corresponding policy statement exists and that there is evidence that a list of authorized users also exists.
- Line 2 of the table, states that the SSP does indicate that “no concrete description or evidence that authorized users are identified” which calls out the need for the missing authorized users list.
Taken in aggregate, the table above does a pretty good job of analyzing these objectives. It references that a policy exists, a procedure is provided, and is very usable for identifying gaps in the evidence (e.g., the missing authorized users list). It also provides the corresponding documents that can help speed up locating evidence in an assessment. Using this analysis as a preliminary screening could be a very good way to speed up the assessment analysis process.
The next set of rows addresses objective 3.1.1[b] – “Determine if processes acting on behalf of users are identified.
assessmentgptresults2
| Reqt. | Reqt. Name | Objective Number | Objective | Document | Status | Confidence | Reasoning |
| 3.1.1 | Authorized Access Control (CUI Data) | 3.1.1[b] | Determine if processes acting on behalf of users are identified | AC.L2-3.1.8 evidence file.txt | Not Met | 92 | The evidence only shows account lockout configurations (AC.L2-3.1.8). It does not demonstrate identification of processes acting on behalf of users (e.g. |
| 3.1.1 | Authorized Access Control (CUI Data) | 3.1.1[b] | Determine if processes acting on behalf of users are identified | Alpha Nine Fabrication SSP.docx | Not Met | 88 | The SSP lists roles |
| 3.1.1 | Authorized Access Control (CUI Data) | 3.1.1[b] | Determine if processes acting on behalf of users are identified | POL-AC-001_AccessControl Policy.docx | Not Met | 88 | The policy identifies users and devices but does not address processes acting on behalf of users (e.g. |
| 3.1.1 | Authorized Access Control (CUI Data) | 3.1.1[b] | Determine if processes acting on behalf of users are identified | PROC-AC-P01_AccountProvisioning Procedure.docx | Not Met | 85 | Procedure addresses user and system account provisioning and reviews but does not identify or require documentation of processes acting on behalf of users (e.g. |
| 3.1.1 | Authorized Access Control (CUI Data) | 3.1.1[b] | Determine if processes acting on behalf of users are identified | SOP-DEVSECOPS-CICD-001_Secure_CICD_ Procedure.docx | Partially Met | 62 | The SOP references automated CI/CD components (e.g. |
| 3.1.1 | Authorized Access Control (CUI Data) | 3.1.1[b] | Determine if processes acting on behalf of users are identified | STD-AC-002_RemoteAccess Standard.docx | Not Met | 80 | The standard covers remote access |
You can notice that like the prior example, there are non-relevant items listed. In this example, rows 1, 2, 5 and 6 refer to items that are not intended to address processes acting on the behalf of users.
As a compliance analyst, if I ignore these rows, the remaining rows (3 and 4) readily identify that none of the documentation addresses processes acting on behalf of users.
Therefore, I can conclude that I need to either supply the missing evidence, or address the issue of processes acting on behalf of users as part of the overall program. Using this tool aids in zeroing on what needs to be done in order to more fully meet compliance requirements.
Observations
The above exercise was artificial and purposefully constrained due to resource limitations and not an example of a real full assessment. In a real-world situation, a much larger, more complete body of evidence would probably be used for an assessment. The results do however provide an indication of the potential of applied AI and some of the areas where improvements can be made.
Importance of Approach
Two approaches were described in this exercise, a fuzzy-based approach and an AI model-based approach. The assumptions used with the two methods used are compared here:
🧩 2. Implicit Model Assumptions
For the Fuzzy Baseline
- Literal matching: Control association depends on explicit keyword overlap (e.g., the words “session timeout” or “account lockout” must literally appear).
- Equal weighting: Every keyword contributes equally to the score (unless you extend the weighting model).
- No semantic reasoning: It does not understand synonyms or paraphrasing — “automatic logout” ≠ “session lockout” unless the same term appears.
- File independence: No cross-file reasoning — each evidence file is assessed in isolation.
For the GPT-5 Assessment
- Semantic linking: GPT-5 infers association based on wording similarity, conceptual meaning, and context (e.g., “user lockout after inactivity” → matches AC-11 even if not labeled as “3.1.11”).
- Granularity: Each document is treated independently. If a file covers multiple objectives, GPT-5 can still assign multiple matches.
- Evidence relevance: GPT-5 assumes that if evidence text semantically aligns with the objective’s intent, it’s valid, even if no explicit control reference (e.g., “password rotation every 60 days” → satisfies IA-5(d)).
- Faith in extracted text: GPT-5 relies entirely on what the PDF/DOCX reader extracted — if OCR or formatting fails, it cannot infer unseen content.
As can be seen the GPT-5 assessment method is more powerful in that it uses semantic linking as opposed to literal matching as its basis. This is evident in the results. It is worth noting that the GPT-5 method took over an hour to process its results. The Fuzzy Baseline method completed in under one minute.
When comparing the two methods, it is evident that the GPT-5 reasoning based method is superior. The Fuzzy Baseline method may however still add value, perhaps as a quick pre-screening to identify what documents are related to which objectives with initial feedback. Tuning the Fuzzy Baseline method can also improve its results with more refined literal matching and better preprocessing of inputs.
This shows that the approach selected can have a large impact on the results. In this case, the assessment was to process all documents by each requirement objective without regard to the type of content of the document. This caused the program to assess documents that had the irrelevant items noted. In effect, these are like false positive indicators of compliance. An improved method could be to tag each document with metadata indicating the parts of the requirements addressed and instruct the AI model to use it as part of the evaluation process.
Another possible trade off of the approach is whether to assess documents independently (assess by document), or assess all documents by each requirement objective (assess by objective). In this example no instruction was given to ChatGPT as to which approach to use, so it simply assessed each document individually, by objective instead of assessing all documents by a single objective which is probably a more natural way for an assessor to go about an assessment.
Iterative Nature of Chat Dialog Development
Interacting with ChatGPT gives feedback on how the model reacts to dialog chats. Interim results can be used to adjust the approach and model used to affect the intended results. The observed behavior of including irrelevant document references provides feedback that improvements can be made through use of metadata tags in document input as in this example. This could allow refinement of results to only relevant items only and potentially improve performance by eliminating evaluations of irrelevant document artifacts. This last point is important.
In this constrained example that consisted of 8 requirements and 28 objectives it took over an hour to process with the ChatGPT model-based approach. A full set of NIST SP800-171 requirements consists of 110 requirements and 375 objectives. Over a much larger body of evidence, the evaluation processing time could increase dramatically. Iterating through the use of the AI model can improve results and is necessary. However, it is also wise to examine the overall architecture used.
For a more powerful system that can process larger amounts of input, we may need to employ alternative architectures such as Retrieval Augmented Generation (RAG) where inputs are preprocessed and multiple evaluation steps are employed to further reduce the processing time needed. This suggests that those that are attempting to get good results from applied AI not only need problem domain expertise (cybersecurity in this case), but also need skills in business process and knowledge engineering.
Conclusion
This exercise provides a sample of applying AI technology to the problem of automating compliance assessments. With additional work to tune the process through chat dialog refinements, better assignment of metadata to documents, alternate AI model use, and improvements in manipulating inputs, etc., we anticipate significant improvements can be readily achieved. We have just scratched the surface of what is possible with this example.
Full automation of compliance assessments is not currently achievable. This is due to the wide variety of environments and different types of inputs in real world environments (see note below “A few words about OSCAL”). As tools to aid in reducing the time and effort to conduct compliance assessments are developed, AI technologies can reduce the time and effort to conduct them. In addition, the quality of assessments can be increased because of the detailed assessments by a computer. The effort spent to develop AI tools is well worth the effort now and the potential going forward is great.
A pending crisis exists in cybersecurity compliance as the requirements for assessments is driven by regulatory agencies. Estimates vary, but by some more conservative estimates, over 80,000 US Defense Industrial Base (DIB) companies will have to conduct CMMC Level 2 cybersecurity assessments over the course of the next three years as the requirement for CMMC requirement assessments are written into DoD contracts. The total DIB base is estimated at over 300,000 companies. Given that the total hours to conduct a compliance assessment is 120 hours (includes initial internal assessment and final external reported assessments), for the initial base of 80,000 companies this represents 9.6 million hours. This does not include the time that the US DoW will have to spend analyzing results. Anything that has the potential to reduce the time and cost to conduct cybersecurity assessments is well worth it.
A few words about OSCAL:
The Open Security Controls Assessment Language (OSCAL) is a NIST lead initiative for a set of hierarchical, formatted, data formats that provide a standardized representation for different categories of security information pertaining to the publication, implementation, and assessment of security controls (ref. OSCAL: the Open Security Controls Assessment Language https://pages.nist.gov/OSCAL/). There are two relevant major parts of OSCAL:
The framework consisting of a controls catalog, system security plan, assessment plan, assessment results and
The standardization of input formats (most notably, file formats and meta data tagging). These offer the potential to solve some of the challenges we observed in this exercise.
The examples noted here where metadata tagging can be used to identify documents and sections related to a given objective to act as filter and an identifier to the AI model for assessment is one key example. OSCAL defines both metadata tagging and the standardized labeling of content data.
Use of standardized data formats can drastically reduce the processing time and potential errors due to the disarray of input data types and the interpretation of their meaning.
We applaud the NIST teams that developed OSCAL and the NIST cybersecurity standards for outstanding work in this area and their contributions to solving problems with cybersecurity and assessments. We believe that the combination of standards like OSCAL and the use of AI models have the potential to achieve full compliance automation in the not-too-distant future. Stay tuned for more blog articles on these and related compliance automation topics.