Evaluation Process

NIST will form an internal selection panel composed of NIST employees for the technical evaluations of the submitted algorithms. This panel will analyze the submitted algorithms and review public comments that are received in response to the posting of the “complete and proper” submissions. The panel will also take into account all presentations, discussions and technical papers presented at the PQC standardization conferences, as well as other pertinent papers and presentations made at other cryptographic research conferences and workshops. NIST will issue a report after each PQC standardization conference. Final selections of cryptosystems will be made by NIST and the technical rationale for these decisions will be documented in a final report. The following is an overview of the envisioned submission review process.

5.A      Overview

Following the close of the call for submission packages, NIST will review the received packages to determine which are “complete and proper,” as described in Section 2 and Section 3 of this notice. NIST will post all “complete and proper” submissions at http://www.nist.gov/pqcrypto for public review. To help inform the public, a PQC standardization conference will be held at the start of the public comment process to allow submitters to publicly explain and answer questions regarding their submissions.

The initial phase of evaluation will consist of approximately twelve to eighteen months of public review of the submitted algorithms. During this initial review period, NIST intends to evaluate the submitted algorithms as outlined in Section 5.B.  NIST will review the public evaluations of the submitted algorithms’ cryptographic strengths and weaknesses, and will use these to narrow the candidate pool for more careful study and analysis.  The purpose of this selection process is to identify candidates that are suitable for standardization in the near future.  Algorithms that are not included in the narrowed pool may still be considered for standardization at a later date, unless they are explicitly removed from consideration by NIST. 

Because of limited resources, and also to avoid moving evaluation targets (i.e., modifying the submitted algorithms undergoing public review), NIST will NOT accept modifications to the submitted algorithms during this initial phase of evaluation.

For informational and planning purposes, near the end of the initial public evaluation process, NIST intends to hold another PQC standardization conference. Its purpose will be to publicly discuss the submitted algorithms, and to provide NIST with information for narrowing the field of algorithms for continued evaluation.

NIST plans to narrow the field of algorithms for further study, based upon its own analysis, public comments, and all other available information. It is envisioned that this narrowing will be done primarily on security, efficiency, and intellectual property considerations. NIST will issue a report describing its findings. Submitters of sufficiently similar algorithms may be asked to merge submissions for the next phase.

Before the start of a second evaluation period, the submitters of the algorithms will have the option of providing updated optimized implementations for use during the next phase of the evaluation. During the course of the initial evaluations, it is conceivable that some small deficiencies may be identified in even some of the most promising submissions. Therefore, for the second round of evaluations, small modifications to the submitted algorithms will be permitted for either security or efficiency purposes. Submitters may submit minor changes (no substantial redesigns), along with a supporting justification that must be received by NIST prior to the beginning of the second evaluation period. (Submitters will be notified by NIST of the exact deadline.) NIST will determine whether the proposed modification would significantly affect the design of the algorithm, requiring a major re-evaluation; if such is the case, the modification will not be accepted. If modifications are submitted, new reference and optimized implementations and written descriptions must also be provided by the announced deadline. This will allow a thorough public review of the modified algorithms during the entire course of the second evaluation phase.

Note: All proposed changes must be conveyed by the submitter; no proposed changes (to the algorithm or implementations) will be accepted from a third party.

The second round of evaluation will consist of approximately twelve to eighteen months of public review, with a focus on a narrowed pool of candidate algorithms. During the public review, NIST will similarly evaluate these algorithms as outlined in the next section. After the end of the public review period, NIST intends to hold another PQC standardization conference. (The exact date is to be scheduled.)

Following the third PQC standardization conference, NIST will prepare a summary report, which may select algorithm(s) for possible standardization, and/or may determine that a third phase of evaluation is needed. This third evaluation process would be structured similarly to the previous two evaluation periods.  Any selected algorithm(s) for standardization will be incorporated into draft standards, which will be made available for public comment.

When evaluating algorithms, NIST will make every effort to obtain public input and will encourage the review of the submitted algorithms by outside organizations. NIST encourages the reviewers to demonstrate their findings and attacks both on the versions with parameters that achieve full security levels, as well as with practical attacks on the provided parameter sets with lower security levels.  The final decision as to which (if any) algorithm(s) will be selected for standardization is the responsibility of NIST.

It should be noted that this schedule for the evaluation process is somewhat tentative, depending upon the type, quantity, and quality of the submissions. Specific conference dates and public comment periods will be announced at appropriate times in the future.  NIST estimates that some algorithms could be selected for standardization after three to five years. However, due to developments in the field, this could change. 

5.B      Technical Evaluation

NIST will invite public comments on all “complete and proper” submissions. The analysis done by NIST during the initial phase of evaluation is intended, at a minimum, to include:

i. Correctness check: The KAT values included with the submission will be used to test the correctness of the reference and optimized implementations, once they are compiled. (It is more likely that NIST will perform this check of the reference code—and possibly the optimized code as well—even before accepting the submission package as “complete and proper.”)

ii. Efficiency testing: Using the submitted optimized implementations, NIST intends to perform various computational efficiency tests.  This could include, for example, the time required for key generation, encryption, decryption, digital signing, signature verification, or key establishment, as well as the size of keys, ciphertext, and signatures.

iii. Other testing: Other features of the submitted algorithms may be examined by NIST.

Platform and Compilers

The above tests will initially be performed by NIST on the NIST PQC Reference Platform, an Intel x64 running Windows or Linux and supporting the GCC compiler.

At a minimum, NIST intends to perform an efficiency analysis on the reference platform; however, NIST invites the public to conduct similar tests and compare results on additional platforms (e.g., 8-bit processors, digital signal processors, dedicated CMOS, etc.). NIST may also perform efficiency testing using additional platforms.

NIST welcomes comments regarding the efficiency of the submitted algorithms when implemented in hardware. During the second evaluation period, NIST may request specifications of some of the algorithms using a hardware description language, to compare the estimated hardware efficiency of the submitted algorithms.

Note: If the submitter chooses to submit updated optimized implementations prior to the beginning of the second round of evaluation, then some of the tests performed may be performed again using the new optimized implementations. This will be done to obtain updated measurements.

Note: Any changes to the NIST PQC Reference Platform will be noted on http://www.nist.gov/pqcrypto.

5.C      Initial Planning for the First PQC Standardization Conference

An open public conference will be held shortly after the end of the submission period, at which the submitters of each “complete and proper” submission package will be invited to publicly discuss and explain their submitted algorithm. The documentation for these algorithms will be made available at the conference. Details of the conference will be posted at http://www.nist.gov/pqcrypto.