Search CSRC

ABOUT: Cyber risk has become a topic of core strategic concern for business and government leaders worldwide and is an essential component of an enterprise risk management strategy. The Software and Supply Chain Assurance Forum (SSCA) provides a venue for government, industry, and academic participants from around the world to share their knowledge and expertise regarding software and supply chain risks, effective practices and mitigation strategies, tools and technologies, and any gaps related to the people, processes, or technologies involved. The effort is co-led by the National Institute...

***Disclaimer: Items in the following lists are provided for research purposes, and do not imply endorsement by NIST.*** U.S. Government Activities / Initiatives Related Standards / Best Practices C-SCRM Research / References Involved Standards Organizations / Associations   U.S. Government Activities / Initiatives Committee on National Security Systems Directive (CNSSD) 505 - "...provides the guidance for organizations that own, operate, or maintain [National Security Systems (NSS)] to address supply chain risk and implement and sustain SCRM capabilities". Comprehensive National...

Comments Received in Response to: Federal Register Notice (June 28, 2013) Computer Security Incident Coordination (CSIC): Providing Timely Cyber Incident Response   Date (2013) Comment Received From Aug. 14 Carbon Black, (Michael Viscuso, CEO) Aug. 14 CERT Coordination Center at the Software Engineering Institute at Carnegie Mellon University (Ryan Meeuf, CERT Coordination Center, Carnegie Mellon Univ.) Aug. 14 C.I.G.N.E.T. (Vishwas Rudramurthy) Aug. 14 Internet Identity (IID) (Chris Richardson, Senior Manager, Federal...

Access control systems are among the most critical security components. Faulty policies, misconfigurations, or flaws in software implementation can result in serious vulnerabilities. The specification of access control policies is often a challenging problem. Often a system’s privacy and security are compromised due to the misconfiguration of access control policies instead of the failure of cryptographic primitives or protocols. This problem becomes increasingly severe as software systems become more and more complex and are deployed to manage a large amount of sensitive information and...

Access control (AC) policies can be implemented based on different AC models, which are fundamentally composed by semantically independent AC rules in expressions of privilege assignments described by attributes of subjects/attributes, actions, objects/attributes, and environment variables of the protected systems. Incorrect implementations of AC policies result in faults that not only leak but also disable access of information, and faults in AC policies are difficult to detect without support of verification or automatic fault detection mechanisms. Most research on AC model or policy...

Access control mechanisms control which users or processes have access to which resources in a system. Access control policies are increasingly specified to facilitate managing and maintaining access control. However, the correct specification of access control policies is a very challenging problem. This problem becomes increasingly severe as a system becomes more and more complex, and is deployed to manage a large amount of sensitive or private information and resources. To provide high security confidence levels for the nation’s critical IT infrastructure, it is important to provide a...

This ACPT version is a beta release, which includes a concise user manual, examples, and Java code. The user documentation and software will be updated in the future. Please check the web site for update information. To download the latest ACPT version (.zip file, May, 15, 2019), please contact: Vincent Hu vhu@nist.gov for the password to unzip the zip file.   The source code is also available. The Access Control Policy Tool (ACPT) was developed by NIST's Computer Security Division in cooperation with North Carolina State University and the University of Arkansas. ACPT is provided free of...

April 27, 2010: NIST SP 800-22rev1a (dated April 2010), A Statistical Test Suite for the Validation of Random Number Generators and Pseudo Random Number Generators for Cryptographic Applications, that describes the test suite.   Download the NIST Statistical Test Suite. July 9, 2014: This update has a few minor corrections to the source code. The first change corrects the non-overlapping template test to make it correctly skip bits when a sequence matches.  The second change is to correct the π values in the overlapping template test. Software Revision History August 11, 2010:...

This information is provided for historical purposes.  Papers Statistical Testing of Random Number Generators; Proceedings of the 22nd National Information Systems Security Conference, October 1999. Presentations Empirical Statistical Testing of RNGs, 1999 RSA Data Security Conference, San Jose, CA, 1/99. Statistical Testing of RNGs, ANSI X9F1 Meeting, Institute for Defense Analyses, Alexandria, VA, 4/99. Statistical Testing of Random Number Generators, The 22nd National Information Systems Security Conference, Crystal City, VA, 10/99.

Research tools to support combinatorial testing. No license is required and there are no restrictions on distribution or use. All software is provided free of charge and will remain free in the future. NIST is an agency of the US Government, so this software is public domain. You are free to include it and redistribute it in commercial products if desired.  To obtain the ACTS tool, please send a request to Rick Kuhn - kuhn@nist.gov  including your name and the name of your organization. No other information is required, but we like to have a list of organizations to show our management where...

A:  All or nearly all failures involve only 1 to 6 factors The key insight underlying combinatorial testing’s effectiveness resulted from a series of studies by NIST from 1999 to 2004. NIST research showed that most software bugs and failures are caused by one or two parameters, with progressively fewer by three or more. That is, they were only revealed when multiple conditions were true.  For example, a 2-way interaction fault could be "altitude = 0 AND volume < 2.2". So testing all 2-way combinations of parameter values could detect this problem. A method called "pairwise testing" has been...

Combinatorial testing is being applied successfully in nearly every industry, and is especially valuable for assurance of high-risk software with safety or security concerns.  Combinatorial testing is referred to as effectively exhaustive, or pseudo-exhaustive, because it can be as effective as fully exhaustive testing, while reducing test set size by 20X to more than 100X. Application   Reference Notes/Abstract Industrial controls, consumer appliances   M Park, H Jang, T Byun, Yunja, "Property-based Testing for LG Home Appliances...

Self-driving cars and autonomous systems of all types are notoriously difficult challenges for software assurance.  Both traditional testing and formal methods are even harder to apply for autonomous systems than in ordinary cases. The key problem is that these systems must be able to function correctly in a vast space of possible input conditions.  For example, autonomous vehicles must deal with lighting, rain, fog, pedestrians, animals, other vehicles, road markings, signs, etc.  Combinatorial methods are uniquely well suited to analysis and testing for this enormous input space, because by...

The tools distributed here are used extensively in testing for security vulnerabilities.   Survey article: Simos, D. E., Kuhn, R., Voyiatzis, A. G., & Kacker, R. (2016). Combinatorial Methods in Security Testing. IEEE Computer, 49(10), 80-83. Introduces CT-based approaches for security testing and presents our case studies and experiences so far. The success of the presented research program motivates further intensive research on the field of combinatorial security testing. In particular, security testing for the Internet of Things (IoT) is an area where these approaches may prove...

NEW:  Combinatorial Coverage Difference Measurement for assured autonomy in critical software.  Autonomous systems are increasingly seen in safety-critical domains, such as self-driving vehicles and autonomous aircraft.  Unfortunately, methods developed for ultra-reliable software, such as avionics, depend on measures of structural coverage that do not apply to neural networks or other black-box functions often used in machine learning.   This problem is recognized and teams are seeking solutions in aviation and other fields. As one notes, "How do we determine that the data gathered to train...

The field of formal methods covers a broad range of mathematically-based techniques for specifying and verifying properties of software and systems.  Formal methods can be very effective for certain classes of problems, but they have gained a reputation for enormous expense.  One of the greatest opportunities for cost-effective use of these methods is the union of formal methods with testing. When a formal specification can be used in generating expected test results, the cost of developing the specification can be offset by a great reduction in the otherwise high cost of producing a test...

SEQUENCE COVERING ARRAY LIBRARY  The sequence covering array construct described below was introduced in: D.R. Kuhn, J.M. Higdon, J.F. Lawrence, R.N. Kacker and Y. Lei, "Combinatorial Methods for Event Sequence Testing",   First International Workshop on Combinatorial Testing, in Proceedings of the IEEE Fifth International Conference on Software, Testing, Verification and Validation (ICST 2012), Montreal, Quebec, Canada, April 17-21, 2012, pp. 601-609.   Preprint Many testing problems involve sequences of operations. For example, an embedded system may accept multiple sensor inputs and...

Combinatorial methods improve security assurance in two ways: Reducing vulnerabilities - Multiple studies show that about two-thirds of security vulnerabilities result from ordinary coding errors that can be exploited (for example, lack of input validation).  By identifying errors more efficiently, combinatorial testing can reduce vulnerabilities as well.  Specialized security testing - We have been able to achieve huge improvements in fault detection for cryptographic software, hardware Trojan horse and malware, web server security, access control systems, and others.   Below are some...

Oracle-free Testing Combinatorial methods make it possible to detect a significant number of faults without a conventional test oracle.  This seemingly impossible task is achieved using two layers of covering arrays with equivalence classes, as shown in this presentation.  Kuhn, D. R., Kacker, R. N., Lei, Y., & Torres-Jimenez, J. (2015, April). Equivalence Class Verification and Oracle-free Testing Using Two-layer Covering Arrays. In Software Testing, Verification and Validation Workshops (ICSTW), 2015 IEEE Eighth International Conference on (pp. 1-4). IEEE. Automated Test Generation...

Also see our user manual for the coverage measurement tool. Measuring Test Quality with Combinatorial Coverage D. Richard Kuhn, NIST, Raghu N. Kacker, NIST, Yu Lei, University of Texas Arlington There are few good methods for evaluating test set quality, after ensuring basic requirements traceability. Structural coverage, mutation testing, and related methods can be used if source code is available, but these approaches may entail significant cost in time and resources. Combinatorial methods make possible an alternative measure of test quality that is directly related to fault...

This research grew out of our 2001 paper on failures in medical device software, which found that the failures were triggered by only 1 to 4 variables interacting. Surprisingly, although "pairwise" testing had been popular for many years, no one had looked at the actual distribution of failures by number of interacting factors. We continued this work and published other papers finding that all, or nearly all, software failures involve interactions among a small number of variables, no more than 6, in thousands of failure reports. Below are some of  our research areas. If you'd like to find out...

Papers Covering Array Library Seminars & Talks & Tutorial Combinatorial Methods For Modeling & Simulation Workshop Papers DOs and DON'Ts of testing

Through quarterly meetings and email list, the Forum provides our members: a venue to exchange information, share ideas and best practices, resources, and knowledge; an ongoing opportunity to leverage the work done in other organizations to reduce possible duplication of effort; and access to a community and network of cybersecurity and privacy professionals across the U.S. federal, state, and local government and higher education organizations.  Quarterly Meetings Refer to the CSRC Events Page for upcoming Forum meetings and registration information.   Forum meetings are open to...

Steven Lipner, Chairperson  Executive Director  SAFECode Term Expires 5/30/2026 Dr. Brett Baker Inspector General for the National Archives U.S. National Archives and Records Administration Term Expires 3/14/2026 Giulia Fanti Assistant Professor Carnegie Mellon University Term Expires 7/8/2025 Jessica Fitzgerald-McKay Co-Lead, Center for Cyber Security Standards (CCSS) National Security Agency Term Expires 3/3/2023 Brian Gattoni Chief Technology Officer within the Cybersecurity  and Infrastructure Security Agency (CISA) Department of Homeland Security Term Expires 8/6/2023...

Below is the schedule for upcoming ISPAB Meetings for 2022: July 13-14, 2022 October 26-27, 2022 Meetings Held in 2022 March 09-10, 2022 Virtual Meeting Federal Register Notice Announcing Meeting Agenda Meeting Minutes Link to March 2022 Event Page: https://csrc.nist.rip/Events/2022/ispab-march-2022-meeting Meetings Held in 2021 December 08-09, 2021 Virtual Meeting Federal Register Notice Announcing Meeting Agenda Meeting Minutes Link to December 2021 Event Page: https://csrc.nist.rip/Events/2021/ispab-december-2021-meeting September 28, 2021 Virtual Meeting Federal Register Notice...