LESSONS LEARNED FROM THE FEDCIRC PILOT

Fran Nielsen, National Institute of Standards and Technology (NIST)
Marianne Swanson, National Institute of Standards and Technology (NIST)

August 1998

ABSTRACT

In less than two years, the Federal Computer Incident Response Capability (FedCIRC), initially a Government Information Technology Services (GITS) Board pilot project, has demonstrated the need for coordinated incident handling government-wide and the success of a virtual team approach. Formally announced in October, 1996, FedCIRC will transition to an operational program administered by the Office of Information Security of the General Services Administration on October 1, 1998. The GITS Board sponsored the FedCIRC pilot for 18 months and six Federal organizations subscribed to the FedCIRC services during the pilot experience. This paper describes the pilot and the lessons learned from it.

BACKGROUND

The need for an incident handling capability that crosses Federal government agency and organizational boundaries has never been greater. Both in the private and public sectors, organizations are becoming more dependent on information technology (IT). In fact, the IT explosion is demonstrated by the complexity of today's computer systems and on the increasing dependency to internetwork these systems. An effect of the IT explosion is the increased frequency of computer security related incidents affecting almost every organization relying on IT to conduct its business. Federal agencies dependence on networking and their rapid and expanding involvement in the use of the Internet and other related technologies lends a sense of urgency to addressing the problem of incident handling. Diverse sources, from the General Accounting Office to congressional hearings to the Presidential Commission on Critical Infrastructure Protection, describe the insecurity of Federal Government information technology (IT) infrastructures as illustrated by the dramatic increase in electronic break-ins in the public and private sectors. The Department of Defense alone had 250,000 hacker attempts on its computer systems last year. Newspapers and television report that computer hackers, computer viruses, and threats imposed on the Nation's computer systems are escalating. At a recent Chief Executive Officer's conference, attendees were told that government and corporate computer break-ins by hackers is a $10 billion-a-year problem.

The FedCIRC pilot was designed to address the near- and long-term incident handling needs of the Federal civilian community, by providing incident handling services to civilian agencies and by building agency competence and self-reliance in incident handling. The military incident handling coordination service is provided by the Department of Defense's ASSIST team and numerous industry teams serve private sector constituencies.

The use of information technology is integral to President Clinton's plan of re-engineering the Federal Government to make its services more accessible, more efficient, and easier to use. In 1993, President Clinton formed the Information Infrastructure Task Force (IITF) to deploy the National Information Infrastructure and Vice President Gore established the Government Information Technology Services (GITS) working group under the IITF "to improve the application of information technology by government agencies." The task of the GITS working group was to implement the IT initiatives outlined in the National Performance Review. The working group published its accomplishments in June 1996 and Executive Order 13011 codified their activities and established the GITS Board in July of that year. Currently, the GITS Board meets monthly and administers the IT Innovation Fund. The Fund was established in fiscal year 1995 using funds from the FTS-2000 long distance telecommunication program. Approximately one percent (1%) of the projected FTS-2000 income goes into the Fund. The Fund's purpose is to "seed" money for innovative IT projects in the Federal community. To be selected as a pilot, projects must demonstrate more efficient and effective delivery of services to the public and should provide cross agency benefits. Additionally, projects should become self-sustaining after two years. Projects are selected by the Innovation Fund Committee, comprised of GITS Board members.

FEDCIRC SELECTED AS GITS PROJECT

Initially proposed to GITS in March of 1996, the idea to establish a government-wide Incident Response Capability (IRC) was not a new one; however, the National Institute of Standards and Technology's (NIST) proposal offered a rich blend of experiences and the promise of immediate incident response capabilities through the use of two existing, well-recognized teams, the Department of Energy's Computer Incident Advisory Capability (CIAC) and the Department of Defense's CERT(SM) Coordination Center (CERT/CC). The proposal envisioned an IRC equivalent to the Department of Defense incident handling team, ASSIST, and recommended a close relationship between the IRC and ASSIST to ensure that all national systems, including support for national security related systems.

On June 3, 1996, GITS granted $2,796,000 to the National Institute of Standards and Technology (NIST) to establish a Federal Computer Incident Response Capability (FedCIRC). The capability would assist Federal civilian agencies in their incident handling efforts by providing proactive and reactive computer security related services. By combining the experience and expertise of NIST's Computer Security Division, CERT/CC, and CIAC, FedCIRC could provide agencies with cost reimbursable, direct technical assistance and incident handling support. NIST planned to subcontract the operational incident handling capability to the CERT/CC and CIAC, keeping the responsibility for operational management and for facilitating the development of incident handling standards and guidelines by utilizing the vulnerability data collected by FedCIRC. NIST also planned to use vulnerability information in the analysis and testing of software and other products.

FEDCIRC OBJECTIVES

The goal of the FedCIRC project was to develop a self-sustaining response capability that met the need of the federal civilian agencies. To that end, the FedCIRC objectives included:

to respond effectively and in a timely manner to security incidents by analyzing the problem, determining the magnitude of the threat, providing technical assistance to identify and close vulnerabilities, notifying sites affected, and issuing advisories to warn of the problem and describe countermeasures;
to expand the limited coverage of existing agency computer response teams by providing coverage for a broader range of incident types and technologies;
to provide agencies with guidelines on implementing vulnerabilities "fixes" and other security controls;
to maintain a 24 hour, 7 days a week response service for emergencies and a "help desk" function for normal business hours;
to facilitate the interaction with law enforcement agencies in the reporting of security incidents involving violations of the law;
to assist federal law enforcement in evidence gathering, where appropriate;
to perform "tiger team" attacks and offer intrusion detection services;
to coordinate information sharing with other incident handling organizations, including the Forum of Incident Response and Security Teams (FIRST);
to develop, distribute, and maintain publicly available security tools, incident handling tools, and data gathering and reporting tools;
to coordinate with vendors and Internet service providers to provide critical security patches and "work-arounds";
to perform vulnerability analysis to identify a vulnerability's root cause in order to identify other potential problems before they occur; and
to keep the federal community aware of the current threats through education in current technology and associated threats, training for security and network administrators on security practices; and awareness through Web sites, ftp services, and guidance documents.

FEDCIRC OPERATIONS

One of the most challenging aspects of FedCIRC was the need to quickly create a virtual, seamless organization that spanned the Nation and offered a focal point for incident response around the clock. NIST's role was the overall management of FedCIRC, while CERT and CIAC performed the more traditional operational roles. Prior to the start of the FedCIRC collaboration, each entity had its own operating procedures and methods of conducting business. To perform as a virtual coast-to-coast team, however, the three FedCIRC collaborators agreed to a set of common procedures for coordinated activities and NIST produced an Operations Manual to describe them.

During the pilot experience, the energies and resources of the FedCIRC team (NIST, FedCIRC-East (CERT/CC), and FedCIRC-West (CIAC)) focused on handling incidents, on educating agencies about the need for incident handling, and on soliciting sponsorship for the continuance of the project.

The cornerstone activity of the FedCIRC pilot was incident handling. The availability of the incident response hotline support (i.e., 5 days a week x 12 business hours -- Monday through Friday, 8:30 a.m. to 9:00 p.m. east coast time -- for immediate response and 24 hours x 7 days a week for emergencies) as well as the collection, analysis, and publication of threat, vulnerability, and other security related data was accomplished by an underlying infrastructure of FedCIRC activities consisting of the following:

alert creation; interaction with other incident handling organizations, law enforcement, and vendors;
threat and trend analysis;
hotline availability;
data tracking;
vulnerability analysis;
report generation;
database maintenance;
guidance documents (e.g., example practices);
web site maintenance; and
technology watch.

In the first year of the pilot, FedCIRC handled 244 incidents affecting thousands of sites. In the first half of fiscal year 1998, FedCIRC responded to 400 incidents affecting tens of thousands of sites. In 1997, eighty-four FedCIRC advisories were produced and as of August 7, 1998, seventy-one FedCIRC advisories have been distributed. The FedCIRC Web site (http://fedcirc.llnl.gov) has been accessed more than 370,000 times. The FedCIRC team gave twenty-one courses, trained thousands of Federal employees and their contractors, and visited scores of organizations to raise their awareness about the need for increased IT security measures, including incident response.

SUBSCRIBERS

The funding model used for the FedCIRC pilot was subscription based. Three yearly subscription fees, paralleling three service levels, were offered: platinum ($250,000 per year), gold ($110,000 per year), and silver ($50,000 per year). The philosophy behind the use of subscription levels was that organizations needing more service (e.g., more hours of dedicated incident handling, assistance to develop an organic incident handling capability, evaluation of particular systems or subsystems) could acquire it, while agencies and organizations requiring less service or merely wishing back-up for "hard to handle" incidents could be covered at a reduced cost to them. Six organizations signed on as FedCIRC subscribers during its pilot phase and over three-quarters of a million dollars of subscription funds help sponsor the FedCIRC pilot after the first year.

The subscribers of the FedCIRC pilot are to be applauded as part of the successful collaboration that demonstrated the feasibility of an incident handling capability crossing agency boundaries. The subscribers were the Bureau of Alcohol, Tobacco and Firearms; the Federal Supply Service of the General Services Administration; the National Finance Center of the Department of Agriculture; the Department of Justice; the Department of State; and the U.S. Customs Service. These organizations recognized the importance of incident response as an integral part of a good IT security program.

FEDCIRC ACCOMPLISHMENTS

The FedCIRC pilot provided computer security incident handling services and support to the Federal civilian sector; promoted sharing of threat and vulnerability information across agency boundaries; provided training and awareness; produced advisories, guidance, and tools for preventing and/or mitigating attacks on vital public IT systems; and assisted victimized agencies in response to incidents and attacks.

FedCIRC's virtual team approach has saved the taxpayer arguably millions of dollars in direct and indirect savings. FedCIRC, much like the insurance companies, must project savings based on preventive measures taken and the small amount of data from actual incidents. By augmenting existing agency teams, FedCIRC reduces the need to develop redundant full function incident response teams. As examples, full function incident response teams for NASA and for the Department of Energy are estimated at one million dollars each for each year. It can then be argued that given the eleven departments within the U.S. Government which do not have full function incident response teams, FedCIRC potentially saves the taxpayer roughly eleven million dollars each year by providing this service to them. Also, it can be presumed that some benefits are achieved via avoidance costs when future attacks are deterred by those notified of a vulnerability or threat and by those who have received a FedCIRC product (e.g., training, tools, threat advisory, web site visit for a security patch).

During the course of the pilot, the FedCIRC team responded to nearly a thousand calls for information and has handled almost 500 incidents affecting tens of thousands of sites. Thousands of Federal IT users, managers, and technical support staff have been exposed to FedCIRC's prescriptive IT security training, such as how to establish an incident handling capability; connecting to the Internet securely; practical intrusion detection for NT, UNIX and other host-based systems; web security; current trends with regard to threats, hackers, and vulnerabilities; information security for managers; and establishing a computer security forensics analysis program. FedCIRC's first annual conference, held after a year of operation, engendered extremely favorable attendee responses based on its low cost and high quality. Indeed, the accomplishments of FedCIRC during its brief pilot experience, and the lessons learned from it, tell an extraordinary success story.

LESSONS LEARNED

Several key lessons were learned from the FedCIRC pilot and they are summarized in the points below.

a virtual team is viable. The virtual team approach works. Three diverse organizations agreed upon operating procedures and successfully performed as a cohesive team. Time and energy spent on establishing the team were critical to its smooth operation and success.
all incidents are not equal. Computer security incidents range in scope and size. The amount of effort to handle incidents varies with the complexity of the incident and with the number of sites affected. Because of this, it is very difficult to attach a cost per incident; however, for most organizations a cost-benefit must be shown prior to support for incident handling.
customers remain unaware of the threat and its potential impact. A significant gap exists between those agencies aware of computer security issues and those that are unaware. Many agencies still have an attitude of "it couldn't happen here" or "it's not happening here." This lesson helped focus FedCIRC on the need for additional training for users, managers, and administrators of Federal IT systems. During the pilot, the FedCIRC team gave twenty-one courses to thousands of Federal employees and their contractors.
variance in security expertise. Agencies are at different stages along the continuum of information security expertise -- some agencies are newly connected, some agencies still cling to mainframe activities. Many agencies are just beginning to attack the security issue. Knowledge and skills of systems administrators range from novice to expert; however, in general, an overall lack of computer security expertise is evident.
the subscription model for supporting an incident response team is inappropriate and unworkable. A dichotomy exists between the expectations of subscribers for special attention and the need of the electronic community for trouble-free networking. Assistance must be available to all federal civilian agencies, not just available to subscribers; and, in an ideal world, all agencies would share the expense of such assistance. Like a fire department that responds to any and all fires, not merely those of taxpayers in good standing, an incident response team must help wherever problems exist, not merely help with subscriber incidents. Incident response is not a stand-alone operation. Subscribers expect special attention, yet the reality of incident response requires that all organizations involved in an incident be helped.

FedCIRC is needed. The Federal civilian community needs assistance and guidance now in handling computer security related incidents. And, a coordinated approach to incident handling is extremely preferable.

FUTURE OF FEDCIRC

The project described in this paper was a pilot, funded by GITS. The pilot phase will soon be over and FedCIRC will become a fully operational activity as of October 1, 1998. The pilot demonstrated the need for coordinated incident handling government-wide and the success of a virtual-team approach; however, the problem of obtaining continued and continuous funding using the subscription model remained problematical. The Chief Information Officers (CIO) Council championed the project and facilitated its transition from proof-of-concept to a mature information security service.

Under the auspices of the Office of Information Security at the General Services Administration, the new FedCIRC will continue to be a collaborative partnership of computer incident response and security professionals who work together to handle computer security incidents and to provide both proactive and reactive security services for the Federal government. While FedCIRC will not replace existing agency or organizational response teams, it will serve as the focal point for Federal civilian agencies when dealing with computer related security incidents.

Return to Previous Page

Return to Incident Handling Homepage

Return to CSRC Homepage Please send comments or suggestions to webmaster-csrc@nist.rip Last Modified: December 2, 1998.