Measure, Share, Compare.

• What percent of my business critical assets have severe vulnerabilities? 
• What is the average time it takes to remediate vulnerabilities of low, medium and high severity? 
• How many hours of exposure to known vulnerabilities did our critical assets experience last month?
  What is the difference between a measurement and a metric?

Although metrics and measurements are often used interchangeably, they are very different concepts.  While both measurements and metrics are generated by direct collection of raw data from operational systems, a measurement only provides magnitude.  It does not provide a basis for comparison or context, and without context, you cannot make meaningful decisions.

An example of a measurement is the number of virus incidents.  While quantitatively, the measurement tells you the volume of virus incidents, it does not tell you the severity of these incidents, where these incidents occur, or if these incidents are increasing in frequency over time.  Knowing the volume of virus incidents does little to tell you if these events are an anomaly or if you need to take corrective action immediately.

Metrics are derived from measurements, but enrich them with statistical analysis and contextual information.  Metrics take into account organizational goals and structure, giving you a firm, fact-based foundation for more successful decision-making.  A metric is a clear-cut indicator of the value of your security processes, and compliance to corporate policies and legislative regulations. 

An example of a metric is the ratio of virus incidents originating from contractors versus employees.  By analyzing the origin of virus incidents, the organization can more accurately assess the effectiveness anti-virus controls for virus signatures, and identify policy or process changes that may need to be made for a particular user-base.

To provide value and meaning, metrics have specific additional requirements that go beyond simply being a measurement of their security environment.

What is the composition of a Security Performance Metric?

A metric must contain the following properties: 

• Useful.  There must be a business need to assess the state and performance of a system so that a decision can be made.  The purpose of a metric is to facilitate the decision-making process.
• Repeatable. A metric must be repeatable, so the impact of actions and decisions can be evaluated over time. Repeatability also allows the results to be independently tested. 
• Flexible. Metrics should constantly evolve to adapt to the needs of an organization.  If the organization's strategy, policy, or process changes, so must the metrics that analyze them.
• Accurate. Since decisions will be made based on the results of the metric, the calculation method must be accurate and yield results in the same way each time. 
• Accessible. A metric is only useful if the stakeholders that need the information can access it. The results of a metric must be readily available. 
• Contextual. A metric must be meaningful and relevant to an organization. Metrics should address issues the organization cares about. 
• Transparent. The purpose, algorithm, and analytics of a metric must be auditable and evident to the audience to ensure that correct decisions can be made. It should be clear what the metric is quantifying and what action should be taken based on the results.

How are Security Performance Metrics used?

Metrics enable you to clearly understand the impact that decisions and changes to security processes have on your organization's security posture.  No security program is complete without metrics. Metrics help you to:

• Determine the goals of the program
• Evaluate the current posture and forward momentum toward goals
• Identify when a corrective course needs to be implemented
• Indicate when goals have been achieved
• Optimize the level of security investment

Without metrics these questions can only be answered intuitively, cannot be easily communicated, and are difficult to manage throughout the organization.

How are metrics used in a scorecard?

A scorecard is a collection of related metrics organized to show correlation between your security process, initiative, or goal and the outcome of these processes, initiatives, and goals.  The purpose of a scorecard is to demonstrate state and security posture. A scorecard answers the who, what, where, and when of a security objective. 

Since scorecards are an integral part of your security program, it's essential to publish and review scorecards on a regular basis.  Reviewing scorecards at regularly scheduled intervals ensures that the context of actions and decisions are not lost when the impact is assessed.

The level of granularity of a scorecard depends upon the audience.  Scorecards are generally reviewed at several levels within your organization and contain varying levels of detail:

• Board Summary Scorecards - Evaluate progress on key company-wide security initiatives. Examples include encrypting all laptops that contain sensitive data or implementing security reviews on all significant systems. Goals are updated annually and progress is reported annually and quarterly. 
• Executive Summary Scorecards - Evaluate progress of key initiatives, and include additional levels of detail for programs that show noteworthy results or where significant activity has occurred. Additional scorecards, as needed, to communicate metric results on specific issues. Quarterly reviews and monthly updates on key indicators. 
• Line of Business Manager Summary Scorecards - Grouped by operational responsibilities, and include additional levels of detail for areas under, or affected by, the operations. Additional scorecards that track the impact of specific system, process, or organizational changes on operations are also provided.  Monthly reviews with weekly updates on key indicators. 
• IT/Security Operational Manager Summary Scorecards - Detailed functional-area scorecards evaluating the outcome of processes. Weekly review of process and state, key indicators and monthly summaries to communicate issues, decisions, and resolutions.