Building Blocks of a Quality SAT Program | EP 007

Common Mistakes in Measuring Security Awareness Programs

Today, I want to share some common mistakes companies make when measuring the effectiveness of their security awareness programs. These include relying on a single piece of information, using easy-to-detect phishing assessments, and creating one-size-fits-all tests. Let’s explore how to build a quality program that actually works.

Measuring Success Correctly

I've observed hundreds of security awareness programs, both at individual companies and with partners. Some do it well, others not so much. Here are the most common mistakes I see:

1. Relying on a Single Metric

Many organizations make the mistake of using just one piece of information—like the overall phishing percentage—to judge the effectiveness of their program. This is similar to measuring your health by only your blood pressure or heart rate. While important, one metric alone doesn't tell the whole story.

For example, cyber insurance carriers and brokers often look for a 0% phish rate, and anything above 7% is considered ineffective. This is not accurate. Effectiveness should be measured using multiple data points, such as:

• ​Phishing performance (ability to recognize social engineering)
• ​General sentiment toward the program
• ​Engagement levels
• ​Frequency of manager follow-ups for incomplete training

2. Easy-to-Detect Phishing Assessments

Phishing assessments are like scrimmages before a real game—they have value if done right. The goal is to simulate real-life scenarios so employees can demonstrate their ability to recognize threats. If assessments are too easy to detect (e.g., poorly crafted emails, obvious domains), employees may recognize the test rather than the social engineering attempt. This doesn't help them build real skills.

3. One-Size-Fits-All Phishing Tests

Every employee is unique and vulnerable to different types of phishing. For example, one might fall for an Amazon gift card scam, while another is susceptible to business email compromise. Real-life example: new employees at our company received phishing attempts impersonating me, asking them to join a meeting or buy gift cards. They contacted me to verify, which was the right move.

When designing awareness training, tailor phishing simulations to the specific threats employees are likely to encounter in their roles or industry. Randomization is better than sending the same test to everyone at the same time.

4. Misinterpreting Data Over Time

Suppose you have a 4% phish rate each month, but it's a different 4% of employees each time. Over a quarter, that's a 12% phish rate. It's important to:

• ​Identify what percentage of your workforce is vulnerable
• ​Determine who repeatedly falls for phishing and needs extra support

Recommendations for Effective Measurement

• ​Track multiple metrics, not just phishing percentage.
• ​Monitor employee engagement with training content.
• ​Avoid sending the same phishing assessment to everyone at the same time.
• ​Analyze who is getting phished and who completes training, focusing on individuals who need more support.

Final Thoughts

To build a high-quality security awareness program:

• ​Use multiple data points to measure effectiveness
• ​Make phishing assessments realistic and challenging
• ​Tailor tests to individual vulnerabilities
• ​Analyze data to identify and support those who need it most

Next time, we’ll discuss how to apply human vulnerability management to your security program, taking your awareness training to the next level.

If you want to learn more about launching effective security awareness training campaigns that engage employees and change habits, check out Phin Security at phinsec.io.