Measuring What Matters: Project Athena Innovation Program’s Real-World Testing

By Sudha Ganesh, Jim Scovell and Clem Wong

For years, the PC industry has faced a central question: what’s the ‘right’ way to test laptops?

Traditional benchmarks have succeeded in measuring raw computing power with highly repeatable tests in lab conditions. But after extensive human-centered research, Intel has found that these benchmarks don’t completely reflect the criteria that matter most to people. The right benchmarks and pure processing speed are still important, but not a complete view of the best laptop experience. People want a laptop that’s always ready, helps them focus and stays responsive as they multi-task throughout the day—all with stellar battery life.

People may care about pure processing speed only to the extent that it impacts their experience, but that does not define the full expectations of a great laptop experience. That’s why, when we launched the Project Athena innovation program in 2019, we anchored our testing approach in solving these human, real-world needs.

And it’s why we regularly update this testing methodology with each successive Project Athena specification to reflect new research on the latest trends. For example, laptops that feature the Intel® Evo™ brand must meet all the testing requirements of the 2nd edition Project Athena spec that aims to match real-world user experience needs.

Intel believes the “right” way to test laptops is by using metrics that measure real-world user experience. Testing results then reflect true user utility and value—which is ultimately what matters to people.

To help the industry deliver a new class of advanced laptops, Intel has developed a one-of-a-kind, automated testing tool that:

1. Focuses on key experiences
2. Reflects real-world workflows and conditions
3. Equips OEMs to optimize platform designs

Key Experiences That Matter

Project Athena rigorously tests designs for three foundational key experiences: consistent responsiveness1, instant wake and a worry-free day of battery life.2 These experiences, drawn from ethnographic research and measured by a set of Key Experience Indicators (KEIs), provide the best basis for impactful laptop innovation.

Starting with People and Their Experiences

Project Athena’s real-world testing is based on extensive, in-depth research to identify the experiences that matter most to people, especially the ambitious and demanding people who expect the best from their laptops, without compromise:

Human-centered innovation: Intel’s research teams boast more than 15 years of global research around human-centered innovation, helping to provide a detailed picture of how people use their laptops, what their pain points are and where innovations can help. In particular, the research focused on deep immersion sessions with people who depend on their laptops to manage their dynamic, fast-paced personal and professional lives.

Identifying key experiences: This research revealed that people want their laptops to help them focus, adapt and always be ready and connected. These are essential to help people stay in their flow, so they can work, connect and play without interruption. To achieve this, we determined that laptops need to wake in less than a second; respond quickly and efficiently to people’s inputs; and have worry-free battery life and charge rapidly—the key experiences for Project Athena.

Defining the Key Experience Indicators (KEIs)

Intel’s technical experts then translated these key experiences into a set of quantitative KEIs—the backbone of our real-world testing methodology:

Focusing on impactful interactions: To select the KEIs, Intel zeroed in on the tasks that have the greatest impact and are the most meaningful to a person’s laptop experience. For the second edition, which is represented by the Evo brand, Intel put a specific emphasis on the connection between local and cloud-based tasks to better reflect today’s agile working environment. Launching email, starting video calls, creating charts and loading browser videos are the kinds of essential interactions that are included in the 25 KEIs in Project Athena’s second edition.

Technical decomposition: Drawing on more than 15 years of Intel’s user testing, Intel’s experts defined time targets—down to the millisecond—for each KEI. This testing uses MOS (mean opinion score) models to determine the exact threshold for a best-in-class experience for each interaction. For example, in Project Athena’s second edition, a laptop must be able to wake in an instant, play a YouTube video quickly or launch Outlook in a matter of seconds. Battery life must last for more than 9 hours under real-world conditions3, while being able to charge 4 hours of power in just 30 minutes with full HD displays.4

Real-World Workflows and Conditions

The Project Athena innovation program measures the KEIs with a realistic “day-in-the-life” workflow in conditions that mirror how people typically use their laptops. This provides a more practical view of performance, as opposed to using only synthetic workflows and artificial lab conditions.

A Rigorous, Realistic Workflow

Unlike benchmarks that are designed to tax specific components such as the CPU or the disk, Project Athena testing is designed to reflect a “day in the life” of a person who depends on their laptop to get things done. The testing tool repeatedly runs a workflow of more than 200 tasks, including the KEIs, which model how people use their laptops every day, based on ethnographic research and telemetry data.

This research revealed that people constantly switch between four “computing modes”: Communication and sharing; personal organization; expert production; and casual creation. This includes everything from less performance-demanding activities, like web browsing, online shopping or reading the news, to more demanding tasks, such as Zoom calls, YouTube streaming or working in the cloud with Google Suite. All of this is incorporated into the workflow with Project Athena’s second edition. 

The testing tool also runs with multiple applications open at once, including constant companions like Outlook, Google Chrome, Spotify, Slack and more. This realistic concurrency tests the laptop’s responsiveness and battery life in the same way as a person who is constantly multi-tasking, browsing the Internet and streaming videos throughout the day.

To ensure the battery-life test is as realistic as possible, we use real telemetry data on communication and sharing, casual creation, organization and expert production workflows. Each test takes about two hours to complete, and we require three unique iterations of the test to ensure consistency.

Intel integrated these real-world insights into the second edition of Project Athena:

Real-World Insights

Improvements in Second Edition Testing

More people doing more multi-tasking Added more constant companions, including Spotify, LinkedIn, Slack and Google Drive
Growing expectations for responsiveness Increased number of KEIs from 15 to 25; increased the responsiveness requirements for all KEIs
Growing expectations for battery life Intensified the workload to make the 9 hours of battery life more meaningful; added delays in workflow that reflect a person’s pauses during day
The rise of the cloud Added KEIs for Google Drive tasks, editing Google Sheets files; added Google Drive as constant companion, along with interacting with Office files on the cloud via Office 365
Adoption of tools for work collaboration and personal connection Added KEIs for Zoom tasks; added Slack as constant companion

Real-World Usage

Percentage of Time Tested based on User-Telemetry

Web Browsing 23%
Productivity 19%
Media Streaming 6%
Collaboration 6%
Idle Time 45%

Real-World Conditions

Project Athena tests in real-world conditions for a more accurate picture of performance. This differs from most benchmarks, which are typically measured in artificial lab conditions—for example, the laptop is plugged into the wall, with a dimmed screen and no Wi-Fi connection. Instead, with Project Athena, the laptop is tested on battery power, on a live network, with common background applications open, the latest software updates, logged into multiple cloud accounts and with the display set to ≥250 nits brightness on LCD panels.

The real-world testing tool runs 15 times to ensure a consistent result for the responsiveness KEIs. Each iteration of the workflow captures changes in responsiveness as the battery drains, and the laptop must exceed a stringent threshold to be verified. If a laptop does not meet these requirements, it undergoes further tuning by Intel and the OEM. In every workflow, the laptop is unplugged, so that Project Athena-verified laptop designs are responsive whenever and wherever they’re needed.

OEMs Empowered to Optimize Platform Designs

Project Athena’s real-world testing provides OEMs with an automated tool that they can use to assess, tune and improve their laptop designs. This equips OEMs—and the entire PC ecosystem—to deliver meaningful innovation, from balanced performance to best-in-class platform technologies.

Driving Innovation Across the PC Platform

OEMs use Intel’s real-world testing tool to optimize performance and battery life, as well as make key decisions about the entire PC platform:

• Drilling into the results: The automated testing tool enables an OEM to view its results for each of the KEIs across multiple runs. Engineers can use this information to identify pain points, understand the problem and tune the system for better performance and battery life.

Beyond the CPU: OEMs can also use the tool to make decisions about components and the overall PC platform, because the test requires more than just processing speed. Laptops must also deliver reliable connectivity for online and cloud-enabled tasks, sophisticated memory and storage for responsiveness, and platform optimization to meet battery life requirements.

Project Athena-verified: Ultimately, each laptop design must meet these KEIs and the spec in order to be verified as a Project Athena laptop5. This ensures that people can buy any Project Athena-verified laptop design with the confidence that it will deliver on their most important key experiences.

Just the Beginning

Project Athena and its unique approach to real-world testing is already helping the ecosystem create complete PC platforms tailored for people’s needs. These laptop designs are tuned, tested and verified to deliver on key experiences under real-world conditions, with a comprehensive set of KEIs that measure performance to the millisecond.

Intel will continue to evolve Project Athena’s testing to reflect trends in how people use their laptops, what they expect and prioritize, and what they don’t even know they need yet.

The Project Athena Innovation Program


Notices and Disclaimers6 7

Product and Performance Information

1

Measured responsiveness of premium Windows OS-based designs while performing 25 workflows predictive of performance under a typical-use environment comprising multiple cloud-based and local apps and web pages including Google* Chrome, Google* G-Suite, Microsoft* Office 365, YouTube and Zoom. Testing conducted on laptops on DC battery ≥30%, connected to 802.11 wireless, and with shipped hardware configurations including Windows* 10 and 250-nit LCD screen brightness. Testing results as of August 2020, and do not guarantee individual laptop performance. Power and performance vary by use, configuration and other factors.

2

For more complete information about performance and benchmark results, visit intel.co.uk/Evo.

3

Time taken to drain from 100% to critical battery level while performing typical workflows in a realistic environment. For more complete information about performance and benchmark results, visit intel.co.uk/Evo.

4

Charge attained from OEM-default shutdown level. For more complete information about performance and benchmark results, visit intel.co.uk/Evo.

5

Based on integrated Intel® Wi-Fi 6 (Gig+) and Thunderbolt™ 4 technology. For more complete information about performance and benchmark results, visit intel.co.uk/Evo.

6

Verified, measured and tested against a premium specification and Key Experience Indicators as part of Intel's laptop innovation program Project Athena. Testing results as of August 2020, and do not guarantee individual laptop performance. Power and performance vary by use, configuration and other factors. For more complete information about performance and benchmark results, visit Intel.co.uk/Evo.

7

Software and workloads used in performance tests may have been optimized for performance only on Intel® microprocessors.

Performance tests, such as SYSmark* and MobileMark*, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit www.intel.co.uk/benchmarks.

Performance results are based on testing as of dates shown in configurations and may not reflect all publicly available updates. See www.intel.co.uk/Evo for configuration details.

No product or component can be absolutely secure.

Your costs and results may vary.

Intel® technologies may require enabled hardware, software or service activation.