Smashing the Watermelon Effect with Experience Metrics in Jira Service Management

Conversing with IT teams and employees often reveals a disconnect between perceived operational success and the true experiences of end-users. This discrepancy, famously known as the “Watermelon Effect”, is not an inevitable problem.

In this article, we’ll explore how integrating Experience Level Agreements (XLAs) alongside traditional Service Level Agreements (SLAs) in Jira Service Management can help identify and remedy this phenomenon.

What is the Watermelon Effect?

The Watermelon Effect occurs when IT services appear green (good) on the outside, typically shown through compliance with SLAs, but are red (problematic) on the inside due to underlying issues that affect user satisfaction.

Essentially, while services might meet operational metrics, they fail to meet the actual needs or expectations of the end-users, leading to dissatisfaction and inefficiencies that are not immediately apparent from the surface metrics.

What causes the Watermelon Effect in ITSM?

In ITSM, the Watermelon Effect can stem from several factors. Root causes vary from organization to organization, but three factors emerge as frequent patterns leading to this disconnect:

• Misalignment of metrics: Often, IT teams focus on operational metrics such as uptime and ticket closure rates, which do not necessarily translate to user satisfaction
• Lack of end-user feedback: Without regular and structured feedback mechanisms, it’s challenging for IT teams to understand the user’s perspective
• Overemphasis on quantitative metrics: Prioritizing quantitative data over qualitative insights often leads to overlooking user experiences and satisfaction

Solving this challenge requires a paradigm shift that traditional approaches do not permit. The emergence of user-centric IT frameworks and tools, such as Experience Level Agreements, can help IT teams move the needle in the right direction.

The Case for using Experience Metrics (XLAs) on top of SLAs

Experience metrics, or XLAs, serve as an essential complement to the traditional SLAs by focusing on the satisfaction and experiences of the service users. XLAs help move the focus from just meeting technical benchmarks to enhancing the actual service experience. By integrating XLAs, organizations can:

• Enhance user satisfaction: Directly address what end-users value most in their interactions with IT services
• Improve service quality: Foster a deeper understanding of the qualitative aspects of service delivery
• Drive operational excellence: Align operational goals with user expectations, leading to services that are not only efficient but also effective and well-received

Ready to take the plunge? Let’s have a look at how you can tackle this challenge and set-up XLAs in Jira Service Management.

How to implement and monitor XLAs in Jira Service Management

What you can do with native Jira Service Management features

Natively, teams can opt to enable CSAT surveys: a single question with a one-to-five star rating emailed to the reporter when a ticket is resolved or closed. This method of surveying is transactional and prioritizes operational efficiency over experience. It serves one purpose: how did ticket MyTicket-34567 go? Jira Service Management presents the results as an time series and average of those surveys.

Also natively, SLA success metrics are present stand-alone as single reports.

However, neither of these on their own have a strong connection to the overarching goals of the organization are a reflection of a service goal or an experience ambition. The goals of an organization are not respond to a ticket within 30 minutes or 2 hours. For instance, a service management team or organization seeks to provide prompt and high-quality services when onboarding new employees so they can be productive on their very first day.

Elements Pulse: the app that makes running a user-centric IT team easy

At Elements, we believe there’s a better way, and we want to empower IT to connect with its users on a level that creates value and delight beyond expectations.

We designed Elements Pulse as the central hub for IT managers and their teams to measure and improve customer satisfaction with IT services directly in Jira Service Management.

Here’s a brief overview of the app capabilities:

Fixing the Watermelon Effect with user-centric IT

To truly smash the Watermelon Effect, IT departments must adopt a user-centric approach.

By embracing XLAs alongside SLAs in Jira Service Management, IT leaders can move beyond the facade of operational success and drive genuine improvements in user satisfaction and service quality. This approach not only resolves the hidden issues within IT services but also aligns operations with the true needs and expectations of the end-users, fostering a more effective, responsive, and user-focused IT environment.

After all, to quote Maya Angelou:

“People will forget what you said, people will forget what you did, but people will never forget how you made them feel.”

As the dynamics of IT service management evolve, so does the need for a more nuanced approach to measuring success. Traditional operational metrics IT has relied on for years tell only part of the story. In a role where success hinges not just on systems running smoothly but on users feeling supported, Experience Level Agreements (XLAs) have emerged as an instrumental tool in gauging the true quality of IT services.

Join us as we explore how XLAs can be effectively utilized within Jira Service Management to enhance customer experience, ensuring that services not only meet operational standards but also deliver genuine satisfaction and value.

What are XLAs?

Experience Level Agreements (XLAs) are a transformative approach to IT service management agreements that prioritize the end-user’s experience over the conventional metrics typically found in Service Level Agreements (SLAs). Unlike SLAs, which focus primarily on the quantity and efficiency of service delivery (e.g., response times, issue resolution times), XLAs measure the quality of the interaction and the satisfaction of the user with those services. They aim to encapsulate the subjective elements of service delivery, ensuring that the services provided are not only efficient but also effective and enjoyable.

SLAs vs XLAs: when IT is a victim of the Watermelon Effect

Traditional Service Level Agreements (SLAs) are often celebrated for keeping things running smoothly — at least on the surface. Yet, they can be deceptive, presenting a facade of efficiency while hiding underlying user dissatisfaction. This disparity is known as the watermelon effect, where metrics look good on the outside (green) but are problematic on the inside (red). XLAs challenge this by measuring what really matters to users — their actual experience and satisfaction, thus providing a truer representation of service effectiveness.

Indeed SLAs, by their very nature, encourage a focus on meeting predefined technical targets, often leading service teams to prioritize compliance over user satisfaction. This can result in scenarios where, although the technical aspects of a service are within agreed parameters, the users may still be dissatisfied due to factors such as poor communication, lack of empathy, or inefficient processes that are not captured by traditional SLAs.

Experience Level Agreements (XLAs), on the other hand, are designed to combat the watermelon effect by shifting the focus from purely operational metrics to the actual experiences of the end-users. XLAs assess the impact of IT services on user productivity, satisfaction, and overall experience. They help organizations measure what really matters to the end-user, such as how supported they feel, how effectively their issues are resolved in context, and their overall engagement with the service team.

Satisfaction Maturity levels: why you should go beyond traditional CSAT surveys

Collecting user feedback through Customer Satisfaction surveys is a good start, and Jira Service Management offers a simple, built-in mechanism to collect CSAT ratings and comments on resolved issues. However, these transactional Customer Satisfaction surveys provide only a snapshot of customer sentiment post-service, which does not accurately represent the overall experience.

XLAs fill this void by offering a continuous, in-depth analysis of user interactions over time, thus providing a more comprehensive view of the service effectiveness across different touchpoints and journeys.

Moving beyond traditional CSAT surveys to a more mature model of satisfaction measurement empowers IT teams to not only react to, but anticipate and meet, user needs more effectively, thus enhancing the overall service quality and user satisfaction.

Satisfaction Maturity levels in IT: The 4 levels

The concept of satisfaction maturity in ITSM can be seen as a journey of evolving focus, from basic operational metrics to a comprehensive, strategic embrace of customer experience. This journey is marked by distinct maturity levels, each representing a deeper integration of customer satisfaction into the service management framework.

At Elements, we envision four levels:

• Level 0 – No Experience Measurement: At this stage, there’s little to no active tracking of user satisfaction or service quality. IT teams are mostly reactive, lacking any structured insights into user satisfaction.
• Level 1 – Reactive Measures: Basic surveys are used after ticket resolution, providing snapshot feedback that focuses on specific interactions. The key measurement is usually a simple rating scale (e.g., 1 to 5 stars), which reflects immediate user reactions rather than deep insights.
• Level 2 – Proactive Engagement: Advanced tracking mechanisms like XLAs begin to be implemented, considering broader user experience factors. This level involves continuous data collection and regular analysis, enabling IT teams to quickly identify and address service issues as they occur. However, the focus remains largely on rectifying negatives rather than creating positive experiences.
• Level 3 – Integrated Experience Focus: XLAs are fully integrated, with continuous improvement cycles and a deep focus on user-centric service management. Customer experience is fully integrated into the IT service strategy, and insights are used not only to address service issues but also to drive service design and delivery. Satisfaction measurement is continuous and multifaceted, involving both quantitative and qualitative data to capture a comprehensive view of user experience.

By advancing through these levels, organizations can ensure that they are not just solving immediate problems but are continuously enhancing the user experience, aligning IT services more closely with user needs and expectations, and fostering a culture of excellence in service delivery. This progression is essential for transforming IT from a support function into a strategic enabler that drives organizational success.  

Measuring Experience in Jira Service Management: How to get started with XLAs

Implementing Experience Level Agreements (XLAs) in Jira Service Management involves a structured approach to integrate user experience metrics into your IT service framework.

While you could go the manual way, stick with the native Jira Service Management features and try to create some custom reports and dashboards, it would require a lot of work and provide a deceptive and incomplete solution. To help IT teams connect with their customers on a level that creates value and delight beyond expectations we’ve built Elements Pulse, the app for Jira Service Management that provides IT with a central hub for customer-centric insights and drives continuous improvement.

Here’s a step-by-step guide to help you get started on measuring and enhancing customer experience effectively with Elements Pulse so your IT team makes the leap from Reactive measure to Proactive Engagement.

Step 1: Define the scope of your analysis

Begin by determining the scope of your analysis: select the Jira Service Management projects to pull data from, specify the time period for your analysis, and choose the categories of indicators against which you want to measure your performance.

Step 2: Understand your performance

Start with a broad overview of your performance using the dashboard hub. Then, delve into each category of indicators—Customer Satisfaction, SLAs, Service Quality, and Productivity—to understand where you and your IT team can make improvements.

Step 3: Create a custom survey

To gather feedback from your users, utilize a survey template to craft a custom survey. Enable agents to initiate this survey from an issue upon resolution or at any significant point in the workflow. Aggregated survey results will be directly accessible within your Pulse survey module.

Step 4: Set Experience goals (XLAs) for each indicator

Establish clear targets for each indicator, drawing on insights from your previous performance, and unite your team around these objectives.

Step 5: Analyze feedback and adapt

Regularly review the feedback and data collected through Elements Pulse. Use this data to identify trends, pinpoint areas of improvement, and assess the success of recently implemented changes. Analysis should lead to actionable insights that drive continuous improvement in IT service delivery and user experience.

Step 6: Report and iterate

Create regular reports to share with stakeholders that highlight key findings, progress against XLAs, and areas needing attention. Use insights from Elements Pulse to create reports that will demonstrate the progress of your experience to key stakeholders. These reports are essential for ensuring transparency and for making data-driven decisions to refine IT services further.

Step 7: Foster a culture of Continuous Improvement

Promote a culture that values user feedback and continuous improvement. Encourage your IT team and stakeholders to regularly review XLA outcomes and to brainstorm solutions for enhancing service delivery. This cultural shift is essential for sustaining long-term improvements in user experience and for achieving the full benefits of implementing XLAs in Jira Service Management with Elements Pulse.

By following these steps, you can effectively measure and improve the customer experience in Jira Service Management using XLAs thanks to Elements Pulse. This approach not only enhances user satisfaction but also aligns IT services more closely with business objectives and user needs.

The Future of ITSM is Customer Centric

By embracing XLAs, companies can ensure their IT operations truly support their most important asset: their users. The transition from SLAs to XLAs does not mean abandoning traditional metrics but rather complementing them with experience-focused measurements. This dual approach ensures that while the service continues to meet operational requirements, it also excels in delivering a superior user experience. This shift not only helps avoid the pitfalls of the watermelon effect but also aligns IT services more closely with broader business goals and user needs, fostering a more supportive and responsive IT service environment.

When managing IT services, the focus often leans heavily towards operational efficiency—how quickly issues are resolved and how many tickets are processed. However, if you’re aiming to truly enhance service delivery, it’s crucial to shift your attention from operational outputs to the actual experiences of your users. After all, the fundamental purpose of IT is to support its customers – employees of the organization- so that they are in the best conditions to do the ‘real job’.

In this article, we’ll explore how you can improve customer experience in Jira Service Management by focusing on the right metrics.

Why measure Experience

In the context of IT Service Management, where user interactions are often problem-driven, a positive experience can significantly mitigate the frustration associated with technical issues.

The quality of the experience your department provides to employees directly impacts the perception of IT in your organization, but it doesn’t stop there. When employee experience with IT is positive, it boosts productivity and promotes a culture of excellence. This can lead to higher engagement, increased employee loyalty, and thus better business performance.

XLAs vs SLA: What’s the difference

Traditionally, Service Level Agreements (SLAs) have been the backbone of IT service contracts, focusing on the timeliness and accuracy of services delivered. SLAs are about the ‘what’ of services—what gets done and when. Experience Level Agreements (XLAs), on the other hand, shift the focus to the ‘how’—how users feel about the service they receive, emphasizing the outcome and value over the output.

While SLAs track metrics like uptime and response times, XLAs might look at user satisfaction scores, ease of use, and how effectively issues are resolved from the user’s perspective. Integrating XLAs into your ITSM framework means prioritizing these human-centric metrics, which more accurately reflect the health and effectiveness of your IT services.

To better understand how Experience Level Agreements (XLAs) differ from traditional Service Level Agreements (SLAs), let’s look at a side-by-side comparison.

The below table illustrates the core distinctions, shedding light on how each type of agreement measures service delivery and success.

To sum up, while SLAs focus primarily on quantifiable operational metrics, XLAs emphasize the overall experience and satisfaction of the end-user, aiming to enhance the perceived value of IT services.

The challenges of measuring customer experience in Jira

Implementing and tracking experience-focused metrics in Jira Service Management can be challenging. The tool is inherently designed around operational efficiency, and while it offers robust functionalities for tracking SLAs, it lacks a module to effectively measure and analyze customer experience metrics.

Challenges include integrating user feedback directly into the service workflow, correlating this feedback with service performance data, and ensuring that this data influences IT service strategies for the better.

Customer experience in Jira: the metrics that matter and can actually be tracked

To truly gauge customer experience in Jira, you need to consider metrics that reflect not just the efficiency but also the effectiveness and emotional response to the service provided.

Here are four essential categories of metrics to consider, each playing a key role in providing a holistic view of your IT operations performance:

Taking into account these four lenses enables a more strategic enhancement of your service delivery, ensuring that your IT services are not just functional but truly aligned with the needs of users.

At Elements, we aim to empower IT to connect with employees on a level that creates value and delight beyond expectations. We designed our app Elements Pulse to provide a central hub for monitoring and improving customer satisfaction in Jira Service Management.

The app includes a comprehensive visual overview dashboard that provides a clear snapshot of your current IT experience status. It offers detailed insights derived from your Jira data, an integrated survey tool for collecting user feedback, and a dedicated module for setting and tracking experience goals through Experience Level Agreements (XLAs).

This robust combination ensures that you have all the tools necessary to effectively measure and enhance user experience within your IT services.

Ready to craft IT experiences that leave a lasting impression?

By shifting focus from traditional SLAs to more reflective and comprehensive XLAs, and by carefully choosing and monitoring the right metrics in Jira Service Management, you can not only meet but exceed user expectations, thereby transforming the IT service experience.

Remember, in the digital age, great service is not just about solving problems—it’s about how your users feel when interacting with your IT service desk.