Measuring the usability of online hotel booking systems: the operational framework
Part two of two
In the first part of this article, we established the conceptual framework: the usability of booking systems is not an attribute of technology, but a relationship between user, system and context. We identified the main impediments documented by empirical research — confusion, slowness, excessive information density, and low trust — and argued why most of these problems belong to the domain of design rather than software.
Now we enter the operational details. How do we measure the usability of a hotel booking system in a rigorous, comparable way that is useful for guiding concrete decisions? The short answer is: not with a single tool. The detailed answer follows below.
Why no single method is sufficient
There is a widespread methodological temptation, even among the most attentive professionals: choosing a single evaluation tool and considering its results sufficient to describe the state of a system's usability. International research has clearly shown that this temptation must be resisted. No single method can capture the quality of an experience in its entirety.
Methods based on behavioural observation — navigation logs, flow analysis, abandonment rates — measure what users do, but not why. Standardised questionnaires measure the subjective perception of the experience, but not behaviour. Expert evaluation identifies structural problems that no user has encountered yet, but cannot replace the observation of what happens when real users interact with the system.
Seffah, Donyaee, Kline and Padda (2006) proposed a consolidated model that integrates objective and subjective measures into a coherent framework, recognising that interaction quality is multidimensional and requires complementary tools to be fully understood. This principle of methodological integration is the foundation of the approach Esario applies when evaluating hotel booking systems.
The four measurement levels
The starting point of any structured evaluation of a booking system is the analysis conducted by experts on the interface, independent of users. This does not mean an impressionistic inspection: it means the systematic application of consolidated principles, starting from Nielsen's (1994) ten heuristic principles, which retain all their operational validity when applied to a contemporary IBE.
- Visibility of system status Does the booking system communicate to the user where they are in the path at all times? Is the completion of step 2 out of 5 clearly indicated? Does the user know what has been confirmed and what is still pending?
- Match between system and the real world Does the system use language and logic that match the user's expectations? Are "check-in" and "check-out" presented in the order and manner the user expects, or is there friction between their mental model and that of the system?
- User control and freedom Can the user easily go back if they made a mistake in selecting dates or room types, without losing all the data already entered?
- Error prevention Is the system designed to make errors difficult to commit — for example, blocking the selection of check-out dates prior to check-in before the error even occurs?
- Recognition rather than recall Does the user have to remember information from one screen to another, or does the system present it visibly and accessibly at every moment in the path?
Expert evaluation conducted systematically through these principles identifies structural problems in interaction architecture — problems that, if left unresolved, will produce difficulties for any user, regardless of their digital experience. Read more: Usability, UX and Accessibility Evaluation →
Testing with real users introduces something that no expert analysis can provide: the reality of the interaction, with all its unexpected turns.
The question of participant numbers has been debated extensively. Nielsen and Landauer (1993) established that five users identify approximately 85% of usability problems in qualitative tests. Faulkner (2003) showed that variability with five users can be very high — 55% to 99% of problems identified — and recommended extending to fifteen for stable results. Spool and Schroeder (2001) showed that serious problems can emerge even after many users have been tested. Moczamy, De Villiers and Van Biljon (2012) suggested 8 to 12 participants to achieve 80% problem identification.
For hotel booking systems, where user types vary significantly in digital literacy and booking frequency, a minimum sample of ten participants selected to represent the actual user profiles of the property guarantees reliable coverage of the main problems.
During test sessions, the metrics collected include:
- Task completion rate: does the user succeed or fail in completing the booking?
- Time taken: correlating stated expectations from Buhalis and Law to the actual data observed in the session.
- Number of errors committed
- Hesitation points: moments of pause that signal uncertainty or confusion, often at step transitions or field validation moments.
- Unanticipated navigation paths: which reveal the gap between the designer's mental architecture and that of the user.
The distance between problems identified by expert evaluation and those emerging from real user testing is itself a valuable measure: it reveals how much internal perception of the system diverges from the reality of the interaction. Esario's usability testing service →
User satisfaction — the subjective dimension of usability according to ISO 9241-11 — cannot be deduced from observed behaviour nor inferred from expert evaluation. It must be measured directly. The standardised tools available have distinct characteristics and scopes of application:
| Instrument | Items | Scale | Best suited for |
|---|---|---|---|
| SUS (Brooke, 1987) | 10 | 0–100 | Comparative evaluations before/after redesign. Score <68 = problems; >80 = good usability. |
| PSSUQ (Lewis, 1995) | 16 | 1–7 | Three sub-scales: system quality, information quality, interface quality. The information quality sub-scale is often the most revealing for IBEs. |
| CSUQ (Lewis, 1995) | 19 | 1–7 | Broader satisfaction scope. "Computer system" is replaced by "website" or "booking system" in hotel research without loss of validity. |
| USE (Lund, 2008) | 30 | 1–7 | Four dimensions: usefulness, satisfaction, ease of learning, ease of use. Measuring usefulness and ease of use together produces more accurate results than measuring separately. |
The choice of tool is never arbitrary: it depends on the construct to be measured, the phase of the product lifecycle, and the user population. A SUS administered at the end of each user testing session provides individual data that can be aggregated in a statistically robust manner. Read more: Esario's evaluation framework →
Alongside data collected in structured testing sessions, there are metrics that can be extracted directly from system logs and analytics — requiring no additional sessions and encompassing data volumes that no laboratory test could approach. For a hotel booking system, the most relevant include:
- Abandonment rate per phase: in which step do users abandon most frequently? Room selection? Personal data entry? The summary screen before payment? Each abandonment point identifies a localised, often solvable design problem.
- Time per phase: times significantly higher than average at a specific step indicate a comprehension or interaction problem at that precise point.
- Atypical navigation paths: users visiting the same page multiple times consecutively are likely confused by the navigation structure. Paths that include exits toward support pages signal problems the system does not resolve autonomously.
- Form error rate: how many users make errors in required fields, and in which fields? Repeated errors in the same field indicate a design issue — ambiguous labels, unclear validation, or uncommunicated required formats.
- Interrupted multi-device sessions: a significant portion of hotel bookings begin on mobile and conclude on desktop, or vice versa. Sessions interrupted during the device transition signal continuity problems that extend beyond a single touchpoint.
The analysis of behavioural metrics does not replace real user testing — it does not reveal the why behind a behaviour, only its frequency and distribution. However, it allows for the precise identification of where to focus attention, optimising the resources allocated for evaluation. Data Analytics service →
Integrating the levels: from analysis to decision
The four evaluation levels are not alternatives. They are complementary, and their integration produces something no single method can provide: a complete map of the system's usability status, with priorities based on evidence and recommendations anchored to specific data.
The integrated process Esario applies to hotel booking systems follows this sequence:
-
1Expert evaluation
Identifies structural problems — those resulting from design choices that will produce difficulties for any user. This phase requires no participant recruitment and yields quick results.
-
2Analytics log analysis
Existing logs are analysed to identify abandonment points and anomalous paths, directing user testing toward the critical areas already signalled by behavioural data.
-
3Testing sessions with real users
Produces direct observational data — completion rates, times, errors, hesitations — accompanied by the administration of standardised questionnaires at the end of each session.
-
4Integrated report
The results of the three levels are integrated into a document that includes: the list of identified problems, classified by severity and frequency; recommendations for intervention, ordered by priority and estimated potential impact on conversion; and baseline metrics to measure improvement post-intervention.
The result is not just an evaluation document: it is an operational tool for making evidence-based design decisions.
The impact on the hotel's digital ecosystem
A booking system with insufficient usability is not just a direct conversion problem. It is a problem that propagates throughout the hotel's entire digital ecosystem.
A user who fails to complete a booking on the direct site does not disappear: they move to an OTA. The cost of that booking — in commissions, lost customer data, and the inability to build a direct relationship with the future guest — is a cost the hotel will pay for every booking its system fails to convert.
Furthermore, a user who completes a booking with difficulty leaves a trace in the overall experience that does not stop at the online process checkout. Research has shown that the pre-stay digital experience influences the overall perception of the service received. In the era of Consumer Generated Media — forums, reviews, social networks — a user frustrated by the booking system becomes a vector of negative information that reaches a scale impossible to manage with traditional marketing tools.
These effects are measurable. This is why measuring the usability of the booking system is not an ancillary activity to be delegated to a moment of operational quiet. It is a strategic priority.
Cited sources
Nielsen & Landauer (1993) · Nielsen (2001, 2012) · Brooke (1987) · Lewis (1995) · Buhalis & Law (2008) · Gilbert, Powell-Perry & Widijoso (1999) · Good & Sambhanthan (2012) · Faulkner (2003) · Spool & Schroeder (2001) · Seffah et al. (2006) · Barnard & Wesson (2003) · Oztekina, Nikov & Zaim (2009) · Lund (2008) · Moczamy et al. (2012)