Smartlook

What can be tested and where: client-side changes through an editor, server-side and feature experiments through SDKs, multivariate and multi-page tests, and personalisation — judged on what the documentation shows rather than on the feature grid.

0 — Not an experimentation tool; the product cannot split traffic or serve variants (analytics, heatmaps or session recording only).

3 — Client-side A/B tests built in a visual editor, with a traffic split and targeting by URL or device.

5 — A/B, split-URL and multivariate tests through a visual or code editor, audience targeting on behaviour and attributes, and either rule-based personalisation or server-side experiments through a documented SDK.

8 — Client-side and server-side experiments through documented SDKs for common languages, multi-page tests, feature-flag experiments sharing one audience and metric model with web tests, mutually exclusive experiment groups, holdouts, and personalisation that can itself be tested against a control.

10 — One experimentation programme across every surface: web, app, server and edge from the same platform, experiment interactions managed, a documented experiment lifecycle from hypothesis to archived result, and a library of past results a team can search.

Which statistics decide the winner and what protects the customer from misreading them. Scored on what the vendor documents: the method by name, how peeking and multiple comparisons are handled, and whether sample ratio mismatch is detected.

0 — No statistical evaluation; the product shows raw counts per variant and leaves the decision to the user.

3 — A results report per variant with the method named (frequentist or Bayesian) and a significance or probability-to-beat figure.

5 — A documented method with confidence or credible intervals shown per variant, and a sample-size or test-duration calculator or stated guidance on when a result may be read.

8 — Sequential testing or another documented protection against peeking, plus correction for multiple metrics or variants, sample ratio mismatch detection, variance reduction such as CUPED, and guardrail metrics that can stop a harmful test.

10 — The statistics are auditable: methodology published in enough detail to reproduce a result, the choice of method explained per use case, raw per-visitor data available for independent re-analysis, and the interface refuses to present an underpowered result as a conclusion.

Whether the test script respects §25 TDDDG and the ePrivacy rules as evidenced on the vendor's own pages: when the script runs relative to consent, what it stores on the device, whether a cookieless or consent-free mode exists, and how visitor identifiers are handled.

0 — No consent handling of any kind: the vendor states that the script runs and stores identifiers regardless of the visitor's consent.

3 — A documented integration with a consent tool or a setting that holds the script until consent, and the cookies or storage keys the script sets listed.

5 — Documented behaviour before and after consent, integration with common consent management platforms, and a published list of cookies and storage keys with their lifetime.

8 — A documented consent-pending mode that holds tracking until consent while variants can still be served, a cookieless or first-party-only option, IP anonymisation and retention of visitor data stated, and guidance on the legal basis for testing in the EU under §25 TDDDG.

10 — Built for §25 TDDDG: the default configuration stores nothing on the device without consent, a server-side or cookieless mode documented end to end, per-visitor data deletable on request, and the vendor states in writing how each mode maps to consent requirements.

The cost the client-side snippet imposes on the page it tests: blocking load, flicker of original content, script weight and the effect on Core Web Vitals — scored on what the vendor measures and publishes, not on "lightning fast".

0 — A synchronous, render-blocking snippet stated as the only way to install the product, with no flicker handling.

3 — An asynchronous or anti-flicker snippet with a timeout, described in the installation documentation.

5 — Asynchronous loading and flicker handling explained with their trade-off, the snippet delivered from a CDN, and its size or loading behaviour documented.

8 — Published performance figures including the impact on Core Web Vitals, a self-hosting or first-party-domain option for the script, per-project bundles containing only active experiments, and a server-side or edge alternative for flicker-sensitive tests.

10 — Performance is a stated commitment: measured overhead published and maintained, flicker eliminated by edge or server-side rendering as a documented path, and tooling that shows the customer what their own configuration costs the page.

Getting results and raw data out: integration with analytics and tag management, export of visitor-level results, warehouse-native analysis, and an API — because an experiment result that cannot be checked in the customer's own data is a claim, not a finding.

0 — A closed dashboard: the vendor states that results cannot be exported or sent to another tool.

3 — Results exportable as a CSV file or report, and an integration with at least one named analytics tool.

5 — Integrations with common analytics and tag managers (Google Analytics, Adobe Analytics, Google Tag Manager or similar), export of results, and a documented API or webhooks.

8 — Visitor-level raw data export or streaming to a data warehouse, warehouse-native analysis on the customer's own metrics, CDP integration for audiences, and an API for managing experiments and reading results with stated limits.

10 — The platform treats the customer's warehouse as the source of truth: metrics defined once and computed there, full historical experiment data exportable in open formats, and a versioned API a team can build its own programme tooling on.

Where visitor data is processed and stored and who the contracting entity is. Independently sourced by the sovereignty pipeline; weighted higher here than in categories that hold only the customer's own data, because the script runs on every visitor to the customer's site and their behaviour is what the platform records.

0 — Vendor, contracting entity and processing of visitor data all shown to be outside the EU, with no transfer safeguard stated.

3 — A named contracting entity and a privacy policy or data processing agreement; EU data residency offered as an option or add-on, or the hosting location not stated.

5 — EU hosting or EU processing of visitor data stated as standard, together with an EU contracting entity or a published subprocessor list, and a data processing agreement offered.

8 — EU contracting entity, EU hosting on named infrastructure including the delivery of the snippet, a subprocessor list published with locations, and a data processing agreement covering the visitor data the script collects.

10 — Sovereign end to end and evidenced: vendor, entity, hosting, snippet delivery and every subprocessor European, certification published, and no visitor data reaching a non-EU party at any point.

A category priced by traffic — monthly tracked users, visitors or impressions — where the tier a site lands in depends on numbers the buyer has to estimate. Whether a buyer can compute the real annual cost including traffic limits, overage, server-side or personalisation modules and seats — from public pages alone.

0 — No public prices at all; every tier is a sales conversation.

3 — A public "from" price, a free tier or one priced tier, while the other tiers or the traffic metric behind the price are not stated.

5 — Prices public per tier with the billing period and the traffic metric (tracked users, visitors or impressions) and its limits stated; modules such as server-side SDKs or personalisation, or overage, may still be "contact sales".

8 — Every tier and module priced publicly with the traffic metric defined, limits, overage rates, seats, minimum term and VAT treatment stated.

10 — Complete price computability: annual invoice derivable for a given traffic volume, set of modules and team size, with overage and every add-on published.