To evaluate a database honestly, test the work your application must perform under conditions it will actually face. A polished demonstration is a useful introduction, but it is not your acceptance test.
Write the Decision Before Running the Test
State why you are considering a change. You might need lower operating effort, more capacity, a particular availability model, or a missing capability. Without that purpose, every interesting feature becomes a reason to extend the evaluation.
Define acceptance criteria for correctness, response time, concurrency, recovery, and cost. Separate requirements from preferences. A faster report does not compensate for a missing transaction guarantee the application depends on.
Include the current system as a baseline where practical. A new product should be compared with a reasonably configured alternative, not a neglected example. Record the tuning effort allowed to each candidate.
Shape the Data Like the Workload
Match the distribution of keys, row widths, NULLs, and historical growth. Uniform random rows often hide skew and hot-key contention. Preserve those statistical properties without exposing real customer information.
WITH Digits AS
(
SELECT n FROM (VALUES (0),(1),(2),(3),(4),(5),(6),(7),(8),(9)) AS d(n)
), Numbers AS
(
SELECT a.n + 10*b.n + 100*c.n AS n
FROM Digits AS a CROSS JOIN Digits AS b CROSS JOIN Digits AS c
)
SELECT n AS EventId,
CASE WHEN n < 800 THEN 1 ELSE 2 + n % 20 END AS TenantId,
REPLICATE('x', 20 + n % 180) AS Payload
INTO #EvaluationData
FROM Numbers;This SQL Server example defines synthetic skew and variable row width. It does not claim to reproduce a real application. Adapt equivalent data generation for each candidate while preserving the same logical dataset.
SELECT TenantId, COUNT_BIG(*) AS rows_per_tenant,
AVG(CONVERT(decimal(12,2), DATALENGTH(Payload))) AS average_payload_bytes
FROM #EvaluationData
GROUP BY TenantId
ORDER BY rows_per_tenant DESC;Verify the generated shape before timing queries against it. Include uncommon but important cases, such as the largest tenant. Average behavior can hide the condition that dominates production support.
Measure Complete Business Operations
Use the actual mix of reads, writes, joins, and transactions. Include client serialization and network paths when the requirement concerns user response time. A database-only measurement answers a narrower question.
SET STATISTICS IO ON;
SET STATISTICS TIME ON;
SELECT TenantId, COUNT_BIG(*) AS event_count
FROM #EvaluationData
WHERE TenantId = 1
GROUP BY TenantId;
SET STATISTICS TIME OFF;
SET STATISTICS IO OFF;The SQL Server messages provide engine-side evidence for this example. Collect equivalent metrics from other products using their supported tools. Do not compare differently defined counters as if their labels guarantee equivalence.
Measure throughput and latency under representative concurrency, including tail latency and errors. Keep the request mix and arrival pattern documented. A system that queues requests can look efficient while users wait longer.
Check Semantics and Compatibility
Verify transactions, isolation, collation, date handling, and numeric behavior with explicit examples. Similar SQL syntax can hide different defaults. Correct results come before a performance ranking.
List application features that need rewriting, including stored procedures, drivers, scheduled work, and operational scripts. Test the migration path for representative objects. A database replacement affects more than table storage.
SELECT SERVERPROPERTY('ProductVersion') AS product_version,
SERVERPROPERTY('Edition') AS edition;
SELECT name, compatibility_level, collation_name,
is_read_committed_snapshot_on
FROM sys.databases
WHERE database_id = DB_ID();Capture corresponding configuration evidence for every candidate. Preserve the exact schema, indexes, and settings used in each run. Otherwise, a later repeat may be a different test wearing the same name.
Include Failure and Recovery
Test interrupted requests, lost connections, and the supported failover process in a controlled environment. Check what the client sees and whether retries duplicate effects. Confirm the committed data afterward.
Rehearse backup and restore or the service's recovery mechanism. Measure the recovery process yourself and record its limitations. A vendor's availability description is not evidence that your application resumes correctly.
Also test the ordinary operational work your team performs each week. Monitoring, access changes, patching, and troubleshooting contribute to the real cost. Include licenses, storage, network transfer, support, and staff effort.
Publish a Decision With Its Limits
Keep scripts, configuration, data-generation rules, and raw observations with the evaluation. Report repeated runs and meaningful variation rather than selecting the most flattering result. Explain any feature or failure mode left untested.
Choose based on the stated requirements and accepted tradeoffs. A candidate can be excellent and still be wrong for this workload. An honest evaluation ends with a defensible decision, not a universal winner.
A database evaluation is not a product contest, it is evidence for one workload's decision.
This post was rewritten from scratch in September 2026. The original, published on 2012-09-02, was a short announcement about something that no longer exists. The address is the same, the subject is now something worth keeping.
Published by Pinal Dave on SQLAuthority. More of my work at pinaldave.com.
