A/B Testing
A/B Testing: Server Configuration & Implementation
A/B testing, also known as split testing, is a method of comparing two versions of a webpage or application against each other to determine which one performs better. This article outlines the server-side configuration considerations for implementing A/B testing within a MediaWiki environment. This assumes a relatively high-traffic deployment where direct database manipulation for A/B testing is undesirable. We'll focus on using a reverse proxy/load balancer for traffic splitting.
Understanding the Concepts
Before diving into configuration, let’s clarify key terms:
- Control Group (A): The existing, unmodified version of the MediaWiki site.
- Variant Group (B): The modified version of the MediaWiki site with the changes being tested.
- Traffic Split: The percentage of users directed to each version (e.g., 50/50, 80/20).
- Sticky Sessions (Session Affinity): Ensuring a user consistently sees the same version (A or B) throughout their session. This is *crucial* for accurate results.
- Conversion Rate: The metric used to measure success (e.g., article views, edits, user registrations).
- mediawiki-a.example.com: The control server.
- mediawiki-b.example.com: The variant server.
- $wgSitename: Must be identical on both servers.
- $wgServer: Must be consistent with the server's hostname.
- $wgSessionCacheType: Should be configured to a cache that is *not* shared between the A and B servers. Using a separate database table or a dedicated caching system (like Memcached or Redis) is essential. Otherwise, sticky sessions will be ineffective.
- $wgCookiePrefix: Consider using a unique prefix for each instance to avoid cookie conflicts, though this is less critical with the load balancer handling session affinity.
- Web Analytics: Use a tool like Matomo or Google Analytics to track conversion rates and user behavior on both versions. Ensure the analytics code is deployed correctly on both servers.
- Server Monitoring: Monitor CPU usage, memory usage, and response times on both MediaWiki servers and the load balancer using tools like Nagios, Zabbix, or Prometheus.
- Load Balancer Logs: Analyze load balancer logs to verify traffic distribution and identify any issues.
- MediaWiki Logs: Review MediaWiki's error logs for any unexpected behavior.
- **Non-Sticky Sessions:** Users switching between A and B will invalidate your results. Double-check your load balancer configuration.
- **Inconsistent Configurations:** Any difference between the servers (besides the test change) can skew results. Use configuration management.
- **Caching Issues:** Aggressive caching can mask changes. Ensure caches are properly invalidated or configured to respect the A/B test. See MediaWiki caching for details.
- **Analytics Errors:** Incorrectly implemented analytics code will provide inaccurate data.
- **Small Sample Sizes:** A/B testing requires sufficient data to reach statistical significance. See Statistical Significance for more information.
- MediaWiki Configuration
- Load Balancing
- Reverse Proxy
- Performance Tuning
- Database Replication - For minimizing impact of testing on primary database.
- Telegram: @powervps Servers at a discounted price
Architectural Overview
The core of our A/B testing setup will rely on a reverse proxy or load balancer (like HAProxy, Nginx, or Apache HTTP Server with appropriate modules). This sits in front of multiple MediaWiki instances. Each instance serves either the control or the variant version. The reverse proxy distributes traffic based on the defined split.
Important Note: The MediaWiki instances themselves should be as identical as possible *except* for the changes being tested. This includes the same extensions, configurations (excluding those related to the A/B test), and database schema. Using a configuration management system like Ansible or Chef is *highly* recommended.
Server Configuration Details
We will use a hypothetical setup with two MediaWiki servers:
These servers will be behind a load balancer, `lb.example.com`.
Load Balancer Configuration (HAProxy Example)
Here's a simplified HAProxy configuration snippet demonstrating a 50/50 split with sticky sessions:
``` frontend http-in bind *:80 mode http option httpclose option forwardfor option httpchk
# Cookie name for sticky sessions cookie JSESSIONID
# Define the backends default_backend wiki-a balance roundrobin
backend wiki-a mode http server wiki-a-1 mediawiki-a.example.com:80 check cookie server wiki-a-1
backend wiki-b mode http server wiki-b-1 mediawiki-b.example.com:80 check cookie server wiki-b-1
# ACL to route 50% of traffic to wiki-b acl is_variant hdr(cookie) -i wiki-b-1 use_backend wiki-b if is_variant ```
This configuration uses cookies for sticky sessions. The `JSESSIONID` cookie will be set by HAProxy to ensure users stick to the same server. Adjust the `cookie` names as needed to avoid conflicts with existing MediaWiki cookies.
MediaWiki Configuration
Minimal changes are required on the MediaWiki servers themselves. Ensure the following:
Technical Specifications - Server A (Control)
| Parameter | Value |
|---|---|
| Server Hostname | mediawiki-a.example.com |
| Operating System | Ubuntu 22.04 LTS |
| Web Server | Apache 2.4 |
| PHP Version | 8.1 |
| MySQL/MariaDB Version | 10.6 |
| MediaWiki Version | 1.40 |
| RAM | 8 GB |
| CPU Cores | 4 |
Technical Specifications - Server B (Variant)
| Parameter | Value |
|---|---|
| Server Hostname | mediawiki-b.example.com |
| Operating System | Ubuntu 22.04 LTS |
| Web Server | Apache 2.4 |
| PHP Version | 8.1 |
| MySQL/MariaDB Version | 10.6 |
| MediaWiki Version | 1.40 |
| RAM | 8 GB |
| CPU Cores | 4 |
Technical Specifications - Load Balancer
| Parameter | Value |
|---|---|
| Hostname | lb.example.com |
| Software | HAProxy 2.6 |
| Operating System | Ubuntu 22.04 LTS |
| RAM | 4 GB |
| CPU Cores | 2 |
Monitoring and Data Collection
A/B testing is useless without proper monitoring. Integrate the following:
Common Pitfalls & Troubleshooting
Further Reading
Intel-Based Server Configurations
| Configuration | Specifications | Benchmark |
|---|---|---|
| Core i7-6700K/7700 Server | 64 GB DDR4, NVMe SSD 2 x 512 GB | CPU Benchmark: 8046 |
| Core i7-8700 Server | 64 GB DDR4, NVMe SSD 2x1 TB | CPU Benchmark: 13124 |
| Core i9-9900K Server | 128 GB DDR4, NVMe SSD 2 x 1 TB | CPU Benchmark: 49969 |
| Core i9-13900 Server (64GB) | 64 GB RAM, 2x2 TB NVMe SSD | |
| Core i9-13900 Server (128GB) | 128 GB RAM, 2x2 TB NVMe SSD | |
| Core i5-13500 Server (64GB) | 64 GB RAM, 2x500 GB NVMe SSD | |
| Core i5-13500 Server (128GB) | 128 GB RAM, 2x500 GB NVMe SSD | |
| Core i5-13500 Workstation | 64 GB DDR5 RAM, 2 NVMe SSD, NVIDIA RTX 4000 |
AMD-Based Server Configurations
| Configuration | Specifications | Benchmark |
|---|---|---|
| Ryzen 5 3600 Server | 64 GB RAM, 2x480 GB NVMe | CPU Benchmark: 17849 |
| Ryzen 7 7700 Server | 64 GB DDR5 RAM, 2x1 TB NVMe | CPU Benchmark: 35224 |
| Ryzen 9 5950X Server | 128 GB RAM, 2x4 TB NVMe | CPU Benchmark: 46045 |
| Ryzen 9 7950X Server | 128 GB DDR5 ECC, 2x2 TB NVMe | CPU Benchmark: 63561 |
| EPYC 7502P Server (128GB/1TB) | 128 GB RAM, 1 TB NVMe | CPU Benchmark: 48021 |
| EPYC 7502P Server (128GB/2TB) | 128 GB RAM, 2 TB NVMe | CPU Benchmark: 48021 |
| EPYC 7502P Server (128GB/4TB) | 128 GB RAM, 2x2 TB NVMe | CPU Benchmark: 48021 |
| EPYC 7502P Server (256GB/1TB) | 256 GB RAM, 1 TB NVMe | CPU Benchmark: 48021 |
| EPYC 7502P Server (256GB/4TB) | 256 GB RAM, 2x2 TB NVMe | CPU Benchmark: 48021 |
| EPYC 9454P Server | 256 GB RAM, 2x2 TB NVMe |
Order Your Dedicated Server
Configure and order your ideal server configurationNeed Assistance?
⚠️ *Note: All benchmark scores are approximate and may vary based on configuration. Server availability subject to stock.* ⚠️