In some ways, this has become the Holy Grail of my automated testing existence.
In the environment that I am currently testing, I have four distinct places where I can run our code. On a development system, on a demo machine, on a staging environment, and in production. I used to maintain four different systems, and those four different systems would float and require tweaking. In an attempt to try to get some sanity and reduce wasted effort, I decided that a better, more Don’t Repeat Yourself (DRY) approach was to figure out how to move my scripts to a system that allowed me to create one set of scripts, and then optimize my environments and my test data so that those scripts could be run anywhere. This is the core behind my approach of Write Once, Test Anywhere (WOTA).
Now, when I say “my approach”, I certainly don’t mean that I’m the first one to think to do this, not by a long shot. I’m also having to make interesting tweaks to various config files so that I can effectively do this. Unlike the development team’s tests, which really only have to focus on one environment to verify that their tests work (note, that’s not a dig, it’s a reality), mine have to work effectively on four different environments. This also helps in the sense that it allows me to see if Acceptance Tests and the methods used to write those tests (and their surrounding features) actually carry through as we apply them to consistently more complex systems. As we get closer to production, we abstract away from running tests on a dedicated workstation and a dedicated machine, and instead start calling on multiple machines that are structured in a cluster, with caching, external and distributed databases, load balancing, etc. The test themselves don’t change, but often, we see different behavior with the same tests depending on the system we are testing.
If this piece is a blinding flash of the obvious, well, it took me a bit of time to figure out a good way to do this. If you are still testing with different bases for each environment, seriously, consider looking at ways you can implement WOTA in your tests.