Testing the Impossible: Rules of Thumb

When people say "we can't test that", they usually mean "... with a reasonable effort". They say "we can't test that because it's using a database" or "we can't test the layout of the UI" or "to test this, we need information which is buried in private fields of that class".

And they are always wrong. You can test everything. Usually with a reasonable effort. But often, you need to take a step back and do the unusual. Some examples.

So your app is pumping lots of data into a database. You can't test the database. You'd need to scrap it for every test run and build it from scratch which would take hours or at least ages. Okay. Don't test the database. Test how you use it. You're not looking for bugs in the database, you're looking for bugs in your code. Saying "but some bugs might get away" is just a lame excuse.

Here is what you need to do: Identify independent objects (which need no other objects stored in the database). Write tests for those. Put the test data for them in an in-memory database. HSQLDB and Derby are your friends. If you must, use your production database but make the schema configurable. Scrap the tables before the test and load them from clean template tables.

So you need some really spiffy SQL extensions? Put them in an isolated place and test them without everything else against the real database. You need to test that searching a huge amount of data works? Put that data in a static test database. Switch database connections during the tests. Can't? Make that damn connection provider configurable at runtime! Can't? Sure you can. If everything else fails, get the source with JAD, compile that into an independent jar and force that as the first thing into the classpath when you run your tests. Use a custom classloader if you must.

While this is not perfect, it will allow you to learn how to test. How to test your work. Testing is always different just like every program is different. Allow yourself to make mistakes and to learn from them. Tackle the harder problems after the easier ones. Make the tests help you learn.

So you have this very complex user interface. Which you can't test. Let alone starting the app takes ten minutes and the UI changes all the time and ... Okay. Stop the whining. Your program is running on a computer and for same inputs, a computer should return the same outputs, right? Or did you just build a big random number generator? Something to challenge the Infinite Improbability Drive? No? Then you can test it. Follow me.

First, cut the code that does something from the code that connects said code to the UI. As a first simple step, we'll just assume that pressing a button will actually invoke your method. If this fails for some reason, that reason can't be very hard to find, so we can safely ignore these simple bugs for now.

After this change, you have the code that does stuff at the scruff. Now, you can write tests for it. Reduce entanglement. Keep separate issues separate. A friend of mine builds all his code around a central event service. Service providers register themselves and other parts of the code send events to do stuff. It costs a bit performance but it makes testing as easy as overwriting an existing service provider with a mock up.

Your software needs an insanely complex remote server? How about replacing this with a small proxy that always returns the same answers? Or at least fakes something that looks close enough to a real answer to make your code work (or fail when you're testing the error handling).

And if you need data that some stubborn object won't reveal, use the source, Luke (download the source and edit the offender to make the field public, remove "final" from all files, add a getter or make it protected and extend the class in the tests). If everything else fails, turn to java.lang.reflect.Field.setAccessible(true).

If you're using C/C++, always invoke methods via a trampoline: Put a pointer somewhere which contains the function to call and always use that pointer instead of the real function. Use header files and macros so no human can tell the difference. In your tests, bend those pointers. The Amiga did it in 1985. #ifdef is your friend.

If you're using some other language, put the test code in comments and have a self-written preprocessor create two versions that you can compile and run.

If all else fails, switch to Python.

So... You want your code to be maintainable.

A great post if you're interested in TDD or testing in general: So... You want your code to be maintainable. by Uncle Bob. Thanks, Bob!

Testing With Databases

When it comes to testing, databases are kind of a sore spot. People like to think that "you can't test when you need a database" or "it's too complicated" or "it's not worth it." I'd like to give you some ideas what you can do when you need to test code that depends on a database. This list is sorted in the order in which I try to tackle the problem:

1. Use POJOs to store the data from the database in the real code and for the tests, create some dummy objects with test data and use them.
2. Make the database layer a plug-in of your application and replace it with a mockup for testing that doesn't need the database and which returns test objects instead.
3. Instead of connecting to the real database, get HSQLDB or Derby and use an embedded or at least local database. I prefer HSQLDB because it's smaller and starts faster (and tests should always be fast) but Derby has more features.
4. Create a second instance of the production database system on a different machine, preferably your own computer.
5. Create another instance of the real database with test data on the same machine as the real database.
6. Use database schemas to create a logical database in the real database, for example if all tables are in the schema APP, create APP_TEST and in your code, add a way to replace the schema name in the SQL statements. If you wrote the DB layer yourself, use a system property which isn't set in production. If you're using Hibernate, walk the mapping objects which are created and replace the table names after loading the production configuration. Field.setAccessible(true) is your friend.

If you can't decide, here are a few hints:

Creating two databases using schemas in the same instance can get you into serious trouble without you noticing. For example, the tests should be able to rebuild the test database from scratch at the press of a button so you can be sure in which state the database really is. If you make a mistake with the schema name during that setup, you'll destroy the real database. You might not notice you did, because the flawed statement is usually hidden under a few hundred others.

Installing a second instance on a different machine might trigger license fees or your DB admin might not like it for some reason. Also, a test database should be very flexible because you'll need to be able to drop and recreate it a dozen times per hour if you need to. Your DB admins might not like to give you the necessary rights to do that. Lastly, this means only one developer can run all the tests at any given point in time because you're all going against the same database. This is bad, really bad. More often than not, you'll have spurious errors because of that.

If you can legally get a copy of the real database on your own machine, that's cool ... until you see the memory, CPU and hard disk requirements plus a DB admin will probably hog your machine for a day or two to install it. Having to run two applications which need 1GB of RAM (your IDE and the DB) with a machine that has only 1GB of RAM isn't going to fun.

For many cases, using HSQLDB or Derby is a good compromise between all forces that pull at you. While that will make your tests slow, they will often run much faster than against the real DB. You can install these as many times you like without any license issues, fees or DB admins bothering you. They don't take much memory or hard disk space and they are under your total control.

Only, they are not the real DB. There might be slight differences, hidden performance issues and other stuff that you won't notice. Don't bother about that, though. If you can test your application, you'll find that you'll be able to fix any problems that come up when you run against the real database in little time. If you can't test your application, thought, well, you're doomed.

I strongly recommend to be able to setup the database from scratch automatically. With Derby, you can create a template database and clone that on the first connection. With HSQLDB, loading data is so fast that you can afford to rebuild it with INSERT statements every time you run the tests.

Still, test as much code as possible without a database connection. For one, any test without a DB will run 100-1000 times faster. Secondly, you're adding a couple more points of failure to your test which are really outside the scope of your test. Remember, every test in your suite should test exactly one thing. But if you can't get rid of the connection, you're testing the feature you want plus the DB layer plus the DB connection plus the DB itself. Plus you'll have the overhead of setup, etc. It will be hard to run a single test from your suite.

At the end of the day, testing needs to be fun. If you feel that the tests are the biggest obstacle in being productive, you wouldn't be the good developer you are if you didn't get rid of them.

One last thing: Do not load as much data as possible! It is a common mistake to think that your tests will be "better" if you have "as much data as possible". Instead load as little data as possible to make the tests work. When you find a bug, add as little data as possible to create a test for this bug. Otherwise, you'll hog your database with useless junk that a) costs time, b) no one can tell apart from the useful stuff and c) it will give you a false feeling of safety that isn't there.

If you don't know which data is useful and which isn't, then you don't know. Loading of huge amounts of junk into your database won't change that. In order to learn, you must start with what you know and work from there. Simply copying the whole production system will only slow you down and it will overwrite the carefully designed test cases you inserted yesterday.

Are Bad Tests Worse Than No Tests At All?

In his article "Are Bad Tests Worse Than No Tests At All?", olivstor writes:

Are the drawbacks to bad tests worse than having no coverage at all? I think the answer is that in the short term, even bad tests are useful. Trying to squeeze a extra life out of them beyond that, however, pays diminishing returns. Just like other software, your tests should be built for maintenance, but in a crunch, you can punch something in that works. It's better to have bad tests than to have untested code.

Tests are like any other code: They can go bad.

In my career, I've found that it's surprisingly hard to write good tests if you have no experience in doing so. People starting to write tests make them too complex, too long, let them have too many dependencies and they take too long to run.

If you're in such a situation, you have to face the fact that you just programmed yourself in a corner and you must spent the effort to get out of there. Tests are no magic silver bullet. They are code and follow the usual rules of coding: When it hurts, something is broken and it won't stop hurting unless you fix it.

So in this sense, I agree that bad tests are better than no tests because they tell you early that you need to fix something. That's what their core purpose is.

Management might argue that you're spending too much time on testing. I've never had a problem to sell myself to them. I usually figure that I spend 50% of my time (or more!) writing tests and 50% actual coding - and I'm still much faster than those who write code 80% of the time or more. What's more: when my code goes into production, it's is rock solid or at least easy to fix when something comes up. In 99% of the cases, the things I need to fix were those which I didn't test.

This is a positive reinforcement loop which drives me to test more and more because it stops the hurting. If your tests cost more than they seem to return, you need to fix them until you get the same positive feeling when you think about them.

Jazoon: Web Tests

In his talk, Dierk König showed ways to test a web applications with canoo webtest. There are several way to come up with a test script: Recording the user actions with a browser plug-in or by writing the script directly, for example.

He also explained the best ways to test an old application (just test what you can, look for NPE's, for example, load all pages, make sure some properties appear). Of course, since the app wasn't designed to be testable, you'll be limited in what you will be able to test.

IFRAMEs, on the other hand, are not a problem, as are AJAX requests. The main issue with AJAX is that while they are still asynchronous, the test framework has some limitations as to what it can test (permutation of request order). Testing special code for IE and FF is not a problem, the framework supports some browser bugs, too. Also, you can have special code to login or to get the app into a certain state and use this code as a kind of "subroutine" in several tests to avoid code duplication.

Internally, webtest will use htmlunit to examine the HTML returned by the app and execute the JavaScript in it. So DOM manipulations can be tested, too.

It's not really suitable for load tests, though, since it lacks the features to run concurrently on several computers at once. Use JMeter for that.

If you need to prepare the DB before the tests, use dbunit.

Testing JavaScript

If you're test mad like me, then the <script> tag in HTML was probably one sore spot for you as it was for me: How to test the damn thing? Well, now, there is a way: John Resig wrote a small script which you can source into Rhino 1.6R6 (or later; 1.6R5 won't work, though. You'll get "missing : after property id"). Afterwards, you'll have window, document, nagivation, even XMLHttpRequest!

Yes, you can actually test AJAX within unit tests, now! TDD fans, start your engines!

Unfortunately, it doesn't emulate browser bugs, yet ;-> But you can fix that. I, for example, had problems to get the code load HTML 3.2 files. Especially this code made the SAX parser vomit:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 //EN">

The fix here is to download some XHTML DTD (like XHTML 1 Strict), put it somewhere (along with the three entity files xhtml-lat1.ent, xhtml-special.ent and xhtml-symbol.ent) and change the DTD to point to the new file:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 //EN" "html/xhtml1-strict.dtd">

In my case, I've put the files in a subdirectory "html/" of the directory I start the tests from. (Hm ... shouldn't this path be raltive to the HTML file?? Well, it isn't.)

Also, the env.js supplied doesn't support forms. Here is my which fix:

function collectForms() {
    document.forms = document.body.getElementsByTagName('FORM');
    
    for (var i=0; i<document.forms.length; i++) {
        var f = document.forms[i];
        f.name = f.attributes['name'];
        //print('Form '+f.name);
        f.elements = f.getElementsByTagName('INPUT');
        
        for(var j=0; j<f.elements.length; j++) {
            var e = f.elements[j];
            var attr = e.attributes;
            
            forEach(attr, print);
            e.type = attr['type'];
            e.name = attr['name'];
            e.className = attr['class'];
            
            _elements[ f.name + '.' + e.name ] = e;
        }
        //print(f.elements);
    }
}

Note: I also had to remove the calls to toLowerCase() for the tag names (*not* the attributes!), too. Otherwise, document.body would return UNDEFINED for me. But that's because I'm stuck with old HTML; If you can convert all tag and attribute names to lowercase, then you're safe.

Lastly, the loading of the document is asynchronous. To fix this, we need a class to synchronize the Java and the JavaScript thread. That one is simple:

package test.js;

import org.mozilla.javascript.ScriptableObject;

import test.ShouldNotHappenException;

public class JSJSynchronize extends ScriptableObject
{
    public Object data;
    public Object lock = new Object ();
    
    @Override
    public String getClassName () {
        return "JSJSynchronize";
    }
    
    public Object jsGet_data() {
        synchronized (lock) {
            try {
                lock.wait ();
            }
            catch (InterruptedException e) {
                throw new ShouldNotHappenException(e);
            }
            
            return data;
        }
    }

    public void jsSet_data(Object data) {
        synchronized (lock) {
            this.data = data;
            lock.notify ();
        }
    }
    
    public Object getData() {
        synchronized (lock) {
            try {
                lock.wait ();
            }
            catch (InterruptedException e) {
                throw new ShouldNotHappenException(e);
            }
            
            return data;
        }
    }

    public void setData(Object data) {
        synchronized (lock) {
            this.data = data;
            lock.notify ();
        }
    }
    
}

ShouldNotHappenException is derived from RuntimeException. After registering that with

        jsjSynchronize = (JSJSynchronize)cx.newObject (scope, "JSJSynchronize");
        scope.put("jsjSynchronize", scope, jsjSynchronize);

in the test, I can use the new jsjSynchronize global variable in JavaScript in wondow.onload:

    public void testAddTextFilters() throws Exception
    {
        setupContext ();
        addScript(cx, scope, new File ("html/env.js"));
        cx.evaluateString(scope, 
                "window.location = 'file:///d:/devm2/globus/abs/webapp/html/testAbsSkuOutput.html';\n" +
      "window.onload = function(){\n" +
      "    jsjSynchronize.data = window;\n" +
      "};\n" +
      "", "<"+getName ()+">", 1, null);
        
        ScriptableObject window = (ScriptableObject)jsjSynchronize.getData();
        System.out.println ("window="+window);

At this point, the document has been loaded and you can access all the fields, elements, etc. Good luck!