Back in may Andrew Binstock wondered "is the popularity of unit tests waning?" Andrew lists 5 indications he observed to support his conclusion:

1. Commercial tools are waning
2. Few OSS products
3. Unit testing is not taught by major Java instructors
4. Few books get published on unit testing
5. xUnit alternatives are completely invisible.

I am not sure I agree with all these observations, nevertheless,in a followup article in Java World Andrew continued to ask "Is unit testing doomed?".
Andrew says that two issues with unit testing are that they seem to offer little value and that the evangelism of unit testing in general and TDD in particular also causes antagonism (I can say from personal experience with introducing unit testing to long time C and C++ developers that, indeed, the initial responses are rather cold)

Andrew also quotes former CEO of Agitar (Jerry Rudisin) saying that "unit testing hasn't taken off as mainstream practice". A survey ran by Scott Ambler for Dr. Dobb's also found that "The more difficult practices, such as TDD and executable specifications, have lower adoption rates than easier practices such as daily scrum meetings." (although another survey by VisionOne found that within companies where agile practices are used 77% use unit testing and about 49% use TDD)

I recently read another interesting post on the subject, this time by Dan North about the "end of endotesting" explaining why the record-reply idium of most mocking framework is not intuitive and more easy to use mocking frameworks like mockito (his example) or Moq and Mocha (my examples) are better.

Now, yesterday I read Roy Osherov's post "Goodbye mocks, Farewall stubs". Roy makes similar observation on the adoption (or lack therof ) of unit testing practices. Roy believes the number one reason holding back unit testing is the learning curve associated with it and suggests we simplify the tools and terminology (and gives a example similar to Dan's mentioned above)

What we see is that both in the Java and the .NET worlds unit tests adoption hit a few snags. The way I see it there are event  additional barriers to wide adoption of unit test and TDD but I'll talk about that in part II where I'll also try to talk a little about what can/should  be done

Powered by ScribeFire.

 

Awhile ago I asked "who tests the test":

" One question I don't hear asked too much is "who tests the tests?" - after all we are writing all this additional code - if we write so many bugs in our production code that we need tests - what are the chances the test code is clean?"

I would add that this is especially true considering that the same person that writes the code also writes its unit tests

One answer I had was making sure to keep the triad of tests->code->acceptance tests. so that each two test the third. This is especially good since, usually, the acceptance/integration tests are not written by the developer who wrote the code.

One answer I got was that tests are usually simple enough to be self evident and when tests contain complex constructs you should probably test them as well.

Basically, tests can have two types of problems

• Not testing the right thing
  
• Not testing the thing right

Code+unit tests +acceptance seem to take care of the first type of problems. The claim that tests are simple should take care of the second type of problems.
However I think that it isn't always true, especially when it comes to using the tests for regression. consider for example, the following two incidents I had  where the the problems were actually with the tests:

1. We have a component that interacts with resources in different URIs, so when resources wake up they register themselves with that component. Resources register two URIs -> one for control and one for interactions. We've added a business rule that both URIs should be on the same host (naturally a new test for that was also added but that not the point) suddenly a bunch of the old tests started failing. Sure enough, I took a look at the new code, found a bug, fixed it, but the tests were still in the  red. In the end it turned out one of the Uris that is registered during the tests had an extra "L" (locallhost instead of localhost) so the tests failed with the new rule - as they should..

2. After writing a piece of code to "just work" it had to be made multi-threaded. I needed to write a test that would make sure the component can handle multiple concurrent requests. Sure, I thought, I'll just run few or even all of the other tests in parallel and be done with it. The only problem was that the other tests were written to be isolated (i.e. new instance per test) running them in parallel had no effect what-so-ever.

What can we do to help us locate problems and bugs with tests?

• remeber that Test code is code - so as you write more tests you should also refine the tests design and refactor the code accordingly. For instance in the first example above, it was easier to diagnose the problem since the faulty setup was done by a single method used by alll the failing  tests (Single Responsibility Principle @ the method level).
• Test-First - One of the reasons test-first is beneficial is because you can actually see the test fail before you implement the code to make it work. This helps you make sure that the test actually tests the behavior you want. When the code is already in place it is harder to make sure if the test works because of the change or not. In the second example running all the tests in parallel everything was still green, which triggered me to write specific tests for checking for multi-threading issues.
  

What do you do?

 

One question I don't hear asked too much is "who tests the tests?" - after all we are writing all this additional code - if we write so many bugs in our production code that we need tests - what are the chances the test code is clean?

The current answer I have is that the code, the tests and the acceptance tests all test each other so if one fails we'll spot the problem in at least one of the others. I hope that this  it is a good enough answer... :)

What do you think?

 

Having just read Doron's Yaacoby's post on some of the insights he got  from reading Tom & Lister's Peopleware, I'd thought I'd repost a couple of posts I wrote about a year ago in my Dr. Dobb's Journal blog (with a few edits):

I just finished reading Software Conflict 2.0: The Art and Science of Software Engineering, by Robert L. Glass.

This is a reprint (with few new retrospective additions) of his 1990 book. While the technology mentioned in the book is outdated, many of the author's ideas and views are still valid. The book is a collection of short articles on various subjects, and one of the more interesting articles is about the cognitive side of design.

Glass explains that research done showed that design includes the (obvious) steps of understanding the problem, decomposing it into goals and objects etc. The essence of design, however, are the mental steps taken by designers:

1. They construct a mental model of proposed solution
2. They mentally execute the model (i.e. simulate the model to see if it solves the problem)
3. If the model isn't good enough (e.g., too simple) replay the simulation to find what wrong and remodel
4. Repeat 1-3 until the model seems to solve the problem

Glass also said that people tend to start with a model that worked on a similar problem and that good designers perform this process extremely fast. As I was thinking how one can train himself to get better at performing this task, it occurred to me that I heard something similar  somewhere... While not the exact process this is very reminiscent of TDD - only TDD makes the process explicit. In a way we can say that TDD will not only help make your design better it would also  trains you to design better altogether.

Robert's book has several other views still relevant today. While it can seem odd that a 16-years old book contains relevant thoughts looking at my bookshelf I see there are quite a few other books (some of which are event older) which contain essential information and brilliant ideas that are still very relevant today. Here are just a few examples:

• Peopleware Second Edition by Tom Demarco and Timothy Lister (Second edition 1999, original 1987). Development should be centered on people and not on methods

• The Mythical Man-Month: 20th Anniversary Edition by Fred Brooks (20th anniversary edition 1995, original 1974). "No silver bullet" don't just add people to a late project it will only slow it down and many more.

• Code Complete by Steve McConnell (Second Edition 2004, original 1993). A great book on coding.
  
  
• An Introduction to General Systems Thinking (Silver Anniversary Edition) by Gerald M. Weinberg (silver Anniversary Edition 2001, original 1975). Observations on thinking processes
  
  
• About Face and About Face 2.0 (3.0 edition 2007, 2.0 Edition 2003, original 1995). UI mistakes and guidance, the second edition also details a scenario based approach to user experience design.
  

Getting back to TDD or at least the idea of test first. It seems (via Rob Keefer pomiet blog)  Gerald Weinberg talked about it more than 35 years ago in  Psychology of Computer Programming :

"By pursuing this test example to the point where he understands the problem, he will not only learn the one thing he did not know, but perhaps will learn others as well, for test programs such as this are often better learning instruments than are production programs.

As a matter of good practice, the test program should be constructed before the 'fix' is made to the production program. In the first place, there will be an all too human tendency to forget about the problem once the production program is working correctly, so we must impose a little discipline on ourselves. Possibly more important, however, is the chance that by the mere act of constructing the test case we shall discover the problem."

Anyway, with so much good advice lying around for years (11-40+) and the fact that only about 30% of the projects are successful (on-time;on budget; on scope) I think one question we should all ask ourselves is -- don't we ever learn?