• Post Reply Bookmark Topic Watch Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
This page was migrated from a javaranch.com.jsp

Evil Unit Testing
by Paul Wheaton




As a contractor, I have now appeared at no less than six different companies beaming with pride that they have
"unit testing"!  And, combined with that great pride is concern that this "unit testing" is turning out to be a
hassle.  "Other people talk about how great unit testing is, but this is really becoming painful.  The tests take 45
minutes to run, and for every little change to the code, we break about seven tests!"



What these folks have is a massive pile of functional tests.  They fell into the popular trap of thinking that because
they run the tests with JUnit, they must be unit tests.  90% of their problems could have been resolved with just a
wee bit of vocabulary.



Software Testing Vocabulary



Unit Test: The smallest amount of testable code.   Often a single method/function,
   sans the use of other methods or classes.  Fast!  Thousands of unit tests can run in ten seconds or less!
   A unit test NEVER uses:

   



         
  • a database
         
  • an app server (or server of any kind)
         
  • file/network I/O or file system;
         
  • another application;
         
  • the console (System.out, System.err, etc.)
         
  • logging
         
  • most other classes (exceptions include DTO's, String, Integer, mocks and maybe a few others).

       


   Unit tests are almost always part of a regression suite.



Regression Suite: A collection of tests that can be run all at once.  An example would
   be that all tests put in a certain directory would all be run by JUnit.  A developer could run a unit test
   regression suite 20 times per day.  Or, they might run a functional test regression suite twice a month.



Functional Test: Bigger than a unit and smaller than a full component test.
Usually exercising several methods/functions/classes working together.  Permitted to take its sweet time:
hundreds of tests could take hours to run. Most functional tests are part of a functional test regression suite.
   Usually run from JUnit.



Integration Test: Testing two or more components working together.
Sometimes part of a regression suite.



Component Test: Running one component by itself.  Frequently done by QA, managers,
   XP customers, etc. This sort of test is not part of a regression suite and it is not run by JUnit.



Component Acceptance Test (C.A.T.): A component test run in front of a crowd of people
   as part of a formal process.  The people in the room collectively decide if the component has met required criteria.



System Test: All components being run together.



System Acceptance Test (S.A.T.): A system test run in front of a crowd of people
   as part of a formal process.  The people in the room collectively decide if the system has met required criteria.



Stress Tests: Another program loads a component, some components, or possibly the entire
   system.  I have seen some small stress tests worked into some regression functional tests - a pretty smart way to test
   some concurrency code!




Mock: A brain dead piece of code used in unit and functional tests to make sure that
   a piece of code you are attempting to test does not use some other bit of production code.  A Mock class typically
   overrides all of the public methods in a production class and is inserted somewhere that might try to use the
   production class.  Sometimes a Mock class implements an interface and replaces production code that implements the same
   interface.



Shunt: Kinda like a mock class that extends production code, only the purpose is NOT to
   override ALL of the methods, but just enough so you can exercise some production methods, while mocking the rest of the
   production methods.  Especially useful if you want to test a class that might try to use I/O.  Your shunt can override
   the I/O methods while testing the non I/O methods.







And now for your opportunity to tell the author he's a nitwit that doesn't know the first thing about this stuff in the coderanch thread unit testing vocabulary.







To clarify the number one struggle shops have with "unit testing":



The Trouble With Too Many Functional Tests

Don't get me wrong.  Functional tests have great value.  I think apps that are well tested would have a regression suite of
functional tests and a collection of non-regression functional tests.  Usually, for every pound of production code, I
would like to see about two pounds of unit tests and two ounces of functional tests (a little bit goes a long ways).
   The problem I see in too many shops
is zero unit tests and a pound of functional tests.



The following two images demonstrate classes using classes that use classes.  There are functional tests that exercise
these classes working together.  Fixing a bug in one class breaks many functional tests ....







I have seen this happen many times.  In one case, a small change broke 47 tests.  Meetings were held to decide if the
bug should just be left in the code!  In the end, it was decided to turn off all tests until the time could be set aside
to fix all of the tests.  Months passed.  Things got mighty stinky ....



The solution is to use unit tests instead:







The result is that the project is more flexible.



Functional Test Misnomers

"By writing only functional tests, I write less test code and exercise more production code!"  True!  But at
   the price of making your project brittle.  Plus, there are some finer points of your application that will be a lot
harder to test without using unit tests.  The best coverage and flexibility can only be found through a mix of unit and
   functional testing that is heavy on unit testing and light on functional testing.




"My business logic is all of these classes working together, so testing just one method is pointless."
I'm suggesting
that you test all of the methods - just separately.  Further, I am not suggesting that you have zero functional tests -
they have their value.




"I don't mind if my unit test suite takes a few minutes to run."
 But do the other people on your team mind?  Does
your team lead mind?  Your manager?  If it take a few minutes instead of a few seconds, do you still run the full suite a
dozen times a day?  At what point do people stop running the tests at all?




"A unit test is anything run by JUnit."
It is true that in our industry, the term "unit test" is subjective.  I think
that what I'm expressing here is the most popular interpretation of the term "unit test".



Check out the forums at The Big Moose Saloon for this thread about functional tests vs. unit tests.






Unit Testing Mock Basics

Here is a really obvious form of unit testing.  Trying all sorts of wacky stuff on one method that doesn't depend on
other methods.





Sure, any putz can unit test that sort of thing.  But most business logic uses other business logic:







   Not only is this calling other business logic, it's calling an application server!  Possibly across a network!
   Thousands of these are gonna take more than ten seconds.  Plus, changes in the EJB stuff could break my tests here!
   So a mock object needs to be introduced.




   If I could just mock out all of the EJB stuff, I'd be sitting pretty.  Hmmmm ....   If the code were to somehow
   get my mock FarmEJBRemote, I'd be in fat city.




   First, to create the mock.   If FarmEJBRemote were a class, I would extend it and override all the methods.  But since
   it happens to be an interface, I'll just make a fresh class and implement all the methods:






   The mock is dumb.  Really dumb.  It just carries data between my unit test and the code I am trying to exercise.



Does this class make you .... uncomfortable?  It should.  There are two things about this sort of class that bugged
me the first time I was exposed to it:  The class attributes are not private, and they have underscores in them.  The first
   time I saw a mock object like this
   I was told "Your unit test code doesn't go into production, so it can cut a few corners."  I dunno ... I want to only write first class
   code all the time!  Not even an hour had passed and I needed to mock java.sql.Connection.  40 methods!  Getters and setters for every
   parameter, return value and counter for every method?  ....  hmmmm .... thinking this through a little ....
   the reason we make the attributes private is for the sake of encapsulation - to hide how things are done on the
   inside so that we can change our business logic later without breaking a bunch of stuff that decided to tap into our
   innards.  But that doesn't really apply to a mock, does it?  By definition, a mock has zero business logic.  Further,
   it doesn't really have anything that it didn't just copy from somebody else.  All mock objects everywhere could
   easily be 100% generated at build time!  ...  So I sometimes still feel a little queasy about this, but in the end
I always end up re-convincing myself that this is the best way.  So it is still "first class code all the time" - it just
   smells a little off.  But it smells better than if I did it the other way.



Now I need to get the code to take in my mock object instead of firing up some application server.  So here's that snippet
of code again and I've highlighted the line of code where I want to use my mock.





First, let's separate that out a bit from the rest of the herd ...





This is gonna hurt a little ....  I will now extend my production class and override getRemote() so I can foist my
mock into this operation.  So I'll need to make one little change ...





If you are a good OO engineer, you should be hopping mad right about now!  Oh sure, violating encapsulation in unit
test code is mighty uncomfortable, but violating encapsulation in production code JUST AIN'T DONE!  (in case you missed it,
   I took out the keyword "private", making the method "package private" - now, anything in the same package can
   see that method)   Again, a long winded explanation might help smooth things over a bit. I'm going to save that
for the java forums and say for now: be ever vigilant about first class encapsulation in your production code ...
but ... once in a while ... you might consider trading a dollar's worth of encapsulation for twenty dollars worth of
testability.  And to salve your pain/shame a bit, you might add a comment:





Now I just write the class so I can return the mock:





Note the weird name:  "Shunt".  I'm not certain, but I think this word comes from electrical engineering/tinkering
and refers to using a wire to temporarily complete a circuit.  At first it sounded really stupid to me, but after a while
I got used to it.



A shunt is kinda like a mock, except you don't override all of the methods.  This way, you mock some methods
while testing others.  A unit test could end up with several shunts all overriding the same class, each testing different
parts of the class.  Shunts are usually inner classes.




   Now for the grand finale!  The actual unit test code!





Now that the basic framework is in place, I just need to add lots of assertions.



TestFarmServlet vs. FarmServletTest:  You gotta join one camp or the other.  The folks from the latter camp make a
   damn good point:  "FarmServletTest" sounds more like a noun and thus more OO.  I am in the former camp.
   I have become addicted to my IDE and enjoy the way it can complete class names
for me.  When I get to the point that I have a rich suite of tests, and my test class names all end with "Test",
   then my IDE makes twice as many suggestions as I want it to.  When my test class names all start with "Test",
my IDE makes exactly the right number of suggestions.




Discuss this stuff in the forum thread called Creating Unit Tests (with mocks and shunts!)









To get an idea of how a project might keep all of this stuff straight, take a gander at a



Sample Directory Structure

This is the sort of directory structure I see all the time:





Folks put their unit test stuff in the same package as the production code they are testing - BUT! in a different
directory structure.  So if you are testing com.javaranch.Str.java found in src/java/com/javaranch/Str.java, you
might have a test class com.javaranch.TestStr.java found in test/unit/com/javaranch/TestStr.java.












Fin

This is just a quick overview of this topic.  But once you have your head wrapped around this, you're ready for

   



         
  • continuous integration
         
  • ant
         
  • improving your layer interfaces while not sacrificing testability (static methods using inner implementation classes)
         
  • testing one class or one test method from your IDE
         
  • test first development (sometimes called test driven development) - should you do it more, than, say 30% of the time?
         
  • Extreme Programming!
         
  • Fitnesse

       





Discuss lost more of this sort of thing in the testing forum at the big moose saloon.

 
    Bookmark Topic Watch Topic
  • New Topic