Writing and doing tests

Testing: Why Bother To Do It At All?
Consider the animal fable story of the Hare and Tortoise. This story can be used as an analogy to the benefits vs. costs of unit testing vs. not doing testing. The Tortoise can be likened to a programmer who does both testing and coding, while the Hare can be likened to a programmer that only codes. The Hare and Tortoise have a race, and a misconception that some programmers seem to have, as mentioned in "Learning To Love Unit Testing: Improving development through the practice of testing before you code", is that coding takes less time than doing both coding and testing. If this were true, one would expect the Hare to win. Sure enough, in the beginning, he easily overtakes the Tortoise, but then his activity level drops to zero because he slows down and rests for a long time. The Tortoise, in spite of having a relatively slow start, keeps moving along steadily and eventually takes over the Hare, thereby winning the race. In other words, while the Tortoise initially had lower productivity than the Hare, he is able to overcome every pitfall stopping him from reaching his goal. This steadiness, vs. the huge slump in productivity attributed to not doing testing (like the Hare) is shown in Figure 1.1: Comparison of Paying-As-You-Go vs. Having a Single Testing Phase, p.9.

Java Portal Development: 
Check out Java Portal Coding Standards for more info about coding and testing, with regards to Java Portal Development. Provides additional info about unit testing, integration testing, and acceptance testing, along with general team communication practices with regards to checking in code, updates, etc.

The Pas Development Environment:
In the Pas Development Environment, within the project Pas Learner Runtime, the two main sub-folders in which source code files are located are the pas-common-apps and the pas-learner-runtime folder itself. The main class files that are built are located within pas-common-apps, and the rest of the source code files that are necessary are located within pas-learner-runtime. For the purposes of this project, the SVN repository is the one to which the files are committed. In this main repository, the source code files exist. Synchronizing of files helps maintain version control (also discussed in Pragmatic Project Automation: How to Build, Deploy, and Monitor Java Applications), and automated tests to be run can be very important prior to actually committing code to the repository.

Unit Testing:

1. What is unit testing?
Unit testing, as the name implies, is determining whether a unit, or portion (of code), executes properly. In other words, it involves testing a particular method or part of a class, rather than the overall functionality of that class. After all, perhaps any problems in the code can be pinpointed to a faulty line in a method. Test cases are designed to setup the unit tests to be run. It is possible that certain test cases may cause a method or function to fail. Pragmatic Unit Testing in Java with JUnit: Chapter 1, mentions the need to determine what one actually wants the code to do, and whether it actually does that or not, prior to and while designing the appropriate unit tests. Unit testing involves an all-or-none evaluation process, in that the testing passes (in which all unit tests pass), or fails (in which one or more unit tests fail).

2. Benefits of doing unit testing prior to coding
Pragmatic Unit Testing in Java with JUnit: Chapter 1 mentions several advantages for doing testing prior to coding. When testing prior to coding, there is no need to step through many lines in the debugger and chase down difficult-to-find bugs. The code can be integrated properly (although it may not necessarily do so), but if it does, pinpointing and fixing (debugging) the source of the problems can be easier. Moreover, it is easier to catch problems relatively early on, rather than in a huge avalanche at the end of the coding process. Also, doing unit testing helps to avoid the problem of collateral damage, in which fixing one problem causes another one to arise, similar to the "Whac-a-Mole" effect (in which one cannot keep up with the demands of striking each and every mechanical mole as it pops up, so that in spite of one's best efforts, one has to concede defeat in the end). Another advantage of doing unit testing + coding, as mentioned in Java Extreme Programming Cookbook, Section 2.3, is that it can help one to better focus on what the code one looks at, writes, or modifies is actually supposed to do; Section 2.3.3 mentions that unit testing helps clarify the code that was written, which can be particularly helpful for the beginning unit tester or someone that is not immediately familiar with what the code is actually supposed to do. Clarification of code can become especially important when designing complicated methods and/or classes. As mentioned in "Learning to Love Unit Testing: Improving development through the practice of testing before you code", unit testing not only helps to clarify the development process, but the code can be well integrated into the whole system, which can greatly help when testing the source code at the application level. Also, it is a good idea for a source code file to have associated unit tests with it, before committing that file (with the latest changes) along with the associated tests to the repository.
Unit testing, in conjunction with coding, has been shown to be quite valuable in the long run. As mentioned in Pragmatic Unit Testing in Java with JUnit, Chapter 1, although initially the process of doing both unit testing and coding may be slow compared to solely doing coding, it will be more time-effective in the long run. Also, it is worth noting that doing testing prior to coding not only helps the programmer, but others on the development team as well, as they will be better able to understand what the code is actually supposed to do and determine the extent to which it accomplishes what it ought to. (Please consult the attachment on testing tradeoffs for more information).

3. Costs of only doing coding (no testing)
The programmer will just have to step through the debugger and get rid of every bug, which will become an increasingly arduous and difficult task as the code progresses further into the development cycle. In other words, even if every bug caught is eventually fixed, the process and effort involved can easily turn out to be much more than if one actually conducted testing along with coding.

4. Re-iteration: Why are unit tests so important?
Some of the benefits of doing them were discussed earlier. But are they always necessary? Well, more is better than less. What if the code is fairly trivial and can be clarified with some comments instead of actual unit testing? It is probably safe to assume that what may be obvious to one person may not be to another, so unit tests can help bring along the most novice programmers for the ride.

5. How are unit tests written?
In Java Extreme Programming Cookbook, Section 2.3.5, the process of testing bugs leads to a test-code-test-code cycle, where "test" refers to writing and running a unit test. Without unit tests, it could very well be a nightmare trying to trace down elusive bugs in one's program, whether one actually wrote that code or someone else did. On the other hand, it can be a good practice to even write tests prior to doing the actual coding, and continue to write tests as the code development progresses.
Section 2.3.6: How do you write tests, describes in greater detail about the intricacies of unit testing. First of all, one should know what the method or class one wishes to test is supposed to do, and the expected outcome from running it. Determining these things helps with forming appropriate testing cases upon which to create tests. If a test fails, an error message should show up to indicate this, along with what the error actually is. A summary report, stating how many tests passed vs. failed, along with the status of each particular test and errors (for any failing test, if any), can also be helpful.
Some important test cases to consider (based on personal experience and discussion with Hiroki):
Important boundary conditions (e.g. ends of an array, such as 0 and length-1 indices).

• Illegal inputs (e.g. passing a number to a method accepting a string)
• passing null or full data structures into a method using that structure
• strings with odd formatting
• many others

As mentioned in section 2.3.8: Testing Improves Design, unit testing helps with planning ahead, to provide better direction for writing the code than otherwise. Also it probably helps to test relatively short methods is probably generally a good practice, because it makes the code easier to read and test than otherwise.

6. Who does unit testing?
This question was raised by Java Extreme Programming Cookbook, Section 2.3.2. As that section explains, the programmers write unit tests, along with doing the actual code development. This makes sense, as it is probably safe to assume that they will know what is actually going in the code, which is needed to write good tests (even if one writes the tests prior to actually coding). Also, pair programming helps with testing as well, because more unit tests can be written, and different people may catch different bugs, think of different test cases to run, or focus on different parts of a particular source code file. In short, the more thorough the unit testing, the better one can ensure that the code actually works and does what it is supposed to do.

JUnit Testing:
In "Learning to Love Unit Testing: Improving development through the practice of testing before you code", JUnit test frameworks are described.

Within a JUnit test framework, a test file can consist of a Java class with the corresponding methods to test. This is probably where looking at and trying out the interface itself can be useful so that one does not have to worry about diving into tons of code with the methods to be tested in the actual source code files. The article does mention that it would be good not to rely on a specific template (interface) to test against. Rather, try to test without this dependence. Also, putting in integration tests may help too. Even if errors show up, they're easier to handle with a test file rather than looking through the entire source code file, so that one doesn't have to look at many errors from all over the place.

Note: for doing JUnit tests, one must have a copy of JUnit downloaded. For more info, please go to JUnit

Designing Unit Tests in PAS:
Writing a basic unit test (source code): Please refer to Chapter 4 in Java Extreme Programming Cookbook. The basic steps involve initializing/instantiating the class, using the setup function to add object variables, writing the actual code, and then tearing down (basically, removing) what is no longer needed so that the memory resources are not tied up.

My hunch is that unit testing can help with piecing together and testing the various parts of PAS (namely, activities and steps, particularly those steps that involve extending of other steps/Java classes).
As to how to actually go about designing a unit test, one should be sure to include the following statements at the top of the testing java file.

package junit.framework;
import junit.framework.TestCase;

Tests Should Be Automatically Built As Much As Possible:
Making more regular, relatively frequent scheduled automated builds can help minimize the chances of detecting and correcting errors before it is too late and one is buried in a huge pile of errors. As mentioned in Pragmatic Project Automation: How to Build, Deploy, and Monitor Java Applications, some ways of performing automated testing (without the need for manual command-line input) are scheduled automation and triggered automation. Scheduled automation involves the automatic execution of a command-line statement, so that a build can be scheduled to run automatically. Triggered automation can be considered special-case automation; that is, if a certain occurrence happens, such as right after a source code file (or otherwise) is committed to the repository, a check is made to ensure that the automated builds don't fail. For any resulting failures, something like red flags along with an error log would be maintained to point out what the problems are (if any). If the automated tests result in failures, perhaps it might be worth not only looking at why they fail, but also consider performing consistent tests to be able to compare one set of test results to another. Commanded automation can help in this, in terms of performing builds that were consistent from one testing round to the next.

The same chapter also mentions three important benefits of performing automated testing: accuracy, consistency, and repeatability. Consequently, the reliability and validity of the testing outcome can be ensured more with automated tests than without them. The different stages of automated testing are outlined in Figure 1.2: Automation Road Map, p.9, as follows: One-Step Builds (manual command), Scheduled Builds, Push-Button Releases, and Installation & Deployment, with monitoring being performed by all team members so that everything is kept under control. Of these, the Push-Button Release stage determines the releasing of a software package the application is in prior to having it being actually installed and deployed.

**It is important for all tests in an testing suite to be able to build successfully prior to committing them to the repository.

Automated testing currently done in the Pas Development Environment:
Currently, there are a few automated tests for the PAS Suite, with regards to the PAS Development Cycle. Automated builds seem to be set up with Ant, in that xml files contain the build statements to be executed. Apparently, automated builds occur for however often they are set to build and run the source code files in the repository for a particular project (pas-common-apps, for example) and then generate a JNLP (Java Network Launch Protocol) file based on the project, to be launched and run in the classrooms (where there is no Eclipse environment within which to build and run the appropriate curnits). Otherwise, in Eclipse, the corresponding jar file is generated once the main class is built, and then the launch configuration launches the project. It is important to note that within Eclipse, the latest changes one makes to a file will affect the output, regardless of whether these changes have been committed to the team repository or not. (This is only particular to the machine one is working on, though, so someone working on another machine won't see the changes unless one has committed them to the repository). If these changes are not committed, they will not show up after the next automated build. If one wants to be sure that one's checked in code has in fact been built, one should check out:
TelsCenter Continuum
To learn more about continuum, visit:
Maven's Continuum

Many more such automated tests need to be created; every source code file should have some tests associated with it. Every person involved in programming should create test files along with coding. Corner testing, or boundary testing, is an important aspect of what should be included in the test files (for example, if a list is passed into a method, one test could be to pass an empty method to the list and see whether that would result in an error or not). Of course, commenting on one's code would also help so that others can understand it, prior to committing the code back to the repository. Perhaps each person creating tests to be added (or not) to the testing suite can also put a short description as to what the tests are supposed to do, in the test source code itself.

Also, when one is asked to commit a file, along with being asked for login and password, a screen can indicate about running builds before actually committing the file.
It is important to be careful and ensure that the application builds and runs prior to committing a file that one has been working on back to the repository.

What if a file was hurriedly/accidentally committed to the repository and something broke?
Fortunately, local history keeps track of all changes made to the committed file, so that one can revert back to a previous version if needed. Of course, it would probably save some frustration, along with time, to have automated builds to ensure that the application is not broken due to modifications in the particular file one is working with and wishes to commit.
Setting Up A Test Suite:
"Learning to Love Unit Testing: Improving Development through the practice of testing before you code" suggests creating a very basic source file (containing the package and importing statements and the general outline of the class), making a subdirectory for holding the test files within whichever area the package is stored in.
To maintain proper control of the unit tests, the following suggestions are made:

• make a subdirectory to hold test files
• make a separate test file per source code file
• be able to control which and how many tests are being run (individual, every test for a file, directory, or system)
• automate the build and runs of tests (this falls into automated testing)

A suggestion made is to use Ant for handling the builds and tests. If a target is assigned, the appropriate file can search for whatever location in which the source code files to be tested are in, and where the corresponding test files are located.

Ensuring successful builds:
In Java Extreme Programming Cookbook, Section 2.6, the need to have a good building environment is emphasized, with some ways to work towards this, as follows:
refactoring code, and performing continuous integration (section 2.6.1). Continuously integrating the application involves building it many times, so that any building failures can be caught relatively early on. Perhaps a script can be set up to perform an automatic build once the latest version of the test suite is checked into the repository. A separate directory can be used to store the latest build.

As for integrating the individual tasks being tested, the tests used to test those tasks can also be integrated into the main build. Dependencies may be an issue of possible concern, with regards to things like inheritance (classes deriving from each other). If code in a method (or its attributes, such as return type, parameters, etc. change, then the resulting places from which that method is called should be looked at and fixed accordingly).

Test-driven development:
As mentioned in Java Extreme Programming Cookbook, Section 2.3.1, test-driven development occurs when one knows that what one has written/modified works with the rest of the application. Apparently, XP (Extreme Programming) teams make use of test-driven development.

A set of steps for performing test-driven development, as outlined in Section 2.3.4: Testing New Features, is as follows (probably worth adhering to as much as possible):
(1) Run the project test suite for the project
If more unit tests are to be written (which they probably are), here's the next set of steps:

(2) If any new functionality is to be added, unit tests can be written for that as well (start with writing a single test)

(3) Ensure that the test compiles

(4) Run the test, see whether it passes or not

(5) If it passes, the new functionality/feature should be implemented

(6) Re-run the test

(7) Repeat the entire process for writing unit tests as often as one feels sufficient/necessary

Once each and every test passes, these tests can be committed to the source code repository. Then, all the tests can be run once again. Regression testing can be done to ensure that the changes that one makes to code do not result in breaking someone else's code or some other file. Every test should be error-free prior to committing them to the repository. Each programmer should do his/her best to ensure that any changes made to the code will not break other places in the project, prior to committing.

Acceptance testing:
As mentioned in Pragmatic Unit Testing in Java with JUnit, Chapter 1, satisfying the needs of the customer or end-user is another important consideration when testing code. This need falls into acceptance testing. For example, suppose that there is a software program for determining the amount of calories one burns after doing some activities, with the input values required for activity, duration, and intensity level. What if this generally works but crashes for special cases (e.g. 0, null)? This is where acceptance testing differentiates from, say, unit testing, in that acceptance testing can still determine that this program is of acceptable (passing) quality, as it generally works. In other words, acceptance testing helps ensure that the application or software package is doing what it is supposed to do at a general level.

As for whom can be acceptance testers, according to The Java Extreme Programming Cookbook, Section 2.3, they need not necessarily be programmers, but perhaps those that understand the essential purpose of the application and can make sure to test whether that purpose has actually been sufficiently fulfilled or not. As mentioned in Section 2.3.9 on acceptance testing, a client or customer can probably do this testing; it is worth noting that acceptance tests do not require 100% success, unlike unit testing. The most important acceptance tests, however, should always work. It is possible for acceptance tests to fail more often than unit tests, just because they are not run as often. For the purposes of testing within the TELS-based projects, as is our case, Freda, Doug, teachers, researchers, etc. fall into the category of the acceptance testers.

Automated acceptance tests, along with automated unit testing, can also be helpful. As mentioned in Java Extreme Programming Cookbook, Section 2.6.2, automated acceptance tests can be placed together into an acceptance suite/battery of tests. Maybe a portion of these tests can be automated from one build to the next, rather than testing all of them all of the time.

Notification:
Team communication is important. Perhaps email can be sent out to the team, as done via continuum, reporting the occurrence of failed tests (and details), assuming that failed tests occur. The person who last committed the source code from which the failed tests occur is responsible for fixing that build failure. Also, although it can be tempting to overlook warnings and just focus on errors, at some point it would be good to also address warnings (e.g. Type mismatch, unsafe type casting, etc). Some warnings are probably more serious than others (for example, unused variables may not be as serious as type mismatching, except perhaps for just taking up extra space). This understanding can help in terms of prioritizing which warnings to mainly address.

References:
Java Extreme Programming Cookbook, by Eric M. Burke, Brian M. Coyner - discusses the benefits of XP, or Extreme Programming. Important ideas discussed include code sharing, test-driven development, and continuous integration. Tools used to performing testing with are also documented in detail (e.g. Ant, JUnit), with implementation/code included. Automated builds and testing also mentioned.

Pragmatic Project Automation: How to Build, Deploy, and Monitor Java Applications - points out how automated builds and tests can help prevent potential breakage of code after committing a working file to the repository. Chapter 1 is a fairly easy-to-read chapter with simple language - familiarity with concepts like version control, integration, builds is assumed
Relevant Chapters Used: Chapter 1, Introduction; Chapter 3, Scheduled Builds (more in-depth description of setting up builds, benefits of doing regular builds)

Pragmatic Unit Testing in Java with Junit - explains unit testing in simple terms, good analysis of benefits vs. costs of doing testing (mainly benefits)
"Learning to Love Unit Testing: Improving development through the practice of testing before you code" by Dave Thomas and Andy Hunt (taken from
http://www.sqtemagazine.com) - provides code examples for doing testing, lists out steps to do in order to place all tests in one place, analysis of misconceptions, potential pitfalls, and rebuttal, to advocate for doing unit testing
[]

http://maven.apache.org/continuum/[
http://www.telscenter.org/continuum/servlet/continuum/]
http://www.junit.org
http://safari.oreilly.com

Other Potential References:

Software Testing by Ron Patton

Kent Beck's, Extreme Programming Explained: Embrace Change (2nd Edition)

Questions, Comments, Feedback:

Please look at the attachments and comments made for this page. Feel free to make any suggestions.