TCK_Contributors_Guide

Guide for contributing a new TCK test (at least an outline of a guide)

Some Basic Test Structure

• The typical test method begins by submitting a job via JobOperator#start.

  • There is no order of execution assumed among the tests, so to do pretty much anything requires a job to have run within the test method.

• In general, the assertion / validation logic (i.e. the check for success/failure) is either performed:

  • within the TestNG test method at various points within the sequence of calls to the job repository (i.e. to JobOperator).

  • OR within the job execution (i.e. within the batch artifacts themselves)

  • OR within some combination of both.

• The tests use the JobOperatorBridge utility provided by the TCK in order to encapsulate the waiting for "termination" (i.e. waiting for COMPLETED, FAILED, or STOPPED).

• Some tests have waits which introduce a timing dependency. I.e. because batch jobs execute asynchronously (and because the stop operation is asynchronous), there is a chance that a conforming implementation will fail a given test because of a timing issue (e.g. job completed before it could be stopped). We deal with this by including a specific, well-documented system property for each such individual wait, in each individual test, to allow the executor to tune for his particular implementation and environment. A default value will be hard-coded as well.

What files make up a single test?

1. First, the TestNG test method

   • Annotated with @Test (imported from org.testng.annotations)

   • Referring to one or more jobs (by JSL name)

2. Second, the JSL (job definition XML)

   • Refers to one or more batch artifacts mostly by a short name (i.e. not the fully-qualified class name) for @ref value.

     • This value will be the CDI bean name, i.e. the value of the @Named annotation on the artifact ( or the equivalent default if @Named is omitted).

     • there will be a corresponding entry for each @ref value in batch.xml

   • I believe each JSL lives in its own file in com.ibm.jbatch.tck/src/main/resources/META-INF/batch-jobs. At one point we were generating some but I think we removed that.

3. Third, the batch artifacts (the readers, writers, batchlets, listeners, etc. implementing/extending the Batch API).

   • All have a package like com.ibm.jbatch.tck.artifacts.*

   • At one time there was a thought to group more generic artifacts that can be reused across many different tests into like com.ibm.jbatch.tck.artifacts.reusable as opposed to those tied to a specific one or two tests which would live in com.ibm.jbatch.tck.artifacts.specialized.

     • Some remnants of this remains, but you can see we’ve not been too organized and have also since created some new child packages like com.ibm.jbatch.tck.artifacts.chunkartifacts.

   • In some cases, there are somewhat clear "mappings" in naming conventions/patterns such that you can understand what artifacts are used in which jobs (JSLs). In general though, this is not a super-easy task and requires some custom digging.

Checklist for new test

1. Add new test class if necessary

   • Follow other class' examples and add methods:

     	private static void handleException(String methodName, Exception e)
     	public void setup(String[] args, Properties props)
     	public void  cleanup()
     	public void beforeTest()
     	public void afterTest()

2. Add test method, JSL, artifacts

   • Annotate with both TestNG @Test and @org.junit.Test

   • Log with org.testng.Reporter

   • Instead of using something typical like JUnit Assert, use the custom (provided in this TCK) assertion util: com.ibm.jbatch.tck.utils.AssertionUtils

     • E.g. AssertionUtils#assertWithMessage

   • Use the TCK utility: com.ibm.jbatch.tck.utils.JobOperatorBridge which itself waits for termination on start and restart

   • Define a String METHOD corresponding to test method name, e.g.:

     	public void testSplitFlowContextPropagation() throws Exception {
          	  String METHOD = "testSplitFlowContextPropagation";

   • NEW: Add new @TCKTest annotation mapping back to spec

     • The existing 1.0 tests don’t include this yet but would like to add at some point if we can get to it

3. Update batch.xml

   • TODO: explain

4. Update TestNG suite XML (if necessary)

   • TODO: explain

5. Verify with RI?

   • TODO: explain

6. Update spec buzilla?

   • TODO: Just trying to sketch out a process, nothing is needed at this point.