Unit Testing on the Mainframe

I'm sure you know that most of the time, unit testing on mainframe is just checking basic stuffs and when it runs fine, the developer says "Unit Test Done!" (I know it because I did it 😊 ).

But most of the times, afterwards when you start integration or functional testing you find plenty of bugs and the developer thinks you work against him/her.

It takes lot of times to explain him/her "my job, as yours, is to deliver quality to the Customer and we are complementary, not enemies!".

And in this context, I think we can say that unit tests cover maximum 20% of the coding.

So, I started a quest called "WTF???? how can we improve unit testing?"

By working on it often, I fall on pages, sites made for "new" technologies but I never find sites talking about "old" technologies.

I find plenty of examples for Java, online activities but never for Cobol and batch stuffs...

I was really frustrated.

One day, I start thinking differently...

There is no real difference in unit testing Java or Cobol maybe we struggle doing this for something else...

And of course, it's one of the reason...

We are not able to correctly unit test because each program and each routine are accessing directly the databases.

And we are not able, in current situation, to bypass or simulate these databases accesses...

So, each time we want to "unit test" a routine or a main program we have first to create data in the databases and it's impossible to have easily re-usable data as we have plenty of integrity rules and dates related checks.

So...

How can we solve this?

For me we must refactor a lot our applications.

1. Remove the access to the database from the routine
   They should receive all the necessary data in their interfaces
    
2. Rewrite the routines
   We have too many exception in all the paragraphs.
   If in case A a functionality is doing this and in case B a functionality is doing max 80% the same with exceptions Then
   Write a new functionality.
   Why?
   Too many risks of side effect and if you do it for functionality B, functionality C which will match 80% of B, will also become an exception of A...
   So, you will struggle in maintenance.
   
   Maybe it will lead sometimes to longer maintenance coding but most of the time you will save a lot of time and effort.
    
3. Rewrite the main programs
   The main programs should only
   - manage restartablity
   - read/write files
   - select information from databases
   - update information in databases (update, insert, delete)
   - call routines based on flows specifications
   - doing error management
   Nothing more
    
4. By doing this, all the business logic will be in the routines no more in main programs.
    
5. Unit tests of each routines will be facilitated
   You can write a batch program calling each routine with all possible input combination and check their output with expected results.
   The parameters given to the batch program will guide if unit testing is full, for a specific routines or for multiple routines.
   
   The output of this batch program will contain information about
   - each calls input / output / expected (status of the test, passed or failed)
   - percentage of passed tests
   - percentage of failed tests
    
6. And the good news is if you do this... you can simplify the CI/CD process in a DevOps approach with automatic deployment.

Proposing/Doing this will of course leads to...

It will cost so much

In fact, most of the time functional testing will overcompensate the lack of unit testing (article will follow) and in functional testing, for applications with long lifecycle (multiple days) retesting a defect costs a lot.

So, we will save money because in unit testing we are the master of the time, we are also able to detect faster regression so... yes, it will cost to refactor the modules, programs and to create all the possible unit tests for them.

But it will save lots of money afterwards... (and very quickly)

1. For each changes, we will be able to retest (automatically) a lot of combinations per routine in one hour instead of doing it in one month (sometimes I exaggerate :) ).
    
2. Maintenance of the coding will be easier
    
3. Managing the changes will also be easier, we really understand the impact of a change. It will not be "I think that by changing this the impact would be...", we will be sure of what we do.
    
4. Functional tests will do what they are supposed to do so, will need less money/time (I'm pretty sure we can safe 50% of budget).
    
5. Managers like to speak about "automating the system tests" because it's the one of the only specific testing phase for which budget is discussed and visible.
   But if we increase the unit test coverage and we decrease the overcompensation done in system test, the budget of the system test will decrease, and we will be able to identify the real stuffs to automate in system testing.

Of course,

It's not something we will do in 1 day, it will take months but stone by stone, step by step we will be able to do this, and we will learn together a better way to do things.

Thanks for reading,

Abaowan