Are we really doing functional testing?

20/01/2020 08:40

Automation, yes but of the right validation done at the right place...

What are we doing?

In March 2017, when we received the trigger "automate everything", we started automated our validations.

We are working on mainframe, with lot of files, databases and a "small" GUI web based which just allow to follow-up transaction status and to act on them.

Our application is generating files used to generate reporting for the final customer or money flows. In these files we have plenty of fields, non-visible via the GUI.

Of course, we can visualize them via another GUI which is our 3270 terminal but... my customer does not have the right plug in for UFT (for example) to do automation with it.

So we started building something by our own (adhoc tools):
- creating SPUFI (batch SQL) to get the data from the databases
- downloading SPUFI results and files generated by the applications to our computer
- using Excel to compare the content of the file with the SPUFI content, by putting in place logic in macro, etc.

I was fully participating to it and I took the harder subject as my programming knowledge is higher than my colleagues.

Suddenly I realized that I was in fact translating the coding on the mainframe (Cobol) to VBA in Excel to allow our automated validations to take place.

I thought... "what are we doing? are we really doing this?"

I was just writing "unit test validation coding" and not "functional test"...

So, I start doubting about our "functional tests"...

I checked all our validations and all of them were "go in file X, check that the field blabla contains value a".

Nearly none of our validations were different from previous example.

So, is it really functional testing?

We play the full data flow in we want to ensure that the data is correct?

Shouldn't we "just" (it's a shortcut) ensure that the applications generate the right number of records in the right files?
And as we are involved in the End To End Testing, we should ensure they are correctly processed afterwards...

Instead of this, we
- find the file generated (ok)
- find the corresponding record for it (nearly ok)
- starts apply the record template on it and
- validate field by field if their content is correct (not ok)

and we do it for all dataflows, so yes... when we know that each file contains around 200 fields that we have thousands of dataflows, no I/we understand why testing phase is so long.

Why do we do this?

It's historical... the quality of the delivery between each test phases was so low, provoking so many instability and so many complaints from external partners in testing that we decided to validate all the fields, each time.

Idea is not bad... but for me (now)... clearly at the wrong place... it should be done in Unit Test.

But when you say this to developers who are doing the unit tests, they look at you and say "Checking the content of a field is functional, not technical. Unit Testing is only ensuring on technical point of view that the program is working fine."

Damn... I'm blocked... This developer is right... (S)he knows nothing about the functionalities.

But... Wait...

The program is a technical answer to a functional design. (S)he knows nothing about the business but functionality, yes (the developer is most of the time also doing the technical design).

The developer should test that this technical answer is working fine so that (eg)

- a field which should contain a date, contains really a date (format, values...)
"20-20-20" is not a valid date, it's common sense (but we received Delivery with Unit Test Done having this issue).

- when a field in a file is a copy of a field from the database it should match.
we received deliveries with status Unit Test done, in which the field was even not selected from the database, the field in the file was always empty but it was not an issue for the developer because the program was working fine (?).

- when a field is calculated by the program, the calculation matches with the functional request

- and last but not least, that all the changes made in the program do not have side effects on behaviour which should remain unchanged (non-regression testing).

When you say this to a developer, (s)he looks at you with empty eyes and tells you "you don't know my job, don't tell me how to do it. I don't tell you how to test."

No one, had the courage to fight this kind of answers so we have added plenty of fields validations in our tests to overcompensate this lack of Unit Testing.

Now

I say, "I will not automate anything on functional testing point of view as long as the unit test are not better done and automated as much as they can."

I don't want to spend money and time doing the same mistake again and again.

I agree to work on subjects like "how can we speed up our test execution?" but I will mainly work with the testers and developers together to create this different approach of unit testing.

It will take time, effort, money... but if we want to enter a DevOps world, to improve our time to market, to improve our quality... We have no choice.

Abaonwan