evolution of assessing

The testing industry in South Africa is one that has seen enormous growth over the last 5 years. As an example, when I joined my company 7.5 years ago, there were 6 of us in the organisation. We now have over 120 people working for us.

So what does this mean? Well, it means that we went from having very little interest in testing and not so many testing jobs available in SA, to the opposite situation really quickly. This in turn meant that suddenly, the amount of money you could earn as a tester went from almost nothing to quite reasonable salaries. So everyone decided to become a tester. We had to try and quickly train people to meet the demands of our clients and so a lot of organisations – ours included started graduate recruitment programs to train graduates in the basic concepts of testing. The bottom line is that we have ended up in a market where people are demanding high salaries but have very little skill and experience to match those salaries. (This I think is partly the reason for a lot of companies outsourcing to other countries but that is a whole other blog).

For a few years now, we have been trying to recruit only the best people to ensure that if we are going to pay a lot (and in turn bill a lot) for someone, they are indeed able to deliver to that salary. This is not an easy task. There are a number of steps to this process, and I just want to focus on one aspect that I am trying to refine.

Most of our clients are large financial institutions with heavy waterfall methods of development that rely a lot on specification driven testing. So one of the key skills that we need, is to be able to successfully analyse a specification and come up with tests or test ideas based not only on the specification, but also the things missing from the specification. To this end, I started to develop an assessment which gave people a specification (which I got as a sample off the internet) and asked some questions. This assessment has now been through a number of iterations and is still evolving. The areas it is evolving in are twofold. Firstly – the way I ask the questions. Secondly - my rubric for assessing the answers. The concept of the rubric was first introduced to me in the BBST Foundations course.

For an example developed by Cem Kaner you can go to http://rubistar.4teachers.org/index.php?screen=ShowRubric&rubric_id=1435804&

The evolution of the assessment thus far has been guided somewhat by the responses I get. This has led me to refine the questions and also try and ask myself what exactly gives me a happy / not happy feeling when I am looking at the response and why. (For me, it starts with an emotional reaction that I then need to analyse and put into critical thinking).

The iterations prior to the ones I am showing here are not worth seeing…(I had to use screenshots as I couldn't see another easier way in blogger to simply attach the files - if you know of one, please let me know :))

My first real iteration was simply a set of questions and an open section for my comments.

My second iteration, I changed the wording of the questions slightly and played around with a rubric but didn’t like the rubric as I felt it was too rigid in some ways and not specific enough in others. (I know – the role things are just crazy).

My third iteration played with the questions some more. And looked at some guidelines for marking.

My fourth iteration played with the questions a lot more and also added more of an outline for a rubric that I hope to evolve some more.

I am posting the evolution of the assessments with the hope that you may have some insights / suggestions for improvement, or questions that may guide my thinking in this area. I have left out the spec portion as to me it doesn’t matter – you could use any spec…as long as it wasn’t perfect and not too long. (I allocate 1.5 hours to the assessment).

p.s. you are welcome to re-use this if you find it useful