The myths and realities of Psychometric Testing

There is no reason why psychometric tools need to be expensive. Once developed, the overhead for assessment testing is not large and the cost of ongoing development requires only time and the employment of suitably skilled individuals to continue research. There are options in this area to create huge or modest overhead cost structures.

If we look atcompetitive markets, the costs for developing psychometric tools are recouped primarily by longevity and usage volumes, not high price structures. Any company whose prices are well and truly above that of the competition is likely to have been a monopoly player in the past. It will continue to play from this position until competition slowly erodes the market advantage it had from being first off the block.

Our position: We aim to make our tools among the most cost effective on the market. We believe there is no reason why clients should pay high set-up costs, high annual licence fees or excessive on-going costs.

Psychological testing is like banking, people change their provider only rarely. The customer will complain about the costs, level of service and hidden fees but in reality it just seems too much trouble to change. Potential new customers, however, do not have to overcome a resistance to change and are more open to decisions based on merit and value.

Market share is always a ploy to sell a product and works on the philosophy of "If it is good enough for them then it is good enough for me." We are social beings, and people inherently have difficulty making decisions on their own! In this way an early test provider becomes synonymous with psychometric testing, further ingraining the myth of 'first is best'.

Our position: We believe that resistance to change will diminish over time as people become more aware of the issues in psychometric testing and the options that are available to them. This is evident in our own growth.

Some test providers have played on this by giving people only enough knowledge to make poor decisions. Nowhere is this more prevalent than in the issue of predictive validity. Predictive validity refers to the ability of a psychometric tool to predict some future measure such as job performance. The question is: "How do we use predictive validity to assess the usefulness of a tool for our organisation?". Test providers usually brush over that question.

Predictive validity is often defined by a correlation (i.e. relationship) between one variable, say a score on a verbal reasoning test and another variable, say job performance. Some consumers know that a coefficient of say, 0.25 and above is deemed useful. How they intend to use this magical figure to improve decision making in their organisation is less certain.

In reality, a personality measure may account for 9% of the difference between individuals' job performance. However, before you start throwing out psychometric testing, remember that interviews predict roughly the same amount of variance of job performance. The following quote from an informed test user demonstrates what predictive validity should be used for:

"Predictive validities shown by test producers are only good for demonstrating that the test has some utility for predicting work in general. It is a starting point not an end point for HR professionals. It provides the necessary support to then trial the tool in your organisation to demonstrate its predictive validity for you".

To demonstrate the usefulness of a test, it is important to evaluate it against your own criteria, rather than blindly accept the validity studies produced inside other organisations that may have little similarity to your own. Some test producers use dubious predictive statistics primarily because the people they are selling to are not statisticians and therefore have difficulty determining the value of the information they are reading. Issues with respect to sampling error, the application of significance testing and the promulgation of misleading correlation data all contribute to the weakening of predictive data for any one organisation

To ensure predictive validity studies apply to your organisation, you need to:

1. Check that a test has some relationship to performance. This is a starting point, like a tick on a checklist. To be of value, a test used for selection must predict the world of work. Also, check any meta-analysis research that combines single tests into joint studies. Does your test relate to the types of tests that were used in these studies? IF SO WE CAN INFER A DEGREE of predictive validity for the tool, given it has been constructed well.
2. Develop your own in-house study. This can be done by using a testing system that stores your data and enables it to be correlated with job performance.
3. Develop a multiple regression equation. This will give you weightings to apply to test scores so that YOU can make predictions of performance.
4. Plug the test scores for one individual into the regression equations that come out and you will have a useful guide for your selection.

All of this, by the way, can be done in-house in less that 5 minutes with the appropriate assessment software that allows you to make strategic rather than transactional decisions on the basis of assessment data.

Our position: As a standard part of the GeneSys software package available worldwide from Psytech's distributors, clients automatically have access to the capability to hold assessment data so that in-house validity studies can be completed.

The value of psychometric testing comes primarily from having collected a lot of data and then using that data to model performance. Test providers have done a disservice to HR professionals by not giving them the capability to collect their own candidate data for later use.

In real terms, this means that most organisations have little empirical data about what predicts performance inside their organisation - despite paying thousands of dollars for assessment testing services over many years. This is an injustice. It is equivalent to having a cash register that only holds the last transaction and does not feed a database of sales over the last month or year, etc. How useful is that?

Our position: We have built our business on making psychometric testing non-transactional and giving the client the capability to collect assessment data on potential as well as existing staff. 

In recent years, we have seen an infiltration of some profiteers into the testing industry. Their mission is simple - "How can we make as much money as possible out of psychometric testing?". They may be funded not by psychologists but by capital venture firms and large corporate backers. These stakeholders know little about testing, and while they may get a few quick wins on the board, they tend not to translate their investment into long term returns. Indeed, the psychologists in those teams may even be the casualties.

Issues of psychometric robustness are slowly being eroded as some people strive for any way possible to make a dollar. This has resulted in tests being supplied over the Internet to anyone, without suitable training, and tests being produced without any validation data available.

Our position: We are a business owned by psychologists who seek to bring you robust solutions that conform to the global body of research into psychometric assessment.

The people best qualified to make psychometric tools are psychometricians (i.e. those people whose skill set is around the measurement of psychological phenomena). Psychologists use the tools but the making of them belongs in the hands of psychometricians, as they have the necessary skills to make meaningful measurement possible.

Our position: We publish psychometric assessment tools developed by Chartered Psychologists who are alsopsychometricians. Our project team always includes a Chartered Psychologist as a final point of accountability.

The Internet provides convenience in many situations and has obvious popular appeal today. Some test providers have prospered in the e-commerce environment and some have not.

Psychometric test interpretation relies on standardised testing conditions. Instruments used for the purpose of self-assessment by the individual have been relatively easy to mount on the Internet (e.g. career or well-being assessments). In staff selection, however, standardisation means that a more controlled testing environment is required and a range of other factors need to be addressed, such as the identity of the respondent and the like. Other issues such as restricted access and data storage also need to be addressed. If one wishes to lift testing from a transactional to a strategic level, authorised access to test data for statistical analysis is a priority. These are but a few of the issues that need to be addressed for reliable, valid, ethical psychometric testing over the Internet.

That is why discerning providers have been careful and deliberate in their preparations and processes before making psychometric tests available on the Internet. In this field, there has been a benefit in learning from the misfortune of those perhaps too quick off the mark.

Our position: Psytech International have had the capability of delivering Internet-based tools for some time, but have held back general release until all the key issues were uncovered and solutions developed. Our position is that the Internet is merely another delivery tool and we will offer Internet based 'testing' only in circumstances where one can define the value of the data coming back.

The concept of personality transcends such artificial boundaries as our work or home life. The flexibility to adapt one's behaviour in different situations or not is merely one aspect of personality, and it is not necessary to create a whole new personality questionnaire to cover it. That is done more for plausibility rather than reliability.

Meta-analysis is the standard in independent research activity and meta-analysis has found that a test which provides good measures of the 'big five' personality traits does predict performance regardless of the setting. This finding is based on the analysis of patterns in the published research in a whole field, rather than on the research of one entity.

Our position: Tests published by Psytechare derived from psychological models that are well supported by rigorous, independent research. We offer the choice of personality tools that are underpinned by the 'big 5' model of personality as well as a 'work personality' profile.

Factor analysis is generally regarded as the most robust statistical process for ensuring the rigour of a psychometric tool. In particular, it allows us to find dimensions of personality that are distinct from each other and to identify questionnaire items that do not overlap.

A fundamental flaw with tests which are not factor analysed is that they tend to be excessively long and claim to measure multiple, independent, dimensions of personality - sometimes as many as 20 - 30 traits. In reality, there is likely to be a high degree of overlap between the scales inside these assessment tools.

Our position: Psytech's tests are built around broadly accepted psychometric theory. This means that there is a strong reliance on the underlying factor structure that supports the tool.

One of the strongest indicators of a test publisher's lack of commitment to psychology is a closed-door policy with regards to accessing technical manuals and/or independent research that supports their tools. If this is made available at all, it tends to be to clients who have already bought into their marketing hype. In this way, technical information is not available for people to make informed decisions, but rather is used to preach to the converted. Moreover, it contravenes the international call for open-access to research. You can read more about this topic at: http://www.soros.org/openaccess. 

Our position: Psytechdoes, and will continue to make the research for their tools publicly available.

Ipsative tests are tools that use a forced choice option and require a person to select from a range of traits the one that is most and least like themselves. Non-ipsative tests require a person to respond to a question by indicating their preference on say, a five or three point scale. 

Put simply, ipsative tests allow the comparison of relative strengths of personality traits within an individual but do not allow comparison between people. The dilemma here of course is that selection is all about making comparisons between people.

Ipsative tests have numerous other problems including:

• These tests cannot be normed and therefore comparisons against groups of people are nonsense.
• The results can not be factor analysed and therefore the personality dimensions being measured must be questioned.
• These tests are subject to input response biases.As a result of these and other issues, the use of ipsative tools for selection has been discouraged by the British Psychological Society. Instead, ipsative tests are of more benefit in the area of individual counselling such as career guidance. 

The second problem is that ipsative tests force the spread of a person's scores and while this may make interpretation easier it does not make it any more valid. It is a classic case of people being misled by false logic and simplistic marketing. Indeed, as discussed above, it is the lack of validity with respect to making comparative judgement that is the key problem with ipsative tools.

The public misperception about the merits of ipsative tools in selection was fostered by publishers some time ago and it has not yet left the marketplace. I quote from Johnson, Wood and Blinkhorn (1988) "…publishers and promoters of these (ipsative tests) are either unaware of, or do not understand, or choose to ignore their limitations". 

For those seriously interested in exploring the misuse of ipsative personality tools we strongly recommended one of the most prominent articles in the area (Blinkhorn, S.F., Johnson, C.E., & Wood, R.F. (1988). Spuriouser and Spuriouser: The use of ipsative personality tests. Journal of Occupational Psychology, 61, 153-162).

Our position: Psytech does not publish ipsative tools for selection. Moreover, our training courses cover issues related to ipsative tools explaining the relative strengths and weakness related to each type of test construction.

Psychometric tools are built to be interpreted in a standardised way. This is why computer-based reporting is not only valid but may be less prone to error than human generated reports. A computer report will always provide a similar reading based on a person's scores and is not impacted by any potential errors such as political bias or stereotyping. 

Computer programmes are not, however, sophisticated enough to make all the required integrations between individual scales but provide a fair first draft of a report. Human input is required in order to make some of the more subtle scale interactions.

The irony is that psychometric tools are easier to interpret than an employment interview. Moreover, just because someone is a psychologist does not mean that they have had the training in psychometric tools required to utilise them to their potential. 

Our position: We do not believe that psychometric tools are solely the domain of psychologists, but we will only distribute psychometric tests to people who have the necessary training and/or academic qualifications to ensure proper interpretation.

The rationale for training or ensuring certain skills is summarised as follows:

1. Ethical: Psychometric tools are used on real people to make decisions that effect their lives. People who complete these tests provide a lot of information about themselves and they are entitled to receive feedback on their results. Skills are needed to do this constructively.
2. Standardisation: As discussed earlier, the usefulness of psychometric tools is that they are administered in a standardised manner so that valid comparisons can be made. This requires some education and training.
3. Legal: Selection can be a litigious activity. It is therefore vital that best practice protocol is rigorously followed in the administration and interpretation of psychological tests as well as the delivery of candidate feedback.
4. Utility: Training teaches people how to maximise the usefulness of test data. In particular, training teaches people how to relate the information from each tool to competency models, selection decisions and people development.
5. Psychological and Human Resource guidelines: Given points 1-4, is it understandable why various Psychological Societies and Human Resource Institutes stipulate that tests should only be made available to trained users.While training is of value, it should not be training simply for training's sake, nor a mechanism for extracting excessive revenue. It is a matter of providing sufficient training to use the tools provided properly, without overkill.

Our position: Psytech and our international distributors provide comprehensive training for all test users.  This has been prepared along with British Psychological Society guidelines with the material adapted for localmarkets. We recognise relevant qualifications and the training offered by other test providers and will accept this on a case-by-case basis.

When a person is compared to a norm group, the size of that group has little relevance to the comparisons that can be made. Instead, it is the relevance of a norm group to your own organisation and the distribution (or spread) of scores inside that norm that is of critical importance. If a norm group is not reflective of the people you are testing and does not differentiate people well, there is NO benefit in knowing that a test publisher has collected norm groups of several thousand people. If a norm group is well built and is found to be reflective of the population of interest, a norm grouping of 200 people may be sufficient! 

One of the key issues with norm grouping is that we must be comparing like with like. Thus, the people inside a norm group must have sat the test in the same conditions and come from like ethnic and educational backgrounds. Only if the conditions of Internet testing are sufficiently controlled do they achieve sufficient standardisation to allow the building of robust norm categories. Many Internet tools do not.

Beyond these criteria, a test publisher should also be able to confirm the gender mix of participants and what roles people were tested for. All of these factors will help define the suitability of one particular comparison group over another. 

As a final note, the most relevant norm comparison group will often be a representative cross section of in-house staff who perform each of the role(s) under review.  

Our position: Our norms are updated on a regular basis. These are provided free to all users and can be broken down according to the bio-data that is collected on each candidate who completes the assessment tools. The GeneSys system also allows test users to create in-house norms in their own system and to examine the make-up of each norm group so as to make an informed decision about its appropriateness to them.

Test producers have looked at various means of extracting additional money from client organisations. This includes everything from having an organisation pay each time a candidate sits a test to every time a report is generated or both.

In reality, once test data is inputted into a scoring system, no additional time commitment is required for a report to be automatically generated. If clients do not wish to incur the often significant reporting charges, they will opt to rely solely on verbal feedback to a candidate or recruiting manager (using just a profile chart) or they may prepare short-form, less comprehensive reports by hand for distribution.

Our position: Our users are only charged ONCE to put a person's data into the GeneSys system. Once the data has been entered or imported, there is no charge to produce a report, analyse the data, or develop in-house norms. We do not subscribe to the practice of charging at each step of the testing process, relying instead on the increased usage that comes from the distribution of tests at a very cost effective price.