Holy trinity number one -

This is the first of the three papers that are frequently cited together in support of the treatment of visual stress in poor readers.

Wilkins AJ, Evans BJ, Brown JA, Busby AE, Wingfield AE, Jeanes RJ, et al. Double-masked placebo-controlled trial of precision spectral filters in children who use coloured overlays. Ophthalmic Physiol Opt 1994 Oct;14(4):365–70.

Pubmed

Arnold Wilkins web page where you can download this reference if you wish

At least this paper was a big step up
from this…..

It would be a good idea to start with the good things about this paper which  was  massive step forward from what went before – the opinions of a self proclaimed revolutionary. At least the authors believed that this subject was amenable to the scientific method and should be tested according to the standards that apply in other areas of science.

It is also important to state that these kind of studies require a prodigious amount of work and studies on dyslexia are particularly difficult. Furthermore, the first randomised controlled trial is often problematic. Unexpected problems and sources of bias are frequently encountered.

So rather than be critical of the study itself I am critical of the ‘spin’ that has been put on the data in the discussion and the way this study has been selectively cited as 'proving' that precision tinted lens can ameliorate visual discomfort in individuals with ‘visual stress’. This study produced negative results that do not support the use of precision tinted lenses to treat  ‘visual stress’ in poor readers.

So what about the paper.

The subjects were 68 poor readers drawn from a school for dyslexics and two state schools.

The criteria for diagnosing visual stress was voluntary sustained used of overlays.

The study design was a crossover one so that subjects received the optimum tint determined with the ‘intuitive colorimeter’ and another tint which was found not to ameliorate their visual distortions. They were randomised to either receive experimental lens or control lens first. As far as we can tell allocation concealment was good. The subjects wore one lens for one month followed by two weeks with no lens, then one month with the other lens

The intuitive colorimeter was used to determine the 

optimum tint and a closely related tint which did not

ameliorate their visual distortions

The two outcome measures were reading accuracy speed and comprehension measured with the Neale Analysis of Reading test and the symptom diaries that participants in the trial kept.

The principle problem with this study was the high rate of dropout and failure to analyse the data on an intention to treat basis. High drop out rate can be a problem with cross-over studies, which of necessity last longer than parallel trials. The latter compares two groups randomised to receive one treatment or the other.

Neale Analysis of Reading Test

Data were available for 45 of 68 subjects. Both placebo group and experimental group improved over the course of the study but there was no statistically significant difference for reading speed, accuracy or comprehension between the two groups.

Symptom diaries

This is where the data starts to be spun. Unfortunately, data was only available for 36 of 68 participants. That is, nearly 50% of the data is missing. While some missing data is the norm; a study with this rate of drop out is 'pretty fatally holed' below the water line. 19 subjects had more good days with the experimental lens and 17 subjects had the same number of good days or more good days with the placebo lens. Without data from more of the other 32 subjects, no meaningful conclusions are possible.

The authors claim that their 'positive result' should not be seen as an artefact of the high rate of attrition. They have no reason for that belief.  It is more likely that those who drop out of a study are obtaining no benefit. This claim constitutes the sort of wishful thinking that good peer reviewing should have eliminated. The study also contains conflicting data. When asked at the end of the study which lens they preferred 22 children preferred the experimental lens and 26 preferred the control lens. 

The strange thing is that the most robust part of the study looking at the Neale Analysis of Reading, with a follow up of 45/68 = 66% is seldom mentioned. Yet the least robust part of the study, looking at symptom diaries, with a follow up of just 36/68 = 53% has been hyped to the point of absurdity. Particularly when you remember that the same study produced conflicting evidence. When asked, a small majority of children preferred the control lens. 

This was really a pilot study (nothing wrong with that) which should have pointed to the way to further better studies.