Supporting Families
  • banner6
  • banner9
  • banner13
  • banner11
  • banner12
  • banner10
  • banner2
  • banner4
  • banner8
  • banner7
Wednesday, 08 July 2020 19:23

Comprehensive Future Fails Test on Grammar Schools and Pupil Premium (twice)

Written by
Revised 13th July
The campaigning organisation Comprehensive Future (CF) has published a lengthy article whose main purpose appears to be to attack me. For the second time, this uses false data they have published relating to grammar schools and Pupil Premium children. The problem dates back to a previous CF article about grammar schools two years ago, which wrongly stated, ‘When asked how many pupils were admitted through these priority policies 80 schools responded, revealing that just 574 disadvantaged pupils were offered admission out of their 12,431 available places... there were 22 selective schools who responded to say they had failed to admit a single disadvantaged pupil through their policies’.  Unfortunately, in order to obtain these figures, the authors of both CF articles used figures from a database that has no basis in reality and then have compounded the fiction by using data taken from the wrong column of the database, to make these false claims about grammar school performance, damaging to the image of these schools. The whole fulfils the well-known IT mantra of ‘Garbage in, Garbage out’, twice over.
 
CF has informed me that their published article is the continuation of what I was told was a confidential email correspondence, about a single phrase in a minor paragraph of an article I wrote earlier this year which they have chosen to open up in this way. That article was also about grammar schools and Pupil Premium, although mainly factual rather than theirs which is polemical. The phrase that CF objected to was: ‘demonstrating the falsehood of a previous claim by them’. The new article alleges that I ‘accuse CF of falsifying data. We refute these allegations and object most strongly to the implication that anyone who is a part of Comprehensive Future would alter or fabricate figures supplied in response to an FOI request’, which of course I didn’t, but this misuse of statistics does beg too many questions,  explored below. 

 It had been clear to me in the first place that something was and remains very wrong with these quoted figures. This led to my comment after I had seen a further FOI from the same author, published last month, exposing the falsehood of the data. However, I chose not to investigate at the time, being happy to move on to more important matters without being aware of all that lay behind the organisation’s false claims. There it rested until I was challenged over my comment. As I examined it further, and CF vainly tried to explain the contradictions, the whole sorry story has eventually unfolded.

Despite the misleading title of the second CF article, 'Comprehensive Future’s policy on Freedom of Information requests and data transparency', it was explicitly written after I had participated in an email exchange between myself and the Campaign Officer of CF, labelled confidential by her. This is confirmed by her last two emails to me in the exchange. The first informed me that the CF Chairman would email me on Monday with the CF view of the matter; the second informed me instead, that the article's publication would be their response. This is certainly an unusual way of conducting a confidential discussion!

The Database
This contains an analysis of the responses to an FOI sent to each of the 163 English grammar schools by CF in the summer of 2017. 162 replied, a remarkable response rate showing their happiness to co-operate in the project, although so badly let down by CF in its enthusiasm. I have not been offered a sight of the questions which were apparently numerous, but the request was sent years before the date you're talking about. We didn't use the data for quite some time - mostly because it was an over-complex request. I didn't write the FOI and I think it was poor, a lot of work to address and confusing’. Yet another question left unanswered is, if the FOI was so poor, why was the data from it published as gospel and accepted by other media outlets as such. The article was published almost exactly two years ago, the relevant information drawn from a database set up in February 2018, (so not years before as falsely claimed). I haven’t had sight of the FOI questions but the author of the database, Joanne Bartley the Campaigns Officer for CF. has now sent me a copy. Although it is too wide to re-produce as an online document, I am happy to forward a copy to anyone who would like to check my own facts. I presume that some of the additional minor discrepancies I have identified may be down to updates on the information provided, and I choose not to highlight these. The relevant section relates to the number of Year Seven PP children offered places in each grammar school in the country for September 2017 entry. 

The relevant data comes from two columns in the database, the first simply detailing the number of offers of places to PP children. A separate column gives the number offered places who qualified only through priority policies

The substance of the two key questions according to the database was as follows, although I accept the actual wording to schools was different. 

a)which asked for the total number of Pupil Premium (and/or Free School Meals) children offered places at grammar schools for September 2017.
       Database Answer: 565 children, with 22 of the 86 schools which replied to this question giving a zero response. The reality is that every one of these schools admitted multiple PP children according to the January census, and it is simply unbelievable to suggest large waves of PP children dropped out of grammar school between offer and starting school three months later.  These figures are close enough to those quoted in the article to make clear this is the response being attempted, although those 86 schools actually admitted 1098 PP places between them. The gap between these two alternatives for the same statistic is far too high to be possible in practice, which is precisely what triggered my initial comment last month.  
 
However, the question described in the original CF article is (b), the answers to which should have formed the basis of the original CF article, but which were replaced by the answers to (a).
 
b) which asked for the number of Pupil Premium (and/or Free School Meals) children offered places who were ‘only’ eligible through the criteria. i.e. who had only been offered places because of their PP status.
Database Answer: 70 children, with 19 of the 35 schools that replied giving a zero response. 30 of these 70 children were claimed by Dartford Grammar School for Girls which is clearly another mistake (total number of PP offers for the school is given as 31) but is just one further indicator of the worthlessness of the data, and astonishingly unnoticed by the database compiler. A simple inspection of the data in both columns compared with the relevant census data shows that, yes indeed far too many schools misunderstood the questions and, with the small number of schools replying to the second question, the whole exercise should have been junked. As admitted, it is clear that both questions are more complex and badly worded than my suggestion above. So, to go ahead and present the outcomes as fact is quite simply disgraceful practice. 

In summary, the numbers in this second set of data, the crux of the whole matter, are not mentioned anywhere in either article. Further, whichever reading is used, it takes the figures are so wildly astray that they are useless for drawing conclusions as CF has done twice.  

The comment in my previous article last month, to which CF took great offence, briefly and generally touched on the reality that 2078 PP children took up places in grammar schools in September 2017, whereas the claimed figure of 574 (565 in the database) from 80 (86 in the database) grammar schools indicated far fewer. Although an important discrepancy, of course, it comes no way near the sum of mistakes made by the organisation which has subsequently emerged.

Another running failure is that there is also a difference between the number of pupils offered places at a school, and the number admitted, as identified through census data, which CF regularly fails to notice or pick up. It is not credible to believe the scale of the difference could be the reality. 

The new CF article
The first paragraph of this alleges that I ‘accuse CF of falsifying data. We refute these allegations and object most strongly to the implication that anyone who is a part of Comprehensive Future would alter or fabricate figures supplied in response to an FOI request’, which moves it all up a notch and of course I didn’t. It is not that the figures were altered. It is now clear that whilst CF has used ones that have no relation to reality, this is probably not deliberate fabrication but an inability to understand their own data! The second paragraph proves to CF’s satisfaction that the discrepancy I found, following their challenge to me, was all down to schools’ confusion and inability to understand the questions, although this was not noticed in drawing up the original database. There may have been confusion brought about by incomprehensible questions, but the fact that no one noticed or worked out that as a consequence the data was rotten and unusable is unforgivable. The subsequent mistake of even using the wrong data from their own findings to come to false conclusions takes the failure to a higher level. ‘Garbage in, Garbage out’.

As it tries to explain away the problem, the new article continues: ‘However, these two FOI requests were asking very different questions. The earlier FOI asked grammar schools how many pupil premium pupils were admitted through their disadvantage priority policy’. It is just that the data quoted did no such thing! This hardly fits with the inflated claims in the latter part of the second article which goes on to provide a lengthy self-justification of its methods, including: ‘We ensure our published facts and figures are checked for accuracy and clarity by an appropriate member of our team before publication’ which they patently did not in this case.

The article then blames schools for the matter: ‘We now understand that the question we asked in our first request may have caused confusion to some schools, and also that interpreting the data needed a clearer understanding of the exact information provided’, without noticing they have created the problem themselves with what would appear to be incomprehensible questions.  I can see no problem with interpreting the exact information from the database if it were correct and valid, but yet again, ‘Garbage in, Garbage out’.

The article goes on: ‘We have learned through our Freedom of Information request that many grammar schools have admission priority policies that simply fail to work, or that admit just 2 or 3 pupils. This seems extraordinary and deserves public attention. We believe a pattern of failure on this scale deserves proper investigation’. This shows another complete misunderstanding of how admission policies work, with many of these schools having already picked up numbers of PP children through their normal admission policies as my original article demonstrates. I do agree, however, that failure on this scale by CF deserves proper investigation! We are also told that CF has ‘James Coombs, an experienced data analyst and statistician, on our steering committee to advise on our use of data. Currently studying for a Master’s in Data Science, James has substantial knowledge of FOI matters, as well as expertise in the interpretation of data’. I hope he was not involved in this travesty.

Background for those few keen to learn more
The recent exchanges arose only because I was approached by email by the organisation’s Campaign Officer, Joanne Bartley, who challenged my initial statement but asked that our correspondence be kept private. I am therefore absolutely astonished that when, after very civil email exchanges between us concluded and she told me to expect a response from the CF Chairman in writing via her work email, the organisation has chosen instead to address the matter in this very public way, taking the opportunity to falsely and publicly criticise me in this way, highlighting their own inadequacies.   

Joanne Bartley, Campaign Officer for CF is the author of the database that held the information, although she still appearing to be unaware of the nature of the key questions on which it was based. Otherwise, she should have explained this from the start. Her database makes it blindingly obvious to me what the problem was, although the second article shows CF still does not. The fact that she wrote to me stating: ‘22 grammar schools told us that they didn't offer a single place to a PP child’ confirms a total misunderstanding by her of her own data. I have written before about such matters on the now-defunct Kent Education Network website. 

In our correspondence, Joanne had offered me a variety of alternative explanations for the discrepancy, none of which stood up to scrutiny, although mainly repeated elsewhere in this article. She tried out that there was confusion between the dates the information had been collected and the school census the following January for the same cohort, that could explain the difference. She also tried ‘the request was sent years before the date you're talking about (untrue). We didn't use the data for quite some time - mostly because it was an over complex request’. When I demolished these, she tried ‘almost everyone involved in CF are volunteers so they deserve some understanding’. Then ‘Maybe the grammar schools misreported? Maybe some didn't understand the question? (suggesting the problem was their fault rather than CF's) ‘ When I quoted an article in the leading educational website Schools Week, supporting my data, CF then wrote to the website where the article now explains that ‘the figures provided for this piece are not accurate because they were based on a poorly-worded freedom of information request which they say was misinterpreted by schools’. It appears that Schools Week has also been wrongly informed by CF! I also came across an article on the BBC website that quoted the same false interpretation of the data forcing them to follow up with an apology as they had wrongly stated that a named school had no PP children. This BBC item also reproduced the same ‘inaccurate figures’ which at the time of writing still stand.

I chose not to respond to the various implicit threats offered in Joanne's emails about my breaching the confidence of our emails (although I never agreed to this!), as I had no intention of taking this public at the time. In passing, I have no idea what is meant by ‘I would respectfully like to remind you that comments and data shared with a flag that they are 'in confidence' should remain soAny journalistic outlet that published or commented on information shared confidentially, would lose trust, and if it became known this would damage their reputation and I expect they'd find people less willing to share information in future.’ I have seen no sign of such a flag even if I knew what it was. There is no indication of what one could do to explain what should happen if CF broke the confidentiality by airing the matter on a website in this way, contrary to their declared intention to respond by email. 

Throughout my aim had been simply to try and understand what the problem was which had provided such negative publicity for grammar schools in this case. I accept that one of the aims of Comprehensive Future is to try and bring about the demise of the selective system which it has every right to do, but surely using false data in this way, knowing the data is not safe, and then sticking to it when the truth is pointed out, is itself very wrong.

And Finally
The second half of the CF article appears to be written mainly to advertise its self-justification for its wonderful approach to data, in contrast to my approach and that of other organisations, It finishes with ' We already make the vast majority of our FOI requests openly via the What Do They Know website and will continue to do so. This approach means that our work is in the public domain for scrutiny, and it also means that it can provide useful evidence for anyone interested in the problems around 11-plus testing and selective education'. I am not sure who that is intended to impress but recommend those interested to visit the What do They Know website, and for example enter '11 Plus Testing' as recommended, in the Search Engine. This produces some 6000 results, each following through to a number of exchanges of different natures, impossible to scrutinise without a vast amount of effort. Under this heading, 'Great Crested Newt License Documents' is not untypical of the range of documents thrown up. A representative of CF has regularly criticised me for not making my vast amounts of information available in this way, rather than my own approach of analysing the key outcomes presented in my information articles explicitly for the benefit of browsers.  For example, you will even find an amplified version of CF's own FOI on census data for PP numbers in grammar schools via the link here. A typical example is Kent Test for Entry in September 2020: Further Analysis, a non-pejorative article on material central to CF, packed with data and other information. Now compare that with what you can find out on 'What do You Know' on the same theme. Frankly, it is also fantasy to regard the latter as a means of scrutiny as CF recommends, or as a source to track down information, although it does remain invaluable for the ease with which a researcher can place an FOI. And finally, finally, for those who have got this far, few articles of mine fail to provide relevant tables of data, so here you will find a list of the grammar schools with the highest and lowest percentage of PP children in Year Seven last January. It throws up some interesting patterns for those with an awareness of grammar schools across the country. Whilst CF appears to have an aversion to publishing facts, perhaps the new statistician will be covering this major omission. 

He could start by publishing the offending database so that CFs work is indeed in the public domain for scrutiny so that each grammar school can see how it is portrayed, and followers of the website can see how useless it is. 

Of course, if Joanne Bartley had not picked me up on my language in the first place, none of this would have come out!  Her comment below makes interesting reading. 

Read 415 times Last modified on Monday, 20 July 2020 13:34

5 comments

  • Comment Link Sunday, 12 July 2020 19:21 posted by Simon Caller

    I support the aims of Comprehensive Future, but your site treads a carefully neutral path through the controversy. THe scandal you have exposed certainly damages the reputation of CF and I wait to see how they will put it right. PETER: In practice, there is no controversy as there hasn't been a political will to abolish selection for many years. I remember meeting the Labour Schools Minister at a private meeting in around 1999 at the time of 'parental ballots'. He told me that gramamr schools were safe under Labour's watch. The following year David Blukett, Secretary of State said: 'I'm not interested in hunting the remaining grammar schools ... I'm desperately trying to avoid the whole debate in education once again'. It has been a fringe discussion ever since.

  • Comment Link Sunday, 12 July 2020 19:08 posted by Gary Radford

    Why do you always manage to come up with some data nuggets? Your grammar school lists via the links at the foot of the article should raise many issues. Pity that Comprehensive Future in their obsession with doing down grammars didn't stick to the facts rather than inventing the 'story' you have destroyed. If only they had published it themselves, instead of expecting interested parties to go hunting through What do They Know. What a ridiculous idea even if one knew it was there and why should one!. PETER: THe first two sets of data are in fact from the CF FOI, but they didn't do anything with it. However, it set me on the path to demonstrating the falsehood of the 'information' in their articles which should now be removed.

  • Comment Link Friday, 10 July 2020 09:38 posted by Gerry Francis

    What a bloody cheek from Jo Bartley. This article shows an almighty cock-up from Comprehensive Future. So what do they do? certainly not apologise or investigate. Instead, try and dismiss the whole thing, but in case this is spotted move on to tactic two. Divert attention to some trivial matter no one will be interested in. Congratulations and keep up the good work. I wait in eager anticipation for the explanation and apology from CF!

  • Comment Link Thursday, 09 July 2020 21:38 posted by Joanne Bartley

    Peter, you've wasted far too many words on all this, please just add this final comment.

    1) You were sent a spreadsheet without the accompanying FOI questions. These are needed to make sense of the data properly. I did not have them at the time I shared data with you.

    2) The reason it matters that CF's data is on the WhatDoTheyKnow.org website is not for random searches as you say here. It is so that people can see our questions asked, and our data recieved, whenever they query it or want to learn more. This public scrutiny avoids all these debates and misunderstandings. Our data is now always open for checks, so everything we do is clear and you don't need to write long articless speculating about what we did, asked or thought. :) PETER: A fascinating total non-response to the key issues in my article summarised by 'the authors of both CF articles have completely misunderstood their own database and quoted from the wrong column in it, as explained below!' I would have thought this was worth a mention, but as the evidence is solid, perhaps you were best to ignore the vast majority of the article and focus on the throw away tailpeice of the article. . Why no repeat or retraction of the false allegation CF has made about me?

  • Comment Link Thursday, 09 July 2020 11:33 posted by Richard Jackson

    What an amazing article, completely destroying CF's integrity and competence. Congratulations.

Leave a comment

Make sure you enter the (*) required information where indicated.
Basic HTML code is allowed.