Growl: Flawed Narconon Study Gets Published Substance Abuse Treatment, Prevention, and Policy | Full text | The NARCONON drug education curriculum for high school students: a non-randomized, controlled prevention trial Please -- all anons with a research background -- tear this to shreds. I'd like to point everyone toward the demographic bias in this study. The gender bias between the control group and the "drank the narconon Kool-Aid" group is so fucking obvious it makes my eyes hurt. Put briefly: this study is massively pre-loaded to yield the results they wanted. If this needs explaining, I'm happy to do so... but I prefer that other Anons with a research background critique this study on their own. Hint: males in the same age group are more prone to risk-taking behavior. See the table at: Table 2 Why is this important? If left unopposed, Narconon is going to quote this study over and over again to "prove" their methodology is scientifically based. We can clearly show it's yet another load of crap.
Re: Growl: Flawed Narconon Study Gets Published Apparently the control group wasn't given any curriculum, so any improvement in the drug ed group may be from the placebo effect (i.e. after taking the narCONon program, they know they're not supposed to do drugs) The survey results tend to be very close to '1' (used substance zero times in previous 30 days). The sample size probably gets much lower then ~1000 if you discard all the people who didn't use at all. I suspect the test subjects didn't understand the questions, because in the control group alcohol use went down between the initial survey and the 6 month survey, but 'being drunk' went up. Even with the crappy experimental setup, the narCONon group only improved a couple percent. But NarCONon claims what, 85% success rate?
Re: Growl: Flawed Narconon Study Gets Published don't really need to say much more but I will because I missed the acknowledgements too So a study funded by ABLE (a scilon front group) finds Narconon (another scilon front group) to work. Funny that.
Re: Growl: Flawed Narconon Study Gets Published I love that they cite psychiatry journals in the reference list.
Re: Growl: Flawed Narconon Study Gets Published What a load of skewed bullshit -- check out the huge age discrepancies, as well. Guess that's another advantage of Scientology's ginormous bank account: they can fund their own badly-designed 'studies' for their front groups in order to trick the public into thinking they're legit and never need any reviews that are truly third-party.
Re: Growl: Flawed Narconon Study Gets Published This "study" discredits itself. There is zero empirical evidence involved and the extremely vague/obscure selection method cannot be reproduced by independent third parties. From a scientific standpoint, those "results" aren't worth more than bar room hearsay. tl;dr not empirical + no reproduction potential = not science (by definition)
Re: Growl: Flawed Narconon Study Gets Published It'd be interesting if there was a way of compiling their fail into a nice easy-to-understand chart. Something to put on the 'Scientology lies' type handouts.
Re: Growl: Flawed Narconon Study Gets Published Stick to bad set designs scientology. Its the only thing you will ever be good at.
Re: Growl: Flawed Narconon Study Gets Published So, who will write a comment on this article or to get someone writing it ?
Re: Growl: Flawed Narconon Study Gets Published don't anyone stand down because of my comment, but I will be sending this study to several friends/contacts in the Addictions field; nothing will get responded to very quickly however... (FUCK I HATE THIS CULT!- sorry just had to say it yet again)
Re: Growl: Flawed Narconon Study Gets Published Junior Research Anon reporting in. Unsurprisingly this study has numerous flaws: - Doesn't define key concept - definition of drug alters according to context. - Extravagent claim re: number of youths newly involved with drugs each year - Doesn't adequately explain how subjects were assigned to groups or how schools were matched for similarity (socio-economic status of students, nature of their location) - Doesn't indicate how representative chosen schools are of schools across US, or how they were chosen - No 3rd party assessment of uniform delivery of treatment - Doesn't indicate whether any of the students had received previous drug education - Is entirely reliant on self-report to assess efficacy of treatment - Conflates "agreement" with "correct understanding" when assessing recall of program - Inappropriate treatment of data - the data collected is ordinal not interval (even the Likert-style categories for drug usage cover widely differing ranges of usage - it cannot be claimed that this data is pseudo-interval). This implies that it's a nonsense to talk about means and even worse to conduct an ANCOVA... - Leaving aside whether means are appropriate, the reader is not given any measure of variance within the groups therefore can't assess whether the groups were sufficiently similar to make comparison reasonable. - There are errors in practically all of the data tables e.g. Table 2: Based on Gender N=779, but based on age N=795, the accompanying text states that N=726. - The adjustments made for ANCOVA are not shown therfore whether they were reasonable cannot be verified - N changes again in Table 4 - 809 this time... - The veracity of some of the self-report data needs to be questioned. It would appear that even in the Drug Education group the use of Crack is nearly as prevalent a prescription amphetamines (I take it this refers to Ritalin etc, but isn't explained). - The differences in drug usage between the groups are at best marginal - this makes further data on the within group variance essential. It is missing... - In table 5 points D7-D9 are incorrectly reported: the control group subjects believe the risk of using marijuana and drinking is greater than the treatment group, but it is reported the other way round. - The text incorrectly reports that those on the drug ed program were more likely to say the could resist pressures to take drugs, whereas Table 9 item 22 shows the opposite to be true. - Statements about Narconon's corporate structure are not relevant to the research question - Conclusion about "reliable reductions" is unjustified - Conclusion about students' actual change of behaviour is unjustfied - Statement that Narconon fills vital need in substance abuse prevention is unjustified I'm planning to send some comments to "Substance Abuse Treatment, Prevention, and Policy" to query its policy for the assessment and acceptance of papers. This one even contains spelling mistakes... (the problems of 'dinking', for instance), which suggests it's not really been reviewed. I'll send a link to Dr Ben Goldacre - The Guardian (UK Paper)'s Bad Science columnist to see if he's interested in this pseudoscientific drek. tl;dr - the word shoddy, doesn't even come close.
Re: Growl: Flawed Narconon Study Gets Published Right. That's the first thing that struck me (and I do have a research background). You can't (or shouldn't) measure the effectiveness of "Narconon drug education" versus NO education. You should compare it to other, conventional drug education programmes. NarCONon often claims 80% or so based on a single study that was done in Sweden by a social worker there (i.e. not a professional researcher). This study was deeply flawed. To begin with, the statistics were those given by the Narconon staff themselves. And the follow-up on whether people were drug-free or not a year later consisted of a single phone call and asking them "Are you currently using drugs". Even so, if you look at the actual numbers, Narconon failed even in this study. Out of 65 addicts, only 14 "completed" the program. Out of those, 11 *claimed* to be drug-free a year later. From this, they claim a 11/14 success rate. "80% who complete the program are still drug-free after a year!". Of course, AFAICT, 'completing the program' is one of the biggest problem in drug-rehab programs in general. So, obviously the more important number is the overall success rate. Which in this case would be around 17%. (Of course, you can't really give percentages for such a small group, the variance is too big; it implies more significant figures than there really are)
Re: Growl: Flawed Narconon Study Gets Published excellent work evaluating the study;;;two extra points to add when I just quickly glanced at it - 1) what are the credentials of the people who did the study? What makes them valid researchers in the field? 2) Who published this? Is this a peer reviewed journal? edit: LOL!! : - There are errors in practically all of the data tables e.g. Table 2: Based on Gender N=779, but based on age N=795, the accompanying text states that N=726. Hilarious!
Re: Growl: Flawed Narconon Study Gets Published Good call on Ben Goldacre. I've just posted this on his forum as well.
Re: Growl: Flawed Narconon Study Gets Published Sent the following to the 3 Editors-in-Chief of this Journal - will see what they come back with. If nothing, then will raise with Biomed. --------------------------------------------- Dear Sirs, Madam, I'm writing to raise a number of concerns regarding a paper published by your journal by Lennox and Cecchini (The Narconon drug education curriculum for high school students: A non-randomized, controlled prevention trial). The paper contains several errors, flaws and unsupported conclusions to the extent that I am surprised that the paper was passed for publication by your peer-review process. Could I invite you to peruse the list below and let me know whether you can appreciate my surprise that this paper was published in your journal? I’d be grateful if you could let me know whether the peer-reviewers of this work considered the points below and if information is available as to how they were explained to the satisfaction of the reviewers? I’d be grateful if you could let me know if you consider the points listed below invalid, or if they are valid whether you believe it is appropriate that a paper containing these shortcomings has been published in your journal. Yours, David Catt Errors: In Table 2, the number of participants, if calculated by gender is 779, whereas if calculated by age it is 795. The text explaining the contents of the table (p 3.) states that this number should be 726. In Table 4 this value has once again changed, this time to 809 participants, which surprisingly is 83 more participants than the authors say took part in the baseline assessment and 6 month follow-up. The data collected is treated as if it were interval, which it is not, since the ranges covered by each category vary widely. The level of measurement achieved is ordinal therefore comparisons based on the mean are questionable. In Table 5 the direction of difference between treatment and control groups is incorrectly reported for points D7-D9. If this comparison is between the attitudes of the groups at the 6-month period then it is clear that the control group views these 3 activities as riskier than the treatment group. The baseline values for attitudes are not reported, therefore it is not possible to determine if the analysis is correct. The text on page 11 states that “students who received the curriculum were more likely to say they could now resist pressures to take drugs compared to those who did not receive this program”. Table 9, item 22 shows that the opposite is actually the case: at 6 months 78.8% of control group agreed, compared with 74.5% of the treatment group. Spelling error 'dinking' on page 7, first paragraph. Flaws: The paper does not explain how the schools were chosen or how they were assigned to treatment conditions. The authors state that attempts were made to match demographic composition, but it is not explained how this task was performed or what criteria were used. These issues prevent the internal validity of the study from being fully examined, would prevent the study being replicated and do not allow readers to assess how representative are the groups of the US high school population in general. Even with the limited information provided about the control and treatment groups it is clear that fundamental differences exist: - The control group has a much higher percentage of males than the treatment group (78% compared with 46%). - The mean age of the control group is approximately 6 months higher than the treatment group. - The ethnic composition of the two groups varies – control group is 31% Asian, compared with 17% for the treatment group, there are twice as many Hawaiians in the treatment group than the control group. Given that the groups vary in several characteristics and that other key information is not presented regarding their assignment to conditions we cannot be confident that comparisons between them are valid. The study states that confidentiality was maintained as to which students had supplied which answers. Given that the reported number of participants varies throughout the paper (e.g. N=958 in table 3, N=809 in table 4) how can we be sure that the comparisons made are actually on results from the same groups of students at baseline and 6-month points? The statistical analyses conducted at the baseline and 6 month points appear to be based on the number of students who actually supplied results at these times – given that there are over 100 fewer participants at the 6-month point it must be queried whether comparisons between the results at these time points are valid. The study compares groups of students receiving the Narconon program with those who receive no drug education at all. Given that the aim of the study is to evaluate Narconon’s efficacy, why is no comparison made to other student drug interventions? The absence of this condition prevents an assessment being made as to whether an observed reduction in drug usage is due to the Narconon program or is simply the consequence of drawing students’ attention to the effects of drug use. The paper does not report any measures of variance within the participants' drug use (e.g. Table 3) and therefore it is not possible to determine whether the groups are sufficiently similar at the start of treatment to warrant comparison. The reader cannot tell whether drug use is limited to a small number of heavy users within the groups or is uniformly distributed. This is particularly important as the authors have elected to conduct an ANCOVA. Requirements for an ANCOVA, in addition to interval or ratio data, are as follows: Error terms must be equally distributed, variance components of separate treatments must be equal, regression slopes must be parallel, linearity between covariates is necessary. (Handbook of Regression and Modelling, Applications for Clinical and Pharmaceutical Industries, Ed: S-C Chow, Chapman and Hall). There is no way of telling whether these assumptions have been met and given the inappropriate prior treatment of the data it seems unlikely that the authors have taken this into account. This renders any conclusions based on the data suspect. The study is entirely reliant on self-report for its results and there is no way of confirming whether the results reflect an accurate statement as to the students’ drug usage. The veracity of the data must be questioned as it appears that there is only a marginal difference in the student’s usage of marijuana and crack (for example 1.22 v 1.07 in table 3, Drug Ed group). The study conflates agreeing with the views of those presenting the program with “correct understanding” when assessing recall of the program’s content. It is entirely possible that a student could have perfect recall of the program’s content and have come to a differing conclusion that would inform their attitudes as reported in Table 9. An example of this is question 13, regarding the intended target of Alcohol ads: Why would an answer indicating that alcohol ads are targeted at those who are legally entitled to purchase these products be considered incorrect? The study does not clearly define the concept that is pivotal to its research i.e. what substances are considered drugs? The program asks questions regarding illegal narcotics, but also appears to include some prescription medications. Is it the case that the researchers consider some medications prescribed by a qualified physician to be in the same category as crack and heroin? In the text (page 11) the dangers of having a dichotomy between “good” and “bad” drugs are discussed. It is unclear as to what is the scope of this definition – is it the authors’ intention that medications such as aspirin, ibuprofen or warfarin be included in this category? If this is the case it must be queried whether it is responsible to seek to encourage attitudes to drugs that may affect students’ continuance with prescribed treatments. One of the researchers was the Executive Director of a Narconon center. The programs were delivered by Narconon employees and not subject to assessment as to the standard nature of program delivery by an independent party. Given the partial manner in which data has been presented (see above) and the absence of information given about the selection of schools and their allocation to groups, this cannot be considered an open, independent study. Unsupported Conclusions: The opening statement that the Narconon program “has thorough grounding in theory and substance abuse theory” is questionable. Numerous specialists in the field of toxicology and pharmacology have suggested that the contrary is the case. An example of this being Prof Michael Ryan’s statement in the High Court in Dublin that Narconon’s techniques are “neither medically safe, nor scientifically justified”. Prof Ryan is Head of the Department of Pharmacology, University College, Dublin. The statement that the program produced “reliable” reductions in drug use is not supported. It must be noted that the control group also showed reductions in their reported drug use in 7 from 21 assessed categories. The use of the word reliable is inappropriate in this case – given the flaws outlined above and that the reductions shown were marginal. The statement that Narconon’s program “fills a vital need in substance abuse prevention” is not supported by this research. No comparison was made with any other drug education programs therefore it cannot be stated that a vital need is being met by this program that would otherwise be left unmet: it is entirely possible that other, more effective programs are available to address this situation.
Re: Growl: Flawed Narconon Study Gets Published That is some awesome work you've done. +9000 internets to you!
Re: Growl: Flawed Narconon Study Gets Published In their tables they also list "Tranquiller"...lulz Maybe a Venusian Train Killer?
Re: Growl: Flawed Narconon Study Gets Published Davey - awesome letter, well written. I read the whole thing. Good work. Please post a reply when you get it - I'm following this thread with a lot of interest. I deal mainly with medical studies - credentials are important, so you know if the people conducting the study are qualified to do the research or not. How can we tell if these people even have a college degree? Also, some studies I read have a disclaimer/disclosure of potential conflicts of interest, like when a doctor involved in a study is consulted by a pharm company or something. I don't know if this stuff is required or not, but it seems like best practice for published research. I've only taken a couple graduate research/stats classes, so I'm not an authority on this.
Re: Growl: Flawed Narconon Study Gets Published I would love to see how this study rack ups against DARE or any other drug education program.
Re: Growl: Flawed Narconon Study Gets Published I got a response from one of the journal's editors - it's of the "Meh, so what" variety, but does include an invitation to debate the paper in the journal. He copied in one of the study's authors, the former Narconon Center director, so it will be interesting to see if she joins the debate and to ask her more about the psychiatric literature that she has cited. Some initial points about his responses marked >> in the text below. I'll thank him for his response, make a few more comments and head to the discussion page on the journal - I invite all Research Anons to join me. We could particularly use backup from both an expert on the use of ANCOVA and someone with a broad knowledge of research in this field to say whether it is indeed mostly as bad as this (as suggested by the Editor). I'll send all of this across to Goldacre - he might be just the man to help us out here. -------------------- <<Removed mod's comment as a courtesy - should have asked for permission beforehand>>
Re: Growl: Flawed Narconon Study Gets Published Mailed the following to Ben Goldacre (for those of you unfamiliar with him see Science | Science | guardian.co.uk) ----------------------------------- Dear Dr Goldacre, I'm writing to send you an example of flawed research that will be cited in support of Narconon's activities within the Education system. This research, published in the journal Substance Abuse Treatment, Prevention and Policy Substance Abuse Treatment, Prevention, and Policy | Full text | The NARCONON drug education curriculum for high school students: a non-randomized, controlled prevention trial, came to my attention since I have taken an interest in the links between Narconon and Scientology as per this article in The Times: Revealed: how Scientologists infiltrated Britain's schools - Times Online I am concerned both about the unscientific background of Narconon's methods, which have been widely derided by toxicologists and physicians, and about Scientology's use of a front group as a means of gaining access to schools. Narconon's literature makes a great deal of studies that have been conducted to prove their program's effectiveness, though the soundness of these studies and their interpretation of results is highly dubious (For a summary please see: Narconon's success rates). With this in mind I reviewed the new study and found it to contain errors, flaws and unsupported conclusions - my mail to the journal's editor and his response are shown below. Given that it is likely that Narconon will use this study to persuade the authorities that they should be given further access to schools, could I ask if you would be interested in reviewing the study yourself and providing your comments? If you would like to discuss Narconon further or are interested in receiving a copy of further correspondence on the current study please let me know. Yours, David Catt
Re: Growl: Flawed Narconon Study Gets Published Well done. Another option would be to contact people promoting programs of this type and ask them to have a look at it. At least one was mentioned earlier.
Re: Growl: Flawed Narconon Study Gets Published Got a reply from Ben Goldacre - he agrees the study is dodgy, but is reluctant to take this on given that he's already besieged by some of the firms whose research he's debunked and he's aware of the scientologists' standard practise... Sent him the mail below, agree that we should contact DARE etc. --------------------------------------------------------- Dear Dr Goldacre, Thanks for your reply - I can understand your reluctance to take this one on. We too are aware of the vicious nature of the scientologists and that's the reason that many of us have adopted anonymity as a defence against them. Being a figure in the public eye and criticising scientology is an entirely different kettle of fish, though their tactics are becoming more widely documented and opposed within the mainstream media. Just for your information, Anonymous is holding a protest at scientology locations here in the UK and across the world on Saturday (May 10th) against their use of the "Fair Game" (Fair Game (Scientology) - Wikipedia, the free encyclopedia) policy to stifle criticism. If you're interested in who we are and why we campaign please visit Enturbulation.org Scientology Activism and Organization. [embarassing fan mail sentence removed...] All the best, David Catt
Re: Growl: Flawed Narconon Study Gets Published Great work Mr. Catt. I am very impressed by the emails you have sent to date. Please keep us updated.
Re: Growl: Flawed Narconon Study Gets Published Very interesting, if not somewhat disappointing response. Not overly surprised that they played it down though - if they admit to the study having major flaws then they would have to admit there's something really wrong with their peer review process
Re: Growl: Flawed Narconon Study Gets Published Biomed Central is one of the new open access "author pays to publish" journals. If this turns out to get retracted or is found to be foul then it will be pretty big news in the scholarly communications community.
Re: Growl: Flawed Narconon Study Gets Published Sent this to the editor (and copied in the author). I'll start preparing the comment, but do we have anybody who knows ANCOVA inside out? I feel that their usage of ANCOVA is unjustified in this case and that their treatment of the data is sketchy, but do we have any senior research anons who can back this up? Does anyone know a sympathetic social scientist / statistician who can give this a look over? -------------------------------------------- RE: Lennox and Cecchini - SATPP 3/1/8 From: David Catt (davidcatt@live.co.uk) Sent: 07 May 2008 16:35:53 To: Stephan Arndt (stephan@avalon.net) Cc: macecchini@ca.rr.com Dear Dr Arndt, Thank you for your response and for your invitation to discuss this research further within the comments section of your journal. I will post a comment highlighting some of the concerns I have over the soundness of this study and would very much appreciate the authors' response. One quick comment on your response - I'd made the point that the number of participants varies throughout the Tables contained within the study and does not tally with how participant numbers are reported in the text. You suggested that this may reflect drop-out rates or that not every participant answered every question. I'd suggest that this doesn't account for why in Table 2 there are more seemingly more participants based on their age than based on their gender. Additionally, I'm struggling to see how there are 81 more participants in Table 9 than the authors say took part in the baseline and 6 month conditions - this implies that 81 people's drug use data was included in results assessing whether their drug use changed who had not had it assessed as part of the baseline. I'll include this in my comment so we can get an understanding of the authors' rationale. Yours, David Catt
Re: Growl: Flawed Narconon Study Gets Published Here's a link to one of the author's CV: http://riklennox.com/Richard Lennox cv October 2007.pdf The CV is from 2006, but I'm guessing an updated version would have new entrries under "grants and funding" from any number of CO$ front groups.
Re: Growl: Flawed Narconon Study Gets Published Sorry for this threadhogging, but to keep you posted, got another mail from the ed. It had suggestions for the tone and content of the comment (don't come in with all guns blazing). Interestingly he said that this manuscript had stirred up some controversy - I've a feeling there are others who have concerns as well.
Re: Growl: Flawed Narconon Study Gets Published Just reading through the study now, after having read the thread... Davey, you rock! I didn't read a single "meh" in Arndt's response. Not in the slightest. He's taking you on point-for-point and playing devil's advocate. That, alongside his invitation to continue correspondence indicates his own interest in continuing the debate. The fact that there are still holes to punch through in his responses can be interpreted as an open invitation (and a prep for arguments against). His advice to be measured was unsolicited = means he wants to keep this going. (Off to read the rah-rah-narCONon blather. My response won't be based on stats, sorry to say. No ANCOVA either.)
Re: Growl: Flawed Narconon Study Gets Published Maria Cecchini She's Ex SO, Ran the Narconon in Boston, not sure if she still does. Here's her completions: Maria Cecchini - Scientology Service Completions | Truth About Scientology Statistics Project
Re: Growl: Flawed Narconon Study Gets Published What? Chalk it up to naivety, but I didn't realize that there were journals where the authors could buy credibility (because we all know published=credibility) it also skews the bias, don't you think?
Re: Growl: Flawed Narconon Study Gets Published Comment up on the article: Substance Abuse Treatment, Prevention, and Policy | Comments | The NARCONON drug education curriculum for high school students: a non-randomized, controlled prevention trial Toned it down a bit so it wouldn't get bounced. Ball back with the authors. It occurred to me that this may interest Dr Dave Touretzky - I'll drop him a mail.
Re: Growl: Flawed Narconon Study Gets Published Great work Davey's Cat! I don't have enough experience with research to really help out here. Maybe this would interest Dr. Drew, the guy WBM and Tory did a radio show with.
Re: Growl: Flawed Narconon Study Gets Published wow. davey's cat, you may have 9001 internets. this thread is epic!
Re: Growl: Flawed Narconon Study Gets Published Keep up the great work Davey's Cat. This is one of my favourite threads at the moment. It could potentially become a nice addition to our arsenal if the study gets discredited. Any time a scilon mentions Narconon and how effective it is and try to back it up with "research" we can always mention how their latest research was completely discredited by the scientific community.
Re: Growl: Flawed Narconon Study Gets Published Actually treating ordinal data as interval is very common, and quite justified from an applied sense in many cases. I am currently working on something involving treating a 5-item likert scale as interval; This is extremely dodgy, but it's what I have to work with. Treating say a 9 item scale as interval is considered fine. i don't know what your level of expertise is, but I have a statistics degree. ANCOVA doesn't sound familiar (I never paid attention to acronyms), but if you think it's worthwhile I can look into it when I get home.
Re: Growl: Flawed Narconon Study Gets Published You sound like just the anon that we need involved here - it would be great if you can bring your stats expertise to bear on this. Stats theory is not my strong suit, but my perceived issue with their treatment of the data as interval is that their intervals are not equally sized i.e. on a Likert scale you can argue the the distance from Strong Disagree to Disagree is the same as from Disagree to Neutral, but on their scale you can't argue that the distance from 1 "No occasions" to 2 "1 to 2 occasions" is the same as from 4 "6 to 9 occasions" to 5 "10 to 19 occasions". I believe that this invalidates treating the data as interval, but I'd like to hear your feedback. They've used Analysis of Covariance, which checking the stats books, has a number of preconditions before it can be used correctly - there's no mention in the text of whether these requirements have been met, nor an indication of whether the procedure's been used correctly. Would you be able to check and comment?
Re: Growl: Flawed Narconon Study Gets Published Yes, I would call BS to that as well. I can't see any reason to consider the distances between those answers to be equal. Get back to you later, relevant book already checked out lawl
