We Got the Science

This is so huge. I can not wait.

This will be epic, i can't wait to see what comes of the results

Coach,

i have been doing CrossFit since early 2006 and it has astounded me to see the incredible growth, not only in population, but in advancement of human performance. i'm sure i can speak for a lot of people in saying that i greatly appreciate what you are doing and what you have made. to go above and beyond and carry out a study like this really speaks to your values. i learn so much from watching your lectures and i hope there will be many more to come in the future, especially about this topic.

like Cody above me stated, i can not wait!

I want in!

I hope they use more then just Paleo and zone in there diet study considering the longest living people on record did not eat a diet like the 2 of these.

YES!

Super excited for this.

Not sure how there gonna get alot of the more complex stuff accurately across the board mind....diet to some of the medical readings.

But these "problems" arent insurmountable and will be overcome im sure.

What a awesome study! AND it could be done overtime years to describe the changes down the road!

(Tony Black not graeme brown).

Which medical journal will this monumental work be published in?

::cough:: CFJ ::cough::

I'm so stoked, I thought coach was just hiding away for the past few months. Keep it up! That is- Revolutionizing fitness, wellness, and health.

Also- Great quote at the end.

This is going to be good!!!

Loving the approach of giving people access to information and allowing them to think/decide for themselves.

Brilliant. I wish I could be a part of the science...
...how about a little neuroscience to dump in on the nutrition - performance links...?

great work Coach! Love the ending quote, absolutely awesome!
sp

Thanks Coach!

A very compelling talk, Coach. This is a remarkable step for our community. I am excited to see what the results will show. To predict something as fundamental as how humans adapt to different elements of conditioning is priceless.

He says "aggregate analysis" not "arrogate analysis".

Super excited!!! I was literally front row for this. Coach, you are very inspiring...

So this is going to be peer reviewed, repeatable, have controls, and published in a medical/scientific jornal, correct? Otherwise, it's "Science" or "Medicine" the same way Creationism, Reflexology, Acupuncture, and Miracle Healing are.

Please help me understand why peer review and publication in a major journal are requirements for a study to be considered anything more than (paraphrasing) 'making stuff up'.

Interesting comment. Two questions:
1. how/what did they eat?
2. how would you know?

Soooo What is this Study? I got nothing from that. Im interested to hear more but need to be a little more succinct please.

Wow, what an exciting project.

Have we made predictions in advance? What are the research questions? Data mining and interpretation will be illuminating but as Greg says prediction is most convincing.

One kibitz: Crossfit is obviously on to something about diet and exercise and it will be good to know more about what that is. In addition, Crossfit is obviously on to something about individual motivation in a community.

It seems to me that ethnographic work in the tradition of medical anthropology and studies of religious life and of the military would provide additional insight.

Best of luck with the research,

Dan

[please excuse the long tirade and the poor formatting]

I love Science as much as the next guy, maybe more. It's the greatest
thing man ever figured out. And I could be the next guy in the "Crossfit
changed my life" videos - I love this stuff, it's gotten me father than I
ever would have been otherwise. So I'm firmly not in the Hater camp.

But this video makes me a little nervous. As far as I can see, it's "we're
going to examine the percentage of the population that is willing to spend
the time and money to participate in the CF lifestyle and see what works
best for them". From my experience working out at a few different CF boxes
across the country, and from having watched all the CF videos I can find,
CF is largely Americans, mostly white, and people to whom fitness is
important enough that they're willing and able to shell out $100-$200/mo in
membership fees (so either rich enough that that's not a big expense, or
devoted enough to the lifestyle that they're willing to make tradeoffs in
their life to fund it).

Will the results be extrapolatable to the world at large? Will the study
claim that they are? As far as I can see, unless the study says "This is a
study of what works best for those who meet the Crossfit demographic" it'll
be overreaching.

And "This will dwarf anything ever conceived of in Framingham or Harvard
Nurses"? For those who haven't seen them, (Framingham) and (Harvard) [both WFS].

To claim that you'll dwarf either one of these multi-decade longitudinal
studies with a single study of a particular self-selected set of exercise
aficionados is to set a particularly high bar for yourself, to say the
least, no matter how much math you throw at it.

To those who are all up in arms about the earlier point about published in
a peer-reviewed journal...the point of peer review (by statisticians,
anthropologists, medical doctors, etc) is to make sure that the study
validates what it says it validates. Although every publishing milieu has
its pitfalls, if this study can't get some serious review by some very
clueful people who have no direct stake in its results, the study won't be
valid. Don't show me a bunch of crossfitters claiming the study is
awesome, show me people who've never heard of crossfit examining the
hypothesis, methodology, data and outcomes.

If this study can do that, it'll be groundbreaking. And I really would like to see that, I think it'd be incredibly important. But the video sets some serious expectations, and it needs to live up to them.

Eric...great post..even with the sketchy formatting.

This just in: Framingham Heart Study and Harvard Nurses about to sell knee-high socks and beanies!

Justin

+ 1 for Eric.

I would also like to hear some specifics. A hypothesis before all of the data mining would be nice. Although a hypothesis would require HQ to risk being wrong when the results are found, and I'm pretty sure HQ has never been wrong...or at least admitted to it.

I definitely like where Coach is heading with this, but hopefully he has a much clearer idea (which I'm actually quite sure he does) than was conveyed in the video. It sounds like an exciting possible study, given the importance the findings could have for this community and fitness and diet enthusiasts at large. That said, you can't just go hunting through a boatload of data looking to confirm your preconceived convictions without any kinds of controls.

Were all of the trainees at a similar athletic level? For how long did trainees adhere to the particular diet? macronutrient ratios? cheat days? can you be certain of diet adherence?

This is certainly ambitious and exciting, and I hope that it is published in a peer-reviewed journal so that it can receive the credit it will likely deserve.

Eric,
A couple things. Most importantly, the purpose of this endeavor is not to create an academic paper. It is to track what is already going on and to determine best practices (think the practice and improvement of mining not geology). We're not asking anyone to perform differently than they are, just tracking what they're doing. The result should eventually be a robust and accurate picture of hundreds of thousands of people's diet and exercise over decades with the resulting impact on performance and health metrics. Very little downside here and tons of upside.

In terms of Framingham, the study was highly flawed (see Ravnskov's explanation here http://www.ravnskov.nu/myth2.htm). But the key here is that we're not trying to convince anyone of anything. We are in the business of improving human performance. The study will give us unparalleled data for optimizing how to do that. The insights we get will be fed back to the community to experiment with. Those experiments will yield more data that will be analyzed similarly. The net result will be more tools and a better vehicle for improving fitness and health.

David Smith is wrong. We don't need to create a hypothesis before mining the data. I suspect he is a high school or college student who has been given a simplistic view of academic science. We are working with experts in data analysis and chaos theory. They have decades of experience finding meaningful correlations. We needn't have foregone conclusions about what we'll find at all.

Furthermore, these notions of peer review are hilarious. Peer review and/or consensus have nothing to do with science. Every major flawed study has been peer reviewed. Something is either right, wrong, valid or bunk no matter who agrees, disagrees, understands or doesn't. Science is outside the realm of opinion.

You seem to be worried about how the study will be used. I'm not. Smart people will see it for what it is and use it as a way to tweak their training and diet. Fools will abuse it any way possible. Nothing we could do would stop that.

With all due respect, it was only BECAUSE of a peer review process that flaws are able to be spotted. Science is self correcting mechanism afterall.

Justin

Great idea
I see it starting before its even out of the gate. The haters are sharpening the long knives. Why does a large % of our country form a negative outlook when someone is just trying to improve something? Anyhow Count me in, looking for data on the over 50 crowd? I would be glad to be a cf lab rat.
Four and a half years in.
Zone diet for 4 years.
Still making gains.

R/ 16th master

Folks such as Robb Wolf and OPT have been doing this for years. I'm willing to bet that their collective findings are exponentially less biased that CFHQ's will be.

I love this idea. I think it's awesome that they are doing this.

I have one comment, however. Coach Glassman says (before compiling the data) that Ho Hos, Ding Dongs, potatoes, and grains are bad foods. Personally, I would like to see the actual data and the results of the research determine what's good and what's bad. I will agree that Ho Hos and Ding Dongs are probably not good for you, but I can't believe that rice - a staple in Asia - is as bad as Hostess products are. I'm worried that we are going into this "scientific" study with a preconceived notion of what the outcome will be. This generally makes the Scientist bend the data to give the results he/she is looking for. This is not true science.

All I'm asking is that we allow the unbiased outcome to speak for itself. If we find that pounding down 5 donuts before every workout gives the best results, then let that knowledge be known. (I'm kidding here, but you know what I mean.) If it turns out that whole grain bread isn't as bad as the Paleo diet says it is, I would like to know that.

Thanks.

Hi Tony-

Thanks for the reply. Your comment makes me quite nervous, it seems to have this "we're going to do what we're going to do and we're already right, neener neener neener" tone to it. I doubt that's how you intended it, but IMO that really is how it comes off. I have no doubt you mean well, and I really am interested in finding out more about this study, but I've got no interest in throwing stones here. Just trying to help.

Some points:

---
"We're not asking anyone to perform differently than they are, just tracking what they're doing. The result should eventually be a robust and accurate picture of hundreds of thousands of people's diet and exercise over decades with the resulting impact on performance and health metrics. Very little downside here and tons of upside."
---

I agree on the upside, and I think such data would be really cool to have, but remember what I said about the population. If you pull only from affiliates and their members, your entire population is affiliates and their members. And I do agree with your implied point that if we learn what works best for crossfitters we can help make crossfitters into better crossfitters. All I meant in my original post was to not claim you were any more than that unless you had the right population. Clearly you've got something bigger in mind than just a single survey and I'd love to see the methodology; I could well be reading too much into the video.

----
"David Smith is wrong. We don't need to create a hypothesis before mining the data....I suspect he is a high school or college student who has been given a simplistic view of academic science".
---

If you decide ahead of time what you're looking for in a massive pile of data, you're going to find it. There are all sorts of tricks one can play when looking for an answer they already have; most of the hard part of studies like you're talking about comes from avoiding those biases.

Some of the more science-oriented Crossfit stuff has been accused of having a confirmation bias, and when you say things like "we don't need a hypothesis, you must be uneducated" you only further that impression. Please remember that the crossfit demographic includes lots of people with significant education in all sorts of science, logic, philosophy, law, physiology, etc (a side effect of box membership costing $100-$200+/month). Not everyone who disagrees with you is by necessity dumber than you are.

It may well be that you need less of a formal hypothesis for data mining than you do before starting drug trials (for example). I'm not a data mining expert. But to claim that people who don't understand or agree with your point are 'simplistic' or undereducated is awfully defensive. Look, the people commenting here are CFJ subscribers, which means we're all devoted (devout?) crossfitters. We're all on your side.

----
Furthermore, these notions of peer review are hilarious. Peer review and/or consensus have nothing to do with science. Every major flawed study has been peer reviewed. Something is either right, wrong, valid or bunk no matter who agrees, disagrees, understands or doesn't.
----

I suppose the fact that thousands of mathematicians/physicans/statisticians/etc over hundreds of years have submitted to and participated in peer review carries no weight with you? Crossfit has this free market approach to things, which I agree with, but if peer review was on balance more bad than good, wouldn't the free market have choked it out by now? Surely the fact that it existed and continues to exist is an indicator of its fitness in the realm of academia?

I also think you hold different beliefs about peer review than I do. Peer review is intended to verify whether your work shows what it purports to show. It's like having someone check your figures when you do your taxes. If you don't want to undergo review by people who have no vested interest in your success it just looks like you're hiding something. "Fermat's Last Theorem" by Simon Singh is worth reading if you want to see how important peer review is even in mathematics, which is perhaps the hardest of the hard sciences. And I'm not asking for you to hand your data over to someone who actively dislikes Crossfit in order to ask them what's wrong with it. But if your want to assert that you're doing science, you need to be willing to take comments on your work.

----
Science is outside the realm of opinion.
----

Nowhere in my first comment did I say 'opinion'. That's your word, not mine, and I tend to agree with you. But peer review serves more purposes than opinion, as I said above. I've heard Crossfit described as "open source fitness", and I think that's a wonderful phrase. But if you're really going to be open source you have to be open to constructive criticism and sceptical input from supporters. To do otherwise is to do your end goal a disservice.

Anyways, I don't want this to be the start of a flamewar; that wasn't my intent in my original posting and it's not the intent in my reply. All I mean to put forth is constructive criticism, take it as you like.

I think it's less about creating an authoritative document on the subject as conducting an internal review of what's in play out there, what's working and what's not. I see a great deal of "the community has decided that x works and y doesn't" and when you look at who "the community" is, it's the people who post on the most on the message board and who put the most effort into slamming the opinions of those who dare to disagree with them.

When it comes to real world results of their methods, Robb, OPT and every trainer can only draw on the data provided by the people they work with, as large as that number is. HQ is in the unique position to be able to draw on the data provided by everyone who participates in the CF Games, both the finals and al of the qualifying rounds. That's a lot of CrossFitters.
And, it covers people of all ages, most geographical areas and all abilities, from the elite athletes who compete in the final event to the people who have a go at a local qualifier as a personal challenge.
Most affiliates do their own local version of this all the time, reviewing the progress, training and eating habits of their clients to refine their coaching. This is that local process taken to a global scale with some rigour applied to the data collection and analysis.

The final result may not be suitable for publishing in a medical journal, but it's a damn useful tool to tell HQ in an impartial way what they should be advising the affiliates to do. It feeds directly back into the L1 certs for example.

Darn good idea.
Don't screw it up. :-)

Eric -

CrossFit gyms started in garages with little to no equipment. Sure, there are many affiliates that charge over $200 / year, but everyone knows there are plenty of CrossFitter's out there doing it at globo gyms or even in their garages who have spent far less. That's the great thing about CrossFit, there is very little startup costs to incorporate the program into your life.

Have you looked at the affiliate listing? It's pretty obvious CrossFit is NOT mainly "white American's".

Additionally, you seem to have forgotten about the members of our Military who do CrossFit with little to no equipment.

Your logic is wrong and I believe CF actually caters to individuals who are not rich and can only afford some weights and a pullup bar.

FRAT!

Actually I did read quite a bit of it (just past Tony"s reply) and WOW people are quick to criticize. Nothing has even been published yet! Coach Just made a simple announcement about the REALLY COOL thing he is doing next. Save your criticism for any actual claims made and/or holes in procedure. I for one am seriously excited that this ability to analyze such a sample set even exists, self-selected or not.

I see the fruits of this as a verified starting point for my own personal self experimentation. After all, what else is science for except self improvement?

I also see this as motivation to be more consistent and rigorous in my own record keeping.

Take the inspiration, add your perspiration, and then make your own evaluation.

I intend to keep an open mind about this and await the results, but I must say that this seems like awfully familiar territory. I've watched many a video that talks about "we've got the data", "evidenced-based", and all the rest of the hyperbole that the critics love to use as ammo. Not that all "haters" need to be argued with, but it would be nice if there were some real independent (vice aforementioned confirmation bias) results that proved/disproved the methodology, at least to a larger degree than we've seen thus far. Granted, what Coach is discussing here seems to be a bigger, albeit ambiguous, effort, it would still be nice to tackle the issue that plagues most folks who are trying to fight the good fight in getting conventional attitudes changed towards fitness where we live/work. Meanwhile "we've" always got experts and data that always seem to be just on the other side of the curtain. Reminds me of a movie that I can't recall the title to, but has an exchange that goes something like this:
"We have top minds looking into it"
"Who?"
"Top. Minds."
So, who are these people?

Ryan,

I own a CrossFit affiliate. My clients are mostly white, middle class to upper middle class Americans. I have (predictably) quite a few friends who own CrossFit affiliates. I'm willing to bet (quite a lot) that most of their clients are white, middle to upper to middle class Americans. This is not to say that CrossFit as an organization or a methodology caters to those people, but it is true that most gyms will probably make most of their money off of this population.

Also, you say Eric's logic is wrong. Can you elaborate?

All of the comments and bickering are quite premature.

Unless you know what the study is, what data is to be collected, how it's going to be used, what conclusions will be drawn from it, etc etc, there is no point in arguing hypotheticals.

It could be great, it could be garbage. To quote Axl Rose, "all we need is just a little patience."

Fantastic idea. Please don't leave the results hidden behind the limited access door that is a $25 a year subscription. "Peer review is the set of mechanisms that enable scholars to have reliable access to the informed opinions of other scholars, in a way that allows that those informed opinions themselves to be subject to similar vetting" (Paul N. Courant, University Librarian and Dean of Libraries at the University of Michigan).

Sorry guys, peer reviewed is the standard at every academic institution on the planet, whether you agree with it or not. I am currently conducting research about goal theory and how it relates to a CrossFit class. It is almost impossible to cite CF Journal articles that have a good amount of science in them because they have not been peer reviewed. Wrong or right, that is the standard by which research is judged.

Open it this study up, share it with the world. If in fact is becomes ground breaking, the world needs to see it. Parent's who shop for food for their kids need to see it, school lunch programs need to see it.

Best wishes to the CF research team and thanks to Coach for the initiative to tackle such a controversial issue.

I'm glad you wrote this. This study must be done properly. Otherwise there will be swarms of RD's annihilating anything published.

Thanks for your comments Eric. I agree with many of your points. I have to disagree with your peer review section though. There appears to be a pretty large delta between what peer review is intended to do and what actually happens in the execution of it. People doing the peer review are equally subject to cognitive bias as the people doing the research. I know I'm basically just some random guy posting on the CFJ, so I'll reference someone else's brief analysis as an example of peer review in the 'health' industry...

Fourth International Congress on Peer Review in Biomedical Publication
Drummond Rennie, MD
Deputy Editor, JAMA
JAMA. 2002;287:2759-2760.

http://jama.ama-assn.org/cgi/content/full/287/21/2759
(wfs)

There's a little more in the article, but here are a couple quotes...

"Sixteen years after the initiative started, we find ourselves in the peculiar position of believing still more in the virtues of peer review, a system we know to be "time-consuming, complex, expensive and . . . prone to abuse,"9 while we acknowledge that the scientific evidence for its value is meager. Indeed, if the entire peer-review system did not exist but were now to be proposed as a new invention, it would be hard to convince editors looking at the evidence to go through the trouble and expense. This dissonance suggests that we are using the wrong tools to study the wrong factors."

-----

"In the 1986 editorial announcing the first peer review Congress, I noted the appalling standards then prevalent despite the existence of peer review3:"

"One trouble is that despite this system, anyone who reads journals widely and critically is forced to realize that there are scarcely any bars to eventual publication. There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print."

"In the last 16 years, efforts to systematize reviews and improve the reporting of trials and meta-analyses have borne considerable fruit. But an unbiased reader, roaming at random through a medical library, would find in abundance all the problems I described in 1986."

-----

This coming from the then (may still be) deputy editor of a major peer-reviewed medical journal who has a vested interest in seeing peer review succeed.

Regardless, I'm looking forward to seeing the work that CF is undertaking and evaluating for myself whether I think it's useful or not.

So I listened. Twice.

Lots of buzzwords, not really clear exactly what's going to be done.

Suffice to say that already categorizing "good" and "bad" food and saying the debate between paleo and zone is "silly", bias has already entered into this argument, UNLESS his "world class scientists from DOD and NSA" aren't listening to this speech.

If this will "blow Framingham out of the water" then I suppose Crossfit will be collecting data for the next 30 years and analyzing hypothesis along the way. Data mining can yield just about anything you like if you're only looking for correlations, not causation. Monte Carlo simulations are done all the time to figure out predictability and probability greater than random chance.

I truly hope this speech clip is just Mr. Glassman's attempt to summarize, crudely, what is really going to be done. Because some of what he says is at best contradictory, or at worse (potentially) a misunderstanding of what "sophisticated mathematics" truly is.

So if cereal is "bad" and "broccoli" is good, can they apply the CF WOD protocol to that and determine if you're more likely to be healthier? At what consumption size and frequency. We are talking about applied statistics, yes?

Maybe I'm the only one who noticed smirks on audience members' faces. Perhaps next clip might be a scientist who will more clearly explain EXACTLY what data will be gathered and what correlations will be analyzed.

Many questions, few answers...yet.

I'm not skeptical about the possibility of collecting a very large body of information from a large subject group over a long period of time. I'm not skeptical about the possibility of a variety of researchers analyzing that body of information from a variety of angles and arriving at potentially useful insights regarding the subjects involved. I like what Coach has done and what Crossfit has become and I apply the concepts in my own life and those of others when given the chance.

That having been established, I can't help but think about all of the sports medicine, biomechanics and exercise physiology studies that I have been involved with and the difficulties that I've seen with basic, foundational collection of data, not to mention the accurate and dispassionate analysis of said data.

I honestly wish everybody all the best in this endeavor, but until we get more information about what this is all about I think it is imprudent to claim that "We Got the Science". A proposal without data is not science. Data without analysis is not science. An analysis without SOME form of independent verification, control or review is not science, no matter how many people happen to agree with it. Without that type of unassailable foundation we will have no more legitimate a claim to scientific truth than many other "cults" out there.

The way Coach is talking this up it won't be enough to just win the hearts and minds of the devoted. He needs to deliver evidence that will convince the skeptical and sway opinion in the scientific realm worldwide. I really, really hope that it happens and I'd be happy to help in any way possible, but until there is something solid to refer to I've got to say that I'm uncomfortable with the vibe. I can't help but feel that there is too little humility and too much arrogance before the enormity of the task at hand.

This is exciting stuff. All CrossFitters should be stoked to be a part of something so epic. Very cool!

For sure Pat! Great Muscle-Up coaching vid today by the way. Will put it to goood use!

Maybe Justin Smith ##7 & 29, Vincent Christensen #18, Eric Osborne ##23 & 33, Mike Essman #27, and Chris Bonner #40 might look with favor on publication in Lancet.

>>Richard Horton, editor of the British medical journal The Lancet, has said that "The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability - not the validity - of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed [jiggered, not repaired], often insulting, usually ignorant, occasionally foolish, and frequently wrong."

http://en.wikipedia.org/wiki/Peer_review, citing from

http://www.mja.com.au/public/issues/172_04_210200/horton/horton.html.

To this we can add Dave Wolf's valuable reference at Comment 42.

Eric Osborne #23. Validation in science refers to, or in my book should refer to, the demonstration of empirical evidence that supports the predictions of a model. Whether that happens is a matter of fact, and has nothing to do with peer review. The point of peer review is what Dr. Horton and Tony Budding #28 said.

Eric Osborne #33. Academia indeed makes peer-review essential, not for science but as recognition for academic advancement. Industry, which advances science at rates impossible in the universities, and which employs many more PhDs, will fire a scientist who reveals what should be industrial secrets.

We can't know what articles failed peer review or why. We do know about peer review that failed to detect junk. Read about the falsified, peer reviewed papers on nanotechnology by Jan Hendrik Schön, winner of an Outstanding Young Investigator Award who later had his PhD revoked. Read about the "Sokol Affair", gobbledygook published in a social science journal intending to break with peer review tradition. Recognize the existence of The Astrological Journal, Culture and Cosmos, Correlation, Astrology and Medicine, journals of the Astrological Association of Great Britain.

>>Contains abstracts of 91 studies, most of them empirical, from four astrological research journals. … At the time the first three journals were the world's only peer-review astrological journals devoted to scientific research … . www.astrology-and-science.com

I presume everyone knows to a moral certainty that astrology has no scientific basis. The existence of peer review in astrology is evidence that peer review proves nothing.

In 2004 Naomi Oreskes conducted a famous study called "Beyond the Ivory Tower: The Scientific Consensus on Climate Change". She wrote,

>>Others agree. The American Meteorological Society, the American Geophysical Union, and the American Association for the Advancement of Science (AAAS) all have issued statements in recent years concluding that the evidence for human modification of climate is compelling.

>>The drafting of such reports and statements involves many opportunities for comment, criticism, and revision, and it is not likely that they would diverge greatly from the opinions of the societies' members. Nevertheless, they might downplay legitimate dissenting opinions. That hypothesis was tested by analyzing 928 abstracts, published in refereed scientific journals between 1993 and 2003, and listed in the ISI database with the keywords “climate change”.

>>The 928 papers were divided into six categories: explicit endorsement of the consensus position, evaluation of impacts, mitigation proposals, methods, paleoclimate analysis, and rejection of the consensus position. Of all the papers, 75% fell into the first three categories, either explicitly or implicitly accepting the consensus view; 25% dealt with methods or paleoclimate, taking no position on current anthropogenic climate change. REMARKABLY, NONE OF THE PAPERS DISAGREED WITH THE CONSENSUS POSITION. Caps added. http://www.sciencemag.org/content/306/5702/1686.full

First, this is remarkable as an example of misunderstanding data. She did not poll scientists to count the numbers who agreed and who disagreed with the AGW model. Her paper was extracted from an essay she presented to the AAAS, and was not peer-reviewed.

Second, what she demonstrated was that none of the peer reviewed journals uncovered by her library search spanning a decade ever published an article critical of AGW. She didn't reveal how many journals were involved, but Ohio State University publishes a list of journals in atmospheric and climate sciences in their collection that are available online.

If we were to extrapolate from Oreskes' study that no such journal ever publish an article critical of AGW, and further assume that being designated a journal means that it employs peer review, then peer review has failed to detect that AGW is a fraud of unparalleled proportions.

However, we do know of published papers critical of AGW. Most famous are those by Baliunas and Soon (Climate Research) and by McIntyre and McKitrick (Energy and Environment). The reaction of the authors of the IPCC report, notably Phil Jones and Michael Mann, was two-fold: consider the offending journals no longer to be peer-reviewed, and to shun articles from those journals. Phil Jones attacked the editor of E&E. He wrote,

>>Dr [Sonja] Boehmer-Christiansen states that it is beyond her expertise to assess the claims made. If this is the case then she shouldn't be sending malicious emails like this. The two Canadians she refers to have never developed a tree-ring chronology in their lives and McIntyre has stated several times on his blog site that he has no aim to write up his results for publication in the peer-review literature. I'm sure you will be of the same opinion as me that science should be undertaken through the peer-review literature as it has been for over 300 years. The peer-review system is the safeguard science has developed to stop bad science being published.

This from a ring master of the all-time poorest excuse for science.

A journal editor's job should be screening to the dictates of language and journal style, and to the precepts of science, but not as Jones suggests to academic frills or obedience to dogma.

Moreover, what M&M criticized was not tree ring thermometry, but faulty statistical data reduction, which had passed peer review to produce the Hockey Stick Reduction that erased the Medieval Warm Period. This little fraud had the beneficial effect in IPCC's Climate Change 2001 of sustaining the AGW claim that the current temperatures were unprecedented, as if being unprecedented added any scientific validity. M&M's criticism was sustained, much to the embarrassment of IPCC. While IPCC did not withdraw the infamous Hockey Stick, it obliterated the result by publishing it in Climate Change 2007 overlaid now by a dozen other reductions.

The academic communities of East Anglia and Penn State rallied around Jones and Mann, exonerating them, and, it goes without saying, the peer review process. Neither academia nor peer review is a warrantor of science.

Because IPCC reports are self-reviewed, they are not peer reviewed. Peer review requires an independent assessment by a person wielding the ultimate, arbitrary authority not to publish.

PS

I omitted that the number of journals in the Ohio State list is a whopping 65.

Also, Oreskes' extract was, as shown, later published in Science, and presumably, therefore, was peer-reviewed.

Vogue, JAMA or somewhere in between. Peer reviewed or the Daily Onion, just get it the widest audience possible. The USDA could use some help!

I’d like to introduce myself and the CF Science Team to the CFJ community. I'm Cliff Lewis, one of the scientists leading the CF data analysis effort along with my colleagues Brons Larson (PhD Mathematics, UC Davis), and Sisinio Baldis (MSEE, University of Washington) who is leading the team developing the software platform for data collection. I am a Physicist and received my Doctorate from UCSD. I have read through the comments and can say I am very impressed by the understanding of the scientific process displayed by the group. One thing that has repeatedly struck me since getting involved with CF is the high level of competency and knowledge demonstrated regarding fitness, nutrition, and physiology. I’m sure I can speak for the team and say we are very excited to interact with the CF community in a scientific capacity.

We are working closely with CF HQ to put together the framework on how we will communicate our process and results to the CF community and how to best promote dialogue regarding those processes and results. In addition, we have started to develop plans on how we can include a robust and satisfactory peer review process. The comments regarding peer review are very well taken. Peer review, in its best construction, can be a very valuable thing in that it keeps scientists accountable and causes them to push themselves to even higher levels of thoroughness in their work.

We will be keeping the CF community in the loop (through the CFJ and other avenues) as these plans mature over the next few months and welcome any further comments, questions or constructive criticism. I believe that an open flow of ideas will produce the best non-biased and most informative results, which I assure you is the primary goal of the CF Science Team.

When Coach set his sights on dwarfing the Framingham and Harvard studies he was aiming too low. If one tries to read official reports on the results and conclusions of those studies, he is going to be disappointed. Why is explained well by Michael R. Eades, MD, famous and popular especially in the CrossFit community for his book, Protein Power, written with his wife, Mary Dan Eades, MD. Read about the "Framingham follies" on his blog. http://www.proteinpower.com/drmike/cardiovascular-disease/framingham-follies/ . For some key information about an unfortunate consequence of the Harvard Nurses's Health Study, see his blog on "Observational studies". http://www.proteinpower.com/drmike/bogus-studies/observational-studies/ . These studies are important and statistically significant, but a disaster for conventional medical health wisdom.

What Coach is trying to do may not dwarf Framingham and Harvard Nurses on the calendar scale, but it could on the number of participants and perhaps even on the number of man years. He could blow away these studies because while they were epidemiology observational studies, his plan is for an experimental study, in itself a rare event in health science. He'll be like a climatologist who can actually vary the weather.

While trainers work with their clients, the subjects, measurably and scientifically improving their individual fitness, the CrossFit health study will be tracking the conventional health parameters of diet, clinical parameters of blood pressure, some of the various blood components clinically predictive of heart attacks, body weight, performance, and other factors from vitamins to viagra. The outcomes of the CrossFit study are no more predictable than were the Framingham and Harvard Nurses studies. Possible outcomes might be evidence that improvements in the blood components lag exercise and conditioning, and that diet is secondary, again the opposite of the conventional wisdom.

Coach is optimistic about his data collection. As a scientist, I must be skeptical, but I wouldn't bet against him. The popularized observational studies relied on a questionnaire, like the Food Frequency Questionnaire that cooled Eades' enthusiasm. Coach is going to have some of the same problems about people being honest about their diet, much less even remembering what they ate. Still, that is background noise in the promised study, and a boundary condition that might be improved by data reduction techniques. In the final analysis, any results will speak for themselves.

Look for the profound result that medical parameters conventionally reckoned to be predictive of good health are instead a consequence of good health brought on by physical conditioning. I'd bet on that.

First of all this is great to do and a reason we all love CrossFit. However, I think it's totally fine for people to have doubts due to the magnitude of the study and not really seeing any specifics. I don't really see anyone on here bashing it just stating doubts. At the very least this will be an interesting read when it is all said and done. Some of my doubts include (and this is without seeing specifics and I'm sure these have been thought of): Glassman uses bold statements quite a bit, so it seems like whatever is found is going to be the new cat's meow no matter what, also it seems like the "silly debate between paleo and zone" is the ultimate motivation of the study or at least what sparked it. Also I do believe there are going to be WAY to many variables to consider to prove the validity of this high un-controlled study IMO the fact that this study is uncontrollable is largely the reason why it has never been attempted. Not trying to be a negative nancy or hate but just stating doubts.

I'll be the first to say it. The Crossfit community very gladly welcomes you and your team with open arms, just as we've done with everyone else. Duh.

Jeffrey Glassman,

"Look for the profound result that medical parameters conventionally reckoned to be predictive of good health are instead a consequence of good health brought on by physical conditioning."

That is not profound. Doctors have long recognized the need for a high quality diet and physical exercise ("conditioning"). The problem, in my mind, is two fold. First, most people are lazy. They look for the easy way out. Going from the couch to Fran is life altering but the initial mental block is huge. Second, doctors assumed, like seemingly everyone else, that they knew how to get in shape and eat healthily. ("Walk for 30-40 minutes, 3-5 days a week. A joke. Unless you're my 88 year old grandpa.)

The real benefits will come from inducing a paradigm shift about what it means to exercise, be fit, and eat healthy. Crossfit and crossfittes seem to get it. Others think this stuff is insane...

Jeffrey Glassman #53 wrote:

"When Coach set his sights on dwarfing the Framingham and Harvard studies he was aiming too low. If one tries to read official reports on the results and conclusions of those studies, he is going to be disappointed. Why is explained well by Michael R. Eades, MD, famous and popular especially in the CrossFit community for his book, Protein Power, written with his wife, Mary Dan Eades, MD. Read about the "Framingham follies" on his blog. http://www.proteinpower.com/drmike/cardiovascular-disease/framingham-follies/ . For some key information about an unfortunate consequence of the Harvard Nurses's Health Study, see his blog on "Observational studies". http://www.proteinpower.com/drmike/bogus-studies/observational-studies/ ."

Let's see... properly reviewed medical studies peer reviewed and published over many decades, or blog posts by people with merchandise to sell their point of view, who typically pick apart said studies but offer nothing in return. I'd much rather be disappointed by limitations of the former than be amused by the latter.

Justin

How can I get a job at CrossFit HQ?? This is cool stuff. I've been in Software & Tech forever and love crossfit!!

Justin Smith #57: You mischaracterize what I said to create a faux choice.

Being "peer reviewed and published over a many decades" appears not to have coaxed any useful results or conclusions from the principal investigators. The best I could find from the PIs was a list of published papers. But what would you expect from purely observational studies?

Discovery is far from the whole of science. Predicting and validating is everything.

And if the fact wasn't clear to you at the outset that these were observational studies, Eades points that out for you. He also points out that these studies had nothing by way of conclusions beyond de-correlating what were a couple of conventional cause and effect relationships.

Those studies offered no positive conclusions, and Eades, as you noted, offered nothing in return -- except, I note, understanding. On the one hand you have less than you started with, and on the other hand nothing but an explanation.

The studies were not faulty. It's just that the expectations produced in the media exceeded the studies' grasps. Whether peer review in these cases was beneficial can't be known. Peer review proceeds in secret to vague, unpublished standards or worse -- dogma.

"Discovery is far from the whole of science. Predicting and validating is everything."

Tony Budding, does this mean Dr. Glassman has a high school-level understanding of science as well? That seems pretty unlikely to me.

I'm glad to see this discussion continuing. I think it's important that concerns are expressed and hopefully addressed by the individuals in a place to act on them. I remain skeptical but am looking forward to seeing what Dr. Lewis and his colleagues come up with.

the disagreement above seems to me be about the level of hyperbole and choice of language which HQ may or may not be attaching to this study.

I would hope that everybody thinks that it will be cool, interesting, useful and very valuable to learn about the results of this study...to learn on a broader scale than we have seen to date on the impact of different forms of exercise at varying levels of intensity, different nutritional philosophies and intake and the myriad combinations therein on a whole variety of factors including
- fitness
- health
- weight
...sleep, emotional well being, work performance etc etc.

I'm certainly realy looking forward to it. I can't wait in fact.

For me, the important thing for HQ is to then be realistic and humble about how they talk about the results.

We don't need to discredit other studies, diets or forms of exercise or to claim to have found the holy grail. The bigger the claims, the more strident the hyperbole, the bigger the invitation to critics to look for flaws, holes and inconsistencies and the greater the chance of alienating some rank and file crossfitters and turning them away from the results. This study won't be flawless - we've seen above that when smart people argue about science, there are plenty of different ways for them to pick holes in the opposing viewpoint.

It doesn't need to be flawless - it just needs to contain a whole load of learning for us to pick up and use. It doesn't need to be about world domination or killing the opposition.

The only tiny part of Crossfit which can occasionally alienate me is the attitude of HQ to occasionally place itself on an island...surrounded by a perceived ocean of ignorance and incompetence. I would hate for this study to be used by HQ to build itself a mountain on that island to use to look down down and demean other conflicting views.

really looking forward to the results

Dr. James L. Chestnut wrote a series of 4 books, a part of his Move Well, Eat Well, Think Well concept, one being "The Innate Diet & Natural Hygiene" Loads of cited research and logical explaination. Greg Glassman and all of you Crossfitter's out there may find it very valuable in your lives and for this project.

Forgot to mention.. Designs for Health, a supplement company, has some great presenters and research on the affects of excercise on hormones etc. in the body for fat vs. weight loss, building muscle etc. Just another resource for you all..

As far as I know the "Sokal Affair" wasn't an attempt by Alan Sokal to break peer review in general but to point out the difficulties of peer reviewing in soft sciences. For more information read: Fashionable Nonsense: Postmodern Intellectuals' Abuse of Science by Sokal and Bricmont.
IMO the peer review is more a problem in soft science (read non-axiomatic sciences) which rely heavily on statics and the interpretation of data (with usually a huge confirmation bias). Sokal's attack was an attack on postmodernism and the misuse of mathematics and physics in postmodern literature and nothing else.
Today the field of statistical analysis has entered nearly every major scientific community from medicine to economics, where people using fancy software have no really clue of what actually is really going on and why. This is the HUGE problem.

Great video, good debate and interesting comments.

I don't know much about Sokal, but Wikipedia says:

"At that time, the Journal did not practice peer review fact-checking and did not submit the article for outside expert review by a physicist."

and

"The Sokal Affair compelled Social Text magazine to establish an article peer review process. In 1996, the magazine did not peer review because the editors believed that an editorial open policy would stimulate more original, less conventional research."

Justin

Exactly the point that I tried to make earlier. Thanks for expressing it more completely.

My bet, Jeff, is that traditional predictive medical metrics and fitness-related predictive metrics will be inextricably intertwined. There will be a positive (or negative as the case may be) interaction between the two insofar as health is concerned. I predict a "virtuous cycle" between traditional metrics (cholesterol, BP, girth) and fitness (WCABTMD) where a feedback loop exists, each driving the other to a next level of positive.

I have stated exactly this in other, off-line communications.

bingo

Dr.James L. Chestnut trains at our affiliate in Victoria, BC. Great guy.

Certainly interested in the outcome of this, but still unsure how this will be any different from all the other observational studies?

Something about using Wikipedia entries to elaborate on the quality of peer-reviewed literature seems fitting here.

I am glad to see this-Properly done it has the potential to make Crossfit one of the biggest impacts on health and fitness this century. The economic and lifespan benefits of a proper lifestyle can be transformational for generations.
Coach grew up with a science oriented background and I have no doubt that he knows the requirements for scientifically rigorous studies and results. The establishment of an inhouse department is a strong first step and again properly done could make careers for interested researchers.

What also needs to be followed is the inherent injury rates from Crossfit. ( I work out at a very well run box with superb coaching and in about 250 members there have been 10 operative rotator cuff injuries-an entirely preventable and significant athletic injury.)
By honest tracking and injury reporting across a large population-injury trends can be examined, causes identified and coaching tweaked to prevent further occurences.
This could potentially eliminate a frequent criticism of Crossfit, make it stronger,applicable to a broader population and in no way diminish its elite results.

Having worked with large and complex data sets I would suggest a few things:

1) Crawl before your walk (or run). Just getting a large sample of good data will be challenging for this endeavor.

2) Once you have a method/process that works on the data collection-side then work on hypotheses. What questions do we want to ask the data?

3) Start with "simple" questions.

4) Be as transparent as possible. Anonymize the names of participants but share as much data as you can. Look at what wikipedia has been able to accomplish as an open community.

It would be amazing to get results on the most efficient programming strategies, hrs of sleep, yoga or no yoga, what to do if you only have 10 min to workout, etc. Too many variables to count but it would be great to get some better understanding of at least a few.

I just want to know what data you'd like us to start tracking. Meals, warmups, WOD performances, weight, blood metrics, sleep, schedule variables... there are so many variables we could track. I know sophisticated statistics can find the needles in a million haystacks, but we still need to know which data to track.

I assume a typical variable to be looked at in this study is heart attack rates. The rate of heart attacks in the general population is well known, and I assume one could measure the rate of heart attacks in the Crossfit population.

Its funny to see Coach Glassman with dark hair! Or any hair for that matter!

Hi guys,

I've been doing Crossfit for exactly one week (so I'm a baby with respect to the training) but I run a federally funded research group in the area of Biomedical Knowledge Engineering (and have been working as a professional scientist for the last 18 years: http://bmkeg.isi.edu/). Pretty much as soon as I started training and saw how careful people are about tracking their data, I thought to myself 'wow, there is a potential gold mine of data here'. I asked my trainer about this exact question, he pointed me to this article and I just watched this video.

This is mainly adressed to the guys doing the work: Cliff Lewis, Brons Larson and Sisinio Baldis. It looks like this is really very interesting, potentially significant but also a little risky.

1) Doing research on human subjects is ethically very very very tricky and we have to go through a whole bunch of carefully mandated controls for *any research* involving other people's data. This could be anything from just asking people questions in a questionnaire and I haven't seen any form of informed consent being used to ask people permission to use their data for research (and the affiliates would need to ask the individuals doing the workouts). This is actually a step you can't skip and you really really should talk to a University's Internal Review Board (IRB) to make sure (a) that you're not breaking the law and (b) that you're not exposing yourself to some whopping legal liability by working on people's data without their consent.

Here are some of the NIH guidelines for this: http://www.hhs.gov/ohrp/. There are groups that can help with this : specifically one of the national centers of biomedical computing provides support for the process of publishing and sharing data: http://idash.ucsd.edu/

2) Science does not work because of a few clever quantitative scientists coming up with 'the answer', it's a massive collaboration between a lot of very well-trained dedicated people with a lot of checks and balances to prevent any one person kidding themselves (sound like any community you know?). I didn't get the impression that there was any thinking of galvanizing researchers across many disciplines to get involved (just a small team of guys, who are probably excellent, but won't be able to accomplish anything like what the overall community can do). This could produce something cool, but really, if you want to make a difference, you should release the data openly and let all the nerds go crazy on it... and not just physicists and mathematicians, you want specialists in kinesthesiology, biomechanics, physiologists and sports scientists to get their teeth into this stuff...

Also, saying "We've got the science by the balls" makes me sit up and go 'oh really?'. It's best to come at this sort of work with the same way you approach anything truly ground breaking, with determination and a healthy level of respect for the likely pitfalls that you'll encounter as you move forward.

3) I always advocate the principle of open data and it looks like this data is not available for use by the broader community. That's a shame. I hope that the developers and the people at CF HQ think about releasing the data broadly. That's what made the Framingham dataset impactful (or other more modern data like ADNI's impact on Alzheimer's Disease Research).

Anyway, this is very cool and very interesting. I think this could very well have the sort of impact that Mr Glassman predicts but not, sadly, without (a) a careful review of the ethical issues, (b) releasing the data in an appropriately anonymized form for open consideration by the broader scientific community.

Gully Burns