IWCABTAMD

By Dr. Steven M. Platek, J. Ryan Porter and Tia Y. Walters

In Reference, Rest Day/Theory

March 28, 2011

PDF Article

Dr. Steven Platek and co. offer up data analysis showing increased performance in Fran, Angie, Cindy and the CrossFit Total.

Constantly varied functional movements executed at high intensity—this is CrossFit.

To create functionally fit individuals is a primary goal of CrossFit, and an efficacious way of measuring or operationally defining fitness is in an athlete’s ability to do more work faster and across variable domains over time—increased work capacity across broad time and modal domains, or “IWCABTAMD.”

For those of us who are embedded in the culture and the workouts, there is little skepticism about this method because we have loads of anecdotal evidence to draw on.
Anecdotal evidence, however, is just that: anecdotal. In order to show the efficacy of any treatment program, one must devise an experiment where progress is tracked over time. Individuals outside the CrossFit community often question the metric for measuring the efficacy of CrossFit. For instance, they’ll ask, “Where’s the data?”

In an effort to demonstrate evidence-based increases in performance, we conducted two small-scale post-hoc studies. In Experiment 1, we analyzed main-site posts for a benchmark CrossFit workout: Fran. These initial data, even in light of the myriad scientific and methodological limitations associated with our approach, still revealed statistically significant increases in performance (decreased Fran times) over time.

In Experiment 2, we contacted Bill Patton, the owner of LogsItAll, an online repository for CrossFitters to log their times, loads and performance and keep track of their progress over time. Bill was kind enough to provide us with a nameless version of his database, from which we extracted data for four benchmark workouts: Fran, Angie, Cindy and the CrossFit Total.

These data confirm the preliminary data from Experiment 1 with a larger sample size. In other words, people get quicker Fran and Angie times, complete more rounds of Cindy and lift heavier loads from doing CrossFit.

Free Download

Comment

23 Comments on “IWCABTAMD”

1

wrote …

Great to see the journal taking this type of approach, would love to see more data from beyondthewhiteboard / logsital used in this way - especially if tied in with games/regionals competitors. How about looking at:

Is there an optimum timeframe for mainsite crossfit programming to derive benefits (i.e do performance gains taper after a certain period, how does this compare to other protocols)

How does it compare against a periodised progressive metcon program or how about linear strength based work with metcon work etc.

Does weekly frequency impact on this (is 3on 1off optimal scheduling, how long is this sustainable for etc). Clearly lots of hard to test variables, so no idea how you would construct a study to cover these points..

Serious money coming into the games means that someone will be trying to figure this all out, it would be excellent for crossfit to have that process opensource.

2

Matt, I agree - the area is ripe with testable hypotheses. This was a first attempt and it turned out amazing! We are currently following up with some studies and analyses for peer-reviewed journals as well.

In fact, our first pre-lim stuff was published in Evolution: Education and Outreach where we argue you can use CrossFit and paleo diet (ancestral-like exercise and eating) to teach people about how evolutionary science works: walk the walk to talk the talk.
SP

3

wrote …

I love seeing this type of work done on Crossfit. There is some much data out there. If anyone ever needs an additional analyst to work on some kind of research like this, feel free to get in touch. It's what I do for a day job.

4

wrote …

Thanks to Dr. Platek and co. for wading into these waters. Looking forward to more articles like this and I, like Matt, will be interested in future comparisons between training protocols as well (nutrition included). Nice start.

5

wrote …

While this is an interesting study on Crossfit it only proves that if you train in the Crossfit methods you get better at Crossfit workouts. This really should come as no surprise. As a general rule I think most people who workout with the Crossfit methods sees regular improvements. A comparison to some form of control group would be far more useful. Say you took Fran/Angie/Etc data points from people who exercise but don't use the Crossfit methods. Granted this data isn't readily available but it's also more useful when trying to "scientifically" prove the effectiveness of Crossfit. We all know Crossfit works for general fitness. Everyone in every box has seen the success stories but the real question that we all want answered is how Crossfit compares to other methods of training.

6

Casey - we're doing that now. COntrols were definitely an issue here, but don't you find it interesting that frequency of movement training and age had no effect on performance increases? I do.

But like I said, we're working on the other part, too. Nutrition as well. Like y'all said, there's potentially tons of data - it's just getting access to it in a meaningful way.

7

In fact, it might be neat to join one of the emerging teams as a data analysis expert - helping to hone team athletes performance with respect to individual differences in frequency of movement training, nutrition, strengths, weaknesses, etc. If any Games contenders are interested contact me.

8

wrote …

When you say there was no correlation between age and improvement. I assume you mean percentage improvements and if that's the case I do find that to be interesting. For that correlation did you attempt to analyze the data individually or did you relabel the data into groups (20-25, 26-30, etc) for analysis. Mattering on your methods you might not have seen trends with age due to the granular nature of age data but if you were to group them into significant blocks trends might start to develop. In either case if the proper analysis was done I find it very interesting that people in the 20-30 age blocks didn't improve more than the 30-40 age blocks. That being said many of the age related factors are somewhat normalized if you assume you measured relative improvements. But even with that the age thing is something to ponder.

9

wrote …

As #5 Casey states, doing Crossfit should make you better at doing Crossfit.

You started to lose me here (paragraph 2): "For those of us who are embedded in the culture and the workouts, there is little skepticism about this method because we have loads of anecdotal evidence to draw on." One proud member of the Flat Earth Society can convincingly tell the other members that world is flat because his map hangs nicely on a wall. Then, I read your example.

What was accomplished by doing this? Is it relevant? Can it be applied to anything? What hypotheses are you planning on properly testing in the future?

Isn't this the type of scientific fluff that CF despises? There are more holes than a box of Cheerios.

10

Casey, we did both types of analyses (limited space to present in the CFJ) and the effect stood up. Cool right?

Matt, I've already discussed the hypotheses we plan to test in the future in a prior post - if they aren't clear, I could outline them in a bullet point list.

CF v non-CF and Non-Functional fitness program
- then covariates: age, sex, nutrition, etc...
It's not really that complicated when you can recruit willing volunteers.
In fact, we're presenting data this weekend on the effectiveness of ancestral lifestyles on cognitive and academic performance. It's quite neat stuff, I think, but I could be daft.

11

I might also note that this is far from a perfect study. This was a first foray into this type of analysis. We all experience the increase in work capacity and wanted to see if the effects were similar across individuals based on sex and age (the data we had easily available to us). What I think is particularly interesting about these data, and maybe it's not clear in the article where we say there was no correlation between time of "angie occurences" and performance increases. Remember this was not all mainsite blog data. the more compelling data come from LogsItAll where people log what they do and that can be completely independent of the mainsite because they are at a box, at home, whatever. So if I do angie today I am better at it tomorrow. If I did angie 4 months ago and in the interim did what I do, I am equally better at it tomorrow. If you don't find that a little interesting: that training non-specific movements and combinations of exercises I am surprised, but I could be missing something. I can't pretend to be an expert about exercise phys, I am by training an evolutionary biologist and cognitive neuroscientist. I probably have no business dipping my nose in this area, but I couldn't help myself.

12

wrote …

Steven-That is pretty cool.

13

wrote …

How are you going to do it *properly*? As in, without numerous flaws in the study, so that any of the results can be considered useful.

14

at a university with ethical approval on volunteers with random assignment, etc etc.
you know, science! ha, just teasing. But seriously, a proper study with controls, as best as one can control things like exercise routine adherence, nutrition program adherence, the nature of this question is one that does not lend well to causal study, unfortunately... nor do humans really?

It's always been my take that you can sit around and design the perfect experiment for which there is no such thing, or you can actually do something. so we are doing something

15

wrote …

Steven-I like the attitude. I don't know about these other guys but I learn the most from doing. I do my fair share of complaining but I recognize the effort and I like what you're trying to accomplish.

16

wrote …

Steven,


Nice article and a great start. As always the naysayers will gather round and be quick to identify the gaps, deficiencies, etc. , and will question the validity of this or that aspect of the work, but in the meantime the scientist will be doing what he does; science. He will revise / refine the hypothesis, develop more robust experimental protocols, and go about improving on the first work, with the second. Accordingly….. bravo to you.


I found the differences demonstrated amongst the four selected WODs interesting, especially with regard to gender. There are multiple references in the article to the absence of significant differentiation between genders with the exception of CFT, and yet when I look at the WOD’s the data presented itself exactly as I would have expected.


Angie, and Cindy are bodyweight based WODs, and accordingly the data would effectively be normalized between genders as a function of the probable much lower mean bodyweight of the female sample set vs the male.


This is not the case with Fran however, and yet once again the data doesn’t present significant discrepancy between the gender sets. Why? Because Crossfit HQ evidently hit the nail right on the head with its prescribed respective load values for each gender. I am relatively certain if you were to determine the mean bodyweight delta in percentage terms between the male and female sample sets that it would closely align with the delta, in percentage terms, between the prescribed loads for each gender, once again normalizing the data.


As for CFT, given the absence of clear time domains, it presents a host of additional problems beyond those presented by the other WOD’s. That being said, I would be interested to examine the improvement demonstrated, in percentage terms, in the CFT data vs. the data associated with the other WODs, in that it might shed light on the ongoing debate within the Crossfit community as to whether or not mainsite programming demonstrates an appropriate balance between strength and metcon training. Now that I think about it, if the data set were exhaustive enough one could perform a similar analysis for each of the ten physical attributes contained within Crossfit’s first model as a means of refining the programming to ensure that similar improvements were demonstrated across all ten attributes resulting in a theoretical ideal programming model to maximize GPP.


Great stuff!

17

wrote …

Here the quote of the central result of the study:
Those data strongly suggest that [...] CrossFit increases work capacity across broad time and modal domains independent of sex, age, and frequency of exposure to the WOD. 

IMHO the above statement (and it's sub statements) are either meaningless, or false. It would be great if the authors would carefully review what they have written and ensure that no myths are perpetuated in the community, especially the assertion that training frequency is irrelevant for the impact the training has. 

I will take those things in turn. Firstly about "the data strongly suggests" point. I believe the authors of the study confuse a failure-to-reject-the-null-hypothesis with an accepting-the-null-hypothesis result. I would probably agree that the data has not proven that there is a difference based on "sex, age etc", but this is not the same as claiming that is has proven that there is no difference - the reality probably being that the data was not good enough to make either claim. 

As to the underlying statements: 
"across broad modal and temporal domains" - probably true, and whilst the study can not really prove it (the athletes will have trained specifically trained for improving their Fran times etc) using those benchmark workouts is probably good enough a proxy. Not really a surprise though: you train things many different things, you get better at many different things. 

"increases work capacity" - probably true (and not really surprising), but the setup is weird making the results unreliable: rather than analysing something like the percentage improvement of every athlete, only the improvement of the average is shown, which is not a very meaningful measure in an inhomogenous group. For example, if one person manages to improve a 20min Fran time to 10min than this has the same overall effect as 20 people increasing their time from 2:00 to 1:30. Whilst the former pretty much everyone can do, 20 people improving from 2:00 to 1:30 is a miracle. A study that treats those two events as being the same can not give credible results. 

"independent of sex, age" - I am not sure what the authors wanted to say here. If the intention was to say something like "everyone got better when training" then this is certainly true, but not really noteworthy. If it was meant more along the lines "men, women, and athletes of all age increase at the same pace" then I doubt that the study can have shown this. In any case, one would have to define "pace" first and the definition would be crucial (eg absolute, relative to initial performance, relative to bodyweight)

"independent of exposure to CrossFit": now this is - excuse my French - b+llocks. You don't train, you dont improve. You train more you improve more, up to the point where you overtrain and where your performance worsens. This has been shown in numerous studies and it is certain to be true. The authors simply fell in their aforementioned trap: just because the data did not show that there was a dependency this does not mean that there was no dependency. 

18

replied to comment from Stefan Loesch

I didn't think the invalid results deserved such a thorough and informed response, but there it is.

19

Michael, thanks bro.
I think people have missed the point here: 1) the CFJ is not a peer-reviewed scholarly journal, I will send HQ the link when I post those data and 2) this is/was a 1st slice at the pie.
It sounds like some people think that it'd been better if we never undertook this endeavor, opinions in my mind. Independent of the flaws, and sure there are numerous flaws with this type of study/description, this is a 1st slice...

To address my inaccuracies when using the word PROVE is useless. That's not what science is about. Nor were we explicitly testing for null hypothesis as you might learn in an undergraduate course on statistics. We simply perusing a set of archival data and drawing connections.

I do VERY MUCH like the comment about % change. That makes a ton of sense and something we might include in future descriptions of this type. However the error bars do provide some of that information, yes on the means (SORRY), and an analysis for outliers revealed no significant outliers. That is, we did not have anyone in our group who improved their Fran time from 15:00 to 1:30.

Thanks again for your kind words Michael. Thank goodness with several peer-reviewed publications I have a thick skin about it all. :-)

20

wrote …

Dr. Platek,

Fantastic look at CrossFit. I loved your article "The Ultimate Painkiller". I cited it in my recent graduate school research paper on goal setting in a CrossFit class. I will try to turn it into a Journal appropriate article soon. Keep up the great work!

Chris

21

wrote …

Steven, keep up the good work. The ones who criticize often are the same ones that rarely contribute.

22

Thor Falk wrote …

Steven, I believe people like that you have undertaken the study. I personally certainly found it an endeveaour well worth it. I believe what people did get a bit cross about was that the results were slightly oversold. Now if I am looking at your paper, I believe what you have found is that when people train they improve. Not a great result, but if you had communicated this together with an explanation about your further plans people would have got excited.

What did make people cross I believe was the "lipstick on the pig" as they used to say in a previous job of mine. There were too many conclusions in the article that did not seem to be backed by the data. Especially the statement about independency of the results of the training frequency did not really make sense I have to admit, and it made the article look slightly dishonest, sort of, I need a result, any result.

I believe we are all looking forward to your further work, but I guess what some people are saying is that if the data is such that no meaningful statements can be made, then it is better to freely admit this rather jumping to conclusions that are not backed by the data. Keep in mind that most readers of the CFJ are probably not scientists and they might take even wrong statements at face value. Or do you really want to ne responsible for people WODing only once a month because science has shown that frequency does not matter? :-)

23

wrote …

First and foremost, great job Steven! A tremendous amount of positive discussion is being participated in because of your (and your colleagues) work. It is a pleasure to see such a well educated and opinionated CF population offering "advice". You make a great point in these comments that the CFJ is not peer-reviewed. The CFJ often spurs great conversations and is beginning to influence more rigorous research that will be peer-reviewed.

Again, great job and thank you.

Leave a comment

Comments (You may use HTML tags for style)