Peer review is the necessary rite of passage that all academics go through before they achieve all of their riches and glory. Actually, just the glory. Well, really, just their name on a paper. But I guess that can be glorious at times.
When you submit one of your ideas to a journal for peer review, and hopefully, publication, what actually happens? In 2016, I submitted a paper to the Journal of Sports Sciences, where I aimed to use a fatigue model I had developed to investigate if Pitch Clocks would increase fatigue levels in Major League Baseball pitchers. This paper went through 3 intense rounds of revisions, faced the brink of rejection, before ultimately being accepted for publication. I wanted to write a little bit about the peer review process for those of you who aren’t familiar with it, just to show you how important the process is to understanding the way the human body works.
When I prepared the manuscript for submission, the journal asked that I remove all Author names and affiliations, to make it a blinded review and keep the authors from biasing the reviewers. This is a good first step – you shouldn’t get a paper accepted just because of the name that wrote it. That being said, I was using a fatigue model that hadn’t been used in practice before, and the only reference to it had my name all over it. I’m pretty sure that the reviewers could have put 2 and 2 together to figure out who the first author was – but at the same time, my name doesn’t really carry much clout. I’m not going to bias anyone.
The next step is to find the reviewers. The journal asks that you recommend three “arm’s-length reviewers” to closely critique your paper. These are researchers that you haven’t been on grants with, published with, or worked with. You very well may know these reviewers in a professional sense, but they should be able to review your paper without having any professional gain or detraction from their career by either accepting, or rejecting your work.
This is where the biggest downside of the peer review process takes place. I submit my first iteration of my manuscript on April 19, 2015. I received a request for revisions and feedback from the reviewers on July 8th, 2015. After resubmitting in July, I then waited until October 26th to receive the next round of revisions. By the time the paper was accepted, and the paperwork was signed to get the manuscript out to the general public – it was mid-February. Nearly a full year had passed from the time I submit the research, to when the public could consume it. This was a bit discouraging, as I had wanted to get the paper out before the Winter meetings so that GMs and rule makers in the MLB could have access to it – in the end, it all worked out, but it was a frustrating time while it happened.
So, what do these reviews look like?
There were 3 reviewers that looked through this manuscript. Each of them provided very detailed comments regarding the manuscript. Reviewer 1 had 10 comments, Reviewer 2 had 1 comment, and Reviewer 3 had 8 comments. Here is a sampling of some of the concerns they had:
The author(s) explore an interesting concept of potential pitcher fatigue within innings/games. With the new pace of play rules adopted by Major League Baseball, this is a very worthwhile endeavour. However, the results and discussion are based upon methods that are unclear. More details about the model used, and how exactly it was applied, are necessary to properly evaluate the contribution of this work. A discussion of the model’s sensitivity to assumptions made would also be beneficial. Finally, an analysis of velocity changes within a given inning or throughout a game may provide a more direct measure of fatigue.
The author(s) predicate their work based on the assumption that joint stability in the elbow is analogous to joint stability in the spine. Majority of references provided are for the spine, not the elbow. The argument that muscular fatigue alters joint position sense in the upper extremity, potentially leading to greater loading on passive tissues, is a much stronger justification for the present work; and hence, should be expanded upon in the introduction and discussion.
I applaud you on your topic selection as I think it is relevant and interesting. Unfortunately, I think you had to make too many assumptions in calculating your results. Also, as injury is ultimately what we are interested in, I found it difficult to relate your findings to increased risk of injury and truly quantify what that relationship is.
Although I agree with the arguments put forth by the authors in the introduction, as well as the intended purpose. I have difficulty seeing the contribution that this study provides to the current literature in its current form.
First, there is a complete lack of information regarding the model used to predict fatigue. From my understanding, the model was built from isometric hand grip, or isometric thumb contractions. This data is not appropriate to model the fatigue from pitching which is a complex movement between flexion/extension and axial rotations of the wrist, and elbow. Some of the muscles they include in their study have complex interactions at both joints (wrist and elbow). If the authors already had some data indicating that it is an appropriate model for flexion and extension of the wrist/elbow joint muscles, it would have been valuable to share. I think for this paper to be accepted this major limitation must be either validated before the fatigue related conclusions can be accepted.
I think the authors have some interesting conclusions to draw from their efforts, and so I am suggesting major revision. However, without this substantial amount of additional work I do not see the manuscript as publishable in its current form. The conclusions they draw are of substantial interest to those in the sports science and medicine occupation, as well as those working in Major League Baseball. Therefore this work must be completed to a high degree of scientific merit, which I hope the authors are willing to provide.
At the time I got these reviews back, I remember thinking – this is just about as harsh of a critique as I’ve received from any manuscript I’ve submitted. At the same time, the comments given by these reviewers were exceptional – every single point would go along way to strengthening the paper. The biggest concern, was that this was a completely theoretical approach to proving pitch clocks could cause an increase in muscle fatigue. This concern was later alleviated when another research group evaluated the physiological properties of fatigue in groups that used pitch clocks.
We took these comments to heart, and made significant changes to our manuscript – the findings remained the same, but we incorporated some of the literature that the reviewers recommended. We also found a few more papers that looked into muscle activation patterns of the forearm during pitching, and included those in to the fatigue model. And then? We waited…
Another large list of comments from the reviewers – once again, these comments were very constructive and bettered the paper. The biggest difference this time around, is that we didn’t necessarily make changes to the paper – but had to explain more in depth some of the assumptions, and limitations our model and study presented. Here is another sampling of a few of the comments the reviewers provided. Reviewer 1 had three major comments, and Reviewer 2 had 32 COMMENTS. Reviewer 3 was fine with the changes we made. I’m sure if you’ve cruised any of academic Twitter, you’ve heard the jokes about reviewer 2. Well… there’s a reason for that! (Reviewer 2 provided excellent comments, that’s for sure).
74% of academics hate reviewer #2. 36% of academics actually are reviewer #2.
— Fake Academic Stats (@FakeAcademStats) May 19, 2017
I appreciate the additions the authors have made to the paper. However, the authors seem to be missing an opportunity to make relationships to injury or biomechanics literature that is relevant to pitching. There is a lot to be said about how reductions in muscle force capabilities can affect the injury mechanism (valgus-varus joint stability). A 5 % force decrease might be substantial or insignificant, depending on the force-length, force-velocity. There is also a comment to be made about the muscles that stem from the common flexor tendon. Epidemiological, and cadaveric models indicate that these dynamic stabilizers are most important in the direction where joint stability is compromised during pitching. All the literature that you suggest stem from joint flexion/extension. This is not the suspected injury direction. In regards to this, I think the discussion of this paper will benefit from more comments on these muscles rather than the brachioradialis. How my comments fit into the comments from other reviewers will decide how much to add and take away.
The work of Holmes and Keir only looked at the flexion extension stiffness of the elbow. This is an inappropriate citation to apply to the valgus-varus stability of the elbow. There exist many other authors who have looked at muscle contributions to valgus-varus joint stability. This is a comment on all mentions of stability in this paper. The authors seem to be including all elbow stabilizers, independent of movement. I would argue that the brachioradialis is not a stabilizer of the valgus-varus movements at the elbow, in fact in the citations the authors reference, it increases UCL strain. The brachioradialis is however, a stabilizer of flexion extension stability. This is stability has never been identified as a reason for the most commonly seen baseball injuries. I urge the authors to be more conservative in their mention to stability in the elbow as a general term. Stability is dependent on movement direction.
-Lin et al. (2007). Muscle contribution to elbow joint valgus stability. J Shoulder Elbow Surg.
-Udall et al. (2009). Effects of flexor-pronator muscle loading on valgus stability of the elbow with an intact, stretched, and resected medial ulnar collateral ligament. J Shoulder Elbow Surg
-Park and Ahmad (2004) Dynamic contributions of the flexor-pronator mass to elbow valgus stability. J Bone Joint Surg Am.
The use of EMG as in an input does not validate a model to all movement types. The normalizing of muscle demands completely ignores the requirements of muscle energetics. With isometric muscle tasks you are limited by blood perfusion issues which are affecting oxygen uptake and metabolite flushing, this is not the same limitation in dynamic muscle contractions. Furthermore, you have built a model on local muscular work. The activity you are applying it to is a whole body. Whole body exercise provides other demands on the body’s energy sources apart from the demands of that local contraction. Simply having a normalized input, does not provide validity to a method. The reader must be made aware of this severe limitation, or citations must show that
1) EMG is appropriate for isometric, eccentric and concentric muscle contraction alike. Which includes being able to measure MVC and current motor unit usage during each movement.
2) That isometric fatigue models can, and have been applied to concentric or eccentric models.
If there exist no citations, then more care must be taken to show the reader the expected limitations. You are either over or under predicting fatigue with your model. I agree with the direction of the results you have found, but the number you land on must be put in perspective.
With more thought into the experimental protocol and analysis…I am unsure if null hypothesis significance testing is appropriate here. You have controlled inputs to your model. Your results are completely dependent on the inputs that you have put in. The only variability you have is the inputs of pace, but those are estimates at best. Since you have each pitcher using the same EMG, these results could essentially be arrived upon by inputting theoretical pitchers (Monte Carlo simulation). I feel confident that statistical analysis is unneeded and do not provide any additional understanding. The effect size is an appropriate tool for identifying important changes in muscle fatigue, but as mentioned elsewhere, a mechanical argument of joint valgus-varus stability is a better.
Once again, these were great insights into the paper. We removed the significance testing from the results section, and included the references on valgus/varus stability.
Our paper was accepted for publication in the Journal of Sports Sciences.
I wanted to write this post for a variety of reasons, but the main one was to discuss just how important this process is to better understanding how the human body works. Without the input from these reviewers, this paper may have been accepted into a lesser journal, but it would not have come close to having the reach and impact that it did in its final state.
Keep this in mind – the reviewers of these manuscripts made no dollars to do these extremely thorough reviews. Not an extra cent over their normal salary. Like I mentioned, reviewer 2 had 30 comments in the second round of revisions for this manuscript – that isn’t something that takes an hour to do. These reviewers were anonymous – I have my thoughts on who they likely are, but they will get no formal credit for the large amount of work they put in to this paper. This was an exceptional case for me, but all of the manuscripts I have submitted for publication have received great attention from the reviewers.
Keep this all in mind when you read about the evils of peer review. The blogging world has been a great experience for me – it allows me to express my ideas and findings to a larger community. However, you can’t replace the process of peer review when it comes to bringing your big ideas into the world. Yes, it takes a long time. Yes, there can be cronyism when it comes to presentations and conferences – but in the world of sports sciences, this process ensures that your ideas are thoroughly tested, and that your results carry the biggest impact that they can.