question regarding a priming experiment: dealing with RTs before target offset

[mmy]

hi there, we conducted a priming lexical decision experiment, and the response time was calculated from the target onset. However when we cleaned the data, we found that some responses were given before the target offset, thus, my question is: how should we deal with these data (to delete them, or delete those response time less than 200ms (however since the target are different in length), or just take them as valid data)? Any advice is appreciated:)

[Daniel]

It's hard to answer in general terms.

The question you must assess, from as unbiased a perspective as you can, is what is best for your data. Will removing that data harm your experiment? Why not just gather more?

If you leave it, is there a fundamental problem with it? Will the results always have an (at least imaginary) asterisk?

If you are uncomfortable with the data, you should remove it. However, I would suggest talking with your advisor, a colleague, or someone else who can comment specifically on the norms of your area of research and what is done in cases like this, and what journals will accept for publication.

In the most extreme cases (not usually for linguistics, but sometimes psychology where research like your might be published) it is actually required by the journal that you pre-register your hypotheses and methodology, so that the results are just that-- results, not new ideas, analyses, etc., just a binary yes/no was the hypothesis supported by the data?

From that very strict perspective, your methodology was not followed, so that data is invalid.

On the other hand, if you don't think reaction time is relevant to the results, maybe it doesn't matter. But for priming I would think it usually is relevant.

The best advice is probably to try to collect more data until you have your original target, excluding that data. (You could also include it in a secondary analysis to see if there is any variation in it, in case that would matter to you.)

If it is not possible to collect more data, then I would suggest checking with your advisor/colleagues to see whether the data should be abandoned or published as-is.

I suppose the main question is how much data you're talking about. If it's a few items, go ahead and remove them. If it's half of the data or something like that, you should probably revise your protocol and collect more data, or accept that the data may not accurately reflect reaction times as desired.

As for 200ms, that's a very short amount of time. Any reaction before then, I am guessing, would be without actually hearing and processing the input. N400 and P600 in ERP research are at 400ms and 600ms respectively, and that's for cognitive processing, not actual reaction times. So 200ms is very fast.

But it sounds like you have a longer utterance, so there is more context than just 200ms, in which case it may not be a problem-- but is the last word (is that the target itself?) important? If they weren't reacting to it, then why was that? Could that actually be explained by priming? Maybe these surprise results are relevant to your research questions. Or maybe the participants are just guessing to get done with the experiment quickly.

(By the way, double check that your equipment is reliable to the level of 200ms, so you don't throw out potentially good data from unreliable equipment, or, better yet, try again using more sensitive equipment. But I assume in this case that's not the problem because you are getting different results on the order of 200ms or less.)

Anyway, it's too hard to answer in general terms here. If you're hesitant to use the data, there's probably a good reason for that though.

[mmy]

Thanks Daniel, therefore, I'd ask is is the threshold of 200ms be calculated from the onset of target or offset of it when you say the cognitive processing?

[Daniel]

It depends, of course. Are you interested in processing related to the first syllable of the word, or the meaning of the whole word? Obviously the offset is required in order to unambiguously understand the word.

You're interested in priming effects, so maybe answering early is an indicating of their expectations, in which case you could include the reaction time in your statistical model. A more complex statistical model with more variables might solve the problem, and you could look to see whether these different factors are individually significant within the model.

But again, 200ms is not a very long time, and the word itself is, I assume, usually also very short (maybe another 200ms?), so that's only 400ms. For some purposes that isn't enough time for someone to process it.

In short, this should have been part of your experimental design. Now all you can do is either abandon this data because it does not fit your original design/expectations, or decide what would have been most reasonable in the first place and include only that data. In other words, if you imagine running this experiment again, what would be your methodology for this data? Would you exclude it? (Depending your specific research questions, I certainly would either exclude it, or try to include it in the statistical model.)

This is why doing some pilot studies is useful to see how the experiment works in real life and whether there are any problems. Maybe the instructions simply weren't clear enough to the participants so you would need to tell them "wait for the beep sound" and that's all. But, again, maybe this data is actually useful to you in some way. If you can't think of the RT data as being useful, I would recommend not using this.