Practical measures of proficiency

[Daniel]

What are your favorite ways to measure proficiency that actually feel effective?
I often wonder whether, for example, I'm "good at" Spanish, and I don't feel like I have a good way to answer that. As a linguist I'm probably harder on myself than most, but I'm also realistic: I'm certainly no native speaker of Spanish and I do have trouble understanding sometimes (perhaps often).

Personally I don't see the value in labels like "intermediate" or "advanced" or even "fluent". (Maybe "beginner" is an exception.)

And official scales like "B2" don't seem to mean much unless you can compare yourself to others on exactly the same scale.


So what's a good solution?

Personally I like thinking about practical aspects:
--Can you give/get directions to the airport? [Also known as the "train station test"]
--Can you read a newspaper?

Of course different measures like that might have very different results (maybe you can read a newspaper but not ask for help, or the opposite), but that seems natural to me.

Based on these tests, I'd say:
I am comfortable living in Spanish.
I can read academic research in most (Germanic, Romance) European languages, at least with a dictionary.


Does that seem reasonable? Any better suggestions?
What's your favorite way of measuring proficiency?

[Corybobory]

I think those are good measurements - for myself, I gauge in terms of being 'conversational' or not, if I can carry on a conversation with a stranger, and  if I can read academic texts with a dictionary or not.

It's hard to define, since language learning can improve in different areas at different rates - as in I'm conversational in Japanese and French (and Spanish at a push), but I struggle to read Japanese, and reading French or Spanish for academic works is fine (with a dictionary!).  I don't think 10 Japanese dictionaries could get me through an academic paper sadly...

[Daniel]

I agree about "conversational"-- I'd describe myself as strongly conversational in Spanish. But sometimes I wonder if others might expect more of my Spanish than is realistic, because "conversational" is almost a technical term like "fluent" with no real definition.

It's hard to define, since language learning can improve in different areas at different rates - as in I'm conversational in Japanese and French (and Spanish at a push), but I struggle to read Japanese, and reading French or Spanish for academic works is fine (with a dictionary!).

Absolutely. But it might be a good thing that a proficiency test/description would take that into account.

I don't think 10 Japanese dictionaries could get me through an academic paper sadly...

I know what you mean. There's also a degree to which being "advanced" in Japanese and "advanced" in Spanish mean very different things. Somehow we want to almost separate the difficulty from the experience. It's hard feeling like we'll always be at the "beginner" level in some languages (like my Arabic, for example), even though that may be true (unless we find the time to go to the country for a while and really put in a lot of specific effort).

[freknu]

This is not something I have thought about before, but being able to read a newspaper is something I would consider a very good indicator of proficiency.

Of course, this does not mean that you have an extensive knowledge slang, colloquialism, idioms, proverbs, academic or technical terminology, and so on. However, newspapers do seem to be a pretty good indicator of the standard language.

It's also important to remember that proficiency is not a "scalar":

• reading proficiency
• writing proficiency
• speaking proficiency
• listening proficiency
• general understanding
• morphology

... and so on.

Of course, it all depends on the newspaper you are reading. An economic (or otherwise academic or technical newspaper) is going to be more difficult.

[Daniel]

Of course, it all depends on the newspaper you are reading. An economic (or otherwise academic or technical newspaper) is going to be more difficult.

Not necessarily! I do relatively well reading Linguistics articles in languages I'd have no ability in otherwise (like Swedish, if I have some patience and a dictionary). [It's possible some of that is a cognate effect, I admit, but not all; context also helps and my specialization in the field and learning that specific jargon.]

As you said: it's not scalar at all. Definitely not. (But on the whole it does feel that way... perhaps we could somehow measure a "core" proficiency, in things like syntactic structures and phoneme contrasts??)

[freknu]

Of course, it all depends on the newspaper you are reading. An economic (or otherwise academic or technical newspaper) is going to be more difficult.

Not necessarily! I do relatively well reading Linguistics articles in languages I'd have no ability in otherwise (like Swedish, if I have some patience and a dictionary). [It's possible some of that is a cognate effect, I admit, but not all; context also helps and my specialization in the field and learning that specific jargon.]

I was talking more in general, not taking into account any specialisation or previous knowledge, the average general population.

As you said: it's not scalar at all. Definitely not. (But on the whole it does feel that way... perhaps we could somehow measure a "core" proficiency, in things like syntactic structures and phoneme contrasts??)

I would are say reading and writing would then be secondary, as language is firstly an oral skill, so perhaps:

• pronounciation
• syntomorphology (basic syntax and inflection)
• vocabulary

Those would probably be three core features that I would concentrate on if I absolutely had to create a constant of proficiency. I'm unsure about weighting, though. It's very difficult to say which one is more important.

[Daniel]

Right. Weighting is the hard part.
I was thinking about some kind of "necessary core", though. I mean, you can't speak/read/do anything with English if you don't know things like SVO order, "of", the contrast between "z" and "s", etc. Beyond modality, beyond genre, a few core things are there. Although perhaps knowing all of the core doesn't really take too much; I'm not sure.
Perhaps a definition: core = generalizable knowledge, rather than specialized details?
(It's not a sharp distinction, of course, but perhaps practical.)



As for specializations, I think it applies to almost everyone: lots of "average" people really like sports and can probably understand a whole lot more of that than an average economics article in a newspaper

[freknu]

As a side note, is it always "weigh-t-ing" when speaking of statistical factors?

[Daniel]

Yes. weighing = measuring weight; weighting = assigning weight. (interesting/odd contrast now that I think about it)

[However there's the non-technical usage "weigh X down" which I suppose is an exception to the rule above. "weighting" is always about sets/lists and relative weights, I think.]

[freknu]

I was thinking about some kind of "necessary core", though. I mean, you can't speak/read/do anything with English if you don't know things like SVO order, "of", the contrast between "z" and "s", etc.

Funnily enough, I'm not sure I would consider z/s that important When it comes to phonology I would be more interested in strict phonemicity, maybe even dropping some minor phonemes.

So [s~z], [θ~ð], [ð~d], and so on, is not something I would necessarily require for basic proficiency.

I guess I'm more inline with being able to understand and make yourself understood being more important that strict correctness. It could certainly be a part of further proficiency, but for basic proficiency I'm somewhat uncertain.

I guess it depends a lot on the definition of proficiency as well.

[Daniel]

Sure. But I meant that it's not modality/genre dependent. If you know s/z then you know English better, simple as that. It's not required for basic proficiency, you're right.

Perhaps for basic proficiency (but still no specialization) consider t/d.
[Or a less practical but still very literal example, "e" vs. "t" ]

[freknu]

For something that would seem quite central to language acquisition, has there not been any linguistic study into proficiency and maybe things like sigma-based proficiency scales?

[Daniel]

'Sigma-based'?

There has been a lot of study. I'm wondering at the moment mostly about intuitive measures. Not for the sake of competition, but just linguistic research, is there some intuitive way to say that you speak German better than I do? (Certainly we'd need to make some assumptions, but ignoring the details...)

[freknu]

'Sigma-based'?

There has been a lot of study. I'm wondering at the moment mostly about intuitive measures. Not for the sake of competition, but just linguistic research, is there some intuitive way to say that you speak German better than I do? (Certainly we'd need to make some assumptions, but ignoring the details...)

I see. Kind of difficult to say.

[Daniel]

Ah, standard deviations, bell-curve, all that. Why would you assume that language learning is in a normal distribution? (It might be, but I'm not sure that's an obvious statistical assumption to make.)