Linguist Forum

Specializations => Language Acquisition => Topic started by: Daniel on December 18, 2013, 03:48:16 AM

Title: Practical measures of proficiency
Post by: Daniel on December 18, 2013, 03:48:16 AM
What are your favorite ways to measure proficiency that actually feel effective?
I often wonder whether, for example, I'm "good at" Spanish, and I don't feel like I have a good way to answer that. As a linguist I'm probably harder on myself than most, but I'm also realistic: I'm certainly no native speaker of Spanish and I do have trouble understanding sometimes (perhaps often).

Personally I don't see the value in labels like "intermediate" or "advanced" or even "fluent". (Maybe "beginner" is an exception.)

And official scales like "B2" don't seem to mean much unless you can compare yourself to others on exactly the same scale.


So what's a good solution?

Personally I like thinking about practical aspects:
--Can you give/get directions to the airport? [Also known as the "train station test"]
--Can you read a newspaper?

Of course different measures like that might have very different results (maybe you can read a newspaper but not ask for help, or the opposite), but that seems natural to me.

Based on these tests, I'd say:
I am comfortable living in Spanish.
I can read academic research in most (Germanic, Romance) European languages, at least with a dictionary.


Does that seem reasonable? Any better suggestions?
What's your favorite way of measuring proficiency?
Title: Re: Practical measures of proficiency
Post by: Corybobory on December 18, 2013, 04:10:01 AM
I think those are good measurements - for myself, I gauge in terms of being 'conversational' or not, if I can carry on a conversation with a stranger, and  if I can read academic texts with a dictionary or not.

It's hard to define, since language learning can improve in different areas at different rates - as in I'm conversational in Japanese and French (and Spanish at a push), but I struggle to read Japanese, and reading French or Spanish for academic works is fine (with a dictionary!).  I don't think 10 Japanese dictionaries could get me through an academic paper sadly...
Title: Re: Practical measures of proficiency
Post by: Daniel on December 18, 2013, 10:58:52 AM
I agree about "conversational"-- I'd describe myself as strongly conversational in Spanish. But sometimes I wonder if others might expect more of my Spanish than is realistic, because "conversational" is almost a technical term like "fluent" with no real definition.

Quote
It's hard to define, since language learning can improve in different areas at different rates - as in I'm conversational in Japanese and French (and Spanish at a push), but I struggle to read Japanese, and reading French or Spanish for academic works is fine (with a dictionary!).
Absolutely. But it might be a good thing that a proficiency test/description would take that into account.

Quote
I don't think 10 Japanese dictionaries could get me through an academic paper sadly...
I know what you mean. There's also a degree to which being "advanced" in Japanese and "advanced" in Spanish mean very different things. Somehow we want to almost separate the difficulty from the experience. It's hard feeling like we'll always be at the "beginner" level in some languages (like my Arabic, for example), even though that may be true (unless we find the time to go to the country for a while and really put in a lot of specific effort).
Title: Re: Practical measures of proficiency
Post by: freknu on December 19, 2013, 01:00:11 AM
This is not something I have thought about before, but being able to read a newspaper is something I would consider a very good indicator of proficiency.

Of course, this does not mean that you have an extensive knowledge slang, colloquialism, idioms, proverbs, academic or technical terminology, and so on. However, newspapers do seem to be a pretty good indicator of the standard language.

It's also important to remember that proficiency is not a "scalar":


... and so on.

Of course, it all depends on the newspaper you are reading. An economic (or otherwise academic or technical newspaper) is going to be more difficult.
Title: Re: Practical measures of proficiency
Post by: Daniel on December 19, 2013, 01:30:44 AM
Quote
Of course, it all depends on the newspaper you are reading. An economic (or otherwise academic or technical newspaper) is going to be more difficult.
Not necessarily! I do relatively well reading Linguistics articles in languages I'd have no ability in otherwise (like Swedish, if I have some patience and a dictionary). [It's possible some of that is a cognate effect, I admit, but not all; context also helps and my specialization in the field and learning that specific jargon.]

As you said: it's not scalar at all. Definitely not. (But on the whole it does feel that way... perhaps we could somehow measure a "core" proficiency, in things like syntactic structures and phoneme contrasts??)
Title: Re: Practical measures of proficiency
Post by: freknu on December 19, 2013, 01:39:39 AM
Quote
Of course, it all depends on the newspaper you are reading. An economic (or otherwise academic or technical newspaper) is going to be more difficult.
Not necessarily! I do relatively well reading Linguistics articles in languages I'd have no ability in otherwise (like Swedish, if I have some patience and a dictionary). [It's possible some of that is a cognate effect, I admit, but not all; context also helps and my specialization in the field and learning that specific jargon.]

I was talking more in general, not taking into account any specialisation or previous knowledge, the average general population.

As you said: it's not scalar at all. Definitely not. (But on the whole it does feel that way... perhaps we could somehow measure a "core" proficiency, in things like syntactic structures and phoneme contrasts??)

I would are say reading and writing would then be secondary, as language is firstly an oral skill, so perhaps:


Those would probably be three core features that I would concentrate on if I absolutely had to create a constant of proficiency. I'm unsure about weighting, though. It's very difficult to say which one is more important.
Title: Re: Practical measures of proficiency
Post by: Daniel on December 19, 2013, 01:51:08 AM
Right. Weighting is the hard part.
I was thinking about some kind of "necessary core", though. I mean, you can't speak/read/do anything with English if you don't know things like SVO order, "of", the contrast between "z" and "s", etc. Beyond modality, beyond genre, a few core things are there. Although perhaps knowing all of the core doesn't really take too much; I'm not sure.
Perhaps a definition: core = generalizable knowledge, rather than specialized details?
(It's not a sharp distinction, of course, but perhaps practical.)



As for specializations, I think it applies to almost everyone: lots of "average" people really like sports and can probably understand a whole lot more of that than an average economics article in a newspaper :)
Title: Re: Practical measures of proficiency
Post by: freknu on December 19, 2013, 01:56:04 AM
As a side note, is it always "weigh-t-ing" when speaking of statistical factors?
Title: Re: Practical measures of proficiency
Post by: Daniel on December 19, 2013, 01:57:37 AM
Yes. weighing = measuring weight; weighting = assigning weight. (interesting/odd contrast now that I think about it)

[However there's the non-technical usage "weigh X down" which I suppose is an exception to the rule above. "weighting" is always about sets/lists and relative weights, I think.]
Title: Re: Practical measures of proficiency
Post by: freknu on December 19, 2013, 02:01:41 AM
I was thinking about some kind of "necessary core", though. I mean, you can't speak/read/do anything with English if you don't know things like SVO order, "of", the contrast between "z" and "s", etc.

Funnily enough, I'm not sure I would consider z/s that important ;) When it comes to phonology I would be more interested in strict phonemicity, maybe even dropping some minor phonemes.

So [s~z], [θ~ð], [ð~d], and so on, is not something I would necessarily require for basic proficiency.

I guess I'm more inline with being able to understand and make yourself understood being more important that strict correctness. It could certainly be a part of further proficiency, but for basic proficiency I'm somewhat uncertain.

I guess it depends a lot on the definition of proficiency as well.
Title: Re: Practical measures of proficiency
Post by: Daniel on December 19, 2013, 02:06:57 AM
Sure. But I meant that it's not modality/genre dependent. If you know s/z then you know English better, simple as that. It's not required for basic proficiency, you're right.

Perhaps for basic proficiency (but still no specialization) consider t/d.
[Or a less practical but still very literal example, "e" vs. "t" ;) ]
Title: Re: Practical measures of proficiency
Post by: freknu on December 19, 2013, 02:18:56 AM
For something that would seem quite central to language acquisition, has there not been any linguistic study into proficiency and maybe things like sigma-based proficiency scales?
Title: Re: Practical measures of proficiency
Post by: Daniel on December 19, 2013, 02:34:33 AM
'Sigma-based'?

There has been a lot of study. I'm wondering at the moment mostly about intuitive measures. Not for the sake of competition, but just linguistic research, is there some intuitive way to say that you speak German better than I do? (Certainly we'd need to make some assumptions, but ignoring the details...)
Title: Re: Practical measures of proficiency
Post by: freknu on December 19, 2013, 02:36:48 AM
'Sigma-based'?

(http://www.innovatingtowin.com/innovating_to_win/images/SixSigmaBellCurve.jpg)

;)

There has been a lot of study. I'm wondering at the moment mostly about intuitive measures. Not for the sake of competition, but just linguistic research, is there some intuitive way to say that you speak German better than I do? (Certainly we'd need to make some assumptions, but ignoring the details...)

I see. Kind of difficult to say.
Title: Re: Practical measures of proficiency
Post by: Daniel on December 19, 2013, 02:39:56 AM
Ah, standard deviations, bell-curve, all that. Why would you assume that language learning is in a normal distribution? (It might be, but I'm not sure that's an obvious statistical assumption to make.)
Title: Re: Practical measures of proficiency
Post by: freknu on December 19, 2013, 02:43:55 AM
Ah, standard deviations, bell-curve, all that. Why would you assume that language learning is in a normal distribution? (It might be, but I'm not sure that's an obvious statistical assumption to make.)

Not necessarily language learning, but perhaps taking a sample of native speakers and statistically comparing your "proficiency" to that.

It would possibly require some sort of standardised test to query specific linguistical features and understanding of your language. It wouldn't surprise me if that was close to a bell curve.

I don't really know if comparing proficiency to some superset of all native features is useful. It's how you compare to a native speaker.
Title: Re: Practical measures of proficiency
Post by: Daniel on December 19, 2013, 02:46:07 AM
Native speakers, yes, I'd expect a normal distribution.
Learners, I'm not sure. Mostly I bet it wouldn't be symmetrical: you'd get a relatively steep increase until the midpoint, then a wide/shallow extension off toward infinity beyond it. The midpoint would be determined by when people on average stop studying/learning, and I don't know where that would necessarily be. Maybe on average at something like the two-years-of-class level?
Title: Re: Practical measures of proficiency
Post by: Corybobory on December 19, 2013, 04:31:25 AM
The Japanaese Language Proficiency Test is a standard test, that in conversation people can say to give a rough idea of fluency.  The problem is, like the problems discussed above, it doesn't give a resolution into whether you might be a strong or weak reader, or better with informal language or academic language.

That's a good point about the bell curve and language learners - there is always going to be more early level learners than intermediates or advanceds. It would be a changing 'language-scape' too, because of trends in langauge learning and language popularity I imagine.

I wonder if there can be a universal proficiency index used for all languages, or if they have to be language individual as they otherwise won't work well.

I was imagining something like the Myers Briggs personality test with the 4 letters that could indicate proficiency broken down into reading writing speaking listening, though that leaves out register.  Hmm...
Title: Re: Practical measures of proficiency
Post by: IronMike on February 28, 2014, 10:13:45 AM
I'd like to add to this discussion as one (hopefully not the only one) who has attended their nation's military language school (for U.S.: Defense Language Institute, or DLI), who was required to take language proficiency exams (Defense Language Proficiency Test) annually from 1986 to just a couple years ago. A couple of notes, none of which are based on science:

1) I like to talk about passive and active proficiency/skills. Under passive I put reading and listening (hold on, I'll explain) and under active, speaking and writing. Now I say passive purely based on my unscientific experience on how I study the language, take proficiency tests in the language, or work in the language. Reading and listening, I'm taking L2 input into my brain, which translates it into my L1 for me to understand. Then, if I want to respond to that input, I switch to active:  My brain takes my L1 response, and has to actively translate that into the L2 so that I can respond to my interlocutor.

2) This was the way I explained it to the military linguists I worked with when I would catch them always studying vocabulary one way:  L2 to L1.  I would tell them to flip their flashcards over and study L1 to L2.  Much harder.  But those who made a concerted effort at studying that way would do better on the speaking proficiency test.

3) Now I know the above can be language specific. In my experience, which was Slavic-heavy, most of my colleagues were very good at reading and listening, but only so-so at speaking. This was mostly due to the job not requiring speaking (and their study habits, as I describe above). But I had friends who were Chinese linguists, who would always do really well on the speaking and listening test and not so well on the reading. But I contend that if they knew the character, they'd be able to understand it passively, but the action of having to study so many characters and learning/memorizing them is an active skill.

4) With regards to practical measures of proficiency, personally I use the term "Berlitz level." If I can order food (beer!), get a hotel room, follow and ask for directions, buy a ticket to something, I say that I'm at "Berlitz level in X-language."  In ACTFL terms, I'd say that's about a 1 to a 1+.  And for most languages, I'm happy with that.
Title: Re: Practical measures of proficiency
Post by: sieledorothy on May 17, 2014, 12:31:21 AM
I think it's all in the ability to converse, to sustain a conversation on and on. If it's with a native speaker of that language, the better.
Title: Re: Practical measures of proficiency
Post by: Guijarro on May 18, 2014, 08:50:45 AM
As I have never thought seriously about the ways to measure proficiency, let me try a lateral thought on it.

Could PROFICIENCY be related to RELEVANCE in some way or other?

I mean,

(1) When I am with my Italian friends having a wonderful time out in, say, Ferrara, I do think that I am proficient in Italian. But when I am trying to read Dante Alligheri in that language and make out something about his ideas, I confess that I am less so.

(2) Similarly, when I write letters to my German friends, I tend to be fluent. But i don't consider myself fluent enough to write a paper on Linguistics in that language.

So, as relevance, fluency could probably be considered a function of the effects I try to achieve against the effort it would cost me in each case.

Does that make sense?
Title: Re: Practical measures of proficiency
Post by: Daniel on May 18, 2014, 09:48:19 AM
I think that's exactly it. "Fluent" of course is a label we apply when we find someone to be in general, across various domains fluent (fluid) in speaking a language. Being fluent in saying "hello" is not enough (or I'd be fluent in maybe 20 languages).

However, the arbitrary tasks you list may not be very reliable across learners or languages. Is the difference between ordering a cup of coffee and writing a linguistics article always constant?

But certainly, I would agree that proficiency relates to ability in accomplishing certain tasks, or just more generally ability to speak the language functionally in certain circumstances.

At the same time, there may be other measures of proficiency such as pronunciation metrics, which may not necessarily impede communication but still mark someone as non-native.
Title: Re: Practical measures of proficiency
Post by: freknu on May 18, 2014, 09:53:32 AM
If the documentary (http://linguistforum.com/linguist%27s-lounge/%28documentary%29-nova-evolution-the-mind%27s-big-bang/) I recently posted bears any weight then I would dare say that gossip would be a true test of fluency ;)

PS. It's towards the end, at 43:50 minutes.