Syllabification

[annacrook]

Hello !

I am currently working on a mechanism for automatic text difficulty assessment for English, French and German. One of the difficulty parameters is a number of syllables per word/sentence. For the former two languages we have the syllabification tools, but not for German. Does anybody have a slightest idea of where I can take it from ? An algorithm, an on/offline service, an API of such a service ?

Thank you,
Anna

[Daniel]

It doesn't seem that hard to build. You'd have to check on all of the details, but as far as I can tell at the moment (just thinking about examples) German syllabification involves the number of vowels except when those vowels are in a diphthong, which is a regular pattern. There are no consonant-only syllables (like English rhythm) that I can think of at the moment (maybe some very unusual words? probably not a major statistical problem), and almost everything is spelled regularly (except for perhaps some borrowed words and any typos). So any sequence of a, e, i, o, u, ä, ö, ü, eu, ie, ei, au, and probably a few more that I'm not thinking of, and that's it. There might be a few cases where two adjacent vowels are in different syllables but that is often indicated as in geh-en. Probably some ge- prefixed words would have e-e across two syllables.

Anyway, to build a 98% accurate algorithm should not be especially difficult.

[annacrook]

Hello djr33 !

Thank you for your interest in the topic. As for the syllabification algorithm, I spoke to my colleagues about your suggestion and they say that when implying such an approach for English language, it gave a somewhat considerable ratio of error. May be it can work better for German, I'll try to research on it. If not, the ideal variant for me would be a database of some 150000 words that, among all, contains the info on the number of syllables for each word. An API for any off/online syllabificator can also match my requirements.

Have a great day and thanx for help,
Anna

[Daniel]

English spelling is substantially more problematic than German. It will still possibly have some problems, but I'm certain it will be better than English.

[jkpate]

The Bonn Machine-Readable Pronouncing dictionary has entries for 141,548 German word forms, with syllable boundaries indicated by ".". It is distributed under a license that allows use for non-commercial, non-military work. If you want to use it for commercial purposes, you might be able to work out a commercial license with the bofu project.

Are you implementing something like Flesch-Kincaid reading levels? Keep in mind that these reading levels offer at most a very coarse assessment of readability, since they completely ignore things like word frequency and syntactic factors.

[Copernicus]

Hello !

I am currently working on a mechanism for automatic text difficulty assessment for English, French and German. One of the difficulty parameters is a number of syllables per word/sentence. For the former two languages we have the syllabification tools, but not for German. Does anybody have a slightest idea of where I can take it from ? An algorithm, an on/offline service, an API of such a service ?

I assume you know about standard measures for that sort of thing such as the Gunning-Fogg Index.

Generally speaking, though, I have a lot of reservations about difficulty assessments that are tied too closely to the phonological aspects of language.  It is true that alphabetic languages have a close coupling with phonological representation in principle, but English graphemics can be quite complex.  When it comes to reading, we are dealing primarily with a visual medium.  Those of us lucky enough to have an alphabetic writing system can associate symbols with single phonemes or strings of them, and that is very useful for building up vocabulary quickly.  That is, you can discover what an unfamiliar spelling is by mentally converting it to pronunciation and then associating it with phonological memory.  However, skilled readers tend to develop visual memories of word shapes and higher level phraseology.  So foreign readers, who do not necessarily have a large phonology-based vocabulary, will not necessarily perceive textual difficulty in quite the same way.