home edit page issue tracker
This page still pertains to UD version 1.

Uppsala Group on Tokenisation

(Chris Manning, Francis Tyers, Hèctor Martínez Alonso, Huner Kaşıkara, Aibek Makazhanov)

>1 token is 1 word

Here we define a token as a space delimited sequence of characters.

In CoNLL-U format, there is a restriction that “Fields must not contain space characters.”.1 This is problematic for a number of languages, as a word may consist of more than one token.

Outcome

We came to the conclusion that the CoNLL-U format should allow multi-token words. The options were to use space itself, another spacing character (e.g. ZWNJ) or to use some punctuation character (such as #, _, ⎵, etc.) The opinion of the group was that as CoNLL-U format is tab-separated, it would not be problematic to allow spaces.

Linguistic examples

Vietnamese

“As a result of influence from the Chinese writing system, each syllable in Vietnamese is written separately as if it were a word. In the past, syllables in multisyllabic words were concatenated with hyphens, but this practice had died out, and hyphenation is now reserved for foreign borrowings.”2

Đảng  Cộng⎵sản  Việt⎵Nam
Party Communist Vietnam

Danh⎵sách      quốc⎵gia  xã⎵hội⎵chủ⎵nghĩa
List.of.names  state     socialist

In a wordlist for Vietnamese extracted from Wiktionary,3 over a third of headwords were multisyllabic.

Tuvan

Мен келген мен.
I   come.PAST SG1

Ол келген.
He come.PAST SG3

Мен келген-дыр     мен.
I   come.PAST.EVID SG1

Turkish

 Eve   geliyor      musun?
 Home  come.PROG    QST.SG2 ?

But:

 Eve   geldin         mi?
 Home  come.PAST.SG2  QST?

1 token is >1 word

Question word

(kaz) Нан бар ма?
(kir) Нан барбы?
(tat) Ипи бармы?
(tyv) Хлеб бар бе? 
(tur) Ekmek var mı?
(chv) Çăкăр пур-и?

"Bread existing [is] QST?", "Is there bread?"

For the Tatar and Kyrgyz examples we can use, for example:

1       Нан       нан     NOUN
2-3     барбы     _       _ 
2       бар       бар     ADJ
3       бы        бы      PART
4       ?         ?       PUNCT

or:

1       Нан       нан     NOUN
2-3     барбы     _       _
2       _         бар     ADJ
3       _         бы      PART
4       ?         ?       PUNCT

Copula

Мен  студентпен.
I    student.COP.SG1 

Onun tutkusu       spor     arabalardı.
His  passion.SG3   sport's  car.PL.COP.PAST.SG3

We can separate the copula using:

1    Onun         o        PRON
2    tutkusu      tutku    NOUN
3    spor         spor     NOUN
4-5  arabalardı   _        _
4    arabalar     araba    NOUN
5    dı           i        AUX
6    .            .        PUNCT

or:

1    Onun         o        PRON
2    tutkusu      tutku    NOUN
3    spor         spor     NOUN
4-5  arabalardı   _        _
4    _            araba    NOUN
5    _            i        AUX
6    .            .        PUNCT

Productive “derivations” (-DAGI, -NIKI, -LIK, -LI, -sIz)

The -DAGI suffix has been described well here.

Ben mavi  arabadakileri      gördüm.
I   blue  car.LOC.KI.PL.ACC  see.PAST.SG1
"I see the ones in the blue car."

Mavi arabadakiler        gazete         okuyor.
Blue car.LOC.KI.PL.NOM   newspaper.ACC  read.PROG.3
"The ones in the blue car read the newspaper."

The idea for these is to allow, for example:

0     Ben             Ben      PRON     NUMBER=SG|PERSON=1|CASE=NOM
1     mavi            mavi     ADJ      _
2-3   arabadakileri   _        _        _
2     _               araba    NOUN     CASE=LOC
3     _               ki       X        NUMBER=PL|CASE=ACC
4     gördüm          gör      VERB     TENSE=PAST|NUMBER=SG|PERSON=1

or:

0     Ben             Ben      PRON     NUMBER=SG|PERSON=1|CASE=NOM
1     mavi            mavi     ADJ      _
2-3   arabadakileri   _        _        _
2     arabada         araba    NOUN     CASE=LOC
3     kileri          ki       X        NUMBER=PL|CASE=ACC
4     gördüm          gör      VERB     TENSE=PAST|NUMBER=SG|PERSON=1

Depending on the language, recovering anything sensible for the sub-surface forms may be more or less difficult.

The -NIKI suffix (or in Turkish -NInki) works similarly to the -DAGI suffix, but for the genitive case.

Ben  adamınkini      gördüm.
I    man.GEN.KI.ACC  see.PAST.SG1
"I saw the man's ones."

The -LI morpheme creates attributives from bare noun phrases:

Бир  палаталы     парламент
One  chamber.LI   parliament
"Unicameral parliament" (not "One-chamberly parliament")

The -LIK morpheme works similarly to the -LI morpheme.

The -sIz morpheme is sometimes called the abessive case, corresponding to the preposition ‘without’. It could also be compared with the -less derivational morpheme in English. It creates attributive (like an adjective), adverbial or substantive phrases from a bare noun. Sometimes words with -sIz in can be lexicalised, for example like “evsiz” (home.SIZ “homeless”).

Ол   хабарсыз  кетеді.
He   news.SIZ  vanished
"He vanished without news."

Kayıt         belgesizlere         2 bin       TL ceza kesilecek.
Registration  document.SIZ.PL.DAT  2 thousand  TL fine cut.PASS.FUT.
"Those without registration documents will be fined 2,000 TL"

Causative

Babam           arabayı   Ali ustaya      yaptırmış
Father.SG1      car.ACC   Ali master.DAT  fix.CAUS.EVID
"My father made master Ali fix the car."

The thoughts were for this construction to use a separate relation, for example nmod:caus or nsubj:caus for the causative subject (causee) of a causative verb.