Results

This is the main ranking of the systems by their macro-averaged LAS F1 score. The ±range corresponds to 95% confidence interval computed with bootstrap resampling. The [OK] means all 81 test sets were parsed, otherwise the number in square brackets is the number of test sets with non-zero LAS. More details are provided at separate pages linked below.

 1. Stanford (Stanford)                    76.30 ± 0.12 [OK]
 2. C2L2 (Ithaca)                          75.00 ± 0.12 [OK]
 3. IMS (Stuttgart)                        74.42 ± 0.13 [OK]
 4. HIT-SCIR (Harbin)                      72.11 ± 0.14 [OK]
 5. LATTICE (Paris)                        70.93 ± 0.13 [OK]
 6. NAIST SATO (Nara)                      70.14 ± 0.13 [OK]
 7. Koç University (İstanbul)              69.76 ± 0.13 [OK]
 8. ÚFAL – UDPipe 1.2 (Praha)              69.52 ± 0.13 [OK]
 9. UParse (Edinburgh)                     68.87 ± 0.14 [OK]
10. Orange – Deskiñ (Lannion)              68.61 ± 0.13 [OK]
11. TurkuNLP (Turku)                       68.59 ± 0.14 [OK]
12. darc (Tübingen)                        68.41 ± 0.13 [OK]
13. BASELINE UDPipe 1.1 (Praha)            68.35 ± 0.14 [OK]
14. MQuni (Sydney)                         68.05 ± 0.13 [OK]
15. fbaml (Palo Alto)                      67.87 ± 0.13 [OK]
16. LyS-FASTPARSE (A Coruña)               67.81 ± 0.13 [OK]
17. LIMSI-LIPN (Paris)                     67.72 ± 0.14 [OK]
18. RACAI (București)                      67.71 ± 0.13 [OK]
19. IIT Kharagpur (Kharagpur)              67.61 ± 0.14 [OK]
20. naistCL (Nara)                         67.59 ± 0.15 [OK]
21. Wanghao-ftd-SJTU (Shanghai)            66.53 ± 0.14 [OK]
22. UALING (Tucson)                        65.24 ± 0.13 [OK]
23. Uppsala (Uppsala)                      65.11 ± 0.13 [OK]
24. METU (Ankara)                          61.98 ± 0.14 [OK]
25. CLCL (Genève)                          61.82 ± 0.13 [OK]
26. Mengest (Shanghai)                     61.33 ± 0.14 [79]
27. ParisNLP (Paris)                       60.02 ± 0.14 [OK]
28. OpenU NLP Lab (Ra'anana)               56.56 ± 0.14 [OK]
29. TRL (Tokyo)                            43.07 ± 0.12 [OK]
30. MetaRomance (Santiago de Compostela)   34.05 ± 0.11 [78]
31. UT (Tartu)                             21.10 ± 0.05 [27]
32. ECNU (Shanghai)                         3.18 ± 0.02 [36]
33. Wenba-NLU (Wuhan)                       0.58 ± 0.01 [46]

Other rankings

Outputs of system and evaluation runs are available from http://hdl.handle.net/11234/1-2424.