Results: 2017 shared task systems applied to 2018 test data
DISCLAIMER: This is not the official ranking of systems participating in the shared task.
The systems from the 2017 shared task applied to the 2018 test data. While this is an interesting comparison, note that the 2017 systems do not comply with the 2018 rules because they were trained on a different release of Universal Dependencies. It may affect the results in both ways: if the annotation of the treebank was improved, the old parser is handicapped. If the training-test split has changed and the old training data are now in test data, the old parser is favored undeservedly.
Also, the 2018 data contain eight new languages, and a number of new treebanks for the other languages. The rules of both tasks required that the parsers be robust enough and produce valid output for any language and treebank code, even if they are unknown. Most parsers ignored this request and simply crashed on unexpected language/treebank code; but even if they did not crash, their output on these languages is probably not interesting. Therefore, we define a subset of “old treebanks”, i.e., those that were evaluated in both years. The tables on this page are macro-averages over the old treebanks. There are also links to separate pages with per-treebank results.
Note that if a team has identical name in the two tasks, it does not necessarily mean that their 2018 system can be considered a new version of the one from 2017. Sometimes a team used completely different software in the two years. On the other hand, in some cases same people from the same institution participated under different team names (e.g., 2017 RACAI is NLP-Cube in 2018).
- Per-treebank LAS results, 2017 systems
- Per-treebank LAS results, 2018 systems
- Per-treebank MLAS results, 2017 systems
- Per-treebank MLAS results, 2018 systems
- Per-treebank BLEX results, 2017 systems
- Per-treebank BLEX results, 2018 systems
LAS ranking 2017 systems
1. Stanford (Stanford) software1-P 77.44 Fin: 2018-07-11-09-43-35 => 2018-07-13-15-06-23 2. C2L2 (Ithaca) software5-P 75.59 Fin: 2018-07-08-23-50-57 => 2018-07-09-14-25-46 3. ParisNLP (Paris) software1-P 72.67 2018-09-06-13-55-00 => 2018-09-16-10-12-46 4. HIT-SCIR (Harbin) software4-P 71.67 Fin: 2018-07-09-21-38-44 => 2018-07-10-16-31-45 5. Koç University (İstanbul) software3-P 70.84 Fin: 2018-07-08-23-52-09 => 2018-07-09-14-26-09 6. TurkuNLP (Turku) software1-P 70.23 Fin: 2018-07-08-23-53-00 => 2018-07-09-14-26-30 7. ÚFAL – UDPipe 1.2 (Praha) software1-P 69.62 Fin: 2018-07-08-23-52-18 => 2018-07-09-14-26-15 8. MQuni (Sydney) software2-P 69.53 Fin: 2018-07-14-11-13-20 => 2018-07-14-13-42-38 9. UParse (Edinburgh) software1-P 69.44 Fin: 2018-07-08-23-52-35 => 2018-07-09-14-26-20 10. fbaml (Palo Alto) software1-P 69.34 Fin: 2018-07-11-16-26-28 => 2018-07-14-08-39-18 11. LyS-FASTPARSE (A Coruña) software5-P 68.76 Fin: 2018-07-08-23-54-07 => 2018-07-11-13-20-34 12. darc (Tübingen) software1-P 68.71 Fin: 2018-07-09-23-06-04 => 2018-07-10-06-32-17 13. BASELINE UDPipe 1.1 (Praha) software2-P 68.44 Fin: 2018-07-08-23-53-34 => 2018-07-09-14-26-39 14. LIMSI (Paris) software2-P 68.23 Fin: 2018-07-09-22-22-27 => 2018-07-11-06-45-11 15. IIT Kharagpur (Kharagpur) software3-P 67.87 Fin: 2018-07-09-21-49-26 => 2018-07-09-23-14-21 16. Wanghao-ftd-SJTU (Shanghai) software2-P 66.72 Fin: 2018-07-09-22-30-33 => 2018-07-09-23-56-19 17. Orange – Deskiñ (Lannion) software1-P 66.57 Fin: 2018-07-08-23-52-43 => 2018-07-09-14-26-25 18. LATTICE (Paris) software7-P 65.48 Fin: 2018-07-10-16-41-29 => 2018-07-11-13-20-14 19. UALING (Tucson) software1-P 64.84 Fin: 2018-07-08-23-55-32 => 2018-07-09-14-27-05 20. METU (Ankara) software2-P 60.78 Fin: 2018-07-08-23-57-03 => 2018-07-09-14-27-13 21. CLCL (Genève) software1-P 59.35 Fin: 2018-07-09-23-20-06 => 2018-07-10-22-25-26 22. IMS (Stuttgart) software2-P 57.98 Fin: 2018-07-09-22-02-06 => 2018-07-12-19-06-17 23. Wenba-NLU (Wuhan) software1-P 57.94 Fin: 2018-07-14-13-37-01 => 2018-07-14-17-02-34 24. OpenU NLP Lab (Ra'anana) software6-P 55.30 Fin: 2018-07-08-23-57-40 => 2018-07-11-07-29-45 25. MetaRomance (Santiago de Compostela) software1-P 28.43 Fin: 2018-07-08-23-57-55 => 2018-07-09-14-27-30 26. UT (Tartu) software1-P 13.84 Fin: 2018-07-08-23-58-12 => 2018-07-09-18-40-06 27. RACAI (București) software1-P 12.38 Fin: 2018-07-12-19-40-43 => 2018-07-14-16-26-55 28. Uppsala (Uppsala) software1-P 10.83 Fin: 2018-07-12-21-07-36 => 2018-07-14-16-26-37 29. ECNU (Shanghai) software1-P 3.08 Fin: 2018-07-08-23-58-31 => 2018-07-10-06-32-10
LAS ranking 2018 systems
1. HIT-SCIR (Harbin) software1-P 82.23 2. TurkuNLP (Turku) software1-P 79.67 3. ICS PAS (Warszawa) software1-P 79.61 4. UDPipe Future (Praha) software1-P 79.54 5. Stanford (Stanford) software2 79.53 6. LATTICE (Paris) software1-P 78.84 7. CEA LIST (Paris) software1-P 78.76 8. Uppsala (Uppsala) software1-P 78.24 9. NLP-Cube (București) software1-P 78.03 10. ParisNLP (Paris) software1-P 77.60 11. AntNLP (Shanghai) software1-P 77.37 12. SLT-Interactions (Bengaluru) software2-P 77.08 13. UniMelb (Melbourne) software1-P 75.26 14. IBM NY (Yorktown Heights) software1-P 74.93 15. LeisureX (Shanghai) software1-P 74.82 16. KParse (İstanbul) software1-P 72.67 17. Fudan (Shanghai) software5-P 72.44 18. BASELINE UDPipe 1.2 (Praha) software1-P 72.05 19. Phoenix (Shanghai) software1-P 71.80 20. BOUN (İstanbul) software2-P 69.73 21. CUNI x-ling (Praha) software1-P 69.57 22. ONLP lab (Ra'anana) software3-P 63.52 23. iParse (Pittsburgh) software1-P 61.32 24. HUJI (Yerushalayim) software1-P 60.64 25. ArmParser (Yerevan) software1-P 52.08 26. SParse (İstanbul) software1-P 2.63
MLAS ranking 2017 systems
1. C2L2 (Ithaca) software5-P 59.78 Fin: 2018-07-08-23-50-57 => 2018-07-09-14-25-46
2. Koç University (İstanbul) software3-P 56.79 Fin: 2018-07-08-23-52-09 => 2018-07-09-14-26-09
3. UParse (Edinburgh) software1-P 55.55 Fin: 2018-07-08-23-52-35 => 2018-07-09-14-26-20
4. LyS-FASTPARSE (A Coruña) software5-P 55.15 Fin: 2018-07-08-23-54-07 => 2018-07-11-13-20-34
5. LIMSI (Paris) software2-P 54.94 Fin: 2018-07-09-22-22-27 => 2018-07-11-06-45-11
6. IIT Kharagpur (Kharagpur) software3-P 54.42 Fin: 2018-07-09-21-49-26 => 2018-07-09-23-14-21
7. MQuni (Sydney) software2-P 53.55 Fin: 2018-07-14-11-13-20 => 2018-07-14-13-42-38
8. TurkuNLP (Turku) software1-P 52.64 Fin: 2018-07-08-23-53-00 => 2018-07-09-14-26-30
9. ÚFAL – UDPipe 1.2 (Praha) software1-P 52.13 Fin: 2018-07-08-23-52-18 => 2018-07-09-14-26-15
10. HIT-SCIR (Harbin) software4-P 51.81 Fin: 2018-07-09-21-38-44 => 2018-07-10-16-31-45
11. darc (Tübingen) software1-P 51.43 Fin: 2018-07-09-23-06-04 => 2018-07-10-06-32-17
12. UALING (Tucson) software1-P 51.17 Fin: 2018-07-08-23-55-32 => 2018-07-09-14-27-05
13. BASELINE UDPipe 1.1 (Praha) software2-P 51.02 Fin: 2018-07-08-23-53-34 => 2018-07-09-14-26-39
14. LATTICE (Paris) software7-P 50.88 Fin: 2018-07-10-16-41-29 => 2018-07-11-13-20-14
15. Wanghao-ftd-SJTU (Shanghai) software2-P 49.49 Fin: 2018-07-09-22-30-33 => 2018-07-09-23-56-19
16. fbaml (Palo Alto) software1-P 48.61 Fin: 2018-07-11-16-26-28 => 2018-07-14-08-39-18
17. CLCL (Genève) software1-P 47.34 Fin: 2018-07-09-23-20-06 => 2018-07-10-22-25-26
18. Wenba-NLU (Wuhan) software1-P 45.11 Fin: 2018-07-14-13-37-01 => 2018-07-14-17-02-34
19. IMS (Stuttgart) software2-P 34.19 Fin: 2018-07-09-22-02-06 => 2018-07-12-19-06-17
20. OpenU NLP Lab (Ra'anana) software6-P 14.23 Fin: 2018-07-08-23-57-40 => 2018-07-11-07-29-45
21. Stanford (Stanford) software1-P 12.56 Fin: 2018-07-11-09-43-35 => 2018-07-13-15-06-23
22. ParisNLP (Paris) software1-P 11.95 2018-09-06-13-55-00 => 2018-09-16-10-12-46
23. Orange – Deskiñ (Lannion) software1-P 10.13 Fin: 2018-07-08-23-52-43 => 2018-07-09-14-26-25
24. RACAI (București) software1-P 9.49 Fin: 2018-07-12-19-40-43 => 2018-07-14-16-26-55
25. UT (Tartu) software1-P 9.40 Fin: 2018-07-08-23-58-12 => 2018-07-09-18-40-06
26. MetaRomance (Santiago de Compostela) software1-P 1.76 Fin: 2018-07-08-23-57-55 => 2018-07-09-14-27-30
27. METU (Ankara) software2-P 0.94 Fin: 2018-07-08-23-57-03 => 2018-07-09-14-27-13
28. ECNU (Shanghai) software1-P 0.00 Fin: 2018-07-08-23-58-31 => 2018-07-10-06-32-10
Uppsala (Uppsala) software1-P 0.00 Fin: 2018-07-12-21-07-36 => 2018-07-14-16-26-37
MLAS ranking 2018 systems
1. UDPipe Future (Praha) software1-P 68.48 2. Stanford (Stanford) software2 68.36 3. TurkuNLP (Turku) software1-P 68.10 4. ICS PAS (Warszawa) software1-P 67.11 5. CEA LIST (Paris) software1-P 66.95 6. HIT-SCIR (Harbin) software1-P 66.67 7. Uppsala (Uppsala) software1-P 65.74 8. NLP-Cube (București) software1-P 64.37 9. LATTICE (Paris) software1-P 63.10 10. ParisNLP (Paris) software1-P 62.32 11. AntNLP (Shanghai) software1-P 62.26 12. SLT-Interactions (Bengaluru) software2-P 61.69 13. UniMelb (Melbourne) software1-P 59.93 14. LeisureX (Shanghai) software1-P 59.74 15. KParse (İstanbul) software1-P 59.35 16. Fudan (Shanghai) software5-P 58.57 17. BASELINE UDPipe 1.2 (Praha) software1-P 58.47 18. Phoenix (Shanghai) software1-P 58.32 19. BOUN (İstanbul) software2-P 56.33 20. CUNI x-ling (Praha) software1-P 55.68 21. ONLP lab (Ra'anana) software3-P 51.12 22. HUJI (Yerushalayim) software1-P 50.99 23. iParse (Pittsburgh) software1-P 50.28 24. IBM NY (Yorktown Heights) software1-P 44.48 25. ArmParser (Yerevan) software1-P 40.25 26. SParse (İstanbul) software1-P 2.26
BLEX ranking 2017 systems
1. Stanford (Stanford) software1-P 67.80 Fin: 2018-07-11-09-43-35 => 2018-07-13-15-06-23
2. C2L2 (Ithaca) software5-P 65.72 Fin: 2018-07-08-23-50-57 => 2018-07-09-14-25-46
3. ParisNLP (Paris) software1-P 62.84 2018-09-06-13-55-00 => 2018-09-16-10-12-46
4. Koç University (İstanbul) software3-P 61.00 Fin: 2018-07-08-23-52-09 => 2018-07-09-14-26-09
5. UParse (Edinburgh) software1-P 59.68 Fin: 2018-07-08-23-52-35 => 2018-07-09-14-26-20
6. MQuni (Sydney) software2-P 59.58 Fin: 2018-07-14-11-13-20 => 2018-07-14-13-42-38
7. LIMSI (Paris) software2-P 58.99 Fin: 2018-07-09-22-22-27 => 2018-07-11-06-45-11
8. IIT Kharagpur (Kharagpur) software3-P 58.08 Fin: 2018-07-09-21-49-26 => 2018-07-09-23-14-21
9. HIT-SCIR (Harbin) software4-P 57.05 Fin: 2018-07-09-21-38-44 => 2018-07-10-16-31-45
10. LATTICE (Paris) software7-P 55.98 Fin: 2018-07-10-16-41-29 => 2018-07-11-13-20-14
11. TurkuNLP (Turku) software1-P 55.51 Fin: 2018-07-08-23-53-00 => 2018-07-09-14-26-30
12. UALING (Tucson) software1-P 54.91 Fin: 2018-07-08-23-55-32 => 2018-07-09-14-27-05
13. LyS-FASTPARSE (A Coruña) software5-P 54.76 Fin: 2018-07-08-23-54-07 => 2018-07-11-13-20-34
14. darc (Tübingen) software1-P 54.45 Fin: 2018-07-09-23-06-04 => 2018-07-10-06-32-17
15. BASELINE UDPipe 1.1 (Praha) software2-P 54.04 Fin: 2018-07-08-23-53-34 => 2018-07-09-14-26-39
16. Wanghao-ftd-SJTU (Shanghai) software2-P 51.87 Fin: 2018-07-09-22-30-33 => 2018-07-09-23-56-19
17. CLCL (Genève) software1-P 51.31 Fin: 2018-07-09-23-20-06 => 2018-07-10-22-25-26
18. ÚFAL – UDPipe 1.2 (Praha) software1-P 49.07 Fin: 2018-07-08-23-52-18 => 2018-07-09-14-26-15
19. Wenba-NLU (Wuhan) software1-P 48.88 Fin: 2018-07-14-13-37-01 => 2018-07-14-17-02-34
20. IMS (Stuttgart) software2-P 40.43 Fin: 2018-07-09-22-02-06 => 2018-07-12-19-06-17
21. OpenU NLP Lab (Ra'anana) software6-P 24.71 Fin: 2018-07-08-23-57-40 => 2018-07-11-07-29-45
22. MetaRomance (Santiago de Compostela) software1-P 22.31 Fin: 2018-07-08-23-57-55 => 2018-07-09-14-27-30
23. RACAI (București) software1-P 10.53 Fin: 2018-07-12-19-40-43 => 2018-07-14-16-26-55
24. UT (Tartu) software1-P 10.23 Fin: 2018-07-08-23-58-12 => 2018-07-09-18-40-06
25. Orange – Deskiñ (Lannion) software1-P 0.17 Fin: 2018-07-08-23-52-43 => 2018-07-09-14-26-25
fbaml (Palo Alto) software1-P 0.17 Fin: 2018-07-11-16-26-28 => 2018-07-14-08-39-18
27. METU (Ankara) software2-P 0.10 Fin: 2018-07-08-23-57-03 => 2018-07-09-14-27-13
28. ECNU (Shanghai) software1-P 0.00 Fin: 2018-07-08-23-58-31 => 2018-07-10-06-32-10
Uppsala (Uppsala) software1-P 0.00 Fin: 2018-07-12-21-07-36 => 2018-07-14-16-26-37
BLEX ranking 2018 systems
1. TurkuNLP (Turku) software1-P 73.18 2. HIT-SCIR (Harbin) software1-P 72.17 3. UDPipe Future (Praha) software1-P 71.73 4. ICS PAS (Warszawa) software1-P 71.67 5. Stanford (Stanford) software2 71.56 6. CEA LIST (Paris) software1-P 68.79 7. LATTICE (Paris) software1-P 68.73 8. ParisNLP (Paris) software1-P 67.67 9. AntNLP (Shanghai) software1-P 67.45 10. SLT-Interactions (Bengaluru) software2-P 66.97 11. UniMelb (Melbourne) software1-P 65.32 12. LeisureX (Shanghai) software1-P 64.98 13. NLP-Cube (București) software1-P 62.49 14. BASELINE UDPipe 1.2 (Praha) software1-P 62.08 15. Phoenix (Shanghai) software1-P 61.91 16. KParse (İstanbul) software1-P 61.40 17. Fudan (Shanghai) software5-P 59.83 18. BOUN (İstanbul) software2-P 59.63 19. CUNI x-ling (Praha) software1-P 59.20 20. HUJI (Yerushalayim) software1-P 54.91 21. iParse (Pittsburgh) software1-P 53.92 22. ArmParser (Yerevan) software1-P 43.46 23. IBM NY (Yorktown Heights) software1-P 35.46 24. Uppsala (Uppsala) software1-P 34.33 25. ONLP lab (Ra'anana) software3-P 30.70 26. SParse (İstanbul) software1-P 2.30