Research visit at University of Sheffield

From November 11 and until November 29th I visited the Natural Language Processing group at the University of Sheffield, in the United Kingdom. The Short-Term Scientific Mission was part of the Transnational Access program of the SoBigData project, funded by the European Union under grant agreement No. 654024. The program provides support to researchers to travel to one of the SoBigData partner centers where they can engage researchers who are experts in their fields.

University building

I visited Prof. Kalina Bontcheva at the University of Sheffield who has done extensive research on Natural Language processing on social media text and, more recently, on spread of rumor veracity prediction, hyperpartisan news detection, and spread of misinformation. During the research visit, I continued my ITU research on multi-lingual stance classification. Stance classification is the task of classifying the stance of the author of a piece of text with respect to another piece of text. In my research I focus on social media content, predominantly from Twitter. For example, a Twitter user may post a claim such as, for example “I heard that the Eiffel tower is made out of cheese”. When other Twitter users react to such tweets, they often manifest a supportive stance (e.g. “I believe it’s all made out of Gouda”), disagreement (e.g. “Been there, checked, it’s definitely made of metal”), ask for clarifications (e.g. “Do you have any link to support that?”), or just simply comment on the reply without necessarily taking a clear stance. Automatically classifying the stance in social media text is an important Natural Language Processing task since stance labels provide information for validating claims and help in identification of rumors and fake news.  This has important implications for societal events such as democratic elections. The task is particularly challenging as it requires that the NLP systems learn to recognize relation patterns between words in tweets and their replies. Moreover, the amount of labeled data is quite small which poses a challenge for modern deep neural networks, such as recurrent neural networks with LSTMs which are known to require lots of training data to learn from. Training such models on languages other than English is even more difficult since there is even less labeled data for other languages. Finally, label imbalance in the data can cause machine learning NLP systems to tend to ignore those kinds of stances that appear less frequently.

With the assistance of Kalina Bontcheva and Carolina Scarton, I focused on understanding the difficulties of evaluating stance classification. Researchers at the University of Sheffield have extensive experience in running shared task competitions on stance classification and rumor identification, and on evaluating systems participating in the competition. One of the major hurdles in both training highly performing systems and in evaluation is label imbalance. Most tweets in currently available data sets are comments (i.e. they do not manifest a strong supporting, denying, or querying stance). Because of this, traditional measurement metrics such as accuracy, precision, recall, and F1 tend to be misleading of the system’s true performance. Therefore, alternative metrics must be used to understand the performance of stance classification systems. Macro-averaged metrics (versions of precision, recall, and F1 that assign equal importance to each kind of stance) are one such option. During the visit in Sheffield, I investigated the output of my own classifiers for stance classification in multiple languages by following Carolina’s work in using confusion matrices to understand where systems fail and why. Confusion matrices make is relatively easy to observe if a stance classifier has a tendency to ignore rare stances and provides information on what kind of stances are confused for each other. I discovered that utilization of class weights allows my classifier to address some of the problems characteristic of label imbalance, and came up with a strategy to perform transfer learning to let the classifiers learn from more data than what is included in the various stance classification data sets currently available. Transfer learning is a technique for training systems based on machine learning on two or more tasks that share some commonalities. The goal is for the system to improve its performance on a target task by benefiting from the similarities with the related tasks.

Now, upon returning to ITU in Copenhagen, I will continue research of the several promising paths identified while in Sheffield to improve the performance of my stance classification architecture.

~ Manuel R. Ciosici

EMNLP 2019

We are present at EMNLP 2019, Hong Kong:

Furthermore, we participated in the organization:
Natalie was EMNLP handbook proofreader
Barbara was Area Chair, and was honored as outstanding AC by the PC – we’re very honored! 

 

Verif-AI: against misinformation – DFF grant for Leon Derczynski

Leon Derczynski has won a 30-month grant from DFF, the Independent Danish Research Fund, who contribute 2.9M DKK of funding to the project. The project, Verif-AI, researches automatic detection of fake news, across many languages.

Misinformation and propaganda threaten our public discussions and try to change our attitudes and behaviours. This kind of manipulation is easy to produce at huge scales, making it hard for fact checkers to detect it and deal with it. This is a problem that we can use artificial intelligence against, and there have been some promising early results, but only really for English.

While there are many different kinds of misinformation and propaganda, one that’s efficient to work against is where there are false claims that we can provide evidence against. So, to improve the situation, Verif-AI investigates cross-lingual ways of finding misinformation, by detecting where claims have been made, and comparing the claims with knowledge bases like Danmarks Statistik.

Using AI tools called adversarial learning and multi-task learning, we can not only build fact verification technology for multiple languages: we can also benefit by adding together data from many different languages. What’s more, once there are models for cross-language fact checking, the project will adapt these to languages for which there’s no fact-checking data at all, bringing fact checking technologies to many languages at once, assisting journalists across the world.

Journalists play an important role in dealing with misinformation, and this is a core part of the project, too: we work with TjekDet at Mandag Morgen and also the EU “WeVerify” project to share news and data to a broad network of journalists, giving Verif-AI has real impact.

Verif-AI starts in 2020 and will run until mid-2022. You can reach the project lead at ld@itu.dk.

NLP@ITU at the first EurNLP summit

We are participating in the first EurNLP summit at Facebook in London this week!

  • Barbara is program co-chair of EurNLP, together with Sebastian Ruder (DeepMind) and with the general chairs from Facebook AI: Sebastian Riedel, Fabrizio Sebastiani and Armand Joulin –  EurNLP organization
  • Natalie is giving the talk “Neural Syntactic Parsing seems so simple. Is it?
  • Poster presentations by Alan and Manuel:

    Alan Ramponi, Barbara Plank and Rosario Lombardo. On the impact of cross-domain edge detection in biomedical event extraction

    Manuel Ciosici, Leon Derczynski and Ira Assent. Characterizing the information content of Brown clusters

Marija Stepanovic joins NLP at ITU

Marija is a new PhD student in computer science working on automatic speech recognition. Her prior education includes bachelor and master studies in theoretical and applied linguistics with a specialization in phonetics, phonology, and cognitive semantics, as well as master studies in computational cognitive science with an emphasis on machine learning and natural language processing. Her research incorporates linguistic, computational, and statistical analyses of spoken and textual data with the aim of identifying and modeling cognitive processes behind recurring concepts and patterns across languages for the purpose of bringing machines closer to truly understanding natural language.

 

At ITU, Marija will be working as a PhD student under the supervision of Barbara Plank. Her project is concerned with improving speech recognition for low-resource dialects of Danish and English through a comparative acoustic analysis of their vowel systems.

Marija

Where to find us: NODALIDA 2019

Being a research group located in the Nordics, ITU NLP has a strong presence at NODALIDA this year, held in Turku. The conference’s general chair is Barbara Plank from ITU NLP, for whose efforts we are all very grateful. You can find us here:

Monday September 30

NLPL Workshop on Deep Learning for Natural Language Processing, 09:00-17:00 PUB2

  • Co-organiser: Leon Derczynski (also first session chair, 09:20-10:00)

Deep Transfer Learning: Learning across Languages, Modalities and Tasks

Barbara Plank. Keynote, 10:30-11:30, NLPL DL4NLP, PUB2

Tuesday October 1

Lexical Resources for Low-Resource PoS Tagging in Neural Times.

Barbara Plank and Sigrid Klerke. Talk: 11:25-11:50, Parallel session A, PUB1

Bornholmsk Natural Language Processing: Resources and Tools.

Leon Derczynski and Alex Speed Kjeldsen. Poster: 16:45-17:45, Poster and demo session, Entrance hall

We introduce language processing resources and tools for Bornholmsk, a language spoken on the island of Bornholm, with roots in Danish and closely related to Scanian. This presents an overview of the language and available data, and the first NLP models for this living, minority Nordic language.

The Lacunae of Danish Natural Language Processing.

Andreas Kirkedal, Barbara Plank, Leon Derczynski and Natalie Schluter. Poster: 16:45-17:45, Poster and demo session, Entrance hall

Danish has received relatively little attention from a technological perspective. In this paper, we review Natural Language Processing (NLP) research, digital resources and tools which have been developed for Danish. We find that availability of models and tools is limited, which calls for work that lifts Danish NLP a step closer to the privileged languages.

UniParse: A universal graph-based parsing toolkit.

Daniel Varab and Natalie Schluter. Demo: 16:45-17:45 

Come by for a chat on how UniParse works and how it may be useful for your research.

Wednesday October 2, 2019

Political Stance Detection for Danish.

Rasmus Lehmann and Leon Derczynski. Talk: 11:10-11:35, Parallel session A: Sentiment Analysis and Stance, PUB1

The presented research concerns identification of the stance towards immigration within quotes from politicians brought in Danish newspapers. Covered in the presentation will be the creation of a dataset of stance annotated quotes from politicians in Danish, the first of its kind, along with the creation of two deep-learning based stance detection models, one using an LSTM architecture and one using a basic feed forward architecture, along with the results of testing these models.

Neural Cross-Lingual Transfer and Limited Annotated Data for Named Entity Recognition in Danish.

Barbara Plank. Talk: 11:10-11:35, Parallel session B: Named Entity Recognition, PUB3

Session chairing – Parallel session A: Text Generation and Language Model Applications
14:00-15:15, PUB1. Leon Derczynski

Joint Rumour Stance and Veracity Prediction.

Anders Edelbo Lillie, Emil Refsgaard Middelboe and Leon Derczynski. Talk: 11:35-12:00, Parallel session A: Sentiment Analysis and Stance, PUB1

We present an end-to-end stance and veracity prediction system that works at SotA level on Danish despite low data, and show that stance-based veracity prediction models can be transferred across languages and platforms with negligible performance drop.

NEALT business meeting

13:00-14:00, PUB1

Watch out for ✨exciting✨ items from ITU NLP here…

 

 

We hope to meet you in Turku!

Rob van der Goot joins NLP at ITU

Rob has a background in information science, but quickly became interested in the field of natural language processing, especially in the problem of building robust models. His expertise lies in automatically deriving syntactic analyses of natural language (parsing), with a focus on low-resource settings. During his PhD, he improved the automatic syntactic analysis of social media texts by first translating it to a more ‘standard’ form (try it yourself: www.robvandergoot.com/monoise). More broadly, he is interested in the automatic processing of all types of language varieties without having explicit training data.
 
Rob will be working at the ITU as a postdoc under supervision of Barbara Plank (partially funded by Amazon), together they will develop natural language processing models for low-resource languages and language varieties.
 
Rob van der Goot

Alan Ramponi joins NLP at ITU

Alan is a Ph.D. student in natural language processing at Fondazione The Microsoft Research – University of Trento COSBI, Italy. His research focuses on unsupervised domain adaptation and deep learning methods for biomedical information extraction from scientific publications. Broadly, his interests are centered on building robust language models which are resilient to domain shift, thus being readily applicable to real-world problems in which the target domain is not known in advance.

Alan will be doing his work as a visiting Ph.D. fellow with Barbara Plank, researching domain adaptation methods for all the stages of the task of biomedical event extraction.

Rasmus Lehmann joins NLP at ITU

We’re very happy to welcome Rasmus Lehmann to NLP at ITU!

Rasmus resides in the cross section between business, communication and technology, with a Bachelor’s degree within organizational communication and economics from CBS, and a Master’s degree within software development, specialized in Business Intelligence and Machine Learning. Rasmus’ interest in the field of NLP was aroused while working on implementing a deep learning-based model for use in rumor identification, and he continued to write his thesis, titled “Stance Detection in Danish Politics”. The focus of this project was to build a dataset of quotes from Danish politicians for use in stance detection in Danish, and applying a deep learning-based approach to solving this classification task. The project was subsequently turned into a submission for the NoDaLiDa 2019 conference on Computational Linguistics, to which the paper was accepted.

Rasmus will be working closely with Leon Derczynski on creating tools for NLP in Danish.

Rasmus Lehmann

Daniel Varab joins NLP at ITU

We are delighted to welcome Daniel Varab back to NLP at ITU! Daniel introduces himself:

“I come from a traditional computer science background and somewhat by chance ended up writing my thesis titled “Contradiction Detection in Natural Languages” in the scope of natural language inference (NLI). This sparked my interest in NLP and has caused all my work since to revolve around the field. I am now two years out after graduation and have spent a year as a research assistant exploring NLI and graph-based dependency parsing, followed by a year at the Danish/Swedish company Karnov Group where I have worked on helping lawyers navigate the ever-growing pile of legislation with the use of NLP techniques. I am now excited to be heading back into academia where I will be working on text summarization together with Natalie Schluter and supporting courses of ITU’s data science bachelor degree.

With regards to interests, I genuinely enjoy work on simple models with well-founded inductive biases, work on so-called less privileged languages, and good old thorough research.”

Daniel Varab