Task 1 : Classification, extraction and normalization of adverse effect (AE) mentions in English tweets

In this task, modified from previous years, systems developed must develop one or more components to classify tweets that contain an adverse effect (AE) or also known as adverse drug effect (ADE), detect the text span of reported ADEs in tweets and map these colloquial mentions to their standard concept IDs in the MedDRA vocabulary (preferred terms). This task presents multiple challenges. Firstly, the classification task needs to take into account class imbalance where only around 7% of the tweets contain ADEs. Secondly, span detection will require advanced named entity recognition approaches. Finally, the resolution task will additionally require choosing a normalized concept from more than 23,000 MedDRA preferred terms.

Participants will be provided with a labeled training set containing tweet texts and ADE annotations with the option of participating in one or more subtasks. This task contains three subtasks in increasing order of complexity:

  • Training data: 18,000 tweets
  • Test data: 10,000 tweets

Register your team here : https://forms.gle/1qs3rdNLDxAph88n6
After registration approval, you will be invited to join the Google group for the task. Link to the dataset is available in the Google groups banner. If you do not receive the invite please request to join the Google group with team name using the link below.
Google groups : https://groups.google.com/g/smm4h21-task-1
Link to Codalab : https://competitions.codalab.org/competitions/28766

Evaluation Period for Task 1 :

Test Dataset Release26th Feb 2021 12:00am UTC
Predictions Due28th Feb 2021 11:59pm UTC (3:59pm PST)
All submissions are automated and time limits are enforced by Codalab. No extensions will be provided.

Subtask 1a : ADE tweet classification

Given a tweet, participants of this subtask will be required to submit only the binary annotations ADE/noADE. A tweet should be assigned the label ADE if and only if it has one or more mentions of an ADE.

Submission format: Please use the format below for submission. Submissions should contain tweet_id and label separated by tabspace in the same order as below. Predictions for each subtask should be contained in a single .tsv (tab separated values) file. This file (and only this file) should be compressed into a .zip file. Please upload this zip file as submission.

tweet_idlabel
699ADE
310NoADE
653NoADE
872NoADE
751ADE
469NoADE
Submission format for Subtask 1a

You may also ignore predictions that are not ADE as shown below.

tweet_idlabel
699ADE
751ADE
Alternative submission format for Subtask 1a

Evaluation Metric : Submissions will be ranked by Precision, Recall and F1-score for the ADE class.

Subtask 1b : ADE span detection (includes annotations for Subtask 1a)

Participants of the ADE span detection subtask will be required to submit both the ADE classification labels and spans of expressed ADE.

Submission format: Please use the format below for submission. Submissions should contain tweet_id, label, start, end and span separated by tabspace in the same order as below. You may use the same submission format for Subtask 1a and start, end and span columns will be ignored. Predictions for each subtask should be contained in a single .tsv (tab separated values) file. This file (and only this file) should be compressed into a .zip file. Please upload this zip file as submission.

tweet_idlabelstartendspan
699ADE111118fatigue
699ADE126130achy
751ADE2641lower back ache
751ADE4352neck ache
751ADE5458hips
751ADE6373knees ache
751ADE7590short of breath
Submission format for Subtask 1b

Evaluation Metric : Submissions will be ranked by Precision, Recall and F1-score for each ADE extracted where the spans overlap either entirely or partially.

Subtask 1c : ADE resolution (includes annotations for Subtask 1a and 1b)

Participants of the ADE resolution subtask will be required to submit ADR classification labels, spans and normalization labels.

Submission format: Submissions should contain tweet_id, label, start, end and span separated by tabspace in the same order as below. You may use the same submission format for Subtask 1a where start, end, span and ptid columns will be ignored. Similarly, you may use the same submission format for Subtask 1b where ptid column will be ignored. Predictions for each subtask should be contained in a single .tsv (tab separated values) file. This file (and only this file) should be compressed into a .zip file. Please upload this zip file as submission.

tweet_idlabelstartendspanptid
699ADE111118fatigue10016256
699ADE126130achy10033371
751ADE2641lower back ache10003988
751ADE4352neck ache10028836
751ADE5458hips10003239
751ADE6373knees ache10003239
751ADE7590short of breath10040604
Submission format for Subtask 1c

Evaluation Metric : Submissions will be ranked by Precision, Recall and F1-score for each ADE extracted where the spans overlap either entirely or partially AND each span is normalized to the correct MedDRA preferred term ID.

Contact information: Arjun Magge (Arjun.Magge@pennmedicine.upenn.edu)

Frequently asked questions (FAQs):

Do I have to participate in all the subtasks?
No. You may choose to participate in one, two or all three subtasks. During evaluation, you will be allowed to make two submissions for each subtask.

How do I access MedDRA preferred terms (PT) and lower level terms (LLT)?
You can download MedDRA from https://www.meddra.org/. It is free for academic institutions and you may subscribe online at https://www.meddra.org/subscription/subscription-form. Once you download MedDRA, you will find PTs, LLTs and their mapping in the llt.asc file.