Social Media Mining for Health 2023 (#SMM4H)


  • Workshop
  • Shared Task 
  • Workshop Program 
  • Past events 
Workshop
SMM4H logo

The Social Media Mining for Health Applications (#SMM4H) Workshop serves as a venue for bringing together researchers interested in automatic methods for the collection, extraction, representation, analysis, and validation of social media data (e.g., Twitter, Reddit) for health informatics. The 8th #SMM4H Workshop invites the submission of papers on original, completed, and unpublished research in all aspects at the intersection of social media mining and health. Paper submissions may consist of up to 4 pages (including references) and must follow the AMIA formatting requirements. In order for accepted papers to be included in the workshop proceedings, at least one author must register for and present at the #SMM4H 2023 Workshop.


Submit workshop papers here: https://softconf.com/n/SMM4H-2023/


Important Dates (tentative)

Submission site open July 31, 2023
Submission deadline August 11, 2023
Notification of acceptance September 15, 2023
Camera-ready papers due September 29, 2023
Workshop November 11 or 12, 2023 (TBA)

Organizers

Graciela Gonzalez-Hernandez, Cedars-Sinai Medical Center, USA

Ari Z. Klein, University of Pennsylvania, USA

Ivan Flores, Cedars-Sinai Medical Center, USA

Abeed Sarker, Emory University, USA

Yuting Guo, Emory University, USA

Juan M. Banda, Georgia State University, USA

Raul Rodriguez-Esteban, Roche Pharmaceuticals, Switzerland

Lucia Schmidt, Roche Pharmaceuticals, Switzerland

Dongfang Xu, Cedars-Sinai Medical Center, USA

Shared Task 

The Social Media Mining for Health Applications (#SMM4H) Shared Tasks address natural language processing (NLP) challenges of using social media data for health informatics, including informal, colloquial expressions, misspellings, noise, data sparsity, ambiguity, and multilingual posts. For each of the 5 tasks below, teams will be provided with annotated training and validation data to develop their systems, followed by 5 days during which they will run their systems on unlabeled test data and upload their predictions to CodaLab. The individual CodaLab site for each task can be found below. Teams may upload up to 2 sets of predictions per task.

Please use this form to register. When your registration is approved, you will be invited to a Google group, where the data sets will be made available. Registered teams are required to submit a paper describing their systems. System descriptions may consist of up to 2 pages (including references) and must follow the AMIA formatting requirements. Teams participating in multiple tasks are permitted an additional page. Sample system descriptions can be found in past proceedings. In order for accepted system descriptions to be included in the proceedings, at least one author must register for and present at the #SMM4H 2023 Workshop. 

Submit system description papers here: https://softconf.com/n/SMM4H-2023/

Training and validation data available April 24, 2023
System predictions for validation data due June 30, 2023 (23:59 CodaLab server time)
Test data available July 10, 2023
System predictions for test data due July 14, 2023 (23:59 CodaLab server time)
Submission site open for system description papers July 31, 2023
Submission deadline for system description papers August 11, 2023
Notification of acceptance September 15, 2023
Camera-ready papers due September 29, 2023
Workshop November 11 or 12, 2023 (TBA)
* All deadlines are 11:59 PM UTC (3:59 PM PST), NO extension will be provided
Task 1 - Binary classification of English tweets self-reporting a COVID-19 diagnosis

To facilitate the use of Twitter data for monitoring personal experiences of COVID-19 in real time and on a large scale, this binary classification task involves automatically distinguishing tweets that self-report a COVID-19 diagnosis (annotated as "1")—for example, a postitive test, clinical diagnosis, or hospitalization—from those that do not (annotated as "0"). By this definition, a tweet that merely states that the user has experienced COVID-19 would not be considered a diagnosis. The training data include the Tweet ID, the text of the Tweet Object, and the annotated binary label. System predictions for the validation and test data should be submitted through CodaLab. Submissions should be formatted as a ZIP file containing a TSV file with only two columns: the tweet_id column first and the label column second, separted by a tab. The TSV file should not be in a folder in the ZIP file, and the ZIP file should not contain any files or folders other than the TSV file. The TSV file should be named prediction_task1.tsv.

 

  • Training data: 7,600 tweets
  • Validation data: 400 tweets
  • Test data: 10,000 tweets
  • Evaluation metric: F1-score for the "positive" class (i.e., tweets that self-report a COVID-19 diagnosis)


 

Contact: Ari Klein, University of Pennsylvania, USA (ariklein@pennmedicine.upenn.edu)

CodaLab: https://codalab.lisn.upsaclay.fr/competitions/12763

Task 2 - Multi-class classification of sentiment associated with therapies in English tweets

There is an abundance of health-related data on social networks, including chatter about therapies for health conditions. These therapies include but are not limited to medication, behavioral, and physical therapies. Social media subscribers who discuss such therapies often express their sentiments associated with the therapies. In this task, the focus will be to build a system that can automatically classify the sentiment associated with a therapy into one of three classes—positive, negative, and neutral. The annotated dataset for this task has been drawn from multiple preidentified Twitter cohorts (chronic pain, substance use disorder, migraine, chronic stress, long-COVID, and intimate partner violence). Thus, there is a high possibility that the therapies are being mentioned by people who are actually receiving/consuming them. The dataset consists of 5000 English Tweets containing mentions of a variety of therapies manually labeled as positive, negative, or neutral with the following approximate distribution: 20%, 14%, and 66%, respectively. The evaluation metric for this task is the micro-averaged F1-score over all 3 classes. The data include annotated collections of posts on Twitter which will be shared in csv files. There are 4 fields in the csv files: tweet_idtherapytextlabel. The training data is already prepared and will be available to the teams registering to participate. The testing data will be released when the evaluation phase starts.

  • Training data: 3009 tweets
  • Validation data: 753 tweets
  • Testing data: TBA
  • Evaluation metric: micro-averaged F1-score

Data Examples

tweet_id therapy text label
15309 meditation Did you know meditation can be one of the most rewarding important things you do in your life? Did you also know it’s impossible to not be able to meditate? For people that believe your mind must somehow go blank you’re wrong unless you’re dead. positive
15262 acupuncture abt to get acupuncture for my migraines for the first time ever & i am terrified neutral


Submission Format

Please use the format below for submission. Submissions should contain tweet_id and label separated by tabspace in the same order as below.

tweet_id label
15309 positive
15262 neutral

The unzipped submission data needs to be named as "answer.txt" and be zipped.

For more information, please refer to https://github.com/codalab/codalab-competitions/wiki/User_Building-a-Scoring-Program-for-a-Competition#directory-structure-for-submissions

Contact: Yuting Guo, Emory University, USA (yuting.guo@emory.edu)

Google Group: smm4h-2023-task-2@googlegroups.com

Codalab: https://codalab.lisn.upsaclay.fr/competitions/12421


Task 3 - Extraction of COVID-19 symptoms in Latin American Spanish tweets

Expanding on an #SMM4H 2022 task involving the classification of Spanish tweets that self-report COVID-19 symptoms, this task focuses on the detection and extraction of COVID-19 symptoms in tweets written specifically in Latin American Spanish. The task includes both personal self-reports and third-party mentions of symptoms, in an effort to generalize the identification of various disease symptoms in Latin American Spanish to both colloquial and formal language domains. The dataset consists of tweets annotated by medical doctors who are native Latin American Spanish speakers, including labels for whether or not the tweet mentions a symptom and the characters offsets of symptoms. In addition, participants will be provided with the dataset for the aforementioned #SMM4H 2022 task, along with BERT-like language models pretrained on Latin American Spanish tweets. The evaluation metric for this task is the strict F1-score for identifying the character offsets of COVID-19 symptoms. The task involves NER offset detection and classification. Participants must find the beginning and end of symptoms. Dataset annotation guidelines: Adapted annotation guideline derived from 2022’s SocialDisNER SMM4H shared task (available https://zenodo.org/record/6983041).

  • Training data: 6,021 tweets
  • Validation data: 1,979 tweets
  • Test data: 2,150 tweets
  • Evaluation metric: Strict F1-score

Submission Format

Tab-separated file with headers, same format used in the validation set.

tweet_id begin end label span
25 131 139 síntoma dolor de cabeza
26 198 201 síntoma nauseas

Contact: Juan Banda, Georgia State University, USA (jbanda@gsu.edu)

CodaLab: https://codalab.lisn.upsaclay.fr/competitions/12901

Task 4 - Binary classification of English Reddit posts self-reporting a social anxiety disorder diagnosis

Because social media is used by patients in every aspect of their daily lives, its analysis presents a promising way to understand the patient’s perspective on their disease journey, their unmet medical needs and their disease burden. Social media listening (SML) can, therefore, potentially support the progress in our understanding of a disease and influence the development of new therapies. SML, however, still has many challenges to overcome in order to do rigorous quantitative studies, one of them being the lack of confirmed diagnosis. The lack of diagnosis information, while difficult, becomes even relevant in the cases of mental disorders, where access to a diagnosis faces many barriers and patients tend to self-diagnose. Social Anxiety Disorder is a good example of this problem: the main barrier that prevents the patients from getting a diagnosis is the disease itself.

For this task we used a dataset extracted from the subreddit r/socialanxiety. The challenge is to build a classifier that correctly identifies patients that report having a positive or probable diagnosis of social anxiety disorder (positive cases labeled as ‘1’) from patients that report not having a diagnosis or the presence of a diagnosis is unlikely or unclear (negative cases labeled as ‘0’). For more details into the class annotation please refer to the annotation guidelines shared in the Google Group.

The dataset consists of 8117 posts written by users who range from 12 to 25 years old. Each row corresponds to a post and contains a unique identifier, the text and the diagnosis label.

  • Training data (75%): 6090 posts
  • Validation data (~8.4%): 680 posts
  • Test data (~16.6%): 1347 posts
  • Evaluation metric: F1-score for the positive class (i.e. posts annotated as “1”)

Table 1 provides sample training data, which includes the Post ID, Post Text, and annotated binary label (Class). Posts were annotated as “1” if the user’s reports having a positive or probable diagnosis of social anxiety disorder. Posts were annotated as “0” if the user reports not having a diagnosis or the presence of a diagnosis is unlikely or unclear.

id text class
1 Even though they know that I've struggled with and was diagnosed with SA, they act like I can just put myself out there and this will all go away. 1
2 Meh, I diagnosed myself long before I saw a therapist. 1
3 She diagnosed me with social phobia, took me off the Zoloft, and instead put me on 20mg of Paxil. 1
4 I was diagnosed with this too. I’m 19 and I’m unable to connect with anyone. I’m out in space hoping that someone will come along who I can look in the eyes and feel comfort. Im deathly afraid of any sort of physical contact with anyone around me. Im trying to work myself out of this but it just gets harder and harder. I have hope that something will someday click and I can move on. Sometimes I just think I’m beyond stupid. So you aren’t alone. 1
5 Yeah even my therapist agreed when I talked to her about this. She had dealt with SAD patients for more than 30 years, and she said in her experience men were absolutely more negatively affected by the disorder than women, especially regarding dating. 1
6 I suffer from SA, I've tried almost everything. Therapy didn't work for me, medicine did nothing. Weed doesn't even help me. Honestly the only that has been working for me, is actually leaving the house and being in public. If you do something enough you eventually get use to it. My SA is a lot more manageable and honestly doesn't really bother me anymore BUT I did develop a "side effect" to anxiety that it horrible. I sweat, a lot under my arms. Every time I leave the house and I don't even feel nervous. Just me thinking about going places makesI me sweat. Its honestly worst than SA because I could at least hide my SA but you can't hide giant sweat marks under your arms. Its so bad its like constant drops rolling down my side just after a couple minutes of driving away from my house. Now I have to find out how to deal with this problem, probably gonna have to be surgery. Anyways basically what you need to do is take your gf out of the house. Start with less busy places and try to do it everyday. My gf has been so supportive for me. She's honestly made doing things outside the house so much easier. I'm sure your gf feels the same with you. Just remember, a pill won't make your problems go away and please explain that to your gf. I had horrible side effects to that shit and a bad withdrawal. Anyways good luck 1
7 same problem. i've been to a therapist, been on meds, but stopped. now i am once again socially fucked. maybe I'll got to our councilor next week. i never liked my therapist, and I have a problem with trusting doctors, dentists, and those types. It seems i see the bad more than the good when it comes to people. i wish you all the luck. I am 19 as well. And reading this, it made me think 'did I write this??' How did that old song go? Tomorrow belongs to me.. yep maybe 1
8 I was just recently diagnosed with ADHD-PI. This is all new to me since I didn't know much about ADHD, I just knew that it was impossible for me to focus and I was constantly daydreaming. The combination of SA and ADHD sucks. Even when I really want to, I find it hard to focus on what people are saying. I get bits and pieces of what people are saying and it's frustrating and makes my SA worse. PM me about this, I'd love to talk to you about this as I don't know anyone else in a similar situation. 0
9 I'm in the same boat as you, I'm pretty sure that I have social anxiety but can't approach someone about it. But I'm planning on talking to my school counselor in September :/ 0
10 I’m a receptionist and it’s basically been forced exposure therapy. It has helped me navigate my anxiety but it’s definitely draining 0
11 Not diagnosed. I never went to a psichyatrist 0
12 self-diagnosed myself and to this day, I never been officially diagnosed with social anxiety. 0

Contact: Lucia Schmidt, Roche Pharmaceuticals, Switzerland (lucia.schmidt@roche.com)

CodaLab: https://codalab.lisn.upsaclay.fr/competitions/13160

Task 5 – Normalization of adverse drug events in English tweets

An adverse drug event (ADE), or adverse drug reaction (ADR), is harm resulting from the use of a medication. In recent years, many studies have begun mining social media for the potential of early detection and novel discovery of ADEs. This task focuses on normalizing ADEs in tweets to their standard concept IDs in the MedDRA vocabulary. 

To enable novel approaches, in contrast to previous iterations of this task, the evaluation metric will no longer require extracting the text span of the ADE. In addition to evaluating systems' performance for all MedDRA IDs in the test set, this year, a second evaluation metric will be based on a zero-shot learning setup, evaluating systems' performance specifically for MedDRA IDs in the test set that were not seen during training. 

Participants will be provided with about 18,000 labeled tweets for training and about 10,000 tweets for testing. The training data include the Tweet ID, the text of the Tweet Object, the annotated binary label for whether or not the tweets contains an ADE, the character offsets of the ADE, the text span of the ADE, and the MedDRA ID—for example: 

tweet_id text label begin end span meddra_id
SMM4H2022uCZV2SRsCe4vzjFm have to go to a doc now to see why i'm still gaining. stupid paxil made me gain like 50 pounds ?? and now i have to lose it ADE 61 68 gaining 10047896
SMM4H2022uCZV2SRsCe4vzjFm have to go to a doc now to see why i'm still gaining. stupid paxil made me gain like 50 pounds ?? and now i have to lose it ADE 91 110 gain like 50 pounds 10047896

Contact: Dongfang Xu, Cedars-Sinai Medical Center, USA (Dongfang.Xu@cshs.org)

CodaLab: https://codalab.lisn.upsaclay.fr/competitions/12941

Workshop Program 
 
Saturday, November 11, 2023
8:30 AM – 8:45 AM Introduction Graciela Gonzalez-Hernandez
8:45 AM – 10:00 AM System Demonstrations Ivan Flores
8:45 AM – 9:00 AM Overview of Shared Task 1: Binary Classification of English Tweets Self-Reporting a COVID-19 Diagnosis Ari Z. Klein
9:00 AM – 9:15 AM Shayona at #SMM4H 2023: COVID-19 Self Diagnosis Classification Using BERT and LightGBM Models Rushi Chavda, Vraj Patel, Darshan Makwana and Anupam Shukla
9:15 AM – 9:30 AM Overview of Shared Task 2: Multi-Class Classification of Sentiment Associated with Therapies in English Tweets Yuting Guo
9:30 AM – 9:45 AM UQ at #SMM4H 2023: Balanced and Explainable Social Media Analysis for Public Health with Large Language Models Yan Jiang, Ruihong Qiu, Yi Zhang and Zi Huang
9:45 AM – 10:00 AM Sentiment Analysis of COVID-19 Survey Data: A Comparison of ChatGPT and Fine-tuned OPT Against Widely Used Sentiment Analysis Tools Juan Antonio Lossio-Ventura and Francisco Pereira
10:00 AM – 10:30 AM Coffee Break
10:30 AM – 12:00 PM System Demonstrations Ivan Flores
10:30 AM – 11:30 AM Keynote Presentation from Academia Luis Rocha
11:30 AM – 11:45 AM Overview of Shared Task 3: Extraction of COVID-19 Symptoms in Latin American Spanish Tweets Juan Banda
    11:45 AM – 12:00 PM   Explorers at #SMM4H 2023: Enhancing BERT for Health Applications through Knowledge and Model Fusion Xutong Yue, Xilai Wang, Yuxin He and Zhenkun Zhou
12:00 PM – 1:15 PM Lunch Break
1:15 PM – 2:30 PM System Demonstrations Ivan Flores
1:15 PM – 1:30 PM Overview of Shared Task 4: Binary Classification of English Reddit Posts Self-Reporting a Social Anxiety Disorder Diagnosis
1:30 PM – 2:30 PM Keynote Presentation from Industry Raul Rodriguez-Esteban
2:30 PM – 3:00 PM Coffee Break
3:00 PM – 4:30 PM System Demonstrations Ivan Flores
3:00 PM – 3:15 PM Overview of Shared Task 5: Normalization of Adverse Drug Events in English Tweets --Dongfang Xu
3:15 PM – 3:30 PM DS4DH at #SMM4H 2023: Zero-Shot Adverse Drug Events Normalization using Sentence Transformers and Reciprocal-Rank Fusion
3:30 PM – 3:45 PM Shared Task Flash Talks
3:45 PM – 4:30 PM Panel on the Future of Social Media Data for Health Research Graciela Gonzalez-Hernandez, Luis Rocha and Raul Rodgriguez-Esteban