SemEval-2022 Task 7

Identifying Plausible Clarifications of
Implicit and Underspecified Phrases in Instructional Texts



Update 10 January 2022: The test data for the evaluation phase is now available below.

The goal of this shared task is to evaluate the ability of NLP systems to distinguish between plausible and implausible clarifications of an instruction. Such clarifications can be critical to ensure that instructions describe clearly enough what steps must be followed to achieve a specific goal. We set up this task as a cloze task, in which clarifications are presented as possible fillers and systems have to score how well each filler plausibly fits in a given context.

Cloze tasks have become a standard framework for evaluating various discourse-level phenomena in NLP. Some prominent examples include the narrative cloze test (Chambers and Jurafksy, 2008), the story cloze test (Mostafazadeh et al., 2016), and the LAMBADA word prediction task (Paperno et al., 2016). In these tasks, NLP systems are required to make a prediction about the filler of a cloze that is most likely to continue the discourse. However, it is not always clear whether exactly one likely filler exists or how plausible different fillers would be.

This task revolves around judging the plausibility of human-inserted and machine-generated fillers in naturally occurring contexts. Specifically, the contexts are instructional texts on everyday scenarios in which clarifications may have been necessary to eliminate possible misunderstandings. Clarifications were identified using revision histories in which it is possible to observe disambiguations of various semantic and pragmatic phenomena, including implicit, underspecified, and metonymic references, as well as implicit discourse relations and implicit quantifying modifiers.

There is no formal registration for the task yet. Anyone interested can join our Google group here.

Data and Examples

The basis of our task is wikiHowToImprove (Anthonio et al., 2020), a collection of revisions of instructional texts from the how-to website wikiHow. For this task, we extract revisions that we believe are likely to represent specific instances of clarifications. As such, each revision in this task represents an option to clarify between possibly multiple meanings. To assess the plausibility of different clarification options, we automatically generate alternatives and ask annotators to rate for each clarification option whether it "makes sense in the given how-to guide" (on a scale from 1 to 5).

The data created with the described approach is now available at the following link. Examples from the trial data are shown in the panels below.

For a given how-to guide (panel), systems participating in the task have to predict the plausibility (1-5) of each filler (listed in each panel's footer).


How to Keep a Band Together

Steps
(...)
4. If a member of the band does bring up something that he or she feels is a problem, do your best to fix it. Whether it's that the bassist wants a longer solo or the keyboardist doesn't like the style of music you're playing, try all you can to work it out.
5. If you can't fix a problem completely, come to a compromise so that everybody is at least somewhat happy with         .

  • the situation (5.0)

  • the outcome (4.5)

  • the music (4.0)

  • your efforts (4.0)

  • someone else (1.0)

How to Lighten Birthmarks Naturally

Lightening Your Birthmark
(...)
2. Rub lemon juice on your birthmark
(...)
* Wash thoroughly with warm water. Pat          dry with a clean towel.
* Repeat this process three times a day.

  • your birthmark (5.0)

  • the area (4.5)

  • the lemon (2.5)

  • lemon juice (2.0)

  • your stomach (1.0)

How to Store Jalapenos

Warnings
* Make sure to wear latex gloves when handling jalapenos or wash          thoroughly after handling. You can get a chemical burn if you don't protect yourself.

  • your hands (5.0)

  • the jalapenos (3.0)

  • your body (1.5)

  • the floor (1.0)

  • your underwear (1.0)

How to Build Deck Stairs

Finishing the Deck Stairs
1. Screw each stringer to          the deck frame with a drill. Use L-brackets and deck screws to attach the stringers to the deck.

  • the top of (5.0)

  • the base of (4.5)

  • the bottom of (4.5)

  • the inside of (3.5)

  • each side of (3.0)

How to Calibrate Your Sprinklers

Steps
(...)
4. Measure the water collected in the cans. Once you turn the water off, collect the cans and pour all the water into one container. The container must be exactly the same          as the containers you used to collect the water. Use a ruler to measure how many inches of water are in the can.

  • diameter (4.5)

  • size (4.5)

  • width (4.0)

  • height (3.5)

  • color (1.0)

How to Ask Someone to Be Your Groomsman

Deciding What to Say or Write
1. Ask in person whenever possible. Receiving an invitation to be a groomsman is exciting.
(...)
2. Send a          card. Even if you’re able to have the conversation in person, it's worth sending a card.

  • wedding (4.5)

  • handwritten (4.5)

  • reminder (2.5)

  • business (1.0)

  • birthday (1.0)

Important Dates

Organizers