TREC Knowledge Base Acceleration

Supporters:

TREC KBA 2014: Streaming Slot Filling

Knowledge Base Acceleration (KBA) is an open evaluation in NIST's Text Retrieval Conference (TREC). KBA addresses this fundamental question:

Given a rich dossier on a subject,
filter a stream of documents to
accelerate users filling in knowledge gaps.

Basic Rules

SSF builds on CCR by requiring systems to complete a basic attribute information on each entity's profile in the KB. In 2014, we have simplified this task by having the assessors create basic profiles for every entity while performing the vital filtering task. For each slot on each entity, assessors will assemble a list of valid string values from the entire time frame.

Systems must identify candidate slot fills as the stream progresses.

The slot equivalence class concept from SSF 2013 has been deprecated and is not used in 2014.

SSF runs use the same run submission format as CCR with two additional fields to specify the slot name and the byte offset of the slot filling passage in the clean_visible text. Generally, slot fills are short phrases, not long paragraphs.

The run submission format requires: task_id: kba-ssf-2014

Metrics

To score an SSF run, we plan to:

  1. Treat each (entity, slot_name) pair as a query
  2. Take all of run's responses (slot fills) from the entire time range
  3. De-duplicate responses keeping the highest confidence score for each string
  4. Sort the list of fill strings for each query by confidence
  5. Compute Mean Average Precision (MAP) on these ranked lists
  6. Possibly other rank-based measures.

This procedures balances two objectives: 1) requiring systems to operate in a streaming fashion, and 2) enabling a clear cut evaluation protocol.

Slot filling is a rich and complex task, and is explored by other open evaluations, e.g. TAC KBP. For KBA SSF, we focus on generating candidate slot fills to present to a human trying to rapidly improve a KB profile. This simplifies evaluation in several ways. By pooling all of the slot fills from the entire time frame and scoring them against the entire list of ground truth slot fills, the metric ignores changing slot values. While we are interested in a system's ability to model evolving slot values, we have elected to focus on the more basic task of finding good slot fills within the hour-by-hour restriction of the KBA task. This means that the task is to find strings that were ever a good slot fill for that (entity-slot_name) pair. In the future, we may introduce revised versions of this task.

Target slots

(This list will be changed/updated during assessing in June 2014):