A short description of the PromptSource tool as summarised in "PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts", published by Stephen Bach (and many others) under the BigScience project as an ACL Demo in 2022.

Timestamps:
00:00 - PromptSource - An IDE and Repo for Natural Language Prompts
02:36 - Prompt creation is not traditional NLP annotation
03:37 - The PromptSource workflow
05:21 - PromptSource plays nicely with 🤗 Datasets
06:17 - Prompt Template Engine: Jinja2
07:56 - User Interface: Dataset Browsing
08:22 - User Interface: Sourcing
09:03 - User Interface: Helicopter view
09:17 - Community Guidelines
10:41 - Case Studies with PromptSource
11:40 - Prior work

Detailed description:
PromptSource is a tool (and collection of prompts) that was developed as part of the BigScience project. The motivation for the tool is that language model prompting is growing in importance in NLP, but few tools exist to help users create, refine and share prompts.

The "Public Pool of Prompts" (P3) gathered with PromptSource (as of September 2022) includes spans more than 2000 prompts spanning 180 datasets.

It's useful to note that creating prompts is quite different to traditional NLP annotation in several ways: prompts are functions (not labels), they apply to datasets (not examples) and variation between prompts is desirable (rather than a nuisance).

PromptSource proposes a simple workflow to meet these challenges. It also works well with 🤗 Datasets so it can be applied to a wide range of existing datasets.

To support flexible prompt editing, the Jinja2 template engine is used. This is a bit more flexible than a rule-based approach but easier to analyse than pure Python code.

PromptSource contributions are governed by community guidelines which encourage the authorship of effective prompts and mandate the inclusion of appropriate metadata with each prompt query.

P3 has already seen usage in other projects such as the massively multitask prompted training employed in T0, multilingual prompting with XGLM and training on in-context learning examples with MetaICL.

Note that while existing NLP annotation tools exist, prompts differ substantially from existing types of NLP annotations, motivating the need for PromptSource.

Topics: #prompting #nlp #bigscience

Slides (pdf): [ Ссылка ]

The PromptSource paper can be found at [ Ссылка ] and the library can be found at [ Ссылка ]

References for papers mentioned in the video can be found at
[ Ссылка ]

свернуть