Skip to content

Commit b99bfc2

Browse files
Urmishnajoungkim
Urmish
and
najoungkim
authored
Prompts for adjunct_island subset of blimp dataset (#736)
* First 2 prompts for blimp * Fixed some template issues that led to conflicting details * Updated templates to avoid packing errors * Added 2 new prompts with a different style of asking question * Add Najoung\'s templates * Added prompts for complex_NP_island * Updated the templates after the requested fixes * Added templates for complex_NP_island via copy scripts * Updated templates for all subsets * Switched to randomized prompts. * Minor changes to prompt names * Fix answer_choices for some templates + better template names * Fixed extra spaces. * Added null prompts (true null and single quotation mark versions) * Minor cleanup * Prompt cleanups: true null prompt, choice randomization + dropping order swapped prompts. Co-authored-by: cookie <nk> Co-authored-by: najoungkim <[email protected]>
1 parent 5dea218 commit b99bfc2

File tree

67 files changed

+17554
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

67 files changed

+17554
-0
lines changed

‎promptsource/templates/blimp/adjunct_island/templates.yaml

Lines changed: 262 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,262 @@
1+
dataset: blimp
2+
subset: adjunct_island
3+
templates:
4+
6e71cd4b-882c-4458-9cd4-fd83f849ad09: !Template
5+
answer_choices: 1 ||| 2
6+
id: 6e71cd4b-882c-4458-9cd4-fd83f849ad09
7+
jinja: '{% set shuffled_order = [0, 1] | random %}
8+
9+
{% set shuffled_order_options = [0, 1] | random %}
10+
11+
Between the two sentences, which one is grammatical? Please answer {% if shuffled_order_options
12+
== 0 %}1 or 2{% else %}2 or 1{% endif %}.
13+
14+
{% if shuffled_order == 0 %}
15+
16+
1: {{ sentence_good }}
17+
18+
2: {{ sentence_bad }}
19+
20+
{% else %}
21+
22+
1: {{ sentence_bad }}
23+
24+
2: {{ sentence_good }}
25+
26+
{% endif %}
27+
28+
|||
29+
30+
{% if shuffled_order == 0 %}
31+
32+
{{ answer_choices[0] }}
33+
34+
{% else %}
35+
36+
{{ answer_choices[1] }}
37+
38+
{% endif %}'
39+
metadata: !TemplateMetadata
40+
choices_in_prompt: true
41+
metrics:
42+
- Accuracy
43+
original_task: true
44+
name: grammatical_between_1_2
45+
reference: ''
46+
7ed733fd-524c-4f00-800f-77a7a862ab4a: !Template
47+
answer_choices: 1 ||| 2
48+
id: 7ed733fd-524c-4f00-800f-77a7a862ab4a
49+
jinja: '{% set shuffled_order = [0, 1] | random %}
50+
51+
{% set shuffled_order_options = [0, 1] | random %}
52+
53+
Which one of the following sentences is grammatical? Please answer {% if shuffled_order_options
54+
== 0 %}1 or 2{% else %}2 or 1{% endif %}.
55+
56+
{% if shuffled_order == 0 %}
57+
58+
1: {{ sentence_good }}
59+
60+
2: {{ sentence_bad }}
61+
62+
{% else %}
63+
64+
1: {{ sentence_bad }}
65+
66+
2: {{ sentence_good }}
67+
68+
{% endif %}
69+
70+
|||
71+
72+
{% if shuffled_order == 0 %}
73+
74+
{{ answer_choices[0] }}
75+
76+
{% else %}
77+
78+
{{ answer_choices[1] }}
79+
80+
{% endif %}'
81+
metadata: !TemplateMetadata
82+
choices_in_prompt: true
83+
metrics:
84+
- Accuracy
85+
original_task: true
86+
name: grammatical_which_one_1_2
87+
reference: ''
88+
8f354d74-5ffb-4cbc-b119-b79f2db00df3: !Template
89+
answer_choices: Yes ||| No
90+
id: 8f354d74-5ffb-4cbc-b119-b79f2db00df3
91+
jinja: '{% set shuffled_order = [0, 1] | random %}
92+
93+
{% set shuffled_order_options = [0, 1] | random %}
94+
95+
"{% if shuffled_order == 0 %}{{ sentence_good }}{% else %}{{ sentence_bad }}{%
96+
endif %}"
97+
98+
Is this sentence a bad sentence of English, {% if shuffled_order_options ==
99+
0 %}Yes or No{% else %}No or Yes{% endif %}? ||| {% if shuffled_order == 0 %}
100+
{{ answer_choices[1] }} {% else %} {{ answer_choices[0] }} {% endif %}'
101+
metadata: !TemplateMetadata
102+
choices_in_prompt: true
103+
metrics:
104+
- Accuracy
105+
original_task: false
106+
name: single_sentence_bad_yes_no
107+
reference: ''
108+
967ca4e2-162f-4365-a1f5-2c87c0e874ce: !Template
109+
answer_choices: A ||| B
110+
id: 967ca4e2-162f-4365-a1f5-2c87c0e874ce
111+
jinja: '{% set shuffled_order = [0, 1] | random %}
112+
113+
{% set shuffled_order_options = [0, 1] | random %}
114+
115+
Between the two sentences, which one is grammatical? Please answer {% if shuffled_order_options
116+
== 0 %}A or B{% else %}B or A{% endif %}.
117+
118+
{% if shuffled_order == 0 %}
119+
120+
A: {{ sentence_good }}
121+
122+
B: {{ sentence_bad }}
123+
124+
{% else %}
125+
126+
A: {{ sentence_bad }}
127+
128+
B: {{ sentence_good }}
129+
130+
{% endif %}
131+
132+
|||
133+
134+
{% if shuffled_order == 0 %}
135+
136+
{{ answer_choices[0] }}
137+
138+
{% else %}
139+
140+
{{ answer_choices[1] }}
141+
142+
{% endif %}'
143+
metadata: !TemplateMetadata
144+
choices_in_prompt: true
145+
metrics:
146+
- Accuracy
147+
original_task: true
148+
name: grammatical_between_A_B
149+
reference: ''
150+
a1997343-0a20-4f8c-8939-a512f0f68485: !Template
151+
answer_choices: Yes ||| No
152+
id: a1997343-0a20-4f8c-8939-a512f0f68485
153+
jinja: '{% set shuffled_order = [0, 1] | random %}
154+
155+
{% set shuffled_order_options = [0, 1] | random %}
156+
157+
"{% if shuffled_order == 0 %}{{ sentence_good }}{% else %}{{ sentence_bad }}{%
158+
endif %}"
159+
160+
Is this sentence a good sentence of English, {% if shuffled_order_options ==
161+
0 %}Yes or No{% else %}No or Yes{% endif %}? ||| {% if shuffled_order == 0 %}
162+
{{ answer_choices[0] }} {% else %} {{ answer_choices[1] }} {% endif %}'
163+
metadata: !TemplateMetadata
164+
choices_in_prompt: true
165+
metrics:
166+
- Accuracy
167+
original_task: false
168+
name: single_sentence_good_yes_no
169+
reference: ''
170+
c45d5bac-968e-4e39-96b7-d18a5ddb225c: !Template
171+
answer_choices: A ||| B
172+
id: c45d5bac-968e-4e39-96b7-d18a5ddb225c
173+
jinja: '{% set shuffled_order = [0, 1] | random %}
174+
175+
{% set shuffled_order_options = [0, 1] | random %}
176+
177+
Read the following two sentences.
178+
179+
{% if shuffled_order == 0 %}
180+
181+
Sentence A: {{ sentence_good }}
182+
183+
Sentence B: {{ sentence_bad }}
184+
185+
{% else %}
186+
187+
Sentence A: {{ sentence_bad }}
188+
189+
Sentence B: {{ sentence_good }}
190+
191+
{% endif %}
192+
193+
Which one is a better sentence of English, {% if shuffled_order_options == 0
194+
%}A or B{% else %}B or A{% endif %}? ||| {% if shuffled_order == 0 %}
195+
196+
{{ answer_choices[0] }}
197+
198+
{% else %}
199+
200+
{{ answer_choices[1] }}
201+
202+
{% endif %}'
203+
metadata: !TemplateMetadata
204+
choices_in_prompt: true
205+
metrics:
206+
- Accuracy
207+
original_task: true
208+
name: better_A_B
209+
reference: ''
210+
d11a37d7-7c42-4ee0-96c5-ffd24ad7073e: !Template
211+
answer_choices: A ||| B
212+
id: d11a37d7-7c42-4ee0-96c5-ffd24ad7073e
213+
jinja: '{% set shuffled_order = [0, 1] | random %}
214+
215+
{% set shuffled_order_options = [0, 1] | random %}
216+
217+
Which one of the following sentences is grammatical? Please answer {% if shuffled_order_options
218+
== 0 %}A or B{% else %}B or A{% endif %}.
219+
220+
{% if shuffled_order == 0 %}
221+
222+
A: {{ sentence_good }}
223+
224+
B: {{ sentence_bad }}
225+
226+
{% else %}
227+
228+
A: {{ sentence_bad }}
229+
230+
B: {{ sentence_good }}
231+
232+
{% endif %}
233+
234+
|||
235+
236+
{% if shuffled_order == 0 %}
237+
238+
{{ answer_choices[0] }}
239+
240+
{% else %}
241+
242+
{{ answer_choices[1] }}
243+
244+
{% endif %}'
245+
metadata: !TemplateMetadata
246+
choices_in_prompt: true
247+
metrics:
248+
- Accuracy
249+
original_task: true
250+
name: grammatical_which_one_A_B
251+
reference: ''
252+
f6cfe60f-a0c5-4ab8-ae7d-1c08d3cdaa2b: !Template
253+
answer_choices: '{{ sentence_good }} ||| {{sentence_bad}}'
254+
id: f6cfe60f-a0c5-4ab8-ae7d-1c08d3cdaa2b
255+
jinja: ' ||| {{ sentence_good }}'
256+
metadata: !TemplateMetadata
257+
choices_in_prompt: false
258+
metrics:
259+
- Accuracy
260+
original_task: true
261+
name: null_prompt
262+
reference: ''

0 commit comments

Comments
 (0)