Exercise#2 Few Shot
Objective
Get hands on experience with iterative design of few shot prompts. Keep in mind that this task can be achieved easily with a fine tuned classifier model.
Task :
- Named Entity Recognition
- List entities with labels [person, city, state, country, animal, airlines]
Output format (JSON)
[
{'entity': 'person-name', 'type': 'person'},
{'entity': 'animal-name', 'type': 'animal'},
{'entity': 'airlines-name', 'type': 'airlines'},
{'entity': 'country-name', 'type': 'country'},
{'entity': 'state-name', 'type': 'state'},
{'entity': 'city-name', 'type': 'city'}
]
Try out Cohere Command-Light model for NER
Note You will be carrying out this exercise in the Cohere playground. Before proceeding register for free access to Cohere API.
In this exercise we would like Cohere Command-Light model to identify entities in a given text string. Although the model generates a JSON, we would want it to generate a JSON in a format defined by us. Use the prompt below to incrementally get the desired outcome. Feel free to experiment !!!
Open the Cohere playground. Link to Cohere Playground
-
Select the model : command-light
-
Change randomness to low value (0.1)
Prompt #1 Zero Shot
Return a list of named entities and their types in the text.
Format response as JSON shown in answer
Text: john travelled with his dog duffy from new jersey to Turkey in a United flight. He met with his friend abdul.
Answer:
Response
- New Jersey missed out as a state
- Turkey missed out as a country
- United missed out as a airline
- Flight added incorrectly as a place
{
"entities":[
{
"name": "John",
"type": "Person"
},
{
"name": "Duffy",
"type": "Animal"
},
{
"name": "Abdul",
"type": "Person"
},
{
"name": "Flight",
"type": "Place"
}
]
}
Prompt #2 One Shot
- Add 1 example to address the location label (to country & city)
Return a list of named entities and their types in the text.
Format response as JSON shown in answer
Text: Anil lives in Paris, the most beautiful city in France
Answer: [{'entity': 'Anil', 'type': 'person'}, {'entity': 'Paris', 'type' : 'city'}, {'entity': 'France', 'type' : 'country'}]
Text: john travelled with his dog duffy from new jersey to Turkey in a United flight. He met with his friend abdul.
Answer:
Output
-
Removed Flight as place
-
New Jersey added as a place - We would like this to be of type state
-
Duffy marked as thing instead of animal
{
"entities": [
{"entity": "John", "type": "person"},
{"entity": "New Jersey", "type": "location"},
{"entity": "Turkey", "type": "location"},
{"entity": "Abdul", "type": "person"},
{"entity": "Duffy", "type": "thing"}
]
}
Prompt #3 Few Shots
Return a list of named entities and their types in the text.
Format response as JSON shown in answer
Text: Anil lives in Paris, the most beautiful city in France
Answer: [{'entity': 'Anil', 'type': 'person'}, {'entity': 'Paris', 'type' : 'city'}, {'entity': 'France', 'type' : 'country'}]
Text: Mary likes to fly Lufthansa
Answer:[{'entity': 'Mary', 'type': 'person'}, {'entity': 'Lufthansa', 'type': 'airlines'}]
Text: john travelled with his dog duffy from new jersey to Turkey in a United flight. He met with his friend abdul.
Answer:
Output
This is the desired output.
[
{'entity': 'John', 'type': 'person'},
{'entity': 'Duffy', 'type': 'animal'},
{'entity': 'United', 'type': 'airlines'},
{'entity': 'Abdul', 'type': 'person'},
{'entity': 'New Jersey', 'type': 'state'}
]
Iterate to test & improve the performance
In real world you would test this with additional prompts to ensure that output is still good. In case additional prompts lead to issues, introduce additional shots/examples. Continue to do it till you get the desired results consistently.
Try out the following 2 prompts one after anothe. You will see some issues, address them by way of instructions and/or examples.
Pinky loves to take her cat Daisy for playdates to her friend Jenny's place.
Jack and Jill went up the hill to fetch a pail of water. This hill is next to the Winchester township in London, UK
Ryan took Ryan Air to visit his aunt in Belgium