About

Our motivation for this demo is twofold:

1. Create a tool to turn a basic question into quality prompts for an LLM
2. Explore the LLMs ability to perform pairwise comparison on two answers to the same question

Generating prompts

LLM prompts for simple tasks often take a familiar format:

1. Start with a high level instruction on the role and tone we want the AI to adopt
2. Include one or more examples of relevant Q&A between the human and the AI (with the speakers named explicitly e.g. Human and Assistant)
3. Ending with Assistant: and optionally some prefilling of the response

Let's try generating a prompt to elicit a relevant and accurate answer to a arbitrary question. In fact we'll generate two versions. They'll be generated by the LLM of course, using a seed prompt.

Making different prompts

We don't want to create the same prompt twice. To introduce variation between the prompts we have two options:

1. Different seed prompts

E.g. one seed prompt can ask the LLM to generate a verbose, formal prompt and another seed prompt could ask the LLM to generate a concise, informal prompt.

2. Same seed prompt with n > 1 and temperature >= 0.9

This way we use one seed prompt but tell the model to generate n possible versions. With the model temperature high enough, we expect the versions to be sufficiently different from one another.