llama.cpp
llama.cpp copied to clipboard
Published
20 hours ago •
ggml-org
ggml-org
llama.cpp + Final Jeopardy
I was browsing reddit and saw this post:
https://www.reddit.com/r/LocalLLaMA/comments/12xkm9v/alpaca_vs_final_jeopardy/
If anyone is interested, it would be great to add such evaluation as an example to llama.cpp and add instructions for running it with different models: LLaMA, Alpaca, Vicuna, etc. and different quantizations.
Here is the original work by @aigoopy which can be a good starting point:
https://github.com/aigoopy/llm-jeopardy
Going to attempt this, should it be in a separate folder (ex. examples/jeopardy) or should it be thrown directly in examples?
In examples/jeopardy, with a README with instructions, question files, eval scripts, plot scripts, etc. would be nice
I extracted the data from the spreadsheet. I don't know if it is necessary to reverse the Jeopardy "answer in the form of a question". I think LLMs should be able to emulate it.
JSON
[
{
"category": "The Oscars",
"answer": "Born in 1932 & the son of a percussionist in the CBS radio orchestra, he's been nominated for 53 Oscars",
"question": "Who is John Williams?"
},
{
"category": "English Literature",
"answer": "It says, \"The mind is its own place, & in itself can make a heaven of hell, a hell of heaven. What matter where, if I be still the same\"",
"question": "What is Paradise Lost?"
},
{
"category": "Writers' Lesser-Known Works",
"answer": "Known for more philosophical works, he wrote the play \"La Mandragola\", in which Florentines are rewarded for immoral actions",
"question": "Who is Niccolò Machiavelli?"
},
{
"category": "Exploration",
"answer": "James Cook's account of a 1774 visit here records an object \"near 27 feet long, and upwards of 8 feet over the breast or shoulders\"",
"question": "What is Easter Island (Rapa Nui)?"
},
{
"category": "The Bill of Rights",
"answer": "England's \"Bloody Assizes\" & a 1685 life sentence for perjury were 2 main origins of this amendment to the U.S. Constitution",
"question": "What is the Eighth Amendment?"
},
{
"category": "Nobel Peace Prize Winners",
"answer": "At times they each lived on Vilakazi St. in Soweto, so it claims to be the world's only street home to 2 Nobel Peace Prize winners",
"question": "Who are Nelson Mandela & Desmond Tutu?"
},
{
"category": "Famous Names",
"answer": "In 1966, the year of his death, he shared plans for an experimental prototype community in Florida",
"question": "Who is Walt Disney?"
},
{
"category": "Geography",
"answer": "Of the 13 nations through which the Equator passes, it's the only one whose coastline borders the Caribbean Sea",
"question": "What is Colombia?"
},
{
"category": "Fashion History",
"answer": "These decorative items get their name from their origin in the port city of Strasbourg, on the border of France & Germany",
"question": "What are rhinestones?"
},
{
"category": "Movies of the '80s",
"answer": "Based on an off-Broadway play with just 3 characters, it won the Best Picture Oscar & the actors in all 3 roles were nominated",
"question": "What is Driving Miss Daisy?"
},
{
"category": "Novelists",
"answer": "A 2012 book review noted subjects that \"sparked his ire\": capital punishment, big tobacco & \"the plight of the unjustly convicted\"",
"question": "Who is John Grisham?"
},
{
"category": "20th Century Eponyms",
"answer": "A 1940 headline about this included \"failure\", \"liability when it came to offense\" & \"stout hearts no match for tanks\"",
"question": "What is the Maginot Line?"
},
{
"category": "City History",
"answer": "Over 700 years after its traditional 1252 founding date, this port city became associated with a psychological response",
"question": "What is Stockholm?"
},
{
"category": "Brand Names",
"answer": "The success of this brand has its roots with a hydrotherapy pump its cofounder created for his son, who had arthritis",
"question": "What is Jacuzzi?"
},
{
"category": "American Authors",
"answer": "In a periodical in 1807, he called New York City \"Gotham, Gotham! Most enlightened of cities\"",
"question": "Who is Washington Irving?"
},
{
"category": "Symbols",
"answer": "In math, it's a rotated V; in society, it's a feeling of some marginalized or underrepresented people",
"question": "What is \"less than\"?"
},
{
"category": "Movie Theme Songs",
"answer": "Monty Norman, the composer of this character's theme, said the staccato riff conveyed sexiness, mystery & ruthlessness",
"question": "Who is James Bond?"
},
{
"category": "American Novelists",
"answer": "He served with an airman named Yohannan in World War II & despite what readers might think, he said he enjoyed his service",
"question": "Who is Joseph Heller?"
},
{
"category": "Medieval Places",
"answer": "One of the participants in an 1170 event at this place said, \"Let us away, knights; he will rise no more\"",
"question": "What is Canterbury, England? (Canterbury Cathedral)"
},
{
"category": "Countries of Africa",
"answer": "At one time a province of the Roman Empire, this kingdom is known to Arabic scholars as Al-Maghrib Al-Aqsa, \"the far west\"",
"question": "What is Morocco?"
},
{
"category": "Statehood",
"answer": "Congress relented in 1890 after this prospective state said it would wait 100 years rather than come in without the women",
"question": "What is Wyoming?"
},
{
"category": "1980s Movies",
"answer": "A writer & producer of this movie said he wanted it to be like a Western or James Bond film, \"only it takes place in the '30s\"",
"question": "What is Raiders of the Lost Ark?"
},
{
"category": "Art Exhibitions",
"answer": "In 1898 what's been called the first blockbuster art show was devoted to him & put on for Queen Wilhelmina's coronation",
"question": "Who is Rembrandt?"
},
{
"category": "Countries of the World",
"answer": "Part of the largest contiguous land empire during the 1200s & 1300s, today it's the world's second-largest landlocked country",
"question": "What is Mongolia?"
},
{
"category": "Literature",
"answer": "A 2006 book was titled \"The Poem That Changed America:\" this \"Fifty Years Later\"",
"question": "What is \"Howl\"?"
},
{
"category": "Invasions",
"answer": "Backed by 14,000 troops, he invaded England to restore, in his words, its \"religion, laws, and liberties\"",
"question": "Who is William of Orange?"
},
{
"category": "Landmarks",
"answer": "After its completion in the late 19th c., it was called \"a truly tragic street lamp\" & a \"high & skinny pyramid of iron ladders\"",
"question": "What is the Eiffel Tower?"
},
{
"category": "Geographic Name's the Same",
"answer": "The busiest passenger port in the U.K., it shares its name with a capital of one of the original 13 states",
"question": "What is Dover?"
},
{
"category": "Names in the Bookstore",
"answer": "This man made lists, perhaps to cope with depression; a set of lists he published in 1852 made his name synonymous with a type of book",
"question": "Who is Peter Mark Roget?"
},
{
"category": "U.S. History",
"answer": "An 1869 presidential pardon was granted to this man, due in part to a plea by the Medical Society of Harford County, Maryland",
"question": "Who is Dr. Samuel Mudd?"
},
{
"category": "American Literature",
"answer": "Letters, pocket knives, C rations & steel helmets are among the tangible items referred to in the title of this modern war classic",
"question": "What is The Things They Carried?"
},
{
"category": "Nonfiction",
"answer": "It has the line, \"The discovery of America…opened up fresh ground for the rising bourgeoisie\"",
"question": "What is The Communist Manifesto"
},
{
"category": "Laws in U.S. History",
"answer": "A radical Republican championed this 1875 act but the Supreme Court struck it down in 1883; a new version was passed 81 years later",
"question": "What is the Civil Rights Act?"
},
{
"category": "Names of Myth",
"answer": "Her brothers, Castor & Pollux, saved her after Theseus stole her away as a kid; a larger force would seek her later in life",
"question": "Who is Helen of Troy?"
},
{
"category": "African Countries",
"answer": "Once Africa's largest country in area, it dropped to third in 2011 when a portion of it declared independence",
"question": "What is Sudan?"
},
{
"category": "The Ancient World",
"answer": "The ancient writer Galen said books on ships arriving to this city's port were seized, originals kept & copies returned",
"question": "What is Alexandria?"
},
{
"category": "Famous Names",
"answer": "For a special 1970s cookbook, he provided one simple recipe -- a can of Campbell's tomato soup & 2 cans of milk",
"question": "Who is Andy Warhol?"
},
{
"category": "People & Places",
"answer": "Thought to descend from people of Southeast Asia, the Chamorro make up this U.S. territory's largest ethnic group",
"question": "What is Guam?"
},
{
"category": "Current World Leaders",
"answer": "In office from 2022, the president of this country has taken so many foreign trips a play on his name is \"Ferdinand Magellan Jr.\"",
"question": "What is the Philippines?"
},
{
"category": "Writers & The South",
"answer": "In 1939 he lived on Toulouse Street in the French Quarter & chose the professional name that bonded him to the South",
"question": "Who is Tennessee Williams?"
},
{
"category": "National Parks",
"answer": "It's named for a river indigenous people called Mi tse a-da-zi, translated by French-speaking trappers as \"Pierre Jaune\"",
"question": "What is Yellowstone?"
},
{
"category": "Sports",
"answer": "In 2010 they introduced the 4-point shot, 35 feet from the basket",
"question": "Who are the Harlem Globetrotters?"
},
{
"category": "The U.S. Military",
"answer": "Losses over Asia in the 1960s led to the establishment of the program known as this at a San Diego naval base in 1969",
"question": "What is \"Top Gun\"?"
},
{
"category": "Art & Science",
"answer": "A craft that visited it was named for Giotto, based on the story that 680 years earlier, the painter depicted it as the Star of Bethlehem",
"question": "What is Halley's Comet?"
},
{
"category": "Words From World War I",
"answer": "\"Cistern\" & \"reservoir\" were suggested names for a secret invention, but the British preferred this less clumsy monosyllable",
"question": "What is \"tank\"?"
},
{
"category": "European History",
"answer": "Until 1806, some German nobles included among their honors the title of \"Elector\" for their role in selecting this personage",
"question": "What is Holy Roman Emperor?"
},
{
"category": "Theater History",
"answer": "In 1904, wearing a harness, actress Nina Boucicault became the first to play this character onstage",
"question": "Who is Peter Pan?"
},
{
"category": "European Cities",
"answer": "Alphabetically the first German city in encyclopedias, it was also the first one taken by the Allies in World War II",
"question": "What is Aachen?"
},
{
"category": "Word Origins",
"answer": "This Sanskrit word referring to a spoken word or phrase comes from a word for \"to think\"",
"question": "What is mantra?"
},
{
"category": "Inventions",
"answer": "1917's \"Elements of Trench Warfare\" said this Old West item was \"difficult to destroy\" & \"difficult to get through\"",
"question": "What is barbed wire?"
},
{
"category": "World War II",
"answer": "Mimi Reinhard, who never learned to type using more than 2 fingers, produced this with 1,100 names, including hers",
"question": "What is Schindler's list?"
},
{
"category": "Mythology",
"answer": "Poseidon carried off the maiden Theophane & turned her into a ewe; their offspring was the source of this mythical object",
"question": "What is the Golden Fleece?"
},
{
"category": "Literature",
"answer": "Published in 2011, P.D. James' final novel, \"Death Comes to Pemberley\", was a sequel to this novel from 200 years earlier",
"question": "What is Pride and Prejudice?"
},
{
"category": "U.S. State Names",
"answer": "5 U.S. states have 6-letter names; only these 2 west of the Mississippi River border each other",
"question": "What are Oregon & Nevada?"
},
{
"category": "Word Origins",
"answer": "Originally relating to a story of suffering, this word now more commonly refers to strong emotion of any kind",
"question": "What is passion?"
},
{
"category": "World Cinema",
"answer": "The 2007 biopic called \"La Môme\" in France, meaning \"The Kid\", was released in the U.S. under this other French title",
"question": "What is La Vie en Rose?"
},
{
"category": "History",
"answer": "Returning home in 1493, Columbus stopped in the Azores at an island with this name, also something he'd lost off the Haiti coast",
"question": "What is Santa Maria?"
},
{
"category": "Landmarks",
"answer": "Pskov & Nizhny Novgorod are 2 of the cities that have a fortress called this",
"question": "What is a kremlin?"
},
{
"category": "Foreign-Born Authors",
"answer": "In the 1950s the New York Times said this author \"is writing about all lust\" & his lecherous narrator \"is all of us\"",
"question": "Who is Vladimir Nabokov?"
},
{
"category": "Astronomy & Geography",
"answer": "At the winter solstice, the sun is in Sagittarius; it once appeared in this constellation, giving a geographic feature its name",
"question": "What is Capricorn?"
},
{
"category": "Television",
"answer": "Mike Post combined the sound of a slamming jail door, an anvil & 100 men stomping on a floor for this series that debuted in 1990",
"question": "What is Law & Order?"
},
{
"category": "British Landmarks",
"answer": "Like Sir Thomas More, 3 16th century English queens are buried at this location",
"question": "What is the Tower of London?"
},
{
"category": "Early American History",
"answer": "In 1692 Increase Mather wrote, \"It were better that ten suspected\" these \"escape, than that one innocent person … be condemned\"",
"question": "What are witches?"
},
{
"category": "Geography Mnemonics",
"answer": "Mimal, sometimes said to be the silhouette of a chef or elf, stands for Minnesota, Iowa, Missouri, and these 2 states",
"question": "What are Arkansas and Louisiana?"
},
{
"category": "Business Milestones",
"answer": "These were first sold in 1908, at a price equivalent to about $27,000 today",
"question": "What is the Ford Model T?"
},
{
"category": "In The Bookstore",
"answer": "The name of this author dead since 2013 now appears on books written by a former U.S. marshal & a former Apache helicopter pilot",
"question": "Who is Tom Clancy?"
},
{
"category": "Historic Art",
"answer": "The artwork once known in France as \"la tapisserie de la Reine Mathilde\" is better known as this",
"question": "What is the Bayeux Tapestry?"
},
{
"category": "Pop Stars",
"answer": "In 2022 she became the first woman to have a Billboard Top 10 album in 5 decades starting with the 1980s",
"question": "Who is Madonna?"
},
{
"category": "Classic Tale Characters",
"answer": "In one 19th century translation, she \"perceived the dawn of day and ceased\" speaking nearly 1,000 times",
"question": "Who is Scheherazade?"
},
{
"category": "USA",
"answer": "Ironically, though this company founded in the 1860s is Moore County, Tennessee's largest employer, Moore is a dry county",
"question": "What is Jack Daniel's?"
},
{
"category": "Historic People",
"answer": "After a 1789 event, he wrote, \"My first determination was to seek a supply of…water at Tofoa, & afterwards to sail for Tongataboo\"",
"question": "Who was William Bligh?"
},
{
"category": "The Movies",
"answer": "Laurence Olivier & Ernest Borgnine were considered for the lead role & Sergio Leone to direct for this film that turned 50 in 2022",
"question": "What is The Godfather?"
},
{
"category": "Continental Geography",
"answer": "Until a 1903 secession, this country's contiguous territory spanned 2 continents",
"question": "What is Colombia?"
},
{
"category": "Foreign-Born Authors",
"answer": "Early in her career she translated romance novels into Spanish, often changing the dialogue to make the heroines smarter",
"question": "Who is Isabel Allende?"
},
{
"category": "Historic Crimes",
"answer": "Saying it was stolen by Napoleon, self-styled Italian patriot Vincenzo Peruggia took it in 1911",
"question": "What is the Mona Lisa?"
},
{
"category": "U.S. Bodies of Water",
"answer": "Continuing a downward trend, in July 2022 it was at 27% capacity, its lowest level since 1937 when it was first being filled",
"question": "What is Lake Mead?"
},
{
"category": "Gods & Goddesses",
"answer": "Each morning she began her ride in her chariot across the sky ahead of her brother Sol, or Helios",
"question": "Who is Aurora (or Eos)?"
},
{
"category": "America At War",
"answer": "Until the Civil War, the Jan. 8 date of this battle of dubious military importance but big morale value was a national holiday",
"question": "What is the Battle of New Orleans?"
},
{
"category": "Children's Books",
"answer": "Its title character is told \"By the time you are real, most of your hair has been loved off…your eyes drop out & you get…shabby\"",
"question": "What is The Velveteen Rabbit?"
},
{
"category": "TV Finales",
"answer": "In a reunion over 40 years in the making, Dolly Parton appeared as an angel named Agnes in the final episode of this comedy in 2022",
"question": "What is Grace and Frankie?"
},
{
"category": "American Poems",
"answer": "In an 1847 poem this character sees her town of Grand-Pré burned, but finally reunites with her beau for a kiss before his death",
"question": "Who is Evangeline?"
},
{
"category": "Famous Names",
"answer": "In 2001 he published a book called \"Banging Your Head Against a Brick Wall\"; in 2002, \"Existencilism\"",
"question": "Who is Banksy?"
},
{
"category": "Children's Lit",
"answer": "The title object of this book \"never looked more beautiful…each strand held dozens of bright drops of early morning dew\"",
"question": "What is Charlotte's Web?"
},
{
"category": "Classic Songs",
"answer": "The shouts of excited children at a 1946 holiday parade are said to have inspired this perennial favorite",
"question": "What is \"Here Comes Santa Claus\"?"
},
{
"category": "Brand Names",
"answer": "Unable to make these candies perfectly round, the confectioner embraced this flawed name for the product",
"question": "What are Milk Duds?"
},
{
"category": "Countries of the World",
"answer": "It's home to 58 UNESCO World Heritage Sites, more than any other country; the sites include a volcano & a lagoon",
"question": "What is Italy?"
},
{
"category": "Action Movies",
"answer": "Its last line is \"If this is their idea of Christmas, I gotta be here for New Year's\"",
"question": "What is Die Hard?"
},
{
"category": "Presidential Facts",
"answer": "Only 3 presidents have married while in office— John Tyler was the first & he was the last",
"question": "Who is Woodrow Wilson?"
},
{
"category": "19th Century Americans",
"answer": "Demonstrating the dignity & humanity of Black Americans, he sat for 160 known photographs, the most of any American in the 19th century",
"question": "Who is Frederick Douglass?"
},
{
"category": "Latin Phrases",
"answer": "Originally, this 3-word phrase referred to when a doctor or apothecary substituted one medicine for another",
"question": "What is \"quid pro quo\"?"
},
{
"category": "1970s Movies",
"answer": "A 1975 premiere of this comedy advertised free coconuts for the first thousand in the audience",
"question": "What is Monty Python and the Holy Grail?"
},
{
"category": "Name's The Same",
"answer": "A cocktail, an island & a WWII venture originally called \"Development of Substitute Materials\" all bear this name",
"question": "What is Manhattan?"
},
{
"category": "U.S. Presidents",
"answer": "He was sworn in twice as President within 2 years, first by his father & then later by a former U.S. President",
"question": "Who is Calvin Coolidge?"
},
{
"category": "Plays",
"answer": "A 1609 story in which an exiled king of Bulgaria creates a sea palace with his magic may have inspired the plot of this play",
"question": "What is The Tempest?"
},
{
"category": "Landmarks",
"answer": "In 2009, during a 20th anniversary celebration, it was called \"an edifice of fear. On Nov. 9, it became a place of joy\"",
"question": "What is the Berlin Wall?"
},
{
"category": "World Capitals",
"answer": "Among its nicknames are the \"City of Classical Music\" &, possibly in honor of a famous resident from 1860 to 1938, the \"City of Dreams\"",
"question": "What is Vienna, Austria?"
},
{
"category": "Language & Its Meanings",
"answer": "Now meaning someone with nocturnal habits, it catches a sleeping dove in Shakespeare's \"…Lucrece\"",
"question": "What is a night owl?"
},
{
"category": "Flags of Our Hemisphere",
"answer": "The stars on this country's flag represent states, 26 of them; unlike the USA's, its \"federal district\" gets its own 27th star",
"question": "What is Brazil?"
},
{
"category": "Names in U.S. History",
"answer": "This father was the only man among the 13 plaintiffs in a class-action case filed in 1951",
"question": "Who is Oliver Brown?"
},
{
"category": "Children's Authors",
"answer": "Reversing the story of this heroine she created, Patricia Maclachlan was born on the prairie but spent much of her life in New England",
"question": "Who is Sarah? (from Sarah, Plain and Tall)"
}
]
I got a notebook here where I use Langchain to create prompts to evaluate the quiz data.
It looks like I was right in that the models can just play Jeopardy without needing to change the data.
https://gist.github.com/SlyEcho/a1e6ac9e44eb48a6769b44b61050b635