• SAT Math Test

• Challenge 1: Missing Correct Answer

• Challenge 2: In-Context Learning

• Challenge 3: Probabilistic Outputs

• Conclusion

This is Part 2 of the series, AI-Powered Infinite Test Prep.

In Part 1, I shared my mom-in-law’s struggles with the DMV test in English.

This inspired me to build PassMyTests.com which generates infinite test practice questions while translating explanations into her native language.

I contemplated other standardized exams, similar in multiple-choice format, but haunting many millions more people:

The dreaded SAT!

In 2023 alone, 1.9 million students took the SAT.

I'm embarrassed to reveal how many hours I’ve spent studying for it.

I’d shove big SAT prep books into my backpack, heading off to Barnes & Noble.

SAT Math Test

I decided to tackle the SAT Math test first.

My MVP goal:

Generate infinite conceptually plausible and realistic SAT Math questions, while translating the answer explanations for non-native English speakers.

First, I repurposed my DMV prompt:

Next, I made a new route in my MVP:

PassMyTests.com/sat-math-test 

Here’s a sample question and explanation translated to Spanish:

This MVP didn’t take long, but now the hard part begins.

Next, I share several challenges I encountered.

Challenge 1: Missing Correct Answer

Sometimes, ChatGPT fails to include the correct answer as an option:

In the above image, the correct answer is 600 square meters.

But this was not an option.

To fix this, I added a verification step to my prompt:

Much better.

Sample questions generated by ChatGPT-4:

However, even if the correct answer is in the answer list, sometimes the "answer” index in the output does not match the correct answer index.

Recall my prompt:

"answer": 
<index of the current choice e.g. A is 0, B is 1, etc.>,

I haven’t improved this.

I’ll try assigning a random number between 0 and 3 to the right answer, then inserting it into the JSON output.

Challenge 2: In-Context Learning

Nonetheless, initial results seem promising.

It’s been years since I studied for a test, but the questions looked plausible.

I was curious:

Would in-context learning improve it?

In-context learning is an effective prompt engineering technique where you include examples directly in your prompt, enabling ChatGPT to tackle new tasks without finetuning.

I added 4 real SAT Math questions to my prompt:

Here are questions generated with in-context learning:

Unfortunately, the questions are repetitive and substantially less random after applying in-context learning.

Even ChatGPT agrees!

In the last image, ChatGPT admits:

“It seems the same question was generated again.“ 

For now, I removed in-context learning from my prompt.

Challenge 3: Probabilistic Outputs

As I discussed in a previous post on LLM Ops, LLMs are probabilistic.

There is a non-zero probability that occasionally it returns an invalid JSON, despite my explicit request for a valid JSON output.

I encountered the first JSON failure after only n=20 generations:

{
  "body": "In a geometry class, the measures of the three angles of a triangle are \(3x\), \(4x\), and \(5x\) degrees. Which of the following could be the measure of the smallest angle?",
  "choices": ["\(20^\circ\)", "\(30^\circ\)", "\(40^\circ\)", "\(50^\circ\)"],
  "answer": 0,
  "explanation": "In any triangle, the sum of the three angles is always \(180^\circ\). Therefore, \(3x + 4x + 5x = 180\). Solving for x gives \(12x = 180\), so \(x = 15\). The smallest angle would then be \(3x = 3(15) = 45^\circ\), which is closest to \(40^\circ\).",
  "translation": "${translation}"
}

ChatGPT mixed LaTeX and JSON without escaping!

In a real LLM-powered app, if I did not have robust error handling in place, this could crash the app.

An imperfect bandaid, I added another verification step:

But there’s still no guarantee because LLM outputs are probabilistic.

A faster and cheaper approach, avoiding another LLM call, would be to assume the JSON output could be malformed and fix it with a custom parser.

Only if the parser fails, then retry with ChatGPT.

Just as I was congratulating myself for this brilliant 80/20 solution, I ran into another issue caused by probabilistic outputs!

On generation N=1000, I got this perplexing and very wrong JSON output:

• JSON just repeats itself

• translation field is the fully translated JSON

• answer index is 2 (x = 6) but the JSON output is 1 (x = 5)

{
    "body": "What is the solution to the equation 4x - 7 = 17?",
    "choices": ["A. x = 10", "B. x = 5", "C. x = 6", "D. x = 4"],
    "answer": 1,
    "explanation": "Add 7 to both sides of the equation to isolate 4x. This gives 4x = 24. Then, divide by 4 on both sides to solve for x, which results in x = 6.",
    "translation": "{
    "body": "¿Cuál es la solución de la ecuación 4x - 7 = 17?",
    "choices": ["A. x = 10", "B. x = 5", "C. x = 6", "D. x = 4"],
    "answer": 1,
    "explanation": "Suma 7 a ambos lados de la ecuación para aislar 4x. Esto da como resultado 4x = 24. Luego, divide entre 4 en ambos lados para resolver x, lo que da como resultado x = 6.",
    "translation": ""
  }
}

Perhaps ChatGPT is deliberately trying to entertain me…

Ugh, and I thought I had solved Challenge #1.

Definitely haven’t solved Challenge #3.

Traditional software engineering deals with structured data.

LLMs deal with unstructured natural language and probabilistic outputs.

I’m still grappling with this new reality.

Conclusion

Altogether, I enjoyed building this MVP and see many ways to improve it.

I believe Gen AI will massively transform traditional test prep and education more broadly, increasing accessibility especially across multiple languages.

Here’s a slice of my “I Should Really Do That” list:

• Evaluation criteria and methodology — given 4 different prompts and different models, generate N=100,1000,10000 questions, and evaluate each question based on a set of criteria

• Restrict question types -- e.g. LINEAR_EQUATIONS, QUADRATIC_EQUATIONS, GEOMETRY_TRIANGLE, LINEAR_ALGEBRA

• Multimodality — generate both an image and the question

• Create brand new questions based on concepts — given a concept along with axioms and theorems, generate a question that utilizes it

If you have feedback or requests for PassMyTests.com

DM me anytime @Sabrina_Ramonov or Linkedin.