• MVP of DMV Test Prep

• Challenges

  • Multiple Modalities

  • Complex Situations

  • Changing Laws and Knowledge

• Ideas for Future Work

• Happy Ending

My mom-in-law failed the DMV test 4 times.

She studied aerospace engineering (Rocket Science).

She worked on project Mir, the first modular space station:

She’s smart.

She knows how to drive.

But English is not her native language.

I speak only English, so I couldn’t fully appreciate her difficulties, studying in a foreign language, taking a test in a foreign language.

Yet, she’s far from alone.

While waiting at the DMV, I saw many others facing the same struggle.

I realized…

This is a real problem Gen AI can solve today.

By combining infinite test question generation and language translation, AI-powered test practice will help non-native English speakers:

• study and learn concepts in their native language

• understand how these concepts translate to English

• practice test questions (often worded to trick you) in English

• for each question, see explanations in their native language and English

So, I started hacking on PassMyTests.com to help my mom-in-law succeed.

MVP of DMV Test Prep

According to Driving-Tests, over 90% of people who use their DMV practice tests will pass the exam.

But this success rate only applies after taking 10+ practice tests.

This statistic also includes native English speakers.

My stupid simple MVP:

1. You choose a language and US state (some answers vary per state)

2. You get a realistic AI-generated question with 4 choices, all in English

3. You select an answer

4. You see an explanation for the correct answer, in both English and translated by AI to your native language

After a few iterations, I arrived at the prompt:

I haven’t done a formal evaluation, but from eyeballing outputs, this instruction seems to increase randomness:

“The question should pull from a random set of 500 questions”

The key term is seems.

I could be fooling myself.

My next post on this project will dive into evaluations.

The variable ${translation} works like this:

• If the user didn’t select a language, then there’s no translation

• If the user selected a language, then we ask ChatGPT to translate: "translation": "Translation of the explanation in Spanish"

Putting it all together, here is my MVP for practicing DMV test questions:

Below is a sample question where the correct answer varies by state.

In California, the answer is 0.08%.

But in Utah, the answer is 0.05%.

Challenges

Multiple Modalities

One challenge is is that only certain types of questions are generated.

I couldn’t generate an image of a street sign or show an image as an answer.

This MVP is limited to text-based question and answers.

No support for other modalities yet.

Here’s a sample DMV test question that involves an image:

Complex Situations

In the same vein, I did not generate complex situational questions.

For example, a question could show a picture of two cards behind a school bus, while another car is turning left from the opposite lane.

Then, the question asks you to determine right of way.

Unfortunately, my MVP neither generates images nor complex situations.

Changing Laws and Knowledge

I grew up learning 10 and 2 as the optimal position to keep my hands on the steering wheel.

But knowledge evolves.

Because of airbags, now the optimal recommended position is 9 and 3.

Laws also change.

Some factual data might be year-dependent, presenting another challenge.

Ideas for Future Work

I see multiple vectors for improvement.

I could slice a DMV handbook, making a RAG that answers factual questions directly from the latest handbook, along with the explanation.

This would address the challenge of non-static laws and knowledge.

Another opportunity is to increase the quality of LLM outputs by having 2 interacting agents:

• one agent generates test questions

• second agent answers test questions

I could have open and closed-book variants, although presumably the test-taking agent should perform better if it’s open-book!

In general, it would be interesting for someone to turn DMV tests into a proper dataset LLMs can be evaluated on.

The non-trivial reasoning tasks are especially intriguing, such as analyzing an image to determine right of way, law violations, and best next actions.

Happy Ending

Relieved to report…

After a week practicing on PassMyTests, my mom-in-law passed her exam!

Her favorite feature — reading the English explanation right above the explanation in her native language.

She already knew most concepts, but often mistranslation caused confusion.

Seeing the English and translated explanations side-by-side helped her understand nuances in the questions, answers, and explanations.

Next up in Part 2 this series, I tackle a much bigger standardized test.

If you have feedback or requests for PassMyTests.com

DM me anytime @Sabrina_Ramonov or Linkedin.