Evaluating OpenAI Whisper's Hallucinations on Different Silences

Fascinating experiment to see how "silence" is transcribed by OpenAI's speech-to-text model, Whisper

Author

Sabrina Ramonov
October 29, 2024

AI hallucinations in healthcare have made recent headlines, as OpenAI’s speech-to-text model (Whisper) has been shown to hallucinate during silences.

I decided to test OpenAI Whisper’s transcription on different types of silence, as well as varying durations, amplitudes, and sample rates.

The results?

Very creepy Shining-level stuff!! 😱 

Try it yourself with the full python script at the end.

Experiments

WHITE NOISE 60 SECONDS DURATION

Côte d'Ivoire Côte d'Ivoire Côte d'Ivoire Côte d'Ivoire

Ivory Coast Ivory Coast Ivory Coast Ivory Coast

TRANSLATION

PINK NOISE 60 SECONDS DURATION

若不小心地踏入水中,便会被水淹没。 若不小心地踏入水中,便会被水淹没。 若不小心地踏入水中,便会被水淹没

If you accidentally step into the water, you will be submerged. If you accidentally step into the water, you will be submerged. If you accidentally step into the water, you will be submerged.

TRANSLATION

BROWN NOISE 60 SECONDS DURATION

Ctrl-Alt-Shift-Alt-E Ctrl-Alt-Shift-A Ctrl-Alt-E Ctrl-Alt-E Ctrl-A Ctrl-A

BROWN NOISE 60 SECONDS DURATION

✔️ Follow me! https://bit.ly//ISCVideo New HD video every 2 days, subscribe and don't forget to click the bell 🔔 to get notifications of all my latest updates. Thank you! 😉👍

VIOLET NOISE 180 SECONDS DURATION

Rwy'n gwybod eich bod chi'n gwneud y pethau yma. Yn ogystal â'r ddwyieithiau, mae'n rhaid i chi ddod o'r ffordd arall. Cymysgwch y pethau yma. Yn y ddau o'r rhai o'r pethau yma, rwy'n rhaid i chi ddod o'r ffordd arall. Rwy'n gwybod eich bod chi'n gwybod y pethau yma. Rwy'n gobeithio, rwy'n gobeithio, rwy'n gobeithio, rwy'n gobeithio, rwy'n gobeithio, rwy'n gobeithio, rwy'n gobeithio. Rwy'n gobeithio y byddwn yn gweithio'n gyflym. Rwy'n gobeithio y byddwn yn gweithio'n gyflym. Rwy'n gobeithio y byddwn yn gweithio'n gyflym.

I know you are doing these things. Besides the routines, you must find another way. Mix things up here. In both of these things, you must find another way. I know you know these things. I hope, I hope, I hope, I hope, I hope, I hope, I hope. I hope we work quickly. I hope we work quickly. I hope we work quickly."

Let me know if there’s anything else you’d like translated or clarified!

TRANSLATION

PINK NOISE 180 SECONDS DURATION

它们将在天气和海水中发生交叉反差,因此,它将在水面上发生交叉反差。 它们将在天气和海水中发生交叉反差,因此,它将在天气和海水中发生交叉反差。 它们将在天气和海水中发生交叉反差,因此,它将在天气和海水中发生交叉反差。 它们将在天气和海水中发生交叉反差,因此,它将在天气和海水中发生交叉反差。 它们将在天气和海水中发生交叉反差,因此,它将在天气和海水中发生交叉反差。 它们将在天气和海水中发生交叉反差,因此,它将在天气和海水中发生交叉反差。 它们将在天气和海水中发生交叉反差,因此,它将在天气和海水中发生交叉反差。 它们将在天气和海水中发生交叉反差,因此,它将在天气和海水中发生交叉反差。 它们将在天气和海水中发生交叉反差,因此,它将在天气和海水中发生交叉反差。 它们将在天气和海水中发生交叉反差,因此,它将在天气和海水中发生交叉反差。 它们将在天气和海水中发生交叉反差,因此,它将在天气和海水中发生交叉反差。 它们将在天气和海水中发生交叉反差,因此,它将在天气和海水中发生交叉反差。 它们将在天气和海水中发生交叉反差,因此,它将在天气和海水中发生交叉反差。 它们将在天气和海水中发生交叉反差,因此,它将在天气和海水中发生交叉反差。

They will undergo cross-contrast in the weather and seawater, so cross-contrast will occur on the water surface. They will undergo cross-contrast in the weather and seawater, so cross-contrast will occur in the weather and seawater. They will undergo cross-contrast in the weather and seawater, so cross-contrast will occur in the weather and seawater. They will undergo cross-contrast in the weather and seawater, so cross-contrast will occur in the weather and seawater. They will undergo cross-contrast in the weather and seawater, so cross-contrast will occur in the weather and seawater. They will undergo cross-contrast in the weather and seawater, so cross-contrast will occur in the weather and seawater. They will undergo cross-contrast in the weather and seawater, so cross-contrast will occur in the weather and seawater. They will undergo cross-contrast in the weather and seawater, so cross-contrast will occur in the weather and seawater. They will undergo cross-contrast in the weather and seawater, so cross-contrast will occur in the weather and seawater. They will undergo cross-contrast in the weather and seawater, so cross-contrast will occur in the weather and seawater. They will undergo cross-contrast in the weather and seawater, so cross-contrast will occur in the weather and seawater. They will undergo cross-contrast in the weather and seawater, so cross-contrast will occur in the weather and seawater. They will undergo cross-contrast in the weather and seawater, so cross-contrast will occur in the weather and seawater. They will undergo cross-contrast in the weather and seawater, so cross-contrast will occur in the weather and seawater.

TRANSLATION

PINK NOISE 120 SECONDS DURATION

如果您不習慣使用手機,您可以使用手機錄影。 如果您不習慣使用手機,您可以使用手機錄影。 如果您不習慣使用手機,您可以使用手機錄影。 如果您不習慣使用手機,您可以使用手機錄影。 如果您不習慣使用手機,您可以使用手機錄影。 如果您不習慣使用手機,您可以使用手機錄影。 如果您不習慣使用手機,您可以使用手機錄影。 如果您不習慣使用手機,您可以使用手機錄影。 如果您不習慣使用手機,您可以使用手機錄影。

If you are not used to using a mobile phone, you can use mobile phone recording. If you are not used to using a mobile phone, you can use mobile phone recording. If you are not used to using a mobile phone, you can use mobile phone recording. If you are not used to using a mobile phone, you can use mobile phone recording. If you are not used to using a mobile phone, you can use mobile phone recording. If you are not used to using a mobile phone, you can use mobile phone recording. If you are not used to using a mobile phone, you can use mobile phone recording. If you are not used to using a mobile phone, you can use mobile phone recording. If you are not used to using a mobile phone, you can use mobile phone recording.

TRANSLATION

BROWN NOISE 120 SECONDS DURATION

Ychwanegwch y gwres yma i ddod i mewn i'r gwasanaethau. Ychwanegwch y gwasanaethau ar y gwasanaethau yma i gyd. Ychwanegwch y gwasanaethau ar y gwasanaethau yma i gyd. Ychwanegwch y gwasanaethau yma. Ychwanegwch y gwasanaethau yma. Ychwanegwch y gwasanaethau yma. Ychwanegwch y gwasanaethau yma.

Add the heat here to enter the services. Add the services to all the services here. Add the services to all the services here. Add the services here. Add the services here. Add the services here. Add the services here.

TRANSLATION

VIOLET NOISE 5 SECONDS DURATION

I can do it. I can do it. I can do it. I can do it. I can do it. I am so good at this! I am so good at this!

VIOLET NOISE 12 SECONDS DURATION

多謝您收睇時局新聞,再會!

Thank you for watching the current affairs news, goodbye!

TRANSLATION

VIOLET NOISE 12 SECONDS DURATION SAMPLE_RATE 16000 AMP 0.5

チャンネル登録をお願いします。

Please subscribe to the channel.

TRANSLATION

BROWN NOISE 12 SECONDS DURATION SAMPLE_RATE 16000 AMP 0.5

ご視聴ありがとうございました

Thank you for watching.

TRANSLATION

WHITE NOISE 12 SECONDS DURATION SAMPLE_RATE 16000 AMP 0.5

Thanks for watching and don't forget to like and subscribe!

BROWN NOISE 36 SECONDS DURATION SAMPLE_RATE 16000 AMP 0.5

......... ....... ............ ............ ............ ............ ............ ............ ..... ... ......... .....

VIOLET NOISE 12 SECONDS DURATION SAMPLE_RATE 16000 AMP 999

Thanks for watching please subscribe and hit that like button....

PINK NOISE 12 SECONDS DURATION SAMPLE_RATE 16000 AMP 0.01

ご視聴ありがとうございました

Thank you for watching.

TRANSLATION

BROWN NOISE 12 SECONDS DURATION SAMPLE_RATE 16000 AMP 0.01

Yn y blynyddoedd, mae'n rhaid i ni ddweud y byddwn ni'n ymwneud â'r rhesymau sydd ar gael yng Nghymru. Diolch.

BLUE NOISE 12 SECONDS DURATION SAMPLE_RATE 16000 AMP 0.01

ご視聴ありがとうございました

Thank you for watching.

TRANSLATION

VIOLET NOISE 12 SECONDS DURATION SAMPLE_RATE 16000 AMP 0.01

you

PINK NOISE 300 SECONDS DURATION SAMPLE_RATE 16000 AMP 0.5

若您需要更詳細的資訊或更多的資訊,請搜尋「雲端網站」。 如果您需要更詳細的資訊或更多的資訊,請搜尋「雲端網站」。 如果您需要更詳細的資訊或更多的資訊,請搜尋「雲端網站」。 如果您需要更詳細的資訊或更多的資訊,請搜尋「雲端網站」。 如果您需要更詳細的資訊或更多的資訊,請搜尋「雲端網站」。 如果您需要更詳細的資訊或更多的資訊,請搜尋「雲端網站」。 如果您需要更詳細的資訊或更多的資訊,請搜尋「雲端網站」。 如果您需要更詳細的資訊或更多的資訊,請搜尋「雲端網站」。 如果您需要更詳細的資訊或更多的資訊,請搜尋「雲端網站」。 如果您需要更詳細的資訊或更多的資訊,請搜尋「雲端網站」。 如果您需要更詳細的資訊或更多的資訊,請搜尋「雲端網站」。 如果您需要更詳細的資訊或更多的資訊,請搜尋「雲端網站」。 如果您需要更詳細的資訊或更多的資訊,請搜尋「雲端網站」。 如果您需要更詳細的資訊或更多的資訊,請搜尋「雲端網站」。 如果您需要更詳細的資訊或更多的資訊,請搜尋「雲端網站」。 如果您需要更詳細的資訊或更多的資訊,請搜尋「雲端網站」。 如果您需要更詳細的資訊或更多的資訊,請搜尋「雲端網站」。 如果您需要更詳細的資訊或更多的資訊,請搜尋「雲端網站」。 如果您需要更詳細的資訊或更多的資訊,請搜尋「雲端網站」。 如果您需要更詳細的資訊或更多的資訊,請搜尋「雲端網站」。 如果您需要更詳細的資訊或更多的資訊,請搜尋「雲端網站」。 如果您需要更詳細的資訊或更多的資訊,請搜尋「雲端網站」。 如果您需要更詳細的資訊或更多的資訊,請搜尋「雲端網站」。 如果您需要更詳細的資訊或更多的資訊,請搜尋「雲端網站」。

If you need more detailed information or additional information, please search for "cloud website." (REPEATED LOTS OF TIMES)

TRANSLATION

VIOLET NOISE 300 SECONDS DURATION SAMPLE_RATE 16000 AMP 0.5

Rhaid i ni gysylltu â'r cwmni'n ysgrifennol i ddod yn rhaid i ni ddod yn rhaid i ni ddod yn rhaid i ni ddod yn rhaid i ni ddod yn rhaid i ni ddod yn rhaid. Rhaid i ni ddod yn rhaid i ni ddod yn rhaid i ni ddod yn rhaid. Rhaid i ni ddod yn rhaid i ni ddod yn rhaid i ni ddod yn rhaid. Diolch yn fawr am wylio'r fideo.

We must contact the company in writing, and we must proceed with this step repeatedly. We must proceed with this step repeatedly. We must proceed with this step repeatedly. Thank you very much for watching the video.

TRANSLATION

PYTHON SCRIPT

from openai import OpenAI
import numpy as np
from scipy.io import wavfile
import time

client = OpenAI()


class NoiseGenerator:
    def __init__(self, duration_seconds=5, sample_rate=44100, amplitude=0.5):
        self.duration_seconds = duration_seconds
        self.sample_rate = sample_rate
        self.amplitude = amplitude
        self.num_samples = int(duration_seconds * sample_rate)

    def generate_white_noise(self):
        """Generate white noise (equal energy per frequency)"""
        samples = np.random.uniform(-1, 1, size=self.num_samples)
        return samples

    def generate_pink_noise(self):
        """Generate pink noise (1/f power spectrum)"""
        # Generate white noise
        white = np.random.uniform(-1, 1, size=self.num_samples)

        # Create pink noise by applying 1/f filter
        f = np.fft.fftfreq(len(white))
        f = np.abs(f)
        f[0] = 1e-6  # Avoid divide by zero
        f = 1 / np.sqrt(f)
        f = f / np.sqrt(np.mean(f**2))  # Normalize

        # Apply filter in frequency domain
        white_fft = np.fft.fft(white)
        pink_fft = white_fft * f
        pink = np.real(np.fft.ifft(pink_fft))

        return pink

    def generate_brown_noise(self):
        """Generate brown noise (1/f^2 power spectrum)"""
        # Integrate white noise
        white = np.random.uniform(-1, 1, size=self.num_samples)
        brown = np.cumsum(white)

        # Normalize
        brown = brown - np.mean(brown)
        brown = brown / np.max(np.abs(brown))

        return brown

    def generate_blue_noise(self):
        """Generate blue noise (f power spectrum)"""
        # Generate white noise
        white = np.random.uniform(-1, 1, size=self.num_samples)

        # Create blue noise by applying f filter
        f = np.fft.fftfreq(len(white))
        f = np.abs(f)
        f[0] = 1e-6  # Avoid divide by zero
        f = np.sqrt(f)
        f = f / np.sqrt(np.mean(f**2))  # Normalize

        # Apply filter in frequency domain
        white_fft = np.fft.fft(white)
        blue_fft = white_fft * f
        blue = np.real(np.fft.ifft(blue_fft))

        return blue

    def generate_violet_noise(self):
        """Generate violet noise (f^2 power spectrum)"""
        white = np.random.uniform(-1, 1, size=self.num_samples)

        f = np.fft.fftfreq(len(white))
        f = np.abs(f)
        f[0] = 1e-6
        f = f**2
        f = f / np.sqrt(np.mean(f**2))

        white_fft = np.fft.fft(white)
        violet_fft = white_fft * f
        violet = np.real(np.fft.ifft(violet_fft))

        return violet

    def save_noise(self, samples, filename):
        """Save noise samples to WAV file"""
        # Normalize and adjust amplitude
        samples = samples / np.max(np.abs(samples))
        samples = samples * self.amplitude

        # Convert to 16-bit integers
        samples = (samples * 32767).astype(np.int16)

        # Save to file
        wavfile.write(filename, self.sample_rate, samples)
        print(f"Created {filename} ({self.duration_seconds} seconds)")


def generate_all_noises(duration=5, sample_rate=44100, amplitude=0.5):
    """Generate all types of noise and save them"""
    generator = NoiseGenerator(duration, sample_rate, amplitude)

    # Generate and save each type of noise
    noises = {
        "white": generator.generate_white_noise(),
        "pink": generator.generate_pink_noise(),
        "brown": generator.generate_brown_noise(),
        "blue": generator.generate_blue_noise(),
        "violet": generator.generate_violet_noise(),
    }

    for noise_type, samples in noises.items():
        generator.save_noise(samples, f"{noise_type}_noise.wav")


# Example usage
if __name__ == "__main__":
    for dur in [12]:
        for amp in [0.5]:
            for sr in [16000]:
                generate_all_noises(duration=dur, sample_rate=sr, amplitude=amp)
                for noise in ["white", "pink", "brown", "blue", "violet"]:
                    audio_file = open(f"{noise}_noise.wav", "rb")
                    transcription = client.audio.transcriptions.create(
                        model="whisper-1", file=audio_file
                    )
                    print(
                        f"=== TRANSCRIPTION FOR {noise.upper()} NOISE {dur} SECONDS DURATION SAMPLE_RATE {sr} AMP {amp} ==="
                    )
                    print(transcription.text)
                    print("===========================================")
                    time.sleep(1)