GuidesGuideCan AI listen to music?Directory

Can AI listen to music? Claude, ChatGPT and Gemini compared

Updated

Of the three, Gemini’s documentation is the only one we read that names music: its API audio guide lists “Emotion detection in speech and music”, and the Gemini app takes audio files. OpenAI’s API has audio-input models too, and ChatGPT’s app accepts audio files on paid plans, framed mainly around transcripts and notes, according to MIXED News’s report on OpenAI’s help pages (October 2026). Claude’s documented inputs are text and images, so to have Claude “listen”, you turn the song into something it can read: numbers, a picture, MIDI, or pictures in order with a tool in Claude Code.

All three can also talk about a well-known song from what they learned in training. That’s knowing about the song, which is different from hearing your file, and the table keeps the two apart.

An AI drafted this page from the sources below and another AI checked it. No person has run a ChatGPT or Gemini test for it: those rows come from the makers’ documentation and, for ChatGPT’s app, a news report. Listen Along’s example listen is a real, public run. Listen Along and Counterparts are ours: Mike LaPeter runs Counterparts.

The four ways an AI could “hear” a song

ClaudeChatGPTGemini
Voice mode

Your speech becomes text and the recording is deleted (Anthropic’s help). Nothing about music.

OpenAI’s Realtime API and GPT-Live take live audio (GPT-Realtime-1.5: “audio in, audio out”). How ChatGPT’s voice mode treats music isn’t in the pages we could read.

Google’s Live API takes raw audio. How the Gemini Live app treats music isn’t in the pages we read.

Upload a song file in the app

The claude.ai upload list has documents and images; no audio type.

Audio files on paid plans, up to 512 MB, aimed at transcripts and notes (MIXED News, Oct 2026).

Yes: 10 minutes of audio in total free, 3 hours on AI Pro or Ultra, 100 MB a file.

Audio in the API

Text and images in; other files only through the code execution tool, as data a program reads.

gpt-audio-1.5 and the Realtime models take audio; the flagship GPT-6 Astra takes text and images; transcription takes files up to 25 MB.

Native audio, music named, 32 tokens per second, up to 9.5 hours a prompt, heard as 16 kbps mono.

Answering from lyrics and training

Yes, for a well-known song.

Yes, for a well-known song.

Yes, for a well-known song.

A few things the table can’t hold:

  • Gemini’s API hears a compressed mix. Its audio guide says audio is “Downsampled to 16 Kbps” and multi-channel audio is “combined to single channel”. It gets the whole song in one go, and it doesn’t get the stereo field: which side of the mix a guitar sits on.
  • ChatGPT’s app facts are second-hand. OpenAI’s help pages refused our download, so the upload row rests on a report by MIXED News, which read those pages on October 6, 2026. The uses OpenAI lists are meetings, interviews and lectures, and long recordings may be processed in smaller sections when Data Analysis is available. Check help.openai.com in a browser before relying on it.
  • Knowing the song can look like hearing it. An AI that recognizes a famous song can describe it from what it learned. One r/Bard user tested the other side in October 2025: they renamed a comedy track to a neutral filename, uploaded it, and wrote that “Gemini thinks it’s a serious track”. To see what an assistant actually hears, give it a recording it can’t know, such as your own or a live take.

What are you really after?

Feedback on my own track

Gemini is the most direct: upload the file in the app or send it through the API and ask about structure, energy or the vocal. Measurement tools are the other route. They put numbers on what a mix engineer listens for, and an assistant writes the critique from them:

  • RoastYourMix takes a WAV or MP3 and gives “a score out of 100, what’s holding the mix back and what to fix first”.
  • audio-analyzer-rs, an MCP server for Claude Desktop and Claude Code, reports key, tempo, dynamics, stereo field and sections, and can compare two tracks.

Expect technical notes more than taste. The top reply in an r/ChatGPT thread on this called AI feedback “mostly technical/pattern-based”. Listen Along usually won’t help with an unreleased track or an instrumental: it needs the song’s synced lyrics on LRCLIB.

What song is this?

A music-recognition app such as Shazam is built for this. A chat assistant helps when the song isn’t playing: describe the bits you remember and let it guess. SoundGuys found a song with ChatGPT that way.

Chords, notes and lyrics

For notes and chords, Spotify’s Basic Pitch, which also runs in the browser at basicpitch.io, turns audio into MIDI, best with one instrument at a time. Read the MIDI in a notation app, or convert it to JSON or text and hand that to an assistant (how, for Claude); we haven’t tested how well an assistant names chords from a full mix. For lyrics, OpenAI’s transcription API and Gemini’s API both transcribe audio, though their docs are written for speech. For a well-known song, published lyrics are quicker and more reliable.

An AI that listens with me

This is what many “can Claude listen to music” threads are about: people want the AI’s own reaction to a song they love. Gemini can give one from the file. Claude users have built their own ways, from converting MP3 to MIDI to JSON to spectrogram images and librosa scripts.

Listen Along, a free tool we make, is one of those ways, built around order. In Claude Code, it splits a recording into the lead voice, the other voices and five instrument parts, measures them, and shows Claude the song 15 seconds at a time as pictures.

Claude writes a note on each stretch, and the note is sealed before the next window opens, “So a wrong guess stays wrong on the page.” The pictures are measured from the audio, with the song’s timed lyrics from LRCLIB under the voice, and no second model describes the song to Claude. At the end you get a page that plays the notes back against the music, like this listen to Gotye’s “Somebody That I Used To Know”.

Claude forgets the listen when the session ends unless it has a memory; we make Counterparts, a memory for Claude Code, and use it for this.

It’s a heavy install: a Mac with an Apple chip, about 5 GB of disk, 8 to 18 minutes to prepare most songs, and a song with synced lyrics. The steps, its costs, and the other Claude routes are in Can Claude listen to music?.

Tokens: why a song costs so differently

Gemini’s API counts 32 tokens per second of audio, so a 4-minute song is about 7,680 tokens. A Listen Along listen to the same song fills about 130,000 tokens of Claude’s context, and every step re-reads the conversation from cache, so a whole song is “a few million cached tokens read against your usage limit”.

The two numbers measure different things. Gemini’s are the audio itself, compressed. Listen Along’s are 16 or so windows of three pictures each, plus the notes Claude writes between them. An analysis MCP server is light on context: audio-analyzer-rs says a full analysis takes “under 1% context window usage”.

Questions people ask

Which AI can listen to music?

Of the three, Gemini’s documentation is the only one we read that names music: Google’s audio guide for the Gemini API lists “Emotion detection in speech and music”, and the Gemini app accepts audio uploads. ChatGPT’s app accepts audio files on paid plans, mainly for transcripts and notes, according to MIXED News’s report on OpenAI’s help pages (October 2026), and OpenAI’s API has audio-input models too, such as gpt-audio-1.5 and GPT-Realtime-1.5. Claude’s documented inputs are text and images, so to have Claude listen you turn the song into numbers or pictures it can read, usually with a tool in Claude Code.

Can ChatGPT listen to music?

ChatGPT’s app takes audio files on paid plans, but nothing we could read says how it handles music. According to MIXED News’s report on OpenAI’s help pages (October 2026), the app accepts audio files up to 512 MB, not on the free plan, and the uses OpenAI lists are meetings, interviews and lectures. In the API, gpt-audio-1.5 and the Realtime models take audio input; OpenAI’s flagship GPT-6 Astra takes text and images. We couldn’t download OpenAI’s help pages, so check them in a browser before relying on the app details.

Can Gemini listen to music?

Yes, by its own documentation. The Gemini app takes audio files (10 minutes of audio in total on the free plan, 3 hours on Google AI Pro or Ultra, up to 100 MB a file). The Gemini API’s audio guide names music: it lists “Emotion detection in speech and music” and answering questions about audio with timestamps. In the API, audio is downsampled to 16 kbps and combined to one channel, so it gets the mix rather than the stereo picture; the app’s help doesn’t say how it processes audio.

Can I play a song to an AI in voice mode?

We found nothing on music in voice mode in the pages we could read. Claude’s voice mode converts your speech to text and deletes the recording. OpenAI’s and Google’s live APIs both take live audio, but neither app’s help that we could read says how voice mode treats music. One Reddit user says ChatGPT’s voice once listened along to music live until that stopped working, which we couldn’t confirm.

Can AI listen to a song and tell me what it is?

For naming a song you hear, a music-recognition app such as Shazam is the tool built for it. Chat assistants are better at the other case: describe what you remember (a line of lyrics, the beat, where you heard it) and let them guess. SoundGuys found a song with ChatGPT that way: “Instead of matching audio patterns, we described what we remembered.”

Can AI listen to my music and give feedback?

Partly. Gemini can take your file and describe it. Measurement tools give numbers (loudness, frequency balance, stereo width, key, tempo) that an assistant can turn into a critique: RoastYourMix scores an uploaded WAV or MP3, and audio-analyzer-rs is an MCP server that hands Claude those numbers. A top reply in an r/ChatGPT thread put the limit well: AI feedback is “mostly technical/pattern-based”.

Can AI listen to a song and tell me the chords?

Use a transcription tool. Spotify’s Basic Pitch turns a recording into a MIDI file of its notes, and its README says it “works best on one instrument at a time”. You can then read the MIDI in a notation app, or convert it to JSON or text and give that to an assistant to name the chords (how, for Claude); we haven’t tested how well an assistant names chords from a full mix. Listen Along, which we make, deliberately doesn’t name chords: “Nothing names the chord or the key”.

Can AI listen to a song and write out the lyrics?

Speech-to-text models will try: OpenAI’s transcription API takes MP3, WAV and other formats up to 25 MB, and Gemini’s API transcribes audio. Their docs are written for speech, and we haven’t tested either on singing over a band. For a well-known song, published lyrics are a quicker route.

How many tokens is a song?

In the Gemini API, 32 tokens per second of audio, so a 4-minute song is about 7,680 tokens. A Listen Along listen in Claude Code fills about 130,000 tokens of context for the same length, plus a few million cached tokens read against your usage limit as each step re-reads the conversation. They measure different things: Gemini’s are compressed audio tokens, and Listen Along’s are pictures of the song plus the notes Claude writes about them.

Is Claude deaf?

It reads rather than hears. Its documented inputs are text and images, and the claude.ai upload list has no audio type, so a song has to be turned into something it reads first: analysis numbers, a spectrogram picture, MIDI, or Listen Along’s window-by-window pictures. Can Claude listen to music? walks through each.

Questions? Ask in r/AIMemorySystems.

References

More guides: all guides.