Ten Seconds Is Enough To Borrow A Voice

You can run a capable voice-cloning model on your own machine in about fifteen minutes, and a clean clip of somewhere between six and twelve seconds is enough to give it a voice to imitate. That is the useful part and the uncomfortable part in the same sentence. Chatterbox, the open-source speech model from Resemble AI, makes local cloning ordinary. This post walks through the Docker setup, how to add your own voices, why running it locally is worth the effort, and why the same ease should make all of us a little more careful about what we trust when we hear a familiar voice.

Getting it running in Docker

Chatterbox itself is a Python library, but the quickest route to something usable is a community server that wraps it in a web interface and an API. The one I'd point to is the Chatterbox-TTS-Server project, which ships Docker Compose files for several kinds of hardware. You need Docker, and for a GPU you need current drivers plus the NVIDIA Container Toolkit so containers can see the card. A CPU-only file exists too, and it's fine for trying things out, though patience becomes a feature.

The setup is short:

git clone https://github.com/devnen/Chatterbox-TTS-Server.git
cd Chatterbox-TTS-Server
docker compose up -d --build

That default file targets NVIDIA RTX 20, 30 and 40 series cards. Newer Blackwell cards, AMD hardware and CPU-only machines each have their own compose file, used with docker compose -f <file> up -d --build. Check the repository for the current list, because this corner of the ecosystem moves quickly.

The first start is slow because the container downloads the models into a named volume for the Hugging Face cache. Later restarts reuse it. Once it settles, the interface is at http://localhost:8004. The compose file also mounts the config file and local voices, reference_audio, outputs and logs folders from your host. That detail matters more than it looks, because it means your voices and generated audio live on your disk rather than inside a container you might rebuild next week and forget about.

Adding a custom voice

The server separates two ideas. Predefined voices are audio files in the voices folder, which you pick by name. Reference audio, kept in reference_audio, is what you use for cloning on demand. In both cases the mechanism is the same: you give the model a short sample, and it generates new speech in that speaker's timbre and manner.

You can drop files into those folders directly, upload them through the web interface, or use the upload endpoints. Then you request speech by pointing at the file. In clone mode, the request looks like this:

curl -X POST http://localhost:8004/tts \
  -H "Content-Type: application/json" \
  -d '{"text": "Welcome back. Here is your weekly summary.", "voice_mode": "clone", "reference_audio_filename": "my_voice.wav"}' \
  --output summary.wav

An OpenAI-compatible endpoint is also available at /v1/audio/speech, which is handy because plenty of existing tools can be pointed at it by changing a base URL.

The quality of the sample matters far more than people expect. Resemble's own examples use a reference of roughly ten seconds, and the working range is six to twelve seconds of a single speaker, recorded close to the microphone, with no music, no room echo and no one talking over them. If you start with a longer recording, cut a clean passage out of it. Something like this does the job:

ffmpeg -i interview.m4a -ss 00:00:12 -t 10 -ac 1 my_voice.wav

Two other things are worth knowing. The language of the sample should match the language you're generating, otherwise the output can pick up the accent of the reference. And the two tuning knobs, exaggeration and cfg_weight, both default to 0.5, which suits most text. For more dramatic delivery, lower cfg_weight to around 0.3 and raise exaggeration to 0.7 or more. Higher exaggeration tends to speed the voice up, and the lower cfg_weight pulls the pace back.

Why local is worth it

The best argument for running this yourself is not cost, although it does stop being a per-character bill. It is that a voice is personal data, and a recording of someone speaking is about as personal as audio gets. Hosted cloning services need you to upload that sample to someone else's infrastructure, agree to someone else's retention policy, and hope the terms stay the same. When the model runs in a container on your own machine, the sample never leaves it. You decide how long it exists and who can read it. For anyone working near healthcare, client confidentiality or anything under a privacy obligation, that difference decides whether the idea is even allowed.

There's a practical side as well. Latency is low, there are no rate limits, and nothing breaks because a vendor retired a model. The licence is MIT, so you can build on it commercially without a legal detour.

The benefits are also very human ones. Someone losing their voice to illness can bank a sample while they still can and keep speaking in something recognisably theirs. A small organisation can produce audio for accessibility, narrating long documents for people who can't comfortably read them. A creator can record one clean take and generate corrections later without setting the microphone up again. A developer can prototype a voice agent without a procurement cycle. None of that is theoretical, and none of it needed a research lab ten years ago.

The other side of ten seconds

Here is the part I don't want to skip past. A six-to-twelve-second sample is about the length of a voicemail greeting, a podcast intro or a short social clip. Most professionals have published far more than that without thinking about it. The barrier to imitating someone has dropped from a studio session to a webinar recording.

The obvious misuse is fraud. A phone call from a relative in trouble, or from a manager asking for an urgent payment, works because we trust the voice before we examine the request. Voice has long been treated as a soft form of identity, and banks and call centres have leaned on that. Cloning undermines the assumption that a familiar voice is evidence of anything. Beyond fraud there is harassment, fabricated statements attributed to public figures, and the quieter harm of someone's voice being used in places they never agreed to.

Open source makes this harder to manage, and I'd rather say so than pretend otherwise. Chatterbox embeds Resemble's Perth watermark in generated audio, designed to survive compression and editing, and a short script can check a file for it. That is a thoughtful safeguard, and it helps platforms and investigators identify output from this model. But it is a detection aid, not a lock. It only helps if someone checks, it only covers audio from this model, and with MIT-licensed code on a machine you control, I wouldn't treat the watermark as something a determined bad actor can't route around. That last point is my own reasoning rather than anything the project claims, but I think it's the sensible default assumption.

So I don't believe the answer is to stop the technology or to shrug at it. The practical answer is consent and habit. Only clone voices you own or have clear permission to use, and write that permission down if the use is anything more than personal. Keep reference clips as carefully as you would keep any personal data, which the local folder layout makes easy to do properly. Label synthetic audio as synthetic where people might reasonably assume otherwise.

The habits on the receiving end matter just as much. Families and small businesses should agree on a way to verify an urgent request that doesn't depend on recognising a voice, such as calling back on a known number or using a shared phrase. Finance teams should treat a voice instruction to move money like any unverified instruction, because that is now what it is.

Where I land

I'd rather understand a capability by running it than hold an opinion about it from a distance. The setup is short, the results are good enough to be useful, and the privacy case for keeping the sample on your own hardware is strong. It also gives you a concrete sense of how little it takes. If a clean ten-second clip and a container are all that stands between a person and a convincing copy of someone else, then the responsibility sits with the people running it, and the scepticism sits with everyone listening.

Both are worth getting comfortable with now, while it still feels slightly novel.