• Paper on arXiv •

Vikentii Pankov, Valeria Pronina, Alexander Kuzmin, Maksim Borisov, Nikita Usoltsev, Xingshan Zeng, Alexander Golubkov, Nikolai Ermolenko, Aleksandra Shirshova, Yulia Matveeva

Abstract

We address zero-shot TTS systems’ noise-robustness problem by proposing a dual-objective training for the speaker encoder using self-supervised DINO loss. This approach enhances the speaker encoder with the speech synthesis objective, capturing a wider range of speech characteristics beneficial for voice cloning. At the same time, the DINO objective improves speaker representation learning, ensuring robustness to noise and speaker discriminability. Experiments demonstrate significant improvements in subjective metrics under both clean and noisy conditions, outperforming traditional speaker-encoder-based TTS systems. Additionally, we explore training zero-shot TTS on noisy, unlabeled data. Our two-stage training strategy, leveraging self-supervised speech models to distinguish between noisy and clean speech, shows notable advances in similarity and naturalness, especially with noisy training datasets, compared to the ASR-transcription-based approach.

[*] this page is intended for research demonstration purposes only.

System architecture

Robustness to noises at inference and general similarity improvement

text	speaker	gender	noise	reference	DINO-VITS	AAM-Softmax-VITS	YourTTS	BYOL-A
HE PASSED THROUGH HENLEY SAINT ALBANS AND CAME SO NEAR TO LONDON AS HARROW ON THE HILL	adi	male	street
THE INFLUENCE WITH THE TIMAEUS HAS EXERCISED UPON POSTERITY IS DUE PARTLY TO A MISUNDERSTANDING	casey	male	marathon
BY QUICK MARCHES THROUGH THESE INACCESSIBLE MOUNTAINS THAT GENERAL FREED HIMSELF FROM THE SUPERIOR FORCES OF THE COVENANTERS	casey	male	outdoor
HER SEA GOING QUALITIES WERE EXCELLENT AND WOULD HAVE AMPLY SUFFICED FOR A CIRCUMNAVIGATION OF THE GLOBE	jess	female	booth
I CAN'T SEE IT IN THAT LIGHT SAID THE OLD LAWYER	jess	female	booth
I DID NOT WRONG MYSELF SO BUT I PLACED A WRONG ON THEE	jess	female	hairdryer
THIS REALITY BEGINS TO EXPLAIN THE DARK POWER AND OTHERWORLDLY FASCINATION OF TWENTY THOUSAND LEAGUES UNDER THE SEAS	jess	female	hairdryer
HER SEA GOING QUALITIES WERE EXCELLENT AND WOULD HAVE AMPLY SUFFICED FOR A CIRCUMNAVIGATION OF THE GLOBE	jess	female	hairdryer
IF SHE COULD ONLY SEE PHRONSIE FOR JUST ONE MOMENT	jess	female	street
IT IS MANIFEST THAT MAN IS NOW SUBJECT TO MUCH VARIABILITY	kallmekris	female	booth
IN SHORT HE BECOMES A PROMINENT FIGURE IN LONDON SOCIETY AND IF HE IS NOT CAREFUL SOMEBODY WILL SAY SO	kallmekris	female	cafe
EVEN SO I HAD JUST RETURNED FROM AN ARDUOUS JOURNEY EXHAUSTED AND BADLY NEEDING A REST	kallmekris	female	cafe
FOR A FULL HOUR HE HAD PACED UP AND DOWN WAITING BUT HE COULD WAIT NO LONGER	kallmekris	female	cafe
IT WAS THE SCARLET LETTER IN ANOTHER FORM THE SCARLET LETTER ENDOWED WITH LIFE	kallmekris	female	cafe
UPON THIS MADAME DEIGNED TO TURN HER EYES LANGUISHINGLY TOWARDS THE COMTE OBSERVING	kallmekris	female	street
THEY WERE CERTAINLY NO NEARER THE SOLUTION OF THEIR PROBLEM	kallmekris	female	waterfall
IT IS SIXTEEN YEARS SINCE JOHN BERGSON DIED	kallmekris	female	waterfall
SOMEONE ELSE TOLD A STORY NOT PARTICULARLY EFFECTIVE WHICH I SAW HE WAS NOT FOLLOWING	kallmekris	female	waterfall
CONCORD RETURNED TO ITS PLACE AMIDST THE TENTS	kallmekris	female	waterfall
THE SALIENT FEATURES OF THIS DEVELOPMENT OF DOMESTIC SERVICE HAVE ALREADY BEEN INDICATED	myra	female	booth
THE PARIS PLANT LIKE THAT AT THE CRYSTAL PALACE WAS A TEMPORARY EXHIBIT	nadine	female	inside
GENTLEMEN TO YOUR POSTS WHEREUPON SAINT AIGNAN AND VILLEROY TOOK THEIR LEAVE	nadine	female	loud
BUT THE MORE FORGETFULNESS HAD THEN PREVAILED THE MORE POWERFUL WAS THE FORCE OF REMEMBRANCE WHEN SHE AWOKE	nadine	female	outdoor
YOU WILL FIND ME CONTINUALLY SPEAKING OF FOUR MEN TITIAN HOLBEIN TURNER AND TINTORET IN ALMOST THE SAME TERMS	nadine	female	outdoor
ALSO A POPULAR CONTRIVANCE WHEREBY LOVE MAKING MAY BE SUSPENDED BUT NOT STOPPED DURING THE PICNIC SEASON	nadine	female	outdoor
WELL MOTHER SAID THE YOUNG STUDENT LOOKING UP WITH A SHADE OF IMPATIENCE	nadine	female	outdoor
LENGTH OF SERVICE FOURTEEN YEARS THREE MONTHS AND FIVE DAYS	nadine	female	restaurant
MISS LAKE DECLINED THE CARRIAGE TO NIGHT	nadine	female	restaurant
WHAT IS YOUR COUNTRY OLAF HAVE YOU ALWAYS BEEN A THRALL THE THRALL'S EYES FLASHED	sawyer	male	crowdstreet
HE BEGAN A CONFUSED COMPLAINT AGAINST THE WIZARD WHO HAD VANISHED BEHIND THE CURTAIN ON THE LEFT	sawyer	male	crowdstreet
YES DEAD THESE FOUR YEARS AN A GOOD JOB FOR HER TOO	sawyer	male	inside
IN EVERY WAY THEY SOUGHT TO UNDERMINE THE AUTHORITY OF SAINT PAUL	sawyer	male	restaurant
YET THESE THOUGHTS AFFECTED HESTER PRYNNE LESS WITH HOPE THAN APPREHENSION	sawyer	male	restaurant

Training from noisy untranscribed data

text	speaker	gender	noise	reference	Whisper transcripts from original training data	Whisper transcripts with 35% of training data augmented with noises (SNR=0db)	HuBERT units from original training data	HuBERT units with 35% of training data augmented with noises (SNR=0db)
AND HOW ODD THE DIRECTIONS WILL LOOK	kallmekris	female	cafe
THE PARIS PLANT LIKE THAT AT THE CRYSTAL PALACE WAS A TEMPORARY EXHIBIT	nadine	female	inside
THE DEMOCRATIC COMMITTEE FIGURED OUT A WAY TO DO THIS	myra	female	street
I DID NOT WRONG MYSELF SO BUT I PLACED A WRONG ON THEE	jess	female	hairdryer
IT WAS THE SCARLET LETTER IN ANOTHER FORM THE SCARLET LETTER ENDOWED WITH LIFE	kallmekris	female	cafe
MY FRIEND'S TEMPER HAD NOT IMPROVED SINCE HE HAD BEEN DEPRIVED OF THE CONGENIAL SURROUNDINGS OF BAKER STREET	kallmekris	female	booth
IN EVERY WAY THEY SOUGHT TO UNDERMINE THE AUTHORITY OF SAINT PAUL	sawyer	male	restaurant
LENGTH OF SERVICE FOURTEEN YEARS THREE MONTHS AND FIVE DAYS	nadine	female	restaurant
THE STOP AT QUEENSTOWN THE TEDIOUS PASSAGE UP THE MERSEY WERE THINGS THAT HE NOTED DIMLY THROUGH HIS GROWING IMPATIENCE	casey	male	booth
NATURE OF THE EFFECT PRODUCED BY EARLY IMPRESSIONS	nadine	female	booth
IT IS SIXTEEN YEARS SINCE JOHN BERGSON DIED	kallmekris	female	waterfall
WHY ARE WE TO BE DENIED EACH OTHER'S SOCIETY	nadine	female	outdoor
MISS LAKE DECLINED THE CARRIAGE TO NIGHT	jess	female	booth
THEY WERE CERTAINLY NO NEARER THE SOLUTION OF THEIR PROBLEM	kallmekris	female	waterfall
STUFF IT INTO YOU HIS BELLY COUNSELLED HIM	kallmekris	female	wind
CONCORD RETURNED TO ITS PLACE AMIDST THE TENTS	kallmekris	female	waterfall
IF SHE COULD ONLY SEE PHRONSIE FOR JUST ONE MOMENT	nadine	female	restaurant
ON ARRIVING AT HOME AT MY OWN RESIDENCE I FOUND THAT OUR SALON WAS FILLED WITH A BRILLIANT COMPANY	kallmekris	female	cafe
JOHN TAYLOR WHO HAD SUPPORTED HER THROUGH COLLEGE WAS INTERESTED IN COTTON	casey	male	booth
FOR A FULL HOUR HE HAD PACED UP AND DOWN WAITING BUT HE COULD WAIT NO LONGER	kallmekris	female	cafe
WHAT IS YOUR COUNTRY OLAF HAVE YOU ALWAYS BEEN A THRALL THE THRALL'S EYES FLASHED	sawyer	male	crowdstreet
IT ENGENDERS A WHOLE WORLD LA PEGRE FOR WHICH READ THEFT AND A HELL LA PEGRENNE FOR WHICH READ HUNGER	nadine	female	outdoor
SATURDAY AUGUST FIFTEENTH THE SEA UNBROKEN ALL ROUND NO LAND IN SIGHT	kallmekris	female	cafe
HE HAD HIS HAND UPON LAKE'S SHOULDER	kallmekris	female	cafe
IT WAS THE INDIAN WHOSE DARK SILHOUETTE APPEARED SUDDENLY UPON HIS BLIND	nadine	female	background
IN A GENERAL WAY THOUGH NOT WHOLLY NOR CONSISTENTLY THESE TWO GROUPS COINCIDE	sawyer	male	booth
HER SEA GOING QUALITIES WERE EXCELLENT AND WOULD HAVE AMPLY SUFFICED FOR A CIRCUMNAVIGATION OF THE GLOBE	jess	female	hairdryer
HE KNEW THE SILVER FLEECE HIS AND ZORA'S MUST BE RUINED	kallmekris	female	cafe
HE BEGAN A CONFUSED COMPLAINT AGAINST THE WIZARD WHO HAD VANISHED BEHIND THE CURTAIN ON THE LEFT	sawyer	male	crowdstreet
YES DEAD THESE FOUR YEARS AN A GOOD JOB FOR HER TOO	sawyer	male	inside
I CAN'T SEE IT IN THAT LIGHT SAID THE OLD LAWYER	jess	female	booth
BUT CAP'N BILL MADE NO SUCH ATTEMPT KNOWING IT WOULD BE USELESS	myra	female	street
THE PITY THAT WE MUST COME AND GO	nadine	female	restaurant
EVEN SO I HAD JUST RETURNED FROM AN ARDUOUS JOURNEY EXHAUSTED AND BADLY NEEDING A REST	kallmekris	female	cafe
ALSO A POPULAR CONTRIVANCE WHEREBY LOVE MAKING MAY BE SUSPENDED BUT NOT STOPPED DURING THE PICNIC SEASON	nadine	female	outdoor
BY QUICK MARCHES THROUGH THESE INACCESSIBLE MOUNTAINS THAT GENERAL FREED HIMSELF FROM THE SUPERIOR FORCES OF THE COVENANTERS	nadine	female	restaurant
MISS LAKE DECLINED THE CARRIAGE TO NIGHT	nadine	female	restaurant
THIS WAS WHAT DID THE MISCHIEF SO FAR AS THE RUNNING AWAY WAS CONCERNED	myra	female	street
CONCORD RETURNED TO ITS PLACE AMIDST THE TENTS	adi	male	booth
HE HAD HIS HAND UPON LAKE'S SHOULDER	nadine	female	outdoor

Broader Impacts

As voice cloning and conversion technologies progress and grow in quality so does the risk of their misuse for malicious purposes. That is why special attention should be payed to developing mitigation strategies. One such strategy is to control the use of voice cloning systems and ensure inability to use them in open-domain speech generation without user consent. Another strategy is the development of fake audio detection systems. In our case the discriminator jointly trained with our DINO-VITS model can serve as the first step to such a solution. For its evaluation we took real speech recordings from the full CHiME3 dataset and then measured its accuracy separately on the outputs of each of the models mentioned in this paper. The F-score was above 95% for all such evalua tions. This shows that GAN discriminators seem to show a potential for efficiently detecting synthesized speech by models that share the core generator architecture with the one they were trained with (HiFi-GAN decoder in our case). However a more extensive study of this issue should be in order so as to fully ensure minimization of risk of harm to the end user related to the said technologies.