PocketTTS non-English languages Cloned voice

[iami2]

Do the non-English language in Pocket-TTS support synthize with cloned voice? I am able to run the code below and generate .english TTS successfully but when generating any of the other languages, I get an audio file with just noise.

                let manager = PocketTtsManager(language: .spanish, directory: modelDirectory, placement: .gpu)
                try await manager.initialize()

                let voiceURL = Bundle.main.url(forResource: "sample voice", withExtension: "m4a")!
                let voice = try await manager.cloneVoice(from: voiceURL)
                
                let dataURL = FileManager.default.temporaryDirectory
                    .appendingPathComponent(UUID().uuidString)
                    .appendingPathExtension("wav")
                let data = try await manager.synthesize(text: "Hola, Que Tal?  Como andas?  Todo bien?", voiceData: voice, temperature: 0.3)//, )
                try data.write(to: dataURL)

[iami2]

FYI, I tried using the built-in voices and they exhibit the same behavior. The only voice that works is "alba"

            let manager = PocketTtsManager(
                defaultVoice: "lola",
                language: .spanish,
                precision: .fp16,
                placement: .gpu
            )
            
            try await manager.initialize()
            
            let data = try await manager.synthesize(
                text: "Hola. ¿Qué tal? ¿Cómo andas? ¿Todo bien?",
                voice: "lola",
                temperature: 0.3,
                deEss: true
            )

[Alex-Wengg]

#797 (comment) i might have a fix for it could you give it a try

[iami2]

Thanks for looking into this bug.

I have tested the following:

1. Generating TTS with built-in voices. This is confirmed fixed.

2. Generating TTS with cloned voice.
   2.1. It does not always work. I find myself adjusting the sample voice audio duration, temperature, deEss to make it work. When it works in Spanish, it does not work for Portuguese.
   2.2. For the instances that do work, some of them cloned voices do not resemble the source voice.

           let dataURL = FileManager.default.temporaryDirectory
               .appendingPathComponent(UUID().uuidString)
               .appendingPathExtension("wav")
   
           
           let manager = PocketTtsManager(language: .spanish24L)
           
           try await manager.initialize()
           let voiceURL = Bundle.main.url(forResource: "sample", withExtension: "wav")!
           
           let voice = try await manager.cloneVoice(from: voiceURL)
           
           let data = try await manager.synthesize(text: "Hola, ¿qué tal? ¿Cómo andas? ¿Todo bien?", voiceData: voice)
           
           try data.write(to: dataURL)
   

[Alex-Wengg]

Do you know if this issue is upstream of the base PyTorch model?

[iami2]

base PyTorch model works fine. Refer to attached generated audio files. I re-tested the built-in voice "Lola" and was able to reproduce the noise as well. Here's my test code.

spanish24L.zip

            let manager = PocketTtsManager(language: .spanish24L)
            
            try await manager.initialize()
            let voiceURL = Bundle.main.url(forResource: "reference_voice", withExtension: "wav")!
            
            let voice = try await manager.cloneVoice(from: voiceURL)
            
            for iter in 0..<5 {
                let dataURL = FileManager.default.temporaryDirectory
                    .appendingPathComponent(UUID().uuidString)
                    .appendingPathExtension("wav")
                let data = try await manager.synthesize(text: "Hola, ¿qué tal? ¿Cómo andas? ¿Todo bien?", voiceData: voice, temperature: 0.7, deEss: true)//, )
                /*
                let data = try await manager.synthesize(
                    text: "Hola, ¿qué tal? ¿Cómo andas?",
                    voice: "lola",
                    temperature: 0.15
                ) */
                
                try data.write(to: dataURL)
            }