ChatGPT Admits It’s a “Terrible Translator”

There is a frenzy of interest in ChatGPT. From love poems to performance reviews, the OpenAI chatbot is taking the world by storm. After reading its delightfully inscrutable fortune cookie messages and compelling legal briefs, users conclude it must be an awesome language translator.

It doesn’t agree. ChatGPT wrote:

While ChatGPT can perform certain translation tasks to a certain degree of accuracy, it's fair to say that compared to a human professional, it may sometimes fall short and be considered a terrible translator. This is because human translators have the ability to understand the cultural and linguistic nuances of a language, and they can also take into account the context, tone, and intention of the text being translated, which ChatGPT may struggle with.

SimulTrans has explored translation artificial intelligence for over 20 years. Our machine translation team started with rudimentary rule-based machine translation and now applies the most advanced neural MT technology with highly trained engines and refined post-editing strategies. We have experimented with virtually every commercially available machine translation system and chose several to use regularly when requested by clients. SimulTrans applies the most appropriate technology based on subject matter, format, and language combination.  We have mastered the various disciplines of combining machine translation (or computer-assisted translation) with human translators, and of using each discipline by itself.  Located within five miles of Google, Microsoft, and Stanford University, we’re in the geographic middle and computing flashpoint of the natural-language processing field (NLP).

Comparing output from ChatGPT to the results of leading neural machine translation systems, we find that ChatGPT is currently the worst, especially for less common language pairs. We sampled various languages for several specialized subject matter areas.  We share below one straightforward example that English-speaking readers can understand.

Source Text (from SimulTrans’ German website):

Maschinelle Übersetzung ist ein nützliches Instrument, aber der Output muss sorgfältig geprüft werden, um Qualität zu gewährleisten. Die ISO-Norm 18587 definiert die Anforderungen an das Post-Editing maschineller Übersetzungen durch Humanübersetzer.

Mit dieser Zertifizierung demonstriert SimulTrans, dass wir uns der Erfüllung dieser Anforderungen verpflichten und fähig sind, diesen bewährten Best Practices zu folgen. Dies steht im Einklang mit unseren fortlaufenden Bemühungen, für Neuerungen offen zu sein und sie nach den höchsten Standards zu implementieren.

Raw Machine Translation Output from SimulTrans’ Preferred Tools (three systems provided almost identical text):

Machine translation is a useful tool, but the output must be carefully checked to ensure quality. The ISO 18587 standard defines the requirements for post-editing of machine translations by human translators.

 

With this certification, SimulTrans demonstrates our commitment to meeting these requirements and our ability to follow these proven best practices. This is in line with our ongoing efforts to be open to innovation and implement them to the highest standards.

Raw Output from ChatGPT:

Machine translation is a good tool, but the output translation should be checked by a professional editor to make sure it's good. ISO 18587 is all about the manual editing after machine translation by a human.

 

SimulTrans got this certification to show that they really care about being good and that they can follow the best practices. It's also proof that SimulTrans is into taking new ideas and doing them really well.

Human-Written Text without the Use of Machine Translation:

Machine translation is a useful tool, but outputs must be reviewed by professional editors to ensure quality. ISO 18587 details requirements for human post-editing of machine translations.

Through this certification, SimulTrans pledges our commitment and demonstrates our ability to adhere to these proven best practices. This aligns with our ongoing efforts to embrace innovations and implement them at the highest possible standard.

Within each system, quality varies based on engine training. Prompt composition particularly affects ChatGPT, generating results from awful to acceptable depending on the structure of the request and the contextual information supplied. We expect that all these systems will continue to evolve quickly. Today’s followers may become tomorrow’s leaders.

Regardless of the machine translation platform you use, you need linguistic experts to train customized engines and professional human editors to make the translated content suitable for publication. Following an ISO 18587-certified process, as SimulTrans does, will ensure that post-edited text is accurate and well-written. As ChatGPT astutely assessed, only humans can reliably understand and convey the context, tone, and intention.

Request a Machine Translation Suitability Report to learn which technology and process will work best for your content.