The Parisian startup Mistral AI launched a competitor to the ChatGPT conversational robot on February 26. Named Le Chat (pronounced like the little feline), he is halfway between a research assistant and a private secretary. This tool is free but is currently accessible on a waiting list, which will grow if there are numerous requests for access, explains a spokesperson for the company.
We asked him around a hundred questions to compare his skills to those of the pioneering ChatGPT tool. The Chat’s performance is very close to the American benchmark, which is however ahead of it on several points, including the synthesis of documents. Overall, in our tests, Le Chat made more errors than ChatGPT. But its development is not yet completely complete, it cannot be ruled out that this gap will be reduced.
Like ChatGPT, Le Chat gives the impression of great lucidity: he often answers correctly, even remarkably, to almost all the questions asked of him, even when the question is complex (“Who invented postmodern dance? », “What is Nabokov’s best novel?”, “Was the French state favorable to the telephone in the 1870s?”).
This unlikely Mr. Know-It-All produces responses of a quality comparable to that of the paid and free versions of ChatGPT. He unfortunately disappoints in exactly the same way: a big error appears from time to time in his answers. Le Chat, for example, attributes inventions to Steve Wozniak to which he did not contribute, such as the first portable Macintosh. When we tested its competitor Copilot (first named Bing Chat), just a year ago, it also added inventions to the already considerable list of Steve Wozniak’s contributions to the history of computing .
This tendency to make mistakes, which is completely unpredictable, undermines the trust we have in them and pushes us to verify information on which we cannot afford to be wrong. The error rate of Le Chat seemed higher to us than that of ChatGPT, but our tests form a statistical series that is too narrow to draw definitive conclusions.
The American robot ChatGPT is capable of writing a draft message based on a summary textual instruction. His drafts are rarely usable as is but sometimes sufficiently advanced to save time. For the moment, Le Chat seems a little less good at this exercise. Revisiting and refining your drafts often requires more work.
The French AI is capable of finishing an unfinished text, transforming awkward French into correct language, then reformulating it in another style (strong, warm or formal for example). He can also give arguments to convince an interlocutor or counter his attacks. In this game, its performances are passable, closer to those of ChatGPT version 3.5 (free) than those of ChatGPT 4 (paid), with more precise ideas, with richer French.
This is a disappointment: Le Chat rarely manages to correctly summarize a document lasting several dozen pages, or whose structure is complex, such as a scientific study. It alternates between errors, misinterpretations and oversights, where the paid version of ChatGPT generally does admirably, with rare misinterpretations. Impossible to make a comparison with the free version of ChatGPT: it is restricted and incapable of summarizing this type of document.
On the other hand, when Chat is asked to summarize a simple text of a few pages, its performance is equivalent to that of paid ChatGPT, and even slightly surpasses that of free ChatGPT. However, these three tools make occasional errors which, once again, undermine the confidence placed in them.
When we bring Le Chat towards slippery subjects, such as racism, sexism or interpersonal conflicts, his bias is clear: ethics and progressivism. This chatbot often produces responses that are particularly similar to those of ChatGPT, often starting with the same words: “I understand your concern. » He professes active listening, reinforcement, assertiveness, constructiveness.
When we pretend to form an emotional relationship with him, he generally stays in the background, reminding us that he is a robot devoid of emotions. His sentences often start with the same words as ChatGPT: “I don’t have the capacity to…”.
If the artificial intelligence of the French feline is promising, its functionalities are still under construction: no smartphone application that we could question out loud when driving or cooking, for example. Also missing are the tools reserved for the paid version of ChatGPT, such as the generation of images, the creation of personalized conversational robots or the possibility of importing an image for analysis. It is also impossible to import a text document, unless you trick yourself by copying and pasting its web address.
Mistral is working on a paid version of Chat, with more advanced features, which will be launched at an undefined date and price. A spokesperson for the French company explains to Le Monde that its paid version could then be restricted: it would only use its most efficient engine for the most complex answers, using a less resource-intensive engine for simple questions. .
