Wednesday, July 19th 2023
OpenAI Degrades GPT-4 Performance While GPT-3.5 Gets Better
When OpenAI announced its GPT-4 model, it first became a part of ChatGPT, behind the paywall for premium users. The GPT-4 is the latest installment in the Generative Pretrained Transformer (GPT) Large Language Models (LLMs). The GPT-4 aims to be a more capable version than the GPT-3.5 that powered ChatGPT at first, which was capable once it launched. However, it seems like the performance of GPT-4 has been steadily dropping since its introduction. Many users noted the regression, and today we have researchers from Stanford University and UC Berkeley, who benchmarked the GPT-4 performance in March 2023, and the model's performance in June 2023 in tasks like solving math problems, visual reasoning, code generation, and answering sensitive questions.
The results? The paper shows that GPT-4 performance has been significantly degraded in all the tasks. This could be attributed to improving stability, lowering the massive compute demand, and much more. What is unexpected, GPT-3.5 experienced a significant uplift in the same period. Below, you can see the examples that were benchmarked by the researchers, which also compare GTP-4 and GPT-3.5 performance in all cases.
Source:
Research Paper (arXiv)
The results? The paper shows that GPT-4 performance has been significantly degraded in all the tasks. This could be attributed to improving stability, lowering the massive compute demand, and much more. What is unexpected, GPT-3.5 experienced a significant uplift in the same period. Below, you can see the examples that were benchmarked by the researchers, which also compare GTP-4 and GPT-3.5 performance in all cases.
9 Comments on OpenAI Degrades GPT-4 Performance While GPT-3.5 Gets Better
With Microsoft buying their way in, I wouldn't be surprised if the whole thing collapsed. The amount of effort and money they put into Bing without it even being able to find material on their own home site is beyond shocking. It speaks volumes.
I'm reminded of the talk given by the two guys who made The Social Dilemma (a documentary about how harmful social media has been to the public), where they talked about the dangers of AI. Contrary to their good take in The Social Dilemma, they did a complete 180 bootlicker turnaround; they said the solution to all the dangers that AI represents is to - get this - centralize control of it to a tiny handful of corporations and government agencies. Surely centralizing something as powerful and influential (and increasingly powerful and influential) as AI won't be horrible for the public, right? I'm sure we can trust western governments and corporations like Amazon and Google to do what's right. Definitely.
But if you seek to put me down, you have to do a little better. A good place to start, is to stop making assumptions.
I'm not anti-Microsoft. I have been running Windows since 3.1 and even have the original 7 3.5" installation disks in my possession. I still run Windows at home and at work. Servers and sensors are running nix, but even the hardcore Linux-guys I work with are happy with Windows, WSL and VSCode.
I made fun of Bing and the department responsible for it. Not MS as a whole.
I consider Bill Gates to be a great man. A lot of people are alive today because of him, and some of the projects he is working on has the potential to make a huge positive impacts for us all.
All of that aside, while I agree Bill Gates is a good guy. He hasnt run MS in over a decade atleast.
Google gets worse at the same rate the internet accumulates rubbish I guess. Shit in = shit out. I use Google Scholar whenever possible. I hadn´t even heard of it a year ago.. imagine my surprise when I tried it. It might not solve my problem right away and its pretty dense reading. But its worth it. No ads, precise wording, peer reviews and no biases. Well, at least not compared to the rest of the web. Stuck in tutorial hell is not fun at all. Its the enemy of motivation.