- cross-posted to:
- [email protected]
- cross-posted to:
- [email protected]
Over just a few months, ChatGPT went from correctly answering a simple math problem 98% of the time to just 2%, study finds. Researchers found wild fluctuations—called drift—in the technology’s abi…::ChatGPT went from answering a simple math correctly 98% of the time to just 2%, over the course of a few months.
For simple language models sure but we’re talking about chatGPT here. OpenAI has some pretty bold claims…
https://towardsdatascience.com/gpt-4-will-have-100-trillion-parameters-500x-the-size-of-gpt-3-582b98d82253
100 trillion bites is 100 terrabytes and if you have any amount of actual data in those parameters then the size of the data could easily get into the petabyte range.
They list the currently available models that users of their API can select here:
https://platform.openai.com/docs/models/overview
They even say that while the main models are being continuously updated (read: re-trained) there are snapshots of previous models that will remain static.
So yes, they are storing and snapshotting the models and they have many different models available with which to perform inference at the same time.
Each parameter corresponds to a single number, so if it’s using 16 bit numbers then that’s 200 TB. They might be using 32 bit numbers (400 TB) but wouldn’t be using anything larger.