AI Weekly Digest #13: OpenAI’s 2023-2024 Roadmap

AI safety, LLM scaling laws, Nvidia’s announcements, Open Source Falcon Model becomes royalties free.

AI Weekly Digest #13: OpenAI’s 2023-2024 Roadmap

Welcome to this new edition of our AI newsletter, where we bring you the latest updates on artificial intelligence!

  • The Main Headlines: AI safety, LLM scaling laws, Nvidia’s announcements, Open Source Falcon Model becomes royalties free.

  • Bonus: Test open source models online - Free colab notebook.

Main Headlines

AI Safety Concerns: The risk of extinction from AI is being discussed alongside other societal-scale risks such as pandemics and nuclear war. Signed by Bill Gates, Sam Altman, Geoffrey Hinton, and many others, the document is calling for mitigation of these risks to be a global priority. The AI community is now torn between those who think that AI safety needs to be tackled urgently, and those who believe that AI is nothing more than smarter autocomplete and argue against giving it such attention and focus for now.

LLM Scaling Laws: Despite claims that the era of giant AI models is over, OpenAI's data suggests that scaling laws for model performance continue to hold. What does it mean? mostly two things:

  • Performance vs. Compute Demand: Larger models still results in better performance, so we still have room for “brut force” model improvements (aka, GPT4++). However, it simultaneously implies a substantial computational load, i.e., longuer training times and decreased inference speed. This is particularly critical as these models find more widespread usage in real-time, operational settings.

  • Future Size Growth Expectations: Given the necessity of balancing model sizes with existing computational power, the rate of growth in model sizes is likely to decelerate. We expect to see more moderate expansions, perhaps two to three times larger, as opposed to the recent leaps of ten to a hundred times.

OpenAI’s Roadmap Focusing on GPT-4 Scaling and Larger Contexts (source):

  • 2023: cheaper and faster GPT-4. As a matter of fact, GPT-4 proved to be too slow and expensive to be used at a reasonable scale. Using it at scale in production would require a significant drop in API costs and in latency.

  • 2023: longer context windows (~100k tokens, and up to 1M tokens)! This will enable scenarios previously impossible to handle with a 4k-8k contexts (e.g., long documents or multiple documents’ simultaneous analysis).

  • 2024: releasing multimodal models will be the focus of 2024 (e.g., GPT4 with both text & images).

  • Note: OpenAI is currently facing a GPU shortage, impacting the speed and reliability of their API. This shortage is also delaying the rollout of longer 32k context and finetuning API. This also explains Microsoft’s investments to design its own AI chips.

Nvidia is on fire these days!:

  • Generative AI used for real-time interaction in video games with NPCs (non playable characters)

  • Nvidia unveiled the DGX H100 AI supercomputer. It will be available for large corporations (Microsoft, Google and so on). “Servers equipped with H100 NVL GPUs increase GPT-175B model performance [aka GPT-3] up to 12X over NVIDIA DGX™ A100 systems while maintaining low latency in power-constrained data center environments.”

  • Nvidia's market cap briefly surpassed $1 trillion becoming the 6th most valued company!

  • Nvidia released research on their Neurolangelo model, which can convert 2D video into 3D objects and elements, potentially useful in game development and video production.

Falcon 40B Release: Falcon 40B, the leading Open Source LLM mode, has been released with an Apache 2.0 license, allowing for open use. Very exciting news for the AI open source community!

Bonus!

Wondering how to test the latest Open Source Models? Bookmark these two links below!

Shout-out to 1littleCoder for this video. It will guide you through the colab notebook!

Enjoy!

That’s it for today! If you made it this far, I’d appreciate a quick feedback 😋! I know there’s room for improvement! So don’t hesitate to share with me the things you liked and those that you didn’t.

Have a great Sunday and may AI always be on your side!