Good morning. It’s Monday, May 6th.

Did you know: On this day in 1998, Apple Computer unveiled the first iMac?

In today’s email:

• Most LLMs Were “Overfitted” on AI Benchmarks

• Prometheus 2 Model

• X Stories by Grok

• 5 New AI Tools

• Latest AI Research Papers

• AI Creates Comics

You read. We listen. Feel free to reply to this email anytime if you’re looking for a specific AI tool, have a story to recommend, or want to share a project you’re working on. I’m always looking for great things to share with this audience of over 54,000 readers.

This edition of the newsletter is brought to you by….

You.

Ads suck.

(Sometimes they’re relevant, but those ones don’t count.)

If you find this publication interesting, you can sign up here to become a premium member for just $5/mo, and receive an ad-free version of the newsletter and become an official supporter of this project.

Thank you for your readership. I love writing this newsletter and will continue to do so until we reach a technological singularity that inhales us all.

Scale AI Just Released New Research Uncovering Significant ‘overfitting’ Of Certain LLMs On Popular AI Benchmarks

Scale AI introduces GSM1k, a benchmark for measuring reasoning in large language models (LLMs). It addresses the opacity of model decision-making by comparing performance on novel and known data. Scale uncovered a problem with a lot of LLMs being trained directly on the benchmarking material, like cramming for a test without legitimately attempting to understand the core concepts of the benchmark’s material.

Through systematic comparison, researchers identify varying levels of overfitting and reasoning across models. This research is vital for sectors like healthcare and finance, where transparent models are imperative. By distinguishing between genuine reasoning and memorization, GSM1k guides interpretability methods and advances machine learning. Read more.

Open-source Model Prometheus 2 Can Evaluate Other Language Models Nearly As Well As GPT-4

Prometheus 2, an open-source language model, has been optimized to evaluate other models, rivaling commercial counterparts like GPT-4. This enhancement enables researchers and developers to objectively measure language model performance and receive detailed feedback for targeted improvements, fostering continuous enhancement in quality and reliability.

Developed by Seungone Kim's team at KAIST AI, Prometheus 2 addresses the lack of transparency and affordability associated with proprietary models, providing independent and transparent evaluations for all. By mastering direct evaluation and pairwise comparison methods, Prometheus 2 supports diverse evaluation criteria, facilitating the optimization of language models for specific applications such as medical advice chatbots. Read more.

X Launches Stories, Delivering News Summarized By Grok AI

X (formerly Twitter) is shaking things up with Grok Stories, a new feature powered by Elon Musk's AI chatbot Grok. Grok curates personalized summaries of trending stories in the Explore section, offering Premium subscribers bite-sized insights alongside each hot topic on the For You tab. Unlike Twitter's past, limited attempts, Grok dives into all major news using AI, providing a comprehensive overview.

The Premium subscription, starting at $8 per month, grants access to Grok, making it a selling point for X. However, Grok’s reliance on X conversations for news summaries raises concerns about the content’s accuracy. Despite controversy, Grok’s Stories represent a novel approach to news delivery, potentially impacting traditional news consumption habits or your friendly neighborhood AI newsletters. Read more.

🖇️ Etcetera: Stories you may have missed

> AI engineers report burnout and rushed rollouts as 'rat race' to stay competitive hits tech industry

> OpenAI prepares its AI safety infrastructure for "advanced AI"

> Microsoft buys cloud growth in Southeast Asia for nearly $4 billion

> ScrapeGraphAI: A Web Scraping Python Library that Uses LLMs to Create Scraping Pipelines for Websites, Documents, and XML Files

> Refined AI approach improves noninvasive brain-computer interface performance

> SK Hynix says high-end AI memory chips almost sold out through 2025

> Here's the 'true solution' to the problem of audio deepfakes, according to the CEO of ElevenLabs

Pressmaster.ai is an AI-powered PR software streamlining public relations with journalist connections and simplified article creation, eliminating traditional hassles and costs.

Creatoor AI simplifies social media video creation with AI avatars. Generate high-quality Instagram Reels from text prompts, subtitles, language dubbing, and AI-generated B-Roll.

PaddleBoat hones sales pitches with AI roleplays. Practice cold-calls with customizable AI buyers, tailored to your business context for effective training.

Assista AI acts as the central nervous system for productivity, integrating AI across apps. Manage tasks effortlessly with voice or text commands, streamlining workflows.

Trag is an AI-powered code review tool for engineering teams. Customizable rules in natural language, automatic bug detection, and autofix suggestions streamline review processes.

📄 Customizing Text-to-Image Models with a Single Image Pair

📄 LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report ​​

📄 StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation

📄 WildChat: 1M ChatGPT Interaction Logs in the Wild

📄 Octopus v4: Graph of language models 

AI Creates Comics

Thank you for reading today’s edition.

Your feedback is valuable. Respond to this email and tell us how you think we could add more value to this newsletter.

Interested in reaching smart readers like you? To become an AI Breakfast sponsor, apply here.