On Thursday, OpenAI took a significant step by releasing what essentially becomes a $200-a-month chatbot. This move has sparked a wave of curiosity and uncertainty within the AI community. The company's new ChatGPT Pro plan provides access to "o1 pro mode," which is claimed by OpenAI to utilize more compute for delivering the best answers to the most challenging questions. It is a souped-up version of OpenAI's o1 reasoning model, aiming to offer more reliable and comprehensive responses related to science, math, and coding.

Unraveling the Mystery of OpenAI's $200-Monthly Chatbot

Initial Reactions and User Queries

Almost immediately after its launch, people began to explore the capabilities of this new chatbot. Many asked it to create various visual elements like unicorns and design unique computer concepts. For instance, one user asked ChatGPT o1 Pro Mode to create an SVG of a unicorn, and another designed a "crab-based" computer. There were also those who waxed poetic on the meaning of life, showing the diverse range of inquiries. However, not everyone was convinced of its superiority. British computer scientist Simon Willison questioned if OpenAI had shared concrete examples of prompts that failed in the regular o1 but succeeded in o1-pro. It's a valid concern given the high price tag.

Performance Benchmarks and Failures

OpenAI's internal benchmarks indicate that o1 pro mode only performs slightly better than the standard o1 on coding and math problems. Even in stricter evaluations where the model had to get the answer right four out of four times, the improvements were not dramatic. For example, in handling Sudoku and an optical illusion joke, both o1 and o1-pro faced difficulties. This shows that despite the increased cost, the performance gains are not as substantial as one might expect.

Intended Audience and Market Fit

The question of who this ChatGPT Pro is for remains. While some experts in specialized fields might find value in it, like the handful of medical researchers at leading institutions who will get free access, it's unclear if there are enough people willing to pay $200 a month for such services. As Ameet Talwalkar, an associate professor of machine learning at Carnegie Mellon and a venture partner at Amplify Partners, pointed out, it seems like a big risk to raise the price tenfold. UCLA computer scientist Guy Van den Broeck also expressed doubts about the price point and whether pricey reasoning models will become the norm.

Limitations and the "Intelligence Overhang"

There are fundamental limitations to the simple, text-based interface of ChatGPT. As Bob McGrew, OpenAI's former chief research officer, noted, users may not know how to get the most value from the "extra intelligence" offered by o1 pro mode. Even with its touted capabilities, ChatGPT Pro is not a slam dunk as evidenced by the early reception on social media. It's a complex issue that requires careful consideration of both the benefits and drawbacks.