Final week, OpenAI unveiled Agent, its new instrument that mixes the capabilities of Deep Research and Operator. Operator was OpenAI’s first attempt at a computer-using model, a mannequin that really can open home windows and click on on consumer interface parts. ChatGPT Agent can try this and extra.
Proper now, ChatGPT Agent is simply obtainable for $200/mo Professional tier subscribers and offers for 400 agent interactions per 30 days. When the $20/mo Plus tier beneficial properties entry to Agent, which ought to be immediately, these customers will get 40 interactions per 30 days.
Additionally: Microsoft is saving millions with AI and laying off thousands – where do we go from here?
(Disclosure: Ziff Davis, ZDNET’s guardian firm, filed an April 2025 lawsuit in opposition to OpenAI, alleging it infringed Ziff Davis copyrights in coaching and working its AI techniques.)
I upgraded my plan from Plus to Professional simply so I might take a look at out the brand new Agent mode and report again to you. On this article, I will present you detailed outcomes from eight complete exams.
TL;DR take a look at outcomes
Earlier than we go into the detailed exams, I will begin with some general TL;DR observations.
Check rely: Up to now two days, I used 25 of the obtainable 400 queries, for a complete of virtually 12 hours of hyper-uber-supercomputer use. No surprise this factor prices $200/month.
Additionally: I found 5 AI content detectors that can correctly identify AI text 100% of the time
Almost each question required a follow-on, so when it comes time for Plus customers, do not assume you can provide Agent 40 tasks. Extra probably, you may be giving it 20-25, and utilizing the remainder of your queries to persuade the Agent to comply with instructions.
End result high quality: In all my exams, Agent appeared to grasp the issue. But it surely failed to supply helpful outcomes for many of the exams. That mentioned, the ultimate take a look at produced outcomes that may solely be characterised as amazingly helpful.
Undertaking scale: Agent cannot deal with huge tasks, the form of information evaluation tasks you actually need an AI to have the ability to deal with. It has bother scrolling by means of internet pages. It will probably’t go to websites which have AI or robots.txt restrictions in place. And lengthy processing exceeds session time allocations, even with the tremendous top-of-the-line gold-pressed latinum Professional version.
Presentation high quality: One of many main pitch factors for Agent is its potential to create spreadsheets and shows. It did okay with spreadsheets, however the graphic high quality of the shows was fairly tough. I anticipate this to vary over time, however do not anticipate Agent to make shows you should utilize with out appreciable cleanup.
Accuracy: AIs hallucinate. The OpenAI crew cautioned about utilizing Agent due to the brand new dangers concerned. Whereas I did get again some outcomes that had been correct, Agent additionally got here again with unforced errors, outcomes it might have simply examined and deemed inaccurate. However no such verification or validation occurred. That mentioned, the ultimate take a look at was correct and reveals what this tech can do when it really works.
Connectors: Agent comes with the power to make use of connectors (through API calls) to hyperlink to Gmail, Google Calendar, Google Drive, Outlook, Dropbox, and extra. I didn’t take a look at out the connectors due to how usually Agent hallucinates or does one thing pretty boneheaded. I simply did not really feel comfy sufficient to present Skynet entry to my accounts. At the least, not but.
Limits: I used to be unable to make use of Agent within the MacOS app. I additionally discovered that Agent stalled onerous once I tried to run it in a number of Chrome tabs without delay. For now, you launch an Agent course of and wait. It is not like Codex, the place you’ll be able to launch a bunch of tasks and are available again later and harvest all the outcomes. However since that functionality exists in Codex, I am certain it can present up quickly in Agent.
That ought to offer you a reasonably good overview. Let’s get began wanting on the eight take a look at outcomes. For every outcome, I’ve included a hyperlink to the session recording, so you’ll be able to see the prompts I used, the detailed outcomes, and watch Agent motive its approach by means of the issue.
Additionally, undoubtedly learn to the top. A few of the early outcomes are pretty unhealthy, however the final one knocks it out of the park. And with that, right here we go.
1. Deciding on merchandise on Amazon
- Understanding of the issue: Strong
- Execution: Each good and unhealthy
- Hallucination: Bizarre church reference, pretend Amazon hyperlinks
- Processing time: 20 + 12 minutes
When OpenAI launched ChatGPT Agent, the crew demoed how they used the instrument to buy wedding ceremony garments and a marriage present. That appeared like a reasonably unusual and impractical software for a super-intelligence, particularly since present registries exist and are extensively used.
As a substitute, I gave Agent a buying challenge I had really extensively researched and accomplished just a few months earlier. I am operating Energy-over-Ethernet cables all throughout my yard to improve my safety system. As such, I am creating a number of customized cables. I already know that doing so requires some key instruments: a cutter to slice the cable, a cable finish stripper, a crimper to connect the RJ-45 ends, and a tester to verify that lengthy cable runs work.
Additionally: How a circuit breaker finder helped me map my home’s wiring (and why that matters)
I gave Agent a immediate asking for 3 configurations: a funds toolset, a “money-is-no-object” resolution, and a candy spot resolution. I requested for hyperlinks, product descriptions, and product pictures.
When you give Agent your immediate, it creates a digital desktop. You possibly can watch it conducting its actions, leaping between a desktop view, a textual content view, and code.
The funds resolution turned out to be a win. Agent discovered a single $34 kit with every thing I requested for. It offered a hyperlink, and even reasoning why it selected that resolution. Sadly, the picture it offered was nothing just like the precise equipment.
The mid-tier and top-tier options had been lower than good. Not one of the hyperlinks labored. The mid-tier candy spot resolution did have a product-accurate picture, however and not using a hyperlink, it wasn’t actually useful.
Sadly, the mannequin beneficial would not really exist on Amazon. The truth is, not one of the mid- or upper-tier merchandise exist on Amazon. It appears to be like like Agent did a pile of internet browsing to search out the merchandise, disregarding my directions to go looking solely on Amazon.
It additionally clearly visited different websites, in all probability gathering mannequin names and descriptions.
Then, when it packaged up its closing suggestions, it simply assigned random Amazon hyperlinks to the outline, although these merchandise and people hyperlinks do not appear to exist on Amazon.
I did request it return and check out once more. When it did, after 12 minutes, it offered many of the identical merchandise, though one of many hyperlinks that had failed earlier did, in reality, level to a product on Amazon within the second run.
Additionally: Coding with AI? My top 5 tips for vetting its output – and staying out of trouble
I am unable to go away this part with out declaring one thing simply plain bizarre. As I used to be watching Agent work, it offered this in its desktop view. I do not even wish to know.
You possibly can watch a replay of the entire session here.
2. Evaluating egg costs
- Understanding of the issue: Strong
- Execution: Did what I requested
- Hallucination: My fault for imprecise prompting
- Processing time: 14 minutes
In discussing ChatGPT Agent, OpenAI confirmed a slide that talked about Instacart as one of many examples that the chatbot is comfy working with. Since my household usually makes use of Instacart, I made a decision to set Agent free and see what it might inform me about egg costs at our native shops.
I did not let Agent have entry to my account, however I shared my ZIP code right here in Salem, Oregon. I advised it to “Please go to all of the grocery shops on Instacart and evaluate egg costs.”
Additionally: How to use ChatGPT to write code – and my top trick for debugging what it generates
It did precisely that. You’ve got heard the phrase Rubbish In, Rubbish Out. Properly, that is what occurs once you ask an AI to take a look at “all of the grocery shops.” I ought to have requested it to look in a 5 or 10 mile radius solely. However I did not.
Agent got here again with 21 shops, starting from close by to as much as nearly 47 miles away. It did accomplish what I requested, evaluating egg costs. With out prompting, it determined to rank the eggs by value. This was good. However when it selected the eggs to rank, it did not at all times select the least costly product from every retailer.
For instance, it beneficial the Good & Collect eggs from Goal at $2.99 a dozen, reasonably than the $1.99/dozen Market Pantry egg, additionally from Goal.
You possibly can watch a replay of the entire session here.
3. Making a PowerPoint slide
- Understanding of the issue: Strong
- Execution: Added the right information level
- Hallucination: Was unable to breed graphic high quality
- Processing time: 10 minutes
Subsequent up is a challenge I did early final week. With Congress specializing in Bitcoin, my editor requested me to replace my Bitcoin investment article, the place I have been monitoring the worth of a $50 Bitcoin funding since 2022.
The worth of my holdings went up, which suggests I wanted so as to add a brand new slide. Every slide provides a date worth on the X axis and a price level on the Y axis. From a PowerPoint fiddling standpoint, that meant shifting over the graphics to make room for the brand new worth and, on this case, adjusting the vertical scale to accommodate a considerable rise in worth.
Additionally: The best free AI courses
Once I did it, it took me about 45 minutes. Since OpenAI mentioned that PowerPoint was one among ChatGPT Agent’s strengths, I wished to see if Agent might save me that point sooner or later.
I uploaded my current slide deck minus the final slide I made for the article. Then I requested Agent to create that slide for me.
Because it labored, the desktop view confirmed the terminal interface. You possibly can see how Agent is placing collectively the code to generate a graphic picture.
Here is what that slide ought to have appeared like (observe: foreshadowing).
Here is what Agent gave me.
To be truthful, Agent clearly understood the issue. It moved the prevailing information factors over to the left to make room for the brand new node. It additionally positioned the brand new Bitcoin merchandise correctly in relation to the prevailing ones, and added each value and share change textual content blocks.
Which means Agent learn and understood the context of my PowerPoint deck’s structure. That, in and of itself, could be very spectacular.
Additionally: The best AI for coding in 2025 (and what not to use)
But it surely failed on including extra scale strains and new Y-axis values. It failed on reproducing the fonts. It failed on correctly putting the textual content blocks. And it pushed the whole graphic up and to the left of the slide.
I am guessing the graphics library that Agent makes use of is not actually as much as the duty of constructing nice graphic modifications. That may undoubtedly enhance over time.
You possibly can watch a replay of the entire session here.
4. Article categorization (methodology II)
- Understanding of the issue: Strong
- Execution: Failed on account of exceeding allowable session time
- Hallucination: Gave me again partial outcomes
- Processing time: 8 minutes + 3 minutes + 21 minutes
Every week for the previous two years, I’ve printed a publication that shares with followers the articles I printed right here on ZDNET for the week. Every publication incorporates a title, hyperlink, and article description.
By pointing Agent to my back issue archive, it might have near 300 article summaries to categorize.
Sadly, Agent bumped into quite a lot of issues of its personal making. It was unable to efficiently scroll by means of the article checklist utilizing JavaScript. Once I advised it to make use of the online interface, it began to, nevertheless it reported, “Sadly, I’ve reached the top of the allotted searching periods for this activity, which suggests I am unable to discover additional pages and acquire the extra information at the moment.”
Additionally: Is ChatGPT Plus really worth $20 when the free version offers so many premium features?
Bear in mind, I am paying $200 a month for OpenAI’s greatest plan, and it nonetheless will not give me sufficient time to search for 300 articles. That is a gotcha, proper there. It is also disappointing as a result of a activity like scrolling again by means of an article archive and performing some tabulating is strictly the form of activity you may give to an assistant. If the AI provides up as a result of it takes too lengthy, then we will not actually depend on AI for all of the assistant kind issues. Nobody desires a fussy, choosy assistant.
In any case, Agent did give me again a spreadsheet and a slide primarily based on the restricted information it was capable of finding earlier than my little request exceeded the hourly energy funds for the Metropolis of Las Vegas (or so I think about).
You possibly can watch a replay of the entire session here.
5. Extract remembered textual content from video
- Understanding of the issue: Partial
- Execution: Did not return full transcript on first run, right on second run
- Hallucination: Determined to do what it wished on first run
- Processing time: 2 minutes
I watch a number of YouTube movies to reinforce my studying and analysis. Plus nothing beats an excellent stress-free video about how pavers are made. Whereas it is pretty straightforward to get a transcript of a full video, whether or not instantly from YouTube or using Apple Voice Memos, finding the place in a video a section you wish to discover can take time.
Here is an instance. When OpenAI introduced Agent in a video, CEO Sam Altman mentioned a few of the cautions and warnings about utilizing ChatGPT Agent mode. I did keep in mind they had been close to the top of the video, however I did not wish to spend time sifting by means of to get the precise quotes.
As a substitute, I delegated that project to Agent. On its first run, it discovered the section simply sufficient, however as an alternative of returning a word-for-word transcript, it returned some quotes, interspersed with its personal evaluation.
Additionally: I mapped my iPhone’s Control Button to ChatGPT – here are 5 ways I use it every day
I clarified what I wished and, on its second run, it gave me precisely what I wanted. On this case, although, it wasn’t that my immediate was unclear. I simply needed to insist a second time that I wished a transcript for the AI to do what I requested.
Sadly, this additional evaluate cycle diminished the time-saving worth to me. I nonetheless suppose utilizing Agent was quicker than if I sifted by means of the video myself. However I needed to assemble a second immediate and anticipate a second outcome, all of which took my time.
Nonetheless, it is a useful instrument.
You possibly can watch a replay of the entire session here.
6. Making a pattern evaluation presentation
- Understanding of the issue: Strong
- Execution: Good, aside from slide visible high quality
- Hallucination: An excessive amount of information to verify or deny assertions
- Processing time: 32 minutes
As a part of my job, it is essential to have the ability to sustain with ongoing tech and enterprise traits. As such, I usually spend days in deep dives, coming in control on new matters.
I wished to see if ChatGPT Agent might save me a while by making ready a report and a full presentation on distant work traits. I advised it that the PowerPoint was destined for my administration crew, so it ought to be complete and professional-looking.
It returned an evaluation doc similar to the outcomes we have been getting from ChatGPT deep analysis. The report incorporates numerous assertions and statistical claims, most of which I haven’t got time to analysis for affirmation.
Additionally: ChatGPT can record, transcribe, and analyze your meetings now
A lot of the top-level conclusions are congruent with my understanding of present work-from-home traits. That mentioned, we’re aware of the mannequin’s propensity for hallucination, so I would be very involved about utilizing any of this information professionally with out extra vetting.
Agent did produce a 17-slide PowerPoint deck that was organized fairly effectively. As with earlier experiments, the graphic era high quality was a bit off. The primary slide really appears to be like fairly good.
However later within the deck, it would not look proper. Discover how the next slide has graphics on prime of textual content, and bullets in entrance of bullets on prime of empty bullets.
Within the following slide, not solely is the textual content operating off the top of the web page, however there isn’t any legend. As such, it is not clear what’s represented by crimson and by blue.
As soon as once more, you’ll be able to see how Python is used to assemble the deck.
Agent does a good job, so I am pretty assured that the AI will get higher over time. Programmatic building of slides primarily based on templates isn’t a brand new know-how. I simply do not suppose OpenAI prioritized slide presentation aesthetics as a part of this launch.
You possibly can watch a replay of the entire session here.
7. Vetting a presentation for accuracy
- Understanding of the issue: Strong
- Execution: Good
- Hallucination: Appears full, nevertheless it’s nonetheless from an AI
- Processing time: 11 minutes + 7 minutes
Properly, this was simply plain enjoyable. I made a decision to present the presentation created within the earlier take a look at to a brand new contemporary ChatGPT Agent session and requested it to validate the claims.
Agent concluded, “A number of quantitative claims—particularly these regarding productiveness/innovation impacts, the scale and development of the gig economic system, charges of facet‑gig participation, and the affect of politics and tradition—couldn’t be verified with accessible proof throughout this evaluate.”
Agent offered an in depth evaluation of every assertion. I’ve summarized the outcomes beneath.
- Adoption timeline: Largely confirmed
- International comparability: Confirmed
- Workforce composition: Confirmed
- Migration: Confirmed
- Mobility of distant employees: Confirmed
- Housing & native economies: Confirmed
- Workplace emptiness & environmental impacts: Largely confirmed
- Social connections & wellbeing: Partly confirmed
- Employer attitudes & return‑to‑workplace mandates: Largely confirmed
- Worker preferences & pay cuts: Largely confirmed
- Productiveness & innovation: Partly confirmed
- Gig economic system & freelancing: Unverified
- Freelancing motivations & challenges: Not strictly factual claims
- Facet gigs & a number of jobs: Unverified
- Demographics & fairness: Partly confirmed / blended
- Political & cultural influences: Partly confirmed / principally unverified
- Different components & coverage panorama: Typically correct however qualitative
As you’ll be able to see, of the 17 information factors, Agent thought-about solely 5 to be absolutely confirmed. Distinction this with how GPT-4o analyzed the outcomes. When GPT-4o was given the identical PowerPoint deck, it thought-about all assertions to be confirmed. You possibly can see GPT-4o’s detailed results here.
Regardless that I used the AI to validate the AI, I in all probability would not be comfy utilizing any of the presumed information in my work with out private, Mark I Eyeball affirmation. Nonetheless, it was a enjoyable train, and interesting to see how completely different the outcomes had been between ChatGPT Agent and ChatGPT 4o.
You possibly can watch a replay of the entire session here.
8. Analyze constructing code for fence set up
- Understanding of the issue: Strong
- Execution: Fairly near good
- Hallucination: None. It obtained all however one graphic good
- Processing time: 4 minutes
Again after we lived in Palm Bay, Florida, we lived on a nook property. The home got here with what might solely charitably be referred to as a fence. We wanted to exchange it, and since we wished privateness, we wished to see simply how a lot fence we might legally set up.
Over the course of a few years, I spent a ton of time going forwards and backwards with the planning workplace in an effort to each perceive what I might do with a fence, and what different options is perhaps obtainable to me.
Since I’ve a number of historical past with this challenge and am very aware of Palm Bay codes (even years after shifting away), I made a decision to level ChatGPT Agent on the drawback.
It took all of 4 minutes to offer an in depth, correct evaluation. It even created working diagrams that illustrated the choices. Based mostly on my expertise, I do know the outcomes to be correct.
ChatGPT Agent produced output that could possibly be used to take this challenge to the following step. Again once I lived in Palm Bay, the equal in all probability took me 20 calls, a ton of emails, and some visits to Metropolis Corridor to provide you with choices. The extent of presentation and group I got here up with wasn’t even shut.
If Agent can up its sport elsewhere to be on a par with this take a look at, then it can have some legs.
You possibly can watch a replay of the entire session here.
What’s all of it imply?
Properly, it certain as heck is not sentient but. At greatest, it is like that administrative assistant you employed as a result of your mother mentioned you needed to rent her cousin’s unemployable slacker child. There are occasional flashes of brilliance, however principally the output looks like the results of each aggressively following instructions and purposely inventing different information.
Is it value $200/month for the Professional program? Not for Agent. At the least not but. Agent is unreliable and usually performs pretty poorly. In a yr or so, I am certain it can get higher. However now? No. The one motive to spend $200 a month on it’s to do what I am doing: testing it to see the place the know-how is immediately.
Keep tuned, as a result of regardless of all of the inaccuracies and drawback areas, this undoubtedly reveals the place AI know-how might go. After all, if an online searching AI Agent is the long run, and all of the content material websites on the market block it as a result of AI is stealing our content material, then we’ll have a really attention-grabbing drawback.
Additionally: I’m an AI tools expert, and these are the only two I pay for (plus three I’m considering)
It is early days, of us. Whether or not it is a know-how that can be a boon to all humanity or a know-how that destroys the web and kills us in our sleep stays to be seen.
However hey, within the meantime, I and the remainder of the ZDNET crew can be attempting to make sense of all of it for you. So hold coming again. We’ll have extra to let you know. I will be tinkering with Agent and I am certain I will have extra to say as effectively.
Have you ever tried ChatGPT Agent but? If that’s the case, did it comply with your directions precisely or veer off into its personal interpretation of the duty? Did it hallucinate or hit the mark? How do you are feeling about giving AI instruments entry to your recordsdata, accounts, or browser? Are you seeing extra worth in this sort of automation, or are you continue to ready for it to change into helpful? Tell us within the feedback beneath.
You possibly can comply with my day-to-day challenge updates on social media. You should definitely subscribe to my weekly update newsletter, and comply with me on Twitter/X at @DavidGewirtz, on Fb at Facebook.com/DavidGewirtz, on Instagram at Instagram.com/DavidGewirtz, on Bluesky at @DavidGewirtz.com, and on YouTube at YouTube.com/DavidGewirtzTV.
