I am fighting with Claude 4 and ChatGPT 4.1 using them in VSCode with Copilot as an Agent (Gemini 2.5 pro fails every time), everything is fine, but they can't figure out a simple PHP application with everything explained in a nice file ,every url is wrong, uses differents patterns of design in every page, broke stuff that worked, iterates in useless testing code that never works and more.
So, probably you don't need to test an AI in a precise exercise but in a real world application, an useful one, there's the place where it belongs and it works awfuly bad at it, i need the AI to follow the instructions and not take every damn wrong path nobody is telling it to take :P