I have tried to use free tier ChatGPT for tasks like helping me with high school algebra, asking it python code questions and helping me write the first draft of a short story. It is absolutely awful. Sure, it’s extremely quick to give answers but it’s spewing out many words but says very little. It hallucinates like crazy for the math and code questions.
Bing with GPT4 is much slower but it’s much more human-like and it’s much more aware what you’re talking about. It hallucinates only 1/10th of the time which is pretty good for a free product.
GPT4 is the thing you should be testing. If you base your impression of what generative AI is capable of on the free tier ChatGPT, you'll be way off base.
My knowledge of Terraform was limited to the basic principles, but I've been using ChatGPT to develop scripts and learn as I go. It's been excellent, and the scripts I've been working on are merrily terraforming AWS with a custom VPC, subnets, internet gateway, security groups, EC2 instances, keypair generation, etc.
The majority of suggestions work first time. The ones that don't are a good learning experience, as you can discuss the issue or error with GPT4 and dig deeper into the causes. For an effective learning experience, it's important to not just accept config or code that you don't understand. This is where the nature of ChatGPT is useful, because you can ask as many followup questions as you like. When learning this way, it's also useful to tweak the custom instructions feature and focus the responses on common or idiomatic approaches.
I'm not sure if your comment was more about asking it to generate a complete Terraform project and verifying that everything runs perfectly first time, but I wanted to mention that it's been highly accurate for me when taking an iterative approach (GPT4 at least).
This mirrors my (very positive) experience migrating a web app to modern technologies, a project that was my first serious use of TypeScript, a new-to-me framework, and a new-to-me build system.
I think anyone who treats LLMs like search engines is setting themselves up for disappointment.
It is quite bad at certain higher-level tasks, like generating exotic language scripts. But for things like Algebra and basic boilerplate programming stuff it is amazing.
Especially for computational stuff, like Math, when using its "Advanced Data Analysis" feature where it doesn't try to hallucinate the answer but generates the code to compute the answer instead.
This hasn't been my experience. In fact GPT4 seems to absolutely excel at DevOps boilerplate automation stuff (GitHub Actions, Docker compose, Cloudformation, etc). It still struggles on more strictly software development stuff like algorithms though.
Bing with GPT4 is much slower but it’s much more human-like and it’s much more aware what you’re talking about. It hallucinates only 1/10th of the time which is pretty good for a free product.