This was supposed to be an article comparing three major AI players in their ability to write technical articles. In preparation, however, I made some mistakes ... :(
When I prepared for the initial article, I asked them to fetch the data from https://github.com/gabr42/ComputeCore/blob/main/ComputeCore.pas and write about it. (See the end of this article for the initial query and results.) At that moment, however, that link was not public yet. All three AIs ignored that and happily wrote about the code they knew nothing about. It was only Claude that complained about the bad link but I missed that while checking the results.
So I decided to rerun the experiment on all platforms. I started by asking the engines the following question:
Can you accesss https://github.com/gabr42/ComputeCore/blob/main/ComputeCore.pas and provide an overview of the code in one paragraph?
Can you accesss https://raw.githubusercontent.com/gabr42/ComputeCore/refs/heads/main/ComputeCore.pas and provide an overview of the code in one paragraph?
Analyzing the answer shows that the answers were not done purely on the basis of the linked code. Both Claude and Gemini have included details that they got from my previous article on ComputeCode.
In the answers below I marked hallucinated misinformation with red and information that was obviously retreived from different sources with blue.
[ChatGPT]
ComputeCore.pas
implements a lightweight, unit-based parallel computation framework for Delphi. It defines an abstraction over task-based parallelism using the IComputeTask
interface, which encapsulates a single unit of computation with a Run
method. The main orchestrator, TComputeCore
, manages a pool of worker threads, balancing execution via TComputeQueue
, a thread-safe task queue. Tasks are submitted through TComputeCore.Run
, which enqueues them and ensures efficient thread utilization without over-subscription. The unit also includes support for cooperative cancellation and shutdown through TComputeCancellationToken
. Designed to be simple and efficient, it offers a foundation for building parallel computation workflows without relying on external libraries like OmniThreadLibrary or the Parallel Programming Library (PPL).[Claude Sonnet 4]
[Gemini 2.5 Pro]
Based on the information I found, ComputeCore is a lightweight parallel processing framework for Delphi. It is designed to simplify the execution of CPU-intensive tasks by distributing them across multiple threads. The main goals of the framework are to run tasks concurrently, allow tasks to create child tasks, and provide a simple interface for developers. It is intended to be a general-purpose framework that can be used in various projects. It will eventually be integrated into the Spring4D project as Spring.Threading
.
My comment: Short and definitely extracted from my previous article and not from the code.
Original prompt
Please write this article.
Answers
Observations
As you can see from the prompt, I was trying to treat the AI as a helpful assistant researcher. I did specify the main guide lines for the article but let the AI pull in all the missing pieces. This approach, however, failed as the AIs were not able to access the code at all.
I have to repeat the experiment, but with a different codebase.
The link to OpenAI ChatGPT gives me a 404. The other two provide the answers ok.
ReplyDeleteIt seems to me that you didn't put sufficient constraints on the original prompt. You must tell the LLM that it must access the original code and base its response on that code. All you said was to cover certain talking points, with no restrictions on where that discussion came from.
The URL for the ChatGPT article is https://github.com/gabr42/DelphiAndAI/blob/main/08%20Write%20me%20an%20application%20about%20ComputeCore/computecore-chatgpt.pdf
ReplyDeleteThank you, I fixed the link!
Delete