Summary Points
- Modern AI coding tools often produce both code and tests from the same ambiguous specification, undermining true verification and validation (V&V).
- Effective V&V requires independent review: the person writing the tests must be separate from the coder to catch ambiguities and disagreements.
- The author’s open-source tool, qikly, automates this separation by having one AI agent specify and generate tests, and another generate code based solely on requirements, ensuring independence.
- This approach prevents “self-confirmation” where code and tests agree because they share assumptions, thereby enhancing meaningful verification—crucial for safety-critical industries.
Advancing Toward Independent Verification
Modern AI tools can generate code and tests from a single specification. This makes development faster and more efficient. However, it also creates a problem. When both the code and tests come from the same source, they share the same biases and ambiguities. This means tests can never truly oppose the implementation. In safety-critical industries, this lack of independence undermines verification efforts. Recognizing this, some developers are exploring ways to automate independent testing. The goal is to ensure tests are unbiased and truly confirm the system works as intended. This shift could revolutionize how teams approach testing, making it both quicker and more reliable.
Understanding Specification and Its Challenges
Specifications guide what a system must do. They often use plain language, which can be vague or ambiguous. For example, a radar specification might say the device must measure distances up to 250 meters. But it doesn’t specify where the measurement starts or if 0 meters is valid. When a developer writes code based on this, they decide how to interpret these ambiguities. Then, a tester from the same team verifies it, often confirming what the developer already assumes. This process risks missing potential issues, because the tester’s perspective isn’t truly independent. Over time, these ambiguities can cause overlooked flaws or misunderstandings, especially as the model improves and becomes more consistent.
Implementing Spec-Driven Test Automation
New AI workflows aim to split the roles of writing code and testing. One team defines the requirements in clear, plain language. Another team writes test criteria based only on these requirements, without seeing the implementation. The challenge then is to generate independent tests from specifications alone. This is where open-source tools come into play. They generate separate code implementations and test suites solely from the requirements. During testing, the code is verified against the criteria, not the actual implementation. If a test fails, developers see what went wrong and can fix it. This approach maintains the separation of duties and leverages AI to replicate the verification process traditionally done by human teams. While promising, it depends on precise specifications and careful handling of ambiguities—a step forward in trustworthy automation.
Continue Your Tech Journey
Explore the future of technology with our detailed insights on Artificial Intelligence.
Access comprehensive resources on technology by visiting Wikipedia.
AITechV1
