Using AI to test logical arguments

[popeyespappy]

Hi guys. It's been a while.

Not very long along on a forum not very far from here, I watched a theist try, unsuccessfully, to use AI to support his proof of the trinity. As a result of that I designed, with the help of several LMM's including ChatGPT, DeepSeek, Claude, and Grok, a set of prompts designed to have AI based LMM's test arguments.

I tested the prompts against several metaphysical & philosophical arguments including the cosmological, design, ontological, and moral arguments for god. Then I tested evolution twice. Once as scientific theory and again as an alternative to design. I thought the results were interesting but not surprising.



Here are the prompts if you are interested in trying them:

Analytical Prompts for Testing Arguments

Introduction
These prompts are designed to rigorously analyze arguments in good faith and mitigate bias during the analysis.
The goal is not to “win,” but to:
• Understand the argument clearly
• Identify strengths and weaknesses honestly
• Improve reasoning for all sides
Use the prompts sequentially. Each builds on the previous.
________________________________________
1. Identify the Structure
Premises
List all explicit premises in the argument as numbered statements. Do not evaluate them.
Hidden Assumptions
Identify all implicit or unstated assumptions the argument relies on.
Formal Structure
Rewrite the entire argument in formal logical form:
numbered premises → intermediate steps → conclusion.
________________________________________
2. Test Validity and Soundness
Validity
If all premises were true, would the conclusion logically follow?
Identify any gaps, unwarranted inferences, or non sequiturs.
Soundness
Evaluate each premise by categorizing it as:
• Empirical claim
• Historical claim
• Interpretive/theological claim
• Philosophical/metaphysical claim
• Definitional claim
Identify where uncertainty or dispute exists.
________________________________________
3. Clarify Concepts & Methods
Definitions
List all key terms and note any ambiguities, inconsistencies, or shifting meanings.
Methodology
Identify the methods of reasoning used (e.g., deductive logic, analogy, inference to best explanation).
List any assumptions underlying those methods.
________________________________________
4. Stress-Test the Argument
Counterargument
Generate the strongest possible counterargument to test the reasoning.
Alternative Interpretations
Provide at least three different ways the same facts, data, or premises could be interpreted.
Stress Test
Test whether the conclusion still holds if key assumptions, definitions, or conditions are changed.
Generalization Test
Check whether the same method could “prove” contradictory or mutually exclusive claims.
If yes, explain why the method may be unreliable.
________________________________________
5. Identify Logical Fallacies
Fallacy Analysis
List any formal or informal fallacies in the argument.
For each fallacy identified:
• Explain where it occurs
• Explain why it is problematic
• Explain what would be required to avoid or correct it
________________________________________
6. Improve the Argument
Steelman
Rewrite the argument in its strongest possible form while preserving the original intent.
Address the major weaknesses identified.
Formal Proof
Present the steelmanned version as a clean, numbered formal proof.
After each premise or inference, label it as:
• Empirically verified
• Widely accepted
• Disputed
• Assumption
• Logical inference
Highlight Weak Points
Identify which specific steps require the greatest additional evidence or justification.
________________________________________
7. Summary Assessment
Provide a balanced overall assessment that includes:
• Major strengths
• Major weaknesses
• Logical gaps
• Well-supported points
• Evidence needed to strengthen the argument
• Whether the argument meets minimal standards of clarity and coherence
This is not the final verdict—it is an integrated summary of the analysis.
________________________________________
8. Final Verdict: Pass or Fail
State clearly whether the argument:
• ✅ Passes
• ⚠️ Partially passes (valid but unsound, or sound but incomplete)
• ❌ Fails
Explain:
• Whether the argument is valid
• Whether it is sound
• Which premises or inferences cause the failure
• What would be required for the argument to pass
This step forces the model to commit to a final determination based on all previous analysis.