Is LLM the Stockfish for Programmers?

Is LLM the Stockfish for Programmers?

(Picture courtesy: Chess.com)

A Chess Engine is a powerful computer program designed to play chess at an exceptionally high level. Chess engines provide strategic insights, suggest optimal moves, and serve as a valuable training partner for professional chess players.

Currently one of the strongest chess engines is Stockfish

I indulge in chess as a hobby and I love the relentless pursuit of self-improvement among competitive chess players. Before Chess Engines professional chess players studied the game by...

  • Learning openings and tactics from classic chess books.

  • Dissecting the playing styles of Grandmasters and understanding their strengths and weaknesses.

  • Playing against a variety of opponents in tournaments and improving their decision-making abilities.

Here is a video of Magnus Carlsen demonstrating his expertise and photographic memory.

Modern Chess Engines have completely revolutionized the way chess is studied.

Check this interview with Hikaru Nakamura ( 8 minutes 20 seconds onwards) where he talks how modern chess engines have completely flipped the equation between a chess player and computer.

So why am I talking about chess?

As programmers, historically we have had books, conferences, open-source code to study, code reviews with seasoned programmers and so on.

However, we never had the equivalent of a chess engine.

Yes, we have static code analyzers that provide valuable insights, but they are not "digital mentors" like a chess engine.

Something that one can tweak continuously and refine solutions.

I hope the following transcript with ChatGPT demonstrates that programmers might be on a similar boat as professional chess players.

I have retained only the essential stuff from the verbose ChatGPT replies.

Program Comprehension

I purposefully provided a poorly written Java method to check if ChatGPT "understands" the program.

Program Improvements

public class Patient {
    public boolean hasHighBloodPressure() {
        return getDiastolicPressure() > 90 || getSystolicPressure() > 140;
    }
    public int calculateHealthScore() {
        int healthScore = 0;
        if (hasHighBloodPressure()) {
            healthScore++;
        }
        if (isSmoker()) {
            healthScore++;
        }
        if (isHyperTensive()) {
            healthScore++;
        }
        return healthScore;
    }
}

Modifying requirements to test ChatGPT's responses

[P]: Consider this... Generally, the threshold values of what is considered to be risk could differ based on a Patient's age, medication etc.

So do you think it is better to have methods such as hasHighBloodPressure() within the Patient class or would you suggest something different?

public class HealthParameters {
    private int diastolicPressure;
    private int systolicPressure;
    private boolean smoker;
    private boolean hypertensive;
    public boolean hasHighBloodPressure() {
        return diastolicPressure > 90 || systolicPressure > 140;
    }
    public boolean isSmoker() {
        return smoker;
    }
    public boolean isHypertensive() {
        return hypertensive;
    }
}
public static int calculateHealthScore(HealthParameters healthParameters) {
    int healthScore = 0;
    if (healthParameters.hasHighBloodPressure()) {
        healthScore++;
    }
    if (healthParameters.isSmoker()) {
        healthScore++;
    }
    if (healthParameters.isHypertensive()) {
        healthScore++;
    }
    return healthScore;
}

Asking for suggestions and evaluating the ChatGPT response

[P]: Wouldn't it be better to have the checks such as "diastolicPressure > 90 || systolicPressure > 140" within the calculateHealthScore() method?

What do we gain by having it in the HealthParameters class?

The current approach that you have suggested reduces calculateHealthScore() to a mere risk score accumulator, isn't it?

public static int calculateHealthScore(HealthParameters healthParameters) {
    int healthScore = 0;
    if (healthParameters.getDiastolicPressure() > 90 || healthParameters.getSystolicPressure() > 140) {
        healthScore++;
    }
    if (healthParameters.isSmoker()) {
        healthScore++;
    }
    if (healthParameters.isHypertensive()) {
        healthScore++;
    }
    return healthScore;
}

Pushing the solution towards "Design Patterns"

public interface HealthParameterCheck {
    boolean isConditionMet(HealthParameters healthParameters);
}
public class HighBloodPressureCheck implements HealthParameterCheck {
    @Override
    public boolean isConditionMet(HealthParameters healthParameters) {
        return healthParameters.getDiastolicPressure() > 90 || healthParameters.getSystolicPressure() > 140;
    }
}
public class SmokingCheck implements HealthParameterCheck {
    @Override
    public boolean isConditionMet(HealthParameters healthParameters) {
        return healthParameters.isSmoker();
    }
}
public class HealthParameterComposite implements HealthParameterCheck {
    private List<HealthParameterCheck> checks = new ArrayList<>();
    public void addCheck(HealthParameterCheck check) {
        checks.add(check);
    }
    @Override
    public boolean isConditionMet(HealthParameters healthParameters) {
        for (HealthParameterCheck check : checks) {
            if (check.isConditionMet(healthParameters)) {
                return true; // At least one condition is met.
            }
        }
        return false; // No conditions are met.
    }
}

public class HealthCalculator {
    private HealthParameterCheck healthParameterCheck;

    public HealthCalculator() {
        // Create a composite check for multiple conditions
        HealthParameterComposite composite = new HealthParameterComposite();
        composite.addCheck(new HighBloodPressureCheck());
        composite.addCheck(new SmokingCheck());
        // Set the composite check as the main health parameter check
        this.healthParameterCheck = composite;
    }

    public int calculateRiskScore(HealthParameters healthParameters) {
        if (healthParameterCheck.isConditionMet(healthParameters)) {
            return 1; // Health risk detected
        }
        return 0; // No health risk
    }

    public static void main(String[] args) {
        // Example usage
        HealthCalculator calculator = new HealthCalculator();
        HealthParameters patientParameters = new HealthParameters(95, 130, true); // Example health parameters

        int riskScore = calculator.calculateRiskScore(patientParameters);
        System.out.println("Risk Score: " + riskScore);
    }
}

Gotcha!

In the previous step, it completely lost the intent of the program. I am trying to get it back on track here...

[P]: I think with the use of a Strategy and Composite, you have changed the meaning of the program. Remember that our intent was not to get a "1" or "0" based on whether a health risk was detected or not. Our goal was to get a cumulative risk score. Also, I see the usage of HealthParameterComposite as an unnecessary abstraction.

import java.util.ArrayList;
import java.util.List;

// HealthParameterCheck interface and its implementations remain the same

public class HealthCalculator {
    private List<HealthParameterCheck> healthChecks;

    public HealthCalculator() {
        // Initialize the list of health checks
        this.healthChecks = new ArrayList<>();
        this.healthChecks.add(new HighBloodPressureCheck());
        this.healthChecks.add(new SmokingCheck());
        // Add more checks as needed
    }

    public int calculateRiskScore(HealthParameters healthParameters) {
        int riskScore = 0;
        for (HealthParameterCheck check : healthChecks) {
            if (check.isConditionMet(healthParameters)) {
                riskScore++;
            }
        }
        return riskScore;
    }

    public static void main(String[] args) {
        // Example usage
        HealthCalculator calculator = new HealthCalculator();
        HealthParameters patientParameters = new HealthParameters(95, 130, true); // Example health parameters
        int riskScore = calculator.calculateRiskScore(patientParameters);
        System.out.println("Risk Score: " + riskScore);
    }
}

This conversation went on until we introduced a HealthCheckRegistry class with rules read dynamically during runtime and instantiated using reflection :)

Key Takeaway

When some commentators on Hacker News dismiss LLM as a linguistically gifted stochastic parrot I do not understand the skepticism.

To me, the fact that I can have a conversation with a marvelous piece of technology to refine code, brainstorm ideas, and debug problems is nothing short of magic.

Just as a novice chess player might struggle with a chess engine, but a seasoned player can harness its power, LLMs can serve as a valuable guide if the programmer knows how to utilize them effectively, discern when they might deviate, and steer them back on track.

In my view, Large Language Models could be the programmer's Stockfish.