Contents

Using OpenRouter to access large models library

Contents

OpenRouter makes it super easy to just get best scores at school by using a large set of models available.

; claude-write-fibonacci

(require '[pyjama.openrouter.core :as openrouter])
(let [response (openrouter/with-prompt "Write Fibonacci in Clojure!")]
  (println
    (-> response :choices first :message :content)))

I actually wanted to evaluate answers from anthropic/claude-3.7-sonnet so that model is the default one for open router.

With a bit of rewriting around, we can get something vastly more generic, where the prompt and the model are simply calling openrouter/with-config.


(defn prompt-for-model[prompt model]
  (let [response (openrouter/with-config {:prompt prompt :model model})]
      (-> response :choices first :message :content)))

(deftest mistral-write-fibonacci
  (println
    (prompt-for-model
      "Write Fibonacci in Clojure!"
      "cognitivecomputations/dolphin3.0-mistral-24b:free"))

From there the obvious next step is to perform execution of the same prompt over different model, and review their response.


(defn get-responses [prompt models]
  (doall (map (fn [model] [model (prompt-for-model prompt model)]) models)))

(defn score-response [prompt qa-pairs scoring-model]
  (map (fn [[model answer]]
         (let [score-prompt (str "Evaluate the following answer:\n\n"
                                 "Question: " prompt "\n"
                                 "Answer: " answer "\n\n"
                                 "Provide:\n"
                                 "- A score from 1 to 10\n"
                                 "- A brief explanation for the score\n"
                                 "- Suggestions for improvement")
               response (prompt-for-model score-prompt scoring-model)]
           [model answer response]))
       qa-pairs))

Some flipping pages in openrouter’s library of model, we would like to compare the output of:

  • deephermes-3-llama-3-8b-preview
  • gemini-2.0-flash-lite-001
  • r1-1776 (deepseek)

We will use dolphin traiend mistral to review and analyse the answers:

  • dolphin3.0-r1-mistral-24b
(let [prompt "Write Fibonacci in Clojure!"
        models ["nousresearch/deephermes-3-llama-3-8b-preview:free"
                "google/gemini-2.0-flash-lite-001"
                "perplexity/r1-1776"]
        scoring-model "cognitivecomputations/dolphin3.0-r1-mistral-24b:free"
        responses (get-responses prompt models)
        scored-responses (score-response prompt responses scoring-model)]
    (write-to-markdown "results.md" prompt scored-responses))

We pretty easily generate this full review, comparing all the different models, and with a nice conclusion on which model to use for further similar work:

# Conclusion

**Best Model:** ` perplexity/r1-1776 `

**Best Answer:**
## Evaluation of the Answer: Fibonacci in Clojure

**Score:** 9/10

**Explanation for the Score:**

The provided answer is excellent and demonstrates a good understanding of Clojure and idiomatic practices.  It offers two distinct and well-explained approaches to generating Fibonacci sequences or numbers:

* **Lazy Sequence:** The `fib-seq` function is a brilliant use of `lazy-seq` and `letfn`. It effectively creates an infinite, lazily evaluated Fibonacci sequence.  This is highly memory-efficient, especially for large sequences, as it generates values only when requested. The use of `lazy-seq` is a core Clojure idiom for creating infinite or large sequences without burdening memory. The `letfn` and recursive structure are also very Clojure-like and readable. The explanation clearly points out its laziness and on-demand generation.

* **Iterative Approach:** The `fib` function efficiently computes the nth Fibonacci number using `loop` and `recur`, which is the standard and most efficient way to perform iteration in Clojure.  It achieves O(n) time complexity and O(1) space complexity, as correctly stated. The explanation is clear, highlighting its efficiency and iterative nature.

**Strengths:**

* **Correctness:** Both implementations are functionally correct and accurately produce Fibonacci numbers/sequences.
* **Idiomatic Clojure:**  The code leverages core Clojure constructs like `lazy-seq`, `letfn`, `loop`, and `recur` in a natural and effective way. This demonstrates good Clojure style.
* **Efficiency:** The lazy sequence is memory-efficient for generating sequences, and the iterative approach is time-efficient for calculating specific nth numbers.
* **Clarity and Readability:** The code is well-formatted, commented, and easy to understand. The explanations are concise and effectively highlight the key features of each approach.

**Minor Point Deduction (leading to 9/10 instead of 10):**

While excellent, the answer could perhaps benefit from a tiny addition:

* **Explicitly mentioning Tail-Call Optimization (though implicitly present):**  While `recur` *is* tail-call optimization in Clojure, explicitly mentioning this in the explanation of the iterative approach could further enhance the answer's depth, especially for readers who might be less familiar with the underlying mechanics.  However, it's by no means a major flaw, as `recur` is understood to be efficient iteration in Clojure.  Alternatively, you could mention that `loop`/`recur` avoids stack overflow issues inherent in naive recursion.

**Suggestions for Improvement:**

1. **Consider adding a brief note on Tail-Call Optimization or `recur`'s efficiency:** As mentioned above, a very minor detail to explicitly connect `recur` to tail-call optimization for even clearer understanding, especially for learners.
2. **Maybe add a quick comparison between the two approaches:**  Explicitly state when each approach is most suitable (e.g., lazy sequence for potentially infinite sequences or when you don't need all elements at once, iterative for getting a specific nth number quickly).
3. **Consider adding a very simple test case:** While not strictly necessary for this basic example, including a simple REPL interaction or `assert` statement could further solidify the correctness in a more formal testing context, although the examples provided are sufficient for demonstration.


**Overall:**

The answer is very strong and demonstrates a good grasp of Clojure and problem-solving.  The two approaches are well-chosen, idiomatically implemented, and clearly explained.  A slight enhancement by explicitly mentioning tail-call optimization in the iterative approach explanation would be a very minor improvement, but the current answer is already excellent.

## Summary
- ` perplexity/r1-1776 ` provided the highest-rated answer.
- Consider using this model for similar tasks in the future.

Test code is on github.