I wrote an LLM prompt generator/optimizer ChatGPT - CO-STAR Prompt Optimizer base on How I Won Singapore’s GPT-4 Prompt Engineering Competition | by Sheila Teo | Towards Data Science

It use the CO-STAR technique:

• C: Context
• O: Object
• S: Style
• T: Tone
• A: Audience
• R: Response

and helps me to create relevant, creative, and precise responses that meet my needs most of the time.

How to Open the Web Console in iOS Safari with Scriptable and Eruda

Sometimes I just want to quickly inspect a webpage on my phone without needing a desktop browser. Unfortunately, iOS Safari doesn’t come with built-in developer tools. But, I found a way around it by combining Scriptable and the web console library Eruda - Console for Mobile Browsers | Eruda Here’s how you can do it too.

The Setup

We’re going to use a small script in the Scriptable app that loads Eruda right into Safari, giving you an interactive console on your iPhone or iPad. Follow the steps below to get everything running.

1. Install Scriptable

First, you’ll need to download the Scriptable app from the App Store. It lets you run JavaScript code directly on your iOS device.

2. Get Eruda

Eruda is a lightweight console for mobile browsers. It’s perfect for inspecting elements and running JavaScript on the fly in Safari.

3. Create the Script

Now, use the code snippet from my gist open web consle on iOS Safari · GitHub to create a script in Scriptable. This script injects the Eruda console into any webpage you’re viewing on Safari.

4. Create a iOS shortcut

create a iOS shortcut so that webpage can be shared to the shortcut to invoke the script.

Using the Console

Once the script runs, Eruda will load at the bottom of the page. Click the icon of gear, and you can inspect elements, execute JavaScript, and generally poke around the site just like you would on a desktop browser.

API price for gpt40-mini is very cheap yet the model is very capable, it is basically a nearly-free boost for the input quality for larger [[LLM]] models.

Here is a prompt I used

Enhance the following text to improve its quality for processing by a larger language model:

1. Correct any grammatical or spelling errors.
2. Improve sentence structure and flow.
3. Clarify any ambiguous or vague statements.
4. Ensure logical coherence and progression of ideas.
5. Remove redundant information while preserving all key points.
6. Maintain the original tone and intent of the text.
7. Do not add new information or alter the core meaning.

Provide the enhanced text in a clear, concise format. If any part of the text is unclear or requires subject matter expertise to interpret, flag it with [NEEDS CLARIFICATION] at the end of the relevant sentence.

It’s so pleasant to use [[nix]] to install and config complex software packages.

Here is how to make emacs org work with latex

config-latex.nix

# https://nixos.wiki/wiki/TexLive
# For a minimal set of packages needed for Emacs Orgmode
{ pkgs, lib, ... }:
let
  tex = (pkgs.texlive.combine {
    inherit (pkgs.texlive)
      scheme-basic dvisvgm dvipng # for preview and export as html
      wrapfig amsmath ulem hyperref capt-of fontspec;
  });
in { home.packages = lib.mkBefore [ tex ]; }

doom-emacs packages.el

(package! org-fragtog)

doom-emacs config.el

(use-package! org-fragtog
  :config
  (add-hook 'org-mode-hook 'org-fragtog-mode))

(after! org
  (setq org-preview-latex-default-process 'dvisvgm)
  (setq org-startup-with-latex-preview t))

I recently wanted to practice some LeetCode and write documents and code in an org file. To quickly test the code, I wanted to use C-c C-c on a src-block to run pytest. I created this snippet to enable that functionality.

(after! org
  (defun org-babel-execute:python-with-pytest (body params)
    "Execute a python source block with pytest if :pytest is specified."
    (if (assq :pytest params)
        (let* ((temporary-file-directory ".")
               (temp-file (make-temp-file "pytest-" nil ".py")))
          (with-temp-file temp-file
            (insert body))
          (unwind-protect
              (org-babel-eval (format "pytest -v -s %s" temp-file) "")
            (delete-file temp-file)))
      (org-babel-execute:python-default body params)))

  (advice-add 'org-babel-execute:python :override #'org-babel-execute:python-with-pytest))

usage example

#+begin_src python :pytest
def test():
    assert Solution().mergeAlternately("abc", "pqr") == "apbqcr"
    assert Solution().mergeAlternately("ab", "pqrs") == "apbqrs"
    assert Solution().mergeAlternately("abcd", "pq") == "apbqcd"

class Solution:
    def mergeAlternately(self, word1: str, word2: str) -> str:
        longest = max(len(word1), len(word2))

        def get_char(i, chs):
            return chs[i] if i < len(chs) else ""

        r = []
        for i in range(0, longest):
            r.append(get_char(i, word1))
            r.append(get_char(i, word2))

        return "".join(r)
#+end_src

I used the built-in tempo to create a template. This allows me to run M-x insert-leetcode-solution, which inserts the template content and places the cursor on the line below “Problem”.

#+begin_src elisp :tangle config.el
(require 'tempo)

(tempo-define-template
  "leetcode-solution"
  '("* Problem"
    n
    p
    n
    "* Note"
    n
    "* Solution"
    n
    "#+begin_src python :pytest"
    n
    "#+end_src"
    n))

(defun insert-leetcode-solution ()
  (interactive)
  (tempo-template-leetcode-solution))
#+end_src

Introducing Transducers: A Powerful Tool for Functional Programming

I recently learned the concept of transducer and implement it in [[Gleam]] language.

GitHub - nohzafk/gtransducer: Transducer in Gleam language

Transducers originated in Clojure, designed to tackle specific challenges in functional programming and data processing. If you’re working with large datasets, streaming data, or complex transformations, understanding transducers can significantly enhance the efficiency and composability of your code.

What Are Transducers?

At their core, transducers are composable functions that transform data. Unlike traditional functional programming techniques like map, filter, and reduce, which are tied to specific data structures, transducers abstract the transformation logic from the input and output, making them highly reusable and flexible.

Key Advantages of Transducers

1. Composability and Reusability

Transducers allow you to compose and reuse transformation logic across different contexts. By decoupling transformations from data structures, you can apply the same logic to lists, streams, channels, or any other sequential data structure. This makes your code more modular and adaptable.

2. Performance Optimization

One of the primary motivations for using transducers is to optimize data processing. Traditional approaches often involve creating intermediate collections, which can be costly in terms of performance, especially with large datasets. Transducers eliminate this overhead by performing all operations in a single pass, without generating intermediate results.

A Python example

import time
from functools import reduce

# Traditional approach
def traditional_approach(data):
    return [x * 2 for x in data if (x * 2) % 2 == 0]

# Transducer approach
def mapping(f):
    def transducer(reducer):
        def wrapped_reducer(acc, x):
            return reducer(acc, f(x))
        return wrapped_reducer
    return transducer

def filtering(pred):
    def transducer(reducer):
        def wrapped_reducer(acc, x):
            if pred(x):
                return reducer(acc, x)
            return acc
        return wrapped_reducer
    return transducer

def compose(t1, t2):
    def composed(reducer):
        return t1(t2(reducer))
    return composed

def transduce(data, initial, transducer, reducer):
    transformed_reducer = transducer(reducer)
    return reduce(transformed_reducer, data, initial)

data = range(1000000)

# Measure traditional approach
start = time.time()
traditional_result = traditional_approach(data)
traditional_time = time.time() - start

# Measure transducer approach
xform = compose(
    mapping(lambda x: x * 2),
    filtering(lambda x: x % 2 == 0)
)

def efficient_reducer(acc, x):
    acc.append(x)
    return acc

start = time.time()
transducer_result = transduce(data, [], xform, efficient_reducer)
transducer_time = time.time() - start

# Results
print(f"Traditional approach time: {traditional_time:.4f} seconds")
print(f"Transducer approach time: {transducer_time:.4f} seconds")
print(f"Traditional is faster by: {transducer_time / traditional_time:.2f}x")

however when executed the transducer version is much slower in Python

Traditional approach time: 0.0654 seconds
Transducer approach time: 0.1822 seconds
Traditional is faster by: 2.78x

Are Transducers Suitable for Python?

While transducers offer theoretical benefits in terms of composability and efficiency, Python might not be the best language for leveraging these advantages. Here’s why:

1. Python’s Function Call Overhead: Python has a relatively high overhead for function calls. Since transducers rely heavily on higher-order functions, this overhead can negate the performance gains that transducers are designed to offer.

2. Optimized Built-in Functions: Python’s built-in functions like map, filter, and list comprehensions are highly optimized in C. These built-ins often outperform custom transducer implementations, especially for common tasks.

3. Efficient Mutation with Lists: Python’s lists are mutable, and appending to a list in a loop is highly efficient. The traditional method of using list comprehensions or filter and map is often faster and more straightforward than setting up a transducer pipeline.

When to Use Transducers

Transducers shine in functional programming languages that emphasize immutability and composability, such as Clojure or Gleam. In these languages, transducers can significantly reduce the overhead of creating intermediate collections and improve performance in complex data pipelines. They’re especially powerful when working with immutable data structures, where avoiding unnecessary copies is crucial for efficiency.

In contrast, Python’s strength lies in its mutable data structures and optimized built-in functions, which often make traditional approaches more performant. However, if you’re working in a functional programming environment where immutability is the norm, or if you need to maintain a consistent API across various data sources, transducers can be a valuable tool.

Conclusion

Transducers are a powerful tool in the right context, but Python’s inherent characteristics—such as function call overhead and optimized built-ins—mean that traditional approaches may be more efficient for typical data processing tasks. If you’re working in a language that deeply benefits from transducers, like Gleam, they can greatly enhance your code. In Python, however, it’s often best to use the language’s strengths, such as list comprehensions and optimized built-ins, for performance-critical applications.

LLM Sampling Techniques: Minimum Probability and Temperature

Minimum Probability Sampling

Definition

Minimum probability sampling is a technique used in language model APIs to balance between diversity and coherence in the model’s output.

How it works

• Sets a dynamic threshold for token selection based on the probability of the most likely token.
• The threshold is a fraction (determined by the min_p value) of the top token’s probability.

Example explanation

Let’s say min_p = 0.1, and we’re generating the next token:

Scenario A:

• Most likely token probability: 95%
• Threshold: 95% * 0.1 = 9.5%
• Only tokens with probabilities ≥ 9.5% are considered

Scenario B:

• Most likely token probability: 10%
• Threshold: 10% * 0.1 = 1%
• Tokens with probabilities ≥ 1% are considered

Adaptive nature

• When the model is very confident (high top probability), the threshold is higher, limiting options to maintain coherence.
• When the model is less certain (lower top probability), the threshold lowers, allowing more diverse options.

Benefits

• Preserves diversity for open-ended choices
• Maintains coherence for deterministic choices (e.g., programming syntax)
• Allows higher temperatures without losing coherence

Temperature in LLM Sampling

Definition

Temperature controls the randomness in token selection during text generation.

Effects of Higher Temperature

1. Increased diversity in outputs
2. Exploration of less likely options
3. Reduced repetitiveness
4. Better performance on open-ended tasks
5. Potential mitigation of model biases
6. Improved resilience to prompt engineering

Challenges

• Maintaining coherence and relevance at higher temperatures

Optimal Use

• Lower temperatures: Tasks requiring high accuracy or factual correctness
• Higher temperatures: Creative or exploratory tasks

Synergy: min_p and Temperature

Combining min_p sampling with higher temperatures allows for:

• Increased creativity and diversity in outputs
• Maintained coherence by filtering out extremely improbable tokens

Key Takeaways

1. min_p sampling adapts token selection threshold based on the model’s confidence.
2. Higher temperatures increase output diversity but risk coherence.
3. Combining min_p with higher temperatures balances creativity and coherence.
4. The optimal sampling strategy depends on the specific task and desired outcome.

Python is a great language but not perfect.

There are some common pitfalls, many of these are legacy issues retained for backward compatibility.

I want to share some of them.

Global Interpreter Lock (GIL)

It’s 2024, but Python still struggles with multi-core utilization due to the Global Interpreter Lock (GIL).

• The GIL prevents multiple native threads from executing Python bytecode simultaneously.
• This significantly limits the effectiveness of multi-threading for CPU-bound tasks in CPython.
• While technically a CPython implementation detail, Python’s lack of a formal language specification means CPython’s behavior is often duplicated in other implementations.

Historically, when Python was created, there were no multi-core CPUs. When multi-core CPUs emerged, OS started to add thread support, the author added a thread interface as well, but the implementation was essentially single-core. The intention was to add real multi-threaded implementation later, but 30 years on, Python still grapples with this issue.

The GIL’s persistence is largely due to backward compatibility concerns and the fundamental changes removing it would require in the language and its ecosystem.

Lack of Block Scope

Unlike many languages, Python doesn’t have true block scope. It uses function scope and module scope instead.

def example_function():
    if True:
        x = 10  # This variable is not block-scoped
    print(x)  # This works in Python, x is still accessible

example_function()  # Outputs: 10

Implications:

1. Loop Variable Leakage:

   for i in range(5):
       pass
   print(i)  # This prints 4, the last value of i
   

2. Unexpected Variable Overwrites:

   x = 10
   if True:
       x = 20  # This overwrites the outer x, not create a new one
   print(x)  # Prints 20
   

3. Difficulty in Creating Temporary Variables: It’s harder to create variables that are guaranteed to be cleaned up after a block ends.

4. List Comprehension Exception: Interestingly, list comprehensions do create their own scope in Python 3.x.

   [x for x in range(5)]
   print(x)  # This raises a NameError in Python 3.x
   

Best practices:

• Use functions to simulate block scope when needed.
• Be mindful of variable names to avoid accidental overwrites.
• Be cautious of the risk of using incorrect variable names in large functions.

Mutable Objects as Default Arguments

This is a particularly tricky pitfall:

def surprise(my_list = []):
    print(my_list)
    my_list.append('x')

surprise()  # Output: []
surprise()  # Output: ['x']

Why this happens:

• Default arguments are evaluated when the function is defined, not when it’s called.
• The same list object is used for all calls to the function.

This behavior:

• Dates back to Python’s early days, possibly for performance reasons or implementation simplicity.
• Goes against the “Principle of Least Astonishment”.
• Has very few practical use cases and often leads to bugs.

Best practice: Use None as a default for mutable arguments and initialize inside the function:

def better_surprise(my_list=None):
    if my_list is None:
        my_list = []
    print(my_list)
    my_list.append('x')

Late Binding Closures

This issue is particularly tricky in loops:

def create_multipliers():
    return [lambda x: i * x for i in range(4)]

multipliers = create_multipliers()
print([m(2) for m in multipliers])  # Outputs: [6, 6, 6, 6]

Explanation:

• The lambda functions capture the variable i itself, not its value at creation time.
• By the time these lambda functions are called, the loop has completed, and i has the final value of 3.

Fix: Use a default argument to capture the current value of i:

def create_multipliers():
    return [lambda x, i=i: i * x for i in range(4)]

This behavior is particularly confusing because it goes against the intuitive understanding of how closures should work in many other languages.

The __init__.py Requirement

In Python 2 and early versions of Python 3, a directory had to contain an __init__.py file to be treated as a package.

• This requirement often confused beginners and led to subtle bugs when forgotten.
• It provided a clear, explicit way to define package boundaries and behavior.

Evolution:

• Python 3.3 introduced PEP 420, allowing for implicit namespace packages.
• Directories without __init__.py can now be treated as packages under certain conditions.

Modern best practices:

1. Use __init__.py when you need initialization code or to control package exports.
2. For simple packages or namespace packages, you can often omit __init__.py in Python 3.

Understanding these pitfalls is crucial for writing efficient, bug-free Python code. While they can be frustrating, they’re part of Python’s evolution and often retained for backward compatibility. Being aware of them will help you navigate Python development more effectively.

Anthropic AI just announced prompt caching

What

Prompt caching is a feature that allows developers to efficiently use large amounts of context or instructions in repeated API calls to Claude. While the entire prompt (including cached content) is sent with each request, the cached portion is processed more efficiently and charged at a reduced rate after the initial setup.

Key benefits:

• Reduced costs: Cached content is charged at only 10% of the base input token price in subsequent requests.
• Improved performance: Potentially faster processing times for large, repeated contexts.
• Enhanced capabilities: You can include more examples, instructions, or background information cost-effectively, leveraging Claude’s in-context learning abilities.

Use cases:

• Chatbots that need consistent, complex instructions
• Coding assistants that reference large codebases
• Q&A systems working with entire books or documents
• Any application requiring extensive, consistent context across multiple interactions

How

1. Initial cache setup: The first request to set up the cache is charged at 125% of the base input token price for the cached portion of the prompt (cache write).
2. Subsequent requests: The cached portion of the prompt is charged at 10% of the base input token price (cached read).
3. The entire prompt, including cached content, is sent with each request but processed more efficiently.
4. Cache has a 5-minute lifetime, refreshed each time the cached content is used.

Important notes

• Prompt caching doesn’t reduce data transfer; the full prompt is sent each time.
• It’s not traditional fine-tuning, but a way to efficiently leverage Claude’s long context window (200k tokens) and in-context learning capabilities.
• Enables customization of model behavior for specific tasks without changing model parameters.