HomeAboutPostsTagsProjectsRSS

Updated
Words1411
TagsRead4 minutes

Django-socio-grpc (DSG) is a framework for using gRPC with Django. It builds upon django-rest-framework (DRF), making it easy for those familiar with DRF to get started with DSG.

Although I decided to go back to DRF after exploring DSG, I chose to do so because I needed to get things done quickly. Using gRPC is considered a potential way to achieve performance gains, and there are some obstacles need to be addressed before going full gRPC. I’m leaving these notes as my learning experience.

The workflow

Using django-socio-grpc (DSG) the workflow is like following:

flowchart TD
  subgraph backend[Backend Side]
	  db-schema[Design Database Schema] --> |create| django-models[Django Models] --> |define| serializers
	  pb-backend[Protocol Buffers]	  
	  django-models --> server-app[Server Application]
	  django-models --> |migrate| database
  end

  subgraph protobuf-repo[Protobuf repository]
	  .proto[.proto] --> |run| buf-lint[buf lint] --> |detect| buf-check[Breaking Change] --> |if pass|protobuf-gen[Generate Code]
	  protobuf-gen --> server-stub[gRPC Server skeleton]
	  protobuf-gen --> client-stub[gRPC Client Stub]
	  protobuf-gen --> mypy-types[.pyi type stub]
  end

  subgraph frontend[Frontend Side]
	  pb-frontend[Protocol Buffers]
		client-app[Client Application] --> |call| client-stub
  end

  serializers --> |DSG generates| .proto

  server-app --> |implement| server-stub
  server-stub --> |serializes| pb-backend

  mypy-types --> server-app

  client-stub --> |serializes| pb-frontend
  pb-backend <--> |binary format over HTTP/2| pb-frontend

Make DSG aware of big integer model fields

Currently DSG (version 0.24.3) has an issue of mapping some model fields to int32 type in protobuf incorrectly due to DRF’s decision, they should be mapped to int64 type.

It’s kind of hard to implement at library level, in application level I implemented something like this, using BigIntAwareModelProtoSerializer as the parent class of the proto serializer will correctly map BigAutoField BigIntegerField PositiveBigIntegerField to int64 type in protobuf.

from django.db import models
from django_socio_grpc import proto_serializers
from rest_framework import serializers

class BigIntegerField(serializers.IntegerField):
    """Indicate that this filed should be converted to int64 on gRPC message.

    This should apply to
    - models.BigAutoField.
    - models.BigIntegerField
    - models.PositiveBigIntegerField

    rest_framework.serializers.ModelSerializer.serializer_field_mapping
    maps django.models.BitIntegerField to serializer.fields.IntegerField.

    Although the value bounds are set correctly, django-socio-grpc can only map it to int32,
    we need to explicitly mark it for django-socio-grpc to convert it to int64.
    """

    proto_type = "int64"

# First, get the metaclass of ModelProtoSerializer
ModelProtoSerializerMetaclass = type(proto_serializers.ModelProtoSerializer)

class BigIntegerSerializerMetaclass(ModelProtoSerializerMetaclass):
    def __new__(mcs, name, bases, attrs):  # noqa: D102 ANN001 ANN204 N804
        # First, let's modify the serializer_field_mapping if it exists
        for base in bases:
            if hasattr(base, "serializer_field_mapping"):
                # Create a new mapping dictionary inheriting from the base
                field_mapping = dict(base.serializer_field_mapping)
                # Update the mapping for BigInteger fields
                field_mapping.update(
                    {
                        models.BigIntegerField: BigIntegerField,
                        models.BigAutoField: BigIntegerField,
                        models.PositiveBigIntegerField: BigIntegerField,
                    }
                )
                # Add the modified mapping to attrs
                attrs["serializer_field_mapping"] = field_mapping
                break

        # Then proceed with the normal class creation
        return super().__new__(mcs, name, bases, attrs)

class BigIntAwareModelProtoSerializer(proto_serializers.ModelProtoSerializer, metaclass=BigIntegerSerializerMetaclass):
    """A ModelProtoSerializer that automatically converts Django BigInteger fields to gRPC int64 fields by modifying the field mapping."""

Major obstacles for using gRPC

The biggest obstacle is that browsers do not natively support gRPC, which relies on HTTP/2.0. As a result, client-side frontend calls to backend services from a browser using gRPC require a proxy, typically Envoy. This setup involves additional overhead, such as configuring a dedicated API gateway or setting up an ingress. Even with a service mesh like Istio, some extra work is still necessary.

The next challenge is how to corporate with existing RESTful services if we chose to add a gRPC service. For communications happen between RESTful service and gRPC service, a gRPC-JSON transcoder (for example Enovy ) is need so that HTTP/JSON can be converted to gRPC. Again some extra work is needed at infrastructure level.

The last part of using gRPC is that data is transferred in binary form (which is the whole point of using gRPC for performance) makes it a little bit harder for debugging.

Conclusion

Django-socio-grpc is solid and its documentation is good. However, the major issue is the overhead work that comes with using gRPC. I will consider it again when I need extra performance and my team’s tech stack is adapted to gRPC.

Updated
Words284
TagsRead2 minutes

This git principle advocates for a workflow that balances a clear main branch history with efficient feature development.

1. Merge into main:

  • Purpose: Keeps the main branch history clean and linear in terms of releases and major integrations.
  • How it works: When a feature is complete and tested, it’s integrated into main using a merge commit. This explicitly marks the point in time when the feature was incorporated.
  • Benefit: main branch history clearly shows the progression of releases and key integrations, making it easier to track releases and understand project evolution.

2. Rebase feature branches:

  • Purpose: Maintains a clean and linear history within each feature branch and simplifies integration with main.
  • How it works: Before merging a feature branch into main, you rebase it onto the latest main. This replays your feature branch commits on top of the current main, effectively rewriting the feature branch history.
  • Benefit:
    • Linear History: Feature branch history becomes a straight line, easier to understand and review.
    • Clean Merges: Merging a rebased feature branch into main often results in a fast-forward merge (if main hasn’t advanced since the rebase), or a simpler merge commit, as the feature branch is already based on the latest main.
    • Avoids Merge Bubbles: Prevents complex merge histories on feature branches that can arise from frequently merging main into the feature branch.

In essence:

  • main branch: Preserve a clean, chronological, and release-oriented history using merges.
  • Feature branches: Keep them clean and up-to-date with main using rebase to simplify integration and maintain a linear development path within the feature.

Analogy: Imagine main as a clean timeline of major project milestones. Feature branches are like side notes. Rebase neatly integrates those side notes onto the main timeline before officially adding them to the main history via a merge.

Updated
Words2026
TagsRead1 minute
Original code from @XorDev
vec2 p=(FC.xy-r*.5)/r.y*mat2(8,-6,6,8),v;for(float i,f=3.+snoise2D(p+vec2(t*7.,0));i++<50.;o+=(cos(sin(i)*vec4(1,2,3,1))+1.)*exp(sin(i*i+t))/length(max(v,vec2(v.x*f*.02,v.y))))v=p+cos(i*i+(t+p.x*.1)*.03+i*vec2(11,9))*5.;o=tanh(pow(o/1e2,vec4(1.5)));

Updated
Words227
TagsRead1 minute

TIL Python use the integer itself as the hash value, except for -1. hash value for -1 is -2.

# For ordinary integers, the hash value is simply the integer itself (unless it's -1).
class int:
    def hash_(self):

        value = self
        if value == -1:
            value == -2
        return value

source

Updated
Words154
TagsRead1 minute

Auto-venv is a Fish shell script that automatically activates and deactivates Python virtual environments when entering/leaving directory that contains virtual environment.

Recently, I added multiple enhancements compare to the upstream version, now it handles edge cases more gracefully:

  • It safely manages virtual environment inheritance in new shell sessions.
  • It prevents shell exits during the activation and deactivation processes.

Updated
Words622
TagsRead2 minutes

A raycast script command to start lowfi

I recently create a Raycast script command to start the lowfi command effortlessly, so I can enjoy the lowfi music in no time and control it just using keyboard.

Here is the gist , just place the lowfi.sh to the Raycast Script Directory . Run it the first time, a wezterm window will be created if lowfi process isn’t running, run it the second time, the wezterm window will be brought to the front.

Problem of using /opt/hombrew/bin/wezterm

While i thought it was a simple task, it tooks me 1 hour to finished it. I encountered a totally unexpected problem: osascript can’t control the wezterm window that running lowfi process.

I installed wezterm using homebrew cask, when launching WezTerm using the Homebrew-installed binary (/opt/homebrew/bin/wezterm), the window was created with a NULL bundleID, which made it impossible for AppleScript/System Events to properly control it. This is because the Homebrew version doesn’t properly register itself with macOS’s window management system.

I was only able to debug this problem thanks to the amazing aerospace command

aerospace debug-windows

The solution was to always use the full application path /Applications/WezTerm.app/Contents/MacOS/wezterm for all WezTerm operations, ensuring proper window management integration with macOS.

When launching WezTerm using the full application path (/Applications/WezTerm.app/Contents/MacOS/wezterm), the window is created with the proper bundleID com.github.wez.wezterm. This allows:

  1. System Events to properly identify and control the window
  2. AppleScript to manipulate the window through accessibility actions (AXRaise, AXMain)
  3. Proper window focusing and bringing to front

Updated
Words293
TagsRead1 minute

To add Mermaid support to Hugo, one key detail not explicitly covered in the official documentation is that code fences must be used to enable rendering hooks for code blocks. If code fences are not enabled, the rendering hook has no effect.

Additionally, the condition for including Mermaid.js does not function as expected. Despite trying multiple approaches, I was unable to make it work reliably and ultimately decided to remove the conditional statement entirely.

complete configuration

hugo.toml

[markup]
[markup.highlight]
codeFences = true # enable render code block hook
noClasses = false # tells Hugo to use CSS classes instead of inline styles

layouts/_default/_markup/render-codeblock-mermaid.html

<pre class="mermaid">
  {{- .Inner | safeHTML }}
</pre>
<script type="module">
  import mermaid from '[](https://cdn.jsdelivr.net/npm/mermaid/dist/mermaid.esm.min.mjs';)
  mermaid.initialize({ startOnLoad: true }); 
</script>

Updated
Words1278
TagsRead4 minutes

Notes on OAuth 2.0

OAuth 2.0 is an open authorization framework that allows applications to access resources on behalf of a user, without exposing the user’s credentials. It enables secure access by providing access tokens for API requests, facilitating delegated permissions across services.

Security:

  • Tokens never exposed to browser
  • Session cookie only contains session ID
  • Session cookie typically expires in 24 hours or on browser close
  • Tokens stored securely on server

User Experience:

  • Seamless token refresh
  • No need to re-login when token expires
  • Single sign-on benefits

Phases

sequenceDiagram
    actor U as User
    participant C as Client App
    participant AS as Auth Server
    participant RS as Resource Server

    note over U,RS: Phase 1: Authorization Code Request
    U->>C: 1. Clicks "Login with Service"
    C->>AS: 2. Authorization Request
    Note right of C: GET /authorize?
client_id=123&
redirect_uri=https://app/callback&
response_type=code&
scope=read_profile&
state=xyz789 AS->>U: 3. Shows Login & Consent Page U->>AS: 4. Logs in & Approves Access AS->>C: 5. Redirects with Auth Code Note right of AS: GET /callback?
code=AUTH_CODE_123&
state=xyz789 note over U,RS: Phase 2: Token Exchange C->>AS: 6. Token Request Note right of C: POST /token
client_id=123&
client_secret=SECRET&
grant_type=authorization_code&
code=AUTH_CODE_123&
redirect_uri=https://app/callback AS->>AS: 7. Validates Auth Code AS->>C: 8. Returns Tokens Note right of AS: {
"access_token": "ACCESS_789",
"refresh_token": "REFRESH_111",
"expires_in": 3600
} C->>U: 9. Set session cookie Note right of C: Cookie contains
session ID only note over U,RS: Phase 3: Resource Access C->>RS: 10. API Request with Access Token Note right of C: GET /api/resource
Authorization: Bearer ACCESS_789 RS->>RS: 11. Validates Token RS->>C: 12. Returns Protected Resource note over U,RS: Phase 4: Token Refresh (When Access Token Expires) C->>AS: 13. Refresh Token Request Note right of C: POST /token
grant_type=refresh_token&
refresh_token=REFRESH_111&
client_id=123&
client_secret=SECRET AS->>C: 14. Returns New Access Token Note right of AS: {
"access_token": "NEW_ACCESS_999",
"expires_in": 3600
}

Token Refreshing Process

sequenceDiagram
    participant B as Browser
    participant C as Client (Your Server)
    participant AS as Auth Server (Google)

    Note over B,AS: Initial Login (happens once)
    B->>C: 1. User visits site
    C->>AS: 2. Redirect to Google login
    AS->>B: 3. Login page
    B->>AS: 4. User logs in
    AS->>C: 5. Auth code
    C->>AS: 6. Exchange for tokens
    Note right of C: Receives:
- Access Token (30 min)
- Refresh Token (long-lived)
- ID Token (user info) C->>B: 7. Set session cookie Note right of C: Cookie contains
session ID only Note over B,AS: Later: Access Token Expired B->>C: 8. Makes request C->>C: 9. Checks token
found expired C->>AS: 10. Uses refresh token
to get new access token AS->>C: 11. New access token C->>B: 12. Serves request Note right of B: User never sees
this process

Token Validation Process

sequenceDiagram
    participant B as Browser
    participant S as Server
    participant R as Redis/DB
    participant AS as Auth Server (Google)

    Note over B,AS: Scenario 1: Valid Token
    B->>S: 1. Request with session_id cookie
    S->>R: 2. Lookup user_id by session_id
    R->>S: 3. Returns user_id
    S->>R: 4. Get tokens for user_id
    R->>S: 5. Returns tokens
    S->>S: 6. Check token expiration
    Note right of S: Token still valid
    S->>B: 7. Return requested resource

    Note over B,AS: Scenario 2: Expired Token
    B->>S: 1. Request with session_id cookie
    S->>R: 2. Lookup user_id by session_id
    R->>S: 3. Returns user_id
    S->>R: 4. Get tokens for user_id
    R->>S: 5. Returns tokens
    S->>S: 6. Check token expiration
    Note right of S: Token expired!
    S->>AS: 7. Refresh token request
    AS->>S: 8. New access token
    S->>R: 9. Store new tokens
    S->>B: 10. Return requested resource

    Note over B,AS: Scenario 3: Invalid Session
    B->>S: 1. Request with invalid session_id
    S->>R: 2. Lookup user_id by session_id
    R->>S: 3. Returns null
    S->>B: 4. Return 401 Unauthorized

token flow

sequenceDiagram
    actor U as User
    participant B as Browser
    participant C as Client App (Your Server)
    participant R as Redis/DB
    participant AS as Auth Server (Google)
    participant RS as Resource Server

    note over U,RS: Phase 1: Initial OAuth Login
    U->>B: 1. Clicks "Login with Google"
    B->>C: 2. Request login
    C->>AS: 3. Authorization Request
    Note right of C: GET /authorize?
client_id=123&
redirect_uri=https://app/callback&
response_type=code&
scope=read_profile&
state=xyz789 AS->>B: 4. Shows Login & Consent Page U->>AS: 5. Logs in & Approves Access AS->>C: 6. Redirects with Auth Code Note right of AS: GET /callback?
code=AUTH_CODE_123&
state=xyz789 note over U,RS: Phase 2: Token Exchange & Session Setup C->>AS: 7. Exchange auth code for tokens Note right of C: POST /token
client_id=123&
client_secret=SECRET&
grant_type=authorization_code&
code=AUTH_CODE_123 AS->>C: 8. Returns Tokens Note right of AS: {
"access_token": "ACCESS_789",
"refresh_token": "REFRESH_111",
"expires_in": 3600
} C->>R: 9. Store tokens Note right of C: Store in Redis/DB:
user_id: user123
access_token: ACCESS_789
refresh_token: REFRESH_111 C->>B: 10. Set session cookie Note right of B: Cookie contains only:
session_id: "sess_xyz" note over U,RS: Phase 3: Subsequent API Requests B->>C: 11. Request with session cookie Note right of B: Cookie: session_id=sess_xyz C->>R: 12. Look up user_id & tokens R->>C: 13. Return tokens C->>RS: 14. API request with access token Note right of C: Authorization: Bearer ACCESS_789 RS->>C: 15. Return resource C->>B: 16. Send response to browser note over U,RS: Phase 4: Token Refresh (Automatic) B->>C: 17. Later request with session cookie C->>R: 18. Look up tokens R->>C: 19. Return tokens (expired) C->>AS: 20. Refresh token request Note right of C: POST /token
grant_type=refresh_token
refresh_token=REFRESH_111 AS->>C: 21. New access token C->>R: 22. Update stored tokens Note right of C: Update Redis/DB with
new access_token C->>RS: 23. Retry API request RS->>C: 24. Return resource C->>B: 25. Send response Note right of B: Same session cookie
continues to work

Updated
Words991
TagsRead3 minutes

Programming concepts in SQL

CTEs (Common Table Expressions) as Variables/Functions

  • Just like how you declare variables or functions in traditional programming, CTEs let you create named result sets that you can reference later
  • Each CTE can be thought of as storing an intermediate result, similar to variable assignment
  • They can be chained together like function calls, where one CTE uses results from another

GROUP BY as Loops

  • In traditional programming, you might loop through data to accumulate results
  • GROUP BY automatically “loops” through rows, grouping related records together
  • The aggregation functions (SUM, COUNT, AVG) work like accumulators inside these implicit loops

Example in programming vs SQL:

# Python equivalent
results = {}
for row in data:
    if row.customer_id not in results:
        results[row.customer_id] = {
            'total_transactions': 0,
            'total_spend': 0
        }
    results[row.customer_id]['total_transactions'] += 1
    results[row.customer_id]['total_spend'] += row.quantity * row.unit_price
-- SQL equivalent using GROUP BY
SELECT 
    customer_id,
    COUNT(*) as total_transactions,
    SUM(quantity * unit_price) as total_spend
FROM sales
GROUP BY customer_id

CASE Statements as If/Switch

  • Similar to if/else or switch statements in programming
  • Can be used in both SELECT and WHERE clauses
  • Can be nested for complex conditional logic

Example comparison:

# Python equivalent
def get_spending_tier(total_spend):
    if total_spend >= 10000:
        return 'Premium'
    elif total_spend >= 5000:
        return 'Gold'
    elif total_spend >= 1000:
        return 'Silver'
    else:
        return 'Bronze'
-- SQL equivalent using CASE
CASE 
    WHEN total_spend >= 10000 THEN 'Premium'
    WHEN total_spend >= 5000 THEN 'Gold'
    WHEN total_spend >= 1000 THEN 'Silver'
    ELSE 'Bronze'
END as spending_tier

Advanced Programming-like Features

  1. Window Functions as Iterators:

SELECT 
    *,
    LAG(value) OVER (ORDER BY date) as previous_value,
    LEAD(value) OVER (ORDER BY date) as next_value
FROM data

This is similar to accessing previous/next elements in an array iteration.

  1. Recursive CTEs as While Loops
WITH RECURSIVE countdown(val) AS (
    SELECT 10  -- Initial value
    UNION ALL
    SELECT val - 1 FROM countdown WHERE val > 1  -- Loop condition and iteration
)
SELECT * FROM countdown;
  1. HAVING as Filter After Processing: Like applying conditions after a loop completes:
SELECT category, COUNT(*) as count
FROM items
GROUP BY category
HAVING COUNT(*) > 10

Key Insights:

Declarative vs Imperative:

Traditional programming is imperative (you specify HOW to do something) SQL is declarative (you specify WHAT you want) The SQL engine optimizes the execution plan

Set-based Operations:

Instead of thinking about individual records, think in sets Operations like GROUP BY process entire sets of data at once This often leads to better performance than record-by-record processing

Composition:

CTEs allow you to break complex logic into manageable pieces Each CTE can build upon previous ones, similar to function composition This promotes code reusability and maintainability