Django-socio-grpc (DSG) is a framework for using gRPC with Django. It builds upon django-rest-framework (DRF), making it easy for those familiar with DRF to get started with DSG.
Although I decided to go back to DRF after exploring DSG, I chose to do so because I needed to get things done quickly. Using gRPC is considered a potential way to achieve performance gains, and there are some obstacles need to be addressed before going full gRPC. I’m leaving these notes as my learning experience.
The workflow
Using django-socio-grpc (DSG) the workflow is like following:
flowchart TD subgraph backend[Backend Side] db-schema[Design Database Schema] --> |create| django-models[Django Models] --> |define| serializers pb-backend[Protocol Buffers] django-models --> server-app[Server Application] django-models --> |migrate| database end subgraph protobuf-repo[Protobuf repository] .proto[.proto] --> |run| buf-lint[buf lint] --> |detect| buf-check[Breaking Change] --> |if pass|protobuf-gen[Generate Code] protobuf-gen --> server-stub[gRPC Server skeleton] protobuf-gen --> client-stub[gRPC Client Stub] protobuf-gen --> mypy-types[.pyi type stub] end subgraph frontend[Frontend Side] pb-frontend[Protocol Buffers] client-app[Client Application] --> |call| client-stub end serializers --> |DSG generates| .proto server-app --> |implement| server-stub server-stub --> |serializes| pb-backend mypy-types --> server-app client-stub --> |serializes| pb-frontend pb-backend <--> |binary format over HTTP/2| pb-frontend
Make DSG aware of big integer model fields
Currently DSG (version 0.24.3) has an issue of mapping some model fields to int32 type in protobuf incorrectly due to DRF’s decision, they should be mapped to int64 type.
It’s kind of hard to implement at library level, in application level I implemented something like this, using BigIntAwareModelProtoSerializer as the parent class of the proto serializer will correctly map BigAutoField BigIntegerField PositiveBigIntegerField to int64 type in protobuf.
from django.db import models
from django_socio_grpc import proto_serializers
from rest_framework import serializers
class BigIntegerField(serializers.IntegerField):
"""Indicate that this filed should be converted to int64 on gRPC message.
This should apply to
- models.BigAutoField.
- models.BigIntegerField
- models.PositiveBigIntegerField
rest_framework.serializers.ModelSerializer.serializer_field_mapping
maps django.models.BitIntegerField to serializer.fields.IntegerField.
Although the value bounds are set correctly, django-socio-grpc can only map it to int32,
we need to explicitly mark it for django-socio-grpc to convert it to int64.
"""
proto_type = "int64"
# First, get the metaclass of ModelProtoSerializer
ModelProtoSerializerMetaclass = type(proto_serializers.ModelProtoSerializer)
class BigIntegerSerializerMetaclass(ModelProtoSerializerMetaclass):
def __new__(mcs, name, bases, attrs): # noqa: D102 ANN001 ANN204 N804
# First, let's modify the serializer_field_mapping if it exists
for base in bases:
if hasattr(base, "serializer_field_mapping"):
# Create a new mapping dictionary inheriting from the base
field_mapping = dict(base.serializer_field_mapping)
# Update the mapping for BigInteger fields
field_mapping.update(
{
models.BigIntegerField: BigIntegerField,
models.BigAutoField: BigIntegerField,
models.PositiveBigIntegerField: BigIntegerField,
}
)
# Add the modified mapping to attrs
attrs["serializer_field_mapping"] = field_mapping
break
# Then proceed with the normal class creation
return super().__new__(mcs, name, bases, attrs)
class BigIntAwareModelProtoSerializer(proto_serializers.ModelProtoSerializer, metaclass=BigIntegerSerializerMetaclass):
"""A ModelProtoSerializer that automatically converts Django BigInteger fields to gRPC int64 fields by modifying the field mapping."""
Major obstacles for using gRPC
The biggest obstacle is that browsers do not natively support gRPC, which relies on HTTP/2.0. As a result, client-side frontend calls to backend services from a browser using gRPC require a proxy, typically Envoy. This setup involves additional overhead, such as configuring a dedicated API gateway or setting up an ingress. Even with a service mesh like Istio, some extra work is still necessary.
The next challenge is how to corporate with existing RESTful services if we chose to add a gRPC service. For communications happen between RESTful service and gRPC service, a gRPC-JSON transcoder (for example Enovy ) is need so that HTTP/JSON can be converted to gRPC. Again some extra work is needed at infrastructure level.
The last part of using gRPC is that data is transferred in binary form (which is the whole point of using gRPC for performance) makes it a little bit harder for debugging.
Conclusion
Django-socio-grpc is solid and its documentation is good. However, the major issue is the overhead work that comes with using gRPC. I will consider it again when I need extra performance and my team’s tech stack is adapted to gRPC.
