Accelerating object serialization by using constraints

About me

• Backend developer.
• ML engineer (former Google Developer Expert).
• Coded in the last 12 months: Python, Go, C/C++, JavaScript.
• Work in Athenian.

Problem

• Backend materializes a big pandas DataFrame.
• Must serialize and submit that DataFrame to memcached.
• pickle.dumps(), to_arrow() are very slow
  • Lots of string columns.
  • datetime and timedelta columns.
  • Nested lists and dicts.
  • Nested numpy arrays of variable length, incl. object dtype.

Custom binary serialization

• 👎 Not portable (CPU), based on memcpy of the internals.
• 👎 Not portable (OS), assumes Linux.
• 👎 Not portable (interpreter), assumes CPython 3.11.x.
• 👎 Not backward-compatible.
• 👎 Not universal, doesn't support all object types.

Custom binary serialization

• 👍 Compact Cython code without feature bloat.
• 👍 Single-pass. Very fast. I mean it.
• 👍 Releases GIL (Cython: nogil).

Releasing GIL

• Cython C API wrappers are disabled, use raw PyObject *.
• 100% sure the models are not accessible anywhere else.
• Only read-only, manually reviewed operations with the underlying objects. Must reimplement the rest.
• GC cannot relocate objects.

Approach to pandas

dtype object

assert PyArray_IS_C_CONTIGUOUS(arr)
        assert PyArray_NDIM(arr) == 1
        assert PyArray_DESCR(arr).kind == b"O"
         
        cdef PyObject **data = <PyObject **> PyArray_DATA(arr)
        for i in range(PyArray_DIM(arr, 0)):
            serialize(data[i])

stdlib containers

for i in range(PyList_GET_SIZE(obj)):
            serialize(PyList_GET_ITEM(obj, i))
         
        while PyDict_Next(obj, &pos, &key, &val):
            serialize(key)
            serialize(val)

Integers, floats

double PyFloat_AS_DOUBLE(PyObject *)
        long PyLong_AsLong(PyObject *)
        void PyArray_ScalarAsCtype(PyObject *scalar, void *ctype)
         
        memcpy(buffer, &value, sizeof(value))

Strings

Internal representation: smart UCS1, UCS2, or UCS4.

memcpy(
            buffer,
            PyUnicode_DATA(obj),
            PyUnicode_GET_LENGTH(obj) * PyUnicode_KIND(obj)
        )

datetime and timedelta

PyDateTime_CAPI: internal representation is a struct, not a timestamp

int PyDateTime_GET_YEAR(PyObject *)
        int PyDateTime_GET_MONTH(PyObject *)
        int PyDateTime_DATE_GET_SECOND(PyObject *)
         
        int PyDateTime_DELTA_GET_DAYS(PyObject *)
        int PyDateTime_DELTA_GET_SECONDS(PyObject *)

Model

@dataclass(slots=True)
        class Movie:
            name: str
            rating: float
            actors: list[Actor]

Problem

Typical paginated request without DB pushdown:

blob = await load_from_cache(key)
        movies = deserialize(blob)
        selected = movies[offset:offset + limit]
        return to_json(selected)

Problem

Slow as hell with our "movies":

blob = await load_from_cache(key)
        movies = deserialize(blob)
        selected = movies[offset:offset + limit]
        return to_json(selected)

Problem - attempt to fix

First call:

movies = await bake_movies(...)
        dicts = to_atoms(movies)
        await store_to_cache(dicts, key)
        return to_json(dicts[:limit]), key

Problem - attempt to fix

n + 1 call:

blob = await load_from_cache(key)
        dicts = deserialize(blob)
        return to_json(dicts[offset:offset + limit]), key

Problem - attempt to fix

First call:

movies = await bake_movies(...)
        dicts = to_atoms(movies)
        await store_to_cache(serialize(dicts), key)
        return to_json(dicts[:limit]), key

Problem - attempt to fix

n + 1 call:

blob = await load_from_cache(key)
        dicts = deserialize(blob)
        return to_json(dicts[offset:offset + limit]), key

to_json_vadim(movies)

'[{"name": "RRR", ...},{"name": "Up", ...},{...'
        [ 1,                  100,                200]
        

Problem - my ideal solution

First call:

movies = await bake_movies(...)
        blob, toc = to_json_vadim(movies)
        await store_to_cache(serialize((blob, toc)), key)
        selected = blob[:toc[limit] - 1]
        return f'{{"key": "{key}", "movies": [{selected}]}}'

Problem - my ideal solution

n + 1 call:

blob, toc = deserialize(await load_from_cache(key))
        selected = blob[toc[offset]:toc[offset + limit] - 1]
        return f'{{"key": "{key}", "movies": [{selected}]}}'

to_json_vadim(movies)

1. Make the serialization specification.
2. Write each movie according to the spec.
3. Reimplement many routines:
   • list, dict
   • int, float to str
   • datetime, timedelta to str
   • str to str: 'esc"ape\n' -> r'"esc\"ape\n"'; utf8

Main trick with __slots__

Serialization spec (Cython)

ctypedef struct SpecNode:
            DataType type
            Py_ssize_t offset
            PyTypeObject *model
            vector[SpecNode] nested

cdef enum DataType:
            DT_INVALID = 0            DT_DT = 7
            DT_MODEL = 1              DT_TD = 8
            DT_LIST = 2               DT_BOOL = 9
            DT_DICT = 3
            DT_LONG = 4
            DT_FLOAT = 5
            DT_STRING = 6

Model - precomputed spec

@constrained_json_spec
        @dataclass(slots=True)
        class Movie:
            name: str
            rating: float
            actors: list[Actor]
            # __json_spec__: ClassVar[PyCapsule]

Advice

• Doing less is doing more.
• Know your tools, learn internals.
• Going low-level is your super weapon.
• But only if you get the architecture right.
• With great power come annoying consequences.

Thank you

• vadim@athenian.co
• @vadimlearning
• @vmarkovtsev
• @vadimlearning

bit.ly/3kM6zcz