Building a library in Go that can be used in Python or Android can be difficult. Building one that can be used for both at the same time is really hard.
Laurent presents a way to simplify the problem and automate most of the process, letting you focus on your business logic.
Laurent explains how he exposed a Go codebase to an Android app and a Python server.
His main motivation was that his code had to be executed on both an Android device and a Django server, on the same datasets. Before this work, the logic was implemented twice, in Python and in Java. After a few months, this led him to (at best) inconsistent results and (at worst) inconsistent bugs.
His team at EcoCO2 putted him in charge to write the code once and for all, in Go, and make it run on both targeted platforms. It was quite a challenge but it can be made really easy! Let's see.
To bring some Go code to Android, it's quite straightforward: install gomobile, just run
gomobile bind and it will build an Android archive out of your Go package.
There's one limitation, though: you can only use a subset of Go types. What's really a pain is that you can't use slices - except byte slices.
To bring Go code to Python is a different story: you can run
go build buildmode=c-shared. Yes : you are building a C shared library, because your data will go from Python to C to Go, and then from Go back to C and to Python.
So it means that you have to write some C code, containing Python headers, pointers, memory mapping, etc... not cool. Actually, your Go code will also have CGo headers, memory mapping and unsafe pointers, before you finally can wrap your API.
The glue code for Python is quite complex and you don't want to write it more than once. It's time to automate!
To write this code for you, Go provides several tools. You should parse your Go code with the
go/parse package and navigate your methods/types programmatically using
go/ast. Then, construct whatever output you want with
text/template and automate it with a
To be fair, you cannot automate it completely because each time you introduce a new method signature, you will need to write some code to handle it in C.
To solve/avoid the problems that we found, we can serialize your API's inputs and outputs!
With serialization, the only type you need to pass around is strings. It's cool with gomobile and minimizes the C code you need to write: there's only one data type and one method signature, all in all. But you will have to write serialization/deserialization code which is bad.
Enter Protobuf. Protobuf is a language that you can use to describe your model. It provides some tools to generate implementations for your types and for their serializers/deserializers in a lot of languages (Go, Java, Python, JS,...).
Even nicer: you can describe your API methods! This was designed to be used for gRPC, which normally generates HTTP server/clients implementations. We can use the same logic to make the protobuf compiler generate our library stubs!
How does the protobuf compiler work? The protobuf compiler is called
protoc and calls generators following a simple convention: if you call
protoc --XX_out=..., it will call whatever program is in your
$PATH and called
Therefore, the mainstream Go code generator is called protoc-gen-go. Guess what: it's opensource, written in Go and what you will find out if you read its source is that it has a plugin system (which was, in fact, created specifically for grpc, it seems)!
It means that we can create a plugin for protoc-gen-go and register it so that when a .proto file is compiled, protoc-gen-go will parse the file, navigate the model and call our plugin on each type/method to let it generate whatever we want.
Let's put everything together.
The big win? Most of this is pure Go, so you don't have to switch multiple times back-and-forth between languages!
There's a lot of information, but to recap:
Laurent wrote two packages to make your life easier:
You can use those right away or study/contribute to them on Github!
A few useful links:
Huge thanks to Nina, Max & Aimad for helping Laurent shape this talk and to his team @EcoCO2 for supporting him in this project.