Exposing Go code to Android and Python

Laurent explains how he exposed a Go codebase to an Android app and a Python server.

Why ?

His main motivation was that his code had to be executed on both an Android device and a Django server, on the same datasets. Before this work, the logic was implemented twice, in Python and in Java. After a few months, this led him to (at best) inconsistent results and (at worst) inconsistent bugs.

His team at EcoCO2 putted him in charge to write the code once and for all, in Go, and make it run on both targeted platforms. It was quite a challenge but it can be made really easy! Let's see.

Exposing your Go code to other platforms

From Go to Android

To bring some Go code to Android, it's quite straightforward: install gomobile, just run gomobile bind and it will build an Android archive out of your Go package.

Problem #1: no slices

There's one limitation, though: you can only use a subset of Go types. What's really a pain is that you can't use slices - except byte slices.

From Go to Python

To bring Go code to Python is a different story: you can run go build buildmode=c-shared. Yes : you are building a C shared library, because your data will go from Python to C to Go, and then from Go back to C and to Python.

So it means that you have to write some C code, containing Python headers, pointers, memory mapping, etc... not cool. Actually, your Go code will also have CGo headers, memory mapping and unsafe pointers, before you finally can wrap your API.

Generating Python wrapper code

The glue code for Python is quite complex and you don't want to write it more than once. It's time to automate!

Go to the rescue

To write this code for you, Go provides several tools. You should parse your Go code with the go/parse package and navigate your methods/types programmatically using go/ast. Then, construct whatever output you want with text/template and automate it with a //go:generate directive.

Problem #2: too many types

To be fair, you cannot automate it completely because each time you introduce a new method signature, you will need to write some code to handle it in C.

Using serialization to solve our problems

To solve/avoid the problems that we found, we can serialize your API's inputs and outputs!

What do you mean by serialization ?

With serialization, the only type you need to pass around is strings. It's cool with gomobile and minimizes the C code you need to write: there's only one data type and one method signature, all in all. But you will have to write serialization/deserialization code which is bad.

Protobuf

Enter Protobuf. Protobuf is a language that you can use to describe your model. It provides some tools to generate implementations for your types and for their serializers/deserializers in a lot of languages (Go, Java, Python, JS,...).

Hacking gRPC to automate further

Even nicer: you can describe your API methods! This was designed to be used for gRPC, which normally generates HTTP server/clients implementations. We can use the same logic to make the protobuf compiler generate our library stubs!

How does the protobuf compiler work? The protobuf compiler is called protoc and calls generators following a simple convention: if you call protoc --XX_out=..., it will call whatever program is in your $PATH and called protoc-gen-XX.

Therefore, the mainstream Go code generator is called protoc-gen-go. Guess what: it's opensource, written in Go and what you will find out if you read its source is that it has a plugin system (which was, in fact, created specifically for grpc, it seems)!

It means that we can create a plugin for protoc-gen-go and register it so that when a .proto file is compiled, protoc-gen-go will parse the file, navigate the model and call our plugin on each type/method to let it generate whatever we want.

The whole process

Let's put everything together.

• From a description of your model+API in Protobuf, you can generate some stubs (you have to write a stub generator)
• From those stubs, you can implement your API.
• From your implementation, you can generate a Python-compatible package (you have to write a wrapper generator)
• Finally, you can use standard build commands to generate Python and Android libraries.

The big win? Most of this is pure Go, so you don't have to switch multiple times back-and-forth between languages!

Conclusion

There's a lot of information, but to recap:

• if you build a cross-platform lib, using protobuf will make it easier, by providing you serialization
• the standard Go tools will help you to generate your code automatically using the abstract syntax tree and templating
• note that the approach shown for Python can work with a lot of languages, as long as they can be bound to C.

Show me some code!

Laurent wrote two packages to make your life easier:

• a stub generator (protobuf => Go)
• a wrapper generator (Go => Python)

You can use those right away or study/contribute to them on Github!

Diving deeper

A few useful links:

• The GoPy project, which inspired me this work: https://github.com/go-python/gopy
• The go-mobile-collection project, which solves the gomobile problem another way: https://github.com/scisci/go-mobile-collection
• The doc for extending Python with C: https://docs.python.org/3.6/extending/extending.html
• Writing a protobuf plugin the hard way: https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.compiler.plugin

Thanks

Huge thanks to Nina, Max & Aimad for helping Laurent shape this talk and to his team @EcoCO2 for supporting him in this project.