JSON logging on App Engine (Flexible, Custom runtimes)

8 Jan 2017

In a flexible App Engine instance, the documentation states that we can output to STDOUT and STDERR from our app, and the output is sent to Cloud Logging and can be seen in Log Viewer.

But I don't want unstructured logging or raw text logging. I want JSON logging, which is much better.

I tried to log a JSON string to STDOUT. But that didn't work because, in Log Viewer, it put the JSON string in the field textPayload.

We need a better way.

In an appengine instance, the fluentd logging agent is configured with managed_vms.conf. The agent is not configured to accept TCP/UDP packet. Therefore, we can't use fluent-logger-scala.

But managed_vms.conf shows that we can write a JSON log to /var/log/app_engine/app/app*.json.

I ssh'ed into an appengine instance and ran echo '{"test": "by-tanin"}' >> /var/log/app_engine/app/app-test.json. And it worked! The log entry is shown in Log Viewer with the field jsonPayload.

The journey wasn't over yet. Our appengine ran within a docker container. We had to figure out how to log to /var/log/app_engine/app/app-test.json from an app within a docker container.

Docker offers a way to mount a host path to a path inside docker container. But this is not configured in Dockerfile. Mounting is configured through an argument of docker run. So, I needed to figure out what the docker run looked like or see its conifguration. And I managed to figure it out (yay!).

The article, Debugging an instance, offers a way to go into a docker container.

After running sudo docker ps, I saw this line:

c40b39fda70b us.gcr.io/test-tanin-selenium/appengine/[email protected]:18b7e6c9c328f22232022535d8b1ef55eb4cad06bae3c5e276d0090768bd9c7c "./docker-entrypoint." 8 hours ago Up 8 hours gaeapp

That looks like my app container because of the word 'gaeapp'. Now I ran docker inspect c40b39fda70b and saw:

"Mounts": [ { "Source": "/var/log/app_engine/app", "Destination": "/var/log/app_engine", "Mode": "rw", "RW": true, "Propagation": "rprivate" }, ... ]

Bingo! I ran echo '{"test": "from-within-container"}' >> /var/log/app_engine/app-test.json, and the line showed up in /var/log/app_engine/app/app-test.json on the host.

This means that our Play application can log JSON strings to /var/log/app_engine/app-play.json. Yay!