how to consume python logs inside a container with Datadog
This weekend I was adding finishing touches to photoRake. Deploying it to railway has been delightful. They're cheaper, faster, and easier to use than the alternatives. You give them a docker image, and they handle the rest.
That said, setting up Datadog for log ingestion was neither obvious nor trivial. Datadog suggests running their agent in a sidecar container, which is sadly not an option for railway apps as of yet. I needed to run my agent inside the container.
I opted to simply install the agent within my docker container. It bloats your container a little bit, but I'd rather do that than manage my own container cluster.
Here's how I got Datadog log ingestion working for photoRake:
- in your Dockerfile, add the following to install the agent:
This will install the agent within a linux based docker image (my base image for photoRake is python:3.9, which runs on linux).
2. Configure the agent with a
datadog.yaml file. My file was copied from the official Datadog repo, with a single modification (line 3) to enable log ingestion .
3. Make a
conf.yaml to help the agent find your application logs. Since photoRake is a python app, I followed these conf.yaml instructions.
3. Copy your configs to their respective locations in your Dockerfile
4. And finally, make sure you chown the copied config files, so the datadog agent can use them
summarize, here is my Dockerfile with the above changes:
FROM python:3.9 ... # Datadog setup RUN DD_AGENT_MAJOR_VERSION=7 DD_API_KEY=<API_KEY> DD_SITE="datadoghq.com" bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script.sh)" COPY conf.yaml /etc/datadog-agent/conf.d/python.d/conf.yaml COPY datadog.yaml /etc/datadog-agent/datadog.yaml RUN chown dd-agent:dd-agent /etc/datadog-agent/datadog.yaml /etc/datadog-agent/conf.d/python.d/conf.yaml ...
And that's it - you've set up a Datadog agent for log consumption within a container.