Back to Telegraf

I am in the middle of producing the podcast, which we are doing together with vanadium23.me, it is called “Два Ивана (название обсуждается)”. One of the things I had to solve was mp3 files hosting.

I went for existing solutions and planned to use Cloudflare as much as possible, because it is what I use for all my DNS needs. R2 was compelling, but it required a credit card. The problem is that I am in Russia and there is no way to use our credit cards on foreign websites. My backup Georgian card is too valuable to waste if I am suddenly going over the free tier. So I looked for other kind-of-free solutions. Thought about Backblaze and others, but then I reallised it shouldn’t be too much trouble to run an S3 server on my own server.

You probably already figured it out, that I am going to write about MinIO, which is open source, written in Go and gained lots of popularity in recent years.

Well, I had to take a little detour and look for the alternative solutions. I stumbled onto Garage. It is written in Rust and has small memory footprint, which is basically the main criteria for a new self-hosted app for me. The docs for Garage are okay. I spent around 30 min tuning my Ansible role for it. I deployed it in a way that it has to external access points: s3.doubleivan.ru and s3.biozz.dev. Because I wanted an S3 app that I can use for my Django experiments. S3 is like a makeshift CDN, if you will.

The main takeaways from the Garage are:

• you don’t need the “web” part from it if you don’t plan to use it a static website hosting
• it doesn’t support anonymous access at the time of writing this

Yeah, the second one killed Garage for me. I had to go for MinIO, because there is no point in CDN if it can’t serve the files anonymously.

I haven’t mentioned Telegraf yet, I know. Stay with me.

So I switched to MinIO, did the same setup as for the Garage with two external hosts and one extra for a nice admin panel. I can close my eyes on the 125 MiB RAM usage versus 24 MiB in Garage’s case, when I see this nice Web UI.

But I didn’t know what was the exact memory usage of MinIO when I deployed it. I couldn’t get cAdvisor to detect the container name. Or maybe it was some MinIO container magic, that deleted it. I couldn’t get my metrics working for it. And I also noticed that my network stats of containers looked suspiciously low.

That was enough for me to say goodbye to cAdvisor and say hello again to Telegraf.

They had a few deprecated options in Docker input plugin, but other than that it was straightforward. I bumped the version to 1.27.1 and deployed it. It didn’t work. The logs said that it didn’t have access to the docker socket. It was a new for me. But a quick google search solved it. I had to put user UID into the config to make sure it had access.

Oh, and they also built arm64 images, so I was happy to delete my custom build step in the Ansible role.

Working with metrics in Grafana was much easier, because Telegraf made very easy. They have “usage”, they have “totals”. You don’t have to make this insanely long rate(usage_x + usage_y - usage_z)[something] query to get started.

I have only Docker input plugin enabled in Telegraf and Prometheus for exporting. We will see how it holds up over time.

My setup before Telegraf used about 82 MiB of RAM:

• node_exporter - 18 MiB
• postgres_exporter - 14 MiB
• cadvisor - 50 MiB

Now with Telegraf taking 40 MiB sans cAdvisor - 72 MiB. Which is good. I will try to use Telegraf for everything again. We will see how it is going to work out for me.

There is one last thing, a quick tip for my fellow Cloudflare users. When you set up a new domain make sure to change SSL/TLS mode to Full (strict), otherwise you will git weird endless redirection behavior when services behind you proxy require HTTPS. It happened to me with MinIO.