Things I like less in life: Servers locking up for hard to explain reasons. Somewhere between a weird Ceph object storage daemon, a coredump handler that freezes and general k8s madness lies the explanation why some nodes end up with a constant iowait, no responses to ps/htop and general madness unless rebooted. I can’t begin to explain how annoyed I am with this looming threat of this happening to a cluster that’s not only hosting internal tooling but real production services.

@monoxyd https://DeepL.com macht bei sowas gemeinhin einen guten Job (bin evtl. biased weil es meine Brötchen zahlt)

@fluepke @isotopp spannend. Das kann evtl. ausreichen um auf Wärmepumpen zu wechseln ohne die Heizkörper zu tauschen.

@343max Weigert sich leider mit meiner Akkoma-Instanz zu reden (die eine Mastodon-kompatible API anbietet).

Back after 2 days of business trip in Amsterdam. I managed to see basically nothing of the city - arrived Sunday night after a delayed train,went right after the training. Not my favorite thing to do.

@nova To add to that: One really has to see to believe how hard a process can get throttled - imagine a well parallelized application seeing all 256 threads my k8s workers offer - and then only being allowed 1-2 of those.

@nova TBH shouldn't even be the developers per se, but the underlying runtimes doing it right. For example .net is aware of the cgroups under k8s and just sets it's threadpools up properly, golang on the other hand is of the opinion that every damn CPU core it can see belongs to it, no matter the cgroups - Uber gladly fixed it with a lib: https://github.com/uber-go/automaxprocs

@nova I would tend to go with @sindarina argument here. 1.4 metric tons for the round trip New York / Keflavik (didn’t check where you are based, so took sth close) - that’s the rough equivalent of 11MWh in Finnland when compared to Iceland (150g/kWh in .fi, 30g/kWh in .is). From that perspective somewhere managed by remote hands is far more green than Iceland but flying there to do sth with it - even when that location isn’t 100% renewable power.

@cablespaghetti @nova Fun fact: We stayed online, the operator of the cable systems even did a test-burial right where the lava would go (it flowed super slow, luckily) to see if the cables can survive it. IIRC they didnt't, but there always was a second link through a different valley or sth. Very weird to watch though, when you regularly get mails updating you of volcanic activity and where the lava goes :D

@nova I feel those requirements and Iceland don't mix suuuuper well sadly, I could also only find Floki who don't seem to even publish pricing for additional traffic.

I'd personally have a look around northern Europe. Not quite as independent, but there are some bigger players in the size range you are looking for. Iceland is sadly very full of bitcoin farms and big players like us buying in bulk, that seems to make it really unattractive to host individuals for a reasonable price.

There's nothing like remote-debugging your parents smart home. Something broke the powerline link to the garage, now their Home Asisstant instance can't set wallbox charging limits anymore m) Time to build an ESP32 based system with local control that at least has a "just charge it" button. Also: Stupid freaking wallbox that needs constant modbus communications to work. How hard wold a "just do what I told you last time" flag be. I know, for coordinating multiple wallboxes and keeping power limits it's safer to not charge if comms are down, but I'm only doing PV optimization here, not load limiting for any critical stuff.

@nova We work with Verne, they subcontract parts of our infra to atNorth. We are talking Megawatt-scale though, I fear they only bother from a few racks up. Floki.net looked good, only saw half of your last stream though, I guess it's a problem of traffic?

Living that corporate swag life

Nachdem ich im Oktober bereits die vierte Impfung gegen Corona mit dem angepassten bivalenten Impfstoff hatte (Walk-in im einzigen noch geöffneten Impfzentrum in Berlin am Ring Center im Friedrichshain), habe ich diese Woche dann auch mal endlich die Influenza-Impfung beim Hausarzt abgehakt.

Ich schreibe das, damit es in der Timeline steht und vielleicht Menschen denken „Ach ja, gute Idee, eigentlich.“

@Twitch @nova I happen to be working for a company that operates a medium sized GPU HPC cluster in Iceland. There are a few things to keep in mind with Iceland: Lava and shipment. We now had 2 vulcanite fissures since I joined where there was lava on a path that crosses the main fiber runs stretching from the south end of the island where atNorth and Verne are located towards Reykjavik - not fun. Same goes for shipping: Getting stuff there is just annoying, especially servers, with their pesky RTC backup batteries. The rest is amazing though. We btw rarely go there and have the amazing people from Verne handle all on-site ops for us. Of course that’s mostly a company thing, 100+ $/h is still cheap compared to flying employees out from Europe.

@wortkomplex Beeindruckend. Sehen aus wie ein Klon von meinen beiden.

Caturday!

@leah Die Debatten die ich wegen Disk IO hatte, wow, ja. "Der Ceph-Cluster is langsam" - ja , der Ceph-Cluster besteht aus 3-fach redundanten HDDs die über Ethernet verbunden sind :D Wie soll es latenzarm sein?

Freude auf den neuen NVMe-only Ceph-Cluster ist real 🤓

@nova I'd love that, running a server basically for myself right now it would be great to have a bit of a federated bubble that's fun to read through.

@rami Nutze privat Home Assistant. Am elegantesten für dein Problem ist das IMHO mit ZigBee und billigen ZigBee Thermometern wie https://amzn.eu/d/0FoOtnk - keine Kabel, aber ab und an Knopfzellen tauschen (gefühlt deutlich < 1/Jahr). Braucht einen Raspi oä, einen ZigBee-Stick und beliebige Thermometer der Art.