Hello,

we’re currently experiencing some issues due to Lemmy containers frequently running out of file descriptors, and we haven’t had the time to properly investigate the root cause yet. For the time being limits have been increased and we’ve already triggered multiple restarts over the past few days to reset the consumption.

this was not our instance automod, please contact the community mods instead

the community had been removed due to the creator being banned and their content having been removed along with it, including their communities. would you like to take it over?

You should persist the auth token, rate limits are by IP. They also don’t exclude successful logins currently. This is a limitation in Lemmy. We have currently configured it to allow 5 logins per IP per day, but if you’re lucky you can exceed that, as the consumption is not synchronized between backends at this time.

Then I don’t know where you’re seeing Cloudflare CAPTCHAs. There shouldn’t be any on any of the API paths.

I’ve checked both URLs you mentioned above and only saw a Python user agent getting blocked with a browser integrity check on one of them in our CF logs. I’ve now added a dedicated rule to skip browser integrity checks on all API paths.

Hello,

use the official Lemmy API for automated interactions, don’t use headless browsers or pretend to be a browser.

Given the amount of false notices in the past (not to us, in general), there is absolutely a ton of automation without human oversight in there, and that was already the case long before AI.

Companies like “CTW Anti-Piracy 2” and “Altitude Film Sales” or "Altitude Film Sales 2” (not sure if there are multiple types of Altitude Film Sales, the Lumen notice lists them with a 2 but the complaint text without) clearly don’t care about the damage they do to their own reputation. In the end, most people won’t care anyway and it won’t meaningfully harm their reputation even if they fuck up on a significantly larger scale. It’s not like they’d have to fear any legal repercussions for these false statements either, as the only part they swear under penalty of perjury is that they are authorized by the copyright holder.

Hi, it’s transferred back.

I think Lemmy might still have some federation issues with that, it’s unlikely to be specific to LW or .ca.

There is a chance it would have fixed itself after some time (24h?), I’m not sure if a mod list refresh is part of the community metadata refresh.

doesn’t seem to be the case, at least there is not enough information to link the two cases.

Hi,

the immediate issue is resolved, and we’re looking to extend our monitoring in the coming days to have better insights on the underlying issue, as well as alerts before it would get to the point of throwing errors.

For some reason Lemmy has been having significantly more open file handles (including e.g. connections) than usual, leading to a denial of service. At this point it’s unknown whether this is caused by a bug in Lemmy, an attack, or something else.

Unlikely, we haven’t changed these limits recently and we haven’t exceeded the limits in the past. While it’s possible to change the limits, it’s not clear yet why they are being exceeded currently.

Hi,

sorry for the late response to this. The immediate issue is resolved, and we’re looking to extend our monitoring in the coming days to have better insights on the underlying issue, as well as alerts before it would get to the point of throwing errors.

For some reason Lemmy has been having significantly more open file handles (including e.g. connections) than usual, leading to a denial of service. At this point it’s unknown whether this is caused by a bug in Lemmy, an attack, or something else.

Hi,

sorry for the late response to this. The immediate issue is resolved, and we’re looking to extend our monitoring in the coming days to have better insights on the underlying issue, as well as alerts before it would get to the point of throwing errors.

For some reason Lemmy has been having significantly more open file handles (including e.g. connections) than usual, leading to a denial of service. At this point it’s unknown whether this is caused by a bug in Lemmy, an attack, or something else.

I don’t know if they changed the server

it’s still the same one.

this is not only used for lemmy.world, but also mastodon.world

this is not correct - they have the same specs i believe, but it’s 2 separate servers.

by now, it probably is also used for other services the Fedihosting foundation hosts

that is also not correct. pretty much all our services are isolated on VM or physical box level currently, although we are looking into a potential migration to kubernetes to use our hardware resource more efficiently, as we have a lot of unused resources with our current setup due to downscaling being a bit challenging with hardware and with the current market in general. lemmy.world and mastodon.world run on baremetal, most other services on hetzner cloud VMs.

you mean creating or updating a post with embedded images is causing issues in some cases?

Hi,

are you seeing this on pages that work when you reload them or is it persistent on certain URLs?

Do you see an error message being returned?

The default Lemmy interface has a part like this below the matrix handles for admins:

The server returned this error: Error. This may be useful for admins and developers to diagnose and fix the error

Unfortunately, errors returned by Lemmy aren’t often that useful in what you see as a user, but it would still help to know which URL you were on and roughly which time, including timezone.

Are you still having issues or just describing what you’ve experienced before?

We know why the error you’re seeing is happening, as that’s a caused by reaching a limit, we just don’t know yet why the limit is being filled up by other stuff yet.

Hello,

we have just fixed an issue with our media service, can you please let us know whether you’re still seeing this?

Hello,

this should be resolved for now. Unfortunately the root cause is still not clear and will require some additional investigation.

Hello,

this should be resolved for now. Unfortunately the root cause is still not clear and will require some additional investigation.