It seems like the whole internet is broken today (at least, for me it is). How am I supposed to live??!??!?!?!?

We've had some ongoing problems with slowdowns and timing out that seanyboy has been working on. The only thing we pick up from Metafilter is the recent posts by Metachat users, and that gets a little notice like "Mefi rss feed down" or something when that information isn't available.

But that's as much as I know about it... seanyboy can answer more better.

The recent posts was several weeks out of date, until a few minutes ago (I know because it list one of my FPPs, and I haven't posted one in quite a while). Now's it up to date.

And now it's back to being weeks out of date.

ah! Well that's interesting. So evidently, instead of getting the "mefi rss feed down" message it's using some kind of cached info or something (and when it had the current information, mefi was back up for a moment, maybe)... Weird. And yeah, maybe a clue there.

Well, "recent posts" has been out of date for weeks. I never mentioned anything, because I figured you guys were seeing the same thing.

Actually... no, this must be something that's happening at certain times. Just the other day on the fake craigslist ad thread here, Jessamyn mentioned that matt had posted it to Metafilter, which she noticed because she saw it on the list of recent mefi posts here.

(Hmm, I do tend to come here mostly if I've gone to the other site, and it's down.)

This si what I see in teh side-bar:

Recent MeFi
posts by
Metachat users

up up down down left right left right B A SELECT start
posted by ninjew

Can you hear the Constitution now?
posted by orthogonality

Darlene Rockey's walk of pain
posted by matteo

Photo tips in your inbox or RSS
posted by dobbs

Every spherical football is a branched cover of the standard one.
posted by Wolfdog

Domesday Book
posted by Marxchivist

Inflamed what?
posted by handee

Language courses, courtesy of the United States government (and some devoted language learners)
posted by anjamu

Baby Don't Fear the Reaper
posted by maryh

High Speed Photography
posted by tellurian

Yeah... that's what I see as well; I just don't know where it's picking that up from.

When did you make your Constitution post?

This is very strange.

My Constitution post was August 17. (That wasn't my last FPP; my last FPP, about the Sunday school teacher fired for being a woman, was deleted by Jessamyn. I also made a post September 5, which was not deleted.)

All the posts listed in "recent posts" were made August 17.

The post ids are ordered descending,. From ninjew's to tellurian's they are:

http://www.metafilter.com/mefi/53975 up up down down left right left right B A SELECT start
http://www.metafilter.com/mefi/53973 Can you hear the Constitution now?
http://www.metafilter.com/mefi/53971 Darlene Rockey's walk of pain
http://www.metafilter.com/mefi/53969 Photo tips in your inbox or RSS
http://www.metafilter.com/mefi/53968 Every spherical football is a branched cover of the standard one.
http://www.metafilter.com/mefi/53966 Domesday Book
http://www.metafilter.com/mefi/53965 Inflamed what?
http://www.metafilter.com/mefi/53963 Language courses, courtesy of the United States government (and some devoted language learners)
http://www.metafilter.com/mefi/53960 Baby Don't Fear the Reaper
http://www.metafilter.com/mefi/53958 High Speed Photography

I bet that's when seanyboy installed the test version of the database on the new server.

I cannot make a new post. It says something about not parsing and gives me some gobbledy gook...

Specklet, that's known and by design; you either have an unclosed tag in your post, or an img tag that's not closed XML-stye.

If you have an img tag, it has to be like this:

<img src="some-url" />

not like this:

<img src="some-url" >

I've noticed the parsing error too, and I've only posted text. I think I may have bolded a word and it's ok when previewing, but on posting I get errors. Weird.

Hmm. Well, that didn't reproduce. :/

Ah, it's probably a permissions problem in the cache folder and so it's not writing newer entries.

That'll be why I keep getting those "Cannot write to rss folder" errors then.
Doh!
... And Fixed.

Thanks dodgy.

A quick update on the situation. (Warning: Technical nonsense ahead) I'm keeping an eye on metachat tonight. It seems to like crashing on a monday (my evening). I suspect that the error is caused by Apache2, but just to check that it is, I'll be checking seanyboy.co.uk when it goes down. If seanyboy is available when metachat isn't it's the PHP. Otherwise, it's Apache2.

My weekend plan was to swap Apache2 out for lighttpd. This would have had the added benefit of speeding the site up. Unfortunately, lighttpd is great for the simple stuff, but seems to require black magic for the rest. Try as I might, I can't get it to compile properly with the pcre-devel extensions. (pcre is the bane of my life) I need these to enable the other websites I have on the my server and make the wiki work correctly. I'd apt-get it, but that doesn't seem to work either.

I'll have another stab at that sometime this week.

So, just to let you know that I am doing stuff, but I'm not as up to speed with all this Linux stuff as I could be. My promise to you all is that the situation will improve.

Now, There's interesting - It appeared to stop on the hour.

Hmmmmm?
*strokes beard*

Is it Rimu doing their backup?

Maybe.
The problem is that the whole site appears to go down, and stay down until apache2 is restarted. It does this on a weekday, usually at a random hour between 8.00pm and 03.00am GMT. Today, it's done it twice. This seems to rule out a daily backup.

Anyway, the syslog shows that there's some dodgy cron.d fragments for php that appear to run on the half hour, and I've disabled these.
"/USR/SBIN/CRON[11859]: (root) CMD ( [ -d /var/lib/php5 ] && find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -print0 | xargs -r -0 rm)"

There's was also a lot of SMTP traffic and 22:04 which I'm investigating. I may move mailing onto another server.

Who knows. It's all a bit of a mystery.

Approaching zero-hour!!!

Well, that was unimpressive.

Hmmmm?

For some reason, I cannot access the profiles of those people who have spaces in their names (Joe Famous and By the Grace of God, for instance). I only get a blank page. I can only see mine by clicking the [Profile (daniel charms)] link next to the comment posting box.

Well I never. You're right. I wonder how long thats been like that.
It has been added to the list.

Heh. Actually, I mailed you about it yesterday (maybe I'm getting thrown to spam sometimes?). Anyway, it's funny, because I said that I know it hasn't been going on that long, since I remember going to Daniel Charms' profile when he did the Daniel Charms post.

Anyway, I bet it's very recent.