Above my pay grade. The problem has been solved by splitting the site across two servers.
Here is the brief version:
Memory usage in server spiked multiple times.
See that blue chart above 4Gb line? That is when server starts heavily swapping and then load accumulates and it become unresponsive.
I was pretty pleased with my Friday posts, so if you got tired of trying then, I hope you will read them now.
readerOfTeaLeaves says: And this is after the server upgrades of last fall ;-)
steelhead23 says: Hmm. I’m not certain this is benign and merely due to interest. I admit that I often pull up NC while having my morning Joe, around 7A Pacific. That’s 10 Eastern. Note how it spiked then dropped well below normal. Might someone be playing games with you? Mark me as suspicious.
Yves Smith says: Yes, spiking at 6 AM does not look organic. Comments don’t really start rolling in till 8 AM, which I think corresponds roughly with page views.
ex-PFC Chuck says: Yves, you should have been damn pleased with your posts yesterday. They were awesome. I’m usually up not much after 5:00 am CST and check your blog first thing. So I read your Investors’ Ox Gored and Romney’s Wife’s Swiss Bank Account posts before the crush, then couldn’t bring up the site when I came back a few hours later. It immediately occurred to me that somebody didn’t want them read, especially the Ox Gored one.
CaitlinO says: I figured it was the specificity of the Calista post which had ruffled the feathers of the peacocks.
Bev says: I thught it was something I said as some of my comments on the story
“Erica Chenoweth: Confronting the myth of the rational insurgent”
posted, then were reduced in number and moved to a different location, then gone.
So, something happened there.
In the face of attempts to censor the web, I figure that it becomes more important every day to say everything that is the best information for the best outcome for regular people everywhere.
Bev says: correction
thought
…….
Thanks so much.
tawal says: I was in there early at 2:30 am on my pc not my cell, which doesn’t scale as well, and it was acting funky then.
John Reilly says: It is hard to tell based on that single graph. You would need to look at the trends over a period of time to see when are the high times daily and the general trend upward over time. Once things get close to memory limits they will have a tendency to get much worse really quickly.
EmilianoZ says: It was hard having to go through the day without NC. I immediately suspected an attack by the banksters.
LeonovaBalletRusse says: Ditto! Can’t let a Good Woman stand up to the Man in public, now can they?
psychohistorian says: I am sorry your site was attacked. Letting another server process some of the traffic load is good thinking from the information you have provided.
What are you going to do when they shut down the intertubes for national security reasons? Myself, I will have withdrawals.
IF says: This graph unfortunately shows symptoms only, not a cause. Could be anything. If your hits went steadily up, maybe you reached critical mass and fell off a cliff. Or maybe a software update caused this. Without supportive logs I am doubtful this was an attack.
Yves Smith says: There was not a software update and Friday is usually the lowest traffic day of the week.
YankeeFrank says: Does NC use any kind of load balancing with redundant servers?
If not, or if those features were turned off, it is possible that some kind of index went wonky for a bit either within the web server itself or in the operating system. A memory management glitch, possibly in some garbage collection (newly freed memory being reclaimed by the system/application framework) routine, could’ve caused some churning due to unavailability of sufficient ram.
Or it could’ve been some kind of DDOS attack. You could tell by looking at what requests were coming in and from where in the web server log. If you see tons of page requests very quickly from the same IP addresses that’s gonna likely be a DDOS. If however the web server log shows little activity I would chalk it up to a memory issue and see if it recurs.
John Reilly says: Many requests from a single IP is not a DDOS attack – its a DOS attack. DDOS implies that the requests are coming from multiple IP addresses (the first D meaning distributed).
2little2late says: I fear it’s my fault. I told several friends what a great site you have. Sorry.
LeonovaBalletRusse says: Lulzsec for the Man. Too much reality on NC.
LeonovaBalletRusse says: Come on, Yves, weren’t we too vocal about Gingrich and his backer?
LeonovaBalletRusse says: YVES, just for fun, have your techie try something: compare the spike on Friday with the traffic at NC on any Saturday (which the Italians call “Sabbato”) over the last six months. Does this exercise reveal something of interest?
ambrit says: Maam;
I too was instantly suspicious when I couldn’t access the NC site this (Friday) morning. I tried several formats, all with the same Unable To Access response results. My first thought was Denial of Service Attack, which seems to be a common assumption among this somewhat jaded crowd you attract. (Sort of like David Brombergs song, “Sharon.”)
Does the ad click rate for the peak usage times approximate that of ‘normal’ usage periods?
Anyway, echoing a comment above, are there any alternate communication schemes in the works for that time, (in the not too distant future I suspect,) when the ‘Enabling Act of 2011′ is activated? Here’s hoping.
E.H. says: Finviz, had the same problem early morning. I too, was thinking of some attack.
sleeper says: Jeez – We are a suspicious bunch.
A denial of service attack on a small barely effective site ?
Maybe but since the major news sites where most folks get their news are pretty well controlled why bother ?
ex-PFC Chuck says: Just because we’re paranoid doesn’t mean they’re not out to get us!
tom allen says: What, there was flatulence? Sorry, that was me. :-P
just me says: My first thought was the MERS story. Advance work. As soon as I heard Schneiderman deal, this was the first place I went to. And with time the essence and news cycles what they are… I’m still wondering: MERS. (Ya think people want to know the true story and not the spin? A big h/t and quelle surprise to Yves.)
RalphR says: Suggest you check your facts:
http://www.nakedcapitalism.com/2011/01/the-20-most-influential-blogs-in-financial-media.html
http://www.nakedcapitalism.com/2010/10/we-are-number-one-on-technorati.html
Greg R says: “I was pretty pleased with my Friday posts”
So was I, prompting another trip to the tip jar.
Jim3981 says: I remember awhile back the NC website went down right after a few posts about China.
I posted something about the Chinese during that time, and for the next day or two I couldn’t log-on to other forums and stuff I normally use.
Some how I figured the two were linked.
par4 says: This is what happens when you live in a third world country. Our Fascist owners couldn’t even keep the railroads running on time, let alone the internet.
Markj says: I’m with steelhead32
It is not uncommon for net usage to become saturated with traffic as folks in the west become active. If your server is on the east coast then at 8 am EST folks in the west are getting up, drinking their coffee and reading the news on line. All this means there is a surge in traffic on your server at this time.
This is good news as it means the NC reader base is growing.
Tyzão says: ah the law of diminishing marginal serval returns — now I understand (but I thought this one goes to 11?)
JerryDenim says: Could people looking for information about the Scheiderman civil suit been the straw which broke your server’s back or was the announcement well after the official crash? I logged on no problem yesterday morning and after hearing of the NY AG’s suit of MERS early in the afternoon I immediately attempted to check Naked Capitalism again for details and analysis but was denied due to the crash. I just assumed there were a lot of people like myself who depend on Yves and company for up to the minute coverage of all things mortgage/bank fraud related and the surge in interest generated by the surprise Scheiderman suit did the site in.
Dave of Maryland says: Why not see if you can find connection between the posts/comments on one day, and the (revenge) outage on the next? (Since there does not seem to be any convincing explanation for it.)
In other words, make a list of inflammatory posts/comments from Thursday and then try them out, one a day, and see if anything connects. Yeah, it’s paranoid, but in crisis times, the paranoid are para-normal.
Bev says: Good idea. Here goes. First of several comments that dissappeared.
To explain why the following article is important for people to recover our democracy in order to recover our economy which is still posted at http://www.nakedcapitalism.com/2012/02/erica-chenoweth-confronting-the-myth-of-the-rational-insurgent-2.html :
http://www.bbvforums.org/cgi-bin/forums/board-auth.cgi?file=/8/81816.html
GLOBAL INTERNET VOTING FIRM BUYS U.S. ELECTION RESULTS REPORTING FIRM
It creates a vertical monopoly, in that you have one provider providing each step of the system, and a horizontal monopoly, with one provider delivering results across the majority of the geographic spectrum.
snip
…the bad news is that this centralizes one middleman access point for over 525 jurisdictions in AL, AZ, CA, CO, DC, FL, KY, MI, KS, IL, IN, NC, NM, MN, NY, SC, TX, UT, WA. And growing.
As local election results funnel through SOE’s servers (typically before they reach the public elsewhere), those who run the computer servers for SOE essentially get “first look” at results and the ability to immediately and privately examine vote details throughout the USA.
In 2004, many Americans were justifiably concerned when, days before the presidential election, Ohio Secretary of State Ken Blackwell redirected Ohio election night results through the Tennessee-based server for several national Republican Party operations.
This is worse: This redirects results reporting to a centralized privately held server which is not just for Ohio, but national; not just USA-based, but global.
snip
also affecting:
the United Kingdom, France, Canada, Norway, Switzerland, United Arab Emirates, South Africa, India and Australia.
Here is the link to the press release regarding SYCTL’s acquisition of SOE:
http://www.marketwatch.com/story/scytl-acquires-soe-software-becoming-the-leadin g-election-software-provider-2012-01-11
………
However, the following from the same forum was gone:
It took 75 years to ban slavery for the first time (this was done 100 years before the USA banned it). This was not “economically feasible” in the global sugar trade. Still, public citizens kept at it till they got what they wanted.
The story above is why I get annoyed at those who play partisan politics with the issue of public controls on elections. (Review again the quote at the end of the article, which is why I put it there.) It is exceptionally difficult, probably not possible, to get installed into office as either a Democrat or a Republican unless the real power super-elites find that person acceptable to their interests, which have a great deal to do with resource control on a global scale.
It is a surprisingly small number of people worldwide who are meaningful players on the global corporate field. There are about 6,000 of them, and while they do include some heads of states, like “president of the United States”, by and large those are considered rotating pieces which are replaceable.
Most real power is held by private corporate directors and the investment banking firms that move their money. There are really no global laws and there is no global enforcement mechanism, so these guys duck in and out of about 70 offshore secrecy havens like Jersey and the Cayman Islands, like Butch Cassidy setting up camp in the Waterpocket fold.
Most of the power is controlled by western corporations, a large portion of them based in the United States, but they only base certain operations here, and create offshore entities whenever they want to dodge regulations or taxes.
Now here’s the problem: These are very pragmatic men. Mostly men, a few women. These are not idealistic men, and they don’t give a hoot about what Thomas Jefferson thought or what the Declaration of Independence says. In fact, when your company depends on resources that are geographically dispersed, it can be doggone risky to let the people decide who governs the territory in which those resources (like oil, minerals, water) are found.
As I say, these are tough, practical men who are used to solving problems, and public control of elections is a bit of a “problem” if you can’t exert enough control by influence over who can run for office (ballot access and campaign finance) or persuasion (mainstream TV media). If that influence fails, and the wrong guy gets nominated, and the public just wants to go out and vote for him anyway, that’s a problem.
In that case, if you’re a practical and ruthless man, you need to control the mechanism of the election itself.
And there you have it.
…….
preceded by:
Why this is important:
http://www.washingtonsblog.com/2012/01/fearful-gopers-failed-dem-prepare-for-jeb-bush-draft.html
Fearful GOPers, Failed Dem Prepare for Jeb Bush Draft
…
next
Bev says: Contributer George Washington often sites political, military, intelligence agents and leaders who want to reopen an investigation into 911. Bravo.
This time with links:
Co-Chair of the Congressional Inquiry Into 9/11 – and Former Head of the Senate Intelligence Committee – Calls for a New 9/11 Investigation
…
High-Level Officials Eager to Spill the Beans About What REALLY Happened on 9/11 … But No One In Washington or the Media Wants to Hear
………
Pentagon Papers Whistleblower Daniel Ellsberg Says that the Government Has ORDERED the Media Not to Cover 9/11
Bev says: As brave a journalist as George Washington is, so too is journalist Lynn Hart:
http://existentialistcowboy.blogspot.com/2012/01/aa-exposes-bushs-big-lie-flight-11-did.html
AA Exposes Bush’s ‘Big Lie’: Flight 11 DID NOT FLY on 911!
by Len Hart, The Existentialist Cowboy
American Airlines is the source for information that AA Flights 11 (North Tower) and 77 (Pentagon) did not fly on 911. If neither flew on 911, the Bush ‘theory’ is a lie.
Bev says: Len Hart
Bev says: Additional to this:
AA Exposes Bush’s ‘Big Lie’: Flight 11 DID NOT FLY on 911!
by Len Hart, The Existentialist Cowboy
American Airlines is the source for information that AA Flights 11 (North Tower) and 77 (Pentagon) did not fly on 911. If neither flew on 911, the Bush ‘theory’ is a lie.
Is this:
Congressional records would show that AA already reported the same to Congress in televised open session.
What is Congress waiting for?
………
Bev says: Bev Harris says about 6,000 elites run (ruin) the world.
Then occupy movement should not be limited by such small porportions as 99% vs 1%.
The world’s population being over 7 billion the math would suggest 1,000,000+ to every 1 elite.
Those are much better odds.
Bev says: http://www.bradblog.com/?p=8889
By Brad Friedman
Recommended #OWS Demand: Let ALL Citizens 18 and Older Vote, On Paper Ballots, Count Them in Public
I offer the following simple “demand” for consideration by OWS, as this one likely underscores almost every other. Or, at least, without it, all other demands may ultimately be rendered moot.
Every U.S. citizen 18 years of age or older who wishes to vote, gets to vote. Period. Those votes, on hand-marked paper ballots, will be counted publicly, by hand, on Election Night, at the precinct, in front of all observers and video cameras.
….
EDA–Election Defense Alliance at
http://electiondefensealliance.org/
The Occupy Movement, Rigged Elections, and the Bastille Line: An Urgent Call To Action
All revolutions, even peaceful ones, require a point of attack capable of breaking through the Wall erected by the Powers-That-Be. The Occupy movement, such a welcome and important revival of democracy, has great potential to “rattle the walls” and change our times. There’s great heat out in the winter cold all over America (and across much of the globe). Focusing all that Heat like an acetylene torch on restoring observable vote counting and honest elections may well be the best, if not the only, chance the Occupy movement has to break the chokehold of the 1%.
…
Lynn Landes
http://www.thelandesreport.com/VotingSecurity.htm
DEMOCRACY DEMANDS TRANSPARENCY, NOT TRUST.
There is no transparency to our current voting system. The only thing candidates and voters can do under current circumstances to find out how citizens really voted is to conduct their own CITIZEN AUDITS. And even if you do prove fraud, it is highly unlikely Congress will ever do anything about something that they have benefited from. Congress has legalized election fraud by allowing, if not mandating, non-transparent voting systems that prohibit direct access to a paper ballot and meaningful public oversight:
ABSENTEE VOTING (1870’s)
SECRET BALLOT (1880’s)
VOTING MACHINES (1890’s)
I believe that there should be only one standard of voting for both our political representatives and voters. I believe that all voting should be open and public – no machines, no absentee or early voting, and no secret ballots. Secret ballots are really an anonymous ballots that corrupt election officials can count any way they want. Why one standard of voting for politicians and another for the public?
Javagold says: zerohedge was working but running slowly all day as well
my guess, something shady was going on !
LeonovaBalletRusse says: Dave and Javagold: Ditto. NC & Co. are making waves, and the Captain of the Titanic is unhappy.
Susan the other says: I know. My computer was slow on all my modestly revolutionary favorite sites. So I just gave up and cleaned out my temporary files and emptied all my caches. I was pretty paranoid about it all. I drank some wine and told myself it was probably the remnants of the latest solar flare. But I don’t believe me.
Matt says: Yves – certainly it seems you’ve found a fix (possibly temporary), but I suspect that you should be able to safely run this site from a single, robust web server. Since you’re running the site off WordPress, I’m assuming your server is Linux/Unix + Apache based, running PHP/MySQL. There is an excellent script (PHP-based) that has been developed to stop DOS and DDOS attacks. I’ve installed it on a server that I run that has ridiculous specs but that was still getting crushed by tons of unwanted bot traffic. This script put an end to all of that. Not sure of policy on posting links herein, but here is one to a discussion of this (free) script – http://www.webmasterworld.com/forum88/10425.htm. NOTE: I have nothing to do with this script other than I use it, nor do I have an affiliation with WebmasterWorld. Tell your tech guy about this and it may save you from having such problems in the future and may make it possible to keep things on a single server. Looking at your raw logfiles might help to diagnose as well.
Final note – if your server has decent specs (processor/hard drive size+RPM/Memory) and proper settings for resource utilization, it’s unlikely that even extremely heavy use from actual/real site visitors would bring your site down.
Bev says: from NC:
“The problem has been solved by splitting the site across two servers.”
How would Yves test that there are not two portals, separated sites to veer uncomfortable information into the ether.
MIWill says: The dreaded blue spike of death.
pebird says: Didn’t need to say anything, I had blamed it on AT&T.
scraping_by says: On Friday, John Carney published a slide show of “The Best Alternative Financial Blogs”. Of course, NC was there. I suspect a lot of Mr. Carney’s readers bookmarked here and a lot of company came to call.
Next time, we’ll see if the guest towels are clear and dry.
http://www.cnbc.com/id/46163886/The_Best_Alternative_Financial_Blogs
Jane says: John Carney’s piece about Alternative Financial Blogs was posted on 27 January 2012, so I doubt that was the reason.
Alex says: Hi Yves. Thanks for adding the extra server, it should help a lot.
However, I think there’s a secondary problem, which is the immense amount of advertising traffic run on the site. Obviously the site is expensive to run and you desperately need the revenue to pay your expenses. (Hey everyone, donate to Naked Capitalism if you’ve got the chance – adding a second server was doubtless quite expensive.) I noticed the my browser (Firefox 9.01 on Ubuntu 10.04) was very slow, and the culprit seemed to be advertising sites that took forever to load.
So I downloaded a list of advertising servers and their domain names, and blocked them from my computer. At that point the site started loading really, really quickly. I did this around 6-7:00 pm Pacific time, so that might or might not coincide with your upgrade. I haven’t had the tiniest urge to block the advertising on the other sites I visit; the problem seems confined to Naked Capitalism.
So you might want to look into which advertising services are causing problems with site load times, and explain to those companies that if they can’t make their systems run faster you’ll find other advertisers.
Just for the record, I am not encouraging anyone else to block advertising sites – Naked Capitalism needs the revenue – this was an experiment to see what the problem is and develop methods to fix it. If you want to help with the problem, your best choice is to make a donation. Sorry for the bad news, but this is how it looks from here.
ForYves says: You got a front-page article in a “subreddit” with 500,000 readers, at about that time.
http://www.reddit.com/r/politics/comments/p95v4/romneys_wife_had_3_million_in_secret_swiss_bank/
Check your referrer logs.
Sundog says: Not only the politics subreddit — that entry (or another linking to the same NC post) was pretty high on Reddit’s main page when I looked at Reddit yesterday morning.
Sundog says: Not arguing that, but I think there would’ve been plenty of legitimate interest among Redditors (and people like me who breeze through there often) in seeing that post.
I hit the link in the entry that made it to Reddit’s homepage (when I saw it yesterday mid-morning PST) and it failed. Felt bad to know a favorite site was not available to who knows how many first-time readers. Reading the post this morning I thought it exemplified some of the best of NC and was even more upset that the link from Reddit’s homepage failed in a situation that could’ve brought many new eyes here.
If the spike did coincide with appearance on Reddit’s home page maybe they could extend some useful cooperation? No doubt Reddit staff are well experienced in coping with the manipulation of web traffic.
Hoping NC will follow up on this by sharing some forensic results & maybe some meta-thoughts on blogging in the silly season.
kravitz says: I think we can all agree it’s been a fascinating news week…
fuzed says: decrease thread limit
LeonovaBalletRusse says: YVES, also the link to the William K. Black piece on CreditWritedowns might have been enough to trigger the shut-out. Surely the NSA is tuned in to you.
Clonal Antibody says: Pages are now being served faster than I have ever seen them being served. This approach is definitely much better than before.
Lambert,
Mor Harchol-Balter had a very good server based algorithm that would speed up web page serving. This was based on her PhD research – the algorithm did not get too much traction, because bandwidth increases made the approach moot for the last 10 years or so. However, I think the time to utilize it is approaching once again.
Jill says: NC: Using the power of numbers, graphs and amazing analytical skills in the service of good!
Keep it up!
Chet says: Yves,
I’m a techie in the business of this website stuff. I’d be happy to talk to your techie about your setup, and find out if something -else- was going wrong. Not that I’d be sure to find it. But, well, if there -is- a glaring error/hole in your site’s configuration, I might be able to help find it. Open offer.
Though I also find it all a bit suspicious, I figure it’s worth eliminating the possibility of technical error.
Posts
All Comments
looks like the balance sheets of the ECB, BoE and the FED