Evil Site Scraper Update and Reader Notice « naked capitalism

Saturday, December 4, 2010

Evil Site Scraper Update and Reader Notice

I wanted to thank readers for their generous and helpful input on the problems I’ve been having with sites ripping off Naked Capitalism content by putting up entire posts in full without permission, and often without attribution.

Even with reader help, however, the process of dealing with this takes effort, and any time I devote to site admin is at the expense of generating new content.

For instance, Ed Harrison e-mailed a very good list of action steps, as well as the form for one of the missives he suggested:

I got one site to take my stuff down. Here are the measures I took

Terms of Service page
E-mail to offender about violation referencing ToS and DCMA
E-mail to host about violation referencing ToS and DCMA
E-mail to advertiser (Google) about violation referencing ToS and DCMA
E-mail to Google News about violation referencing ToS and DCMA
E-mail to Twitter (using their twitter account) about violation referencing ToS and DCMA
Summary RSS feed only (just temporary)

Dan Duncan advocated the “don’t get mad, get even” strategy:

While you sort it out, always include several internal internal links to other posts. As long as you have internal links to your other work, then at least the scraped content will get you deep links to your back pages.

Other considerations: Instead of a simple HTAccess denial—ie simply denying access from the offending IP address— do an HTAccess “re-write”. By doing this, you don’t block access…rather, you send the asshole “false” content of your choice. It could be a HUGE file of jibberish like “hy^&GBHBDFNLG#$&H%” …or even better send them “The Best of DownSouth”! ["Please Yves of Naked Cap, we won't ever scrape your site again. Please, just-make-it-stop! We're begging you!"] [Of course, you are more than welcome to send them my commentary as well.]

Or, you could send the scraper into an infinite loop with something like this in HTAccess:

RewriteCond %{REMOTE_ADDR} ^123.123.123
RewriteRule ^(.*)$ http://domain.tld/feed

Replace the IP address with that of the scraper and replace the feed URL with the feed from the scraper’s site. That would actually be amusing. If you do this, please let us know what happens.

Here are some other good blacklist options from a helpful site:

http://perishablepress.com/press/2009/02/03/eight-ways-to-blacklist-with-apaches-mod_rewrite/

Also, beyond the Cease and Desist, you need to file DMCA Reports with the Search Engines.

http://www.mcanerin.com/EN/articles/copyright-03.asp

And finally, since they are scraping to game Google go to Google:

http://googlewebmastercentral.blogspot.com/2008/06/duplicate-content-due-to-scrapers.html

Again, the tech torture sounds great but is above my pay grade, and the tech people I know weren’t certain how to go about implementing it.

So I am instead opting for the cut the Gordian knot approach. I’m going to implement limited RSS syndication. It is the simplest solution (as in scrapers generally pull content off RSS feeds, this should cut way back on abuse) and to the extent anyone republishes my limited RSS, I will probably get more out of it than they do (as in they won’t get much content, and it will also drive some traffic back to NC).

Another reason this change may be a plus is, perversely, it will remove disincentives against putting up more than five posts in a day. Being in Recent Items listing has a huge impact on comments levels. Once an item drops off Recent Items, comments by readers pretty much cease too. Since I like getting comments, on evenings when I might have material for, say, six posts, I don’t see the point in putting up that many, since some will be ignored. Using excerpts (which means the main www.nakedcapitalism.com page will also show excerpts) will give site visitors another way to navigate through posts.

It also offers some advantage in terms of my typo proneness. I often catch typos after posting, as do readers, and believe it or not, I do correct them. But RSS grabs the first published version, a too often buggy 1.0 release. To my knowledge, the only way to get a new version up is to repost, and you still have the earlier version in the feed also. So having a much shorter version in RSS makes the odds much higher that it will be clean (since I do look at the opening sentences more closely than the guts of the post).

I’m not keen about this change, but some sites go this route. Jesse and Krugman only have headlines in their RSS; Felix Salmon in his Portfolio days only had a limited RSS feed (Update: Felix wrote to tell me that was not accurate; he did have excerpts on his Portfolio site, but had required them to provide a full RSS feed); FT Alphaville only gives the first line of its articles. And at least in my case, it did not make me less inclined to read them.

Where I may have a problem is with e-mail subscribers. My tech guy did not know whether going to a truncated feed would lead to e-mail subscribers getting that version (anyone who knows what has to be done in WordPress to have e-mail subscribers get full text versions, please e-mail me at yves@nakedcapitalism.com). I hope there is a solution, but we may not have it in hand.

I plan to start early next week and hope you will be like the change.

Separately, I finally have my new MacBook Air with all my old data on it and it really is cute and tiny. The cursor worringly skitters when I’m not using it. So far, it seems like a harmless problem, but I’m keeping an eye on it. I’m still getting over the fact that the sharper 13.3″ screen is at least as usable as the 15.4″ screen on my old laptop. I’ve also plugged in my antique monitor (a 20″ Apple Studio from 2002, we Yankees don’t like getting rid of things that still function) and it’s instructive to see the two of them side by side.

But as nice as this is, I still think the apex of desktop computing was the NeXT, with Improv (what a phenomenal spreadsheet) and the old WordPerfect.

78 Comments:

Fraud Guy says:

December 4, 2010 at 1:59 am

Wish I still could use Word Perfect 5.x

For someone who learned how to type by learning how to type, I could format with keyboard commands faster than I can remember.

tango says:

December 4, 2010 at 2:01 am

Sorry to see you are going to truncate your RSS feeds, but I do understand the problem. Most likely I will skip clicking through on a percentage of posts I would normally read in Netnewswire.

The reason being some of your material is over my head, or too “inside baseball” so now I will have to make less informed decisions on whether to click through.

So it goes.

Ben says:

December 4, 2010 at 2:08 am

If you feel like you have to then sure, abridge the RSS feed, but I’ll have to unsubscribe and come to the site directly. RSS really only works for me with the full text and excepts also make the feed pretty useless in offline reader applications.

Thanks for all your work, I am extremely grateful. I read your writing daily..

epc says:

December 4, 2010 at 2:16 am

Every site that I’ve read which has switched to truncated feeds from full-content feeds has seen their site traffic drop.

Instead of doing this, I’d recommend two things:
- monetize your feedburner feed using Google’s feedsense or adsense for feeds, or something like pheedo
- hardcode a copyright blurb into each post in the feed with a link back to the original article.

If you do truncate feeds, please don’t take the defaults from your blog tool, which is usually 65-120 characters. Instead ensure that your complete lead paragraph gets posted, with a “Read more” link. Too often feeds get truncated down to one or two sentences and honestly if those sentences don’t grab me I never click through to the original site.

Swarm The Banks says:

December 4, 2010 at 2:22 am

I did not realize you were having a problem. Whenever I notice a blog that lifts “too much” of a blog article from elsewhere, I will post a comment raising concern over their procedure.

Check out Swarm The Banks to see how I list RSS feeds to sites like yours.

And you readers, you can do a simple google check for this and other blogs, Take an approximately 8-10 word phrase, put quotes around the entire phrase, and then see how many sites show up with that exact same phrasing.

Then, see if they offer a link through and only a portion of the article. If they don’t, then leave a polite warning explaining their mistake.

Josh says:

December 4, 2010 at 2:34 am

Regarding monetization of NC … I’ve thought about donating a few times but always stop when I get to the point that I have to create a PayPal account. You might get more donations if you permitted people to donate directly from their credit card without having to go through PayPal.

Swarm The Banks says:

December 4, 2010 at 2:39 am

Is this the kind of issue you are talking about?

uh oh, no link, full article from Karl Denninger, Denninger article also then quotes this site as well, making it sound like Denninger wrote the article for the blog that appears to have lifted it.

I think I posted about how they were posting entire articles with no link backs, (even posting an entire article with a link back is not “good enough”). lol, I don’t know if my comment is still on the site or not, I could not find it.

I think I had posted it for the particular article above but the comment is not to be found. (it would have been under Alessandro Machi)

Yves Smith says:

December 4, 2010 at 6:45 am

Yes, that’s precisely the behavior in question. But I’m disabling your link, you’ve just effectively link-whored for them and helped their Google rating.

Rich says:

December 4, 2010 at 2:41 am

Noooo! Sad panda today.

I use an offline reader on my iPod touch. I have truncated feeds on it, including FT Alphaville, but these rarely get read.

This is going to be a big hit to my “Finance Must Reads” section. If other bloggers follow suit I might have to go back to mainstream media!! Aaargh! [Hmm, Conspiracy?]

Still, on the plus side I won’t be so obliged to read everything posted.

And yeah, monetize the feeds regardless.

Thx for the words.

Richard Kline says:

December 4, 2010 at 2:53 am

So Yves, while I enjoy having entire articles posted up, I strongly endorse your changes, for two reasons. First, anything that makes you tear your hair less makes for more and better content for us out here, so if it’s good for you I’m all for it. Second, I’ve very much seen the issue of ‘sunsetting’ on posts which drop beyond the five-limit, and anything which gets around that is a major plus. I often don’t read NC on some days, as you know, but I do tend to go though all the back posts when I get up-to-synch. In doing so I actually comment in threads that are a day or two old because of the quality of the post, of the discussion, or just because. Other times, a good discussion seems under way when the post drops over the horizon, and that suffocates furter interaction. I’m sure that others have had this exerience, too. Collapsing posts means more stay available means smile on my face. There’s the further issue that many of your posts are highly topical—foreclosure fraud and the Gulf-eating Blob for instance—where having several days recent posts really makes a difference in following the plot. All to the good.

Glad to hear your Bookin’ on Air. A light ‘n’ CuteTop at your fingertips means blogging has never been breezier, right? And it isn’t just you Yankees, as I have an old 14 lb MacPro laptop still powered up as a dee-ee-eep back-up. (I’m sure some real techies out there have legacy Rube Goldberg rigs to put us to shame.) It’s all progress . . . .

Jojo says:

December 4, 2010 at 2:53 am

I live by RSS and I don’t have any problem with truncated feeds.

I have yet to read any really cogent justification as to why some people have a problem with less than full feeds. With so many people running on broadband connections these days, what is so difficult about clicking on the link and going to the web page to read the full article?

Rich says:

December 4, 2010 at 3:07 am

Jojo, it makes it fiddly for offline reading really and ,at the risk of sounding EXTREMELY lazy, my reading on an iPod will go from swipe-read-swipe to swipe-click-wait-read-back to reader-swipe!

It’s the little things :)
- Jojo says:
  
  December 4, 2010 at 4:32 am
  
  So then I assume that you don’t read any major MSM sites I guess, because most of them to my knowledge only print a partial RSS feed (like the NY Times, for instance)?
  - Rich says:
    
    December 4, 2010 at 7:07 am
    
    Not sure that’s relevant, but the only truncated MSM feeds I have in my reader are FT Alphaville and Ambrose Evans Pritchard @ the Telegraph, both of which I’d say suffer to get my attention, because the way I’ve chosen to consume written media.
    
    Both the Guardian and the BBC here in the UK both have full feeds, and individual authors can be viewed to help specialize and get the “daily me” affect.
    
    Personally, I’m happy to fill the rest of my reader with blogs and “newer” media sites like Techcrunch, ReadWriteWewb, The Monday Note (I work in IT) some Huffpost writers. All of which have full feeds.
    
    Several bloggers I respect – including Yves – also curate daily links which is probably the main way I access MSM nowadays.
    
    The sky isn’t falling, it’s all still on this website. The only thing that will change is how I will consider each truncated RSS feed, rather than blindly – and gleefully! – “swiping” into each article not caring if my ipod was connected or not, that’s all.
    
    BTW EPC in a post above piqued my interest in site traffic falling when RSS feeds are truncated. I came across this article from Felix Salmon who advocates full feeds nowadays (unlike his portfolio days, mentioned above by Yves.)
    
    http://blogs.reuters.com/felix-salmon/2010/03/11/why-did-nick-denton-truncate-gawkers-rss-feeds/
    
    A further read around the subject suggests truncated feeds can actually be restored using some websites. So perhaps this move may affect (some) readers rather more than the scrapers it’s aimed at.
    
    Anyways, I’m interested to see if this change solves Yves problem and whether it affects site traffic in anyway.
    - Rich says:
      
      December 4, 2010 at 10:32 am
      
      There’s nothing difficult in clicking on a the link from the truncated RSS feed. Truncating RSS feeds doesn’t particularly affect me using a PC screen either.
      
      But after so many years we’re finally moving out of the desktop era and Naked Capitalism isn’t optimized for a small phone-size screen.
      
      Killing full RSS feeds prevents using an RSS reader as a “fix” to the poor text resolution and navigation of the site.
      
      I use mobileRSS, it really revolutionised my consumption, to the extent I was rarely browsing on the web from my work desktop and any queue – except at the bar – meant I’d happily browse a random article from a device tucked in my jacket pocket.
      
      I’m a bit sad that this I’ll lose that option for this site. It’d be nice to know that these mobile issues will be considered in conjunction to the truncated RSS feed.

Mogden says:

December 4, 2010 at 2:56 am

I understand your problem, but the solution sucks for many readers, including me. It is much nicer to read posts on a mobile device in a feed reader than jumping all over the place.

Is it really that horrible if they scrape your content?

Rich says:

December 4, 2010 at 2:57 am

“Every site that I’ve read which has switched to truncated feeds from full-content feeds has seen their site traffic drop.”

As much as hate the change surely it will increase site traffic?

Do you mean RSS feeds count as site traffic and subscriptions likely cancelled?

MichaelC says:

December 4, 2010 at 3:11 am

FWIW, I think this is key as you think this through:

Another reason this change may be a plus is, perversely, it will remove disincentives against putting up more than five posts in a day. Being in Recent Items listing has a huge impact on comments levels. Once an item drops off Recent Items, comments by readers pretty much cease too. Since I like getting comments, on evenings when I might have material for, say, six posts, I don’t see the point in putting up that many, since some will be ignored. Using excerpts (which means the main http://www.nakedcapitalism.com page will also show excerpts) will give site visitors another way to navigate through posts.

Although its hard to fathom that you can do 5+ postings a day, if you’re game for it, enable the site to handle more.(although I think you need to strike a good balance, ZH is a good proxy for overextension) Your hardcore fans will be grateful, and others will pick up the extra posts and do the circulation for you. The sleazy plagarists may be annoying, but you’re making it easy for legit allies to cross post efficiently (firedog/BR/FT, etc)to the right target audience.

As the author you’re concerned with circulation, (RSS feeds etc), but I think your commenting audience is more low tech, which I think is a good thing given the quality of the comments.

Yves Smith says:

December 4, 2010 at 4:34 am

Remember, 5+ posts normally includes guest/cross posts. Sometimes I get 2-3 good guest submissions on days when I might have 3-4 posts of my own.
Fractal says:

December 4, 2010 at 8:07 pm

“your commenting audience is more low tech, which I think is a good thing given the quality of the comments”

Ha! speak for yourself, mouse breath!

Jeremy says:

December 4, 2010 at 3:16 am

The soft underbelly of all these content scrapers, archives of libelous rants masquerading as humor, and the other bottom feeders of the web is their need to make money. I noticed one of the scrapers you complained about is running Google ads, for instance. Make a stink, and Google will cut them off.

Michele says:

December 4, 2010 at 4:28 am

I have been putting together a site (Not active yet) with links and a short description of useful news articles. I was planning to add several links to your articles and I just wanted to make sure this is alright with you before I do it.

It will be in the format of the linked title, author name and date, and a couple lines describing the contents.

If there is any problem with this, just let me know and I’ll take care of it.

Thanks.

rjs says:

December 4, 2010 at 5:06 am

in my experience, email subscribers get the truncated feed, which necessitates a site visit…

Hans Suter says:

December 4, 2010 at 5:22 am

to punish scrapers you’ll loose readers, very old media behavior.

Yves Smith says:

December 4, 2010 at 6:29 am

To be quite blunt, I earn pretty much no income off RSS readers. RSS only readers presumably do not comment, so I’m really not sure what value they really have to me. Ever since I got my new ad service early in 2010 (which has performed well for me, the guys running it clearly know what they are doing and did not overpromise, unlike pretty much everyone else in that business) they’ve been urging me at every turn to go to truncated RSS as a way to increase my income. So from an income optimization standpoint, I should have made this change long ago. I spend enough time on this I might as well cater to the readers that contribute in a tangible way.

As Barry Ritholtz says, embrace the churn.
- Petey says:
  
  December 4, 2010 at 6:43 am
  
  I think you are doing the correct thing.
  
  There are a certain subset of readers who like full RSS feeds who will be initially upset, but they will be able to change their reading behavior.
  
  There are plenty of sites I regularly read that have truncated RSS feeds. The interested reader can still easily and pleasantly follow those sites.
- Rich says:
  
  December 4, 2010 at 7:35 am
  
  Blunt is good, and it’s fair.
  
  I’m sure everyone who regularly reads this site, regardless of how they read it, wants you to be financially rewarded for you work.
  
  Allow me to feel somewhat ashamed for using RSS to read your work then. I didn’t actually notice until now that you didn’t have ads in your feed and I guess it pays bugger all anyway.
  
  Apologies for the accusation, but it does feel that this might be the main reason for the truncated RSS. Fair enough. paidcontent.co.uk does quite a good job collating all the latest business models. For the record yourself, Ritholtz and Mauldin I would gladly subscribe to, within reason. I’d be so much intellectually poorer though if every blogger went that route. Or skint :)
  
  And you’re probably right in that RSS users don’t engage as much. This is a first for me on this site.
- Hans Suter says:
  
  December 4, 2010 at 10:31 am
  
  please read what Felix Salmon has to say about truncating: http://blogs.reuters.com/felix-salmon/2010/03/11/why-did-nick-denton-truncate-gawkers-rss-feeds/
  - Jojo says:
    
    December 4, 2010 at 3:01 pm
    
    Please read what the users say in the link you provided. The response is mixed and it is indeterminate IF truncated feeds actually hurt. No one was able to quote any study links.
    
    Additionally, reference was made to Twitter. WHY is 140 characters good enough for Twitter but not for RSS?
    
    Again, I don’t really care what Yves does with the RSS feed one way or another.
    
    But I get annoyed at reactionary responses with no real justification for the response or action.
    - Rich says:
      
      December 5, 2010 at 6:02 pm
      
      Additionally, reference was made to Twitter. WHY is 140 characters good enough for Twitter but not for RSS?
      
      Whilst they both generate through traffic to a website, Twitter is more random and has an associated opinion value.
      I don’t think you can rely on Twitter to reference all of a websites posted articles, which is primarily how RSS readers are used.
      
      To truncate RSS or not has been discussed elsewhere.
- Mogden says:
  
  December 4, 2010 at 12:00 pm
  
  I am a RSS only reader and I comment.
  - Yves Smith says:
    
    December 4, 2010 at 9:00 pm
    
    You have to come to the site to make comments, ergo you really aren’t “RSS only”:-)
    - Phil Groce says:
      
      December 5, 2010 at 10:48 pm
      
      The lack of a “reply” link in the RSS feed has something to do with that. Although one still has to come to the site to post the comment. Ah, pedantry. :)
      
      I’m an RSS reader, and I’ve thought about consuming the site for the comments. I’m sure they’re very informative. But I have to force time into my schedule to read this site — which is worth it to me. I can’t justify adding considerably more time to go through the comments — which are also of high quality, but not as consistently.
      
      I wish you had an easier way to monetize me. You have provided a positive externality in educating us RSS readers, and for that I thank you.
      
      Please consider a site redesign for mobile devices. The white space to the left and right of the main column is fine for PC’s, but it makes reading awkward on mobile devices.
      
      I should contribute. Time to add money to Paypal, I suppose.
- Economists Do It With Models says:
  
  December 4, 2010 at 1:04 pm
  
  I’ve read your feed for a while now (without going directly to the site, in case you were curious), and I think you make a lot of good points, especially where the mortgage mess is concerned. I also intended to purchase your book when I am done with what I am currently reading. However, I changed my mind when I saw that I was of no value to you since I didn’t contribute that $.01 per view or whatever to your ad revenue, and I’d hate for you to be incorrect in your valuation.
  
  On a related note, I recommend the book “Free” by Chris Anderson, particularly the chapter on the perils of nickel-and-diming customers on a zero marginal cost product.
  - Yves Smith says:
    
    December 4, 2010 at 3:41 pm
    
    Honestly, you read my content for free, on which I spend HOURS every day, you say you were going to buy my book (on which I have yet to see a penny beyond my advance, but incrementally, I’d make about the value of a cup of coffee when I have sold enough books to recoup the advance, and I am close to that point) and now you are pissed and want to punish me? (and you also overstate my remark, I said it didn’t appear RSS only readers have any value to me). And you can’t see how selfish your attitude is?
    
    To put it more simply, you can’t consider yourself to be a “customer” if you generate no income. There may be no marginal cost, but there is no marginal revenue either. And you as a self-proclaimed economist should know that and recognize that your argument is disingenuous.
    
    I tell you that the way you participate in the site appears to have no economic value to me, and instead of either giving a counterargument (that there is some benefit I don’t see) or saying you are sympathetic, you decide ot punish me. Again, why should I accommodate someone with such an openly selfish attitude?
    - Anonymous Jones says:
      
      December 4, 2010 at 5:02 pm
      
      Not that you need my help, but you missed one of main flaws in his argument. Your motivation for the change had nothing to do with “nickel and diming.” It had to do with trying to shut down the site scraping. So there is, in fact, a marginal cost to providing the RSS feed (your time and effort in defeating site scrapers whose job is made easier by the RSS feed). In any event, you made him look like a fool anyway; just wanted to point out that he is even more daft and ignorant than you exposed. Always a reminder to people like “Economists Do It…”, thinking before typing is often best practices.
  - Michele says:
    
    December 4, 2010 at 4:52 pm
    
    I am curious as to why you would change your mind about purchasing a book that you obviously believe to be of value to you.
    
    Having your feelings hurt (if that is what this is about) doesn’t seem to be a reasonable excuse for not expanding your knowledge IMHO.
    - Yves Smith says:
      
      December 4, 2010 at 5:39 pm
      
      I separately doubt he would buy the book. It’s been out since March and has gotten good to stellar reviews (albeit in secondary venues) and very good word of mouth (I think it’s selling more on that than normal PR). If he is still thinking about it, he is hardly committed. This just seemed like a convenient threat to lob at me.
- MarcoPolo says:
  
  December 4, 2010 at 1:20 pm
  
  I’m an RSS reader and I use the tip jar.
  - Yves Smith says:
    
    December 4, 2010 at 3:42 pm
    
    Thank you! I’m sorry that this change will inconvenience you.

Agent Vanilla says:

December 4, 2010 at 6:55 am

If you go ahead with this idea, I would reeeeeally appreciate an improvement in your font color. In RSS, you titles are a strong deep blue, your text, a strong black.

Orange is a terrible color for reading. And your text seems vaguely dark gray. Makes it hard to read. I never would click through, unless I wanted to comment, because it was easier on the eyes in Google Reader. I am not vision impaired, but definitely vision strained after 12 hours staring at a computer.

Try viewing your site in Google Reader and check yourself. If you are looking for it, the quality difference is noticeable.

Jojo says:

December 4, 2010 at 3:02 pm

I agree. I don’t like the orange at all. And the text does look gray and should be black.
- Yves Smith says:
  
  December 4, 2010 at 3:47 pm
  
  I have pinged my WP guy a zillion time about the complaints about grey. He says it’s set up to be black. I see the text as black. If any people who know WP have any clues, I’m all ears.
  - Jon H says:
    
    December 4, 2010 at 4:32 pm
    
    The font is a little thin. At small sizes, perhaps that contributes to an appearance of being grey.
    
    PS: I agree about NeXT! Still own two of them.
    - Fractal says:
      
      December 4, 2010 at 8:20 pm
      
      yeah it’s probably the font, not the text color.
      
      me personally, I look forward to the hot orange, since it’s the same color as DKos, but awesomely more intellectual and more entertaining (and intelligible) comments, notwithstanding what mouse breath said @3:11 am (MichaelC)
  - Jojo says:
    
    December 4, 2010 at 11:30 pm
    
    I fiddled with the brightness controls on my monitor and yes, the text is black. Sorry.
    
    But the ORANGE has got to go….
    - Agent Vanilla says:
      
      December 7, 2010 at 8:43 am
      
      Yeah, seems you are using Optima as the font. In comparison to other fonts, it comes through rather thinly even if same font size. Here is a comparison of fonts by usage. Optima has a rather low frequency rate on Windows machines, less than 3%. You can use the following chart to compare Optima’s visibility against other fonts. Certainly seems “lighter” to me compared to other fonts.
      
      http://www.codestyle.org/css/font-family/sampler-WindowsResultsFull.shtml
      
      Might I suggest a nice Tahoma in 10pt font? Your WP guy should be able to help you make the change to the resource Cascading Style Sheets (“CSS”) in addition to your other Too Doo’s. I think this would intensify the Orange as well. It probably is suffering from the same thinning effect as the black Optima.

Kevin Smith says:

December 4, 2010 at 7:25 am

Excellent post, that’s a keeper. Great compendium of useful info. Thanks to all who contributed.

Get a trackball for your laptop … it will make your work a lot easier. The classic [if it is still on the market] is the USB MicroSoft Trackball Explorer, which has 4 programmable buttons you can set to click, double-click, close window or hide window [or whatever other commands you fancy].

James O'Keefe says:

December 4, 2010 at 8:27 am

As a predominantly RSS reader I will miss the unabbreviated posts. I don’t think this is the right way to go since it hurts fans as well, but it is your site.

On another issue, the NeXT Station was a beautiful computer and Lotus Improv is still the best spreadsheet to ever be created. Last I looked someone had created a spreadsheet like it but I think it is over $800.

Yves Smith says:

December 4, 2010 at 3:49 pm

It’s Quantrix, which is $399 for a version that does not export to Excel and something like $1000 for the version that does. Improv was just fantastic. I’m no fan of spreadsheeting, given my horrid inaccurate typing but it was a pleasure in Improv.

And Dan Duncan gives the solution below. I’m not going to promote it, but loyal comment readers who want their RSS too have their fix.

Siggy says:

December 4, 2010 at 9:11 am

It’s your site and you should do what serves you best.

Truncated feeds are fine with me. You could also dispense with RSS altogether and I would check the site anyway.

The fact of scrapping and the extent of it is troubling and it speaks to something more profound as to the propensity to plagerize.

Dan Duncan says:

December 4, 2010 at 9:47 am

You may want to include a section on your site to this link which is quite helpful for readers of any RSS content who want to turn the abbreviated posts into unabbreviated:

http://www.online-tech-tips.com/computer-tips/turn-partial-text-rss-feeds-into-full-text-the-easy-way/

It’s pretty straightforward. [I have absolutely no connection whatsoever to any site referenced in this comment.]

Obviously, you would like to come up with a solution on the publisher end, as opposed to having the reader go through these steps. Nevertheless, it is useful advice that can be applied to other blogs besides just yours’ and it would be appreciated.

As for getting a full text of the article in the email distributions….Joost de Valk at Yoast.com is a respected SEO expert lauded by Jeremy Shoemaker among others. http://yoast.com/about-me/ If anyone could help, it’d be de Valk.

And for what it’s worth, this from Matt Cutts, Google’s lead “anti-spam” engineer on link-backs in your RSS FEED:

Don’t cloak the link or make the anchortext spammy, but otherwise….Cutts said syndicating articles with a link to the original article was smart:

“Stephan Spencer: When one’s articles or product info is syndicated, is it better to have the syndicated copies linked to the original article on the author’s site, or is it just as good if it links to the home page of the author?

Matt Cutts: I would recommend the linking to the original article on the author’s site. The reason is: imagine if you have written a good article and it is so nice that you have decided to syndicate it out. Well, there is a slight chance that the syndicated article could get a few links as well, and could get some PageRank. And so, whenever Google bot or Google’s crawl and indexing system see two copies of that article, a lot of the times it helps to know which one came first; which one has higher PageRank.

So if the syndicated article has a link to the original source of that article, then it is pretty much guaranteed the original home of that article will always have the higher PageRank, compared to all the syndicated copies.”

Yves Smith says:

December 4, 2010 at 4:10 pm

Thanks, this is very helpful.
- T. Rex Bean says:
  
  December 5, 2010 at 5:15 pm
  
  Yes, it is. I read this site via RSS (and have, on occasion, commented). I suscribe to many feeds and much prefer the full versions. However, I certainly have no right to complain about Ms. Smith’s decision. I benefit from her hard work and bring her no profit. But I will certainly try Mr. Duncan’s suggestion to obtain a full feed if that is permissible.
  - T. Rex Bean says:
    
    December 5, 2010 at 9:43 pm
    
    Update: I tried that site that purports to ensure a full RSS feed but with no success. All I get is the feed for my own RSS subscriptions.

Kyle Julien says:

December 4, 2010 at 10:09 am

“RSS only readers presumably do not comment, so I’m really not sure what value they really have to me.”

Frankly and respectfully, that statement kind of rubbed me the wrong way. This is only my 2nd or 3rd comment here; I don’t spend a lot of time commenting on blogs, and have never felt like I could enhance the conversation much by appearing in the comments section. The value may not be measurable or concrete, but I have learned a great deal reading this blog and ‘consuming your content’ has changed the way I think about economic matters. Surely the value of your work and of your loyal readers can be measured in something other than click-throughs.

(And, if I weren’t a reader of your RSS feed, I wouldn’t have bought your book.)

I understand the need to change your site to ensure fair compensation for the significant work that goes into it, as well as the need to maintain control over that work in the age of automated digital reproduction. Changing to limited feeds will be an inconvenience to me, but I get the reasons.

But I don’t think it is going to be helpful to think through such choices based upon a distinction between regular commenters and regular readers.

Thank you for your hard work.

Kyle

Rich says:

December 4, 2010 at 11:08 am

I’d like to 2nd this sentiment.

Originally I was drawn to this and other financial blogs when it became apparently that not only did the majority of the mainstream media fail to give sufficient heads up to the impending crisis, they were arguably complicit in some of the issues, as noted by their oversized property supplements (here in the UK at least.)

Once the realisation hit, I backtracked and questioned who were calling it and have been loyally following ever since.

But I can add very little to many the articles that usually go waaay over my head. A lot of current issues raised here are’t even occurring in the same country as me, but I find myself egging on the “little guy and gal” bloggers who are arguably doing the MSM’s job again.

Yves. Alluding to financial reason for making the RSS change has also hit a raw nerve. Not because it is wrong, but because I always felt that old media would be the ones doing all the adjusting and good bloggers would get by nicely and continue to be a problem for MSM by merely free and being easily accessible, in any format.

I think I’ve peppered this thread enough to keep schtum in what will arguably more infrequent visit.
- Yves Smith says:
  
  December 4, 2010 at 3:57 pm
  
  I’m sorry if you take offense, but I suggest you look at what you are effectively saying. I spend the equivalent of a full time job on this site. You basically argue that bloggers should continue to labor for free to upend the paid MSM. I’ve been blogging for four years now, and I have to tell you that the churn in blogs is huge. Many really good bloggers have dropped off the map or have cut back their post levels. There is a very large number that simply don’t exist any more. And the ones with my sort of traffic rank do not have comparable business models. Barry and Mish have synergies with their day jobs, so they aren’t doing much blog-specific research, just dedicated writing (and Barry now posts much less often on his own, he has a lot of content from other authors). CR is retired, and he’s been very smart about his content (for instance, spreadsheets related to data releases, he can dump new release date in and generate a new chart). ZH has a wealthy backer. Mark Thoma has tenure and his site also has synergies with his day job. Felix is on a payroll. By contrast, I am self employed, and as the saying goes, “The difference between being self employed and unemployed is thin indeed.”
  
  So while the tone of your comment is reasonable, the substance is that you are begrudging my getting some compensation for a large amount of work, and my maintaining the discipline of posting pretty much every day, no matter what my competing demands are. And you ALSO deriving the benefits of my 30 years of experience in the financial services industry.
  
  In a cafe, someone approached Picasso, and asked him to draw something on a napkin. He complied. The person tried to take it and Picasso pulled it back and asked for $10,000. The person stuttered, “That took you only a couple of minutes.” Picasso replied, “No, it took a lifetime.” I’m no Picasso, but the same principle applies in reduced form, the value of this site goes beyond the considerable hours I put into it.
  
  You are unable to see that you want me to work for you for free. I know some RSS readers do not fall in this camp, per Marco Polo, they have contributed via tips and I am sorry they will be affected too. But your attitude, that I’m the selfish one here, is really very revealing.
  - Rich says:
    
    December 4, 2010 at 7:00 pm
    
    [Did Picasso draw pictures on the back of cheques so they wouldn't be cashed? ]
    
    Yeah, I was petilly wanting a straight-up “I’d like to get some more reward for all this effort, so here’s the deal” post, rather than something dropped in the comments of a post concerning a solution against scrapers. It’s cool, I’ve got it – Sorry!
    
    This is your site and you have every right to run it as you wish and expect justified rewards. I hope you find a balance that works for you. Getting that balance has got everyone scratching their heads for a considerable time now and I’m slightly concerned that truncated RSS feeds will neither resolve the scraper issue but could also be detrimental to reward. I’ve no idea. we’ll see how it pans out.
    
    I don’t want bloggers to labour for free. I was – very badly – trying to suggest that I’ve viewed the situation of influencial bloggers with rose-tinted specs and am unnerved that this view is not true.
    
    I’ve hoped an ad supported model and opportunities as a result of owning a popular site was more than sufficient reward for popular bloggers, enabling continued free distribution of valued comment and opinion, whilst MSM are forced to erect counter productive paywalls and lose some of their (perhaps) unhealthy influence.
    
    This sites still free, it’s cool, I’m merely inconvenience, but the prospect of truncated RSS has made me think I’ve been way too optimistic.
    
    The emergence in the last few years of new form factors will to some extent shift how we can (freely) consume media. That’s been on my mind since the iPad caused Murdoch to act like one of Pavlov’s dogs at the prospect of resumed control. Quite how much bloggers can upend MSM is less apparent to me.
    
    Right. Scrapers. Bastards. Thanks for a great site. Will honestly get round to buying your book.
    - Paul Repstock says:
      
      December 5, 2010 at 2:11 am
      
      All of this discusion is really no different from the music and movie download conflicts. Even Stephen King gave up on the public trust model.
      
      While I somewhat doubt a large financial impact on Yves Smith, I also have no patience with theft, plagarism, or copyright infringment. There is no reason for any of us to reward criminal behavior. Perhaps Yves could watermark her articles in such a way that Google would be forced to remit to Naked Capitalism rather than the scrapers…:)
      
      Yves does the work so she should get paid. However, as others have alluded, the point of the internet and specially the blogosphere is to spread knowledge and to give the Average Joe a soapbox. My concern is that given the current trend to handheld devices, anything like converting to the RSS format will narrow the audience unecessarily.

Sufferin' Succotash says:

December 4, 2010 at 10:40 am

Well, I admit to ripping off that snapshot of the cute l’il duckies in the wash basin, but I DID credit NC.

:)

PunchnRun says:

December 4, 2010 at 10:56 am

For any who read as I do, what you plan will have no effect. Perhaps I represent only some less gadget-ridden readers, but I use Google Reader for all RSS feeds. I have no mobile web-enabled devices, but then I don’t need to be mobile.

With Google Reader I can see much of the originally posted material but I don’t like the style, and I don’t see comments. The comments on NC are as often useful to me as the original. So I always use Reader purely to see what is interesting and then jump to the site to read. For NC, it’s all interesting. Since the Reader shows me all the original posts, I never have the problem of losing them as they scroll off the “recent posts” set.

Jojo says:

December 4, 2010 at 3:08 pm

That is a good point about the comments. For me,on many blogged articles, the comments are more interesting than the article content. And you have to go to the web page to read the comments.

If you are not reading comments (even if you don;t comment yourself), you are missing out on a great deal of information!

dimitris says:

December 4, 2010 at 11:01 am

Noooo! Another sad mobile-rss-reading panda here.

- Is there a tech reason why ads can’t be added to the full-text rss feed?
- Before taking the “drastic” step of truncating the feed, why not try e.g. the copyright blurb and/or reporting the scrapers to Google first?
- To keep comments alive longer, can’t the limit of 5 “recent stories” simply be raised?
- As seen above, full text feed workarounds already exist. What’s to keep the scrapers from using them?

Reed says:

December 4, 2010 at 11:13 am

I’m with Fraud Guy on WordPerfect 5.x. Simple commands, code placement made sense and was easy to edit, and macros were a breeze. Sigh…

Francine McKenna says:

December 4, 2010 at 11:47 am

My experience with scrapers:

1)I post a limited RSS feed via feedburner and my wordpress blog. Some folks like Felix Salmon have told me they do not subscribe to any limited RSS because they do not click on sites only read feeds. I, too, would like to know how to push full feeds only to my feedburner enabled wordpress blog subscribers.

2)I have a Huffington Post account but repost my stuff there very sparingly. Typically I only report legal related first. Surprisingly a lot of lawyers read Huff. All the liberal class action guys, who I want to cultivate. As son as I post on Huff I get a prolif of pingbacks to my internal links from scrapers. I complained to Arianna’s right hand person. “Why are you pushing out full feeds?” I asked him. Because we don’t see a way around it. Nature of net, he said. I don’t agree.

3) I am very internally linky. My posts circle back to my previous posts often to get full narrative and context. But many scrapers not only do not attribute but take out all links. So much for that.

4) I think the Google route is a great idea. I will do that when I have a chance.

5) I have found legit sites that are scraping full and put up attribute only in title. Some are also on Twitter. When I have shamed them publicly, they stop.

6) My bigger beef is sites like NPR, USA Today, others who use a service called Daylife.com to pull in content to populate their news aggregation/forum pages. Although it is limited blurbs and links back, I think it’s dumb, not curated by their internal digital professionals and not promoted. I could send them more traffic then they are sending me if they signed an agreement, worked with me to select appropriate and relevant posts, and paid me for my content.

Bob Alexander says:

December 4, 2010 at 12:31 pm

hmmm … never occurred to me to read anything other than the originating site! Feels better … like I’m getting the real thing. Copied stuff is way too easy to change. (That would be an argument for Truncated feeds! Anyway, I don’t rely on the RSS anyway from anybody!)

spc says:

December 4, 2010 at 2:07 pm

Apropos monetization, Yves, You should denounce add blocking !!
Furthermore, maybe time has come to consider paywall. Paywalls are hot topic these days.
For instance 50 $ per month for membership, one post a week for free as a apetizer.
This might work ! You could blaze a new trail of econ blogsphere.

Each time You truncate a rss feed Wall Street banker suffocates a hooker.

Jojo says:

December 4, 2010 at 3:34 pm

Paywalls are a big FAIL for most companies. The WSJ has been successful with theirs but I think that is because they have somewhat unique audience demographics. Most others don’t have such an advantage:

==========
After Three Months, Only 35 Subscriptions for Newsday’s Web Site

http://www.observer.com/2010/media/after-three-months-only-35-subscriptions-newsdays-web-site
=========

New paywall costs the Times 66% of its internet readership

Charge for access to website sees newspaper’s online audience fall to 33% of its previous size – better than many had expected

http://www.guardian.co.uk/media/2010/jul/18/times-paywall-readership

============

I don’t participate in sites with paywalls because you cannot easily share links to their articles.

When the NY Times puts up their paywall, I will drop them from my RSS feed and replace them with WaPo or the LA Times or some other major non-paywall site.
- Paul Repstock says:
  
  December 5, 2010 at 2:26 am
  
  I am living proof of the “Fail of Paywalls”. Specially with WSJ. I don’t like The Journal and if I happen to encounter a headline from them put out by some othr media outlet like Yahoo, I normally ignore it or if interested, just copy and paste the headline into my search engine. Almost always one will get the same information from other sources, for free.
  
  I know I’m bad..:(. But, Gosh Darn, Their content usually isn’t worth paying for. And besides, they are running various types of ads on the articles anyway.
- Rich says:
  
  December 5, 2010 at 6:21 pm
  
  I think it should be noted that FT.com is profitable I believe.
  
  What you wrote is interesting for two reasons.
  
  Firstly, what you do once you encounter a new paywall – gravitate to the next available free site – is probably the major reason reason for fail.
  
  (Out of interest what will you do if every news site erects a paywall?)
  
  Secondly, you highlighted what we’re losing as these paywalls come into effect, which is a free reign on any news content and the ability to generate further value through linking and comment.
  
  I’d imagine there will be a lot more specialisation in the next fews years (paywalled up) and a lot of consolation in the “old media” companies.
  
  I’m quite intrigued what will happen to the BBC here, which is effectively subsidised, as more of the paywalls are put up.

matthew slaughter says:

December 4, 2010 at 2:39 pm

what the !@*# is RSS and burnfeeder? jesus ive been online since 1994 i think maybe im getting too old for this %%$#.

people read this site from their mobile phone?

i dont understand why the phone people, who monetize ##*#** ring tones for chrissake, can’t figure out how to monetize these ‘feeds’ or whatever the %$*# they are called. pull 1 penny out of someones phone account everytime they visit ‘feed xyz.com’, and give 0.8 cents to the blogger. oh well.

emca says:

December 4, 2010 at 2:40 pm

From the the Felix Salmon article linked above by ToddG:

“I’ve grappled with the issue myself over the years. I’ve yet to see one solid case study that shows going to full feed dramatically increased someone’s traffic AND bottom line. The Guardian example above isn’t proof of anything. Any number of things could have sparked their rise, and they say as much.”

Outside personal opinions in the matter, little seems to be offered (even in Felix’s links). My own is; sorry, tried it a few times, but no longer subscribe to RSS shorthand from any site; the deluge of information in list form without organizational outline for browsing was tiresome.

What Yves has proposed sounds reasonable and correct. I doubt whether any commenter is willingly to take the time and effort to voluntary scan the web for articles scraped from NC or send necessary e-mails to perpetrators.

As far as increasing or maintaining traffic to a site, I would give the nod to content first, and presentation second. If more emphasis can be placed here, then RSS feeds will pale in significance. I suspect this is what “The Guardian” experience referenced above really shows.

ChrisPacific says:

December 5, 2010 at 5:14 pm

I might have tried some less dramatic things first to see if they made a difference, but it sounds like there were some other factors involved. If you are not making a living from this blog then you should be, and I support anything that facilitates that. Maximizing my convenience as a reader is an important but secondary consideration.

If you’ve added Terms and Conditions for the feed (wasn’t 100% clear from the post whether you had done so) it’s still not clear to me where to find it. I would like to see a link to it in an obvious place somewhere, preferably right next to the feed subscription section or even as an interim page that must be read before you can subscribe (maybe it’s already there as I didn’t go through the entire process to see).

Re: your earlier comment about Econned, I confess I’m in the “interested but haven’t purchased yet” camp. That’s largely due to procrastination and not being sure where to find it for sale locally – I’ve been planning to get it from Amazon but have been delaying until I had enough other items to help offset the international shipping cost. Although I see a Borders buy link above and we do have one of them here, so I should check there when I do my Christmas shopping.

Indigenous Centurion says:

December 5, 2010 at 8:22 pm

”
cursor worringly skitters when I’m not using it. So far, it seems like a harmless
”

Is your palm touching the touch-pad? Is your touch-pad below or beside the keys? What model number you have? Your cursor moves without your touching the mouse device? If, then your computer could be a robot. Have your tech check the MBR, master boot record and check to see if BIOS has MBR protection turned on. You don’t want to do anything financial from a ‘bot computer. Take care!

Reader says:

December 6, 2010 at 1:07 am

I hope that a brief period of truncated RSS feeds puts a dent in the site scraping and that you’ll reconsider after that, and restore the full feed.

I certainly understand the frustration about site scraping and don’t mean to minimize it. If you need to take measures to combat that, so be it.

To the extent your concerns are about monetizing the RSS feeds, though, I am somewhat surprised that you’ve been advised that reducing the content on RSS will INCREASE site revenue. Is there no possibility of adding advertisements into the full-text feed? This seems like a good way to thread the needle; I know I’ve seen these before on other full-text feeds. Are those revenues so minimal as to be not worth pursuing? I hope the possibility has at least been investigated, as the full text RSS feed is a valuable resource and I hate to lose it.

Just so you understand how some readers are impacted by this, withdrawing the full text RSS feed is almost like putting the full text behind a pay wall. I know that’s an imperfect comparison and somewhat an exaggeration, but it does have a similar emotional impact to lose that resource. It’s not that RSS users are freeloaders who want something for nothing; all the RSS users I know are perfectly content to read full streams with ads included. It’s just that once your reading habits and the technology have adapted to reading the full text in-stream, it’s quite difficult to go back. For instance, many mobile devices (and Blackberrys in particular) have decent enough RSS capabilities, but render full blog pages very slowly and inconveniently.

In any event, the content of your blog is spectacular and I hope you may in the future be able and willing to offer it via full RSS feeds without negatively impacting your bottom line. And for the record, I would never let your decision on this issue affect my decision to buy or read your book; the above poster claiming he would refuse to buy your book now is nuts. Please don’t think all of your RSS-bound fans think that way.

br says:

December 9, 2010 at 6:52 am

No! Please dont give us crippled feeds. I read your site and lots of other stuff via a feedreader (Google reader in this case).

RSS, Atom, feeds etc are really useful, innovative Internet tools, unlike Facebook. I dont get all the fuss about that although the movie was great.

The fact that a feed strips ads, sidebars, comments etc is of course a plus.

If you try to restrict your feed or load it up with ads and other irrelevant junk we are just going to find another way to get the stuff we want. You have to live with this reality.

The commenters (which I dont receive via feed) are often excellent and it is this that brings me to your site regularly.

Make the service great and try to find a way to make money some other way. To cripple the service in an effort to make a buck does seem very ‘old media’ indeed to me.

However I agree that there is something rotten with a system where a talent like yourself struggles to make a living. You should be a high-paid journalist or a tenured Professor.

But as Chomsky says the Prophets only became admired years later. At the time for their cutting and unwelcome remarks they were despised as enemies of the King and the powers that be.

hibi says:

December 13, 2010 at 4:55 pm

This makes me think of a certain former executive in my workplace, head of retail. She was so concerned about the possibility of fraud, she would consider stopping it our number one priority. She’d change procedures and software to minimize fraud, without realizing that many of her changes didn’t really prevent fraud at all, but they also lowered sales. If what she really wanted was to limit retail fraud, the best thing she could do was to close the retail stores altogether: The problem is that no good businessman wants to minimize fraud as a major objective: The major objective is to maximize profits, fraud or not. Minimizing fraud is only worth it if it increases the bottom line.

So you have to ask: How much are scrappers really costing me? How is my income, other than actual ads from the website, being affected by the website itself? Is the website supposed to be a major income source, or mostly an advertisement vehicle for other endeavors?

It’s often very difficult to answer such questions unemotionally, but to do so is the only way to use economical thinking in the matter: Think of all the people that lose sight of the goal, and spend so much time preventing behavior they don’t like that they forget that what they are really about is promoting behavior they like. It applies to bloggers just as well as business executives, musicians or independent video game developers.

Subscribe to: Post Comments