April 25, 2017

Amazon Review Revisited

For those who came in late…

 About seven months ago I came up with this idea of running a little experiment with Amazon’s review policy. A new wave of complaints had cropped up about reviews being deleted or blocked, and—as they tended to-there wasn’t a lot to them past “Amazon took down my review of XxX!” Supposedly, the new rumor at the time was that everyone’s favorite online megastore was surfing social media sites, looking for potential connections between reviewers and reviewees—and using them as a reason to delete reviews.

It struck me that I’d been hearing about review policies for years, but never seen any hard data on them. It always came back to he said this, she said that, lots of people had it happen to them. There were never any hard facts.

So… I decided to find some.

I reviewed thirty books I’d really loved. One every day for the month of August. I listed out all the social media connections between me and each author. I even did a handful of control reviews—ones that should get pulled regardless of social media connections. And I listed all of it out for everyone to see. And tweeted about it. And talked with folks on Twitter about it.

A month after my little experiment ended, nothing had happened. No warnings, no deletions, no reprisals… absolutely nothing. Even on the control reviews, which really should’ve been removed under every possible version of the review guidelines. I left it at that and decided to check in six months later.

Which is… right about now.

How many control reviews finally got spotted by Amazon’s algorithms? How many warnings were issued? Did my account get frozen?

Pretty much across the board… nothing’s changed.

All thirty reviews are still up, including all the blatant control reviews.  Heck, two of the control reviews even have “people found this review helpful” checks.  I never heard a peep from Amazon. Even with the tweets.

Seven months since the first review, Scott Sigler’s Alight (great series, check it out).  Scott and I have known each other for almost our years, if memory serves. We follow each other on Twitter, we’re both with Random House subsidiaries, we’ve done panels together, he even interviewed me last year at WonderCon in front of an audience of about three or four hundred people. There is absolutely, no question a connection between us.

That review is still up.

I feel pretty comfortable saying the social-media scanning algorithm is either a myth or reaaaaaaally poorly written. If it can’ t find a connection between me and Scott, it’s pretty inept. Same holds for me and the next two authors on the list—Chuck Wendig and Eloise Knapp. There’s social media connections and shared blurbs galore. Heck, with both of them I think there are pictures floating around. Incriminating pictures, for these purposes.

And yet… the reviews are still there.

So, yeah, the social media bot probably isn’t real. I wouldn’t bet anybody’s life on it, but the evidence sure seems to point that way.

I think there’s another possible conclusion we can draw here, too. I might be stretching here, so bear with me. Feel free to point out flawed logic.

The control reviews have nothing to do with the social media bot. As I mentioned above, just as they are they violate the basic rules for reviews. And all six of them are still there. Yeah, six examples isn’t a great number for a data pool, but considering the 100% survival rate…

I think getting reviews pulled doesn’t have anything to do with the reviews themselves. I think it has to do with me. Or at least, my account. Last time, one of the spitball-hypotheses I tossed out was that Amazon only applied its all-seeing eye to accounts based on suspicious activity or complaints about said account. I’m more inclined to lean that way after six months of no activity.

So if my review of Yakko’s Yappy Dog Omnibus gets quickly pulled, I think it’s more likely because of something else I did in the past than anything about this particular review.

But, again, other ideas are always welcome.

If I happen to notice anything happen with these reviews, I may revisit this again. Barring that, though, I’m probably done with it. Feel free to share the data with anyone next time you hear about reviews being pulled.

Or, in the spirit of science, repeat the experiment and share your results.

            Two months since I first started all this.  The goal was simple—we’ve all heard anecdotal stories about reviews being deleted for a number of reasons, but they tend to be kind of random and rarely have a lot of other information about them.  Also, Amazon’s policies change a lot and seem to go through… well, random enforcement.  I wanted to create a big set of data that people could refer back to when they talk about such things.
            I did this by taking thirty books I’d read over the past year (thirty really good books, to be clear) and doing a review a day for the entire month of August. Okay, almost the entire month.  I recorded the title, the author, the day the review posted, and every social media or publicity connection I could think of to said author (supposedly, this is one of the big things Amazon keeps an eye on).
            It’s been a little over thirty days since the last of those review posted. 
            What’s happened in the weeks since then?
            Okay, lots of stuff.  But as far as this goes…?
            Well, I went back and checked all the reviews.  They’re all still up as I write this.  Six of them even got marked with the little “X out of Y people found this review helpful.”  One of those is a control book, too.
            I’ve heard nothing from Amazon. Nada.  Zip.  No warnings or alerts or even a mild slap on the wrist.  Nothing on my account or in my email.
            And keep in mind—some of these reviews should be deleted.  They blatantly violate the review rules. There’s a bunch of control reviews where I have a big conflict of interest by offering my “unbiased” thoughts.  Heck, I even admit in them that they’re biased.
            Plus—I haven’t exactly been secretive that I’m doing this. I’ve mentioned it on Facebook and on Twitter, and it was shared/retweeted a fair amount. More than a few of the authors mentioned their reviews publicly, and I’ve usually mentioned this little experiment in the responses.  I’m not going to say this was trending anywhere, but things haven’t been dead-quiet, either.
            So if there’s a social media bot/algorithm searching social media for connections… it’s doing a pretty poor job. 
            Anyway, what can we learn from all this?
            A few ideas…
            Firstis that there might be more to the reviews that have been deleted than we’re being told.  Maybe I logged in to my Amazon account through my author-friend’s computer and some bot registered that?  Or possibly that we share the same IP, depending on just how close I am to said author-friend.  Perhaps I’m very, very bad at sockpuppeting?  Maybe I wrote in all caps and set off a different bot?  There’s so many things that could be a possible trigger, it’s hard to be sure exactly why something was deleted.
            This feeds into my second idea which is that my reviews might only get pulled when someone reports them to Amazon.  Perhaps having the same last name as the author, related or not, made someone shout “J’accuse!”  Maybe somebody’s a bit timid and was offended by some of the colorful terms I used to show how much I liked this book.  Possibly it’s a new form of clever attack by paranoid folks—I can’t write a nasty review of your book to bring down its rating, but I could tell Amazon those two very positive reviews were actually written by your best friend/significant other/somebody you paid.  Heck, if I’m trading reviews with you, it’s even possible the deletion is an attack against me, not you.  How often have we seen some crazed nut chase somebody around social media responding to any and everything they post…?
            Third,  over the past year or three I’ve sometimes wondered if this is actually a clever trick by Amazon to encourage self-policing.  I mean, if we all know our potentially nepotistic reviews are going to be taken down, we probably won’t waste time putting a lot of them up, right?  Right there, that could cut 50% or more of potentially troublesome reviews—and all it cost them was a press release about their latest policy.
            I know I did this for ages.  There’s about a two or three year stretch where I didn’t write any reviews because everyone had me convinced Amazon would pull them immediately.  And I had stuff to do so… why use up that time? Instead I’d often get in touch with the author somehow, let them know how much I liked their book, and offer a blurb if either of us thought my name could offer any weight for them. 
            But I didn’t write any reviews.
            Fourthis something Chuck Wendig suggested to me.  After the reviews went up, he got in touch on Twitter and bounced an interesting idea off me, based (I believe) off a few observations and some of the more… aggressively negative reviews a few of his books have attracted.  His thought was that the automatic deletion is more likely to happen to people who’ve had reviews deleted before.  If one of my earlier reviews was reported for breaking one of the rules, Amazon would be more likely to apply their uber-algorithm to my later reviews.
            This actually makes sense. More than a few folks have pointed out the raw amount of data the algorithm would have to process for every review of every book on Amazon (easily, say five million), and then cross-referencing them with every social media contact said author has (we could probably say, what, five thousand as an average, since Amazon is counting both ways).  By my rough math, that’s like a batrillion calculations.  It’s not a complicated thing to do if you’re just searching for a connection, but as brute-force work goes that’s a fair amount of number-crunching. 
            However, if we’re going to limit it to authors/reviewers who’ve already been reported “manually,” so to speak, those numbers probably shrink by a very large percentage.
            If that was true (and again, we’re just spitballing—it’s barely a hypothesis), it might explain why some people have reviews that never even post while others (like myself) can put up a couple dozen with no problem—even on the ones that should be problematic.
            Which of these are true?  No idea.  There’s a bit of potential overlap.  All four of them fit the small amount of information I was able to glean from this.  And there’s probably other theories that would fit, too.  I’d love to hear your thoughts on any of them, or on your own based on what we’ve got here.
            What we can say is that Amazon definitely isn’t deleting all reviews.  Not immediately and especially not based off social media connections.  We’ve got thirty examples to prove that right here. 
            So the next time someone tries to tell you that a bunch of reviews get deleted for no reason, you can point them to this
            Which I think brings us to the end of this little experiment.  The links are all there if anyone wants to check back at any point to see if anything’s happened. Maybe I’ll check back in six months (April or so)  just as a late follow-up to see if anything’s happened.  And if anything happens before then, I’ll definitely let you know.
            Come back in two days when I’m going to talk about…
            Well, maybe three days.  I’ll get to it eventually.
            Until then… go write.