8 replies to this topic

[britfrog]

Yes this is going to be controversial ! However it is not aimed at any one particular person so dont get on your high horses

It is how I perceive things here being a newbie per se.

IT is quite clear that many software companies appreciate a good product review and it is also quite clear that a good mutley review is a reasonably prized thing to put on their websites.
However I do feel that we give away our 10 point reviews a little too easily. in recent months we have given several 10 point reviews for products that are better than average but are they really worth ten points? come on they were not that good!
what will happen when we get a really good product that is say like the pmdg 737 much better than anything before will we give it 11 out of 10? if we give it only 10 then the makers of that product will feel cheated as it received the same marks as a reasonably good product, therfore they will not hold mutley reviews very highly.
I suggest we reserve the 10 pointers for products that are really landbreaking and make it a maximum of 9 for all others this way software houses will really appreciate a mutley 10 points, they have to work for it knowing it wont be given away lightly and will be more appreciated if it is given.
right over to the peanut gallery to see what you say

[ddavid]

Here's my tanner's worth:

I've given 3 products 10/10 - they were:

1. Treescapes - Darren at Earth Simulations has done a terrific job - he's transformed the 'flat' photo-scenery of the U.K. into a living landscape. No other add-on comes close.

2. Narsarsuaq-X - this freeware scenery for Bluie West One, by Ulrik Motzfeldt, is incredibly detailed, with 4 seasonal variations and a seamless interface with a much modified set of ground textures, plus a dramatic new mesh for the area. This add-on compares to Orb-X's standards - and is FREE - hence the full marks.

3. Take On Helicopters - not really a main line flight sim - more of a game, but the model detail, representation of the Seattle area, helicopter flight dynamics, story line and mission variety make this product unique. I was totally spell bound doing the Review - immensely enjoyable - not everybody's cuppa, but top marks from me.

Oh yes - after a very embarassing first Review (no names, no pack drill) where I marked the product at 6/10, much to the chagrin of the developer - I now buy the product prior to Review. I believe this gives me a degree of independence such that I can say what I think about the product.

Lastly, every Review has to be personal - and therefore subjective. It's not really feasible to implement a set of rules for Reviewers to follow - I've thought about suggesting this but now feel that a set of rules would just inhibit the Review process. And, anyway, Joe and Rob vet each Review before publication - that's a good 'back stop' to the process.

Thanks for raising this topic - it is worth discussing. I hope my comments make sense!

Cheers - Dai.

[rob16584]

When we write the reviews we do it from our own view point and make sure that we take into account

a) What the product is trying to do
b) Value for money
c) Competition
d) Personal Experience

A good example would be of Dai's review of Treescapes compared to my review of Concorde X. Neither can compare to the other because they are totally different and aimed to different market segments, but they achieve what they set out to do far better than anything else which is available.

We try to be fair an consistent with every review, and if we do come across any problems we will go back to the developer prior to writing/publishing the review. There have been cases where we have worked with the developer to improve the add-on, and in one case we gave the product such a low score that the developer held up their hands to a poor effort and is going to re-do the add-on using our review as a starting point on where to improve. There has been another occasion where the developer tried to bully us into giving a higher score than the add-on deserved, but we wouldn't change the score. The developer asked us to withdraw the review from the website, which was done, but it still shows in the review section and you can request a copy of it to be emailed.

At the end of the day the score is always going to be subjective and is dependant on a lot of factors: system spec, time spent with add-on, complexity of add-on etc... and we try to allocate the reivews based on the review preferences. For example Dai is more into helicopters than I am so I would never take on a helicopter reivew as I wouldn't be able to give a thorough appraisal of it. We pride ourselves on giving un-biased reviews, but there are a lot of fantastic add-ons out there at the moment and they are worthy of the scores we award.

[brett]

Very good answers except for one thing that caught my eye. Why would you pull a review off the site because a delveloper asks you too. That dosen't sound like something that should be done. Good or bad a review should be impartial and the developer should have to live with it and fix his product accordingly, then request another review. I think you already go far beyond the line of duty by showing them the review before it is posted. Reviews should be used to keep the developers on their toes and put out a totally tested product, after all, they are charging money for them. An honest review of a released product should be different then beta testing by qualified folks in the sim field. JMO

[mutley]

Hi britfrog,

Thanks for your feedback, I really appreciate it.

I agree, if we were to award everything a 10 it would make a nonsense of the scoring system and just water-down the importance. As Rob pointed out, we have been getting some very good products recently and products which have stood the test of time, Jess' ViennaX for instance.

Rob's and my guidance to our reviewers is that every review starts with a 10 and get docked 1 point for a major let down or bug and 0.5 for a minor problem, then looked at and scored in relation to its competitors or predecessors.

Therefore, if the reviewer finds no problems and really likes the product then a 10 is very likely. However, Rob and I as editors look at the points raised and comments given and if not fair, or conversely too "fan-boy" then we will question the reviewers scores.

Even if the product gets a 10 it may not get our highest accolade for product excellence, so far, since we started the reviews in earnest, only 11 products (Out of 179) have achieved that.

Now that you have raised the point we will be more vigilant on the scoring, thanks for pulling us up on that one, it's good to know you are paying attention!

Regards,

Joe

[mutley]

Very good answers except for one thing that caught my eye. Why would you pull a review off the site because a delveloper asks you too. That dosen't sound like something that should be done. Good or bad a review should be impartial and the developer should have to live with it and fix his product accordingly, then request another review. I think you already go far beyond the line of duty by showing them the review before it is posted. Reviews should be used to keep the developers on their toes and put out a totally tested product, after all, they are charging money for them. An honest review of a released product should be different then beta testing by qualified folks in the sim field. JMO

Hi Brett,

We pulled the FSC Lancaster review Dai mentioned above as it was donated by the publisher/ developer who took real personal exception at the review outcome and although points were proven by Dai the developer wouldn't accept the criticism, so we pulled it to avoid and possible comeback. However, if you want to read it you can find it in our reviews listings!

[britfrog]

Morning all,
Thanks for taking this in the manner for which it was meant I did not want it to appear to be character assasination. I like this site as it does not suffer from some of the political cronyism evident on another site that will remain unnamed. Whilst I fully accept the points raised in answer by the above contributors especially that it is a personal review there are many of the hoypoloi out there that may understand this who then buy a recommended product only to find their opinion differs fom our reviewers However I am glad that I have brought this to everyones attention and that we have debated it without getting our necks in a sling.
May I therefore suggest that as it would appear that 10's are going to remain fairly/reasonably common that we retain the mark of 10+ for truly exceptional stuff?

[mutley]

Hi BF,
Yes I agree with the 10+ for exceptional products, they are the ones that get the award,
Thanks again for your feedback it certainly helps us reflect on our performance!
Cheers,
Joe

[ddavid]

Yep - the '10+' score exists as the Mutley's Hangar Award for Excellence - the 11 out of 179 that Joe mentions.

Cheers -Dai.