My question was more one of clarification because I couldn't clearly see the intent of your statement. You focus your comment on price and in the same context talk about fairness and impartiality. What about all the other considerations and assessments which are done against a product? In the case of aircraft add-ons, an MH review makes assessments across seven discrete categories, all weighted equally.
You also erroneously quote an average $50.00 (sic) price tag for an A2A product. This is misleading in itself as the $50.00 (sic) A2A product is a single platform licence, i.e. it is either for FSX or Prepar3D. All Alabeo / Carenado products are a multi platform licence for FSX, FSX:SE, and Prepar3D. A2A don't even offer such a comparable multi platform product with the closest similar product being the "Academic Bundle" product for FSX and Prepar3D, priced at $69.99. So, on that basis, and using your example of the A2A Comanche and the Carenado PA-42 Cheyenne III, the real price differential is $30.00 not $10.00. But let's be serious, regardless of developer, these two real world aircraft are like the proverbial "chalk and cheese". The real world counterparts of the Piper PA-24 Comanche are the Beechcraft A36, V35B, and F33A Bonanza, Cessna 210, and the Mooney M20. Note that I have specifically chosen these real world counterparts as they are specific models in the Alabeo and Carenado product range. Therefore, these will provide a more accurate and realistic product price comparison. The aforementioned Alabeo / Carenado products are priced across $26.95, $29.95, and $34.95. Now we start to see the reality of a better like for like comparison and, on price alone, at better than two to one in all instances, the A2A Comanche isn't smelling too good.
So, now let's get back to the subject of reviews, the other point in your comment.
From a reviewer's perspective, and at least since my time as Head of Reviews at MH, regardless of the add-on type, aircraft, scenery, etc., we have never, and will never do a direct comparison against other products in the review for a number of reasons:
It is extremely rare to be able to compare like for like products. I've already put the issue of price to bed, so let's look at some other areas. Again, for example, what in the A2A Accu-Sim range do you compare the Carenado PA-42 Cheyenne III to? Other than it being an add-on aircraft, generically, they are completely diverse products. The Accu-Sim products have a higher level of systems complexity, whereas the PA-42 Cheyenne III is a more technically complex aircraft to model. In the rare instance where you may get similar products from different developers, what is the time gap between the development of the respective products? The later product would be expected to benefit from advances and developments in modelling, texturing, and systems coding over an older product, thereby garnering an advantage, but also possibly costing more in development time and, therefore, a higher retail price.
Developers often have different business models, aims, or development criteria for their products. In the case of the A2A Accu-Sim range, at least in part, it is aimed at real world professional flight training schools. Hence the $799.00 price tag for the licence of the commercial versions. The cheaper versions of the same product are aimed at the more serious and procedural flight simmer. As further examples, the same can be said for the Majestic Dash-8 Q400 product range, and the PMDG product range. The level of realism and systems modelling in all of these products are aimed at the more serious and procedural flight simmer. Neither Alabeo or Carenado aim their products at this sector of the market. They do aim, and have set a certain default standard, for an extremely high quality in visual modelling, something they do consistently well. From a business model perspective, Alabeo started with a stated aim of "bringing a different type of flying experience to people who may otherwise never get the chance to experience it", with "lower priced" aircraft, Alabeo products "combine the joy, the challenge, and the fun of flying". Consequently, there are certain inaccuracies or deficiencies in their modelling, particularly in the cockpit. However, not one of these inaccuracies or deficiencies prevent the aircraft from being flown. Regardless, Alabeo and Carenado are constantly criticised for not producing to a level of realism they never claimed or intended to produce.
MH reviews are structured to make quantitative and qualitative assessments against certain scoring categories (more easily done for aircraft and utility add-ons than scenery) to provide a measured degree of objectivity. For example, physical performance against the documented and real world aircraft's performance data, accuracy in the visual model against the real world counterpart, the quantity and quality of supplied documentation, etc.. This measured degree of objectivity is intended to provide a basis upon which readers can make certain indirect comparisons with other products. This is not always a precise methodology but it is considered the most consistent and equitable approach. I will also point out that over the history of MH Reviews, encompassing 386 reviews by 28 reviewers, the scoring, in the main has been consistent, not bad considering no T scoring has ever been applied.
Any reviewed product stands alone to be assessed on its own merits against the scoring categories. To do any form of comparative analysis involving quantitative and qualitative assessments requires the compared products to be reviewed equally. This is restrictive on so many fronts, the least of which is time alone. Secondly, copies of reviewed products are often paid for out of the reviewer's own pocket. In situations where press copies are obtained, the developer understands that we do not use a comparative analysis format for the review.
MH reviews start from the premise of being written for the average flight simmer. Who is the average flight simmer you might ask? Well, let me start by saying I don't consider anyone who has posted in this thread to date to be your average flight simmer, and it certainly would exclude the (extreme) individuals on the AVSIM Carenado (UNOFFICIAL) Forums. All of us once were your average flight simmer but we have all long transcended that level as is evidenced in the level of discussion and subjects of these posts. In some instances, the product itself will dictate the level of its audience, a review of a PMDG product would be a good case in point.
I will finish with some slightly more personal views. The reviewers at MH do what they do for the general benefit of MH members and the broader flight sim community as a whole, as a service to help inform and / or provide a basis upon which to help fellow flight simmers make a decision on a product. I am proud to stand with each of the reviewers and totally respect them for their knowledge and the work they produce. We get very little personal gratification out of the process, and the writing of each review, editing, and preparation for the web site involves tens of hours of our personal time and effort when we could be doing something more enjoyable with our time. We also do it, putting our work out into the public domain, knowing and accepting we will be criticised by all and sundry who have a differing opinion and are not prepared to or cannot accept other's opinions. We take criticism from individuals, both members of MH and others from the broader flight sim community, who more than often have very little understanding of our review process and review structure and have probably never written a review or article in their life, but sure as shit flows downhill, they will bombastically tell you that you are wrong. They justify themselves with meaningless claims like, "...I have 5,000 hrs on VATSIM..." as if it is meant to give the individual more credibility. Others will bombastically claim to be real world pilots with an extensive flight log of hours on certain types, which, whilst it gives them real world credibility, they expect, to the point of demanding, the same veneration in the flight sim world. I'm sorry, and I make no apologies to such people, but respect is earnt and given when it's due. Then you will get the gutless individual, and I use the term gutless deliberately, who will use the MH contact form to question the credibility of a reviewer by stating the reviewer "...is obviously not a real world pilot..." on the basis that the reviewer did not identify all the missing real world functionality on the ProLine21 avionics suite. In this instance, the provided functionality was more than enough to monitor the required basic systems and fly the aircraft model as intended by the developer, and yet this man among men hid behind a false email address, such that a response could not be sent to him. Now, I have been labelled and called many things in my life, and people see me as being blunt, rude, and often lacking empathy. For many reasons I am all of them, partly because I'm an Australian, and as such we simply tell it like it is, no pretence, pretentiousness, and more often than not without malice, it's just simply called the truth, but misinterpreted or called rudeness by those who can't accept it. Lacking empathy, sure, because sometimes life just does that to you and the more you put yourself in the public domain, the more you bear the brunt of unwarranted criticism and put downs. But I will continue to put myself out there in that public domain for as long as I think I still have something to share and offer advice on or assistance with based on my experience. If people don't like the advice or disagree, or if I am wrong, that's fine, I am man enough to accept it, but I will also defend my position when they are wrong, and often, that's when the same level of acceptance is not reciprocated.
I have strayed somewhat from a simple response, and nothing written is intended as a slight, personal or otherwise. It is merely offered for a deeper understanding and awareness of where the content and level of detail in the response comes from. All examples of individuals quoted in the above paragraph come from real incidents experienced in my time at MH.