Dec 18, 2023: Version 2.0 True Score System

Our True Score system now utilizes Bayes Theroem, a much more comprehensive, probablistic model that utilizes machine learning to adapt scores on the fly.

To attain the most accurate and insightful product score, we employ advanced Hierarchical Bayesian Models (HBMs). These models are adept at pooling information from a multitude of sources, both from customer-generated reviews and expert assessments. Expert review websites are carefully evaluated based on a rigorous and systematic ranking scheme, where each website undergoes a thorough analysis by seasoned professionals. These individual expert scores are then integrated across different review platforms.

The utilization of HBMs allows us to merge these expert evaluations coherently, accounting for possible variations and biases inherent in different reviewing methodologies. This consolidation of expert scores adds a layer of sophistication to our models, enabling us to glean deeper insights into the true quality of each product. The hierarchical structure of the models ensures that both expert and customer reviews are weighed appropriately, lending a balanced, unbiased viewpoint that is representative of both professional and public opinion.

Hierarchical Bayesian Models are sophisticated machine learning tools employed in a wide variety of contexts, including healthcare analytics and recommendation systems like those used by Spotify and Netflix. These models navigate complex data structures effectively, synthesizing multiple layers of information to produce robust conclusions. This is particularly vital when dealing with sparse or inconsistent data sets, as the models can “borrow strength” from the entire pool of data, thereby mitigating the impact of outliers or skewed distributions.

By leveraging the capabilities of HBMs, we achieve more than simple review aggregation; we offer a nuanced synthesis that accounts for variability, uncertainty, and the multidimensional nature of product quality. These models are designed to adaptively update as new data are accrued, offering a dynamic, continuously refined product evaluation. This state-of-the-art analytical approach provides an exhaustive, ongoing assessment of product quality, serving as an invaluable tool for market analysis and product development.

March 6, 2023: Version 2.0 Category Trust Score Created

Note: While it may seem odd that our v2.0 was created before v1.5, this is not an error! The intent of our v2.0 confidence research was to create a new and improved process with the Category side of our criteria, but it is a time-consuming, top-to-bottom process.

The General side of the criteria remained the same as it was in v1.0 aside from adjusted point values. We wanted to avoid leaving the gaps v1.0’s category research had, however, so v1.5 was created after the fact to update existing research with new data from some of the most important parts established in v2.0.

As of this writing (July 12, 2023), a majority of our current research uses the v1.5 model, but will be updated over time to v2.0.

• The biggest change in this update is the addition of our Custom Questions to break out what were previously just two generic queries regarding quantitative and benchmarked testing.
  • In this system of “Custom Questions” (or CQs for short), for each category (e.g. best TVs, best gaming mice, etc.) we research leading reviews and buying guides from top authorities in the category like RTINGS, PCMag and GearLab to see what the experts test for. We also search ecommerce listings and Google keywords to see what consumers care about most. Informed by this research, we brainstorm 3-10 questions that outline what we should be looking for in each review or buying guide to prove that a given publication really knows their stuff. These fulfill the same purpose as the questions they replaced, in a more granular and specialized manner.
  • We try our best to keep these CQs focused on quantitative data that can be measured in hard numbers, concrete and defined units. However, for certain categories such as blenders or VPNs, the nature of the products and services in question means that the best we can reasonably do is to scrutinize what is reported on qualitatively, such as the texture of a blender’s smoothie or the user experience and security features of a VPN, in service of the same goal to determine who is and isn’t a trustworthy expert in the consumer space.
• Previously, we were checking for real-world photos of products in either buying guides or product reviews. As of v2.0, we now check buying guides and reviews separately, as many publications will satisfy this question in the latter but not the former.
• Formerly, great consideration was granted to how long the author of a given buying guide had been writing in the industry. Obviously, a well-tenured writing staff is generally a vote of confidence for a publication, but we found that during the research process, scores for the same site could vary wildly based on which article happened to be picked for research. A few steps were taken to rectify this problem:
  • Where before we had multiple incremental questions checking for a tenure ranging between 3 months and 10 years, we now only make one single check for an author who’s been writing for the category for at least one year.
  • Using formulaically assembled Google search links, we have standardized the way we search for articles on each site, helping reducing variance in the research process.
  • Instead of just one product review, we try to look for three in the category from each site, and separately denote whether each claims to test or not.
• In addition to checking for the presence of a testing methodology in the category, we also check if it was marked with the date it was published or last updated. Many of the categories we cover include emerging or otherwise fast-moving technologies that improve near-daily, and so in the world of tech reviews it’s important to keep your information up to date. This is also a factor of transparency, which feeds into a publication’s overall trustworthiness. A testing methodology with no listed date on it is a hit to its veracity.
• The score values of each criteria have been adjusted.
• Along the same lines of our confidence research for publications, we implemented a similar process for reviewing YouTube videos and channels to be able to cover videos that appear in Google searches.

May 23, 2023: Version 1.5 Category Trust Score Created

• Score Weights adjusted to reflect our focus on evidence based testing, both visual and reported through text data, charts and graphs.
  • Point values of Phase 1.5 Research were converted to a 100-point scale.
  • Qualification Category of Performance changed from 22% to 10%
  • Visual Evidence Category of Performance changed from 33% to 30%
  • Methodology Category of Performance changed from 22% to 20%
  • Testing Proof Category of Performance changed from 22% to 40%
• Removed questions from the Testing Proof section that are replaced and covered in greater detail by our new custom questions segment.
  • Quantitative product test results and/or category scores in PR and/or BG}
  • BONUS: Quantitative benchmark (comparative) product tests in PR and/or BG exist}
• Added a question to Methodology to help show the focus of our research: getting to the bottom of who does and doesn’t test.
  • Do they claim to test?
• Added questions to Testing Proof designed to look for proof that any claims of testing a publication makes are properly supported:
  • Do they provide correct units of measurement to help support that they actually tested?
  • Did the reviewer demonstrate that the product was tested in a realistic usage scenario? 
  • (e.g. assessing a pair of headphones’ sound quality with different genres of music, riding an e-bike on various inclines, blending ice in a blender, etc.)”
  • Custom Questions (Up to 10 per category)
    • Criteria of this and all following custom questions determined per category relating to relevant aspects that should be tested to demonstrate technical understanding of the products; these are often tests of performance criteria by way of measurement or use case, such a testing for brightness, how well something crushes ice, or how much debris a vacuum picks up
    • These questions are connected to performance criteria/categories of performance
    • Each category receives a different number of custom questions, but the sum point value of all of the custom questions is always the same, regardless of whether there are 3 or 10
  • Based on the custom questions and your own assessment, is their claim to test truthful?