Printer Testing Methodology & Analysis

Updated: Jun 10, 2024 7:44 PM

TABLE OF CONTENTS

Key Takeaways_

  • 63% of printer reviewers are untrustworthy. That means only 37% of publications actually test!
  • Only 24 reviewers publish trusted printer reviews, including RTINGs, Which?, and TechGear Lab. These 24 reviewers’ trusted testing methodologies and reviews fuel our Performance Criteria indicators and True Scores.
  • ZDNet, Business Insider, and Wired are the top three most popular fake reviewers who earned failing Trust Ratings yet claim to test printers.
  • We spent 47 hours total evaluating 65 printer reviewers for thorough testing and transparency.

Printer Category Analysis

Printers have been around for decades, yet there are few legitimately tested reviews available online. After three years of research, our investigation and ongoing research have revealed that 63% of printer reviewers don’t properly test them, lacking proof of thorough testing.

How do we know what to test? Our process involves a meta-analysis of trusted expert testing methodologies, determining qualitative and quantitative tests, and validating them against real customer review pain points.

Our Dataset Explained

To calculate our proprietary Trust Ratings, we have real people research each reviewer, filling out a total of 67 indicators to gauge how they practice transparency and quantitative, proven testing.

  1. Authenticity / 9 indicators: Publication staff are real humans.
  2. Data Science / 18 indicators: Reviewer uses a thorough, numerical scoring system to differentiate products from each other.
  3. Integrity / 7 indicators: Reviewer promotes editorial integrity and prioritizes genuinely helping consumers.
  4. Clarity / 11 indicators: Content is structured to effectively communicate product information to consumers.
  5. Qualification / 1 indicator: The publication actually claims to test the category.
  6. Expertise / 8 indicators: The reviewer is an experienced expert in a category.
  7. Visual Evidence / 3 indicators: The content includes visual proof to support the reviewer’s product testing and/or usage claims.
  8. Testing Evidence / 9-11 indicators: The reviewer tested the product and provided their own quantitative measurements from their testing.

That’s how we test the testers to determine who people can trust. Once we scored all 65 reviewers, we grouped them into six classifications based on their Trust Rating and how truthful any testing claims were.

Here’s an overview of the six Trust Rating Classifications for printers:

CLASSIFICATIONNUMBER OF REVIEWERS
Highly Trusted (≥90 Trust Rating)2 (3.07%)
Trusted (70-89 Trust Rating)14 (21.53%)
Mixed Trust (60-69 Trust Rating)8 (12.30%)
Low Trust (50-59 Trust Rating)26 (40.0%)
Not Trusted (0-49 Trust Rating)6 (9.23%)
Fake Reviewers (Claim to test is not truthful & <60 Trust Rating)11 (16.92%)
Total Reviewers Analyzed65

Of the reviewers we examined, we found that 11 reviewers (16.92%) are fake; these 11 reviewers claimed to perform testing as part of their review process but provided insufficient testing data to back up their claims. The most common (40.0%) type of reviewer was Low Trust reviewers, who earned failing Trust Ratings from 50-59%.

The total cost for setting up and conducting printer testing can range from $135 – $2,225, not including any recurring costs for the facilities or labor. These costs aren’t very expensive, so the testing equipment shouldn’t be a significant barrier for many reviewers. Perhaps this is why the number of untrusted reviewers is lower than categories that are more expensive to test, such as TVs and soundbars.

tech gear lab photo print under microscope
A tester at Tech Gear Lab assesses the fine details of a photo print using a microscope. (Source: Their How We Tested Home Printers page.)

What Tests are Quantitative and/or Qualitative? 

Quantitative tests feature something that is either measurable or binary in nature, which includes tests that measure sound pressure level and tests that produce frequency response graphs.

Qualitative tests, on the other hand, are based on things that can’t be properly measured or are a “feeling” someone gets. This includes things like ease of use, setup, and soundstage.

How Often should Tests be Updated?

Tests only need to be updated as often as new tests for printers (for new features) are developed, which is infrequently.

The Top 3 Reviewers Who Are Certified Testers

When we started Phase 1 of our research in 2021, it was clear who the most thorough, transparent printer testers are. These three below earned the highest Trust Ratings in the printer category, providing detailed photo evidence, test charts, graphs, and video demonstrations.

RankReviewerTrust RatingWhy They Earned Their Trust Rating
GR video game font 196.28%
RTINGS provides extensive quantitative test results, detailed graphs, videos, videos, and real images to demonstrate their comprehensive printer testing. They answered all four of our performance criteria indicators.
GR video game font numbers 1-15which-logo92.36%Which? provides several quantitative printer test results, though they’re hidden behind a paywall. They have video content and they answered two of our performance criteria indicators.
GR video game font numbers 1-15gear lab logo88.23%Techgear Lab provides real images that include the testing equipment, helpful bar charts comparing quantitative test results, short video clips demonstrating how to use the printers, and a testing methodology. They answered two of our performance criteria indicators.

These three publications above along with the rest of our passing printer reviewers are our trusted data sources that help inform the True Score system. We average the reviewers’ test results across products and present the averages in our guides and reviews. That way you can quickly gauge the true performance of a printer across Performance Criteria:

After Phase 1 of our research, we realized that each category was scored the same way with the same set of indicators. We wanted to further refine our indicators by adding unique Performance Criteria Indicators for each individual category, since categories can be very different from each other.

To determine what these indicators would be for each category, we investigated the testing methodologies of each publication that has one in the category:

We tried to keep the Performance Criteria Indicators as fair as possible, meaning the selected criteria aren’t too difficult to test.

More than one reviewer should have already tested the select criteria, thus making it eligible to become a Performance Criteria Indicator.

The Top 3 Testers’ Products Compared

We dove deeper into the top three testers’ product reviews, and we noticed their detailed quantitative scoring systems that spanned across use cases, categories of performance, and specific performance criteria.

The depth and rigor they applied in evaluating printers allowed us to provide a benchmark for printer testing that we’d apply to Phase 1.5 of our research across the same 65 reviewers.

RTINGsWhich?Tech Gear Lab
Brother MFC-J4335DW
Family Usage: 8.2
Small/Home Office: 8.2
Student: 8.2
Black & White: 8.5
Photo: 7.3
Black & White Documents: 9.0
Color Documents: 9.0
Build Quality: 8.0
Brother MFC-J4335DW
Print quality and speed: Hidden
Printing costs: Hidden
Scanning quality and speed: Hidden
Copying quality and speed: Hidden
Ease of Use: Hidden
Brother MFC-J4335DW
Overall: 4.5/5
Printing speed: 11 simplex pages per minute
Operating cost: $5.50/ream of B&W text
Canon Pixma TR8620A
Mixed Usage: 7.3
Dialogue/TV Shows: 7.7
Music: 7.9
Movies: 6.7
Stereo Frequency Response: 7.5
Latency: 9.1
Build Quality: 8.5
Canon Pixma TR8620A
Did not review (DNR)
Canon Pixma TR8620A
Overall: 65
Color: 7.0
Black & White: 6.0
Print Capabilities: 6.0
Operating Cost: 7.0
Set Up: 6.0
Canon Pixma MG3620
Family Usage: 7.6
Small/Home Office: 6.2
Student: 7.3
Black & White: 6.7
Photo: 7.7
Black & White Documents: 8.5
Color Documents: 7.5
Build Quality: 5.5
Canon Pixma MG3620
DNR
Canon Pixma MG3620
Overall: 49
Text Quality: 3.5
Graphics & Photos: 7.6
Ease of Use: 5.9
Operating Cost: 4.2
Printing Speed: 1.8

We noticed that RTINGs, Which, and Tech Gear Lab all evaluate print quality, speed, and operating cost, which was a big indicator that we should apply these Categories of Performance to our own Testing Methodology and True Scores system for printers.

The most trusted reviewers won’t always test the exact same criteria. RTINGS tests several criteria that no other reviewer tests, like Color Accuracy. Despite not being commonly tested, Color Accuracy measures how precisely a printer reproduces colors without deviation. It’s a critical factor in assessing print quality, which is why we decided to add it and other important criteria to our Testing Methodology.

Type of Tests & The Best Testers for Each

As we looked further into the top three printer reviewers’ content, we identified which ones excelled in certain types of tests. Various types of product tests provide a comprehensive understanding of a product’s performance and suitability.

Type of TestWhy It’s ImportantBest Reviewers That Have This Test
Photo TestsHigh-quality photo tests are crucial as they provide clear visual evidence of design, build, and size, allowing consumers to assess aesthetics and practicality.A. RTINGs (Paywalled)
B. TechGearLab
Test Charts and GraphsTest charts and graphs are invaluable for displaying precise printer performance metrics, such as printing speed and color accuracy, offering a quantitative analysis that aids in comparing different models.A. RTINGs (Paywalled)
B. TechGearLab
Video TestsThey are a visual demonstration of the printer’s performance in real-world scenarios, providing a dynamic assessment of printing speed and color accuracy in action.RTINGs (Paywalled)

Together, all of these tests enable a well-rounded assessment, aiding consumers in making more informed and confident purchasing decisions. Notice how RTINGs has all of these tests, which contribute to its leading printer Trust Rating.

Best Expert Testers

Meet the leading expert testers, certified professionals who are the driving force behind the most reliable printer reviews on our top 3 reviewer sites:

  1. Thomas Szwedska – RTINGs Profile, LinkedIn, No Twitter
  2. Oliver Trebilcock – Which Profile, MuckRack, No Twitter
  3. Genaveve Bradshaw – Tech Gear Lab Profile, No LinkedIn/MuckRack, No Twitter

Testing Equipment Used

This is the equipment that the top testers used to gauge the quality of printers:

ImageEquipmentWhat It MeasuresWhy It’s ImportantPrice
stopwatchStopwatch or TimerPrinting Speed (pages per minute)Determines the speed at which a printer can produce black and white and color documents$5 – $25
nix spectro 2
ColorimeterColor Accuracy (dE)Assesses how accurately a printer can reproduce colors, critical for high-quality print outputs$100 – $1,200
PrinterPrinting OutputThe main device being tested for print speed, print quality, and operating cost$30-$1,000

Aside from the colorimeter, printers isn’t the most expensive category to test. It’s way cheaper to properly test a printer versus a TV or soundbar, which requires several thousands of dollars worth of equipment. The percentage of trustworthy reviewers in this category is higher than the amount of trustworthy reviewers in those other categories.

It’s simple to calculate printing speed and operating costs, so there’s no excuse for reviewers to claim to test these printers when they didn’t test a single criterion.

All printer reviewers should be able to add the easier test results into their content. Or if the fake reviewers can’t for some reason, they should say that they “researched” their printers versus saying “tested”.

The Top 3 Printer Reviewers That Failed

The following three printer reviewers are fake reviewers who claim to test printers and earn the most printer keywords. We recorded this printer keyword dataset from Ahrefs in June of 2024.

RankReviewerTrust Rating# of Printer KeywordsWhy They Failed
GR video game font 117.71%6,403ZDNet provides some real images in their product reviews, but they don’t test the criteria that truly matter, such as print speed and operating cost. They did not answer any of the Performance Criteria indicators so their claim to test is not truthful.
GR video game font numbers 1-1534.29%5,991Though they have some real images, Insider only uses qualitative tests on their printers they review, failing to answer all of our Performance Criteria indicators. Thus, their claim to test is not truthful, making them a Fake Reviewer.
GR video game font numbers 1-1523.58%3,611Wired doesn’t provide real images of the printers nor any quantitative test results. Only qualitative conclusions from their claims of using the printers. They did not answer any of the Performance Criteria indicators, so their claim to test is not truthful

By earning a failing Trust Rating and claiming to test, these reviewers are publishing fake printer reviews that aren’t properly tested.

We suggest that these publishers earn a higher Trust Rating by improving their testing standards, practice transparency, and publish more quantitative test data that would answer our Performance Criteria Indicators. Or if that’s not possible, they should change any mentions of “testing” in their printer reviews to “researched”.

Categories Of Performance & Performance Criteria

From our Phase 1 of research, we identified the important Performance Criteria used by the most trusted printer reviewers’ product reviews and Testing Methodologies. There are 4 Performance Criteria for printers.

Performance Criteria are individual standards, measurements, or benchmarks used to evaluate and assess the performance of a product for a specific use or purpose. These fuel our Trust List which lists the printer reviewers you can trust and can’t trust.

It’s important to note that not every performance criteria that apply to printers is covered here, only the ones that are most important to the most prevalent use cases for printers.

The section below helps explain the Performance Criteria of printers along with their units of measurement and why they’re important to test.

Printer Categories of Performance

When deciding the Performance Criteria, we looked at who are our trusted reviewers along with customer pain points we found in customer reviews:

Eventually we learned which Performance Criteria matter the most to users and are tested by our trusted reviewers:

Category Of Performance# of Performance CriteriaDefinition
Printing Performance3The efficiency and quality when a printer produces documents and images. It evaluates the speed, accuracy, and reliability of the printer in delivering printed materials.
Cost Efficiency1The overall financial effectiveness of operating a printer. It includes long-term costs, including expenses for ink or toner, paper, energy consumption, and maintenance.

Printing Performance and Cost Efficiency represent two fundamental Categories of Performance for printers, and each are made up of Performance Criteria which address aspects of a printer’s functionality and user experience.

PRINTING PERFORMANCE

This refers to the efficiency and quality when a printer produces documents and images. It evaluates the speed, accuracy, and reliability of the printer in delivering printed materials. This consists of three Performance Criteria.

CoST EFFICIENCY

This is the overall financial effectiveness of operating a printer. It includes long-term costs, including expenses for ink or toner, paper, energy consumption, and maintenance.

Usage Cases

After identifying the most important Categories of Performance and Performance Criteria, we categorized them according to real-world applications of document printing, photo printing, and educational and marketing materials.

Document Printing

In business and office settings, they are essential for printing reports, contracts, invoices, and other business documents. At home, they are used for printing school assignments, personal documents, and recipes.

Tests: Black and White Print Speed, Color Print Speed, Operating Cost

Photo Printing

Photo printing is another significant use, both for professional photographers and hobbyists who need high-quality prints, as well as for individuals who want to print family photos and personal memories.

Tests: Color Print Speed, Color Accuracy, Operating Cost

Educational and Marketing Materials

Marketing materials such as brochures, flyers, posters, and business cards are frequently printed by small businesses and marketing agencies.

Educational institutions rely on printers to produce worksheets, exam papers, and other educational resources, and home-schooling parents use them for preparing teaching materials and assignments.

Tests: Black and White Print Speed, Color Print Speed, Color Accuracy, Operating Cost

Performance Criteria Indicators

During Phase 1.5 of our research that we initiated in 2023, we fine-tuned our Trust Rating process further. The latest phase involves the addition of Performance Criteria indicators tailored to individual categories since the type of testing can vary greatly across categories.

These are indicators designed to cut to the core of what a consumer looks for in a product that guarantee that a product was not only used by a reviewer, but also properly tested to provide meaningful data and insights that a consumer can use to make a purchase decision. For this category, our Performance Criteria indicators are:

  1. Does the publication run quantitative tests for mono (B&W) print speed? (seconds, minutes, in/s or cm/s)
  2. Does the publication run quantitative tests for color (CMYK) print speed? (seconds, minutes, in/s or cm/s)
  3. Does the publication run quantitative tests for color accuracy (dE)?
  4. Does the publication run quantitative tests for cost per print?

Wrap-Up

Our Printer Testing Methodology underscores our unwavering commitment to transparency and accuracy in evaluating printer performance. These Performance Criteria fuel our Trust Ratings that we calculate for each reviewer during our ongoing research, which we analyze further in our Printer Trust List. The crazy fact that 63% of printer reviewers lack enough testing emphasizes the need for our meticulous approach.

The Trust Ratings are the bedrock of our Printer True Scores, which is the most accurate product score on the web. Below are our printer reviews sorted by highest True Scores. We invite you to learn more about how we score products with True Scores and how they empower you to make informed buying decisions online.

We also invite expert reviewers to contribute to a more trustworthy and transparent review system. By adopting stringent testing standards and/or supporting our efforts, you help elevate the quality of printer reviews and ensure that consumers can make informed choices based on reliable information.

Let’s commit to higher standards together. Together, we can transform the landscape of printer reviews for the better.

Courtney Fraher Avatar

Latest Reviews

Learn More About Printers