The Complexity of Cupping: Ensuring Accuracy and Credibility Through Alignment and Calibration

(Editor’s note: This article originally appeared in the January/February 2018 issue of Roast Magazine. Photos by Julia Luckett Photography. Links have been added for context.)

Cupping is fun! It’s an exciting process of quality discovery that is approachable and easy to understand.

Most coffee professionals have participated in coffee cuppings or at least attempted to sniff, slurp, spit and savor. Those who do not taste coffee as part of their professional responsibilities often aspire to learn. Product tasters are envied and admired for their seemingly pleasurable jobs and the sophisticated skills necessary to complete their work. Cheese, whiskey, ice cream, wine — the list of products that require professional tasters is endless.

There is a mystique that surrounds coffee tasters, specifically their ability — both innate and acquired — to intimately understand coffee’s sensory attributes and accurately articulate their perceptions. The process is easily understood and accessible to consumers, enthusiasts and professionals alike. At its simplest form, anyone can take a sip of coffee and offer an opinion. The complexity of cupping begins with the credibility of the taster and the accepted accuracy of his or her opinion.

The Basics

While studying culinary arts at Johnson & Wales University in Providence, Rhode Island, I was taught to consider various flavor aspects of the food and beverage products we evaluated. Flavor refers to the combination of aroma and taste. Initial training was conducted on what were then considered the four basic tastes, which have since been expanded to five: sweet, salt, bitter, sour and umami (the taste of glutamates, often referred to as savory). We were trained to evaluate aroma on every food and beverage item we encountered, including herbs and spices. Because humans can detect millions of unique aromas, this type of training should be ongoing for all professional tasters.

While tasting, students were instructed to consider all the perceived attributes. For quality control and product development, we were trained to consider aroma, texture, firmness, base taste, spice levels and spice identification, sweetness levels, salt levels and aftertaste. If we were evaluating from the consumer perspective, we added visual analysis, including plate composition, product color, size, shape, height and artistic quality of the garnishes. This training was essential as sensory analysis and critical evaluation of food products is used extensively throughout the food and beverage industry.

When it comes to the complexities relevant to the sensory evaluation of coffee, in my judgment there are five disciplines that determine an effective coffee cupper:

1. The physical ability to identify flavors (aroma and taste) and their intensity
2. A strong sensory memory to recall flavors, determine profiles and score consistently
3. Effective communication skills to articulate flavors and other sensory attributes
4. The ability to quantify the quality and intensity of flavors and determine a numerical value on a sensory scale
5. “Cupper’s Zen,” the ability to calmly focus one’s attention on the coffee’s sensory attributes without distraction, prejudice or emotional bias

“I would argue that what you refer to as ‘Cupper’s Zen’ is one of the most important skills to develop as a cupper,” says Dana Foster, a trader at Atlas Coffee Importers in Seattle. “We have relationships that span over a decade. The real skill lies in the ability to keep that love separate from what is revealed on the cupping table. In the end, quality must prevail.”

Training

Aspiring cuppers train on these disciplines through experiential education and classroom programs.

“These are learned skills that come with practice,” says Foster. “And practice. And lots more practice.”

Formative assessments are used to provide ongoing feedback, similar to apprenticeships, working side-by-side with experienced, calibrated cuppers to learn the necessary skills. Summative assessments are intended to evaluate the cupper’s educational progress or professional expertise using an established set of standards, and typically involve a formal proficiency exam to obtain a license, employment or promotion. Both formative and summative assessments require the ability to determine the same score and sensory description for the same coffee during different cupping sessions.

Understanding the difference between cupper alignment and cupper calibration is critical for individual skills assessments as well as the credibility of a sensory panel. Many people incorrectly interchange these words and functions. Cupper alignment refers to the similarity of sensory scores and descriptions among cuppers on a panel. Alignment identifies the range between lowest and highest scores, and is best used to identify outliers (cuppers whose scores are not consistent with the majority of the panel). Alignment is conducted cupper-to-cupper and does not determine the accuracy of sensory scores.

“Sharing individual results is extremely powerful and benefits everyone in the group,” says Beth Ann Caspersen, coffee quality manager at Equal Exchange in Massachusetts.

To prevent less experienced cuppers from being influenced by more senior or experienced panel members — regardless of the order in which scores are reported to the group — scores should not be changed during an alignment discussion.

Unlike alignment, calibration compares an attribute to a recognized standard. Initial training on flavor standards is used to gain familiarity with flavor perceptions and is the basis for summative assessments. Cuppers align with each other; cuppers calibrate to standards. Examples of standards used for training are the basic organic acids found in roasted coffee (acetic, citric, lactic, malic, phosphoric and quinic); herbs, spices, fruits, nuts and vegetables also may be used.

Calibration proficiency tests are conducted as blind evaluations using standard flavors that are available globally. Tools for calibration include the World Coffee Research (WCR) coffee lexicon, Le Nez du Café coffee aromas and FlavorActiV flavor standards.

When scores and profiles are discussed and compared between cuppers at the end of a cupping session, that’s an example of alignment, not calibration.

Common Language Issues

Coffee is a global product; supply chains are far more sophisticated now than in years past; and producers, mills and exporters are as interested as roasters and baristas in coffee profiles. Each country, whether producing or consuming, has established expectations and flavor preferences based on cultural differences and foods available for reference.

Foods typically are not universal in flavor and quality, making calibration without controlled standards challenging. Flavor terroir is the concept of the taste of place, specifically the impact of geography, weather and horticulture. The same food grown in different terroirs will not have the same flavor. Because of this, terms commonly used in consuming countries to describe flavors may be unknown to other stakeholders in the supply chain, or even to cuppers in other consuming countries.

Given this challenge, “moving toward a universally agreed-upon lexicon that can be taught across cultures, like WCR’s sensory lexicon, becomes imperative,” says Andi Trindle Mersch, director of coffee at Philz Coffee in San Francisco.

Scoring

All sensory scales are arbitrary. Each sensory scale is created based on the personal choice or research conclusions of an individual, committee or group. Once created, the success of a sensory scale is determined by the number of people or groups who adopt it as an analysis standard.

The most basic coffee scale consists of a three-point score: 1 is under quality, 2 is at quality, and 3 is above quality. This may sound overly simplistic, but a basic scale to identify expectations below, at or above quality remains an often-used professional tool in the industry. Conversely, the Specialty Coffee Association (SCA) cupping form is complex and requires theoretical and practical training to master. Many roasting companies use elements from different systems to meet the needs of their businesses.

“At Philz Coffee, we rely on a combination of scoring systems,” says Trindle Mersch, “like the SCA form and an in-house scale that aligns to specific target profiles for our green bean purchasing.”

Assigning a number to a taste perception is not an easy task. The first question typically is whether the number represents quality or intensity. Next, are the numbers universal for all coffees, or relevant only to the individual expectations of each producing country, processing type, species or variety? Finally, should the numbers represent the product’s sensory profile or likability? Likability scales, often referred to as hedonic analysis, are most commonly used for product development and consumer insight groups; however, likability also may be part of a buyer’s approval consideration.

A coffee’s sensory profile, likability and balance are all part of the scoring equation at Equal Exchange, says Caspersen.

“You might cup a sample that is an 83,” she explains. “It has good characteristics, but you don’t really like the coffee. It’s balanced, sweet, clean, uniform and a solid cup of coffee, but you as a buyer don’t like it for your product, so while your score may align with the seller, you don’t like it enough to buy it.”

Numeric scoring can be as simple as a three-point scale, as previously presented, or as complex as a 100-point scale or beyond. It is important to note that when numeric scoring is delineated to fractions — for example, quarter-points or tenths — the complexity and level of detail greatly increases. As the level of complexity and detail increases, so does the difficulty of training for calibration and alignment. Most people prefer numeric scales with an odd number of points so there’s a definitive midpoint.

“When assigning a score, it is important that all parties understand the definitions of the scale,” says Dorothea Hescock, senior processing manager at Massimo Zanetti Beverage USA in Virginia. “The scoring system chosen must align with the purpose of the cupping. Scoring finished goods may differ from scoring green coffee for purchase.”

The first step in creating a coffee cupping form is to determine the attributes for analysis. Basic attributes for analysis were evident in the first Specialty Coffee Association of America (SCAA, now SCA) cupping form, published in The Coffee Cuppers Handbook by Ted Lingle (third edition, 2001). These included fragrance/aroma, acidity, flavor, body, aftertaste and balance. The scores for this form are based on intensity perceptions, not quality. This was a comprehensive revision of The Basics of Cupping Coffee, also by Lingle, published in 1986 by the Coffee Development Group, which simply presented a chart with six sensory categories — fragrance, aroma, nose, aftertaste, taste and body, as well as taints and faults — along with predefined attributes for each, all ranked using an intensity scale.

Other cupping forms have been designed to identify profile attributes using only a low-, medium- or high- intensity designation, without calculating a quantitative score. In 2004, Roast published its own cupping evaluation form, which expanded on the SCAA version, and a cupping form that followed the SCAA’s format but added descriptive adjectives for each category. Cup of Excellence developed its own cupping score sheet with slightly different categories, including aroma, clean cup, sweet, acidity, mouthfeel, flavor, aftertaste, balance and overall.

The ultimate goal of any scoring sheet is to capture the intrinsic quality of the coffee accurately and efficiently. The cupper should concentrate on sensory perceptions, not the details of the data collection form.

The advent of the Coffee Taster’s Flavor Wheel in 1995 and the revision of the SCAA cupping form in 2003 changed the focus of sensory scoring from intensity to quality, and incorporated standards to be recognized for flavor identification. The Coffee Taster’s Flavor Wheel was created by Lingle as a visual representation of the information published in The Coffee Cupper’s Handbook. The revision of the SCAA cupping form was driven by the International Coffee Organization’s desire to promote specialty coffee in the 1990s.

Coffee professionals Susie Spindler, George Howell, Don Holly and Lingle worked to develop a comprehensive cupping form that would be relevant to specialty coffee and resilient enough to continue to be useful over time. The Cup of Excellence cupping form was a result of these early collaborations. The Coffee Quality Institute (CQI) continued to revise and update the tool, and following the approval of the SCAA Technical Standards Committee in 2003, it became the “official” cupping form for Q graders and SCAA training programs.

In 2004, CQI was ready to begin field testing training and examinations for the Q program. The Roasters Guild Executive Council was the first group to take the exams, in February of that year. The Q training program combines calibration to aroma standards and alignment with other professional cuppers regarding scoring and flavor identification. This program was groundbreaking and innovative — a global training effort to align the farm-to-cup supply chain and all commercial stakeholders with a common language and quality expectation for specialty coffee. The Q program continues today and is well respected for aligning cuppers to an established language.

In 2010, CQI and SCAA initiated a review regarding the evaluation and scoring of sweetness in coffee. Much serious consideration and many discussions about this have taken place over the past few years; however, changes for sweetness have yet to be approved. The last revision of the SCAA cupping form occurred in 2011, when the SCAA Technical Standards Committee confirmed that continuing to use the 10 existing sections to evaluate coffee quality was the appropriate approach for cupping. Also confirmed was the scoring for defects, clean cup and uniformity, specifically how defects would negatively affect the final score. An important change to the SCAA cupping form in 2011 was the addition of vertical scales to identify intensity of acidity and body to help cuppers distinguish the relationship between intensity and quality.

The new Coffee Taster’s Flavor Wheel — developed by researchers at the University of California, Davis, and industry professionals from SCA using the WCR coffee lexicon — is based on academic research and sensory science techniques. Given this new tool, the common language for identifying the sensory attributes of coffee has changed, and professional cuppers are working to calibrate to the new standards.

“This is an important first step,” says Caspersen, “and as a living document this will need to change over time to become more global, which in turn will help to further calibrate cuppers across the supply chain.”

Preparation

Reverting back to my culinary training, the most efficient way to describe the importance of preparation is, “If you cook it differently, it will taste differently.” In this instance, cooking refers to both roasting and preparing the samples to be cupped.

Cupping is a scientific analysis. As such, great care must be taken to control the processes that develop flavor so as not to introduce variables that will affect the outcome of the analysis. The results of cupping have enormous financial impact on a coffee’s supply chain. Managing the process to ensure consistency in roasting, grinding and dosing is critical to the accuracy and credibility of the test results. Further preparation control includes water quality, temperature and volume.

Many companies self-determine their standard operating procedures for coffee cupping based on their individual needs or “quality personality.” When sellers and buyers are in agreement regarding how a coffee is to be evaluated, and both align their processes for uniformity, the results will correlate. Conversely, when buyers and sellers prepare and evaluate their coffees using different processes and protocols, the results cannot be correlated and conflict may arise in determining quality.

Emerging coffee companies and small-scale operations throughout the farm-to-cup supply chain may not have the resources to operate a formal quality control laboratory, as the costs for equipment and staffing may be prohibitive. In these instances, the company must recognize the potential variables related to cupping and consider the variables introduced by its own operation in determining coffee quality. These companies should coordinate with suppliers and customers to align results and mitigate any inaccurate quality determinations.

Emotional Bias

Not part of the cupping score sheet, emotional bias nevertheless has a strong influence on the quality perception of coffee. Familiarity with suppliers or having an affinity for a region or producer can influence a cupper’s perceptions. Experiential knowledge gained during origin tours, a compelling story regarding the coffee’s provenance, or simply the excitement of touring a coffee-producing country can influence a sensory determination. Emotional bias will influence purchasing decisions; however, because it is based on an individual’s experiences, it cannot be aligned or calibrated and, as such, is not included in scoring on cupping forms.

“I won’t make any final purchasing decisions until I’ve cupped the lot or lots back at the home lab with my team of qualified cuppers, in order to rule out the unintentional bias that can occur when you are cupping with producers on sourcing trips,” says Jennifer Roberts, director of trading at Atlas Coffee Importers in Seattle.

Trindle Mersch recommends double-blind cupping whenever practical to remove the potential for emotional bias.

Maintaining skills and expertise is a challenge. Over time, coffee cuppers may develop sensory errors or biases, or allow preference to influence their quality perceptions. A top priority for coffee cuppers is to remain in alignment with other professional cuppers, within a sensory panel, or even outside one’s company.

“Cupping with a panel helps us to stay both aligned and calibrated,” says Roberts. “Regular cupping with our clients and producer partners is also a great way to align expectations across the supply chain.”

Errors of magnitude are common when cuppers do not regularly align for numeric scoring; non-perceiver errors occur when a cupper does not perceive an attribute and scores the coffee differently than other panelists; and incorrect identification errors result when a cupper does not regularly calibrate to flavor standards and fails to identify a flavor accurately. To maintain skills and expertise, coffee cupping should be a regular activity, and should include both alignment with other cuppers and calibration to flavor standards.

Coffee cuppers often have great job satisfaction — but we must never forget the significant responsibility inherent in the task.

“While we often have a great time in the cupping room, we always remember we have a serious job to do,” says Foster. “Professional cupping also comes with great pressure, as it is often the very last stop in the supply chain.”

Mastering the complexity of cupping requires regular training to maintain advanced skills, while trying not to be too obvious about how enjoyable the job really is.

As Hescock says, “The heart and soul of the coffee industry is in the cup!”