Skip to content

measures

measures #

Convenience methods for calculating a number of similarity error measures between one or more reference and hypothesis sentences. These measures are commonly used to measure the performance of an automatic speech recognition (ASR) system.

The following measures are implemented:

  • Word Error Rate (WER), which is where this library got its name from. This has long been (and arguably still is) the de facto standard for computing ASR performance.
  • Match Error Rate (MER)
  • Word Information Lost (WIL)
  • Word Information Preserved (WIP)
  • Character Error Rate (CER)

Note that these functions merely call jiwer.process_words and jiwer.process_characters. It is more efficient to call process_words or process_characters and access the results from the jiwer.WordOutput and jiwer.CharacterOutput classes.

cer #

cer(
    reference=None,
    hypothesis=None,
    reference_transform=cer_default,
    hypothesis_transform=cer_default,
)

Calculate the character error rate (CER) between one or more reference and hypothesis sentences.

Parameters:

Name Type Description Default
reference Union[str, List[str]]

The reference sentence(s)

 None
hypothesis Union[str, List[str]]

The hypothesis sentence(s)

 None
reference_transform Union[Compose, AbstractTransform]

The transformation(s) to apply to the reference string(s)

 cer_default
hypothesis_transform Union[Compose, AbstractTransform]

The transformation(s) to apply to the hypothesis string(s)

 cer_default

Returns:

Type Description
float

The character error rate of the given reference and hypothesis sentence(s).
Source code in src/jiwer/measures.py 
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
def cer(
    reference: Union[str, List[str]] = None,
    hypothesis: Union[str, List[str]] = None,
    reference_transform: Union[tr.Compose, tr.AbstractTransform] = cer_default,
    hypothesis_transform: Union[tr.Compose, tr.AbstractTransform] = cer_default,
) -> float:
    """
    Calculate the character error rate (CER) between one or more reference and
    hypothesis sentences.

    Args:
        reference: The reference sentence(s)
        hypothesis: The hypothesis sentence(s)
        reference_transform: The transformation(s) to apply to the reference string(s)
        hypothesis_transform: The transformation(s) to apply to the hypothesis string(s)

    Returns:
        (float): The character error rate of the given reference and hypothesis
                 sentence(s).
    """
    output = process_characters(
        reference, hypothesis, reference_transform, hypothesis_transform
    )

    return output.cer

mer #

mer(
    reference=None,
    hypothesis=None,
    reference_transform=wer_default,
    hypothesis_transform=wer_default,
)

Calculate the match error rate (MER) between one or more reference and hypothesis sentences.

Parameters:

Name Type Description Default
reference Union[str, List[str]]

The reference sentence(s)

 None
hypothesis Union[str, List[str]]

The hypothesis sentence(s)

 None
reference_transform Union[Compose, AbstractTransform]

The transformation(s) to apply to the reference string(s)

 wer_default
hypothesis_transform Union[Compose, AbstractTransform]

The transformation(s) to apply to the hypothesis string(s)

 wer_default

Returns:

Type Description
float

The match error rate of the given reference and hypothesis sentence(s).
Source code in src/jiwer/measures.py 
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
def mer(
    reference: Union[str, List[str]] = None,
    hypothesis: Union[str, List[str]] = None,
    reference_transform: Union[tr.Compose, tr.AbstractTransform] = wer_default,
    hypothesis_transform: Union[tr.Compose, tr.AbstractTransform] = wer_default,
) -> float:
    """
    Calculate the match error rate (MER) between one or more reference and
    hypothesis sentences.

    Args:
        reference: The reference sentence(s)
        hypothesis: The hypothesis sentence(s)
        reference_transform: The transformation(s) to apply to the reference string(s)
        hypothesis_transform: The transformation(s) to apply to the hypothesis string(s)

    Returns:
        (float): The match error rate of the given reference and
                 hypothesis sentence(s).
    """
    output = process_words(
        reference, hypothesis, reference_transform, hypothesis_transform
    )

    return output.mer

wer #

wer(
    reference=None,
    hypothesis=None,
    reference_transform=wer_default,
    hypothesis_transform=wer_default,
)

Calculate the word error rate (WER) between one or more reference and hypothesis sentences.

Parameters:

Name Type Description Default
reference Union[str, List[str]]

The reference sentence(s)

 None
hypothesis Union[str, List[str]]

The hypothesis sentence(s)

 None
reference_transform Union[Compose, AbstractTransform]

The transformation(s) to apply to the reference string(s)

 wer_default
hypothesis_transform Union[Compose, AbstractTransform]

The transformation(s) to apply to the hypothesis string(s)

 wer_default

Returns:

Type Description
float

The word error rate of the given reference and hypothesis sentence(s).
Source code in src/jiwer/measures.py 
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
def wer(
    reference: Union[str, List[str]] = None,
    hypothesis: Union[str, List[str]] = None,
    reference_transform: Union[tr.Compose, tr.AbstractTransform] = wer_default,
    hypothesis_transform: Union[tr.Compose, tr.AbstractTransform] = wer_default,
) -> float:
    """
    Calculate the word error rate (WER) between one or more reference and
    hypothesis sentences.

    Args:
        reference: The reference sentence(s)
        hypothesis: The hypothesis sentence(s)
        reference_transform: The transformation(s) to apply to the reference string(s)
        hypothesis_transform: The transformation(s) to apply to the hypothesis string(s)

    Returns:
        (float): The word error rate of the given reference and
                 hypothesis sentence(s).
    """
    output = process_words(
        reference, hypothesis, reference_transform, hypothesis_transform
    )

    return output.wer

wil #

wil(
    reference=None,
    hypothesis=None,
    reference_transform=wer_default,
    hypothesis_transform=wer_default,
)

Calculate the word information lost (WIL) between one or more reference and hypothesis sentences.

Parameters:

Name Type Description Default
reference Union[str, List[str]]

The reference sentence(s)

 None
hypothesis Union[str, List[str]]

The hypothesis sentence(s)

 None
reference_transform Union[Compose, AbstractTransform]

The transformation(s) to apply to the reference string(s)

 wer_default
hypothesis_transform Union[Compose, AbstractTransform]

The transformation(s) to apply to the hypothesis string(s)

 wer_default

Returns:

Type Description
float

The word information lost of the given reference and hypothesis sentence(s).
Source code in src/jiwer/measures.py 
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
def wil(
    reference: Union[str, List[str]] = None,
    hypothesis: Union[str, List[str]] = None,
    reference_transform: Union[tr.Compose, tr.AbstractTransform] = wer_default,
    hypothesis_transform: Union[tr.Compose, tr.AbstractTransform] = wer_default,
) -> float:
    """
    Calculate the word information lost (WIL) between one or more reference and
    hypothesis sentences.

    Args:
        reference: The reference sentence(s)
        hypothesis: The hypothesis sentence(s)
        reference_transform: The transformation(s) to apply to the reference string(s)
        hypothesis_transform: The transformation(s) to apply to the hypothesis string(s)

    Returns:
        (float): The word information lost of the given reference and
                 hypothesis sentence(s).
    """
    output = process_words(
        reference, hypothesis, reference_transform, hypothesis_transform
    )

    return output.wil

wip #

wip(
    reference=None,
    hypothesis=None,
    reference_transform=wer_default,
    hypothesis_transform=wer_default,
)

Calculate the word information preserved (WIP) between one or more reference and hypothesis sentences.

Parameters:

Name Type Description Default
reference Union[str, List[str]]

The reference sentence(s)

 None
hypothesis Union[str, List[str]]

The hypothesis sentence(s)

 None
reference_transform Union[Compose, AbstractTransform]

The transformation(s) to apply to the reference string(s)

 wer_default
hypothesis_transform Union[Compose, AbstractTransform]

The transformation(s) to apply to the hypothesis string(s)

 wer_default

Returns:

Type Description
float

The word information preserved of the given reference and hypothesis sentence(s).
Source code in src/jiwer/measures.py 
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
def wip(
    reference: Union[str, List[str]] = None,
    hypothesis: Union[str, List[str]] = None,
    reference_transform: Union[tr.Compose, tr.AbstractTransform] = wer_default,
    hypothesis_transform: Union[tr.Compose, tr.AbstractTransform] = wer_default,
) -> float:
    """
    Calculate the word information preserved (WIP) between one or more reference and
    hypothesis sentences.

    Args:
        reference: The reference sentence(s)
        hypothesis: The hypothesis sentence(s)
        reference_transform: The transformation(s) to apply to the reference string(s)
        hypothesis_transform: The transformation(s) to apply to the hypothesis string(s)

    Returns:
        (float): The word information preserved of the given reference and
                 hypothesis sentence(s).
    """
    output = process_words(
        reference, hypothesis, reference_transform, hypothesis_transform
    )

    return output.wip