Seizures are the defining signature of epilepsy. Because they manifest as abnormal electrical activity in the brain, they can be detected using electroencephalography (EEG) – in theory. In practice, seizure detection is notoriously difficult. Patterns vary widely, are often subtle, and are frequently obscured by noise. Distinguishing true epileptiform activity from normal EEG variants requires years of training.

Enter machine learning. Seizure detection has become a prime target for automation, and the motivation is obvious: trained EEG specialists are scarce, clinical workloads are growing, and rapid interpretation is often critical. The result is an intense race among EEG manufacturers and software developers to build automated detection systems.

With each new system, the numbers look better – sensitivities climb, false alarm rates drop, algorithms appear increasingly reliable. But look closer, and problems become evident: most of these claims are impossible to compare. Studies use different datasets, different definitions of what counts as a detected seizure, and different evaluation procedures. Without accepted benchmarks or reporting standards, performance metrics can easily create an illusion of progress.

This blog series examines the methodological loopholes behind that illusion. We’re not focused on how seizure-detection algorithms work – but on how their performance is measured, reported, and sometimes manipulated. Our goal is to help physicians, researchers, and decision-makers cut through the fog surrounding automated seizure detection claims.

A series of 4 blogs will cover a wide range of topics:

• How to Define Seizure Detection
• Basic Concepts of Seizure Detection
• The Critical Role of Training Data in the Development of Seizure Detection Algorithms 
• Quantifying Seizure Detection

Blog 1 of 4

How to Define Seizure Detection 

Signal Detection Theory: The Foundation

Medical diagnostic tools are typically evaluated using signal detection theory, which assumes an “ideal observer” can optimally separate signal from noise. In seizure detection, the signal represents the predominant ‘abnormal’ EEG pattern during true electrographic seizures, while noise encompasses everything else—including ‘normal’ background EEG and other types of abnormal non-epileptoform activity.

According to this theory, signals can be separated from noise as long as they don’t completely overlap. For example, EEG amplitude generally increases during seizures. By analyzing the distribution of amplitude values during seizure versus non-seizure periods, we can identify patterns that distinguish these two states.

However, reality is more complex. Motion artifacts can produce high-amplitude signals that mimic seizures (see Figure 1), leading to false positive detections. This is why sophisticated algorithms are necessary to achieve high specificity in real-world applications.

The Contingency Table

When scanning an EEG and making binary seizure/non-seizure decisions, four outcomes are possible:

• True Positives (TP): Correctly identifying seizures
• True Negatives (TN): Correctly identifying non-seizure segments
• False Positives (FP): Incorrectly flagging non-seizure activity as seizures
• False Negatives (FN): Missing actual seizures

These four categories form the basis for calculating performance metrics. This table is also often called a ‘confusion matrix’ and it provides an indispensable visual snapshot of classifier performance, revealing not just whether an algorithm works, but precisely how and where it succeeds or fails [4]. 

Impact of false assessments in Seizure Detection

An ideal seizure detection algorithm would produce a matrix with strong diagonal values (correct classifications) and zeros in all off-diagonal cells (no errors). In reality, the goal is to maximize correct detections while minimizing mistakes – but not all mistakes carry equal weight. False positives and false negatives have different clinical consequences.

False Negatives. Missing several individual seizure events within an EEG study might prompt a medication adjustment but typically won’t alter a patient’s epilepsy diagnosis. Though missing life-threatening non-convulsive status epilepticus in an ICU patient altogether can be catastrophic. 

False Positives. Conversely, false detections can lead to misdiagnosis of epilepsy in healthy individuals, affecting their quality of life, employment, and driving privileges. In operational settings, excessive false alarms create another problem: alarm fatigue. When clinicians are inundated with false alerts, they may become desensitized and slower to respond to real emergencies, which is why precision matters as much as sensitivity.

What is “Detection”

One critical yet often overlooked aspect is the definition of “detection” itself. To validate the accuracy of an algorithm for seizure detection, human expert physicians score a given set of EEGs and their readings are then compared to the automated results.   

The simplest approach uses temporal proximity-based detection: counting a ‘detection’ when the algorithm-identified event falls within any time window (Δt) of the expert-annotated onset (Figure 2a). There does not have to be a complete overlap to receive a 100% score. A more sophisticated class of methods uses the overlap between the machine-detected and expert-annotated seizure intervals. In this class, most studies use the binary “Overlap” (or short “OVLP”) method (Figure 2b), which considers any overlap between the detected and expert-annotated seizure interval as a hit (TP) [3]. However, the most accurate quantification of match is the percentage overlap or Time-aligned Event Scoring (TAES) method between expert-annotated and algorithm-detected seizure intervals (Figure 2c). These three different approaches can yield dramatically different results [6]. 

A proximity approach might count any partial detection as a correct “hit” while an overlap approach would reveal a more differentiated picture. For example, a 30-second temporal overlap between a human expert annotation and the algorithm score on a 60 seconds seizure would represent only a 50% overlap. As a result, a study reporting 95% sensitivity using proximity or OVLP detection might show only 70% average overlap with the percentage overlap approach. In all three cases, the algorithm performed equally ‘well’, but these results are reported with a different intent. Using the proximity approach helps to communicate how well the algorithm finds a similar number of seizure events. The overlap approach, however, also is able to make a statement about how similar the algorithm is to the expert human read in its temporal accuracy. Neither approach is wrong, but not considering these detection details can lead to widely different assessments of an algorithm’s efficiency. 

It is entirely possible that a proximity-based detection overinflates the perceived performance of an algorithm by the mere circumstance that only the edges of the algorithm read aligns with the expert human assessment. Assessing the same algorithm outcome using the percent overlap approach would yield lower scores. 

We will address the question of proximity-based vs. interval matching under the chapter “Temporal Considerations: Seizure Onset and Offset.”

While most seizure detection performance reports use proximity or OVLP methods, Zeto’s automated algorithm applies the more conservative percent overlap method for precisely quantifying seizure detection. It requires not only determining the seizure onset but also seizure offset times, and extra information is not always available from annotations of open-access EEG seizure databases.    

Limitations

Throughout this series, we won’t cover the algorithms behind seizure detection methods themselves -several recent reviews do that [1, 7, 9, 12].

We also won’t attempt a systematic comparison of detection systems. Commercial algorithms are proprietary and often patent-protected, making them impossible to evaluate directly. In contrast, academic algorithms are often open-source and more comparable – but with half the landscape hidden behind trade secrecy, any head-to-head comparison would be fundamentally incomplete.

That gap is itself part of the problem. Instead of comparing algorithms, we focus on something more tractable: the loopholes and biases in reporting practices that shape how all of these seizure detection methods are perceived and judged.

References:

1. Bai, L., Litscher, G., & Li, X. (2025). Epileptic Seizure Detection Using Machine Learning: A Systematic Review and Meta-Analysis. Brain sciences, 15(6), 634. https://doi.org/10.3390/brainsci15060634
2. Baghdadi, A., Fourati, R., Aribi, Y. et al. A channel-wise attention-based representation learning method for epileptic seizure detection and type classification. J Ambient Intell Human Comput 14, 9403–9418 (2023). https://doi.org/10.1007/s12652-023-04609-6
3. Gotman, J., Flanagan, D., Zhang, J., & Rosenblatt, B. (1997). Automatic seizure detection in the newborn: methods and initial evaluation. Electroencephalography and clinical neurophysiology, 103(3), 356–362. https://doi.org/10.1016/s0013-4694(97)00003-9
4. Khurshid, D., Wahid, F., Ali, S., Gumaei, A. H., Alzanin, S. M., & Mosleh, M. A. A. (2024). A deep neural network-based approach for seizure activity recognition of epilepsy sufferers. Frontiers in medicine, 11, 1405848. https://doi.org/10.3389/fmed.2024.1405848
5. Kunekar, P., Gupta, M.K. & Gaur, P. Detection of epileptic seizure in EEG signals using machine learning and deep learning techniques. J. Eng. Appl. Sci. 71, 21 (2024). https://doi.org/10.1186/s44147-023-00353-y
6. Lee, K., Jeong, H., Kim, S., Yang, D., Kang, H. C., & Choi, E. (2022). Real-time seizure detection using EEG: a comprehensive comparison of recent approaches under a realistic setting. arXiv preprint arXiv:2201.08780.
7. Li, W., Wang, G., Lei, X., Sheng, D., Yu, T., & Wang, G. (2022). Seizure detection based on wearable devices: A review of device, mechanism, and algorithm. Acta neurologica Scandinavica, 146(6), 723–731. https://doi.org/10.1111/ane.13716
8. Paneru B, (2025) Epileptic Seizure Detection based on Different Events with XAI and Early Aid System for Patient Aid doi: https://doi.org/10.1101/2025.10.03.25337233
9. Slama, K., Yahyaouy, A., Riffi, J., Mahraz, M. A., & Tairi, H. (2025). Comprehensive review of machine learning and deep learning techniques for epileptic seizure detection and prediction based on neuroimaging modalities. Visual computing for industry, biomedicine, and art, 8(1), 27. https://doi.org/10.1186/s42492-025-00208-8
10. Torkey, H., Hashish, S., Souissi, S., Hemdan, E. E. D., & Sayed, A. (2025). Seizure Detection in Medical IoT: Hybrid CNN-LSTM-GRU Model with Data Balancing and XAI Integration. Algorithms, 18(2), 77.
11. Wilson, S. B., Scheuer, M. L., Plummer, C., Young, B., & Pacia, S. (2003). Seizure detection: correlation of human experts. Clinical neurophysiology : official journal of the International Federation of Clinical Neurophysiology, 114(11), 2156–2164. https://doi.org/10.1016/s1388-2457(03)00212-8
12. Zhang X, Zhang X, Huang Q and Chen F (2024) A review of epilepsy detection and prediction methods based on EEG signal processing and deep learning. Front. Neurosci. 18:1468967. doi: 10.3389/fnins.2024.1468967
    

Blog 2 of 4

In the first blog of our Understanding Seizure Detection series, we summarized different approaches on how to define “seizure detection”. In this blog we introduce different ways to help understand and visualize the performance of a seizure detection algorithm. The Receiver Operation Characteristics (ROC), and F1-Score are two popular methods used for that purpose. 

The ROC Curve

The Receiver Operating Characteristic (ROC) curve visualizes the trade-off between detecting true seizures and generating false alarms across different detection thresholds. As the separability between signal and noise increases, the area under the ROC curve (AUC) approaches 1. When signal and noise completely overlap, the AUC approaches 0.5. The AUC value, also called choice probability, provides a sensitive measure of how well an EEG feature can differentiate seizure from non-seizure activity (Figure 1).

In other words, a high area under the curve signifies that a seizure algorithm is accurately detecting a seizure. An AUC of 1 equals a perfect 100% accurate categorization. An AUC of 0.5, however, would signify that the algorithm would only be detecting a seizure by chance. AUC values under 0.5 would be even worse, signifying that an algorithm identifies the target signal even below chance. 

To find out how we derive useful performance metrics from contingency tables and confusion matrices, see our next chapter on Understanding Performance Metrics.

Understanding Performance Metrics

Sensitivity and Specificity

Diagnostic accuracy is typically expressed through sensitivity, specificity, False Positive Rate, and Precision:

• Sensitivity (Recall or Positive Percent Agreement): The proportion of actual seizures correctly identified (TP / [TP + FN])
• Specificity (Negative Percent Agreement): The proportion of non-seizure activity correctly identified (TN / [TN + FP])
• False Positive Rate: (1-Specificity)
• Precision: The accuracy of positive predictions (TP / [TP + FP])

The problem of Ground Truth: Inter-Rater Agreement

Performance metrics assume we have an objective “ground truth” to compare against. In seizure detection, this ground truth comes from expert EEG readers with years of experience. However, this foundation becomes shaky when we examine inter-rater agreement.

Research shows that even expert physicians can disagree substantially. In our own study, three independent experts reviewing the same EEG data agreed on only 54% of seizures (Figure 3). This high variability in  inter-rater assessments  raises serious questions about the reliability of sensitivity, specificity, and F1 scores. 

It is crucial to maintain the statistical representativeness of the physicians reviewing the test data. First of all, the rating physicians must be blinded to each others’ seizure annotations, otherwise one physician’s seizure annotation may influence others’. It is also recommended to elect independent readers of geographically and professionally distant areas, because physicians trained in the same school, working as residents under shared supervision, or have been co-workers may bias the inter-rater agreement, and decrease the representativeness of expert opinion. A study aiming for objectivity should mitigate these types of biases. 

Addressing the Uncertainty

Several approaches can help manage this uncertainty:

• Conservative approach: Use only seizures identified by all experts (intersection). This yields high confidence ratings but excludes many valid seizure episodes if they were missed by just one expert. It provides the smallest number of validated seizures.
• Moderate approach: Require agreement from more than one but not all experts. This balances confidence with data availability. 
• Liberal approach: Include all seizures identified by any expert. This maximizes data but reduces consensus. It provides the largest number of validated seizures.

Ideally, consensus scoring across many raters provides a more robust ground truth. However, this variability means that no published F1 score has absolute validity. These metrics change when different expert readers are involved, making claims of 100% recall or F1 scores questionable and sample-dependent. 

Finally, inter-rater variability is highly sensitive to the method we determine the overlap between expert readers’ definitions of seizures. If we apply the “percent overlap” (discussed in Blog 1: “How to Define Seizure Detection”) between seizure events defined by their onset and offset times, the disagreement can be significant even if the experts agree on the onset but disagree on the offset time. Therefore, reported inter-rater agreements are usually high (>90%) because most investigators consider an agreement perfect when the seizure onset asynchrony does not exceed a predefined ΔT interval (usually a few seconds) and ignore the offset asynchrony. This topic will be further elaborated next.

Temporal Considerations: Seizure Onset and Offset

Seizures have relatively well-defined onset times but much less clear offset times. Seizures often taper gradually, making the endpoint a matter of clinical judgment rather than objective measurement. Some clinicians consider the seizure to be ended when the EEG returns to near-normal, while others use the transition from periodic to aperiodic patterns as the endpoint.

This ambiguity particularly affects the diagnosis of electrographic status epilepticus (ESE), which is defined by total seizure duration thresholds (10% continuous or 12% fragmented seizures over an hour). Since offset time uncertainty directly impacts duration calculations, it can influence critical clinical decisions.

The Electrode Coverage Dilemma

An often-underestimated factor in seizure detection is electrode montage and coverage. Clinical EEG systems range from partial limited montage (headbands, single electrodes) to full montage coverage (headcaps or individually attached electrodes with standardized electrode positions). Most comply with the 10-20 system, which ensures reproducible electrode placement.

Reduced montage electrode coverage introduces an inherent bias by failing to capture all possible seizures, particularly focal seizures confined to brain regions outside the electrode array. As a result, reduced montages cannot establish a complete ground truth for seizure occurrence. Nevertheless, some studies define ground truth using reduced montages rather than full electrode coverage. As a result, it becomes easier to report perfect sensitivity and high specificity when the ground truth is limited to partial montages. This is particularly likely when algorithm performance is evaluated against expert consensus derived from the same reduced electrode set, which may miss seizures occurring outside the monitored regions. In other words, if the ground truth includes only seizures visible within the partial montage—and excludes seizures occurring beyond it—then failing to detect those excluded seizures will not negatively affect performance scores. Readers should therefore be cautious about overvaluing high sensitivity or specificity reported for algorithms based on limited montages.

In contrast, several studies recommend to define the ground truth using expert annotations based on full-montage EEG recordings [1, 3, 5, 7, 9]. When evaluated against this more comprehensive reference, partial montage (reduced-electrode-set) approaches yielded more moderate performance, with reported sensitivities around 75% and specificities near 97%. Notably, none of these studies claimed perfect sensitivity. But that is also a more realistic and statistically plausible outcome, because this more prudent assessment strategy is not subject to the inherent bias of partial montage approaches.

Given the inherent bias of using reduced montage EEG as ground truth, it is not surprising that some algorithms based on partial montages report 100 % sensitivity. Such results beg the question how they can be even better than the concordance between blinded human readers evaluating either reduced montage or full coverage EEG records – it can’t and this simply illustrates the inherent bias of such partial montage approaches. To bring some clarity to the source of contradicting results that approaches like that may produce is a recent multi-center study (AccuRASE) evaluating seizure detection performance of partial montage based seizure detection methods and found only 29% sensitivity (low-to-moderate), but relatively high, >90% specificity [8]. This study used full montage ground truth as opposed to a partial montage.  

The above example illustrates how important it is to maximize the objectivity of ground truth dataset. The misrepresentation of ground truth may also create the false impression that reduced montages perform as well or even better than full montages. Please note, that  specificity (false positive ratio) is relatively immune to the reduced montage bias, because it is only affected by the true negatives and false positives; neither of which is related to missed seizures.  

In other words, by reducing the available electrodes to assess the ground truth, sensitivity assessments based on partial montages risk over-estismate true sensitivity values. Approaches based on full montage ground-truth data have a higher likelihood to score lower sensitivity values simply because of the inclusion of additional ground truth channels in the central head regions. Full montage assessments may well be much more sensitive as a result but that won’t be reflected in their sensitivity values compared to values achieved by partial montage approaches. The simple take-away is that reducing ground-truth coverage introduces a bias towards higher sensitivity scores.

Achieving unbiased performance metrics is paramount in seizure detection. However, performance evaluation is only half the story—the training dataset plays an equally critical role. To understand how training data fundamentally shapes seizure detection performance, see Blog 3: “The Critical Role of Training Data”.

References

1. Asif, R., Saleem, S., Hassan, S. A., Alharbi, S. A., & Kamboh, A. M. (2020). Epileptic seizure detection with a reduced montage: A way forward for ambulatory EEG devices. IEEE Access, 8, 65880-65890. https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9050500
2. Backman, S., Cronberg, T., Rosén, I., & Westhall, E. (2020). Reduced EEG montage has a high accuracy in the post cardiac arrest setting. Clinical Neurophysiology, 131(9), 2216–2223. https://doi.org/10.1016/j.clinph.2020.06.021
3. Frankel, M. A., Lehmkuhle, M. J., Spitz, M. C., Newman, B. J., Richards, S. V., & Arain, A. M. (2021). Wearable Reduced-Channel EEG System for Remote Seizure Monitoring. Frontiers in neurology, 12, 728484. https://doi.org/10.3389/fneur.2021.728484
4. Grant, A. C., Abdel-Baki, S. G., Weedon, J., Arnedo, V., Chari, G., Koziorynska, E., Lushbough, C., Maus, D., McSween, T., Mortati, K. A., Reznikov, A., & Omurtag, A. (2014). EEG interpretation reliability and interpreter confidence: a large single-center study. Epilepsy & behavior : E&B, 32, 102–107. https://doi.org/10.1016/j.yebeh.2014.01.011
5. Lin, Y. C., Lin, H. A., Chang, M. L., & Lin, S. F. (2025). Diagnostic accuracy of reduced electroencephalography montages for seizure detection: A frequentist and Bayesian meta-analysis. Neurophysiologie clinique = Clinical neurophysiology, 55(2), 103044. https://doi.org/10.1016/j.neucli.2025.103044
6. Little, S. C., & Raffel, S. C. (1962). Intra-rater reliability of EEG interpretations. The Journal of nervous and mental disease, 135, 77–81. https://doi.org/10.1097/00005053-196207000-00010
7. Ma, B. B., Johnson, E. L., & Ritzl, E. K. (2018). Sensitivity of a Reduced EEG Montage for Seizure Detection in the Neurocritical Care Setting. Journal of clinical neurophysiology : official publication of the American Electroencephalographic Society, 35(3), 256–262. https://doi.org/10.1097/WNP.0000000000000463
8. Sheikh, Z. B., Dhakar, M. B., Fong, M. W. K., Fang, W., Ayub, N., Molino, J., Haider, H. A., Foreman, B., Gilmore, E., Mizrahi, M., Karakis, I., Schmitt, S. E., Osman, G., Yoo, J. Y., & Hirsch, L. J. (2025). Accuracy of a Rapid-Response EEG’s Automated Seizure-Burden Estimator: AccuRASE Study. Neurology, 104(2), e210234. https://doi.org/10.1212/WNL.0000000000210234
9. Westover, M. B., Gururangan, K., Markert, M. S., Blond, B. N., Lai, S., Benard, S., Bickel, S., Hirsch, L. J., & Parvizi, J. (2020). Diagnostic value of electroencephalography with ten electrodes in critically ill patients. Neurocritical Care, 33(2), 479–490. https://doi.org/10.1007/s12028-019-00911-4

Blog 3 of 4

Data Source and Representativeness

In the first two blogs, we overviewed the basic terminology and quantitative evaluation of seizure detection (1 – How to Define Seizure Detection, 2 – Basic Concepts of Seizure Detection). Here we turn our attention to the importance of datasets.

The training dataset fundamentally determines where an algorithm will work effectively. An algorithm trained exclusively on ICU data – where patients are often sedated with minimal motion artifacts – will generate excessive false alarms in routine EEG settings where patients move freely. Conversely, algorithms trained only on awake patients will generate a lot of false pathological slow oscillations in critical care settings, where those slow oscillations are natural delta waves of the sedated brain. Similarly, none of these seizure detection algorithms will work for patients in transport (airlifted or ambulance cars) unless the EEG is denoised for the mechanical and electromagnetic artifacts before processing.

Public Datasets

Several public databases support algorithm development, but each of them has advantages and disadvantages depending on the seizure detection objectives.

Bonn EEG Database

Description: Contains 600 expert-annotated single-channel seizure segments [1].

Limitations: It is a very limited dataset (1 channel each) with minimal artifacts, hence limiting generalizability. In addition, no seizure offset annotations. 

Temple University Hospital (TUH) Database

Description: Includes the TUH EEG Epilepsy Corpus (TUEP) and TUH EEG Seizure Corpus (TUSZ) with thousands of annotated recordings from various settings (routine, EMU, ICU), including both seizure and non-seizure data. The different TUH datasets overlap.

Limitations: It is an overall good resource, but it contains a broad quality range of EEGs recorded in diverse settings, from routine to ICU data. It is good for developing robust, quality-insensitive seizure detection algorithms, but has a limited use for parameter fine-tuning.  

Harvard Electroencephalography Database

Description: Contains 284,343 EEG studies from 109,178 patients across four sites, including annotated ICU, EMU, and routine EEG datasets [2]. It is a huge database (210 TB), almost ideal for foundational model development.

Limitations: However, the seizure annotations made by human experts are very limited and lack seizure offset times. Moreover, it contains long ICU recordings (> 8 hours) with very few seizures, which takes us to the next problem.

Seizure Prevalence Effects

Seizure prevalence dramatically affects performance metrics. Consider a test database consisting of 100 samples of 20-second EEG segments from patients undergoing 33-minute routine EEG recordings with rare seizure occurrences: average 1 true seizure in 100 EEG samples. If we run the data on a new seizure detection algorithm that detects 10 seizures (including the one true seizure), it would achieve:

• Recall = 100% (found the only true seizure)

• Precision = 10% (9 false positives)

• F1 = 0.18 (severely impacted by false positives)

This demonstrates why 100% sensitivity alone is very misleading—it overlooks the critical importance of false positive rates.

Considering that seizure prevalence varies significantly across clinical settings, we have a good reason to balance the composition of our test and training datasets:

• Routine EEG: Low prevalence (30-minute recordings in outpatients)
• Epilepsy Monitoring Unit (EMU): High prevalence (medications withdrawn to capture seizures)
• ICU: High prevalence (continuous monitoring for status epilepticus)

When evaluating published results, always consider the patient cohort and seizure prevalence, as they fundamentally affect precision and F1 scores.

Data Heterogeneity and Specialization

Age and State Considerations

EEG characteristics vary dramatically across clinical settings, demographics and populations:

• Pediatric EEG differs from adult EEG
• Neonatal EEG is fundamentally different from both
• Sleep EEG requires different approaches than awake EEG

An F1 score reported for adults may be meaningless when applied to neonates. Similarly, algorithms trained on awake patients cannot be directly applied to sleep EEG without adjustment.

The Strategic Choice

Generalist approach: For broad applicability across diverse settings, train on similarly diverse datasets with large numbers of validated examples.

Specialist approach: For specific populations (e.g., neonates), focus training on population-specific datasets for optimal performance.

Sample Size Matters

Best Practices for Transparent Reporting

Performance evaluation of seizure detection methods is not straightforward. Many parameters can significantly impact published results, and incomplete reporting leads to misinterpretation and irreproducibility. Key issues include:

• Inter-rater variability
• Seizure onset-offset uncertainty
• Electrode coverage variations
• Training data representativeness
• Sample size and prevalence effects

Conclusion

No single number can capture the complexity of seizure detection performance. The published F1 scores, sensitivity values, and other metrics that appear definitive are actually contingent on numerous methodological choices—from how ground truth is established to which patient populations are studied.

As the field moves toward larger annotated databases and more diverse training sets, we can develop increasingly robust foundational models. However, transparency in reporting remains essential. By understanding the factors that influence performance metrics, clinicians and researchers can better evaluate new technologies and make informed decisions about their implementation.

The goal isn’t skepticism about all published results, but rather informed interpretation that recognizes both the promise and limitations of current methods. Only through rigorous, transparent reporting can we enable fair comparison between models and drive genuine progress in automated seizure detection.

In the next blog, we address the central challenge of quantifying seizure burden (SB) and explain why reliable seizure detection is the limiting factor for both real-time alerting and longitudinal clinical assessment. Read the next blog: “Quantifying Seizure Detection“.

References

1. Andrzejak, R. G., Lehnertz, K., Mormann, F., Rieke, C., David, P., & Elger, C. E. (2001). Indications of nonlinear deterministic and finite-dimensional structures in time series of brain electrical activity: Dependence on recording region and brain state. Physical Review E, 64(6), 061907
2. Zafar, S., Loddenkemper, T., Lee, J. W., Cole, A., Goldenholz, D., Peters, J., Lam, A., Amorim, E., Chu, C., Cash, S., Moura Junior, V., Gupta, A., Ghanta, M., Fernandes, M., Sun, H., Jing, J., & Westover, M. B. (2025). Harvard Electroencephalography Database (version 4.1). Brain Data Science Platform. https://doi.org/10.60508/k85b-fc87

Blog 4 of 4

Seizure detection algorithms

The first three blogs in this series introduced the quantitative terminology used to evaluate seizure detection and emphasized the critical role of training and testing datasets. Here, we address the central challenge of quantifying seizure burden (SB) and explain why reliable seizure detection is the limiting factor for both real-time alerting and longitudinal clinical assessment .

Read more in our related blog posts:

• How to Define Seizure Detection
• Basic Concepts of Seizure Detection
• The Critical Role of Training Data in the Development of Seizure Detection Algorithms 

Current State of Commercial Automated Seizure Detection

Not all clinical EEG platforms incorporate automated seizure detection capabilities, but the most widely deployed commercial systems feature modular software architectures that support optional seizure recognition modules. These solutions are implemented either as third-party plugins or proprietary extensions developed by the platform manufacturer. Notable examples of third-party integrations include Persyst’s seizure detection module for Natus NeuroWorks™ and Nihon Kohden EEG systems [1], and encevis’ cloud-based detection for Zeto’s NeuroPulse™ AI-powered cloud platform [2]. Ceribell developed Clarity™, a proprietary detection algorithm integrated with their rapid-response EEG system [3,4]. A recent review features the hardware capabilities of these point-of-care EEG systems [5].  

Given that both SB tracking accuracy and electrographic status epilepticus (ESE) alerting reliability are fundamentally determined by seizure detection performance, we focus our analysis on comparing detection algorithms across available solutions. Our evaluation is limited to published performance metrics and, where source code has been made publicly available, direct comparative testing on standardized datasets.

Table 1 (below) presents published performance characteristics of widely used commercial seizure detection systems. It is critical to note that these performance ranges derive from studies conducted on heterogeneous datasets with varying patient populations, recording conditions, and seizure types. (We provided recommendations in Blog 3 under “Best Practices for Transparent Reporting” to improve the consistency of reported characteristics.) Consequently, direct performance comparisons based solely on published metrics are subject to significant dataset bias, as we discussed in Blog 3, “The Critical Role of Training Data“.

Grand Truth Paradox

When evaluating seizure detection algorithms, the most critical methodological decision is how “ground truth” is defined. In medical device diagnostics, ground truth is assumed to represent the objective reality against which algorithm performance is measured. However, in seizure detection, this assumption is often fragile (see Blog 2: “The problem of Ground Truth: inter-rater agreement”). An incorrectly defined or biased ground truth can artificially inflate or suppress sensitivity and specificity. Studies that use different definitions of ground truth therefore become difficult to compare, even if their reported metrics appear similar. Because the contingency table to calculate the key performance metrics is directly affected by the ground truth (see Blog 1), the numbers enter in the table determines the sensitivity and specificity. Hence, by manipulating the ground truth one can easily enhance the performance.

“Ground truth” is not “Gold truth”

In some other diagnostic fields, an external biological standard exists (MRI, CT, etc). For example, a radiological suspicion of a tumor can be confirmed or refuted by biopsy where histopathology serves as an independent validator. In these situations, time and patient outcome of the disease state will help inform about the reality of a preceding assessment. 

Seizure detection lacks such an independent confirmatory modality. The EEG pattern itself is the phenomenon of interest. Consequently, ground truth in seizure detection is anchored only to expert interpretation. This creates an epistemological limitation: the reference standard for what is seizure and what is not is inherently subjective, even when systematically constructed.

Human Expert Consensus as Ground Truth

In electroencephalography (EEG), ground truth is typically established through expert physician annotation. As discussed previously (Blog 2: “The Problem of Ground Truth: Inter-Rater Agreement“), careful expert selection, representativeness, and blinded review are essential to mitigate bias. Multi-rater consensus is commonly used to strengthen reliability. Yet consensus does not guarantee correctness.

Inter-rater agreement varies across types of EEGs (duration, clinical settings, partial or full montages and patients conditions) and across signal conditions. Agreement is typically higher when the signal-to-noise ratio is strong and the seizure is stereotypical. It decreases if the EEG is full of artifacts or contaminated by noise making it ambiguous and hard to read for human experts —precisely the scenarios where automated systems may be superior.

The goal to obtain independent and blinded scores by expert readers in EEG cannot be understated. Readers who belong to the same school of thought through identical training, will necessarily provide more similar ratings, than readers with more diverse backgrounds. Scores that are not blinded may also create an inherent interpersonal conflict between readers if they are financially or professionally dependent on one another. Neurology fellows reading under the supervision of their attending physician in an un-blinded situation may be prone to agree with their mentors scores artificially increasing the inter-rater reliability – human psychology dictates at minimum an unconscious bias towards wanting to please one’s superior.

Finally, at the moment a method requires subjective human ratings, principles of social science apply in full, suggesting that independent inter-rater scores follow a normal distribution with an average and set standard variation [6]. Even if rating scores tend to converge to a commonly high agreement in clear text-book examples of a seizure signal, perfect inter-rater agreement at 100% would be a statistical abnormality when including less clear, less ideal and more edge-case seizure morphologies into the assessment. Perfect inter-rater reliabilities, therefore become a warning sign on their own.  If they are perfectly convergent, then the question begs what detection criteria were deployed (See Blog 1: proximity, overlap, percentage overlap), or what dataset characteristics might have contributed to achieve this level of alignment. Perfect inter-rater reliabilities in EEG, on diverse, multi-facetted datasets, in a truly independent, larger and diverse group of readers, are in-probable.

The Paradox of Algorithmic Superiority

We mentioned above that “… consensus does not guarantee correctness.” This is another source of confusion. A paradox emerges when a machine-learning algorithm demonstrates consistent disagreement with expert annotations—yet appears clinically plausible, internally consistent, and reproducible across datasets.

Only by recognizing and managing the “ground truth paradox” can we generate trustworthy, comparable, and clinically meaningful performance metrics.

The Relationship Between Sensitivity and Ground Truth

One of the clearest warning signs of bias in seizure detection studies is a mismatch between reported sensitivity and inter-rater agreement. The problem arises when vendor-reported sensitivity exceeds the level of agreement among expert reviewers. For example, if two independent EEG experts agree only ~80% of the time on whether a segment contains a seizure (see Blog 2, Figure 3), how can an algorithm claim 99% sensitivity against that same reference?

If Dr. X and Dr. Y agree on seizure presence 80% of the time, then an algorithm achieving 99% sensitivity must be nearly perfectly aligned with one expert while necessarily disagreeing with the other in a substantial fraction of cases. In other words, the 99% sensitivity cannot simultaneously apply to both experts unless one expert’s ratings are excluded from the reference standard.

Constructed Ground Truth and Inflated Metrics

A common methodological solution is to define “ground truth” as only those seizures agreed upon by two or three experts. These consensus events are then treated as 100% true positives (see Blog 2: “conservative,” “moderate,” and “liberal approaches” to consensus).

While this practice increases internal consistency, it also reshapes the denominator of sensitivity. By excluding disputed events, the evaluation may preferentially retain clear, high signal-to-noise seizures and remove those ambiguous cases that lower inter-rater agreement. The resulting contingency table no longer reflects the full clinical reality—it reflects a filtered subset.

Why This Matters

The absence of practical reporting standards in this area has encouraged competitive sensitivity claims, often producing discrepancies between vendor-sponsored and independent studies. These discrepancies are frequently rooted not in statistical error, but in differences in dataset composition and ground truth construction.

Ultimately, real-world clinical performance will arbitrate these claims. Sensitivity that depends heavily on how disagreement cases were handled may not translate into consistent bedside performance.

Sensitivity cannot meaningfully exceed the reliability of its ground truth. When it appears to do so, the issue lies not in the algorithm—but in how the reference standard was constructed and interpreted.

Encouraging Examples of Testing Seizure Detection Methods

A limited number of studies provide examples of rigorously conducted, real-world validation of seizure detection systems using large cohorts. One such example is the 2025 study by Sheikh et al. [7]. This retrospective observational study evaluated the performance of an automated seizure burden estimator (ASBE) integrated into a rapid-response EEG (rr-EEG) system for detecting clinically significant electrographic seizure activity, particularly electrographic status epilepticus (ESE), without requiring immediate expert interpretation. Key methodological features included a multi-center design and independent (blinded) expert review.

Methodological highlights:

• The study reviewed consecutive clinical rr-EEG recordings performed at multiple hospitals between 2019 and 2021.
• Each EEG was independently and blindly reviewed by three human experts.
• A reference standard was defined as at least 2 of 3 reviewers agreeing on whether seizure activity was present.
• The main performance metrics were the negative predictive value (NPV) and positive predictive value (PPV) of the automatic seizure burden estimator for detecting or excluding ESE at various seizure-burden thresholds (e.g., >1%, >10%, >20%, >50%, >90%).
• The only caveat was that the grand truth was quantified based on a partial montage instead of a full montage EEG, which introduced a positive sensitivity bias.

Key Findings: The study concluded that a very high negative predictive value (≈99%) at the cost of sensitivity drop. It means, it rarely misidentified seizure activity when it was absent. Interestingly, the study revealed that  many seizures were missed when they were present, occasionally with sensitivity as low as 29%.

Related Considerations:
These results point to two considerations: High negative predictive values are to be expected in rare clinical events such as seizures. Negative predictive values also increase with the length of EEG studies, if the seizure events are limited in duration and frequency, i.e. such as might be found after successful early administration of AEDs. To primarily focus on negative predictive values to determine the clinical viability of an algorithm may introduce a bias driven by dataset length. The value of negative predictive scores also becomes increasingly inconclusive if AED were administered at any point during the recording.  Finally, the balance of sensitivity and specificity in this device depends heavily on the chosen seizure-burden threshold, underlining the interpretational nuance when deploying such tools in practice.

Clinical Implications: Rapid-response systems with automated estimators may enhance early seizure detection in acute or resource-limited settings, but should not be viewed as replacing expert EEG review. Where expert consensus (ground truth) is low or subtle, algorithm performance must be contextualized against the variability in human interpretation — as explained above. In other words, the argument seems valid that automated estimators are a helpful triaging aid but are not a full replacement for human interpretation at this time.

References:

1. Scheuer, M. L., Wilson, S. B., Antony, A., Ghearing, G., Urban, A., & Bagić, A. I. (2021). Seizure detection: Interreader agreement and detection algorithm assessments using a large dataset. Journal of Clinical Neurophysiology, 38(5), 439–447. https://doi.org/10.1097/WNP.0000000000000709 
2. Zeto, Inc. (n.d.). Internal testing of Encevis AI-Powered seizure detection [Unpublished internal report].
3. Vespa, P. M., Olson, D. M., John, S., Hobbs, K. S., Gururangan, K., Nie, K., Desai, M. J., Markert, M., Parvizi, J., Bleck, T. P., Hirsch, L. J., & Westover, M. B. (2020). Evaluating the clinical impact of rapid response electroencephalography: The DECIDE multicenter prospective observational clinical study. Critical Care Medicine, 48(9), 1249–1257. https://doi.org/10.1097/CCM.0000000000004428 
4. Kamousi, B., Karunakaran, S., Gururangan, K., Markert, M., Decker, B., Khankhanian, P., Mainardi, L., Quinn, J., Woo, R., & Parvizi, J. (2021). Monitoring the burden of seizures and highly epileptiform patterns in critical care with a novel machine learning method. Neurocritical Care, 34(3), 908–917. https://doi.org/10.1007/s12028-020-01120-0 
5. Herman S. T. (2027). Hardware Technology for Point-of-Care EEG: A Comprehensive Review. Journal of clinical neurophysiology: official publication of the American Electroencephalographic Society, 43(3), 191–203. https://doi.org/10.1097/WNP.0000000000001240
6. Grant, A. C., Abdel-Baki, S. G., Weedon, J., Arnedo, V., Chari, G., Koziorynska, E., Lushbough, C., Maus, D., McSween, T., Mortati, K. A., Reznikov, A., & Omurtag, A. (2014). EEG interpretation reliability and interpreter confidence: a large single-center study. Epilepsy & behavior : E&B, 32, 102–107. https://doi.org/10.1016/j.yebeh.2014.01.011
7. Sheikh, Z. B., Dhakar, M. B., Fong, M. W. K., Fang, W., Ayub, N., Molino, J., Haider, H. A., Foreman, B., Gilmore, E., Mizrahi, M., Karakis, I., Schmitt, S. E., Osman, G., Yoo, J. Y., & Hirsch, L. J. (2025). Accuracy of a rapid-response EEG’s automated seizure-burden estimator: AccuRASE study. Neurology, 104(2), e210234. https://doi.org/10.1212/WNL.0000000000210234
8. Haider, H. A., Esteller, R., Hahn, C. D., Westover, M. B., Halford, J. J., Lee, J. W., Shafi, M. M., Gaspard, N., Herman, S. T., Gerard, E. E., Hirsch, L. J., Ehrenberg, J. A., LaRoche, S. M., & Critical Care EEG Monitoring Research Consortium. (2016). Sensitivity of quantitative EEG for seizure identification in the intensive care unit. Neurology, 87(9), 935–944. https://doi.org/10.1212/WNL.0000000000003034 
9. Wilson, S. B., Scheuer, M. L., Emerson, R. G., & Gabor, A. J. (2004). Seizure detection: Evaluation of the Reveal algorithm. Clinical Neurophysiology, 115(10), 2280–2291. https://doi.org/10.1016/j.clinph.2004.05.018 
10. Ganguly, T. M., Ellis, C. A., Tu, D., Shinohara, R. T., Davis, K. A., Litt, B., & Pathmanathan, J. (2022). Seizure detection in continuous inpatient EEG: A comparison of human vs automated review. Neurology, 98(22), e2224–e2232. https://doi.org/10.1212/WNL.0000000000200267 
11. Trinka, E., Cock, H., Hesdorffer, D., Rossetti, A. O., Scheffer, I. E., Shinnar, S., Shorvon, S., & Lowenstein, D. H. (2015). A definition and classification of status epilepticus—Report of the ILAE Task Force on Classification of Status Epilepticus. Epilepsia, 56(10), 1515–1523. https://doi.org/10.1111/epi.13121 
12. Fürbass, F., Ossenblok, P., Hartmann, M., Perko, H., Skupch, A. M., Lindinger, G., Elezi, L., Pataraia, E., Colon, A. J., Baumgartner, C., & Kluge, T. (2015). Prospective multi-center study of an automatic online seizure detection system for epilepsy monitoring units. Clinical Neurophysiology, 126(6), 1124–1131. https://doi.org/10.1016/j.clinph.2014.09.023 

People often assume that all FDA-reviewed devices go through the same process. In reality, the U.S. Food and Drug Administration (FDA) uses multiple regulatory pathways, and the terms like FDA cleared, FDA approved, and FDA granted have specific meanings. 

Understanding this distinction is important for healthcare professionals and patients, particularly in fast-moving fields such as neurology, digital health, and AI-enabled innovation.

Overview: Key Regulatory Pathways

At a high level, FDA clearance applies to most low-to-moderate-risk medical devices, while FDA approval applies to high-risk medical devices. A third pathway is De Novo, where the FDA “grants” a new device type for certain novel low-to-moderate risk devices with no predicate. These pathways differ significantly in risk classification, evidence requirements, review timelines, and cost. As a quick takeaway:

• FDA Cleared (510(k)): FDA determined the device is substantially equivalent to a legally marketed “predicate” device for its intended use.
• FDA Granted (De Novo): FDA classified a novel low-to-moderate risk device type when there is no suitable predicate.
• FDA Approved (PMA): FDA reviewed evidence intended to demonstrate a reasonable assurance of safety and effectiveness for many high-risk (Class III) devices.
• Exempt: Most low-risk (Class I) and some Class II devices do not require premarket review and can go directly to market, though they are subject to other controls.

People often say “FDA approved” as shorthand, but for many medical devices the more precise terms are cleared (510(k)) or granted (De Novo).

What Is FDA Approval?

FDA approval refers to the Premarket Approval (PMA) process, which is the FDA’s scientific and regulatory review to evaluate whether there is a reasonable assurance of safety and effectiveness for most Class III (highest-risk) devices.

This process requires manufacturers to provide independent scientific evidence, usually from extensive clinical trials, demonstrating a reasonable assurance that the device is safe and effective for its intended use.

The FDA provides an official overview of the PMA process on its website:
https://www.fda.gov/medical-devices/premarket-submissions/premarket-approval-pma

Key Characteristics of FDA Approval

FDA approval:

• Applies to life-sustaining or life-supporting devices
• Requires extensive clinical trials
• Involves a detailed review of safety and effectiveness
• Has longer timelines and significantly higher costs
• Often includes post-market study requirements

Examples of FDA-Approved Devices

• Implantable deep brain stimulation (DBS) systems 
• Implantable neurostimulators used for epilepsy management and other neurological conditions
• Implantable continuous brain monitoring devices, such as long-term intracranial EEG systems used to detect and track seizure activity

These devices are typically implantable, life-sustaining, or life-altering, which places them in the FDA’s Class III (high-risk) category, requiring full FDA approval rather than clearance.

Example: A new implantable deep brain stimulation device for Parkinson’s disease must complete multi-year clinical trials and receive FDA approval via the PMA pathway before being marketed.

What Is FDA Clearance?

FDA clearance means a medical device has been reviewed through the 510(k) premarket notification process and found by the FDA to be substantially equivalent to an already legally marketed device. Such comparator devices are called predicate devices.

The FDA does not independently determine safety and effectiveness from scratch. Instead, it evaluates whether the new device is at least as safe and effective as an existing device already on the market.

The FDA provides an official overview of the 510(k) process on its website: https://www.fda.gov/medical-devices/premarket-submissions-selecting-and-preparing-correct-submission/premarket-notification-510k

Key Characteristics of FDA Clearance

FDA clearance:

• Applies primarily to Class II (moderate-risk) devices
• Requires comparison to a predicate device
• Typically relies on bench testing, software validation, and performance data
• Clinical trials are often not required
• Is faster and less expensive than FDA approval

Examples of FDA-Cleared Devices

Common FDA-cleared devices in the EEG and brain-monitoring industry include:

• Non-implantable EEG acquisition systems used in clinical, ambulatory, and point-of-care settings
• EEG analysis and visualization software, including cloud-based and AI-assisted interpretation tools
• EMG and evoked potential (EP) systems used for neurological diagnostics
• Neurophysiological patient monitoring systems for seizure detection and brain-state monitoring
• Wearable and portable brain-monitoring devices designed for remote, home, or outpatient use

These devices are typically classified as moderate-risk (Class II) and are cleared through the 510(k) pathway by demonstrating substantial equivalence to existing EEG and neurodiagnostic technologies already on the market.

Example:
A company develops a new EEG monitoring system that uses established signal-acquisition technology and has the same clinical indications as an existing EEG device. If it can demonstrate substantial equivalence to the existing device, it may receive FDA clearance through the 510(k) pathway.

What Is FDA “Granted” (De Novo)?

If there is no appropriate predicate device, a company may pursue the De Novo pathway for certain novel low-to-moderate risk devices. If granted, the FDA establishes a new device type, classifies it as Class I or Class II, and establishes any special controls required for Class II devices. Future devices of the same type can then use the 510(k) pathway.

Key Differences at a Glance

Common Misconceptions

“FDA cleared” does not mean “FDA approved.” These terms are not interchangeable, but both indicate the medical device can be legally marketed in the United States. Whether FDA cleared or approved, these devices’ manufacturers remain subject to quality system regulations, labeling requirements, post-market surveillance, and adverse event reporting.

An “FDA approved” label doesn’t inherently mean a device is better than an “FDA cleared” one. Those terms reflect FDA-defined regulatory pathways that are determined by a device’s risk classification, how well it fits within existing device types, and whether there’s an appropriate predicate, not a manufacturer’s preference for one label over another.

For many moderate-risk devices, the 510(k) pathway is available when a manufacturer can demonstrate the new device is substantially equivalent to a legally marketed predicate. In this pathway, the demonstration of substantial equivalence serves as a proxy for a direct demonstration of reasonable assurance of safety and effectiveness.

For example, an EEG system that can leverage a well-established, “gold standard” predicate may reach the market via 510(k) by demonstrating substantial equivalence to that predicate’s performance and intended use.

For PMA devices, showing similarity to a predicate is not considered sufficient evidence because the PMA standard is affirmative: the manufacturer must provide valid scientific evidence demonstrating a reasonable assurance that the device is safe and effective for its own specific intended use.

In higher-risk contexts, the FDA generally does not rely on substantial equivalence as a proxy for safety and effectiveness. Small differences in design, materials, software, human factors, or clinical workflow can meaningfully change the risk profile and clinical performance, and the consequences of unexpected failure are higher. As a result, the FDA typically requires direct evidence (bench, animal when relevant, and often clinical data) that addresses the device’s specific risks and demonstrates that the benefits outweigh those risks for the intended population and use.

Bottom line: ‘cleared’ versus ‘approved’ is about which regulatory route applies given FDA definitions and risk-based requirements. It is important to note that the fundamental legal standard for all pathways is to provide a reasonable assurance of safety and effectiveness; the pathway simply defines the method used to demonstrate it. The meaningful comparisons between devices come from their indications for use, claims, and supporting evidence, not from the cleared/approved label alone.

Final Thoughts

FDA clearance and FDA approval serve different but equally important roles in ensuring medical device safety in the United States. Clearance focuses on equivalence and performance for moderate-risk devices, while approval requires direct proof of safety and effectiveness for high-risk technologies.

For many innovations in neurology and digital health-especially those based on EEG technology-FDA clearance through the 510(k) pathway is the appropriate and established regulatory process, not a shortcut.

Disclaimer: This article is for informational purposes only and does not constitute legal or regulatory advice. The information provided is a general overview of FDA regulatory pathways and should not be relied upon as a substitute for consulting with a qualified regulatory professional. The FDA’s regulations and guidance are subject to change.