The Optimal Input-Independent Baseline for Binary Classification: The Dutch Draw

Before any binary classification model is taken into practice, it is important to validate its performance on a proper test set. Without a frame of reference given by a baseline method, it is impossible to determine if a score is `good' or `bad'. The goal of this paper is to examine all baseline methods that are independent of feature values and determine which model is the `best' and why. By identifying which baseline models are optimal, a crucial selection decision in the evaluation process is simplified. We prove that the recently proposed Dutch Draw baseline is the best input-independent classifier (independent of feature values) for all positional-invariant measures (independent of sequence order) assuming that the samples are randomly shuffled. This means that the Dutch Draw baseline is the optimal baseline under these intuitive requirements and should therefore be used in practice.

Comparison of different automatic solutions for resection cavity segmentation in postoperative MRI volumes including longitudinal acquisitions

In this work, we compare five deep learning solutions to automatically segment the resection cavity in postoperative MRI. The proposed methods are based on the same 3D U-Net architecture. We use a dataset of postoperative MRI volumes, each including four MRI sequences and the ground truth of the corresponding resection cavity. Four solutions are trained with a different MRI sequence. Besides, a method designed with all the available sequences is also presented. Our experiments show that the method trained only with the T1 weighted contrast-enhanced MRI sequence achieves the best results, with a median DICE index of 0.81.

The Dutch Draw: Constructing a Universal Baseline for Binary Prediction Models

Novel prediction methods should always be compared to a baseline to know how well they perform. Without this frame of reference, the performance score of a model is basically meaningless. What does it mean when a model achieves an $F_1$ of 0.8 on a test set? A proper baseline is needed to evaluate the `goodness' of a performance score. Comparing with the latest state-of-the-art model is usually insightful. However, being state-of-the-art can change rapidly when newer models are developed. Contrary to an advanced model, a simple dummy classifier could be used. However, the latter could be beaten too easily, making the comparison less valuable. This paper presents a universal baseline method for all binary classification models, named the Dutch Draw (DD). This approach weighs simple classifiers and determines the best classifier to use as a baseline. We theoretically derive the DD baseline for many commonly used evaluation measures and show that in most situations it reduces to (almost) always predicting either zero or one. Summarizing, the DD baseline is: (1) general, as it is applicable to all binary classification problems; (2) simple, as it is quickly determined without training or parameter-tuning; (3) informative, as insightful conclusions can be drawn from the results. The DD baseline serves two purposes. First, to enable comparisons across research papers by this robust and universal baseline. Secondly, to provide a sanity check during the development process of a prediction model. It is a major warning sign when a model is outperformed by the DD baseline.

Jasmine: A New Active Learning Approach to Combat Cybercrime

Over the past decade, the advent of cybercrime has accelarated the research on cybersecurity. However, the deployment of intrusion detection methods falls short. One of the reasons for this is the lack of realistic evaluation datasets, which makes it a challenge to develop techniques and compare them. This is caused by the large amounts of effort it takes for a cyber analyst to classify network connections. This has raised the need for methods (i) that can learn from small sets of labeled data, (ii) that can make predictions on large sets of unlabeled data, and (iii) that request the label of only specially selected unlabeled data instances. Hence, Active Learning (AL) methods are of interest. These approaches choose speci?fic unlabeled instances by a query function that are expected to improve overall classi?cation performance. The resulting query observations are labeled by a human expert and added to the labeled set. In this paper, we propose a new hybrid AL method called Jasmine. Firstly, it determines how suitable each observation is for querying, i.e., how likely it is to enhance classi?cation. These properties are the uncertainty score and anomaly score. Secondly, Jasmine introduces dynamic updating. This allows the model to adjust the balance between querying uncertain, anomalous and randomly selected observations. To this end, Jasmine is able to learn the best query strategy during the labeling process. This is in contrast to the other AL methods in cybersecurity that all have static, predetermined query functions. We show that dynamic updating, and therefore Jasmine, is able to consistently obtain good and more robust results than querying only uncertainties, only anomalies or a ?fixed combination of the two.

A Ray-based Approach for Boundary Estimation of Fiber Bundles Derived from Diffusion Tensor Imaging

Diffusion Tensor Imaging (DTI) is a non-invasive imaging technique that allows estimation of the location of white matter tracts in-vivo, based on the measurement of water diffusion properties. For each voxel, a second-order tensor can be calculated by using diffusion-weighted sequences (DWI) that are sensitive to the random motion of water molecules. Given at least 6 diffusion-weighted images with different gradients and one unweighted image, the coefficients of the symmetric diffusion tensor matrix can be calculated. Deriving the eigensystem of the tensor, the eigenvectors and eigenvalues can be calculated to describe the three main directions of diffusion and its magnitude. Using DTI data, fiber bundles can be determined, to gain information about eloquent brain structures. Especially in neurosurgery, information about location and dimension of eloquent structures like the corticospinal tract or the visual pathways is of major interest. Therefore, the fiber bundle boundary has to be determined. In this paper, a novel ray-based approach for boundary estimation of tubular structures is presented.

Benchmarking the Quality of Diffusion-Weighted Images

We present a novel method that allows for measuring the quality of diffusion-weighted MR images dependent on the image resolution and the image noise. For this purpose, we introduce a new thresholding technique so that noise and the signal can automatically be estimated from a single data set. Thus, no user interaction as well as no double acquisition technique, which requires a time-consuming proper geometrical registration, is needed. As a coarser image resolution or slice thickness leads to a higher signal-to-noise ratio (SNR), our benchmark determines a resolution-independent quality measure so that images with different resolutions can be adequately compared. To evaluate our method, a set of diffusion-weighted images from different vendors is used. It is shown that the quality can efficiently be determined and that the automatically computed SNR is comparable to the SNR which is measured manually in a manually selected region of interest.

Ray-Based and Graph-Based Methods for Fiber Bundle Boundary Estimation

Diffusion Tensor Imaging (DTI) provides the possibility of estimating the location and course of eloquent structures in the human brain. Knowledge about this is of high importance for preoperative planning of neurosurgical interventions and for intraoperative guidance by neuronavigation in order to minimize postoperative neurological deficits. Therefore, the segmentation of these structures as closed, three-dimensional object is necessary. In this contribution, two methods for fiber bundle segmentation between two defined regions are compared using software phantoms (abstract model and anatomical phantom modeling the right corticospinal tract). One method uses evaluation points from sampled rays as candidates for boundary points, the other method sets up a directed and weighted (depending on a scalar measure) graph and performs a min-cut for optimal segmentation results. Comparison is done by using the Dice Similarity Coefficient (DSC), a measure for spatial overlap of different segmentation results.

A Semi-Automatic Graph-Based Approach for Determining the Boundary of Eloquent Fiber Bundles in the Human Brain

Diffusion Tensor Imaging (DTI) allows estimating the position, orientation and dimension of bundles of nerve pathways. This non-invasive imaging technique takes advantage of the diffusion of water molecules and determines the diffusion coefficients for every voxel of the data set. The identification of the diffusion coefficients and the derivation of information about fiber bundles is of major interest for planning and performing neurosurgical interventions. To minimize the risk of neural deficits during brain surgery as tumor resection (e.g. glioma), the segmentation and integration of the results in the operating room is of prime importance. In this contribution, a robust and efficient graph-based approach for segmentating tubular fiber bundles in the human brain is presented. To define a cost function, the fractional anisotropy (FA) is used, derived from the DTI data, but this value may differ from patient to patient. Besides manually definining seed regions describing the structure of interest, additionally a manual definition of the cost function by the user is necessary. To improve the approach the contribution introduces a solution for automatically determining the cost function by using different 3D masks for each individual data set.