LIVIVO - The Search Portal for Life Sciences

zur deutschen Oberfläche wechseln
Advanced search

Search results

Result 1 - 10 of total 53

Search options

  1. Article ; Online: Common Information Components Analysis.

    Sula, Erixhen / Gastpar, Michael

    Entropy (Basel, Switzerland)

    2021  Volume 23, Issue 2

    Abstract: Wyner's common information is a measure that quantifies and assesses the commonality between two random variables. Based on this, we introduce a novel two-step procedure to construct features from data, referred to as Common Information Components ... ...

    Abstract Wyner's common information is a measure that quantifies and assesses the commonality between two random variables. Based on this, we introduce a novel two-step procedure to construct features from data, referred to as Common Information Components Analysis (CICA). The first step can be interpreted as an extraction of Wyner's common information. The second step is a form of back-projection of the common information onto the original variables, leading to the extracted features. A free parameter γ controls the complexity of the extracted features. We establish that, in the case of Gaussian statistics, CICA precisely reduces to Canonical Correlation Analysis (CCA), where the parameter γ determines the number of CCA components that are extracted. In this sense, we establish a novel rigorous connection between information measures and CCA, and CICA is a strict generalization of the latter. It is shown that CICA has several desirable features, including a natural extension to beyond just two data sets.
    Language English
    Publishing date 2021-01-26
    Publishing country Switzerland
    Document type Journal Article
    ZDB-ID 2014734-X
    ISSN 1099-4300 ; 1099-4300
    ISSN (online) 1099-4300
    ISSN 1099-4300
    DOI 10.3390/e23020151
    Database MEDical Literature Analysis and Retrieval System OnLINE

    More links

    Kategorien

    Order via subito

    This service is chargeable due to the Delivery terms set by subito. Orders including an article and supplementary material will be classified as separate orders. In these cases, fees will be demanded for each order.

  2. Book ; Online: Alpha-NML Universal Predictors

    Bondaschi, Marco / Gastpar, Michael

    2022  

    Abstract: Inspired by Sibson's $\alpha$-mutual information, we introduce a new class of universal predictors that depend on a real parameter $\alpha \geq 1$. This class interpolates two well-known predictors, the mixture estimator, that includes the Laplace and ... ...

    Abstract Inspired by Sibson's $\alpha$-mutual information, we introduce a new class of universal predictors that depend on a real parameter $\alpha \geq 1$. This class interpolates two well-known predictors, the mixture estimator, that includes the Laplace and the Krichevsky-Trofimov predictors, and the Normalized Maximum Likelihood (NML) estimator. We point out some advantages of this class of predictors and study its performance from two complementary viewpoints: (1) we analyze it in terms of worst-case regret, as an approximation of the optimal NML, for the class of discrete memoryless sources; (2) we discuss its optimality when the maximal Renyi divergence is considered as a regret measure, which can be interpreted operationally as a middle way between the standard average and worst-case regret measures. Finally, we study how our class of predictors relates to other generalizations of NML, such as Luckiness NML and Conditional NML.
    Keywords Computer Science - Information Theory
    Publishing date 2022-02-25
    Publishing country us
    Document type Book ; Online
    Database BASE - Bielefeld Academic Search Engine (life sciences selection)

    Full text online

    More links

    Kategorien

    Inter-library loan at ZB MED

    Your chosen title can be delivered directly to ZB MED Cologne location if you are registered as a user at ZB MED Cologne.

  3. Book ; Online: From Generalisation Error to Transportation-cost Inequalities and Back

    Esposito, Amedeo Roberto / Gastpar, Michael

    2022  

    Abstract: In this work, we connect the problem of bounding the expected generalisation error with transportation-cost inequalities. Exposing the underlying pattern behind both approaches we are able to generalise them and go beyond Kullback-Leibler Divergences/ ... ...

    Abstract In this work, we connect the problem of bounding the expected generalisation error with transportation-cost inequalities. Exposing the underlying pattern behind both approaches we are able to generalise them and go beyond Kullback-Leibler Divergences/Mutual Information and sub-Gaussian measures. In particular, we are able to provide a result showing the equivalence between two families of inequalities: one involving functionals and one involving measures. This result generalises the one proposed by Bobkov and G\"otze that connects transportation-cost inequalities with concentration of measure. Moreover, it allows us to recover all standard generalisation error bounds involving mutual information and to introduce new, more general bounds, that involve arbitrary divergence measures.

    Comment: Submitted to ISIT 2022
    Keywords Computer Science - Information Theory ; Computer Science - Machine Learning ; Mathematics - Functional Analysis ; Mathematics - Probability
    Publishing date 2022-02-08
    Publishing country us
    Document type Book ; Online
    Database BASE - Bielefeld Academic Search Engine (life sciences selection)

    Full text online

    More links

    Kategorien

    Inter-library loan at ZB MED

    Your chosen title can be delivered directly to ZB MED Cologne location if you are registered as a user at ZB MED Cologne.

  4. Book ; Online: Simultaneous Computation and Communication over MAC

    Frey, Matthias / Bjelaković, Igor / Gastpar, Michael C. / Zhu, Jingge

    2024  

    Abstract: We study communication over a Gaussian multiple-access channel (MAC) with two types of transmitters: Digital transmitters hold a message from a discrete set that needs to be communicated to the receiver. Analog transmitters hold sequences of analog ... ...

    Abstract We study communication over a Gaussian multiple-access channel (MAC) with two types of transmitters: Digital transmitters hold a message from a discrete set that needs to be communicated to the receiver. Analog transmitters hold sequences of analog values, and some function of these distributed values (but not the values themselves) need to be conveyed to the receiver. For the digital messages, it is required that they can be decoded error free at the receiver with high probability while the recovered analog function values have to satisfy a fidelity criterion such as an upper bound on mean squared error (MSE) or a certain maximum error with a given confidence. For the case in which the computed function for the analog transmitters is a sum of values in [-1,1], we derive inner and outer bounds for the tradeoff of digital and analog rates of communication under peak and average power constraints for digital transmitters and a peak power constraint for analog transmitters. We then extend the achievability part of our result to a larger class of functions that includes all linear, but also some non-linear functions.
    Keywords Computer Science - Information Theory
    Subject code 003
    Publishing date 2024-01-30
    Publishing country us
    Document type Book ; Online
    Database BASE - Bielefeld Academic Search Engine (life sciences selection)

    Full text online

    More links

    Kategorien

    Inter-library loan at ZB MED

    Your chosen title can be delivered directly to ZB MED Cologne location if you are registered as a user at ZB MED Cologne.

  5. Book ; Online: Lower Bounds on the Bayesian Risk via Information Measures

    Esposito, Amedeo Roberto / Vandenbroucque, Adrien / Gastpar, Michael

    2023  

    Abstract: This paper focuses on parameter estimation and introduces a new method for lower bounding the Bayesian risk. The method allows for the use of virtually \emph{any} information measure, including R\'enyi's $\alpha$, $\varphi$-Divergences, and Sibson's $\ ... ...

    Abstract This paper focuses on parameter estimation and introduces a new method for lower bounding the Bayesian risk. The method allows for the use of virtually \emph{any} information measure, including R\'enyi's $\alpha$, $\varphi$-Divergences, and Sibson's $\alpha$-Mutual Information. The approach considers divergences as functionals of measures and exploits the duality between spaces of measures and spaces of functions. In particular, we show that one can lower bound the risk with any information measure by upper bounding its dual via Markov's inequality. We are thus able to provide estimator-independent impossibility results thanks to the Data-Processing Inequalities that divergences satisfy. The results are then applied to settings of interest involving both discrete and continuous parameters, including the ``Hide-and-Seek'' problem, and compared to the state-of-the-art techniques. An important observation is that the behaviour of the lower bound in the number of samples is influenced by the choice of the information measure. We leverage this by introducing a new divergence inspired by the ``Hockey-Stick'' Divergence, which is demonstrated empirically to provide the largest lower-bound across all considered settings. If the observations are subject to privatisation, stronger impossibility results can be obtained via Strong Data-Processing Inequalities. The paper also discusses some generalisations and alternative directions.
    Keywords Computer Science - Information Theory ; Computer Science - Machine Learning
    Subject code 000
    Publishing date 2023-03-22
    Publishing country us
    Document type Book ; Online
    Database BASE - Bielefeld Academic Search Engine (life sciences selection)

    Full text online

    More links

    Kategorien

    Inter-library loan at ZB MED

    Your chosen title can be delivered directly to ZB MED Cologne location if you are registered as a user at ZB MED Cologne.

  6. Book ; Online: Generalization Error Bounds for Noisy, Iterative Algorithms via Maximal Leakage

    Issa, Ibrahim / Esposito, Amedeo Roberto / Gastpar, Michael

    2023  

    Abstract: We adopt an information-theoretic framework to analyze the generalization behavior of the class of iterative, noisy learning algorithms. This class is particularly suitable for study under information-theoretic metrics as the algorithms are inherently ... ...

    Abstract We adopt an information-theoretic framework to analyze the generalization behavior of the class of iterative, noisy learning algorithms. This class is particularly suitable for study under information-theoretic metrics as the algorithms are inherently randomized, and it includes commonly used algorithms such as Stochastic Gradient Langevin Dynamics (SGLD). Herein, we use the maximal leakage (equivalently, the Sibson mutual information of order infinity) metric, as it is simple to analyze, and it implies both bounds on the probability of having a large generalization error and on its expected value. We show that, if the update function (e.g., gradient) is bounded in $L_2$-norm and the additive noise is isotropic Gaussian noise, then one can obtain an upper-bound on maximal leakage in semi-closed form. Furthermore, we demonstrate how the assumptions on the update function affect the optimal (in the sense of minimizing the induced maximal leakage) choice of the noise. Finally, we compute explicit tight upper bounds on the induced maximal leakage for other scenarios of interest.

    Comment: Updated to fix an error in Theorem 4 (asymptotic analysis)
    Keywords Computer Science - Machine Learning ; Computer Science - Information Theory ; Statistics - Machine Learning
    Publishing date 2023-02-28
    Publishing country us
    Document type Book ; Online
    Database BASE - Bielefeld Academic Search Engine (life sciences selection)

    Full text online

    More links

    Kategorien

    Inter-library loan at ZB MED

    Your chosen title can be delivered directly to ZB MED Cologne location if you are registered as a user at ZB MED Cologne.

  7. Book ; Online: Fantastic Generalization Measures are Nowhere to be Found

    Gastpar, Michael / Nachum, Ido / Shafer, Jonathan / Weinberger, Thomas

    2023  

    Abstract: We study the notion of a generalization bound being uniformly tight, meaning that the difference between the bound and the population loss is small for all learning algorithms and all population distributions. Numerous generalization bounds have been ... ...

    Abstract We study the notion of a generalization bound being uniformly tight, meaning that the difference between the bound and the population loss is small for all learning algorithms and all population distributions. Numerous generalization bounds have been proposed in the literature as potential explanations for the ability of neural networks to generalize in the overparameterized setting. However, in their paper ``Fantastic Generalization Measures and Where to Find Them,'' Jiang et al. (2020) examine more than a dozen generalization bounds, and show empirically that none of them are uniformly tight. This raises the question of whether uniformly-tight generalization bounds are at all possible in the overparameterized setting. We consider two types of generalization bounds: (1) bounds that may depend on the training set and the learned hypothesis (e.g., margin bounds). We prove mathematically that no such bound can be uniformly tight in the overparameterized setting; (2) bounds that may in addition also depend on the learning algorithm (e.g., stability bounds). For these bounds, we show a trade-off between the algorithm's performance and the bound's tightness. Namely, if the algorithm achieves good accuracy on certain distributions, then no generalization bound can be uniformly tight for it in the overparameterized setting. We explain how these formal results can, in our view, inform research on generalization bounds for neural networks, while stressing that other interpretations of these results are also possible.

    Comment: 34 pages, 1 figure. Minor fix: subsection 6.2 -> section 7
    Keywords Computer Science - Machine Learning ; Computer Science - Neural and Evolutionary Computing ; Statistics - Machine Learning
    Subject code 006
    Publishing date 2023-09-24
    Publishing country us
    Document type Book ; Online
    Database BASE - Bielefeld Academic Search Engine (life sciences selection)

    Full text online

    More links

    Kategorien

    Inter-library loan at ZB MED

    Your chosen title can be delivered directly to ZB MED Cologne location if you are registered as a user at ZB MED Cologne.

  8. Book ; Online: Lower-bounds on the Bayesian Risk in Estimation Procedures via $f$-Divergences

    Vandenbroucque, Adrien / Esposito, Amedeo Roberto / Gastpar, Michael

    2022  

    Abstract: We consider the problem of parameter estimation in a Bayesian setting and propose a general lower-bound that includes part of the family of $f$-Divergences. The results are then applied to specific settings of interest and compared to other notable ... ...

    Abstract We consider the problem of parameter estimation in a Bayesian setting and propose a general lower-bound that includes part of the family of $f$-Divergences. The results are then applied to specific settings of interest and compared to other notable results in the literature. In particular, we show that the known bounds using Mutual Information can be improved by using, for example, Maximal Leakage, Hellinger divergence, or generalizations of the Hockey-Stick divergence.

    Comment: Submitted to ISIT 2022
    Keywords Computer Science - Information Theory ; Mathematics - Probability ; Mathematics - Statistics Theory
    Publishing date 2022-02-05
    Publishing country us
    Document type Book ; Online
    Database BASE - Bielefeld Academic Search Engine (life sciences selection)

    Full text online

    More links

    Kategorien

    Inter-library loan at ZB MED

    Your chosen title can be delivered directly to ZB MED Cologne location if you are registered as a user at ZB MED Cologne.

  9. Book ; Online: Finite Littlestone Dimension Implies Finite Information Complexity

    Pradeep, Aditya / Nachum, Ido / Gastpar, Michael

    2022  

    Abstract: We prove that every online learnable class of functions of Littlestone dimension $d$ admits a learning algorithm with finite information complexity. Towards this end, we use the notion of a globally stable algorithm. Generally, the information complexity ...

    Abstract We prove that every online learnable class of functions of Littlestone dimension $d$ admits a learning algorithm with finite information complexity. Towards this end, we use the notion of a globally stable algorithm. Generally, the information complexity of such a globally stable algorithm is large yet finite, roughly exponential in $d$. We also show there is room for improvement; for a canonical online learnable class, indicator functions of affine subspaces of dimension $d$, the information complexity can be upper bounded logarithmically in $d$.
    Keywords Computer Science - Machine Learning ; Computer Science - Information Theory
    Publishing date 2022-06-27
    Publishing country us
    Document type Book ; Online
    Database BASE - Bielefeld Academic Search Engine (life sciences selection)

    Full text online

    More links

    Kategorien

    Inter-library loan at ZB MED

    Your chosen title can be delivered directly to ZB MED Cologne location if you are registered as a user at ZB MED Cologne.

  10. Book ; Online: The Gaussian lossy Gray-Wyner network

    Sula, Erixhen / Gastpar, Michael

    2020  

    Abstract: We consider the problem of source coding subject to a fidelity criterion for the Gray-Wyner network that connects a single source with two receivers via a common channel and two private channels. The pareto-optimal trade-offs between the sum-rate of the ... ...

    Abstract We consider the problem of source coding subject to a fidelity criterion for the Gray-Wyner network that connects a single source with two receivers via a common channel and two private channels. The pareto-optimal trade-offs between the sum-rate of the private channels and the rate of the common channel is completely characterized for jointly Gaussian sources subject to the mean-squared error criterion, leveraging convex duality and an argument involving the factorization of convex envelopes. Specifically, it is attained by selecting the auxiliary random variable to be jointly Gaussian with the sources.

    Comment: 5 pages, 2 figures. It is presented at the 54th Annual Conference on Information Sciences and Systems (CISS), Princeton, NJ, USA, March 18-20, 2020. arXiv admin note: text overlap with arXiv:1912.07083
    Keywords Computer Science - Information Theory
    Publishing date 2020-02-02
    Publishing country us
    Document type Book ; Online
    Database BASE - Bielefeld Academic Search Engine (life sciences selection)

    Full text online

    More links

    Kategorien

    Inter-library loan at ZB MED

    Your chosen title can be delivered directly to ZB MED Cologne location if you are registered as a user at ZB MED Cologne.

To top