LIVIVO - The Search Portal for Life Sciences

zur deutschen Oberfläche wechseln
Advanced search

Search results

Result 1 - 10 of total 20

Search options

  1. Article ; Online: Overcoming catastrophic forgetting in neural networks.

    Kirkpatrick, James / Pascanu, Razvan / Rabinowitz, Neil / Veness, Joel / Desjardins, Guillaume / Rusu, Andrei A / Milan, Kieran / Quan, John / Ramalho, Tiago / Grabska-Barwinska, Agnieszka / Hassabis, Demis / Clopath, Claudia / Kumaran, Dharshan / Hadsell, Raia

    Proceedings of the National Academy of Sciences of the United States of America

    2017  Volume 114, Issue 13, Page(s) 3521–3526

    Abstract: ... of artificial intelligence. Until now neural networks have not been capable of this and it has been widely thought ... to overcome this limitation and train networks that can maintain expertise on tasks that they have not ... that catastrophic forgetting is an inevitable feature of connectionist models. We show that it is possible ...

    Abstract The ability to learn tasks in a sequential fashion is crucial to the development of artificial intelligence. Until now neural networks have not been capable of this and it has been widely thought that catastrophic forgetting is an inevitable feature of connectionist models. We show that it is possible to overcome this limitation and train networks that can maintain expertise on tasks that they have not experienced for a long time. Our approach remembers old tasks by selectively slowing down learning on the weights important for those tasks. We demonstrate our approach is scalable and effective by solving a set of classification tasks based on a hand-written digit dataset and by learning several Atari 2600 games sequentially.
    MeSH term(s) Algorithms ; Artificial Intelligence ; Computer Simulation ; Humans ; Learning ; Memory ; Mental Recall ; Neural Networks, Computer
    Language English
    Publishing date 2017-03-14
    Publishing country United States
    Document type Journal Article ; Research Support, Non-U.S. Gov't
    ZDB-ID 209104-5
    ISSN 1091-6490 ; 0027-8424
    ISSN (online) 1091-6490
    ISSN 0027-8424
    DOI 10.1073/pnas.1611835114
    Database MEDical Literature Analysis and Retrieval System OnLINE

    More links

    Kategorien

    In stock of ZB MED Cologne/Königswinter

    Zs.A 1148: Show issues Location:
    Je nach Verfügbarkeit (siehe Angabe bei Bestand)
    bis Jg. 1994: Bestellungen von Artikeln über das Online-Bestellformular
    Jg. 1995 - 2021: Lesesall (1.OG)
    ab Jg. 2022: Lesesaal (EG)

    Order via subito

    This service is chargeable due to the Delivery terms set by subito. Orders including an article and supplementary material will be classified as separate orders. In these cases, fees will be demanded for each order.

  2. Article ; Online: Natural Way to Overcome Catastrophic Forgetting in Neural Networks

    Alexey Kutalev

    Современные информационные технологии и IT-образование, Vol 16, Iss 2, Pp 331-

    2020  Volume 337

    Abstract: The problem of catastrophic forgetting manifested itself in models of neural networks based ... we want to propose alternative approaches for overcoming catastrophic forgetting, based on the total ... us to successfully overcome the problem of catastrophic forgetting. Unfortunately, although we were aware ...

    Abstract The problem of catastrophic forgetting manifested itself in models of neural networks based on the connectionist approach, which have been actively studied since the second half of the 20th century. Numerous attempts have been made and various ways to solve this problem have been proposed, but until very recently substantial successes have not been achieved. In 2016, a significant breakthrough occurred – a group of scientists from DeepMind proposed the method of elastic weight consolidation (EWC), which allows us to successfully overcome the problem of catastrophic forgetting. Unfortunately, although we were aware about the cases of using this method in real tasks, it has not yet obtained widespread distribution. In this paper, we want to propose alternative approaches for overcoming catastrophic forgetting, based on the total absolute signal passed through the connection. These approaches demonstrate similar efficiency as EWC and, at the same time, have less computational complexity. These approaches have a simpler implementation and seem to us to be essentially closer to the processes occurring in the brain of animals to preserve previously learned skills during subsequent training. We hope that the ease of implementation of these methods will serve their wider application.
    Keywords neural network ; catastrophic forgetting ; elastic weight consolidation ; back propagation ; total absolute signal ; Electronic computers. Computer science ; QA75.5-76.95
    Subject code 006
    Language Russian
    Publishing date 2020-09-01T00:00:00Z
    Publisher The Fund for Promotion of Internet media, IT education, human development «League Internet Media»
    Document type Article ; Online
    Database BASE - Bielefeld Academic Search Engine (life sciences selection)

    Full text online

    More links

    Kategorien

    Inter-library loan at ZB MED

    Your chosen title can be delivered directly to ZB MED Cologne location if you are registered as a user at ZB MED Cologne.

  3. Book ; Online: Natural Way to Overcome the Catastrophic Forgetting in Neural Networks

    Kutalev, Alexey

    2020  

    Abstract: ... we would like to propose an alternative method of overcoming catastrophic forgetting based on the total ... of neural networks. Although we know about the cases of using this method to preserve skills when adapting pre ... Not so long ago, a method was discovered that successfully overcomes the catastrophic forgetting ...

    Abstract Not so long ago, a method was discovered that successfully overcomes the catastrophic forgetting of neural networks. Although we know about the cases of using this method to preserve skills when adapting pre-trained networks to particular tasks, it has not yet obtained widespread distribution. In this paper, we would like to propose an alternative method of overcoming catastrophic forgetting based on the total absolute signal passing through each connection in the network. This method has a simple implementation and seems to us essentially close to the processes occurring in the brain of animals to preserve previously learned skills during subsequent learning. We hope that the ease of implementation of this method will serve its wide application.

    Comment: 9 pages, 3 figures
    Keywords Computer Science - Machine Learning ; Statistics - Machine Learning
    Publishing date 2020-04-27
    Publishing country us
    Document type Book ; Online
    Database BASE - Bielefeld Academic Search Engine (life sciences selection)

    Full text online

    More links

    Kategorien

    Inter-library loan at ZB MED

    Your chosen title can be delivered directly to ZB MED Cologne location if you are registered as a user at ZB MED Cologne.

  4. Book ; Online: Overcoming catastrophic forgetting in neural networks

    Kirkpatrick, James / Pascanu, Razvan / Rabinowitz, Neil / Veness, Joel / Desjardins, Guillaume / Rusu, Andrei A. / Milan, Kieran / Quan, John / Ramalho, Tiago / Grabska-Barwinska, Agnieszka / Hassabis, Demis / Clopath, Claudia / Kumaran, Dharshan / Hadsell, Raia

    2016  

    Abstract: ... of artificial intelligence. Neural networks are not, in general, capable of this and it has been widely thought ... to overcome this limitation and train networks that can maintain expertise on tasks which they have not ... that catastrophic forgetting is an inevitable feature of connectionist models. We show that it is possible ...

    Abstract The ability to learn tasks in a sequential fashion is crucial to the development of artificial intelligence. Neural networks are not, in general, capable of this and it has been widely thought that catastrophic forgetting is an inevitable feature of connectionist models. We show that it is possible to overcome this limitation and train networks that can maintain expertise on tasks which they have not experienced for a long time. Our approach remembers old tasks by selectively slowing down learning on the weights important for those tasks. We demonstrate our approach is scalable and effective by solving a set of classification tasks based on the MNIST hand written digit dataset and by learning several Atari 2600 games sequentially.
    Keywords Computer Science - Machine Learning ; Computer Science - Artificial Intelligence ; Statistics - Machine Learning
    Publishing date 2016-12-02
    Publishing country us
    Document type Book ; Online
    Database BASE - Bielefeld Academic Search Engine (life sciences selection)

    Full text online

    More links

    Kategorien

    Inter-library loan at ZB MED

    Your chosen title can be delivered directly to ZB MED Cologne location if you are registered as a user at ZB MED Cologne.

  5. Book ; Online: Overcoming Catastrophic Forgetting in Graph Neural Networks

    Liu, Huihui / Yang, Yiding / Wang, Xinchao

    2020  

    Abstract: Catastrophic forgetting refers to the tendency that a neural network "forgets" the previous learned ... dedicated to overcoming catastrophic forgetting problem and hence strengthen continual learning in GNNs ... on convolutional neural networks (CNNs), where the input samples like images lie in a grid domain, but have largely ...

    Abstract Catastrophic forgetting refers to the tendency that a neural network "forgets" the previous learned knowledge upon learning new tasks. Prior methods have been focused on overcoming this problem on convolutional neural networks (CNNs), where the input samples like images lie in a grid domain, but have largely overlooked graph neural networks (GNNs) that handle non-grid data. In this paper, we propose a novel scheme dedicated to overcoming catastrophic forgetting problem and hence strengthen continual learning in GNNs. At the heart of our approach is a generic module, termed as topology-aware weight preserving~(TWP), applicable to arbitrary form of GNNs in a plug-and-play fashion. Unlike the main stream of CNN-based continual learning methods that rely on solely slowing down the updates of parameters important to the downstream task, TWP explicitly explores the local structures of the input graph, and attempts to stabilize the parameters playing pivotal roles in the topological aggregation. We evaluate TWP on different GNN backbones over several datasets, and demonstrate that it yields performances superior to the state of the art. Code is publicly available at \url{https://github.com/hhliu79/TWP}.

    Comment: Accepted by AAAI 2021
    Keywords Computer Science - Machine Learning ; Computer Science - Artificial Intelligence
    Subject code 004 ; 006
    Publishing date 2020-12-10
    Publishing country us
    Document type Book ; Online
    Database BASE - Bielefeld Academic Search Engine (life sciences selection)

    Full text online

    More links

    Kategorien

    Inter-library loan at ZB MED

    Your chosen title can be delivered directly to ZB MED Cologne location if you are registered as a user at ZB MED Cologne.

  6. Book ; Online: Overcoming Catastrophic Forgetting in Graph Neural Networks with Experience Replay

    Zhou, Fan / Cao, Chengtai

    2020  

    Abstract: Graph Neural Networks (GNNs) have recently received significant research attention due ... present the Experience Replay based framework ER-GNN for CGL to alleviate the catastrophic forgetting ... them when learning new tasks to mitigate the catastrophic forgetting issue. We propose three experience node ...

    Abstract Graph Neural Networks (GNNs) have recently received significant research attention due to their superior performance on a variety of graph-related learning tasks. Most of the current works focus on either static or dynamic graph settings, addressing a single particular task, e.g., node/graph classification, link prediction. In this work, we investigate the question: can GNNs be applied to continuously learning a sequence of tasks? Towards that, we explore the Continual Graph Learning (CGL) paradigm and present the Experience Replay based framework ER-GNN for CGL to alleviate the catastrophic forgetting problem in existing GNNs. ER-GNN stores knowledge from previous tasks as experiences and replays them when learning new tasks to mitigate the catastrophic forgetting issue. We propose three experience node selection strategies: mean of feature, coverage maximization, and influence maximization, to guide the process of selecting experience nodes. Extensive experiments on three benchmark datasets demonstrate the effectiveness of our ER-GNN and shed light on the incremental graph (non-Euclidean) structure learning.

    Comment: 9 pages, 7 figures
    Keywords Computer Science - Machine Learning ; Statistics - Machine Learning
    Subject code 006
    Publishing date 2020-03-22
    Publishing country us
    Document type Book ; Online
    Database BASE - Bielefeld Academic Search Engine (life sciences selection)

    Full text online

    More links

    Kategorien

    Inter-library loan at ZB MED

    Your chosen title can be delivered directly to ZB MED Cologne location if you are registered as a user at ZB MED Cologne.

  7. Book ; Online: Correlation of the importances of neural network weights calculated by modern methods of overcoming catastrophic forgetting

    Kutalev, Alexey

    2022  

    Abstract: ... allow EWC method to overcome the catastrophic forgetting of neural networks perfectly? ... Comment: 15 ... the importance of neural network weights for use in the EWC method. Despite the significant difference ...

    Abstract Following the invention in 2017 of the EWC method, several methods have been proposed to calculate the importance of neural network weights for use in the EWC method. Despite the significant difference in calculating the importance of weights, they all proved to be effective. Accordingly, a reasonable question arises as to how similar the importances of the weights calculated by different methods. To answer this question, we calculated layer-by-layer correlations of the importance of weights calculated by all those methods. As a result, it turned out that the importances of several of the methods correlated with each other quite strongly and we were able to present an explanation for such a correlation. At the same time, for other methods, the correlation can vary from strong on some layers of the network to negative on other layers. Which raises a reasonable question: why, despite the very different calculation methods, all those importances allow EWC method to overcome the catastrophic forgetting of neural networks perfectly?

    Comment: 15 pages, 7 figures
    Keywords Computer Science - Machine Learning ; 68T07 ; I.2.6
    Publishing date 2022-10-24
    Publishing country us
    Document type Book ; Online
    Database BASE - Bielefeld Academic Search Engine (life sciences selection)

    Full text online

    More links

    Kategorien

    Inter-library loan at ZB MED

    Your chosen title can be delivered directly to ZB MED Cologne location if you are registered as a user at ZB MED Cologne.

  8. Book ; Online: Overcoming Catastrophic Forgetting by XAI

    Nguyen, Giang

    2022  

    Abstract: Explaining the behaviors of deep neural networks, usually considered as black boxes, is critical ... articulate how catastrophic forgetting happens, particularly showing which components of this famous network ... Forgetting Dissector (or CFD) to explain catastrophic forgetting in continual learning settings. We also ...

    Abstract Explaining the behaviors of deep neural networks, usually considered as black boxes, is critical especially when they are now being adopted over diverse aspects of human life. Taking the advantages of interpretable machine learning (interpretable ML), this work proposes a novel tool called Catastrophic Forgetting Dissector (or CFD) to explain catastrophic forgetting in continual learning settings. We also introduce a new method called Critical Freezing based on the observations of our tool. Experiments on ResNet articulate how catastrophic forgetting happens, particularly showing which components of this famous network are forgetting. Our new continual learning algorithm defeats various recent techniques by a significant margin, proving the capability of the investigation. Critical freezing not only attacks catastrophic forgetting but also exposes explainability.

    Comment: Master of Science Thesis at KAIST; 24 pages; Keywords: continual learning, catastrophic forgetting, XAI, attribution map, interpretability
    Keywords Computer Science - Machine Learning
    Subject code 006
    Publishing date 2022-11-25
    Publishing country us
    Document type Book ; Online
    Database BASE - Bielefeld Academic Search Engine (life sciences selection)

    Full text online

    More links

    Kategorien

    Inter-library loan at ZB MED

    Your chosen title can be delivered directly to ZB MED Cologne location if you are registered as a user at ZB MED Cologne.

  9. Article ; Online: Overcoming Catastrophic Forgetting in Continual Learning by Exploring Eigenvalues of Hessian Matrix.

    Kong, Yajing / Liu, Liu / Chen, Huanhuan / Kacprzyk, Janusz / Tao, Dacheng

    IEEE transactions on neural networks and learning systems

    2023  Volume PP

    Abstract: Neural networks tend to suffer performance deterioration on previous tasks when they are applied ... forgetting, a significant challenge in continual learning (CL). To overcome the catastrophic forgetting ... mitigates catastrophic forgetting and outperforms existing regularization-based methods. ...

    Abstract Neural networks tend to suffer performance deterioration on previous tasks when they are applied to multiple tasks sequentially without access to previous data. The problem is commonly known as catastrophic forgetting, a significant challenge in continual learning (CL). To overcome the catastrophic forgetting, regularization-based CL methods construct a regularization-based term, which can be considered as the approximation loss function of previous tasks, to penalize the update of parameters. However, the rigorous theoretical analysis of regularization-based methods is limited. Therefore, we theoretically analyze the forgetting and the convergence properties of regularization-based methods. The theoretical results demonstrate that the upper bound of the forgetting has a relationship with the maximum eigenvalue of the Hessian matrix. Hence, to decrease the upper bound of the forgetting, we propose eiGenvalues ExplorAtion Regularization-based (GEAR) method, which explores the geometric properties of the approximation loss of prior tasks regarding the maximum eigenvalue. Extensive experimental results demonstrate that our method mitigates catastrophic forgetting and outperforms existing regularization-based methods.
    Language English
    Publishing date 2023-07-21
    Publishing country United States
    Document type Journal Article
    ISSN 2162-2388
    ISSN (online) 2162-2388
    DOI 10.1109/TNNLS.2023.3292359
    Database MEDical Literature Analysis and Retrieval System OnLINE

    More links

    Kategorien

    Order via subito

    This service is chargeable due to the Delivery terms set by subito. Orders including an article and supplementary material will be classified as separate orders. In these cases, fees will be demanded for each order.

    Inter-library loan at ZB MED

    Your chosen title can be delivered directly to ZB MED Cologne location if you are registered as a user at ZB MED Cologne.

  10. Book ; Online: Overcoming Recency Bias of Normalization Statistics in Continual Learning

    Lyu, Yilin / Wang, Liyuan / Zhang, Xingxing / Sun, Zicheng / Su, Hang / Zhu, Jun / Jing, Liping

    Balance and Adaptation

    2023  

    Abstract: ... on overcoming catastrophic forgetting of old tasks in gradient-based optimization. However, the normalization ... With limited access to old training samples, much of the current work in deep neural networks has focused ... to 7.68%, 6.86% and 4.26% on Split CIFAR-10, Split CIFAR-100 and Split Mini-ImageNet, respectively ...

    Abstract Continual learning entails learning a sequence of tasks and balancing their knowledge appropriately. With limited access to old training samples, much of the current work in deep neural networks has focused on overcoming catastrophic forgetting of old tasks in gradient-based optimization. However, the normalization layers provide an exception, as they are updated interdependently by the gradient and statistics of currently observed training samples, which require specialized strategies to mitigate recency bias. In this work, we focus on the most popular Batch Normalization (BN) and provide an in-depth theoretical analysis of its sub-optimality in continual learning. Our analysis demonstrates the dilemma between balance and adaptation of BN statistics for incremental tasks, which potentially affects training stability and generalization. Targeting on these particular challenges, we propose Adaptive Balance of BN (AdaB$^2$N), which incorporates appropriately a Bayesian-based strategy to adapt task-wise contributions and a modified momentum to balance BN statistics, corresponding to the training and testing stages. By implementing BN in a continual learning fashion, our approach achieves significant performance gains across a wide range of benchmarks, particularly for the challenging yet realistic online scenarios (e.g., up to 7.68%, 6.86% and 4.26% on Split CIFAR-10, Split CIFAR-100 and Split Mini-ImageNet, respectively). Our code is available at https://github.com/lvyilin/AdaB2N.

    Comment: Accepted by NeurIPS 2023
    Keywords Computer Science - Machine Learning
    Subject code 006
    Publishing date 2023-10-13
    Publishing country us
    Document type Book ; Online
    Database BASE - Bielefeld Academic Search Engine (life sciences selection)

    Full text online

    More links

    Kategorien

    Inter-library loan at ZB MED

    Your chosen title can be delivered directly to ZB MED Cologne location if you are registered as a user at ZB MED Cologne.

To top