Green-shift in continual learning for Industry 4.0

Research project (sustainability)

With Artificial Intelligence (AI) and Machine Learning (ML) exploding in popularity, it has seen a drastic increase in use across multiple domains where machine learning models serve crucial roles. As an example, in the manufacturing domain, AI and ML models are used to improve the efficiency of the manufacturing process and reduce the waste produced as a result. Where the increase in popularity has caused the prevalence of AI and ML to increase, the growth of Big Data can be attributed as one of the factors responsible for the growing size of data that the models are trained on. This in turn increase the time it takes to train these models as well as the resources required to do so. This cost is also applied when updating the model is necessary, as in the manufacturing domain, where models trained on sensor data can over time become inaccurate due to sensor drift caused by for instance contextual changes. This will then usually require retraining the models from scratch on an updated dataset which includes this recent data.

In addition to an increase in hardware requirements and time, this also increases the energy spent on training these models, thus contrasting with the growing concern over the negative effects that increasing computing can have on the environment. As the world at large looks towards more sustainable solutions, Artificial Intelligence and Machine learning should also seek to adapt and contribute to this new green shift.

To reduce the cost of retraining, one possible approach is Continual Learning, which is an emerging machine learning method that is growing in relevance. Continual Learning is a method which allows for incremental learning, allowing for existing models to be updated with new data. This retraining can occur without having to reuse the old data in its entirety, thus facilitating a more efficient retraining of the model. However, as Continual Learning is a new and challenging field there are difficulties. Retraining on incremental data can easily lead to catastrophic forgetting, degrading the model to where it is unable to benefit from being trained on previously acquired data. Similarly, determining when to retrain a model is also crucial to minimize the use of computing resources.

The aim is to continually adapt the model based on new data acquired from the changed context of the sensors without discarding the benefits of having a model which is trained on a large amount of data collected over significant period of time. This incremental update of the model should occur at appropriate interval to ensure a reliable model while keeping the system as sustainable as possible.

Research Topic 1

Explore how Continual Learning can contribute to mitigating emissions by reducing the energy consumed when retraining ML models. By leveraging existing projects like Erdre, research the effect of Continual Learning when contrasted with regular ML methods.

Required knowledge and skills

• Experience with ML/AI tools and frameworks (TensorFlow, Keras, PyTorch, etc.)
• Good programming skills in one of the languages: Python

References

Erdre - Erroneous data repair for Industry 4.0
https://github.com/SINTEF-9012/Erdre

Embracing Change: Continual Learning in Deep Neural Networks
https://doi.org/10.1016/j.tics.2020.09.004

German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, Stefan Wermter, Continual lifelong learning with neural networks: A review,
https://doi.org/10.1016/j.neunet.2019.01.012

Energy and Policy Considerations for Deep Learning in NLP
https://doi.org/10.18653/v1/P19-1355