Advancements in deep learning are driven by training models with increasingly larger numbers of parameters, which in turn heightens the computational demands. To address this issue, Mixture-of-Depths (MoD) models have been proposed to dynamically assign computations only to the most relevant parts of the inputs, thereby enabling the deployment of large-parameter models with high efficiency during inference and training. These MoD models utilize a routing mechanism to determine which tokens should be processed by a layer, or skipped. However, conventional MoD models employ additional network layers specifically for the routing which are difficult to train, and add complexity and deployment overhead to the model. In this paper, we introduce a novel attention-based routing mechanism A-MoD that leverages the existing attention map of the preceding layer for routing decisions within the current layer. Compared to standard routing, A-MoD allows for more efficient training as it introduces no additional trainable parameters and can be easily adapted from pretrained transformer models. Furthermore, it can increase the performance of the MoD model. For instance, we observe up to 2% higher accuracy on ImageNet compared to standard routing and isoFLOP ViT baselines. Furthermore, A-MoD improves the MoD training convergence, leading to up to 2x faster transfer learning.

Emotion recognition is the identification of emotions usually through verbal communication and facial expressions such as happy, angry, sad, etc. Not only on the basis of a wide spectrum of moods, but different emotions can also be recognized in order to track mental health of as many people as possible for societal well being. Inside positive it detects specific emotions like happiness, satisfaction, or excitement -depending on how it’s configured. The main principles involved in the implementation of our sentiment recognition system that identifies various emotions: anger, happiness, depression, neutral, etc. are audio content and identification of the emotion associated with it. The application developed takes audio input, applies Mel-Frequency Cepstral Coefficients (MFCC) algorithm on it, compares them with those of the content of the existing audio file database depicting various human sentiments, and presents output in the text the emotion expressed by the user. The input from testing was gathered and meaningful spectral coefficients were extracted and stored in a database for comparison with future audio samples. The application extracts the coefficients of the external audio sample and matches it with those present in the database. MFCC algorithm is used to extract the spectral coefficients which are good and can be used for feature matching purposes discarding any static and background noise if present. We have done comparative analysis on our models for their performance evaluation, using four classification metrics and also presented the confusion matrix for better understanding.