The glass transition temperature (T$_g$) is the temperature, after which the supercooled liquid undergoes a dynamical arrest. Usually, the glass network modifiers (e.g., Na$_2$O) affect the behavior of T$_g$. However, in aluminosilicate glasses, the effect of different modifiers on T$_g$ is still unclear and show an anomalous behavior. Here, based on molecular dynamics simulations, we show that the glass transition temperature decreases with increasing charge balancing cations field strength (FS) in the aluminosilicate glasses, which is an anomalous behavior as compared to other oxide glasses. The results show that the origins of this anomaly come from the dynamics of the supercooled liquid above T$_g$, which in turn is correlated to pair excess entropy. Our results deepen our understanding of the effect of different modifiers on the properties of the aluminosilicate glasses.