Are you at the very back? Maybe you could make a sign. And if not, you could maybe get a bright colored sweatshirt or something that is obvious whether you're signaling or not and put it on when you're stressed and then just leave the room like you have to go to the bathroom. If you're in college, you don't have to ask for permission, right? It may be a little strange that your teacher follows you shortly after, but he could just be checking on you, and at least you won't hear if your classmates say anything, which I doubt they would anyway because a lot of people have trouble with math regardless of age or disability, and from my memory of college, it's often a lot less gossipy than high school, especially early morning and evening classes. Good luck!

I'd try like waving or pointing towards the door as an indication that you need to have a chat outside or maybe

I used to have what was called a timeout card. You would just discreetly show it to the teacher and they would know that it means u need a few minutes outside of the classroom. It helped me especially when I was nonverbal and didn’t know how to tell my teacher I needed a break