Enemies in Metal Gear Solid V have four main behavior states - the state of full calmness, curiosity, increased vigilance and full alarm. The guard might switch to the state of curiosity when he noticed a suspicious person, but couldn't identify it as you or when he heard something suspicious. If you manage to hide fast enough, the alarm won't be raised. However, the guard might decide to check the territory where he noticed something. In that case you can act in one of three ways - quickly move away from the place where the enemy is going, wait for him and neutralize him (a risky method, but might prove to be effective) or distract his attention. An empty clip dropped to another place is the best way for the latter, especially since you can decide precisely where it will land (you can, for example, point the enemy in a completely different direction).
Increased vigilance usually means more trouble, especially since the nervous guard very often informs his colleagues about what's worrying him and they gang together for the "hunt". This state usually activates when you make some larger mistake, for example when you shoot from a weapon without a suppressor or when you allow a body of a knocked-out or killed guard to be found. Similarly to the state of curiosity, you can quickly get away from the zone that the enemies will want to search. You can also hide someplace, for example in the bushes, and wait. After some time, the situation calms itself and the guards will go back to their routine patrols.
A state of full alert starts when you are detected by an enemy and you didn't manage to eliminate him quickly enough. That person will then start an alarm and the attack on you will start. Most of all, you should know how the alert raising mechanism works. By default, only the most immediate vicinity of the specific opponent is being alarmed. If that specific opponent has been in the open area, far away from any developments, there is a chance that only several of the enemies will respond to the alarm. If that enemy has been inside the base, most probably all of the opponents stationed there respond to the alarm. There is one more obstacle, regarding the alarms. The personnel of a given base may contact the other bases and smaller outposts, which may result in the heightened alertness of the larger portion of the map. You can remedy this by sabotaging communications devices in bases, beforehand, e.g. by blowing up antennas on rooftops, or by cutting off power supply.
The best response to the alarm is escape, especially that the opponents that you eliminate may soon be replaced by new ones. Apart from that, longer fights may deteriorate your ranking at the end of the mission and lead to lowering Heroism Score.
One of the new additions in Metal Gear Solid V is Reflex Mode, which is an addition really handy in sneaking. It automatically slows time whenever you are detected by a guard. It gives you few seconds to shoot that person. If you succeed, then the alarm won't be raised.
Reflex Mode has two large cons. First, using it is limited, don't count that you can move through the game world as you want and ignore any reconnaissance. Second, using Reflex mode will strip you of the bonus score after completing the mission and thus it might make it harder for you to obtain the highest S rank. If you don't want to use this option at all, you can deactivate it in the main menu.