Interesting question. There is no official API for discovering if the messaging menu has changed it's state. You can do it by looking at the DBus interface of the indicator-messages-service. A shell script way is like this:
$ gdbus call --session --dest com.canonical.indicator.messages --object-path /com/canonical/indicator/messages/service --method com.canonical.indicator.messages.service.AttentionRequested
There is also a signal "
AttentionChanged". You can track that to determine the state as it updates.
Also, if you're interested in the individual items in the messaging menu all their information is exported using libindicate. It is possible to have another libindicate listener on DBus that gets all of the information and aggregates it in a different way.