If you have a master password, then Firefox will need you to input that password in order to use any of your stored passwords. There is no way around this, because the stored passwords are actually encrypted using a key derived from the master password.
So it is not possible to have a master password, but set Firefox only to ask for it if you want to "view" the stored passwords. It will still need to ask for it when you want to "use" them, too.
When you describe it happening at "random" times, it is probably happening when you first visit a website for which you have a stored password. If you are logging in to some websites without it asking for your master password, it will be because that website has made your login persistent (eg using cookies) negating the need to show a login form on that website. Or it will be because the browser has already asked you for your master password recently, so it has no need to ask again.
What you can do is ensure that it will only ask you for the master password once per browser session, which is controlled by the config setting
security.ask_for_password as mentioned elsewhere (set it to
Or, you can disable the master password. Your passwords will still not be shown visibly by default when you are using them, only when you go in and view them in the password manager. There is nothing wrong with finding the right balance between security and convenience for you - if you are annoyed by needing to put in your master password too much, and don't mind if Firefox can use your saved passwords without it, then a master password is not for you. Unfortunately it won't then be able to make the passwords invisible to users of your browser, though.