What is a secret manager (e.g hashicorp vault) and why do applications need one?

What is a secret → Secret can be defined as a piece of sensitive information that will help to authenticate(who you are)/authorize(what you can) to a given system? Examples of secrets are passwords application use to login to an application or database, API credentials, encryption keys, key store passwords e.t.c

So how do applications manage secrets all these days?

• Plain text → Store secrets as plain text in property files.
• Base64 Encoding→ Store passwords in base64 encoding format. This will provide a sense of security that we are not storing secrets in plain text, but still, anyone can decode encoded secrets
• Encryption → Using encrypted secrets in property files, which applications will decrypt on the fly in code.
• Database → Using database table (e.g auth table) to maintain secrets.
• Application or deployment environment → Store secrets in the deployment environment(like Kubernetes, Jenkin) which will then be referred from applications deploying those environments applications
• Hardware Security Modules (HSMs) → These are external devices like plugin cards, that will be attached to the server, which will be mainly used to store secrets of type digital keys, mainly.

So what is the issue with the above mechanisms?

• Applications are responsible for secret security. If we have…