Ways to manage configuration files between multiple environments

Question

Problem

We use Java WAR files and save configuration files in s3 codes. Our environments: DEV, QA, Stage and PROD have their own configuration files and s3 buckets. If I add a new field, for example, "Polling_RATE = 5000", it must be added manually for each env, because these configuration files also store passwords, therefore they cannot be attached to the application or stored inside Github . Not every engineer has access to each env, so you should remember to inform top-level engineers (DEVOPS) before the implementation date of prod in order to add a new field for the application to work. This is a really dirty process now.

Question

Is there a utility or architectural design template designed to solve this problem? How are you "configuration sensitive" sensitive configuration fields that cannot be saved on github?

Answer 1

Recognizable problem.

Typically, configuration fields with sensitive information, such as passwords, change much less frequently than non-sensitive configuration fields. A possible solution is to split the configuration into two parts:

• Configure this environment but does not contain sensitive information. I would advise you to keep these files together with the source code and, if possible, generate the files and automatically upload them to the configuration repository (S3 in your case) during the build. They must be versions and are tied to the version of your application.
• Configuration containing confidential information. Considering the issue, not all team members are allowed to read / write this information. You can save them in S3 with specific access rights so that only authorized members can access them. You will need a mechanism to merge the files together when you deploy or modify the application to read from different configuration files.
  However, this solves only part of your problem. The ops parties will still have to make changes when changing sensitive configuration keys. Whether this is acceptable depends on how sensitive key keys change frequently.

An alternative to S3 could be to run a private Git repository (e.g. AWS CodecCommit ). You will have better version control and easier access for developers to make changes since you are already using Git. You still have to fix the permissions for the separation between dev and ops, or let it happen (since DevOps is trust and collaboration, this might be a good idea). You can apply a similar scheme here, as I described above.

Another solution might be to move the configuration of sensitive values ​​from property files to the system configuration. When you are already using a training system like Puppet or Chef, it will be natural for the ops guys. Or set all sensitive values, such as passwords, as environment variables and ask the application to read it as system properties.

Hope this helps!

Answer 2

We used dynamodb to store configuration values. The advantage of this approach is that the values ​​are easily read from the console and checked. Another advantage is that we periodically check the values ​​from dynamodb, so if any value needs to be changed, we just change it, and the application automatically selects a new value, and does not start it again. Sensitive values ​​are stored in encrypted form using KMS keys, and only the ec2 role on which the application is running has the right to decrypt this key. We have improved the Netflix archiaus project to suit our needs. Maybe you can check it out.