How To Structure Pulumi Projects
Firstly, let me get this out of the way. There is no hard and fast rule or requirement for structuring Pulumi projects. Now that that's out of the way, let's look at couple of approaches and compare them with each other. Hopefully, you'll come out of this a bit clearer on what you want to do for your projects.
What Are Projects?
Projects are a grouping of stacks. You can have as many stacks as you want in a project. Projects define the runtime environment and other settings that apply to all the stacks contained within. There is no limit to the number of projects you can have in your account either. The project configuration settings exist in the Pulumi.yaml which will always be created regardless of whether or not you have per-stack configuration. On the other hand, the per-stack configuration file which takes the form Pulumi.<stack name>.yaml will only be created by the Pulumi CLI when you run the pulumi config set ... command for the first time for a stack.
The project configuration file is the entrypoint for the CLI to understand some basic things about your project. See the full configuration reference for Pulumi.yaml to learn about all the available options.
In a team-based environment, you may be looking to control access to your stacks. Just remember that you can only restrict access to stacks but not to the whole project. The workaround for that is, if your project is in an organization that you are a member, ask the org admin to change the default stack access policy to None at the org-level. That ensures no one gets access to stacks, by default, and have to be explicitly granted access for the stacks on which they want to operate.
Individual Projects
Let's say you have a database and a network project. The idea is that each project manages only those resources that the project name implies. Or perhaps the project name is "core" or "foundational" -- then the idea is that all of your core/foundation cloud resources are in that project. Then you'd have multiple stacks for each of those projects. You can leverage Pulumi's StackReference feature to depend on an upstream stack's outputs. Yes! You can read a stack's output using stack references. They are very useful.
Monolith Project Or The "Everything Bagel"
Most developers start this way. This is sort of the default approach everyone starts with and most end up sticking with as well, even beyond the prototype phase. With this approach you'd end up having your entire infrastructure in a single project with a stack for each environment. That is, test, staging, prod etc. It's not necessarily a bad thing. You could still modularize your code by following the idiomatic way for the language you are using. A couple of things can influence your decision to move away from this. For example, you may want to give granular access to resources managed by the project, that is, you may be looking to restrict access to networking and DB resources to a core group of developers. Or, you may want to group less-frequently changing resources into a separate project.
Quick Comparison
| Individual Projects | Monolith Projects |
|---|---|
| Each project contains only a specific set of resources | All of your resources are in a single project |
| Use stack references to refer to resources across projects | Refer to other resources locally |
| Updating upstream projects requires coordination in your CI/CD service (hopefully you are using one!) | Since everything is local to the project coordination is not required |
| Ability to limit access to all stacks of a project that contains sensitive resources | Limited access restriction since everything is in the same project |
Whichever approach you choose, it is likely that you'd want to create multiple stacks per-project, especially if you are supporting the full application lifecycle, i.e. dev/individual -> QA -> Staging -> Prod. It is rare that you'd want to name the stack something other than its environment designation (there are exceptions to this but you decide what those are.) Stacks are a great way to separate configuration. It is what allows you to deploy similar infrastructure, repeatedly and consistently simply by changing the configuration. Note that I emphasized the word similar there. That's because, it is also often the case that you'd want staging and production to be as similar as possible which could mean provisioning the same machine type, size etc. But you don't need the same level of commitment with the other environments which means your infrastructure across environments are most likely to have a slight variation. This is one of the implicit assumptions most developers make when working with a general-purpose programming language, being able to have conditionals and using other programming patterns they are used to in their application code. It's one of the things that makes your infrastructure codebase flexible.