Read docker-compose docs , in the docker-compose.yml file it commandrefers to the actual command executed inside the container, not the parameters you pass to docker run. Your example translates to:
rstudio:
image: rocker-hadleyverse
ports:
- "8787:8787"
environment:
- USER=foo
- PASSWORD=bar
links:
- db
docker-compose up -d.