Currently, if a PR adds a CHANGELOG.md entry and the base is updated with another entry, the PR will have a merge conflict. We should somehow solve this, either by not requiring changelog entries (and automatically generating them when needed), or some other way.