I think I found a way myself, without any transcoding, with some inspiration from the Coxys answer. MP4Box allows me to do this: MP4Box -cat file1.m4a -cat file2.m4a output.m4a. It does not save metadata, but for my purposes this is just fine.
Now, given Coxys answer, are there any pitfalls with a file that I haven't opened yet?
source
share