The second command looks right. The h264 raw video data is a bit complicated because it has two characteristics - alignment and stream-format, which can vary. h264parse can convert h264 data to the form needed for various h264-related GStreamer elements.
avdec_h264 - decoder element. You do not want to decode the data, as you apparently do not display it. You put the h264 encoded data from the RTSP stream into the mp4 container file.
If the file does not play, you should make sure that the stream is good or try other media players and see if they work (mplayer, Media Player, Quicktime, whatever).
You can also try muxing into the matroska container file using the "matroskamux" element.
source share