How to parse access block in h.264

Question

I am working on a project that needs to cut off some Access units in an H.264 raw elementary stream, for example, remove 4 access blocks and play the remaining video.

To do this, I took the Access unit Delimiter ( NAL Unit Type:9 ) as the border for the Access Unit and cut the video, but the video ended with packet loss. But if I took the Sequence parameter set ( NAL Unit type :7 ) as the border, the resulting video playback without packet loss.
Someone please help me solve this problem: where should I disconnect the video?

Answer 1

Since you did not describe your stream in a bit more detail, for example, what NALUs are present and how the material is formatted, some general recommendations are given:

You always need SPS / PPS as they contain information on how to decode frames. I assume that you get the application B stream stream, in this case, cache the last SPS / PPS until a new one appears, decode the remaining images with the old one, and then apply the new one.

If you have AUD NALU, you already have the end of the last access unit, and you can easily split the video and don’t have to do much.

If you need to dig deeper, here is an excerpt from the specifications regarding the order of the NAL blocks and how to detect the first VAL NAL block of the primary encoded image.

From ITU-T H.264 (05/2013)

7.4.1.2.3 Order of NAL blocks and coded pictures and association for access blocks

This section indicates the order of the NAL units and encoded images and the association for the access unit for encoded video sequences that correspond to one or more profiles specified in Appendix A and are decoded using the decoding process specified in clauses 2-9.

An access unit consists of one primary encoded image, zero or more corresponding backup encoded images, and zero or more non-VCL NAL units. The relationship of VAL NAL units with primary or redundant encoded images is described in section 7.4.1.2.5.

The first access block in the bitstream begins with the first NAL block of the bitstream.

The first of any of the following NAL units after the last VAL NAL unit of the primary encoded image indicates the start of a new access unit:

• access block block NAL block (if present),

• NAL sequence parameter set (if present)

• NAL image parameter set (if present)

• SEI NAL unit (if present)

• NAL units with nal_unit_type ranging from 14 to 18 inclusive (if any),

• the first VCL NAL block of the primary encoded image (always present).

The limitations for detecting the first VAL NCL block of the primary encoded image are specified in clause 7.4.1.2.4.

The following restrictions must be met by the order of the encoded images and non-VCL NAL units within the access unit:

• When an NAL unit of an access separation unit is present, it must be the first NAL unit. There should be no more than one access block block NAL block in any access block.

• If any NAL SEI units are present, they must precede the primary encoded image.

• When a SEI NAL unit containing a buffer period SEI message is present, the buffer period SEI message should be the first payload of the SEI message of the first SEI NAL block in the access unit.

• The primary encoded image must precede the corresponding redundant encoded images.

• If redundant encoded images are present, they should be ordered in ascending order of redundant_pic_cnt.

• If there is a NAL block with a set of sequence parameters, it must be the next NAL block after the set NAL sequence parameter has the same seq_parameter_set_id value as in the NAL block for the sequence parameter set.

• If one or more coded fragments of the auxiliary coded image are present without separating the NAL units, they must follow the main coded image and all redundant encoded images (if any).

• When the end of the NAL sequence is present, it must follow the main encoded image and all redundant encoded images (if any) and the entire encoded fragment of the auxiliary encoded image without separation of the NAL blocks (if any).

• When an end-of-stream NAL unit is installed, it must be the last NAL unit.

• NAL units having nal_unit_type equal to 0, 12, or in the range of 20 to 31 inclusive, shall not precede the first VCL NAL unit of the primary encoded image.

(NOTE 2 — The NAL sequence parameter set or NAL picture parameter set may be present in the access unit, but cannot follow the last VCL NAL unit of the primary encoded image in the access unit, as this condition indicates the start of a new access unit.)

(NOTE 3 — If an NAL unit having a nal_unit_type of 7 or 8 is present in the access unit, it may or may not be indicated in the encoded images of the access unit in which it is present, and may be mentioned in the encoded images of subsequent access units.)

The structure of access blocks that do not contain any NAL blocks with nal_unit_type equal to 0, 7, 8, or in the range from 12 to 18 inclusive or in the range from 20 to 31 inclusive, is shown in Figure 7-1.

7.4.1.2.4 Detection of the first VAL NCL block of the primary encoded image

This section describes the syntax restrictions of the VCL NAL block, which are sufficient to allow detection of the first VAL block NCL of each primary encoded image for encoded video sequences that correspond to one or more profiles specified in Appendix A and are decoded using the decoding process specified in clauses 2-9.

Any NAL unit with an encoded slice or a data section with a section of encoded sections The NAL unit of the primary encoded image of the current access unit must be different from any NAL unit of the encoded fragment or Node data unit of the encoded AAL fragment of the primary encoded image, the previous access unit in one or more of the following ways:

• frame_num is different in meaning. The frame_num value used to verify this condition is the frame_num value that appears in the slice header syntax, regardless of whether this value is set to 0 for later use in the decoding process due to memory_management_control_operation equal to 5.

(NOTE 1 - A consequence of the above statement is that a primary encoded image having frame_num equal to 1 cannot contain memory_management_control_operation equal to 5 unless any other condition listed below is satisfied for the next primary encoded image following him (if any).

• pic_parameter_set_id differs in value.

• field_pic_flag is different in meaning.

• bottom_field_flag is present in both and differs in meaning.

• nal_ref_idc differs in value when one of the nal_ref_idc values ​​is 0.

• pic_order_cnt_type is 0 for both and either pic_order_cnt_lsb is different in value, or delta_pic_order_cnt_bottom is different in value.

• pic_order_cnt_type is 1 for both and either delta_pic_order_cnt [0] is different in value, or delta_pic_order_cnt [1] is different in value.

• IdrPicFlag is different in meaning.

• IdrPicFlag is 1 for both, and idr_pic_id is different in value.

(NOTE 2 - Some of the VAL NAL units in the backup encoded images or some NAL units other than the VCL (for example, the access restriction unit NAL unit) can also be used to detect the boundary between the access units and can therefore help in detecting the start of a new primary encoded image .)

Answer 2

The access block delimiter is optional as far as I know. And the h.264 stream often contains only one set of sequence parameters, at the beginning of the stream.

I do not think that you can safely find the boundaries of access blocks based only on the header information of the block. You will need to parse some information from the slice header. The frame_num parameter identifies the image and, therefore, the access module, but you may need even more information from the bitstream to parse it.

You can try your luck by assuming that one access module always contains only one fragment (nal unit types 1-5), although this is definitely not standard.