Packetized elementary stream

Packetized Elementary Stream (PES) is a specification in the MPEG-2 Part 1 (Systems) (ISO/IEC 13818-1) and ITU-T H.222.0 that defines carrying of elementary streams (usually the output of an audio or video encoder) in packets within MPEG program streams and MPEG transport streams. The elementary stream is packetized by encapsulating sequential data bytes from the elementary stream inside PES packet headers.

A typical method of transmitting elementary stream data from a video or audio encoder is to first create PES packets from the elementary stream data and then to encapsulate these PES packets inside Transport Stream (TS) packets or Program Stream (PS) packets. The TS packets can then be multiplexed and transmitted using broadcasting techniques, such as those used in an ATSC and DVB.

Transport Streams and Program Streams are each logically constructed from PES packets. PES packets shall be used to convert between Transport Streams and Program Streams. In some cases the PES packets need not be modified when performing such conversions. PES packets may be much larger than the size of a Transport Stream packet.

While above flags indicate that values are appended into variable length optional fields, they are not just simply written out. For example, PTS (and DTS) is expanded from 33 bits to 5 bytes (40 bits). If only PTS is present, this is done by catenating 0010b, most significant 3 bits from PTS, 1, following next 15 bits, 1, rest 15 bits and 1. If both PTS and DTS are present, first 4 bits are 0011 and first 4 bits for DTS are 0001. Other appended bytes have similar but different encoding.

...
Wikipedia