使用FFMPEG（LibavCodec）将H264视频解码的问题
https://stackoverflow.com/questions/3493742
-
28-09-2019 - |
italiano
english
français
española
中国
日本の
العربية
Deutsch
한국어
Português
Russian题
我将avcodeccontext的profile_idc，calter_idc，celtefta et freateData_size设置为SDP的profile-level-id-id et sprop-parameter-set。
我将编码切片，SPS，PPS和NAL_IDR_SLICE数据包的解码分开：
在里面：
uint8_t start_sequence [] = {0，0，1}; int size = recv（id_de_la_socket，（char*）rtproceive，65535,0）;
编码切片：
char *z = new char[size-16+sizeof(start_sequence)];
memcpy(z,&start_sequence,sizeof(start_sequence));
memcpy(z+sizeof(start_sequence),rtpReceive+16,size-16);
ConsumedBytes = avcodec_decode_video(codecContext,pFrame,&GotPicture,(uint8_t*)z,size-16+sizeof(start_sequence));
delete z;
结果：消耗> 0和gotpicture> 0（通常）
SPS和PPS：
相同的代码。结果：消耗> 0和gotPicture = 0
我认为这很正常
当我找到一个新的夫妇SP/PPS时，我将使用此数据包的有效载荷及其大小更新Extradata和外部_size。
nal_idr_slice：
NAL单位类型为28 => IDR框架，因此我尝试了两种解码方法
1）我将第一个片段（无RTP标头）带有序列0x000001，然后将其发送到avcodec_decode_video。然后，我将其余的片段发送到此功能。
2）我将第一个片段（无RTP标头）带有序列0x000001，并将其余的片段与其串联。我将此缓冲区发送到解码器。
在这两种情况下，我都没有错误（消耗> 0），但是我没有检测到框架（gotPicture = 0）...
问题是什么 ？
解决方案
在RTP中，所有H264 i-Frames（IDR）通常都碎裂。当您收到RTP时，首先必须跳过标头（通常的前12个字节），然后进入NAL单元（第一个有效负载字节）。如果NAL为28（1C），则表示有效负载代表一个H264 IDR（i-Frame）片段，并且您需要收集所有这些片段以重建H264 IDR（i-Frame）。
由于MTU和更大的IDR而发生碎片化。一个碎片看起来像这样：
片段开始= 1：
First byte: [ 3 NAL UNIT BITS | 5 FRAGMENT TYPE BITS]
Second byte: [ START BIT | END BIT | RESERVED BIT | 5 NAL UNIT BITS]
Other bytes: [... IDR FRAGMENT DATA...]
其他片段：
First byte: [ 3 NAL UNIT BITS | 5 FRAGMENT TYPE BITS]
Other bytes: [... IDR FRAGMENT DATA...]
要重建IDR，您必须收集此信息：
int fragment_type = Data[0] & 0x1F;
int nal_type = Data[1] & 0x1F;
int start_bit = Data[1] & 0x80;
int end_bit = Data[1] & 0x40;
如果 fragment_type == 28 然后有效载荷遵循它是IDR的一个片段。接下来的检查是 start_bit 设置（如果是），那么该片段是序列中的第一个。您可以使用它来重建IDR的NAL字节 (3 NAL UNIT BITS) 并将它们与第二个有效载荷字节中的最后5位相结合 (5 NAL UNIT BITS) 所以你会得到这样的字节 [3 NAL UNIT BITS | 5 NAL UNIT BITS]. 。然后将NAL字节首先写入一个清晰的缓冲区，并带有来自该片段的所有其他以下字节。请记住要以顺序跳过字节，因为它不是IDR的一部分，而是仅标识片段。
如果 start_bit 和 end_bit 是0，然后将有效载荷（跳过第一个有效载荷字节标识片段）到缓冲区。
如果start_bit为0，而end_bit为1，则意味着它是最后一个片段，您只需将其有效载荷（跳过第一个标识片段的字节）到缓冲区中，现在您已重建IDR即可。
如果您需要一些代码，请在评论中询问，我会发布它，但是我认为这很清楚如何做... =）
关于解码
今天，我想到了为什么您会在解码IDR时出现错误（我认为您已经重建了它很好）。您如何构建AVC解码器配置记录？您使用的LIB是否具有自动化？如果没有，您还没有听说过，继续阅读...
指定AVCDCR以允许解码器快速解析他们解码H264（AVC）视频流所需的所有数据。数据如下：
- profileIdc
- profileiop
- leveliDC
- SPS（序列参数集）
- PPS（图片参数集）
所有这些数据均在SDP中的RTSP会话中发送： profile-level-id 和 sprop-parameter-sets.
解码配置文件级ID
Prifile级别的ID字符串分为3个子字符串，每个2个字符长：
[PROFILE IDC][PROFILE IOP][LEVEL IDC]
每个子字符串代表一个字节 在base16中呢因此，如果配置文件IDC为28，则意味着基本10中的实际40。稍后，您将使用base10值来构建AVC解码器配置记录。
解码的Sprop参数集
Sprops是通常的2个字符串（可能是更多），逗号分开，并且 基本64编码呢您可以解码它们两个，但没有必要。您的工作只是将它们从base64字符串转换为字节数组以供以后使用。现在，您有2个字节阵列，第一个阵列US SPS，第二个是PPS。
构建AVCDCR
现在，您需要构建AVCDCR的所有内容，首先要制作新的干净缓冲区，现在按照此处解释的顺序在其中写下这些东西：
1-具有价值的字节 1 并代表版本
2-个人资料IDC字节
3- prifile iop byte
4-级IDC字节
5-带有值0xff的字节（Google AVC解码器配置记录以查看这是什么）
6-值0xe1的字节
7-短与SPS阵列长度的价值
8- SPS字节阵列
9-具有PPS数组数的字节（您可以在Sprop-Parameter-Set中拥有更多的字节）
10-缩短了以下PPS数组的长度
11 -PPS数组
解码视频流
现在，您有了字节数组，该阵列告诉解码器如何解码H264视频流。我相信，如果您的LIB不从SDP构建它，您需要这个...
其他提示
我不知道您的其余实现，但是您收到的“碎片”似乎是NAL单位。因此，每个人都可能需要nalu启动代码（00 00 01 或者 00 00 00 01）当您将bitstream发送到FFMPEG之前，请附加。
无论如何，您可能会发现H264 RTP数据包的RFC有用：
http://www.rfc-editor.org/rfc/rfc3984.txt
希望这可以帮助！
我有这个 @的实现 https://net7mma.codeplex.com/ 对于C＃，但到处都是相同的过程。
这是相关代码
/// <summary>
/// Implements Packetization and Depacketization of packets defined in <see href="https://tools.ietf.org/html/rfc6184">RFC6184</see>.
/// </summary>
public class RFC6184Frame : Rtp.RtpFrame
{
/// <summary>
/// Emulation Prevention
/// </summary>
static byte[] NalStart = { 0x00, 0x00, 0x01 };
public RFC6184Frame(byte payloadType) : base(payloadType) { }
public RFC6184Frame(Rtp.RtpFrame existing) : base(existing) { }
public RFC6184Frame(RFC6184Frame f) : this((Rtp.RtpFrame)f) { Buffer = f.Buffer; }
public System.IO.MemoryStream Buffer { get; set; }
/// <summary>
/// Creates any <see cref="Rtp.RtpPacket"/>'s required for the given nal
/// </summary>
/// <param name="nal">The nal</param>
/// <param name="mtu">The mtu</param>
public virtual void Packetize(byte[] nal, int mtu = 1500)
{
if (nal == null) return;
int nalLength = nal.Length;
int offset = 0;
if (nalLength >= mtu)
{
//Make a Fragment Indicator with start bit
byte[] FUI = new byte[] { (byte)(1 << 7), 0x00 };
bool marker = false;
while (offset < nalLength)
{
//Set the end bit if no more data remains
if (offset + mtu > nalLength)
{
FUI[0] |= (byte)(1 << 6);
marker = true;
}
else if (offset > 0) //For packets other than the start
{
//No Start, No End
FUI[0] = 0;
}
//Add the packet
Add(new Rtp.RtpPacket(2, false, false, marker, PayloadTypeByte, 0, SynchronizationSourceIdentifier, HighestSequenceNumber + 1, 0, FUI.Concat(nal.Skip(offset).Take(mtu)).ToArray()));
//Move the offset
offset += mtu;
}
} //Should check for first byte to be 1 - 23?
else Add(new Rtp.RtpPacket(2, false, false, true, PayloadTypeByte, 0, SynchronizationSourceIdentifier, HighestSequenceNumber + 1, 0, nal));
}
/// <summary>
/// Creates <see cref="Buffer"/> with a H.264 RBSP from the contained packets
/// </summary>
public virtual void Depacketize() { bool sps, pps, sei, slice, idr; Depacketize(out sps, out pps, out sei, out slice, out idr); }
/// <summary>
/// Parses all contained packets and writes any contained Nal Units in the RBSP to <see cref="Buffer"/>.
/// </summary>
/// <param name="containsSps">Indicates if a Sequence Parameter Set was found</param>
/// <param name="containsPps">Indicates if a Picture Parameter Set was found</param>
/// <param name="containsSei">Indicates if Supplementatal Encoder Information was found</param>
/// <param name="containsSlice">Indicates if a Slice was found</param>
/// <param name="isIdr">Indicates if a IDR Slice was found</param>
public virtual void Depacketize(out bool containsSps, out bool containsPps, out bool containsSei, out bool containsSlice, out bool isIdr)
{
containsSps = containsPps = containsSei = containsSlice = isIdr = false;
DisposeBuffer();
this.Buffer = new MemoryStream();
//Get all packets in the frame
foreach (Rtp.RtpPacket packet in m_Packets.Values.Distinct())
ProcessPacket(packet, out containsSps, out containsPps, out containsSei, out containsSlice, out isIdr);
//Order by DON?
this.Buffer.Position = 0;
}
/// <summary>
/// Depacketizes a single packet.
/// </summary>
/// <param name="packet"></param>
/// <param name="containsSps"></param>
/// <param name="containsPps"></param>
/// <param name="containsSei"></param>
/// <param name="containsSlice"></param>
/// <param name="isIdr"></param>
internal protected virtual void ProcessPacket(Rtp.RtpPacket packet, out bool containsSps, out bool containsPps, out bool containsSei, out bool containsSlice, out bool isIdr)
{
containsSps = containsPps = containsSei = containsSlice = isIdr = false;
//Starting at offset 0
int offset = 0;
//Obtain the data of the packet (without source list or padding)
byte[] packetData = packet.Coefficients.ToArray();
//Cache the length
int count = packetData.Length;
//Must have at least 2 bytes
if (count <= 2) return;
//Determine if the forbidden bit is set and the type of nal from the first byte
byte firstByte = packetData[offset];
//bool forbiddenZeroBit = ((firstByte & 0x80) >> 7) != 0;
byte nalUnitType = (byte)(firstByte & Common.Binary.FiveBitMaxValue);
//o The F bit MUST be cleared if all F bits of the aggregated NAL units are zero; otherwise, it MUST be set.
//if (forbiddenZeroBit && nalUnitType <= 23 && nalUnitType > 29) throw new InvalidOperationException("Forbidden Zero Bit is Set.");
//Determine what to do
switch (nalUnitType)
{
//Reserved - Ignore
case 0:
case 30:
case 31:
{
return;
}
case 24: //STAP - A
case 25: //STAP - B
case 26: //MTAP - 16
case 27: //MTAP - 24
{
//Move to Nal Data
++offset;
//Todo Determine if need to Order by DON first.
//EAT DON for ALL BUT STAP - A
if (nalUnitType != 24) offset += 2;
//Consume the rest of the data from the packet
while (offset < count)
{
//Determine the nal unit size which does not include the nal header
int tmp_nal_size = Common.Binary.Read16(packetData, offset, BitConverter.IsLittleEndian);
offset += 2;
//If the nal had data then write it
if (tmp_nal_size > 0)
{
//For DOND and TSOFFSET
switch (nalUnitType)
{
case 25:// MTAP - 16
{
//SKIP DOND and TSOFFSET
offset += 3;
goto default;
}
case 26:// MTAP - 24
{
//SKIP DOND and TSOFFSET
offset += 4;
goto default;
}
default:
{
//Read the nal header but don't move the offset
byte nalHeader = (byte)(packetData[offset] & Common.Binary.FiveBitMaxValue);
if (nalHeader > 5)
{
if (nalHeader == 6)
{
Buffer.WriteByte(0);
containsSei = true;
}
else if (nalHeader == 7)
{
Buffer.WriteByte(0);
containsPps = true;
}
else if (nalHeader == 8)
{
Buffer.WriteByte(0);
containsSps = true;
}
}
if (nalHeader == 1) containsSlice = true;
if (nalHeader == 5) isIdr = true;
//Done reading
break;
}
}
//Write the start code
Buffer.Write(NalStart, 0, 3);
//Write the nal header and data
Buffer.Write(packetData, offset, tmp_nal_size);
//Move the offset past the nal
offset += tmp_nal_size;
}
}
return;
}
case 28: //FU - A
case 29: //FU - B
{
/*
Informative note: When an FU-A occurs in interleaved mode, it
always follows an FU-B, which sets its DON.
* Informative note: If a transmitter wants to encapsulate a single
NAL unit per packet and transmit packets out of their decoding
order, STAP-B packet type can be used.
*/
//Need 2 bytes
if (count > 2)
{
//Read the Header
byte FUHeader = packetData[++offset];
bool Start = ((FUHeader & 0x80) >> 7) > 0;
//bool End = ((FUHeader & 0x40) >> 6) > 0;
//bool Receiver = (FUHeader & 0x20) != 0;
//if (Receiver) throw new InvalidOperationException("Receiver Bit Set");
//Move to data
++offset;
//Todo Determine if need to Order by DON first.
//DON Present in FU - B
if (nalUnitType == 29) offset += 2;
//Determine the fragment size
int fragment_size = count - offset;
//If the size was valid
if (fragment_size > 0)
{
//If the start bit was set
if (Start)
{
//Reconstruct the nal header
//Use the first 3 bits of the first byte and last 5 bites of the FU Header
byte nalHeader = (byte)((firstByte & 0xE0) | (FUHeader & Common.Binary.FiveBitMaxValue));
//Could have been SPS / PPS / SEI
if (nalHeader > 5)
{
if (nalHeader == 6)
{
Buffer.WriteByte(0);
containsSei = true;
}
else if (nalHeader == 7)
{
Buffer.WriteByte(0);
containsPps = true;
}
else if (nalHeader == 8)
{
Buffer.WriteByte(0);
containsSps = true;
}
}
if (nalHeader == 1) containsSlice = true;
if (nalHeader == 5) isIdr = true;
//Write the start code
Buffer.Write(NalStart, 0, 3);
//Write the re-construced header
Buffer.WriteByte(nalHeader);
}
//Write the data of the fragment.
Buffer.Write(packetData, offset, fragment_size);
}
}
return;
}
default:
{
// 6 SEI, 7 and 8 are SPS and PPS
if (nalUnitType > 5)
{
if (nalUnitType == 6)
{
Buffer.WriteByte(0);
containsSei = true;
}
else if (nalUnitType == 7)
{
Buffer.WriteByte(0);
containsPps = true;
}
else if (nalUnitType == 8)
{
Buffer.WriteByte(0);
containsSps = true;
}
}
if (nalUnitType == 1) containsSlice = true;
if (nalUnitType == 5) isIdr = true;
//Write the start code
Buffer.Write(NalStart, 0, 3);
//Write the nal heaer and data data
Buffer.Write(packetData, offset, count - offset);
return;
}
}
}
internal void DisposeBuffer()
{
if (Buffer != null)
{
Buffer.Dispose();
Buffer = null;
}
}
public override void Dispose()
{
if (Disposed) return;
base.Dispose();
DisposeBuffer();
}
//To go to an Image...
//Look for a SliceHeader in the Buffer
//Decode Macroblocks in Slice
//Convert Yuv to Rgb
}
还有其他各种RFC的实现，可以帮助使媒体在MEAVIRELENT或其他软件中播放，或者只是将其保存到磁盘上。
正在写入容器格式。
