基于三维深度信息的人体动作运动轨迹识别V11.docx

基于三维深度数据的人肢体动作运动轨迹识别Recognition of Human Body Action Movement Trajectory Based on the Three-dimensional Depth Data摘要：本文研究了传统人的肢体动作运动轨迹的识别技术，总结了传统识别技术的缺点，然后在此基础上将人的肢体动作的三维深度数据和三维环境下的运动历史图像（3DMHI）相结合，计算出运动历史图像的不变矩作为肢体动作的特征向量，应用于人的肢体动作运动轨迹的识别问题上来。文中详细阐述了基于三维深度数据的人的肢体运动轨迹识别技术的算法理论和实现方案，最后通过对比识别实验结果，验证了基于三维深度数据的人的肢体运动轨迹识别方法具有更好的鲁棒性和更好的准确性。关键词：机器视觉识别；人的肢体动作识别；三维运动历史图像1 引言随着机器人技术的迅猛发展，机器人视觉跟踪技术和自然人机交互技术也开始成为了机器人技术研究领域的重要研究方向。而人的肢体动作是一种自然且直观的人际交流模式，人的肢体动作运动轨迹的识别也理所当然地成为了实现新一代自然的人机交互界面中的不可缺少的一项关键技术，特别是针对一些残障人士，只需要通过人的肢体动作就能给轮椅和残障辅助设备下达指令，更显的尤为便利。之前针对人肢体动作运动轨迹识别的人机交互研究主要侧重于人体皮肤颜色建模，连续动态动作的基于图像属性的鲁棒性特征的提取，然而由于人肢体动作本身具有的多样性、多义性、以及时间和空间上的差异性等特点，因此传统的动作运动轨迹识别研究都有很大的局限性。本文就尝试将人体的三维深度数据引入到人的肢体运动轨迹的识别上面来，开创性的将传统方法拓展到三维空间，这样将减小环境光照、衣物遮挡和人体肤色与背景色的影响，使得计算机对人的肢体运动轨迹的识别更准确和更好的鲁棒性。2 问题描述人的肢体运动轨迹识别问题，即是将传感器实时捕获的人的肢体真实运动轨迹与预先定义好的样本轨迹相匹配的问题。传统方法是应用隐马尔科夫模型来进行真实运动轨迹与模板运动轨迹的匹配。图1.人的肢体动作二维图像基于二维图像的隐马尔科夫模型，如图1所示，通过隐马尔科夫模型进行样本轨迹的匹配，过程如图2所示。但是基于二维的识别有如下的几个难点：(1)光照：当光照发生变化时，人体的亮度信息会发生变化，传感器捕获的图像容易受到自然光和人工灯光的影响。(2)遮挡：由于在识别过程中，肢体运动轨迹可能会被静止的背景区域或者是眼镜、帽子等物体所遮挡，遮挡会产生识别信息的丢失，给识别的可靠性带来了很大的影响。(3)背景：在实际识别过程中，如果人体运动区域与背景区域的颜色、纹理或者形状相似，也会增大识别的难度。图2.基于二维图像的隐马尔科夫模型基于三维深度数据的隐马尔科夫模型，因为引入了三维深度数据，虽然可以有效地去除背景光源照度的影响，和不同目标人员肤色基准值的影响，识别过程不被光照、遮挡和背景等环境因素所影响，但是计算量大，训练效率低下，容易陷入局部最优值等问题，一直制约其在实时监控领域的应用。为了解决这些问题，这里我们将动作历史图像（MHI, Motion History Images）和人的肢体三维深度数据相结合，得到描述人的肢体动作的能量图像MEI，如图3所示，计算运动历史图像MHI的七个不变矩作为肢体动作特征向量，最后建立起肢体动作模板集合，也就是计算出这些肢体动作特征向量集的均值向量和协方差矩阵，识别阶段，通过Mahalanobis距离来衡量新输入的肢体动作与已知的肢体动作模板之间的相似性，只要计算出的Mahalanobis距离在规定的阈值范围之内都可以认为动作识别成功。这样既排除了光照、遮挡和背景等环境因素的影响，又很大程度上提高了识别过程的实时性和准确性。图3.基于三维深度数据的运动能量图像3 问题求解Problem solving3.1 肢体动作的三维运动历史图像表征Body movements characterized by three-dimensional motion history image本文应用将传统的基于二维图像的动作历史图像进行改进，使之与三维深度数据相结合，达到共同表征三维肢体动作信息的目的。In this paper, to characterize the 3D motion information ,application of traditional motion history image based on two-dimensional image is improved to combine the three-dimensional depth data .运动历史图像作为时间差分法的一个分支，时间差分法是将连续的图像序列中比较两个或者三个相邻帧对应像素点发生的相对变化，得到差分图像进而阈值化来提取图像中的运动区域。本文引入三维深度数据，所以采用改进后的差分方法如下：The motion history image is a branch of time difference method，the method is to get different image by a continuous image sequence comparisons between two or three adjacent pixels corresponding to the frame ，then to threshold to extract moving regions in an image.This paper introduced the 3D data, the improved differential method is as follows:Dx,y,z,n=Ix,y,z,n-1-2Ix,y,z,n+I(x,y,z,n+1)其中：Ix,y,z,n表示第n帧图像中三维空间位置x,y,z处的像素灰度值，Dx,y,z,n是连续3帧差分后的结果，代表了人体肢体动作发生变化的区域，将Dx,y,z,n阈值化如下：Among them，Ix,y,z,n represents the Pixel gray value in the position (x,y,z) in three dimensional space, Dx,y,z,n is the result of the three consecutive frames difference and it also represents human body movement changed area.The threshold of Dx,y,z,n is as follows:Bx,y,z,n=1 Dx,y,z,n0 otherwise其中是选择的阈值，值过低则不能有效抑制图像中的噪声，值过高则会抑制图像中有用的变化。 is the choice of the threshold. If its value is too low ,it can not effectively suppress noise in images,but if its value is too high,it will inhibit the image changes of useful.肢体运动的三维运动历史图像MHI的产生如下：Three dimensional motion history image of body movement(MHI) is made as follows:Hx,y,z,t= Bx,y,z,t=1max(0,Hx,y,z,t-1-1) otherwise运动历史图像MHI不仅反映了肢体动作的外在形状，也反映了肢体动作发生的方向和状态，在运动历史图像MHI中，每个像素的灰度值都与该位置肢体动作的持续运动时间成比例，最近发生的肢体动作姿态的灰度值最大，灰度值的变化体现了肢体动作运动发生的方向。The motion history image MHI reflects not only the external shape of body movements, but also reflects the direction and state of body movements . In the motion history image,the gray value of each pixel is in proportion with the duration of the body movement in the position . The recent body gestures have the maximum gray value. Gray value changes reflect the direction of the body movements. 图4.肢体运动的三维运动历史图像MHI Fig4 Three dimensional motion history image of body movement(MHI)3.2 肢体动作的运动历史图像不变矩计算Calculation of the invariant moment for the motion history image 这种基于肢体的三维运动历史图像MHI表征方法虽然简单快捷有效，但对观察点的位置角度比较敏感，为了克服这一缺点，所以本文选取了不变矩作为肢体动作的运动历史图像的特征向量。不变矩的方法是一种比较经典的图形图像特征提取方法，它的平移不变性、伸缩不变性和旋转不变性能很好地排除观察点的位置和角度的影响。Although the three-dimensional motion history image method based on MHI limb is simple and efficient, it is sensitive to the position of observation point . In order to overcome this shortcoming, this paper selects the invariant moments as eigenvector for the motion history image. The method of invariant moment is a classical method to extract image feature, its translation invariance, scaling invariance and rotation invariance properties well rule out the impact on the position and angle.我们得到肢体动作的三维运动历史图像MHI之后，为了计算它的不变矩，先将他分别在XY平面（如图5）、YZ平面（如图6）和XZ平面（如图7）进行投影。这样就可以得到对于同一个肢体动作的三维运动历史图像的三个视图，然后别对这三个主视图进行不变矩的计算。To get the invariant moment,after we get the three-dimensional motion history image, we projected it in the XY plane, YZ plane and XZ plane. This method can be for the three views of a 3D motion history image with one gesture. Then we did the calculation of moment invariants for the three main view. 图5.肢体运动的MHI的XY面投影 Fig5. XY surface projection of the MHI 图6.肢体运动的MHI的YZ面投影Fig6. YZ surface projection of the MHI 图7.肢体运动的MHI的XZ面投影Fig7. XZ surface projection of the MHI对一个尺寸为MN的数字图像fx,y，其中p+q阶矩mpq被定义为：For a size of M * N digital image f(x,y), the p+q order moment mpq is defined as follows:mpq=x=1Ny=1Mfx,yxpyq其中p,q=0,1,2,Among them, p,q=0,1,2,其p+q阶中心矩pq定义为： p+q order central moment pq is defined as follows:pq=x=1Ny=1Mfx,y(x-x)p(y-y)q其中x,y表示物体图像上的点，x,y是物体的质心：(x,y) represents the object image point, x,y is the object centroid:x=m10m00，y=m01m00。再通过零阶中心矩00对其余各阶中心矩进行归一化可以得到运动历史图像的归一化中心矩：Then through the normalizing of the central moment by the zero order central moments ,we can get the normalized center moment of the motion history image. pq=pq00r, r=p+q+22, p+q=2,3,4,HuMK利用二阶和三阶归一化中心矩的线性组合，得到了7个不变矩构成的不变矩组，对于图像的平移、旋转和缩放均保持不变，这个不变矩组如下：HuMK get seven invariant moments based on the linear combination of two order and three order normalized central moment.The image translation, rotation and scaling are unchanged and the invariant moments are as follows:M1=20+02M2=20-022+4112M3=30-3122+321-032M4=30+122+21+032M5=30-31230+1230+122-321+032+03-32103+2103+212-312+302M6=20-0230+122-21+032+41130+1221+03M7=321-0330+2130+122-321+032-(312-30)(03+21)03+212-3(12+30)2因为不变矩值较小，一般通过取绝对值的对数进行数据压缩，所以实际采用的矩值需要按照以下公式修正：Because the moment invariant is smaller, it is compressed by the absolute value of the logarithm and so the actual values need to be corrected in accordance with the following formula.Mk=logMk, k=1,2,3,4,5,6,7经过修正之后的不变矩组依然具有平移、旋转和缩放不变性。The invariant moments still has a translation, rotation and scaling invariance after amendment.通过对三个方向上的投影图像的计算，我们将得到一个37的特征值矩阵，这个特征值矩阵就是每一个肢体动作的运动历史图像的特征向量。Through the calculation of the projection images in three directions, we will get a 3 * 7 eigenvalue matrix.This eigenvalue matrix is the eigenvector for motion history volume.3.3 肢体动作的运动历史图像识别The motion history image recognition of body movements在识别过程中首先对人的肢体动作进行样本采集，建立一个训练模板库，以用于得到肢体动作的标准特征向量。In the process of recognition ,we collect the sample of human body movement first and build a training template library so that we can get the standard eigenvector.对每个相同的肢体动作，让多个不同的人参与反复执行多次，得到每个肢体动作的多组三维运动历史图像MHI，并从中计算出特征向量，再计算出这些特征向量的均值与协方差矩阵，建立起每个肢体动作的模板。For each of the same body movement, different people are involved in and they repeat several times. We get multiple groups of 3D motion history image for each movement and then get the mean of these eigenvectors and the covariance matrix.After doing this, each gesture template is established.然后对于新的肢体动作计算与标准动作模板之间的Mahalanobis距离，Mahalanobis距离的计算公式如下：For the Mahalanobis distance between new movement calculation and standard action template,it is defined as follows:2=f-rTc-1(f-r)其中是Mahalanobis距离，f是肢体动作运动历史图像的不变矩特征向量，r是已训练的特征向量的均值向量，c是已训练的特征向量集的协方差矩阵。 is Mahalanobis distance, f is the eigenvector of motion history image, r is the mean vector of the eigenvectors trained. c is the covariance matrix of the eigenvectors trained.识别过程中，可以利用经典的AdaBoost算法根据每个不变矩的阶数确定一个阈值，然后通过Mhalanobis距离来衡量新输入的肢体动作和已训练得到的肢体动作模板之间的相似性，只要计算出的Mahalanobis距离在规定的阈值范围之内都可以认为是匹配成功，如果匹配模板不止一个，则选择距离最小的那个作为成功匹配的模板。In the recognition process, an optimal threshold is determined according to the order of each moment invariants using the classical AdaBoost algorithm.Then we use Mhalanobis distance to measure similarity between the new input gestures and body movements which have been trained by template.If the Mhalanobis distance is within the scope of the provisions of the threshold,it can be considered as a successful match. If we get more than one template matching,we choose the minimum distance as the template.4 实验结果The experimental results4.1 数据预处理Data preprocessing在普通实验室环境下进行人的肢体动作运动轨迹识别实验。实验中，实验者保持身体正面向前，垂直于水平面，并且距离Kinect传感器1.2米到2米。本文对监测到的肢体动作进行了去抖动处理，记录前一帧判断所得中心位置数据，再和当前帧的中心位置数据进行比较，如果两者偏差在设定的阈值范围之内，就将当前帧视作小幅度内的抖动，继续显示前一帧的位置数据。This trajectory recognition experiments is did in normal laboratory environment.In the experiment,people should keep the body facing forward, perpendicular to the horizontal plane and be about 1.2 meters to 2 meters to the Kinect. In this paper, we debounce the physical movements monitored and record the center position data of the prior frame to compare with the center position data of the current frame. If the deviation is within the threshold range,we can show the position data of the prior frame to ignore the jitter of the current frame. 在识别肢体动作运动轨迹的时候如果直接采用真实运动的轨迹，则在运动开始和运动结束的时候会出现无效帧，而运动中间部分则是有效帧，为了去掉开始和结束部分的无效帧，本文采用了去抖动处理，在运动开始和结束部分运动位移会降低，这里直接作为起始点和禁止点进行处理。When using the real time trajectory,invalid frames will appear at the beginning and end of the movement. In order to remove the invalid part and get the the middle part,we debounce the physical movements,so the motion part displacement will decrease and all the frames can be used.实验中让4个人分别做出4种肢体动作，分别如图8、图9、图10和图11所示，每种动作反复执行10次，这样对于每种肢体动作就产生了40个样本，每个肢体动作持续5s到15s，图像大小为1200*900。In the experiment ，we make four people to do four kinds of body movements, as shown in figure 8, figure 9, figure 10 and Figure 11.Each action is repeated 10 times and it generate 40 samples for each body movement.Every movement last five to fifteen seconds and the image size is 1200*900.图8.动作A的三维运动历史图像Fig8.motion history image for motion A图9. 动作B的三维运动历史图像Fig9.motion history image for motion B图10. 动作C的三维运动历史图像Fig10.motion history image for motion C图11. 动作D的三维运动历史图像Fig11.motion history image for motion D选用每种肢体动作的前20个样本进行训练，得到肢体动作标准模板，其余的20个样本作为测试样本，分别用传统隐马尔科夫模型方法（3DHMM）和三维运动历史图像方法（3DMHI）进行识别实验。We choose the first 20 samples of each kind of movement for training to get the standard movement templates and we choose traditional hidden Markov model (3DHMM) and three-dimensional motion history image (3DMHI) to test the other 20 remaining samples.4.2 结果数据分析the results of data analysis为了验证对于环境光照因素的鲁棒性，这里对于不同光照条件下分别进行试验，表1是分别在普通光照环境下和弱光环境下每种动作的识别准确率。我们可以看出，在弱光条件下传统方法识别准确率急剧下降，结合肢体动作的三维深度数据的三维运动历史图像方法，即使在弱背景光源照度的情况下，系统依然能很好地捕获人体东部运动的轨迹，实验证明了新方法具有较好的鲁棒性。In order to verify the robustness of the ambient light factor, here we test for different lighting conditions. Table 1 are the recognition accuracy in normal light and the weak light for every action. We can see that recognition accuracy rate declined sharply using the traditional method in low light conditions. The 3D motion history image method with 3D depth data will capture the trajectory of human motion very well even in the case of weak background light. The experiment shows that the new method has a better robustness.表1.不同光照条件下动作识别率Fig1. Action recognition rate under different light conditions动作种类普通光环境识别率弱光环境识别率3DHMM3DMHI3DHMM3DMHI动作A0.850.910.600.89动作B0.840.900.540.88动作C0.800.890.550.86动作D0.830.890.530.87动作种类Movement types普通光环境识别率recognition rate under ordinary light environment弱光环境识别率recognition rate underweak light environment3DHMM3DMHI3DHMM3DMHIMovement A0.850.910.600.89Movement B0.840.900.540.88Movement C 0.800.890.550.86Movement D0.830.890.530.87而且对于不同体色和穿戴衣物条件下,上文方法依然可以做到自动检查，不用再根据目标人体的穿戴衣物或者肤色，手动调整体色基准值。Also for different body color and clothes conditions,the above method still can automatically check. There is no need to adjust the color reference value manually according to the target body clothes or skin color.5 结论本文通过将三维深度数据与运动历史图像相结合的方式来进行人的肢体动作运动轨迹识别，开创性的将传统方法拓展到三维空间，将目标对象的三维深度数据信息和运动历史图像相结合，既很好地克服了传统方法的识别方式的目标对象肤色影响、背景光源照度强弱影响、帽子口罩等服饰遮挡影响等种种缺点，又达到了实时监控环境下的实时性和准确性的要求。最后通过实验验证了该方法具有更准确的识别率、更快的实时响应速度和更好的鲁棒性。