多模态预训练模型在OPPO端云场景的落地实践.pdf

DataFunCon#2023多模态预训练模型在OPPO端云场景的落地实践陈宸-OPPO研究院-高级算法工程师80387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-29Contents目录端侧图文检索技术研究图文生成&理解模型的应用优化文图生成模型的端侧轻量化80387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-29端侧图文检索技术研究80387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-29query1:和女朋友去迪士尼query2:山顶婚纱照一句话搜索的意义：一句话搜索的意义：用户体验：真正解决用户想什么就能搜什么想什么就能搜什么的痛点，“智慧搜图，搜你所想”。依托于大模型预训练大模型预训练技术技术，不再依赖于标签不再依赖于标签的迭代和扩展的迭代和扩展https:/ CLIPCLIP（OpenAIOpenAI）的图文理解能）的图文理解能力力。其二，高效搜索速度高效搜索速度。相比动辄翻上十几分钟半个小时的相册，现在无论从桌面下拉智慧搜索、打开相册、或是用语音助手，都只需要一句话就能搜到想要的图片，系统级地提升了找信息的效率。因此因此如何实现大模型在端侧的轻量化部署有重大的意义如何实现大模型在端侧的轻量化部署有重大的意义。大模型轻量化端侧技术落地的难点：大模型轻量化端侧技术落地的难点：1.压缩多模态大模型并确保精度确保精度。这并非简单用剪枝或量化等方法，直接压缩几倍模型大小就能搞定。毕竟对于端侧而言，算力有限的情况下，能部署的模型大小是往往只能达到大模型大模型的几十分之一的几十分之一。2.与算法模型升级相对应的，需要在端侧实现一个性能鲁棒的向量检索引擎，保证大模型下端向量检索引擎，保证大模型下端的工程性能的工程性能80387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-29端侧图文检索技术研究算法优化CLIP双塔模型ALBEF单流模型单双流多教师蒸馏架构损失函数检索引擎的计算分位两部分：1.离线部分：扫描相册所有图片，通过图片编码器将所有图片转成向量；并经过fp16量化存储成Nx200的fp矩阵2.在线部分：每次输入query，通过文本编码器将query转成向量；并经过fp16量化降低计算内存；最后通过矩阵相乘计算query向量跟所有图片的相似分数，并通过排序输出一个有序列表。Lei,Youbo,et al.MCAD:Multi-teacher Cross-modal Alignment Distillation for efficient image-text retrieval.arXiv preprint arXiv:2310.19654(2023).80387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-29端侧图文检索技术研究学术集效果各种蒸馏方法的效果对比Model nameimage modeltext modelfusion modelimage encoding timeretrieval timeparameter numbertestsetplatformCLIPVIT-L/1412-layer transformerdot product11.0ms32.5ms427.62Mfilckr5KV100 GPUALBEFVIT-B/166-layer transformer6-layer transformer7.6ms265ms(k=16)1945ms(k=128)3865ms(k=256)419.12Mfilckr5KV100 GPU自研小模型mobileVitV2-1.54-layer TinyBertdoc product3.8 ms14.1 ms25.9 Mfilckr5KV100 GPU自研小模型mobileVitV2-1.54-layer TinyBertdoc product17.3 ms14.6 ms25.9 Mfilckr5KMTK DX3大小模型的性能对比80387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-29端侧图文检索技术研究真实场景效果数据量：1111个用户真实相册共个用户真实相册共2 2万万+图片图片，手写5400+query5400+query数据分布：测试集R1R5R10MRmAP010.47280.6710.74950.63110.6080020.49560.7580.82510.69290.5306030.40190.56650.61080.52640.4889040.45320.68470.73890.62560.6048050.58430.7530.79520.71080.6428060.53230.68550.750.65590.5890070.350.52940.60880.49610.4771080.64170.80830.84170.76390.5943090.59650.68420.71930.66670.5622100.51210.70590.76470.66090.5441110.56540.74180.7810.69610.6336平均0.48480.67680.73600.63250.584080387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-29端侧图文检索技术研究细粒度优化Doveh,Sivan,et al.Teaching structured vision&language concepts to vision&language models.Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2023.细粒度属性词替换+hard negative sampling+LwF抗遗忘算法80387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-29文图生成&理解态模型的应用优化80387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-29图文生成&理解模型的应用优化中文文生图大模型继续预训练如何做高质量低成本的继续预训练如何对齐中文的语言文化如何提升生成图像的细节质量Parameter efficient adapterOrthogonal FinetuningQiu,Zeju,et al.Controlling text-to-image diffusion by orthogonal finetuning.Thirty-seventh Conference on Neural Information Processing Systems.2023.80387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-29中文语境迁移效果图文生成&理解模型的应用优化中文文生图大模型继续预训练收敛速度80387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-29图文生成&理解模型的应用优化中文文生图大模型继续预训练Finetune 模型鸳鸯双栖蝶双飞，满园春色惹人醉LoRAControlnetSSD1.3B小模型一只超级可爱的兔子穿着僧侣服装，肖像照，皮克斯动画SDXL inpainting青花瓷版的恐龙在长椅上西湖，塔和瀑布，日出江南，夏日湖畔的一个村庄一个漂亮的亚洲女孩，电影灯光3D电影，4k，高度细致，男人坐在马桶上读报带着墨镜的猫咪手里拿着剑，在恶魔城堡里，仙剑奇侠风格LatentCM80387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-29图文生成&理解模型的应用优化通用优化应用壁纸生成春节档热度top1春节档热度top3文生图模型+超分辨率生成2k高清壁纸80387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-29图文生成&理解模型的应用优化通用优化应用锁屏杂志生成文生图模型+微调LLAVA+LLM 生成图文并茂的杂志Liu,Haotian,et al.Visual instruction tuning.arXiv preprint arXiv:2304.08485(2023).80387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-29图文生成&理解模型的应用优化通用优化应用Zhang,Pan,et al.Internlm-xcomposer:A vision-language large model for advanced text-image comprehension and composition.arXiv preprint arXiv:2309.15112(2023).Internlm-xcomposer训练框架80387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-29图文生成&理解模型的应用优化垂域优化-人像垂域AI模型画人的几个问题：1.人脸人手等身体部位的崩坏。2.过于精致标准，渲染过度光滑，在质感上失真。3.细粒度属性和文本描述的不对齐。80387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-29图文生成&理解模型的应用优化垂域优化-人像垂域构建细粒度的人像属性数据80387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-29图文生成&理解模型的应用优化垂域优化-人像垂域U-Net中模块与图像中特征的对应关系，可用于指导LoRA微调的参数选择厚嘴唇薄嘴唇80387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-29图文生成&理解模型的应用优化垂域优化-人像垂域小鼻子大鼻子80387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-29图文生成&理解模型的应用优化垂域优化-人像垂域细眉毛粗眉毛80387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-29图文生成&理解模型的应用优化垂域优化-人像垂域垂域微调经验：1.大量数据粗调，增加模型对新概念的泛化能力2.少量高质量数据精调，提升模型的图片生成质量人脸修复逻辑：穿着华丽盔甲的玄幻战士与巨龙激战，雷霆与火焰交织在一起。（随机6张，无cherry-pick）80387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-29图文生成&理解模型的应用优化垂域优化-古风人像效果古道边一骑红尘客正巍然马背，身披白色斗篷，踏寂静落阿叶（随机6张，无cherry-pick）树丛中，翩翩少女，红衣绿裙，手提花伞，踏泥寻径，仿佛踏入了一幅画卷（随机6张，无cherry-pick）80387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-29图文生成&理解模型的应用优化垂域优化应用广告营销工具（内测版）80387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-29图文生成&理解模型的应用优化文字渲染-问题定义如何在文生图模型中渲染出正确的文字？Ma,Jian,et al.GlyphDraw:Learning to Draw Chinese Characters in Image Synthesis Models Coherently.arXiv preprint arXiv:2303.17870(2023).80387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-29图文生成&理解模型的应用优化文字渲染-算法GlyphDraw训练框架GlyphDraw推理框架数据集图文对数量文字数量中文数据集792k3.3M 字英文数据集1.9M2.3M wordsMa,Jian,et al.GlyphDraw:Learning to Draw Chinese Characters in Image Synthesis Models Coherently.arXiv preprint arXiv:2303.17870(2023).GlyphDraw数据集构建80387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-29图文生成&理解模型的应用优化文字渲染-客观效果Ma,Jian,et al.GlyphDraw:Learning to Draw Chinese Characters in Image Synthesis Models Coherently.arXiv preprint arXiv:2303.17870(2023).80387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-29图文生成&理解模型的应用优化文字渲染-主观效果Ma,Jian,et al.GlyphDraw:Learning to Draw Chinese Characters in Image Synthesis Models Coherently.arXiv preprint arXiv:2303.17870(2023).80387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-29图文生成&理解模型的应用优化个性化生成-问题定义Ma,Jian,et al.Subject-diffusion:Open domain personalized text-to-image generation without test-time fine-tuning.arXiv preprint arXiv:2307.11410(2023).如何使用一张参考图像快速生成新图片并平衡保真度和泛化性？80387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2023-11-2980387241 2