I also have this request.

1/ sorry, the company has no further plan for ffmpeg due to license issue (there's a workable downstream rkmppenc patch in buildroot sdk through, you can check my github rockchip_mirrors repo's buildroot branch for it)

2/ 3588's rga is kind of unusable for now, it has more limits(alignment and function) and new api that is broken in most of common usages. so i would prefer to use gpu if you know how to do it.

3/ enc and dec are running on different hardware modules, so i think so(might affect each other by ddr performane)

4/ don't know much about the enc, but 3588 has multiple dec hardware modules, so guessing multiple dec sessions would take advantage of that.

@herman would know better about the hardware and performance.

i would prefer to use gstreamer. the ffmpeg here is just for some special usage(official chrome/firefox/mpv)

Thanks for your info.

@JeffyCN Hi! I'm planning to add video transcoding support for our open source Jellyfin media server on rockchip SBC, in addition to our Intel, AMD and Nvidia graphics card support. We have maintained many custom patches here for our ffmpeg.

Seeing that librga is introduced with this commit, any plans to implement rkmpp's video encoder to ffmpeg? Or is there a staging patch in the mailing list?

rockchip-linux/mpp#157 There used to have a ffmpeg fork for encoder but it have been removed.

Also, to make librgba can be accepted by upstream, it should be used as a filter to chain with decoders and encoders. Such as scale_librga=w=1280:h=720:format=yuv420p.

We need at least hardware accelerated:

• decoding
• resizing
• subtitle blending
• encoding

It would be awesome If deinterlacing and HDR2SDR tonemapping are also supported via librga. Thus we can minimize the memory copy-back and the cpu usage. Any not supported hardware fix-functions can be compensated by using OpenCL, which can import memory directly from drm prime via Arm's extension.

I have to say that RK3588 is a beast when it comes to multi-way video tasks comparing with my already sold Raspberry pi 4. I've ordered a Rock 5b but it will still take several months before shipping. Hope it won't let me down.

Here are two more questions about RK3588:

• Are the H2645 encoding/decoding engines separate hardware that won't interfere each other too much when using both at the same time?
• Does 8k@30fps encoding/decoding == single session 1080p@480fps == 16 sessions 1080p@30fps?

@nyanmisaka Hi, I have tested the gstreamer with the official patch on RK3568, the decode and encode all run well. I also want to use the hardware transcoding for the Jellyfin Server running on the RK3568 chip. Is it possible to replace the ffmpeg with gstreamer in the Jellyfin code, so that the Jellyfin Server could transcode with the gstreamer? I consider the parameters of ffmpeg are so many and complex that the replacement would also be complex.

Where did you get the RK3568 board? I ordered the Rock 5 a few months ago, but it's scheduled to ship in October.

gstreamer is not as versatile as ffmpeg in terms of VOD streaming, so pure code replacement doesn't seem realistic. The decoder and encoder need to be implemented in ffmpeg for transcoding to work.

Where did you get the RK3568 board? I ordered the Rock 5 a few months ago, but it's scheduled to ship in October.

gstreamer is not as versatile as ffmpeg in terms of VOD streaming, so pure code replacement doesn't seem realistic. The decoder and encoder need to be implemented in ffmpeg for transcoding to work.

@nyanmisaka I got it from the taobao. The rockchip official don't support the ffmpeg on encoding. Do you have plan to implement in ffmpeg for transcoding to work ? If you have the plan, I could share my board.

i would prefer to use gstreamer. the ffmpeg here is just for some special usage(official chrome/firefox/mpv)

Where is gstreamer patches(?) or whatever for rockchip then?
How would one achieve full hardware transcoding of mpeg2ts mpeg2i -> h264 (i or p) possibly with a hardware resize in between, with gstreamer?

At least ffmpeg patches are easy to find and somewhat easy to build, but gstreamer looks like a complete unsupported mess.

How would one achieve full hardware transcoding of mpeg2ts mpeg2i -> h264 (i or p) possibly with a hardware resize in between, with gstreamer?

Here's a ffmpeg fork that contains scale_rga hw scaler.
h264 and hevc 8bit -> h264 8bit works fine but 10bit -> 8bit CSC is kind of broken since it's not finished yet.

I am using that. scale_rga (besides lacking documentation, so I have no idea what I'm doing) isn't working.
My input is mpeg2 1080i , which isn't getting deinterlaced, but even if it did, -vf=scale_rga:1280:720 or whatever just dumps out errors. That project has issue tracking disabled so I dont even know where I would go to report this.

rga_api version 1.7.0_[1]
 RgaBlit(1387) RGA_BLIT fail: Invalid argument
 RgaBlit(1388) RGA_BLIT fail: Invalid argument
fd-vir-phy-hnd-format[11, (nil), (nil), (nil), 0]
rect[0, 0, 1440, 1080, 1440, 1088, 2560, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
fd-vir-phy-hnd-format[14, (nil), (nil), (nil), 0]
rect[0, 0, 1280, 960, 1280, 960, 2560, 1843200]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
This output the user patamaters when rga call blit fail
[scale_rga @ 0x5562995820] RGA failed (code = -22)
free(): double free detected in tcache 2
Aborted

Well... I tried with a mpeg2 source and it fails too with the same logs but it works fine with h264 source.
File an issue to librga repo so as rockchip devs can handle this.

edit: You can also try updating librga to the latest 1.8.1 and see if it helps.

scale_rga filter will output hardware frame, if you want to pass it to a software encoder, you must append hwdownload,scale filter.

Here are examples https://github.com/jjm2473/ffmpeg-rk/wiki/%E7%BC%96%E8%A7%A3%E7%A0%81 .

And detail parameters decribe here https://github.com/jjm2473/ffmpeg-rk/wiki/RGA%E6%BB%A4%E9%95%9C .

scale_rga filter will output hardware frame, if you want to pass it to a software encoder, you must append

I'm aware of this, however, even with default settings and hardware frame coming from RK decoder and going into RK encoder, scale_rga just errors out with stuff similar to the post above.

scale_rga filter will output hardware frame, if you want to pass it to a software encoder, you must append

I'm aware of this, however, even with default settings and hardware frame coming from RK decoder and going into RK encoder, scale_rga just errors out with stuff similar to the post above.

What device you'r using? Did you check /dev/rga,/dev/dri existed?

@trollcop can you share the video file, and the command you'r using?

Just tried it again with a booted board. The above line works to encode (there are issues related to MPP library version, but that's unrelated).
RGA still fails with the following:

dmesg:

[  844.299672] rk_vcodec: vpu_service_ioctl:2149: error: unknown vpu service ioctl cmd 40086c01
[  844.334315] rga2: unknown ioctl cmd!
[  844.334808] rga2: req argument is inval
[  844.335256] rga2: req argument is inval

ffmpeg-rk:

[mpeg2_rkmpp @ 0x556be58060] Decoder noticed an info change (1440x1080), stride(1440x1088), format=0, mode=7
librga fail to get driver version! Legacy mode will be enabled.

rga_api version 1.7.0_[1]
 RgaBlit(1387) RGA_BLIT fail: Invalid argument
 RgaBlit(1388) RGA_BLIT fail: Invalid argument
fd-vir-phy-hnd-format[14, (nil), (nil), (nil), 0]
rect[0, 0, 1440, 1080, 1440, 1088, 2560, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
fd-vir-phy-hnd-format[17, (nil), (nil), (nil), 0]
rect[0, 0, 960, 720, 960, 720, 2560, 1036800]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
This output the user patamaters when rga call blit fail
[scale_rga @ 0x556bc3acf0] RGA failed (code = -22)
free(): double free detected in tcache 2
Aborted

I don't have a 3399 device to test.

rga_api version 1.7.0?

What the 1.7.0 from?
the newest rga is 1.3.1 ( https://github.com/JeffyCN/rockchip_mirrors/blob/linux-rga/version.h.template )

What the 1.7.0 from?

This is all from default images provided by firefly: https://en.t-firefly.com/doc/download/page/id/31.html
I used ubunto20

I have no idea, its preinstalled:

librga-dev is already the newest version (2.2.0-1).

librga-dev - Userspace interface to Rockchip RGA 2D accelerator
librga-dev-rk356x - Userspace interface to Rockchip RGA 2D accelerator
librga2 - Userspace interface to Rockchip RGA 2D accelerator
librga2-dbgsym - debug symbols for librga2
librga2-dbgsym-rk356x - debug symbols for librga2
librga2-rk356x - Userspace interface to Rockchip RGA 2D accelerator

Here is log from rk3568

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'test.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    creation_time   : 1970-01-01T00:00:00.000000Z
    encoder         : Lavf52.64.2
  Duration: 00:00:10.00, start: 0.000000, bitrate: 6836 kb/s
  Stream #0:0[0x1](und): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 6664 kb/s, 24 fps, 30 tbr, 30 tbn (default)
    Metadata:
      creation_time   : 1970-01-01T00:00:00.000000Z
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
  Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 166 kb/s (default)
    Metadata:
      creation_time   : 1970-01-01T00:00:00.000000Z
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (h264_rkmpp) -> h264 (h264_rkmpp))
Press [q] to stop, [?] for help
[h264_rkmpp @ 0x3869900] Decoder noticed an info change (1920x1080), stride(1920x1088), format=0
rga_api version 1.3.1_[11] (228dc67 build: 2022-11-12 07:32:02 base: )
[h264_rkmpp @ 0x38cdac0] Unsupport profile -99, force set to 100
[h264_rkmpp @ 0x38cdac0] Unsupport level -99, force set to 51
Output #0, matroska, to '/dev/null':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf59.8.100
  Stream #0:0(und): Video: h264 (H264 / 0x34363248), drm_prime(tv, progressive), 1280x720, q=2-31, 200 kb/s, 30 fps, 1k tbn (default)
    Metadata:
      creation_time   : 1970-01-01T00:00:00.000000Z
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
      encoder         : Lavc59.12.100 h264_rkmpp
frame=   83 fps= 77 q=-0.0 Lsize=      89kB time=00:00:03.46 bitrate= 210.9kbits/s dup=0 drop=1 speed=3.21x    
video:88kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 1.812635%

can you try librga-dev-rk356x and librga2-rk356x ?

I am based on this RGA version https://github.com/JeffyCN/rockchip_mirrors/tree/linux-rga#%E9%80%82%E7%94%A8%E8%8A%AF%E7%89%87%E5%B9%B3%E5%8F%B0

What do you mean "try" ? I have to rebuild ffmpeg, etc?
3399 is not 356x, different platform? I would rather use official package but I'm confused why it's not working.

Your sample file is H264 in, H264 out.
Here's example of the mpeg2ts I'm using as input.
https://drive.google.com/file/d/1-6IWMrTg0281np_nP_nwDbKk7QvzvO9n/view?usp=sharing

Here is prebuilt mpp,rga,ffmpeg for debian https://github.com/jjm2473/ffmpeg-rk/releases/tag/v1.0

And this is jellyfin docker image supports MPP, try it https://hub.docker.com/repository/docker/jjm2473/jellyfin-mpp

@trollcop Is the ffmpeg you are using comes from official package? I don't think the official ffmpeg has 'scale_rga' filter

@trollcop Is the ffmpeg you are using comes from official package? I don't think the official ffmpeg has 'scale_rga' filter

No, it's built from source from your repository: https://github.com/jjm2473/ffmpeg-rk/

that... prebuilt mpp,rga,ffmpeg release is packaged extremely strangely, I will take a look, but my build options are nearly identical to yours, so I suspect they both built similarly. RGA is included officially at
106008 Jan 7 2022 /usr/lib/aarch64-linux-gnu/librga.so.2.1.0

You should build libRGA from this tree https://github.com/JeffyCN/rockchip_mirrors/tree/linux-rga#%E9%80%82%E7%94%A8%E8%8A%AF%E7%89%87%E5%B9%B3%E5%8F%B0 , it supports RK3399 as document says, and ffmpeg-rk is based on that rga version.

If you can run docker, just test jellyfin-mpp, docker container is not depends on your libRGA version

@trollcop https://dev.t-firefly.com/thread-120601-1-1.html

@trollcop https://dev.t-firefly.com/thread-120601-1-1.html

Hey nice, thank you that fixed it! I guess I don't need to rebuild RGA after all.

Is there any reason https://github.com/jjm2473/ffmpeg-rk/ has issues disabled? There's other stuff I'd like to figure out (seems to be libmpp version differences too) but I feel bad spamming unrelated project with it.

@trollcop Oh, I have never noticed that, maybe github auto disable issues when forked, enable it now

@trollcop Oh, I have never noticed that, maybe github auto disable issues when forked, enable it now

Thank you for your help, I will move to the other project with further issues!

Folks, sorry for updating this thread, but @jjm2473 is unresponsive in his repo, and his repo is NOT working on my NanoPI R5C (RK3568B2).

I really wish to build ffmpeg to support this architecture, but it seems amazingly complex and undocumented.

ANY help would be greatly appreciated, as official friendlyarm email channels are also unresponsive. (3 weeks constantly pinging them without any single reply).

@trollcop: did you solved this on which device?

Thank you.

@danergo you should check your kernel config first.

check /proc/mpp_service, /dev/rga, /dev/dri or /dev/dma_heap existed on your system, if so MPP should work.

if you have truble with compile FFmpeg, try this docker image first https://hub.docker.com/r/jjm2473/jellyfin-mpp

@danergo I have add some english docs https://github.com/jjm2473/ffmpeg-rk/wiki#eng

@jjm2473: thank you.

How can I check the kernel config?

I have installed friendlyarms Debian Bullseye core from official website.

Those files are existing on my system.

Ffmpeg is building now without libx264 I'm gonna test this later.

Thank you for English versions, I appreciate it.

This is crazy.

I compiled with this:

./configure --prefix=/usr --pkg-config=pkg-config --enable-shared --enable-pthreads --enable-zlib --disable-doc --enable-debug --disable-lzma --disable-vaapi --disable-vdpau --disable-outdevs --enable-lto --enable-neon --enable-vfp --enable-gnutls --enable-libopus --enable-hardcoded-tables --enable-libmp3lame --enable-gpl --disable-libx264 --enable-version3 --enable-nonfree --enable-libdrm --enable-librga --enable-rkmpp --enable-libfdk-aac

--disable-libx264 !!

But when I run:

ffmpeg: error while loading shared libraries: libx264.so.155: cannot open shared object file: No such file or directory

./configure --prefix=/usr --pkg-config=pkg-config --enable-shared --enable-pthreads --enable-zlib --disable-doc --enable-debug --disable-lzma --disable-vaapi --disable-vdpau --disable-outdevs --enable-lto --enable-neon --enable-vfp --enable-gnutls --enable-libopus --enable-hardcoded-tables --enable-libmp3lame --enable-gpl --disable-libx264 --enable-version3 --enable-nonfree --enable-libdrm --enable-librga --enable-rkmpp --enable-libfdk-aac --disable-libopenh264 --disable-vaapi --disable-vdpau --disable-decoder=h264_v4l2m2m --disable-decoder=vp8_v4l2m2m --disable-decoder=mpeg2_v4l2m2m --disable-decoder=mpeg4_v4l2m2m

Still missing libx264.so.155 !!!

I had to manually copy my libx264.so.160 to libx264.so.155 to run your ffmpeg:

$ sudo ffmpeg -loglevel debug -benchmark -vcodec h264_rkmpp -i ../Big_Buck_Bunny_1080_10s_30MB.mp4 -map 0:v:0 -f null -
ffmpeg version N-104499-g646d5566ac Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --prefix=/usr --pkg-config=pkg-config --enable-shared --enable-pthreads --enable-zlib --disable-doc --enable-debug --disable-lzma --disable-vaapi --disable-vdpau --disable-outdevs --enable-lto --enable-neon --enable-vfp --enable-gnutls --enable-libopus --enable-hardcoded-tables --enable-libmp3lame --enable-gpl --disable-libx264 --enable-version3 --enable-nonfree --enable-libdrm --enable-librga --enable-rkmpp --enable-libfdk-aac --disable-libopenh264 --disable-vaapi --disable-vdpau --disable-decoder=h264_v4l2m2m --disable-decoder=vp8_v4l2m2m --disable-decoder=mpeg2_v4l2m2m --disable-decoder=mpeg4_v4l2m2m
  libavutil      57.  7.100 / 57.  7.100
  libavcodec     59. 12.100 / 59. 12.100
  libavformat    59.  8.100 / 59.  8.100
  libavdevice    59.  0.101 / 59.  0.101
  libavfilter     8. 16.100 /  8. 16.100
  libswscale      6.  1.100 /  6.  1.100
  libswresample   4.  0.100 /  4.  0.100
  libpostproc    56.  0.100 / 56.  0.100
Splitting the commandline.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument 'debug'.
Reading option '-benchmark' ... matched as option 'benchmark' (add timings for benchmarking) with argument '1'.
Reading option '-vcodec' ... matched as option 'vcodec' (force video codec ('copy' to copy stream)) with argument 'h264_rkmpp'.
Reading option '-i' ... matched as input url with argument '../Big_Buck_Bunny_1080_10s_30MB.mp4'.
Reading option '-map' ... matched as option 'map' (set input stream mapping) with argument '0:v:0'.
Reading option '-f' ... matched as option 'f' (force format) with argument 'null'.
Reading option '-' ... matched as output url.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option loglevel (set logging level) with argument debug.
Applying option benchmark (add timings for benchmarking) with argument 1.
Successfully parsed a group of options.
Parsing a group of options: input url ../Big_Buck_Bunny_1080_10s_30MB.mp4.
Applying option vcodec (force video codec ('copy' to copy stream)) with argument h264_rkmpp.
Successfully parsed a group of options.
Opening an input file: ../Big_Buck_Bunny_1080_10s_30MB.mp4.
[NULL @ 0x55665172b0] Opening '../Big_Buck_Bunny_1080_10s_30MB.mp4' for reading
[file @ 0x5566517720] Setting default whitelist 'file,crypto,data'
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55665172b0] Format mov,mp4,m4a,3gp,3g2,mj2 probed with size=2048 and score=100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55665172b0] ISO: File Type Major Brand: isom
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55665172b0] Unknown dref type 0x206c7275 size 12
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55665172b0] Processing st: 0, edit list 0 - media time: 1024, duration: 153600
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55665172b0] Offset DTS by 1024 to make first pts zero.
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55665172b0] Setting codecpar->delay to 2 for stream st: 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55665172b0] Before avformat_find_stream_info() pos: 4676 bytes read:32768 seeks:0 nb_streams:1
[h264 @ 0x5566517dd0] nal_unit_type: 7(SPS), nal_ref_idc: 3
[h264 @ 0x5566517dd0] nal_unit_type: 8(PPS), nal_ref_idc: 3
[h264 @ 0x5566517dd0] nal_unit_type: 7(SPS), nal_ref_idc: 3
[h264 @ 0x5566517dd0] nal_unit_type: 8(PPS), nal_ref_idc: 3
[h264 @ 0x5566517dd0] nal_unit_type: 6(SEI), nal_ref_idc: 0
[h264 @ 0x5566517dd0] nal_unit_type: 5(IDR), nal_ref_idc: 3
[h264 @ 0x5566517dd0] Format yuv420p chosen by get_format().
[h264 @ 0x5566517dd0] Reinit context to 1920x1088, pix_fmt: yuv420p
[h264 @ 0x5566517dd0] no picture
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55665172b0] All info found
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x55665172b0] After avformat_find_stream_info() pos: 1162504 bytes read:1162504 seeks:0 frames:1
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '../Big_Buck_Bunny_1080_10s_30MB.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    title           : Big Buck Bunny, Sunflower version
    artist          : Blender Foundation 2008, Janus Bager Kristensen 2013
    composer        : Sacha Goedegebure
    encoder         : Lavf57.63.100
    comment         : Creative Commons Attribution 3.0 - http://bbb3d.renderfarming.net
    genre           : Animation
  Duration: 00:00:10.00, start: 0.000000, bitrate: 24563 kb/s
  Stream #0:0[0x1](und), 1, 1/15360: Video: h264 (High), 1 reference frame (avc1 / 0x31637661), yuv420p(progressive, left), 1920x1080 (1920x1088) [SAR 1:1 DAR 16:9], 0/1, 24559 kb/s, 30 fps, 30 tbr, 15360 tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
Successfully opened the file.
Parsing a group of options: output url -.
Applying option map (set input stream mapping) with argument 0:v:0.
Applying option f (force format) with argument null.
Successfully parsed a group of options.
Opening an output file: -.
Successfully opened the file.
[h264_rkmpp @ 0x556653cb20] Initializing RKMPP decoder.
[h264_rkmpp @ 0x556653cb20] Failed to initialize MPP context (code = -1).
[h264_rkmpp @ 0x556653cb20] Failed to initialize RKMPP decoder.
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (h264_rkmpp) -> wrapped_avframe (native))
Error while opening decoder for input stream #0:0 : Unknown error occurred
bench: maxrss=22968kB
[AVIOContext @ 0x556651f7f0] Statistics: 1162504 bytes read, 0 seeks

Failed to initialize MPP context (code = -1). !!!

$ ls -all /dev/mpp_service
crw-rw-rw- 1 root root 241, 0 Aug  7  2022 /dev/mpp_service

$ ls -all /dev/rga
crw-rw-rw- 1 root root 10, 119 Aug  7  2022 /dev/rga

I don't have / want docker. I just want an ffmpeg can use this *** architecture's hw.

[h264_rkmpp @ 0x556653cb20] Initializing RKMPP decoder.

You realize, you're OPENING it with mpp decoder, right? Last I checked that was kinda buggy and needed RGA support.
You want to open with software decoder and encode with MPP.

If I use sw decode and HW encode, I'd lose a lot of resource IMHO. Is there any ffmpeg-fork which can use Rockchip's architecture fluid, and correct?