Video for Linux Two - Video Effects Specification

Video for Linux Two is a set of APIs and standards for handling video devices on Linux. Video for Linux Two is a replacement for the Video for Linux API that comes with the kernel.

This document is the specification for Video for Linux Two video effects devices. This is a companion document to the V4L2 Devices document.
 

Devices

A V4L2 video effects device is a device that can do image effects, filtering, or combining of two or more images or image streams, for example video transitions or wipes. Applications send data to be processed and receive the result data either with read() and write() functions, or through memory-mapped buffers. Using memory-mapped buffers is essential if there are two input images.

Refer to the Device document for the device file names used by effects devices.

Multiple Opens per Device

Drivers can support non-I/O opens in addition to an effect context, or multiple opens where each open is an independent effect device context, as makes sense for the device.
 
 

Query Capabilities - VIDIOC_QUERYCAP

This ioctl call is used to obtain the capability information for the device. See the Device document.

Codec drivers have a type of V4L2_TYPE_FX. Effects devices will probably support the read() and write() calls indicated by the V4L2_FLAG_READ and V4L2_FLAG_WRITE flags. An effects driver can also set V4L2_FLAG_STREAMING if it supports I/O through memory-mapped buffers, and V4L2_FLAG_SELECT if it supports the select() call.
 
 

Enumerating Supported Effects - VIDIOC_ENUMFX

An application can query the list of supported image effects using the VIDIOC_ENUMFX ioctl. The application fills in the index field of a struct v4l2_fxdesc, and then passes it to the VIDIOC_ENUMFX ioctl, which fills in the rest of the fields. The application should use index values from 0 on up; the ioctl will return an error when the index goes out of range.
 

Selecting an Effect - VIDIOC_G_EFFECT, VIDIOC_S_EFFECT

Select the desired effect by calling the VIDIOC_S_EFFECT ioctl with the index of the effect as the parameter. The VIDIOC_G_EFFECT ioctl takes a pointer to an integer, and returns the currently selected effect.
 
 

Enumerating Supported Image Formats - VIDIOC_ENUM_PIXFMT

An application can query the list of supported image formats with the VIDIOC_ENUM_PIXFMT ioctl. The application fills in the index field of a struct v4l2_fmtdesc, and then passes it to the VIDIOC_ENUM_PIXFMT ioctl, which fills in the rest of the fields. The application should use index values from 0 on up; the ioctl will return an error when the index goes out of range.

The struct v4l2_fmtdesc structure is defined in the Capture spec.
 
 

Selecting The Image Format - VIDIOC_G_FMT, VIDIOC_S_FMT

Refer to the Device document for information regarding the video image format structure, struct v4l2_format. Use V4L2_BUF_TYPE_EFFECTSIN for the type field of v4l2_format. Typically, all streams use the same format, so setting the format on one stream sets it on all streams.

These get and set the format of the images which are going to be processed. All input and output images are the same format. These ioctls work as described in the Capture spec.

Remember that the input data is the data input to the effect, and the output data is the processed data coming out of the effect, which means, paradoxically, you write the input data to the device and read the output data from the device.

If you are planning on using memory mapped device buffers, know that VIDIOC_S_FMT is illegal if any device image buffers are currently memory mapped. VIDIOC_S_FMT undoes the effect of VIDIOC_REQBUFS.
 
 

Memory-Mapping Device Buffers - VIDIOC_REQBUFS, VIDIOC_QUERYBUF

See the Device document for instructions on allocating memory-mapped device buffers. For video data input and output buffers, use V4L2_BUF_TYPE_EFFECTSIN (, V4L2_BUF_TYPE_EFFECTSIN2, ...) and V4L2_BUF_TYPE_EFFECTSOUT respectively.
 
 

Writing and Reading Images - write(), read()

Convert images by sending them to the driver with the write() call, and get the converted images by reading them with read(). Each call to write() or read() should be used to transfer a complete frame. The application may have to write several frames before the first frame is available for reading because a filter or effect may involve combining or processing several images in sequence. The read() function will return the error EPERM if the read cannot complete because more input frames are needed.

If an application is both writing and reading, it should use non-blocking I/O calls to prevent it from deadlocking. It is possible for one program to write data while another reads the output. Non-blocking write() will return the error EAGAIN if a frame or frames need to be read before more data can be accepted. Non-blocking read() will return the error EAGAIN if a conversion is in progress, but not yet complete. Note that this is a different condition from the more-input-frames-needed error condition mentioned in the previous paragraph, which returns the EPERM code. Read() and write() do not work if memory-mapped I/O is in use.
 
 

I/O Through Pre-Allocated Buffers - VIDIOC_QBUF, VIDIOC_DQBUF, VIDIOC_STREAMON, VIDIOC_STREAMOFF

This mode is supported if the V4L2_FLAG_STREAMING flag is set in the struct v4l2_capability.

This mode uses memory-mapped buffers for transferring video data to and from the driver. First set up the image format, and allocate the buffers as described in the section titled Memory-Mapping Device Buffers in the Device document. You will need at least two sets of buffers, one set for the original data, another for the converted data. Use buffer type V4L2_BUF_TYPE_EFFECTSIN (, V4L2_BUF_TYPE_EFFECTSIN2, ...) for the original data buffers and V4L2_BUF_TYPE_EFFECTSOUT for the converted data buffers.

Streaming data to and from the device with these ioctls works as described in the Codec spec. Except that an effect may have more than one input image stream.
 
 

Waiting for Frames Using select()

The driver supports the select() call on its file descriptors if the V4L2_FLAG_SELECT flag is set in the struct v4l2_capability. If streaming mode is off, select() returns when there is data ready to be read with the read() call and/or if the driver is ready to accept new input data through the write() call. If streaming mode is on, select() returns when there is a buffer ready to be dequeued. The caller should be sure there is a buffer queued first.
 
 

Controls

See the Controls section of the Device document. Video effects typically have several user-tweakable controls. These controls will be exposed through the V4L2 controls ioctls. In general, the set of controls will be different for each effect, so the application must select the effect before enumerating the controls. The controls field of the struct v4l2_fxdesc object for the effect indicates how many controls the effect has. To enumerate the controls, call VIDIOC_QUERYCTRL with id values starting from V4L2_CID_EFFECT_BASE, and counting up until the indicated number of valid control id's have been identified.

It is normal when streaming data to an effects device to change the effect controls' values on a frame-by-frame basis on the fly. To do that you use an effect control stream. The effect control stream works like the data streams, but the buffers contain control information instead of pictures. The buffer type is V4L2_BUF_TYPE_FXCONTROL. The effect control stream runs in parallel with the effects-in stream(s). The format of the data in the fxcontrol buffer consists of a list of control id-value pairs, or stated another way the buffer contains an array of struct v4l2_fxcontrol structures. There may be any number of control values specified in a buffer. The bytesused field of the v4l2_buffer will indicate how many controls are in the buffer; the bytesused will be sizeof(struct v4l2_fxcontrol) * number of controls.
 

Non-I/O Opens

Non-I/O opens can query device properties and operate controls, but cannot perform I/O operations or change related driver settings such as the image format.