EEEM011 Image and Video Compression Assessment
Guidelines
• You are assumed to possess elementary programming skills in Unix/C and the ability to work in a X-Windows workstation environment. If you don’t have such skills you ought to prepare yourselves (by self-study if necessary) before starting this assessment. Skills you gained through the lab assignment will be very useful to complete this assessment.
• You should take great care not to modify any of the attributes of the software which you will use for lab simulations such as the name, location or access rights of a file or a directory. The easiest way to comply with this requirement is to work in your own directory and copy (not move) everything you need from its original location to this directory. Then you can proceed to edit, modify and compile as necessary.
• Obvious exceptions to the above rule are executable files and large data files such as images or sequences which you will use as input. Output images (i.e. the results of your experiments, if applicable) should reside in your own directory.
• You should avoid copying any part of the assessment output under any circumstance. You should also bear in mind that if such cases are reported or found out your will be automatically allocated zero marks.
• Total Marks for this assessment will be 85%.
• Suggested time to spent on completing this assessment will be approximately 15- 20 hours.
Introduction
All work must be submitted no later than 4pm 18th August 2020. The report should be of reasonable content, if necessary, a brief summary of key concepts relevant to the design/experiment but avoiding material of introductory, tutorial or theoretical nature. Emphasis is placed on experimental ability, interpretation, analysis and presentation of results.
Assessment submission must always be electronic via SurreyLearn.
Part A: IMAGE CODING (Marks 30)
Preparation and warm-up – please review all lecture notes and refresh basic concepts in still image coding.
In this part of the assignment you will demonstrate your knowledge and practical understanding of the basic image coding concepts and techniques, including entropy and entropy coding, predictive coding, vector quantisation (VQ), transform coding, variable-length coding (Huffman and arithmetic coding), residual coding, image coding quality (PSNR) and rate distortion curve. Please document very clearly your workings and explain clearly all decisions and what facts or assumptions your decisions are based on. You do not need to use any software – all questions can be solved with just pen and paper. However, you can also use spreadsheet or write your own software, but you have to describe what your software does and upload all source code and executables together with your submission (simply zip all documents and code into a single file). Please explain clearly your thinking, as 15% of marks is allocated for this. Please note there may be several ways to solve the problems below, you do not always have to find the most optimal solution, but your solution must be logical, and it must be clear how you arrived to your solution.
A.1 Lossless image coding, entropy and entropy-based coding (total 15 Marks)
Your first task is to design a lossless image coding scheme, for a set of “Texture” images which are 8 by 8 pixels in size. Sample of 3 images are shown in Figure A1 below, you can assume that each of them represents perfectly the statistics of all images in the dataset. (For example, if a value 2 does not appear in your One of the tree images is allocated to you, based on you URN number. You can find which image is yours by taking the last two digits from your student number and calculate the reminder from division by 3. This should give you either 0, 1, or 2 – say if you obtain number 2, your will work with Image 2. Please write down your calculation on your report.
Task A.1 (i) – 2 points
Design a suitable lossless binary representation that can be used to represent pixel values (do not apply any entropy coding at this stage, but try to obtain as efficient binary representation
as possible). Show how to encode your image into the representation you just designed and state how many bits are required per pixel. How many bits are required to encode each image in the dataset?
Task A.1 (ii) – 3 points
Now design an entropy coding to reduce the number of bits required. Encode the first raw of the image using your scheme and calculate the number of bits required to encode all values in the first raw. Now decode the first raw and demonstrate that you indeed obtained a lossless encoding. Calculate the compression ratio for the first raw and for your method.
Task A.1 (iii) – 3 points
Compare the compression rate you achieved with the best lossless compression that can be achieved, assuming that neighboring pixels are statistically independent. Comment on the result.
Task A.1 (iv) – 3 points
Now assume that neighboring pixels are correlated horizontally in pairs, that is the first two pixels are correlated (pixel 1 and 2), then the next two are correlated (pixel 3 and 4) and so on. Calculate the theoretical limit on the lossless compression that can be achieved when you take this correlation into account.
Task A.1 (v) – 4 points
Based on the results achieved in the previous tasks, design the best lossless compression scheme that you can think of and comment on its performance versus the theoretical limit. To score points, you should improve on the compression performance achieved in A.1 (ii).
A.2 Lossy coding, prediction, (15 Marks)
Task A.2 (i) – 8 points
Your task is now to design a lossy compression scheme for your image. The pipeline will consist of two stages:
• Stage 1 –Design a suitable VQ-based coding, based on your observations from A.1 (iv) [do not apply entropy coding yet]
• Stage 2 – Design entropy coding for the VQ indices.
Now connect Stage 1 and Stage 2 to form a complete codec and analyse its performance. For the entire codec you designed, please calculate the PSNR and compression ratio achieved for your image.
Task A.2 (ii) – 3 points
Now change some elements in your design to change the compression ratio (image quality is also likely to change). Calculate the PSNR again and draw the rate distortion curve showing the two operating points for your codec. The rate distortion curve should be in terms of PSNR as a function of number of bits required to encode the image.
Task A.2 (ii) – 4 points
Please calculate residual errors for one operating point and design a suitable residual compression scheme to turn your codec into a lossless codec. Explain your design. Calculate the overall number of bits required to encode your image in a lossless way and compare with the lossless codec you designed in part A1. Draw conclusions. Check if you scheme works for one of the remaining images (if you worked with Image 1 you can check on Image 0 or Image 2).
Part B: VIDEO CODING (Marks 25) Part B: VIDEO CODING (Marks 25)
In this part of the assignment H.264/AVC and HEVC standards will be used. The test sequences and the codecs for H.264 and HEVC are available in https://goo.gl/fkY58C. For the 3D part, the resources are available in https://shorturl.at/fgvy8. Ensure that you make a local copy of these before beginning the assignment.
Preparation: Review H.264/AVC and HEVC standards
The video sequences provided for these questions are as follows: “Foreman.yuv”, “highway.yuv”, “poznan_street.yuv”
“kimono”, “cafe”
“basketballDrill”
All sequences are with a source bit depth of 8 bits and of 4:2:0 chroma sub-sampling format sampled at 30fps.
H.264 / AVC encoder details:
The encoder is lencod.exe, its syntax and available options can be checked by; % lencod.exe –h.
Go through the options in default configuration file and settings (“encoder.cfg”). An example of how to run this code is given below.
% lencod.exe -d encoder.cfg
HEVC encoder details:
The encoder is TAppEncoder.exe. Go through the options in default configuration file and settings (“encoder_lowdelay_P_main.cfg”). An example of how to run this code is given below.
% TAppEncoderStatic -c encoder_lowdelay_P_main.cfg -i “highway_cif.yuv” -wdt 352 -hgt 288 -fr 30 -f 10
In addition, go through the software manual provided for HEVC to get a better understanding about the software configurations for the encoder.
During the execution important statistics are sent to the standard output. Ensure that you navigate to corresponding folders and grant appropriate permissions for executable files prior to the beginning of assignments.
6
: CIF (352 x 288)
: HD (1920×1080) : WVGA (832×480)
1.1 Rate control (13 Marks)
(i) Select one of the CIF video sequences and encode 50 frames (as indicated by the Table 1) using H.264 encoder at a constant Quantization Parameter (QP) as defined by Table 1. Use group of picture (GOP) structure I-P-P-P. Plot PSNR and bit-rate as a function of frame number.
Second leftmost digit of student number (URN)
CIF Sequence
Quantization Parameter
Starting Frame
0
“Foreman.yuv”
20
5
1
“highway.yuv”
22
5
2
“poznan_street.yuv”
25
5
3
“Foreman.yuv”
28
10
4
“highway.yuv”
30
10
5
“poznan_street.yuv”
32
10
6
“Foreman.yuv”
35
20
7
“highway.yuv”
38
20
8
“poznan_street.yuv”
40
20
9
“highway.yuv”
15
30
Table 1: Information needed to encode the video
Example: if your student number (URN) is “6978015” the second most leftmost digit is “9”, so you will use second row in above table: Sequence to be used will be “highway.yuv” and QP=15, with starting frame 30.
(ii) Assuming the data rate 1.2 Mbits/s, using the rate controlling functionality available in the H.264 encoder, encode the same video sequence to obtain the selected bit rate, with parameter “RateControlEnable” set to 1. Set “InitialQP” to a value below the QP used in section (i). Plot PSNR and number of bits as a function of frame number in the corresponding plots for section (i).
(iii) Encode the same video selected in (i) using the same bit rate and initial QP used in (ii) with HEVC encoder. Disable the hierarchical bit allocation and LCU level rate control within the encoder. Compare the coding efficiency and rate control performances of the two codecs.
(iv)Encode the same video with the rate controlling enabled HEVC encoder with following configurations and discuss the performances and results.
a. Hierarchical bit allocation disabled with LCU level rate control enabled
7
b. Adaptive bit allocation with LCU level rate control enabled.
(v) With the aid of the PSNR and bit rate curves discuss the merits and demerits of using rate controlling.
(vi) Give an example scenario to illustrate the need for the rate controlling functionality.
[ Mark Distribution: (i) – 3 Marks, (ii) – 3 Marks, (iii) – 3 Marks, (iv) – 2 Marks, (v) – 1 Mark, (vi) – 1 Mark]
1.2 3D video coding
Three Dimensional (3D) video has the ability to give its viewers an added dimension of experience with depth perception. In 3D video, two disparate images are displayed at the same time for each eye of a viewer, known as stereoscopic views.
3D Video Representation Techniques
There are two popular types of 3D video representation. One is the left and right representation and the other is the colour plus depth representation.
In the left and right representation, two views for the two eyes are captured by two cameras and are sent to the display. The capturing cameras are kept apart at a distance equal to separation between the two eyes. At the display, both left and right views are displayed simultaneously. The next type of representation is known as the colour plus depth representation. In this type of 3D video, a color image and a per-pixel depth map is used to represent 3D video as shown in figure 1. The depth map is a luminance image, whereas the colour image contains luminance as well as chrominance information.
Fig. 1. Colour image and its corresponding depth map
The depth map signals the per-pixel depth of the corresponding color image. At the display, the depth map is used to generate two views for the two eyes by a process known as Depth Image Based Rendering.
8
(i) To view the process of Depth Image based rendering, run the Matlab script ‘ColorplusDepth2LeftandRight.m’ in the folder by the name DIBR. For this download the entire DIBR folder on to your machine and then change the runtime directory of Matlab to the DIBR folder. Provide the results for one frame and comment on the artificially synthesised left and right views.
3D Video Compression
3D video compression will deal with either encoding the two views separately in left and right video or encoding the colour image and depth map in case of the color plus depth representation.
(ii) Using the video encoder provided, compress 50 frames from the video streams View1.yuv and View2.yuv separately at 4 Quantization Parameter (QP) Settings using a codec of your choice (H.264 or H.265) and stating which codec is used. Select the first QP value as defined by Table 2. For the next 3 QPs, take by adding 10, 20, and 30 respectively to the first QP. For example, if your first QP is 13, your four QP values for the question will be: 13, 23, 33, and 43.
Second leftmost digit of student number (URN)
Quantization parameter
Starting Frame
0
9
10
1
10
10
2
11
10
3
12
20
4
13
20
5
14
20
6
15
30
7
16
30
8
17
30
9
18
40
Table 2: Information needed to encode the video
Fill the following table from the data obtained.
QP
View 1
View 2
Total Bit Rate
Avg. PSNR
Bit
PSNR_L
Bit
PSNR_R
Rate
Rate
9
Total Bit Rate = Bit Rate for View 1 + Bit Rate for View 2 Avg. PSNR = (PSNR_L+PSNR_R)/2 (for Y PSNR)
(iii) Using the Matlab script introduced in question (i) , synthesize left and right views for 50 frames corresponding to View1.yuv and View1_Depth.yuv.
(iv) Using the same video encoder, compress 50 frames from the video sequence View1_Depth.yuv at the same set of QPs used to compress View1.yuv.
(v) Synthesize left and right views for 50 frames from the compressed View1.yuv and compressed View1_Depth.yuv at each of the QP considered.
(vi) Obtain the PSNR between the left and right views synthesised from the compressed video streams (in Question 5) and the views synthesised from the uncompressed video streams (in Question 3).
N.B. yuvpsnr.m can be used to measure the PSNR between two views.
Total Bit Rate = Bit Rate for View 1 + Bit Rate for View1_Depth Avg. PSNR = (PSNR_L+PSNR_R)/2 (for Y PSNR)
(vii) Plot the Total bit rate Vs. Avg. PSNR on the same plot using the data obtained in Q.ii and Q.vi.
(viii) Discuss the merits and demerits of the two representation techniques investigated in this section.
[ Mark Distribution: (i) – 2 Marks, (ii) – 2 Marks, (iii) – 2 Marks, (iv) – 2 Marks, (v) – 1 Mark, (vi) – 2 Marks, (vii) – 1 Mark]
QP
Bit Rate For View1
Bit Rate for Depth
Synthesis Quality
Total Bit Rate
Avg. PSNR
PSNR_L
PSNR_R
10
Part C: Research on Emerging Media Systems: (30 Marks)
In this part of the assessment you are required to carry out a comprehensive research on one of the following topics. The students are expected to write a 10-12 page report and a 15-minute recorded presentation with a minimum of 15 slides. Both contributions should be submitted to Surrey Learn and marks will be allocated as per following guidelines.
Report – 15 Marks (Clarity, quality of the research and innovations proposed)
Presentation– 15 Marks (Clarity of slides and quality of contents and standard of presentation)
The topics for the presentation are as follows:
1. 5G for video streaming (Group 1)
Video streaming has many challenges including the latency and the bandwidth. The streaming quality further depends on the number of connected devices at a given instance. 5G, with its innovative technologies such as small cells and massive MIMO, can help solve many of the problems in video streaming. Discuss how 5G can effectively be used in future video applications. Your research shall focus one or more such applications.
2. Innovative business concepts for YouTube video (Group 2)
With over 1 million users and 5 billion videos on YouTube being watched everyday, the exploitation of YouTube content for research and business purposes is still at a preliminary stage. Your research shall focus on how YouTube can be used to generate new business concepts with the help of video compression, streaming and computer vision-based technologies. Your research shall refer one or more such applications.
3. Increasing Quality of Experience (QoE) of users with MPEG-DASH over CCN (Group 3)
Users of media content possess divergent capabilities, such as different devices and networks. HTTP AdaptiveStreaming(HAS)considersthesedifferencesandofferssamecontentsinmultiple encodings. Dynamic Adaptive Streaming over HTTP (DASH) is the standard developed by MPEG
11
for HAS. However, the user is limited to accept the hard-coded encoding parameters for segment generation in MPEG DASH, which can hinder user experience. Discuss the client-side and server- side optimizations used in DASH and implementation of DASH over CCN (Content Centric Network) to ensure high user experience.
4. Future prospects of Real-time high volume interactive media distribution (Group 4)
Internet has already revolutionized media consumption. The video streaming takes nearly 58% of total download. Netflix, an online entertainment video portal already consumes 15% of the download bandwidth. The user expectations with respect to the quality of media consumption keep escalating. In meeting the needs, human reacted with implementation of powerful computer clusters, massive storage and huge bandwidths. The state-of-the-art systems have already deployed media clouds for computing, media processing and content delivery. However, user expectations continue to increase. Discuss how would the technology develop and adapt to the increasing expectations of the users.
5. Quality degradation caused by projection techniques in omni-directional videos – (Group 5)
The omni-directional videos known as 3600 videos are being captured and streamed using state- of-the-art technologies. During the process, the omni-directional videos use various projection techniques to project the video onto a 2D surface before encoding it. The encoded video is then decoded and projected back to the spherical dimension for users to view. The projections cause additional degradation in the quality of the video. How should this issue be addressed? What are the alternatives for projection techniques. Discuss them in detail.
6. Versatile Video Coding (VVC)- (Group 6)
Envisioned as HEVC’s successor, Versatile Video Coding (VVC) is another collaboration between MPEG and VCEG set to be completed by end of 2020. VVC aims for a 30 to 50 percent bit-rate reduction for the same perceptual quality as HEVC, but with an estimated 10 times or more encoding complexity compared to its predecessor. Research on how VVC planned to achieve the expected coding gain and why complexities have been increased.
12
7. Versatile Video Coding (VVC)- (Group 7)
Envisioned as HEVC’s successor, Versatile Video Coding (VVC) is another collaboration between MPEG and VCEG set to be completed by end of 2020. VVC aims for a 30 to 50 percent bit-rate reduction for the same perceptual quality as HEVC, but with an estimated 10 times or more encoding complexity compared to its predecessor. Research on how VVC will be able to help emerging businesses.
8. AV1 Codec – (Group 8)
As you may have gathered, AV1 is a next-generation video codec that has been developed by AOMedia. Its founding members included Amazon, Cisco, Google, Intel, Microsoft, Mozilla and Netflix who decided to solidify the separate Daala, Thor, and VP10 codecs that they were working on. The goal of AV1 is to provide an open and royalty-free format that is optimized for the web and has the potential to scale based on modern demands. Research on the motivation for such a codec against emerging VVC and challenges such a codec faces from businesses.
9. Impact of COVID-19 on video coding, video communications and video systems – (Group 9)
The global video conferencing industry is expected to gain traction during and after the coronavirus outbreak. As the enterprises and government organizations are considering video conferencing as an ultimate solution to connect with remote workers, customers, and employees; and, at the same time, it prevents direct contact with the people. It also further help to reduce unnecessary time on traveling and saves lots of time in running businesses and governance. World will be changing after this pandemic. Research on COVID-19 impact on video coding, video communications and video systems and future businesses on these domains.
10. Impact of COVID-19 on creating new media based SMEs – (Group 10)
13
The global video conferencing industry is expected to gain traction during and after the coronavirus outbreak. As the enterprises and government organizations are considering video conferencing as an ultimate solution to connect with remote workers, customers, and employees; and, at the same time, it prevents direct contact with the people. It also further help to reduce unnecessary time on traveling and saves lots of time in running businesses and governance. World will be changing after this pandemic. Research on possible new business opportunities that may arise in media domain from COVID-19 pandemic.
Second leftmost digit of student number (URN)
Group
0
1
1
2
2
3
3
4
4
5
5
6
6
7
7
8
8
9
9
10
Table 3: Project Allocation
Example: if your student number (URN) is “6978015” the second most leftmost digit is “9”, so you need to research on Group 10 topic.
14