MULTIMEDIA SYSTEMS
MULTIMEDIA SYSTEMS
DESIGN – CS576
DESIGN – CS576
DR. PARAG HAVALDAR havaldar@usc.edu
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 1
YOUR INSTRUCTOR!
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 2
COURSE WEB PAGE ON DEN TIMES – Mon 6:40 pm – 10:10 pm
Class Location: Online for Fall 2020 Office Hours: After class Mondays*,
And by email appointment
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 3
Teaching Assistants:
Haiwei Chen
Email : haiweich@usc.edu Phone: (919)536-8476 Office Hours:
Tue – 3:00pm – 5:00pm
Thu – 3:00pm – 5:00pm
Location : Online, a webex/zoom link will be created
Gongshan Chen
gongshan@usc.edu
(213) 608-2607
Tue – 1:00pm – 3:00pm
Wed – 1:00pm – 3:00pm
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 4
COURSE DETAILS
PREREQUISITES
Good Programming Skills
Basic Math Skills
Helpful – Signal Processing, Graphics, Networks…
COURSE GRADE DECISION
One Term Exam – 35 % (Mon Nov 9th) Assignments, Project 60%
2 to 4 Theory/Programming Assignments (30%)
Project, done in a group of 2-3 students (30%) Guest Lecture Participation – 5%
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 5
BOOKS
List of Recommended Books:
Ze Nian Li , Mark S. Drew, Fundamentals of Multimedia, Prentice Hall, 2004 S.V. Raghavan, S.K. Tripathi, Networked Multimedia Systems: Concepts,
Architecture, and Design. Prentice Hall, 1998
F. Kuo, W. Effelsberg, J.J. Garcia-Luna-Aceves, Multimedia Communications:
Protocols and Applications. Prentice Hall PTR, 1998
David S Taubman, Micheal W. Marcellin, JPEG 2000 – Image Compression,
Fundamentals, Standards and Practice, Kluwer Academic Publishers 2002 Mohammed Ghanbari, Video Coding – An Introduction to Standard Codecs.
The Institution of Electrical Engineers (IEE), London, UK, 1999.
A. Puri, T. Chen (eds.), Multimedia Systems, Standards, and Networks. Marcel
Dekker, 2000
Ming-Ting Sun, Amy R. Reibman (eds.), Compressed Video over Networks.
Marcel Dekker, 2000
Marin Bosi and Riach E. Goldberg, Introduction to Digital Audio Coding and
Standards, Kluwer Academic Publishers 2003
Foley, Van Dam, Feiner, Hughes, Computer Graphics – Principles and
Practice, Second Edition. Addison-Wesley – 1990.
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 6
REQUIRED BOOK
Multimedia Systems – Algorithms, Standards and Industry Practices. -Parag Havaldar and Gerard Medioni
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 7
INTRODUCTION
INTRODUCTION
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 8
EXAMPLES AND !EXAMPLES
Reading a newspaper YES/NO
Describing a Picture to your friend NO
Video Game Playing and Multiplayer Games YES Riding a bicycle NO
Video Conferencing YES
Visiting your doctor YES/NO
Watching Television YES
Assembling a car in a garage NO
Listening to Radio NO/YES
Having a phone conversation NO
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 9
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 10
INTRODUCTION
Historical Perspective
When was the word multimedia created? Timeline of information creation and distribution
Multimedia Data and Information
Contains a mixture different types of media – text, images, video, audio, graphics
Definition and media types have been changing
Multimedia Systems Generation
Processing Storage
Distribution Rendering
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 11
HISTORICAL PERPESPECTIVE OF MEDIA
Age
Time and Era
Type of Information
Storage medium
Mode of Distribution
Prehistoric
15000BC
Sounds, Gestures Painting
Rocks, cave walls
–
Ancient
500 BC
Alphabets, Drawing
Invention of paper
People delivering messages,
Horse back
Middle Ages
400-1000 AD
Letters, Writing
Books
Beginning of a postal system
Renaissance
1300- 1800 AD
News, paintings, magazine
Books, Libraries
Printing press, steam engines, automobiles
Modern World
1900 AD
Morse Code, radio, Photographs, Movies
Film, Magnetic Tapes, Phonograph
Telegram service, wireless radio waves
Electronic
1950- 1980
Telephone, Television, Fax, Computers
Electronic memory, cassette tapes. LP records
Radio and TV Broadcasting, Satellite Communication
Digital
1980 to present day
Computers, Digital Video, Surround Sound
Hard Disks, CDROMs, DVDs, IPTV, Facebook,
Ethernet, Wireless Networks, Optical networks, Cell phone networks
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 12
COMPONENTS OF A MULTIMEDIA SYSTEM Capture devices –
Video camera, audio microphone, keyboard etc.
S/W Processing Elements –
S/W for content creation, compression, encryption etc.
Storage devices – CDROMS, Hard disks, Memories
Distribution network –
Ethernet (10-100 Mbs), ATM, Fiber Optics, Wireless
Processing devices
CPUs, Set Top boxes, workstations, DSP hardware
Display / Rendering devices –
HiRes Monitors, Speakers, HDTV, Projectors, Printers
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 13
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 14
MEDIA TYPES – AN “IN”COMPLETE TAXONOMY
Current Media Types
Text – Hypertext
Images – Static & Dynamic
Audio – Speech, Music
Video – Movies, Documentaries
2D Graphics – Vector Graphics, 2D Sprites 3D Graphics – Games
Future Media Types
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 15
TEXT
This is a line of text to explain that text does convey
information!
Hypertext
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 16
IMAGES – GRAY & COLOR
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 17
IMAGES – FAX
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 18
IMAGES – STITCHED TOGETHER
Mosaic example Panorama example
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 19
IMAGES – STEREO
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 20
VIDEO
How do you describe video ?
creamedgates.mpa
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 21
AUDIO
Audio Media is of various kinds
CD Quality (uncompressed)
Mp3 compressed audio takefive.mp3 Speech – JFKCELP8kHz.wav
MIDI example – furelise.mid
How do you describe audio ?
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 22
2D GRAPHICS
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 23
3D GRAPHICS
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 24
MEDIA TYPES – CONCLUSION
We have seen a lot of media types that are currently used; there may be others in future depending on
Need for information
Capture device technologies
Rendering devices and technologies
Need for standards
Many media types, having many formats
Information has to be easily interchanged and
displayed
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 25
BREAK
STRECTH YOUR LEGS
STRECTH YOUR LEGS
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 26
EXAMPLES IN MULTIMEDIA
ImmersiveMedia – Interactive Video Augmented/Virtual Reality
Industry Example, Oculus VR, Holoportation Deep Learning with media – Real time Face Replace,
Synthesizing Obama, GAUGAN
Movies, Animation & VFX Pipeline
Performance capture technologies
Display Technologies- Auto stereoscopic Displays
InFORM – Make the digital physical
Research Progress – the Visual Microphone, Cocktail Party,
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 27
INHERENT QUALITIES OF MULTIMEDIA
Digital Always
Mixture of different media types Interactive
Multimedia Data is huge
Real Time Issues
Synchronization Issues
– Intra media Time dependencies
Inter media Time dependencies
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 28
BACK TO EXAMPLES AGAIN
Reading a newspaper .
Describing a Picture to your friend .
Video Game Playing and Multiplayer Game . Riding a bicycle .
Video Conferencing .
Visiting your doctor .
Watching Television .
Assembling a car in a garage .
Listening to Radio .
Having a phone conversation .
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 29
MULTIMEDIA CLASSIFICATIONS
Static Vs Dynamic
Type – Real time Vs Orchestrated
Linear Vs Nonlinear
Person-to-Person Vs Person-to-Machine
Distribution
Single user (CD ROM and Computer)
Peer to Peer (Teleconferencing between two
addresses)
Peer to Multi Peer (Internet, Corporate Networks)
Broadcast (Cable Network)
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 30
FORCES DRIVING THE “MULTIMEDIA REVOLUTION”
Digitization of all information – text/audio/video documents, libraries, distributed nature of informtion
Evolution of data networks and communication standards with increasing availability of bandwidth on demand
Hardware – Faster processors, large capacity storage devices, smaller mobile computing devices.
Sofware – New algorithms, structures that deal with distributed queries
Better User Interfaces – hand held devices, sensors, displays Digitization of virtually everything ….
Ubiquitous access of information
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 31
TECHNOLOGICAL ASPECTS Organizing, Storage and Retrieval, Distribution, Playback
Techniques for compression Algorithms
Standards
Communications Aspects
Downloading and Streaming
Synchronization
Layering of Signals
QoS – traffic, delays, packet loss, sync
Access to multimedia signals
“natural” spoken language queries media conversion tools
multimodal user interface
distributed and collaborative access
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 32
COURSE MAP Lecture 1 (Chapter 1) –
Introduction to Multimedia and Course Map
Lecture 2 (Chapters 2 & 3) –
Data Acquisition and Media Processing Basics
Signal Processing Basics, time/spatial and frequency domain analysis,
Sampling & quantization – aliasing effects. Bit rate
Representational aspects of media – images, audio, video, graphics.
Lecture 3 (Chapter 4) –
Fundamentals of Color Theory & Displays Color and Color Perception
Camera Color Calibration using CMFs
Color Spaces – types, uses and applications
Color Displays, Monitors and Calibration – CRT, LCD, LED, OLED Color Quantization
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 33
Lecture 4 (Chapter 6) –
Information Theory and Generic Compression Techniques Coding Theory
Lossless techniques – Huffman, Arithmetic, Lempel Zev etc.
Lossy techniques – Predictive Coding, Transform Coding, Wavelets, Hybrid etc. Introduction to perceptual analysis
Lecture 5 (Chapter 7 and Research Papers) –
Media Compression – Images Representation Issues
Generic Image Compression algorithms – DCT, Wavelets, Fractals Fourier Representation, DCT & Wavelet theory
Standards – JPEG, JPEG2000, GIF etc.
Image Dithering, Compressive Sensing
Lecture 6 (Chapter 8 and Research Papers) –
Media Compression – Video
Issues in representation, spatial & temporal domain.
Generic compression algorithms and analysis (MPEG techniques)
MPEG standards – MPEG1, MPEG2, MPEG4: various video Profiles and AVC ITU standards – H.261, H.263, H.264, H.265 (HEVC)
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 34
Lecture 7 (Chapter 9 and Research Papers) –
Audio Processing
Representation and capture Issues
Generic compression algorithms and analysis (MPEG techniques) MPEG (1, 2, 4) standards – mp3, AAC, CELP
ITU standards – G.72x
Dolby AC3, AC5
Surround Sound, THX, Spatial Audio, Dolby Atmos
Lecture 8 (Chapter 10 and Research Papers) –
2D/3D Graphics Content Creation, 3D Compression &
Recent Trends in 3D Representation issues
Geometry Transformations in 2D and 3D
Rendering Pipeline – modeling, lighting, transformations, scan line Animation techniques
Special Effects & Gaming Technology discussions
3D Compression
Modern applications of computer graphics and computer vision – image based rendering, panoramic images and cameras
3DTV, stereoscopic content
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 35
Lecture 9 – Deep Learning in Media, Basics and Recent
Trends
Overview of Machine Learning/Deep Learning
Deep Learning Networks with Media
Traditional Networks architectures used – CNN, R-CNN, RNN, AutoEncoders, GANs
Examples of recent progress in media and animation Programming setup and project
LECTURE 10 – Media Security & Digital Rights Management Watermarking – definition, generic schemes, specific to MPEG world
Encryption – requirements, common rules, encryption related to MPEG world DVD Encryption rules
Watermarking/Encryption Architectures – digital movie distribution pipeline, session based architectures
Term Exam – Monday Nov 9th
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 36
Lectures there after to be held at ICT Open House at ICT
Real Case studies from the Industry
Blending CG and Real Imagery – geometrically & photometrically Digital Characters and Virtual Actors
Light Stage Data Acquisition
Advanced Technologies to create virtual actors
Natural Language Queries
Virtual and Augmented Reality
Multimodal Media Analysis
Multimedia Metadata, MPEG7 and Metadata Management Standards of Metadata – MXF, TV-Anytime, Dublin Core, Examples of multimedia databases
Game Pipelines and In Game workflows
Other Relevant Areas to be covered if we have time
MPEG4 & Applications, MPEG 21 Frameworks Introduction as a object oriented framework
Representation Features – Audio Visual Objects, Scene Graphs Compositional, Synchronization & Delivery Features Compression features of each media object representation Applications around MPEG4, MPEG DASH
MPEG 21 – distribution of content over a variety of networks
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 37
MPEG 21
MPEG 21 – distribution of content over a variety of networks
Digital Items and Digital Item Transactions across different networks
Multimedia Databases
Multimedia Frameworks and Industry wide Multimedia Deployments Current Trends – MPEG DASH, HEVC, DCI
Industry Outlook & Job Scenarios, Discussions on the future of multimedia …
Dr. Parag Havaldar – CS576 Lecture 1 8/25/2020 Page 38