Expanded Perception and Interaction Centre
Applied Hybrid Analytics and Computer Vision Application for Research and Industry
Tomasz Bednarz
Director @ EPICentre, UNSW | Simulation & Modelling CCC Lead, CSIRO
+ collaborators from CSIRO, UNSW, QUT, Kyushu University, JCU, ACEMS
Power of Simulation
Source: https://twitter.com/northmantrader/status/905143410927034369
Role of visualisation
• Visual Analytics and Automated/Statistical Methods are complementary approaches for Data Analysis.
• Automated/Statistical Methods approaches work best for problems
• complete information
• well-defined questions.
• Visual Analytics benefits from Human Traits that are difficult to replicate:
• Pattern Recognition
• Contextual Knowledge
• Intuitive Intelligence
• Prediction
3 | Visualisation Grammar | EPICentre
“Visualization …is necessary but not sufficient”
– Jeffrey Sheer
“The purpose of visualisation is insight, not pictures.” – Ben Shneiderman
“The purpose of computing is insight, not numbers.” – Richard Hamming
“Realise that everything connects to everything else.”
– Leonardo da Vinci
Milgram’s Reality Virtuality Continuum
The area between the completely real and completely virtual, consists of both augmented reality, where the virtual augments the real, and augmented virtuality, where the real augmented the virtual.
P. Milgram and A.F. Kishino, Taxonomy of Mixed Reality Visual Displays, IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994.
Immersive and Big Data Visualisation
@tomaszbednarz
Firing the Imagination
8 | Reaching Through the Looking Glass | Craig James
TELE-OPERATION
Role of sensors
• Sensors provide critical information regarding many different processes, and are necessary for monitoring, tele-operation and automation
• Sense environmental conditions that humans cannot:
• Electromagnetic: thermal infrared, radar, …
• Others: pressure, force, distance, orientation, …
• Constantly monitor set points, thresholds and exceptions – many times need constant processing
Shearer Position Measurement System
This sensor provides 3D position and orientation for underground mining equipment. It comprises of an inertial navigation unit, controller and power management system.
In this example, the sensor is mounted inside a shearer used in longwall coal mining.
Underground Mining
Video Streaming
Inertial Navigation
Thermal Infrared
Ranging Laser
Shiploader
Bulk Carrier
Shiploader
NEW LOCATION:
CENTRAL CONTROL ROOM (CCR)
OLD LOCATION: SHIPLOADER CABIN
Shiploader | CSIRO Project
Spherical Projection Example
• Spherical digital video camera
• Collects video from >80% of the full sphere
• 12MP resolution at 15fps
• Six separate 2MP Sony CCD (1616×1232)
http://www.ptgrey.com
Overview | Tomasz Bednarz
3-D Polygon Mesh
Spherical Projection Application
Shiploader | CSIRO Project
Visual Analytics for Mining
• Hemispherical dome
• Calibration software
• Remote data streaming
Shiploader | CSIRO Project
Sensors Temporal Grouping
Transform chain
Feature Model Update Display Extraction
Pan & tilt rotations
Position & orientation
+
Long travel, slew & luff rotations
Distance & angle
Force & acceleration
Sensor
Position & orientation
Position & orientation
+ +
Frame
Mounting Boom
World
Data Fusion
+
Rockbreaker
Teleoperating Rockbreaker
Rockbreaker VR
Bumblebee2 and Phoenix
The Phoenix skid-steer, remotely controlled test platform
Immersive Environments
HSI Experiment
Human System Integration for remote collaboration
Developing and exploring the value of Immersive Telepresence environments for supporting remote mining tasks
Developing and evaluating a Wearable Augmented Reality System for supporting remote guiding in mining
Developing new interaction techniques for mixed reality UIs, using eye gaze, head motion and gesture based interactions
Glaze Control
Remote Guiding System Using Projection
Expanded Perception and Interaction Centre
EPICentre
a heritage building at UNSW Art & Design in Paddington Sydney
Expanded Perception and Interaction Centre
• VisualAnalytics
• HighPerformanceVisualisation
• Visualisation + Simulation + AI
• VirtualRealityandAugmentedReality
• Human in the Loop, HCI Research
• InterdisciplinaryResearch
Partnership
EPICentre – Expanded Perception and Interaction Centre | To engage or organise a tour please contact Tomasz Bednarz (Tomasz.Bednarz@data61.csiro.au)
XR LAB EPICYLINDER
COMPUTER GRAPHICS LAB DOME LAB
DIGITAL TWINS LAB
EPICENTRE LABS
EPICYLIDNER
EPIcylinder
56 display cubes
12 IR tracking cameras 32.1 channel audio
340o 3.0m tall
6.4m diameter
EPICylinder Infrastructure
CREATIVE MATH
CREATIVE MATH
POINT CLOUDS
GENOMICS VIEWER
Cell Parameters
Dimension Reduction
Immersive and Big Data Visualisation
Cell Parameters
Dimension Reduction
Immersive and Big Data Visualisation
Large-Scale Multiple Views
Immersive and Big Data Visualisation
Collaborative Visual Analytics
Immersive and Big Data Visualisation
MASSIVE NETWORKS
Graphics primitives
46 | Visualisation Grammar | EPICentre
Example pipeline
47 | Visualisation Grammar | EPICentre
MASSIVE NETWORKS
Expanded Perception and Interaction Centre
Digital Twins
Digital Twins
• “A digital twin is a virtual representation of a physical object or system” SAP
• “A Digital Twin is an integrated multiphysics, multiscale, probabilistic simulation of an as-built vehicle or system that uses the best available physical models, sensor updates, fleet history, etc., to mirror the life of its corresponding flying twin” NASA
• “Digital twin refers to a digital replica of potential and actual physical assets (physical twin), processes, people, places, systems and devices that can be used for various purposes” GE
Source: https://ciowatercooler.co.uk/what-are-digital-twins/
Digital Twins
Types of Digital Twins
• STATUS TWIN
Typically used for basic condition monitoring applications such as dashboards and simple alerting systems. It indicates operating parameters and is generally created with visualization tools.
• OPERATIONAL TWIN
Provides more extensive information that is typically used in decision support by operators, reliability engineers, and other decision-makers. It is linked to a set of actions or workflows where users can interact with the twin and change operating parameters where control capability is allowed.
• SIMULATION TWIN
Leverages different types of simulation or artificial intelligence capabilities to
predict, forecast, or provide insight into future operational states. You can use it for predictive maintenance or to improve the recovery yield of a processing plant.
Digital Twins | “The Ultimate Guide to Digital Twins” XMPRO
SYDNEY MAPS
RED CENTRE UNSW
SA2019.SIGGRAPH.ORG
CONFERENCE 17-20 November 2019 – EXHIBITION 18-20 November 2019 – BCEC, Brisbane, AUSTRALIA
HoloCity
HoloCity aims to integrate big transport data feeds into meaningful insights in a collaborative mixed reality environment
Now we can interact with AR urban city models globally, and rapidly, through free API services, e.g. MapBox …
https://www.mapbox.com/augmented-reality/
HOLO CITY
Immersive Prototyping
ASKAP Project
BLOOD
HYBRID ANALYTICS FOR CONSERVATION
Great Barrier Reef health: a local, national and international community concern
Citizen scientists can help obtain needed data to improve
reef health monitoring
Video – Citizen Scientists and the Reef – 2 minutes
Prototype Citizen Science Platform
• Create spatial predictive models
• Engage community in
monitoring the reef
• Develop novel visual experiences of reef environments
• Elicit expert information using contributed images
• Evaluate the utility of the models
Prototype
System Components
• Spatial Modelling
• Web Interface for Citizens
• Virtual Reality Experiences
• Workflow Framework
• Key Point: general nature of framework
• Potential to apply to other domains; river health, invasive species, amongst others
Web Interface for Citizens
Spatial Modelling
Workflow Framework
Virtual Reality
Immersive and Big Data Visualisation
Saving Jaguars – VR, Gaming, GPUs, Stats
Saving Jaguars
COMPUTING / IMAGING
Image Segmentation
Multiple Myeloma deadly cancer of blood plasma cells Facts
• Despite a rash of new drugs and advances in stem-cell therapy, this rare bloodborne cancer is still an almost certain death sentence
• A cure remains a long way off
• Plasma cell cancer
• Collections of abnormal cells accumulate in bones, where they cause bone lesions (abnormal areas of tissue), and in the bone marrow where they interfere with the production of normal blood cells
• The disease develops in 1–4 per 100,000 people PY
Micro-CT scans reveal bone damage from myeloma in a 5T2MM-bearing mouse.
Overview | Tomasz Bednarz
Level Set Applications
Level Set
160
140
120
100
80 60 40 20
0
Max speedup 2D ~70 X
152.24 sec
Governing equation:
¶fé Ñfù = – Ñf êa D( x ) + (1 – a )Ñ × ú
¶tê Ñfú ëû
2.12 sec
8.94 sec
15.43 sec
C2050 opt
C2050 Quadro Xeon
FX 580
E5520
1800 iterations, a = 0.001
900 iterations, a = 0.06
900 iterations, a = 0.95
3D Bone Model – OpenGL
Interactive volume visualisation – OGL + OCL
Overview | Tomasz Bednarz
Cloud-based Image analysis and processing toolbox
Available now: http://cloudimaging.net.au, see our demos
Overview | Tomasz Bednarz
Cloud based imaging
Research Community 1
Australian Synchrotron and Advanced CT Research Community
CSIRO Supported Imaging Tools
Research Community 2
Advanced MRI/PET Research Community
Template Design
Templates
Templates
Templates
Research Community 3
Neuroscience Research Community
Customisations
Research Community 4
Other Biomedical Applications
Toolsusing underlying workflow management
Virtual Laboratory
Characterisation VL (MASSIVE)
HCA-Vision
X-TRACT
MILXView
Workflow Management Framework
Parallel Execution Monitoring Fault-Tolerance Data Management
Public Cloud
NeCTAR Research Cloud
GeneralPurposeImagingToolsfromVL
Overview | Tomasz Bednarz
GPU
GPU
CPU
CPU
Data Data
Data
GPU
CPU
Workflow Editor Tools
IAW
IAW
IAW
Imaging Applications and Services
Visualisation Enabled Results
Cloud-based Image analysis and processing toolbox
Currently using WebGL based 3D viewer Slice:Drop http://slicedrop.com Need for WebCL/WebGL→for interactive parameters tuning
Magnetic Levitation
Diamagnetic levitation: Flying frogs and floating magnets Simon MD, Geim AK
JOURNAL OF APPLIED PHYSICS 87 (9): 6200-6204 Part 3 2000
Immersive and Big Data Visualisation
What is a magnetic force?
f = m B2 = B2 mag.force =(m /2m)B2 =B2
2m 2m gravi. force g 2m g
Paramagnetic(χm>0) material is attracted to a magnet.
Diamagnetic(χm<0)material is repelled from a magnet.
10-Tesla superconducting magnet has 400T2/m.
For oxygen gas, magnetic force is about 45 times of gravity force.
For water, magnetic force is about 1/3 of gravity force.
Enclosure vs magnet position
0 Tesla
5 Tesla
g
magnet
H O
T 10 Tesla
84
Thermochromic Liquid Crystals - for visualisation and measurements
⚫ TLCs
⚫ Color
⚫ The
⚫ Particles
⚫ Shelf
⚫ Efficient
⚫ Colours
:
smectic
,
nematic
and
play
time
months
: 50
choresteric phases
-
temperature
range
: 0.5
enough
-
20
o
for
C
response
problems
–
: 3ms,
typical
only
–
thermal
slurry
–
diameter
: 30
time
-
:6
60
m
-
life
litre
: red
tank
–
65ml
yellow
needs
orange
green
–
blue
violet
Particle Image Thermometry & Velocimetry
• TLCparticlestreatedaspassivetracersduetotheirneutralbuoyancywithrespecttowater
• TermochromicLiquidCrystals:
• Color-temperature play range: 0.5 – 20 oC
• The response time: 3 ms
• Particle diameter: 30-60 m
• Shelf-life time: 6 months
• Efficient: 50 litre tanks needs only 65 ml slurry
• Colours: red–orange–yellow–green–blue–violet
• ColourCalibrationinCondution:
Particle Image Thermometry & Velocimetry
• Whitelightsheet
• RGBcolourimage
• Image acquisition • Image maps input • Cross correlation • Peakdetection
• Sub-pixelinterpolation • Vectoroutput
•
Colour maps→Temperature maps 300
KXN-20/30 calibration
points
240 180 120
60
0
18.8 19.8
unsharp mask
Dt
20.8 21.8 22.8 23.8
temperature [oC]
hue [degree]
Hue vs Temperature vs WB
360
300
240
180
120
60
0
18.9 19.9
20.9 21.9 temperature
22.9 23.9
5200K 6000K 7000K 3000K
color temperature of the camera
hue
Reservoir model
Thermal forcing
24
23
22
21
20 d 19
18 17
ai bh
cj g
f ek
0 7 14 21 28 35 42 49 56 63 70
Time [min]
Temperature [ oC]
Exchange Flows in Reservoirs – Cooling Case
• Water circulation in reservoirs is driven by thermal gradients changing during day and night cycles.
Exchange Flows in Reservoirs – Diurnal Case
24
23
22
21cgj ) 20 df
(a )
ai bh
(b
(c)
(d )
(e )
(f)
(g)
(h )
(i) (j)
(k )
19
18 ek 17
Pr Gr
= 6.82,
4 10
= 3.52
×
0 7 14 21 28 35 42 49 56 63 70
Time [min]
Dt
unsharp mask
PIV result
Temperature [ oC]
T [K]
T0
heating phase
DT/ 2
DT/ 2
t [s]
cooling phase
PP
Gr = 1e7
Expanded Perception and Interaction Centre
Thank you for your attention