IT
…………………………………………………………………………………3
………………………………………………………………………………………………3 ……………………………………………………………………………………………….3 ……………………………………………………………………………..3 …………………………………………………………………………………….4 ……………………………………………………………………………………………….5 ………………………………………………………………………………………………………5 ……………………………………………………………………………………………….8
………………………………………………………………………………………………8 …………………………………………………………………………………….8 ……………………………………………………………………………10
………………………………………………………………………………..12 …………………………………………………………………………………………12
………………………………………………………………………………………….12 …………………………………………………………………………………………14
ETL………………………………………………………………………………………………….14 …………………………………………………………………………………………16
linux……………………………………………………………………………………………………………16 …………………………………………………………………………………………….20
SQL …………………………………………………………………………………………………………….20
…………………………………………………………………………………………25 …………………………………………………………………….25 ………………………………………………………………………………………………………..27 mvc …………………………………………………………………………………………….29 ………………………………………………………………………………………………………..33 ………………………………………………………………………………………35
………………………………………………………………………………………36 ………………………………………………………………………………………….36 …………………………………………………………………………………..40 ……………………………………………………………………………………………..44 hadoop…………………………………………………………………………………47
………………………………………………………………………………..49
……………………………………………………………………..49 ………………………………………………………………………………………….49 …………………………………………………………………………………………………49
……………………………………………………………………………….51 1
…………………………………………………………………………………………………….51 …………………………………………………………………………………………………51 ……………………………………………………………………………………………………………52
……………………………………………………………………………….53
……………………………………………………………………………….54
…………………………………………………………………………55 ………………………………………………………………………………………….55 AlphaGo ……………………………………………………………………………………….56
………………………………………………………………………………..56
…………………………………………………………………………………………56 …………………………………………………………………………………………………56 ……………………………………………………………………………………………..57 …………………………………………………………………………………………………58
…………………………………………………………………………………………59 ……………………………………………………………………………………………………………59 …………………………………………………………………………………………………….60 …………………………………………………………………………………………………….62 …………………………………………………………………………………………………….62
………………………………………………………………………………..63 …………………………………………………………………………………………63 …………………………………………………………………………………………65 …………………………………………………………………………………………66 …………………………………………………………………………………………67 ……………………………………………………………………………….68
2
: ;
;
bit:bitByteKB MBGBTBPBEBZBYBBBNBDB
1 Byte 8bit 1 KB 1,024 1 MB 1,024 1 GB 1,024 1 TB 1,024 1 PB 1,024 1 EB 1,024 1 ZB 1,024 1 YB 1,024 1 BB 1,024 1 NB 1,024
1024210:
Bytes
KB 1,048,576 Bytes MB 1,048,576 KB
GB 1,048,576 MB
TB 1,048,576 GB
PB 1,048,576 TB
EB 1,048,576 PB
ZB 1,048,576 EB
YB 1,048,576 ZB
BB 1,048,576 YB
3
1 DB 1,024 NB 1,048,576 BB
TB PB PB
; PC ; ; web PC
RFID ;; ;;
4V:VolumeVelocity VarietyValue
Volume: TB PB
Velocity:
Variety:
Value:
4V 4V VisualizationValidity 6V
4
; ; ;;
……
5
DIKW
DIKW : ??Where is the wisdom we have lost in knowledge? Where is the knowledge we have lost in information?1982 12 Information as a Resource 1987 :Management Support Systems: Towards Integrated Knowledge Management 1989 From Data to WisdomHuman Systems Management
DIKW
DIKW ;
6
DIKW :
1:;
2: ;
3:; 4:; 5:; 6: ?
:??? ? ?
7
cloudcomputing
8
;
Date Asset
Data asset management DAM
8 2 8
:
Google 100
AmazonIBMYahoo
;
9
: ;
:
1.MAIM2M Application Integration, MaaS; 2.MaaSM2M As A Service, MMO, MultiTenants
: 1.:PoP
;
2. MVNOMMO
SOA :TaaS everyTHING As A Service
Cloud Security :
Server
1.
:
2.
Facebook
3.
4.
LastPass
10
5.
6.
;
7.
? 4
8.
9.
10.
4
cloud computing
X360 LIVEPS PS NETWORK wii wifi
11
, , ,
:IaaS PaaSSaaS
IaaSInfrastructureasa Service: Internet : PaaSPlatformasa Service:PaaS SaaS PaaS SaaS PaaS SaaS
SaaS : SaaSSoftwareasa Service: Internet Web
:
12
;
CCC Java 20 80
13
bug
ETL
ETL ETL BI BI ETL 13 ,ETL BI
ETL : ETL ODSOperational Data Store ETL ETL TTransform ETL 23 DWData Warehousing
ETL ETL Oracle OWBSQL Server 2000 DTSSQL Server2005 SSIS Informatic SQL ETL SQL ETL SQL ETL
14
ETL
Extract
, DBMS,
1 DW
DBMSSQLServer Oracle DW Select
2 DW
ODBC SQL Server Oracle .txt .xls ODS
3.txt,.xls
4
, ODS
Cleaning
1: Excel
2: SQL ETL SQL
15
3:
Excel ETL ,
Transform
1: XX0001, CRM YY0001
2:
3: ETL
ETL ETL :
ETL
ET L
linux
Linux Unix POSIX UNIX CPU UNIX 32 64 Linux Unix
1991 10 5 Linux Linus Torvalds comp.os.minix Linux 1994 3 Linux 1.0 17 GPLGeneral Public License GNU
Linux : 1 2
16
3
4
5 :Android
PDA
Linux : roothomeusrbin
linux :
bin
boot
dev
etc
home
lib
mnt
proc
root
sbin
tmp
usr var
: rootlocalhost , root
linux Linux :
cd cdhome ;:home
cd root root ;cd ..;cd .; …; cd homtest cd
17
:
ls .
ls a ,. pwd
mkdir mkdir test rmdir
rm rm rf test.txt r f
cp ,cp old.txt tmpnew.txt ;
r
mv , mv old.txt new.txt
touch touch test.txt
Useradd useradd wugk userdel Groupadd groupadd wugk1 groupdel
find find home name test.txt,
:
find name
findhome name .txt ;home .txt
vi vi :
vi i
esc ::wq
:q! q! cat cat test.txt test.txt more cat
cat more ,: cat test.txt more text
echo echo ok ok
chmod UNIX 3 3 chmod
:
u User;
g Group;
o Other
18
;
a All; r 4;
w 2;
x 1;
0;
s :
Linux :GroupotherLinux root etcpasswd etcshadow etcgroup !
linux
chown D id
chown
c changes:v;
f quite silent:;
h nodereference:
;
R recursive:
;
v version:; dereference:h;
help:; reference:
; version:
Crontab linux
1 crontab e crontab
19
2 crontab crontab l
3crontab :
command
1 159 1
2 1230 0
3 131
4 112
5 060
6
4 : usrlocaletcrc.dlighttpd
apache
30 21 usrlocaletcrc.dlighttpd restart 21:30 apache
SQL
SQL SQL IBM San Jose SQL Structured Query Language sequel IBM DB2 RDBMS
SQL , SQL
SQL :;; ;;
SQL:DML DDL
SQL : DML
DDL
SQL SQL
SQL DML : SELECT
UPDATE
restart shell
20
DELETE
INSERT INTO
SQL DDL
SQL DDL :
CREATE DATABASE
ALTER DATABASE
CREATE TABLE
ALTER TABLE DROP TABLE
CREATE INDEX DROP INDEX
CREATE VIEW
DROP VIEW
CREATE TABLE CREATE TABLE
1 , 2 , …….
: Person :LastName FirstNameAddress Age:
CREATE TABLE Person
LastName varchar, FirstName varchar, Address varchar, Age int
CREATE INDEX
Unique Index
CREATE UNIQUE INDEX
ON
UNIQUE
CREATE INDEX : CREATE INDEX
ON
21
: PersonIndex Person LastName :
CREATE INDEX PersonIndex
ON Person LastName
SQL WHERE JOIN
CREATE VIEW
CREATE VIEW viewname AS
SELECT columnnames
FROM tablename
WHERE condition
: CurrentProductList Products
SQL : CREATE VIEW CurrentProductList AS SELECT ProductID,ProductName
FROM Products
WHERE DiscontinuedNo
insert into
:
insert into 1 2 3 values 1 2 3 insert into select 1 2 3 from update
:
update set where
delete :
delete from where delete where
truncate table :
truncate table
truncate table where
SQL select
select from where order by asc desc
:select distinct company from orders; : orders company
:select from a
22
: a
:select i,j,k from a where f5
: a f5 i,j,k3
AND OR
: SELECT FROM Persons WHERE FirstNameThomas AND LastNameCarter
: AND Carter Thomas :
SELECT FROM Persons WHERE firstnameThomas OR lastnameCarter
: OR Carter Thomas :
AS
:select name as from a where xingbie
: a name name
:select name from a where email is null
: a email name ;SQL is null is not null
:select name, as from Student
: a name
:top percent 1:select top 6 name from a
: a name 6top 2:select top 60 percent name from a : a name 60percent
:order by , asc , desc :select name
from a
where chengji60
order by desc
: a chengji 60 name ; ASC
like
:like char varchar
:select from a where name like
: a name
between
:select from a where nianling between 18 and 20 : a nianling 18 20
in
23
:select name from a where address in ,, : a address name
group by
:select studentID as ,AVGscore as : score
from score : score
group by studentID
: score strdentID strdentID score ;select
having
:select studentID as ,AVGscore as : score
from score : score
group by studentID
having countscore1
: countscore1 where having
where :select a.name,b.chengji
from a,b
where a.nameb.name
: a b name a name b chengji
from join…on :select a.name,b.chengji
from a inner join b
on a.nameb.name
:
:select s.name,c.courseID,c.score
from strdents as s
left outer join score as c
on s.scodec.strdentID
: strdents score on score strdentID strdents sconde
:select s.name,c.courseID,c.score from strdents as s
right outer join score as c
24
on s.scodec.strdentID
: strdents score on strdents sconde score strdentID
unionunion all
UNION SELECT UNION SELECT
SELECT SELECT columnnames FROM tablename1
UNION
SELECT columnnames FROM tablename2
:UNION UNION ALL
SELECT columnnames FROM tablename1
UNION ALL
SELECT columnnames FROM tablename2
UNION UNION SELECT
EXISTS
EXISTS True False
EXISTS
: EXISTS subquery
: subquery SELECT COMPUTE INTO
:Boolean TRUE FLASE
EXISTS IN select from TableIn where existsselect BID from TableEx where BNAMETableIn.ANAME
select from TableIn where ANAME inselect BNAME from TableEx
NOT EXISTS EXISTS NOT EXISTS WHERE
:
;
25
:
: !
?? ?
! ;
: get : get
:
26
: ? ; ;
java ?
java
JVM
Java Virtual Machine java
JVM java JVM JVM
java java JVM
java JVM
JVM
java
27
JAVAEE:Java Platform Enterprise Edition web ;
JAVASE:Java Platform Standard Edition ;
JAVAME:Java Platform Micro Edition ;
JDK
Java Develop Kits java
JVMJava API
JRE
Java Run Environmentjava JVM java
API
API
Application Programming Interface java
?? API java
java
:java C C
:java java
:
:java
: java JVM
:java
:java
:java
:HTMLCSS Javascript
HTMLCSSJavaScript
HTML Hyper Text Markup LanguageHTML
CSS Cascading Style Sheets, HTML :selector property:value :
JavaScript
28
HTML CSS JavaScript
HTML CSS javascript CSS
?HTMLHTML ?:
!DOCTYPE html PUBLIC W3CDTD XHTML 1.0 TransitionalEN http:www.w3.orgTRxhtml1DTDxhtml1transitional.dtd
html xmlnshttp:www.w3.org1999xhtml
head
meta httpequivContentType contenttexthtml; charsetutf8
titletitle
link relstylesheet typetextcss hrefmycss.css script typetextjavascript srcmyjs.jsscript
head
body pp body
html
HTML W3C XHTML1.0 utf8 mycss.css javascript myks.js
mvc
MVC Model View Controllermodelview controller
MVC M V C
MVC : M V Windows M V
29
V View html MVC
M model MVC
C controller
30
MVC :
MVC jspservletjavabean
JavaBean
JSP Form
Serlvet
MVC :
Struts2 :Struts2 MVC web Struts2 Web Web Web Struts2 Web MVCStruts2 MVC
filterdispatcher
FilterDispatcher FilterDispatcher URL struts.xml Action Action FilterDispatcher Filterservlet web Struts2 web.xml FilterDispatcher J2EE
Action
FilterDispatcher Action Action
31
Struts2 Action web Servlet API Action HttpServletRequest HttpServletResponse Action
Result
Struts2 jsp freemarkervelocity jfreechart JasperReports XML HTML XSLT
MVC :
1.
MVC
2.
MVC
WEBHTTP wap
3.
MVC MVC Java HTML JSP
4.
WEB
MVC :
1. MVC
MVC
MVC 2.
3.
MVC MVC
4.
MVC
5.
32
OS
1;
2 OS ; 3 OS
;
4;
5
LAN
web
:
1Remote Procedure Call 2MesSAgeOriented Middleware 3object RequeST Brokers
OTM API
33
Web Tomcat
Tomcat
Tomcat Apache Servlet Servlet JSP Web Tomcat Tomcat
Tomcat HTTP Web Tomcat ApacheHTTP Apache HTTP C HTTP Web ; HTTP web server Tomcat XML
Tomcat
bin Tomcat .sh Unix ; .bat Windows
conf Tomcat logs Tomcat webapps webapp web
web webapp
METAINF
MANIFEST.MF WEBINF
classes
.class
.xml
lib
.jar
web.xml
userdir
METAINF
WEBINF
class
class xml
jar
Web
userfiles
webapp: war webapp
METAINF:METAINF
WEBINF:Java web
WEBINFclasses: Java class WEBINFlib: jar WEBINFweb.xml:web
servlet
34
Web service
Web Service XML Web Service WebService Internet Intranet : SOAP Web WSDL UDDI
:
Web :1 ;2Web URL
Web Web Web Web Web Web ;Web Web SOAP Web ;Web Web Web UDDI : Web Web Web Web : Web ; Web ;
Web service
MQ
MQ Message Queue
35
MQ webservice
MessageQueue
MQ MQ,
MQ MSMQ ,Apache ActiveMQ mq
, ,, ,
1, ,
2 PC , ,
3,
4,,
,,, Hadoop
, :
1? ?
2, ,?
3?
36
? 4
?? 5?? 6?
? 7?
CPU ? ,
3 : 1
,, ,, , Oracle SQL Server , ,:
1
,,, , ,, , ,,
2
,, , , ,, , hash
,
2
, ,XML HTML , Google FileSysten,GFS GFS : Client Master Chunk server
1 Client
Client GFS ,, ,,
2 Master
Master GFS ,, Chunk Name Space,
37
, 64 , Chunk , Master Chunk heartbeat,
3 Chunk server
, ChunkGFS , 64MB Chunk, Chunk Block , 64KB Chunk 64 GFS , Chunk 3 Chunk Server , GFS
3
, , , schema , , NOSQL
NOSQL , ACIDA NOSQL , Non Relational Database
1Map, ,,
2 CAP , , Elasticity
3,, ,
4 NOSQL ,
Hdfs
Hadoop HDFS
MR
HDFS MasterSlave HDFS NameNode DataNode Hadoop2.2
NameNode hadoop NameNode DataNode HDFS
DataNode NameNode DataNode DataNode NameNode
38
NameNode HDFS NameNode
:HDFS
:NameNodeDataNodeClientNameNode DataNode Client
:
1 Client NameNode
2 NameNode Client
DataNode
3Client block DataNode
block DataNode :
1Client NameNode
2NameNode DataNode
3Client
HDFS : : Block NameNode
DateNode DataNode DataNode DataNode Rack DataNode Rack
39
,
1 load balancing
2 load sharing , ,
, , , ,, ,,,
, , , NP ,
: , , , , ,, ,
, , ;, ,, , , ,, , , , , ,
40
YARNYet Another Resource Negotiator ResourceManagerRM ApplicationMasterAMJob Job DAG
YARN MapReduce Storm Spark S4 :
1
CPU Container YARN Container YARN Container
2:
YARN Cgroups Container
YARN Mapreduce V1 MapReduce V2 YARN ResourceManager NodeManager NodeManger Container Container ApplicationMaster ApplicationMaster Task
YARN
YARN :
YARN MasterSlave ResourceManager
41
NodeManager ApplicationMaster Container
1ResourceManagerRM
NM AM Container AM Container:
2Scheduler
ContainerShceduler ApplicationMaster Container CapacitySchedulerFairScheduler
3Applications Manager
AM AM Container
4NodeManager NM
NM RM Container ; AM Container
5ApplicationMaster AM
AM ApplicationMaster ResourceManager NodeManager Task MapReduce YARN Mapreduce YARN SparkStorm YARN application
6Container
Container YARN CPU AM RM RM AM Container YARN Container Container
YARN
YARN YARN :
AM 13;
AM 47
42
YARN
1 YARN AM AM ; ApplicationMaster unix YarnClient
2RM Container NM Container AM;
3AM RM RM AMRMClientAsync.CallbackHandlergetProgress RM
47;
4AM RPC RM ; AMRMClientAsync , AMRMClientAsync.CallbackHandler
5 AM NM ; ContainerLaunchContext Container
6NM JAR ;
7 RPC AM AM ; ApplicationMaster NM NMClientAsync object NMClientAsync.CallbackHandler
8AM RM
YARN
YARN Container
YARN :
RM AM: push AM
43
AM ;
AM : YARN
ResourceManager Container
YARN
Capacity Scheduler:multitenant Hadoop Capacity Scheduler queue queue queuehierarchical queues
Capacity : Hierarchical Queues
:ACL :
resoucebased scheduling:
queue : queue
:ACL queue queue
Fair Scheduler: FAIR CPU
,, , Flynn
Flynn
1966 , Michael J.Fynn ,, Flynn
44
Flynn , ;
Single Instruction stream and Single Data stream,SISD
SISD ,, , ,, IBM PC
SIMD , SIMD, Intel MMXTM SSE Streaming SIMD ExtensionsSSE2SSE3SSE4 AVX Advanced Vector Extensions SIMD
MISD , ,
Multiple Instruction Stream and Multiple Data Strean,MIMD
MIMD , Intel MIMD
,
1 Shared Memory Access , UMA Uniform Memory Access, SMP Symmetric Multiprocessing,
2 distributed memory access
3 distributed and sharedmemory access
NUMA Non Uniform Memory Access,, ,,
Hadoop MapReduce HDFS MR Hadoop HDFS MR HDFS MR MR HDFS
Singe Instruction stream and
Multiple Data strean,SIMD
Multiple Instruction stream and
Single Data strean,MISD
45
MR : keyvalue
keyvalue MR Map Reduce map keyvalue keyvalue MR key value reduce Reduce key value reduce value value value reduce value value
map
46
k1,v1 map map keyvalue map k2,v2k2,v2 merge shuffle reduce reduce k3,v3 HDFS
hadoop
:
HDFS HBASE
HDFS : NameNode DataNode HDFS Hive HDFS SQL MapReduce
HBase :HBase HBase HRegion HBase Master
Kafka Kafka topic Kafka topic producers topics consumerKafka broker
MapReduceYarn Hadoop Hadoop
47
HiveSparkSqlImpala SQL Hadoop
spark Spark Streaming Spark Apache HadoopHBase Kafka RDD
Storm Eclipse Public License 1.0,Storm Storm Hadoop Storm
Hive Hadoop sql sql MapReduce SQL MapReduce MapReduce
Hvie Hadoop ETL Hadoop Hive SQL HQL SQL MapReduce mapper reducer mapper reducer
Hive SQL HQL Hive Hive Hive Online Hive Hive
Java MapReduce apiPig MapReduce Pig Pig MapReduce Join
ZooKeeper Google Chubby Hadoop Hbase :
ZooKeeper
ZooKeeper Java C
48
,, ,
, , , , ,, , ,, ,
1
2
:1 ;2 ;3;4
3
49
:
;
KJ PDPC
4
:1 ;2 ;3
;4
;5
statistics
descriptive statistics ,,
inferential statistic
: :
1 arithmetic mean
2 3 4
5
50
, 6
XY ,A1AA1A…AAB1AB1A…AB., ,A,i1,2,…,m,Bj1,2,…,n XY X tuples Y
Apriori FPgrowth
;
SVM
generalized linear model wxb w b wxb ywxb L wxb ppLwxb, p 1p L logistic L logistic softmax
SVMSVMSupport Vector Machine SVM
51
: :
Pcx Pxc P c Px
Pcx: x c Pc: c Pxc: c x
Px:x Pcx
x c
ID3C4.5CARTRFGBDT
clustering analysis : classlabeled data,; , taxonomy formation,
Kmeans : k
c :
1 c ;
2 k c
; 3;
4 c 23
Kmeans
52
CRISPDMcrossindustry standard process for data mining 6
business understanding
:
data understanding
:
data preparation
:
53
modeling
evaluation
deployment
SPSS
PSS SPSS SPSS EXCEL SPSS SPSS Logistic Probit
54
SPSS
SAS
SAS SAS GUI
R
R GNU C FORTRAN
Python
Python Python numpy scipy pandas matplotlib sklearn tensorflow
: 3040 :
55
AlphaGo
2016 3 Google DeepMind AlphaGo 4 1 AlphaGo AlphaGo
; :
:
1 2 3
;
Metadatadata about data
property ; ;;
56
; ;
:
1
2 MPP Apache Hadoop Hadoop NoSQL
3
4 ETL ; ;
5
:
?
??
??
?
? ?
:
Accuracy:
57
Completeness:
Consistency:
Uniqueness:
Integration:
Conformity:
:
1 ;
2 ;
3 ;
4 ;
5
58
:
1Confidentiality
secrecy
2Integrity
RSA X 2 2Xmalleably
3Availability
59
:
1:
2:
3:
4:
5: 6 7:
8:
:
1
2
3
4
5
6
7
8
60
:
1
2
3
4NAS
NAS NAS NAS
5
6
IT
7SAN
SAN
8
9
61
3 :
kryptos graphein Ron Rivest : :
1
visible watermarking
Steganography metadata
2
3
:
62
CDROM :
;
:
3.0
63
;,
2017
:
1
,
2
,
3
,,
,
:
1:
2:,
,,
3:,
,,, ,, ,,,
4:,
, , 3 4 , ,
5:,
,, ,
:
1:3G 4G
2: ITV
3:
4:
64
2017
:
1:,
;,,
,;, ,
2:,
,, ,
;
3:MB,
:, ;:
4:,,
OMB,, ,
,,
5:,
,
6:
65
:
1 IT
,
2,
,
3,
MSS ,
ITV ITV
:
1 2
3 4 5
6
7 8 9 10
:
1:
2 :
66
3ITV:EPG VOD EPG VOD ITV
:
IVR
67
1:
2:
3:
4:
68