程序代写代做 android javaEE hbase Java data mining Fortran graph html C hadoop gui database Excel jvm Hive javascript IT

IT

…………………………………………………………………………………3
………………………………………………………………………………………………3 ……………………………………………………………………………………………….3 ……………………………………………………………………………..3 …………………………………………………………………………………….4 ……………………………………………………………………………………………….5 ………………………………………………………………………………………………………5 ……………………………………………………………………………………………….8
………………………………………………………………………………………………8 …………………………………………………………………………………….8 ……………………………………………………………………………10
………………………………………………………………………………..12 …………………………………………………………………………………………12
………………………………………………………………………………………….12 …………………………………………………………………………………………14
ETL………………………………………………………………………………………………….14 …………………………………………………………………………………………16
linux……………………………………………………………………………………………………………16 …………………………………………………………………………………………….20
SQL …………………………………………………………………………………………………………….20
…………………………………………………………………………………………25 …………………………………………………………………….25 ………………………………………………………………………………………………………..27 mvc …………………………………………………………………………………………….29 ………………………………………………………………………………………………………..33 ………………………………………………………………………………………35
………………………………………………………………………………………36 ………………………………………………………………………………………….36 …………………………………………………………………………………..40 ……………………………………………………………………………………………..44 hadoop…………………………………………………………………………………47
………………………………………………………………………………..49
……………………………………………………………………..49 ………………………………………………………………………………………….49 …………………………………………………………………………………………………49
……………………………………………………………………………….51 1

…………………………………………………………………………………………………….51 …………………………………………………………………………………………………51 ……………………………………………………………………………………………………………52
……………………………………………………………………………….53
……………………………………………………………………………….54
…………………………………………………………………………55 ………………………………………………………………………………………….55 AlphaGo ……………………………………………………………………………………….56
………………………………………………………………………………..56
…………………………………………………………………………………………56 …………………………………………………………………………………………………56 ……………………………………………………………………………………………..57 …………………………………………………………………………………………………58
…………………………………………………………………………………………59 ……………………………………………………………………………………………………………59 …………………………………………………………………………………………………….60 …………………………………………………………………………………………………….62 …………………………………………………………………………………………………….62
………………………………………………………………………………..63 …………………………………………………………………………………………63 …………………………………………………………………………………………65 …………………………………………………………………………………………66 …………………………………………………………………………………………67 ……………………………………………………………………………….68
2

: ;
;

bit:bitByteKB MBGBTBPBEBZBYBBBNBDB
1 Byte 8bit 1 KB 1,024 1 MB 1,024 1 GB 1,024 1 TB 1,024 1 PB 1,024 1 EB 1,024 1 ZB 1,024 1 YB 1,024 1 BB 1,024 1 NB 1,024
1024210:
Bytes
KB 1,048,576 Bytes MB 1,048,576 KB
GB 1,048,576 MB
TB 1,048,576 GB
PB 1,048,576 TB
EB 1,048,576 PB
ZB 1,048,576 EB
YB 1,048,576 ZB
BB 1,048,576 YB
3

1 DB 1,024 NB 1,048,576 BB
TB PB PB

; PC ; ; web PC
RFID ;; ;;
4V:VolumeVelocity VarietyValue
Volume: TB PB
Velocity:
Variety:
Value:
4V 4V VisualizationValidity 6V

4

; ; ;;

……

5

DIKW
DIKW : ??Where is the wisdom we have lost in knowledge? Where is the knowledge we have lost in information?1982 12 Information as a Resource 1987 :Management Support Systems: Towards Integrated Knowledge Management 1989 From Data to WisdomHuman Systems Management
DIKW

DIKW ;

6

DIKW :
1:;
2: ;
3:; 4:; 5:; 6: ?
:??? ? ?
7

cloudcomputing
8

;

Date Asset
Data asset management DAM

8 2 8

:

Google 100
AmazonIBMYahoo

;

9

: ;
:
1.MAIM2M Application Integration, MaaS; 2.MaaSM2M As A Service, MMO, MultiTenants
: 1.:PoP
;
2. MVNOMMO
SOA :TaaS everyTHING As A Service

Cloud Security :
Server
1.
:
2.
Facebook
3.

4.
LastPass
10

5.

6.
;
7.
? 4
8.

9.

10.
4

cloud computing

X360 LIVEPS PS NETWORK wii wifi
11

, , ,

:IaaS PaaSSaaS
IaaSInfrastructureasa Service: Internet : PaaSPlatformasa Service:PaaS SaaS PaaS SaaS PaaS SaaS
SaaS : SaaSSoftwareasa Service: Internet Web

:

12

;

CCC Java 20 80

13

bug

ETL
ETL ETL BI BI ETL 13 ,ETL BI
ETL : ETL ODSOperational Data Store ETL ETL TTransform ETL 23 DWData Warehousing
ETL ETL Oracle OWBSQL Server 2000 DTSSQL Server2005 SSIS Informatic SQL ETL SQL ETL SQL ETL
14

ETL
Extract
, DBMS,
1 DW
DBMSSQLServer Oracle DW Select
2 DW
ODBC SQL Server Oracle .txt .xls ODS
3.txt,.xls
4
, ODS
Cleaning

1: Excel
2: SQL ETL SQL
15

3:
Excel ETL ,
Transform

1: XX0001, CRM YY0001
2:
3: ETL
ETL ETL :
ETL
ET L

linux
Linux Unix POSIX UNIX CPU UNIX 32 64 Linux Unix
1991 10 5 Linux Linus Torvalds comp.os.minix Linux 1994 3 Linux 1.0 17 GPLGeneral Public License GNU
Linux : 1 2
16

3
4
5 :Android
PDA
Linux : roothomeusrbin
linux :

bin
boot
dev
etc
home
lib
mnt
proc
root
sbin
tmp
usr var
: rootlocalhost , root
linux Linux :
cd cdhome ;:home
cd root root ;cd ..;cd .; …; cd homtest cd

17

:
ls .
ls a ,. pwd
mkdir mkdir test rmdir
rm rm rf test.txt r f
cp ,cp old.txt tmpnew.txt ;
r
mv , mv old.txt new.txt
touch touch test.txt

Useradd useradd wugk userdel Groupadd groupadd wugk1 groupdel
find find home name test.txt,
:
find name

findhome name .txt ;home .txt

vi vi :
vi i

esc ::wq

:q! q! cat cat test.txt test.txt more cat
cat more ,: cat test.txt more text
echo echo ok ok
chmod UNIX 3 3 chmod
:
u User;
g Group;
o Other
18

;
a All; r 4;
w 2;
x 1;
0;
s :
Linux :GroupotherLinux root etcpasswd etcshadow etcgroup !
linux
chown D id

chown

c changes:v;
f quite silent:;
h nodereference:
;
R recursive:
;
v version:; dereference:h;
help:; reference:
; version:
Crontab linux
1 crontab e crontab
19

2 crontab crontab l
3crontab :
command

1 159 1
2 1230 0
3 131
4 112
5 060
6
4 : usrlocaletcrc.dlighttpd
apache
30 21 usrlocaletcrc.dlighttpd restart 21:30 apache

SQL
SQL SQL IBM San Jose SQL Structured Query Language sequel IBM DB2 RDBMS
SQL , SQL
SQL :;; ;;
SQL:DML DDL
SQL : DML
DDL
SQL SQL
SQL DML : SELECT
UPDATE
restart shell
20

DELETE
INSERT INTO
SQL DDL
SQL DDL :
CREATE DATABASE
ALTER DATABASE
CREATE TABLE
ALTER TABLE DROP TABLE
CREATE INDEX DROP INDEX
CREATE VIEW
DROP VIEW
CREATE TABLE CREATE TABLE

1 , 2 , …….

: Person :LastName FirstNameAddress Age:
CREATE TABLE Person
LastName varchar, FirstName varchar, Address varchar, Age int

CREATE INDEX
Unique Index

CREATE UNIQUE INDEX
ON

UNIQUE

CREATE INDEX : CREATE INDEX
ON

21

: PersonIndex Person LastName :
CREATE INDEX PersonIndex
ON Person LastName

SQL WHERE JOIN
CREATE VIEW
CREATE VIEW viewname AS
SELECT columnnames
FROM tablename
WHERE condition
: CurrentProductList Products
SQL : CREATE VIEW CurrentProductList AS SELECT ProductID,ProductName
FROM Products
WHERE DiscontinuedNo
insert into

:
insert into 1 2 3 values 1 2 3 insert into select 1 2 3 from update

:
update set where

delete :
delete from where delete where
truncate table :
truncate table
truncate table where
SQL select

select from where order by asc desc

:select distinct company from orders; : orders company
:select from a

22

: a
:select i,j,k from a where f5
: a f5 i,j,k3
AND OR
: SELECT FROM Persons WHERE FirstNameThomas AND LastNameCarter
: AND Carter Thomas :
SELECT FROM Persons WHERE firstnameThomas OR lastnameCarter
: OR Carter Thomas :
AS
:select name as from a where xingbie
: a name name

:select name from a where email is null
: a email name ;SQL is null is not null

:select name, as from Student
: a name
:top percent 1:select top 6 name from a
: a name 6top 2:select top 60 percent name from a : a name 60percent
:order by , asc , desc :select name
from a
where chengji60
order by desc
: a chengji 60 name ; ASC

like
:like char varchar
:select from a where name like
: a name
between
:select from a where nianling between 18 and 20 : a nianling 18 20
in
23

:select name from a where address in ,, : a address name
group by
:select studentID as ,AVGscore as : score
from score : score
group by studentID
: score strdentID strdentID score ;select
having
:select studentID as ,AVGscore as : score
from score : score
group by studentID
having countscore1
: countscore1 where having

where :select a.name,b.chengji
from a,b
where a.nameb.name
: a b name a name b chengji
from join…on :select a.name,b.chengji
from a inner join b
on a.nameb.name
:

:select s.name,c.courseID,c.score
from strdents as s
left outer join score as c
on s.scodec.strdentID
: strdents score on score strdentID strdents sconde

:select s.name,c.courseID,c.score from strdents as s
right outer join score as c
24

on s.scodec.strdentID
: strdents score on strdents sconde score strdentID
unionunion all
UNION SELECT UNION SELECT
SELECT SELECT columnnames FROM tablename1
UNION
SELECT columnnames FROM tablename2
:UNION UNION ALL
SELECT columnnames FROM tablename1
UNION ALL
SELECT columnnames FROM tablename2
UNION UNION SELECT

EXISTS
EXISTS True False
EXISTS
: EXISTS subquery
: subquery SELECT COMPUTE INTO
:Boolean TRUE FLASE
EXISTS IN select from TableIn where existsselect BID from TableEx where BNAMETableIn.ANAME
select from TableIn where ANAME inselect BNAME from TableEx
NOT EXISTS EXISTS NOT EXISTS WHERE

:

;
25

:

: !
?? ?

! ;

: get : get

:

26

: ? ; ;

java ?
java
JVM
Java Virtual Machine java
JVM java JVM JVM
java java JVM
java JVM
JVM
java
27

JAVAEE:Java Platform Enterprise Edition web ;
JAVASE:Java Platform Standard Edition ;
JAVAME:Java Platform Micro Edition ;
JDK
Java Develop Kits java
JVMJava API
JRE
Java Run Environmentjava JVM java
API
API
Application Programming Interface java
?? API java
java
:java C C

:java java
:

:java
: java JVM
:java
:java
:java

:HTMLCSS Javascript
HTMLCSSJavaScript
HTML Hyper Text Markup LanguageHTML
CSS Cascading Style Sheets, HTML :selector property:value :
JavaScript

28

HTML CSS JavaScript
HTML CSS javascript CSS
?HTMLHTML ?:
!DOCTYPE html PUBLIC W3CDTD XHTML 1.0 TransitionalEN http:www.w3.orgTRxhtml1DTDxhtml1transitional.dtd
html xmlnshttp:www.w3.org1999xhtml
head
meta httpequivContentType contenttexthtml; charsetutf8
titletitle
link relstylesheet typetextcss hrefmycss.css script typetextjavascript srcmyjs.jsscript
head
body pp body
html
HTML W3C XHTML1.0 utf8 mycss.css javascript myks.js
mvc
MVC Model View Controllermodelview controller
MVC M V C
MVC : M V Windows M V

29

V View html MVC
M model MVC
C controller

30

MVC :
MVC jspservletjavabean
JavaBean
JSP Form
Serlvet
MVC :
Struts2 :Struts2 MVC web Struts2 Web Web Web Struts2 Web MVCStruts2 MVC
filterdispatcher
FilterDispatcher FilterDispatcher URL struts.xml Action Action FilterDispatcher Filterservlet web Struts2 web.xml FilterDispatcher J2EE
Action
FilterDispatcher Action Action

31

Struts2 Action web Servlet API Action HttpServletRequest HttpServletResponse Action
Result
Struts2 jsp freemarkervelocity jfreechart JasperReports XML HTML XSLT
MVC :
1.
MVC
2.
MVC
WEBHTTP wap
3.
MVC MVC Java HTML JSP
4.
WEB
MVC :
1. MVC
MVC
MVC 2.

3.
MVC MVC
4.
MVC
5.
32

OS

1;
2 OS ; 3 OS
;
4;
5
LAN
web

:
1Remote Procedure Call 2MesSAgeOriented Middleware 3object RequeST Brokers
OTM API
33

Web Tomcat
Tomcat
Tomcat Apache Servlet Servlet JSP Web Tomcat Tomcat
Tomcat HTTP Web Tomcat ApacheHTTP Apache HTTP C HTTP Web ; HTTP web server Tomcat XML
Tomcat
bin Tomcat .sh Unix ; .bat Windows

conf Tomcat logs Tomcat webapps webapp web
web webapp
METAINF
MANIFEST.MF WEBINF
classes
.class
.xml
lib
.jar
web.xml
userdir

METAINF
WEBINF
class
class xml

jar
Web

userfiles
webapp: war webapp

METAINF:METAINF

WEBINF:Java web

WEBINFclasses: Java class WEBINFlib: jar WEBINFweb.xml:web
servlet
34

Web service
Web Service XML Web Service WebService Internet Intranet : SOAP Web WSDL UDDI
:
Web :1 ;2Web URL
Web Web Web Web Web Web ;Web Web SOAP Web ;Web Web Web UDDI : Web Web Web Web : Web ; Web ;
Web service
MQ
MQ Message Queue

35

MQ webservice
MessageQueue
MQ MQ,
MQ MSMQ ,Apache ActiveMQ mq

, ,, ,
1, ,
2 PC , ,
3,
4,,
,,, Hadoop
, :
1? ?
2, ,?
3?

36

? 4
?? 5?? 6?
? 7?
CPU ? ,
3 : 1
,, ,, , Oracle SQL Server , ,:
1
,,, , ,, , ,,
2
,, , , ,, , hash
,
2
, ,XML HTML , Google FileSysten,GFS GFS : Client Master Chunk server
1 Client
Client GFS ,, ,,
2 Master
Master GFS ,, Chunk Name Space,
37

, 64 , Chunk , Master Chunk heartbeat,
3 Chunk server
, ChunkGFS , 64MB Chunk, Chunk Block , 64KB Chunk 64 GFS , Chunk 3 Chunk Server , GFS
3
, , , schema , , NOSQL
NOSQL , ACIDA NOSQL , Non Relational Database
1Map, ,,
2 CAP , , Elasticity
3,, ,
4 NOSQL ,
Hdfs
Hadoop HDFS
MR
HDFS MasterSlave HDFS NameNode DataNode Hadoop2.2
NameNode hadoop NameNode DataNode HDFS
DataNode NameNode DataNode DataNode NameNode
38

NameNode HDFS NameNode
:HDFS
:NameNodeDataNodeClientNameNode DataNode Client
:
1 Client NameNode
2 NameNode Client
DataNode
3Client block DataNode
block DataNode :
1Client NameNode
2NameNode DataNode
3Client
HDFS : : Block NameNode
DateNode DataNode DataNode DataNode Rack DataNode Rack
39

,
1 load balancing
2 load sharing , ,
, , , ,, ,,,
, , , NP ,
: , , , , ,, ,
, , ;, ,, , , ,, , , , , ,
40

YARNYet Another Resource Negotiator ResourceManagerRM ApplicationMasterAMJob Job DAG
YARN MapReduce Storm Spark S4 :
1
CPU Container YARN Container YARN Container
2:
YARN Cgroups Container
YARN Mapreduce V1 MapReduce V2 YARN ResourceManager NodeManager NodeManger Container Container ApplicationMaster ApplicationMaster Task
YARN
YARN :
YARN MasterSlave ResourceManager
41

NodeManager ApplicationMaster Container
1ResourceManagerRM
NM AM Container AM Container:
2Scheduler
ContainerShceduler ApplicationMaster Container CapacitySchedulerFairScheduler
3Applications Manager
AM AM Container
4NodeManager NM
NM RM Container ; AM Container
5ApplicationMaster AM
AM ApplicationMaster ResourceManager NodeManager Task MapReduce YARN Mapreduce YARN SparkStorm YARN application
6Container
Container YARN CPU AM RM RM AM Container YARN Container Container
YARN
YARN YARN :
AM 13;
AM 47
42

YARN
1 YARN AM AM ; ApplicationMaster unix YarnClient
2RM Container NM Container AM;
3AM RM RM AMRMClientAsync.CallbackHandlergetProgress RM
47;
4AM RPC RM ; AMRMClientAsync , AMRMClientAsync.CallbackHandler
5 AM NM ; ContainerLaunchContext Container
6NM JAR ;
7 RPC AM AM ; ApplicationMaster NM NMClientAsync object NMClientAsync.CallbackHandler
8AM RM
YARN
YARN Container
YARN :
RM AM: push AM
43

AM ;
AM : YARN

ResourceManager Container

YARN
Capacity Scheduler:multitenant Hadoop Capacity Scheduler queue queue queuehierarchical queues
Capacity : Hierarchical Queues
:ACL :
resoucebased scheduling:
queue : queue
:ACL queue queue
Fair Scheduler: FAIR CPU

,, , Flynn
Flynn
1966 , Michael J.Fynn ,, Flynn
44

Flynn , ;
Single Instruction stream and Single Data stream,SISD
SISD ,, , ,, IBM PC
SIMD , SIMD, Intel MMXTM SSE Streaming SIMD ExtensionsSSE2SSE3SSE4 AVX Advanced Vector Extensions SIMD
MISD , ,
Multiple Instruction Stream and Multiple Data Strean,MIMD
MIMD , Intel MIMD

,
1 Shared Memory Access , UMA Uniform Memory Access, SMP Symmetric Multiprocessing,
2 distributed memory access
3 distributed and sharedmemory access
NUMA Non Uniform Memory Access,, ,,
Hadoop MapReduce HDFS MR Hadoop HDFS MR HDFS MR MR HDFS
Singe Instruction stream and
Multiple Data strean,SIMD
Multiple Instruction stream and
Single Data strean,MISD
45

MR : keyvalue
keyvalue MR Map Reduce map keyvalue keyvalue MR key value reduce Reduce key value reduce value value value reduce value value

map
46

k1,v1 map map keyvalue map k2,v2k2,v2 merge shuffle reduce reduce k3,v3 HDFS
hadoop
:
HDFS HBASE
HDFS : NameNode DataNode HDFS Hive HDFS SQL MapReduce
HBase :HBase HBase HRegion HBase Master
Kafka Kafka topic Kafka topic producers topics consumerKafka broker
MapReduceYarn Hadoop Hadoop
47

HiveSparkSqlImpala SQL Hadoop
spark Spark Streaming Spark Apache HadoopHBase Kafka RDD
Storm Eclipse Public License 1.0,Storm Storm Hadoop Storm
Hive Hadoop sql sql MapReduce SQL MapReduce MapReduce
Hvie Hadoop ETL Hadoop Hive SQL HQL SQL MapReduce mapper reducer mapper reducer
Hive SQL HQL Hive Hive Hive Online Hive Hive
Java MapReduce apiPig MapReduce Pig Pig MapReduce Join
ZooKeeper Google Chubby Hadoop Hbase :
ZooKeeper
ZooKeeper Java C
48

,, ,
, , , , ,, , ,, ,

1

2

:1 ;2 ;3;4

3

49

:
;
KJ PDPC

4

:1 ;2 ;3
;4
;5

statistics
descriptive statistics ,,
inferential statistic
: :

1 arithmetic mean
2 3 4
5

50

, 6

XY ,A1AA1A…AAB1AB1A…AB., ,A,i1,2,…,m,Bj1,2,…,n XY X tuples Y
Apriori FPgrowth

;
SVM
generalized linear model wxb w b wxb ywxb L wxb ppLwxb, p 1p L logistic L logistic softmax
SVMSVMSupport Vector Machine SVM
51

: :
Pcx Pxc P c Px

Pcx: x c Pc: c Pxc: c x

Px:x Pcx
x c
ID3C4.5CARTRFGBDT

clustering analysis : classlabeled data,; , taxonomy formation,
Kmeans : k
c :
1 c ;
2 k c
; 3;
4 c 23

Kmeans

52

CRISPDMcrossindustry standard process for data mining 6
business understanding

:
data understanding
:
data preparation
:

53

modeling

evaluation

deployment

SPSS
PSS SPSS SPSS EXCEL SPSS SPSS Logistic Probit
54

SPSS
SAS
SAS SAS GUI
R
R GNU C FORTRAN
Python
Python Python numpy scipy pandas matplotlib sklearn tensorflow

: 3040 :
55

AlphaGo
2016 3 Google DeepMind AlphaGo 4 1 AlphaGo AlphaGo

; :

:
1 2 3

;

Metadatadata about data
property ; ;;

56

; ;
:
1
2 MPP Apache Hadoop Hadoop NoSQL
3
4 ETL ; ;
5

:
?
??
??
?
? ?
:
Accuracy:
57

Completeness:
Consistency:
Uniqueness:
Integration:
Conformity:

:
1 ;
2 ;
3 ;
4 ;
5

58

:

1Confidentiality
secrecy
2Integrity
RSA X 2 2Xmalleably
3Availability

59

:
1:
2:
3:
4:
5: 6 7:
8:

:
1
2
3
4
5
6
7
8

60

:
1

2

3

4NAS
NAS NAS NAS
5

6
IT
7SAN
SAN
8

9

61

3 :
kryptos graphein Ron Rivest : :
1
visible watermarking
Steganography metadata
2
3

:
62

CDROM :
;
:

3.0
63

;,

2017

:
1
,
2
,
3
,,
,
:
1:

2:,

,,
3:,
,,, ,, ,,,

4:,
, , 3 4 , ,

5:,
,, ,

:
1:3G 4G
2: ITV

3:
4:
64

2017

:
1:,
;,,
,;, ,

2:,
,, ,
;
3:MB,
:, ;:

4:,,
OMB,, ,
,,
5:,
,
6:

65

:
1 IT
,
2,
,
3,
MSS ,

ITV ITV
:
1 2
3 4 5
6
7 8 9 10
:
1:
2 :
66

3ITV:EPG VOD EPG VOD ITV

:
IVR

67

1:

2:
3:
4:
68