CS代写 • Command-line interface • Programmatic access

Programmatic access: Common parts

Filesystem reference
First things first: Get a reference to the underlying filesystem.
abstract class org.apache.hadoop.fs.FileSystem
→ org.apache.hadoop.fs.LocalFileSystem
→ org.apache.hadoop.fs.DistributedFileSystem
→ org.apache.hadoop.fs.HftpFileSystem → org.apache.hadoop.fs.FTPFileSystem → org.apache.hadoop.fs.s3.S3FileSystem
→ org.apache.hadoop.fs.kfs.KosmosFileSystem
Note: In general you should strive to write your code against the FileSystem abstract class, to retain portability across filesystems

Filesystem reference import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem; Configuration conf = new Configuration();
conf.set(“fs.defaultFS”, “hdfs://namenode.fqdn:8020”); // Not necessary when // appropriate config. files
FileSystem fs = FileSystem.get(conf); […]
fs.close();
• Configuration just a wrapper for java.util.Properties
// exist on the client side
• Configuration’s default constructor loads information from local config. files
• Use an explicit namenode URI if executing code on a remote cluster and no such files have been provided (line 5)
• Use conf.addResource(…) to load non-standard config. (XML) files

• URI-awaresyntax
• Secondstep:geta“reference”toafile/directory path
– org.apache.hadoop.fs.Path
– Path f = new Path(“file:///user/username/file.txt”);
– Pathf=new Path(“hdfs://localhost:8020/user/username/file.txt”);
– Path f = new Path(“/user/username/file.txt”);
– Path f = new Path(“file.txt”);

Programmatic access: Basic operations

// File already exists. Do something…
Create empty file Create/overwrite file
boolean isCreated = fs.createNewFile(file); // … or …
FSDataOutputStream out = fs.create(file);
// … or …
Do not overwrite file.txt
Directory/File Creation
import org.apache.hadoop.fs.FSDataOutputStream; […]
Path dir = new Path(“/path/to/dir”);
if (fs.mkdirs(dir) == false) // Create directory structure, if not there
// Error creating directory structure. Bail out…
Path file = new Path(“/path/to/dir/file.txt”); // … or …
Absolute path Filename relative to dir !!
Path file = new Path(dir, “file.txt”); if (fs.exists(file))
FSDataOutputStream out = fs.create(file, false); // … or …
FSDataOutputStream out = fs.create(file, new FsPermission(“u=rw,g=r,o-rwx”), true, 1048576, (short)2, 134217728L, null); // ?!?

Input • Read data from a file:
import java.io.InputStream;
import org.apache.hadoop.fs.FSDataInputStream;
InputStream in = fs.open(file);
// … or …
FSDataInputStream in = fs.open(file);
String str = in.readUTF(); int i = in.readInt();
double d = in.readDouble();
in.close();
• Also check out org.apache.commons.io.IOUtils

Output • Write data to a file:
import org.apache.hadoop.fs.FSDataOutputStream; […]
FSDataOutputStream out = fs.create(file); out.writeUTF(“1024”); out.writeInt(1024);
out.writeDouble(1.0);
out.close();

public class FileSystemDoubleCat {
public static void main(String[] args) throws Exception {
String uri = args[0];
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(URI.create(uri), conf);
FSDataInputStream in = null;
in = fs.open(new Path(uri));
IOUtils.copyBytes(in, System.out, 4096, false); }
finally { IOUtils.closeStream(in);
Here’s the result of running it on a small file:
% hadoop FileSystemDoubleCat hdfs://localhost/user/username/lyrics.txt Mesmerises one of the wedding guests. Stay here and listen to the nightmares of the Sea.
And the music plays on, as the bride passes by. Caught by his spell and the Mariner tells his tale. Driven south to the land of the snow and ice, to a place where nobody’s been.

• Moving around within a file only supported for reading
FSDataInputStream in = fs.open(file);
in.seek(1024); // Go to position 1024 (in bytes) String someString = in.readUTF(); // Read a string in.skip(1024); // Skip next 1024 bytes
int position = in.getPos(); // Get current position in.seek(0); // Go back to start
• HDFS built for streaming access ⇒ Seeking not a good idea (expensive)!
• Seeking not available for writes ⇒ Only append is supported…
FSDataOutputStream out = fs.append(file); out.writeUTF(“some text”);
… but not by all FileSystem subclasses!

public class FileSystemDoubleCat {
public static void main(String[] args) throws Exception {
String uri = args[0];
Configuration conf = new Configuration(); FileSystem fs = FileSystem.get(URI.create(uri), conf);
FSDataInputStream in = null;
in = fs.open(new Path(uri));
IOUtils.copyBytes(in, System.out, 4096, false); in.seek(0); // go back to the start of the file IOUtils.copyBytes(in, System.out, 4096, false);
finally { IOUtils.closeStream(in); }
Here’s a mock result of running it on a small file:
% hadoop FileSystemDoubleCat hdfs://localhost/user/username/lyrics.txt
Mesmerises one of the wedding guests. Stay here and listen to the nightmares of the Sea.
And the music plays on, as the bride passes by. Caught by his spell and the Mariner tells his tale. Driven south to the land of the snow and ice, to a place where nobody’s been.
Mesmerises one of the wedding guests. Stay here and listen to the nightmares of the Sea.
And the music plays on, as the bride passes by. Caught by his spell and the Mariner tells his tale. Driven south to the land of the snow and ice, to a place where nobody’s been.

Directory/file renaming and deletion Path dir = new Path(“/path/to/dir”);
Path file = new Path(dir, “file.txt”);
// Move file.txt to …/other/dir and rename it
Path dst = new Path(“/path/to/other/dir/otherfile.txt”); fs.rename(file, dst);
// Delete the new file
fs.delete(dst);
// Delete the original directory recursively
fs.delete(dir, true);

• Apache Hadoop Documentation – User Guide:
– Command-Line Interface:
Further Reading
http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html
– Architecture:
http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html
http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html
– Java API:
http://hadoop.apache.org/docs/stable/api/index.html
(look under org.apache.hadoop.fs) • Cloudera Library
– http://www.cloudera.com/content/cloudera/en/documentation.html#ClouderaDocumentation • Google…

程序代写 CS代考 加微信: powcoder QQ: 1823890830 Email: powcoder@163.com