Package dev.vortex.spark.io

Class HadoopReadable

java.lang.Object

dev.vortex.io.PooledReadable

dev.vortex.spark.io.HadoopReadable

All Implemented Interfaces:

dev.vortex.io.NativeReadable, Closeable, AutoCloseable

public final class HadoopReadable
extends dev.vortex.io.PooledReadable

A NativeReadable that serves Vortex's reads from a Hadoop FileSystem.

Nested Class Summary

Nested classes/interfaces inherited from class dev.vortex.io.PooledReadable

dev.vortex.io.PooledReadable.PositionalStream, dev.vortex.io.PooledReadable.ScratchBytes

Method Summary

Modifier and Type	Method	Description
static HadoopReadable	open(org.apache.hadoop.conf.Configuration conf, String path)	Opens a readable over path, stating the file for its length.
static HadoopReadable	open(org.apache.hadoop.conf.Configuration conf, String path, long length)	Opens a readable over path.
protected dev.vortex.io.PooledReadable.PositionalStream	openStream()

Methods inherited from class dev.vortex.io.PooledReadable

close, length, name, readFully

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

open

public static HadoopReadable open(org.apache.hadoop.conf.Configuration conf,
 String path)

Opens a readable over path, stating the file for its length.

open

public static HadoopReadable open(org.apache.hadoop.conf.Configuration conf,
 String path,
 long length)

Opens a readable over path. A non-negative length is taken as authoritative — the driver already stats every file when it plans the scan, so executors need not stat them again.

openStream

protected dev.vortex.io.PooledReadable.PositionalStream openStream()
                                                            throws IOException

Specified by:

openStream in class dev.vortex.io.PooledReadable

Throws:

IOException