Package dev.vortex.spark.io

Class VortexFile

java.lang.Object

dev.vortex.spark.io.VortexFile

All Implemented Interfaces:

Serializable

public final class VortexFile
extends Object
implements Serializable

A Vortex file and, when a listing already reported it, its size on storage.

See Also:

• Serialized Form

Field Summary

Fields

Modifier and Type	Field	Description
static final String	EXTENSION	The extension every Vortex data file must carry.
static final long	UNKNOWN_LENGTH	Size of a file nothing has stat'ed yet.

Constructor Summary

Constructors

Constructor	Description
VortexFile(String path, long length)

Method Summary

Modifier and Type	Method	Description
static boolean	hasVortexExtension(String path)	Returns whether path names a Vortex data file.
long	length()
String	path()
static void	requireVortexExtension(String path)	Fails unless path names a Vortex data file.
static VortexFile	unsized(String path)	A file whose size is not known yet.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

EXTENSION

public static final String EXTENSION

The extension every Vortex data file must carry.

See Also:

• Constant Field Values

UNKNOWN_LENGTH

public static final long UNKNOWN_LENGTH

Size of a file nothing has stat'ed yet. Whoever opens it pays for the stat, if it needs one at all.

See Also:

• Constant Field Values

Constructor Details

VortexFile

public VortexFile(String path,
 long length)

Parameters:

path - the path as the caller spelled it, not as Hadoop qualifies it

length - size in bytes, or UNKNOWN_LENGTH for a path named directly rather than listed

Method Details

path

public String path()

length

public long length()

unsized

public static VortexFile unsized(String path)

A file whose size is not known yet.

hasVortexExtension

public static boolean hasVortexExtension(String path)

Returns whether path names a Vortex data file.

Spark's file index hides names that begin with _ or ., but it keeps _metadata and _common_metadata, and it keeps every other extension. The connector reads only files that carry EXTENSION, so this is the one test that decides what belongs to a Vortex dataset. The readers skip every other listed file.

requireVortexExtension

public static void requireVortexExtension(String path)

Fails unless path names a Vortex data file.

Throws:

IllegalArgumentException - if the path does not carry EXTENSION