Package dev.vortex.spark.read

Class VortexPartitionReaderFactory

java.lang.Object
org.apache.spark.sql.execution.datasources.v2.FilePartitionReaderFactory
dev.vortex.spark.read.VortexPartitionReaderFactory
All Implemented Interfaces:
Serializable, org.apache.spark.sql.connector.read.PartitionReaderFactory
public final class VortexPartitionReaderFactory extends org.apache.spark.sql.execution.datasources.v2.FilePartitionReaderFactory implements Serializable
Produces one Vortex reader for each file selected by Spark's file index.

A listed file without VortexFile.EXTENSION is not part of the dataset and reads as empty.

See Also:

  • Constructor Summary

    Constructors
    Constructor
    Description
    VortexPartitionReaderFactory(org.apache.spark.sql.catalyst.FileSourceOptions fileOptions, VortexIo io, VortexOptions formatOptions, org.apache.spark.sql.types.StructType dataSchema, org.apache.spark.sql.types.StructType readDataSchema, org.apache.spark.sql.types.StructType readPartitionSchema, org.apache.spark.sql.sources.Filter[] pushedFilters, boolean caseSensitive)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    org.apache.spark.sql.connector.read.PartitionReader<org.apache.spark.sql.vectorized.ColumnarBatch>
    buildColumnarReader(org.apache.spark.sql.execution.datasources.PartitionedFile file)
     
    org.apache.spark.sql.connector.read.PartitionReader<org.apache.spark.sql.catalyst.InternalRow>
    buildReader(org.apache.spark.sql.execution.datasources.PartitionedFile file)
     
    org.apache.spark.sql.catalyst.FileSourceOptions
    options()
     
    boolean
    supportColumnarReads(org.apache.spark.sql.connector.read.InputPartition partition)
     

    Methods inherited from class org.apache.spark.sql.execution.datasources.v2.FilePartitionReaderFactory

    createColumnarReader, createReader

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • VortexPartitionReaderFactory

      public VortexPartitionReaderFactory(org.apache.spark.sql.catalyst.FileSourceOptions fileOptions, VortexIo io, VortexOptions formatOptions, org.apache.spark.sql.types.StructType dataSchema, org.apache.spark.sql.types.StructType readDataSchema, org.apache.spark.sql.types.StructType readPartitionSchema, org.apache.spark.sql.sources.Filter[] pushedFilters, boolean caseSensitive)

  • Method Details

    • options

      public org.apache.spark.sql.catalyst.FileSourceOptions options()
      Specified by:
      options in class org.apache.spark.sql.execution.datasources.v2.FilePartitionReaderFactory

    • buildReader

      public org.apache.spark.sql.connector.read.PartitionReader<org.apache.spark.sql.catalyst.InternalRow> buildReader(org.apache.spark.sql.execution.datasources.PartitionedFile file)
      Specified by:
      buildReader in class org.apache.spark.sql.execution.datasources.v2.FilePartitionReaderFactory

    • buildColumnarReader

      public org.apache.spark.sql.connector.read.PartitionReader<org.apache.spark.sql.vectorized.ColumnarBatch> buildColumnarReader(org.apache.spark.sql.execution.datasources.PartitionedFile file)
      Overrides:
      buildColumnarReader in class org.apache.spark.sql.execution.datasources.v2.FilePartitionReaderFactory

    • supportColumnarReads

      public boolean supportColumnarReads(org.apache.spark.sql.connector.read.InputPartition partition)
      Specified by:
      supportColumnarReads in interface org.apache.spark.sql.connector.read.PartitionReaderFactory
      Overrides:
      supportColumnarReads in class org.apache.spark.sql.execution.datasources.v2.FilePartitionReaderFactory