Package dev.vortex.spark.read

Class VortexAggregateReaderFactory

java.lang.Object
org.apache.spark.sql.execution.datasources.v2.FilePartitionReaderFactory
dev.vortex.spark.read.VortexAggregateReaderFactory
All Implemented Interfaces:
Serializable, org.apache.spark.sql.connector.read.PartitionReaderFactory
public final class VortexAggregateReaderFactory extends org.apache.spark.sql.execution.datasources.v2.FilePartitionReaderFactory implements Serializable
Produces one footer-backed partial COUNT(*) row per Vortex file.

A listed file without VortexFile.EXTENSION is not part of the dataset and produces no row.

See Also:

  • Constructor Summary

    Constructors
    Constructor
    Description
    VortexAggregateReaderFactory(org.apache.spark.sql.catalyst.FileSourceOptions fileOptions, VortexIo io, VortexOptions formatOptions, org.apache.spark.sql.types.StructType aggregateSchema, org.apache.spark.sql.types.StructType partitionSchema, org.apache.spark.sql.connector.expressions.aggregate.Aggregation aggregation)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    org.apache.spark.sql.connector.read.PartitionReader<org.apache.spark.sql.vectorized.ColumnarBatch>
    buildColumnarReader(org.apache.spark.sql.execution.datasources.PartitionedFile file)
     
    org.apache.spark.sql.connector.read.PartitionReader<org.apache.spark.sql.catalyst.InternalRow>
    buildReader(org.apache.spark.sql.execution.datasources.PartitionedFile file)
     
    org.apache.spark.sql.catalyst.FileSourceOptions
    options()
     
    boolean
    supportColumnarReads(org.apache.spark.sql.connector.read.InputPartition partition)
     

    Methods inherited from class org.apache.spark.sql.execution.datasources.v2.FilePartitionReaderFactory

    createColumnarReader, createReader

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • VortexAggregateReaderFactory

      public VortexAggregateReaderFactory(org.apache.spark.sql.catalyst.FileSourceOptions fileOptions, VortexIo io, VortexOptions formatOptions, org.apache.spark.sql.types.StructType aggregateSchema, org.apache.spark.sql.types.StructType partitionSchema, org.apache.spark.sql.connector.expressions.aggregate.Aggregation aggregation)

  • Method Details

    • options

      public org.apache.spark.sql.catalyst.FileSourceOptions options()
      Specified by:
      options in class org.apache.spark.sql.execution.datasources.v2.FilePartitionReaderFactory

    • buildReader

      public org.apache.spark.sql.connector.read.PartitionReader<org.apache.spark.sql.catalyst.InternalRow> buildReader(org.apache.spark.sql.execution.datasources.PartitionedFile file)
      Specified by:
      buildReader in class org.apache.spark.sql.execution.datasources.v2.FilePartitionReaderFactory

    • buildColumnarReader

      public org.apache.spark.sql.connector.read.PartitionReader<org.apache.spark.sql.vectorized.ColumnarBatch> buildColumnarReader(org.apache.spark.sql.execution.datasources.PartitionedFile file)
      Overrides:
      buildColumnarReader in class org.apache.spark.sql.execution.datasources.v2.FilePartitionReaderFactory

    • supportColumnarReads

      public boolean supportColumnarReads(org.apache.spark.sql.connector.read.InputPartition partition)
      Specified by:
      supportColumnarReads in interface org.apache.spark.sql.connector.read.PartitionReaderFactory
      Overrides:
      supportColumnarReads in class org.apache.spark.sql.execution.datasources.v2.FilePartitionReaderFactory