Package dev.vortex.spark.write

Class SparkToArrowSchema

java.lang.Object
dev.vortex.spark.write.SparkToArrowSchema
public final class SparkToArrowSchema extends Object
Utility class for converting Spark SQL schemas to Arrow schemas.

This enables the conversion of Spark DataFrames to Arrow format for writing to Vortex files.

  • Method Summary

    Modifier and Type
    Method
    Description
    static dev.vortex.relocated.org.apache.arrow.vector.types.pojo.Schema
    convert(org.apache.spark.sql.types.StructType sparkSchema)
    Converts a Spark StructType schema to an Arrow Schema.
    static boolean
    supportsDataType(org.apache.spark.sql.types.DataType dataType)
    Returns whether Vortex's Spark writer can represent this Spark SQL type.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Method Details

    • supportsDataType

      public static boolean supportsDataType(org.apache.spark.sql.types.DataType dataType)
      Returns whether Vortex's Spark writer can represent this Spark SQL type.

    • convert

      public static dev.vortex.relocated.org.apache.arrow.vector.types.pojo.Schema convert(org.apache.spark.sql.types.StructType sparkSchema)
      Converts a Spark StructType schema to an Arrow Schema.
      Parameters:
      sparkSchema - the Spark schema to convert
      Returns:
      the corresponding Arrow schema