Packages

c

io.delta.storage

LogStore

abstract class LogStore extends AnyRef

:: DeveloperApi ::

General interface for all critical file system operations required to read and write the Delta logs. The correctness is predicated on the atomicity and durability guarantees of the implementation of this interface. Specifically,

  • Atomic visibility of files: If isPartialWriteVisible is false, any file written through this store must be made visible atomically. In other words, this should not generate partial files.
  • Mutual exclusion: Only one writer must be able to create (or rename) a file at the final destination.
  • Consistent listing: Once a file has been written in a directory, all future listings for that directory must return that file.

All subclasses of this interface is required to have a constructor that takes Configuration as a single parameter. This constructor is used to dynamically create the LogStore.

LogStore and its implementations are not meant for direct access but for configuration based on storage system. See https://docs.delta.io/latest/delta-storage.html for details.

Since

1.0.0

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. LogStore
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new LogStore(initHadoopConf: Configuration)

Abstract Value Members

  1. abstract def isPartialWriteVisible(path: Path, hadoopConf: Configuration): Boolean

    :: DeveloperApi ::

    :: DeveloperApi ::

    Whether a partial write is visible for the underlying file system of path.

    Since

    1.0.0

    Exceptions thrown

    IOException if there's an issue resolving the FileSystem

  2. abstract def listFrom(path: Path, hadoopConf: Configuration): Iterator[FileStatus]

    :: DeveloperApi ::

    :: DeveloperApi ::

    List the paths in the same directory that are lexicographically greater or equal to (UTF-8 sorting) the given path. The result should also be sorted by the file name.

    Since

    1.0.0

    Exceptions thrown

    FileAlreadyExistsException if path directory can't be found

    IOException if there's an issue resolving the FileSystem

  3. abstract def read(path: Path, hadoopConf: Configuration): CloseableIterator[String]

    :: DeveloperApi ::

    :: DeveloperApi ::

    Load the given file and return an Iterator of lines, with line breaks removed from each line. Callers of this function are responsible to close the iterator if they are done with it.

    Since

    1.0.0

    Exceptions thrown

    IOException if there's an issue resolving the FileSystem

  4. abstract def resolvePathOnPhysicalStorage(path: Path, hadoopConf: Configuration): Path

    :: DeveloperApi ::

    :: DeveloperApi ::

    Resolve the fully qualified path for the given path.

    Since

    1.0.0

    Exceptions thrown

    IOException if there's an issue resolving the FileSystem

  5. abstract def write(path: Path, actions: Iterator[String], overwrite: Boolean, hadoopConf: Configuration): Unit

    :: DeveloperApi ::

    :: DeveloperApi ::

    Write the given actions to the given path with or without overwrite as indicated. Implementation must throw java.nio.file.FileAlreadyExistsException exception if the file already exists and overwrite = false. Furthermore, if isPartialWriteVisible returns false, implementation must ensure that the entire file is made visible atomically, that is, it should not generate partial files.

    Since

    1.0.0

    Exceptions thrown

    FileAlreadyExistsException if the file already exists and overwrite is false

    IOException if there's an issue resolving the FileSystem

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. def initHadoopConf(): Configuration

    :: DeveloperApi ::

    :: DeveloperApi ::

    Hadoop configuration that should only be used during initialization of LogStore. Each method should use their hadoopConf parameter rather than this (potentially outdated) hadoop configuration.

  12. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  13. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  14. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  15. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  16. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  17. def toString(): String
    Definition Classes
    AnyRef → Any
  18. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  19. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  20. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from AnyRef

Inherited from Any

Ungrouped