Type Members

1.  trait CloseableIterator[T] extends Iterator[T] with Closeable

   :: DeveloperApi ::

   :: DeveloperApi ::

   An iterator that may contain resources which should be released after use. Users of CloseableIterator are responsible for closing the iterator if they are done with it.

   Annotations

   @DeveloperApi()

   Since

   1.0.0

2.  class GCSLogStore extends HadoopFileSystemLogStore with Logging

   :: Unstable ::

   :: Unstable ::

   The LogStore implementation for GCS, which uses gcs-connector to provide the necessary atomic and durability guarantees:

   1. Atomic Visibility: Read/read-after-metadata-update/delete are strongly consistent for GCS.

   2. Consistent Listing: GCS guarantees strong consistency for both object and bucket listing operations. https://cloud.google.com/storage/docs/consistency

   3. Mutual Exclusion: Preconditions are used to handle race conditions.

   Regarding file creation, this implementation: - Opens a stream to write to GCS otherwise. - Throws FileAlreadyExistsException if file exists and overwrite is false. - Assumes file writing to be all-or-nothing, irrespective of overwrite option.

   Annotations

   @Unstable()

   Note

   This class is not meant for direct access but for configuration based on storage system. See https://docs.delta.io/latest/delta-storage.html for details.

3.  class IBMCOSLogStore extends HadoopFileSystemLogStore

   :: Unstable ::

   :: Unstable ::

   LogStore implementation for IBM Cloud Object Storage.

   We assume the following from COS's FileSystem implementations: - Write on COS is all-or-nothing, whether overwrite or not. - Write is atomic. Note: Write is atomic when using the Stocator v1.1.1+ - Storage Connector for Apache Spark (https://github.com/CODAIT/stocator) by setting the configuration fs.cos.atomic.write to true (for more info see the documentation for Stocator) - List-after-write is consistent.

   Annotations

   @Unstable()

   Note

   This class is not meant for direct access but for configuration based on storage system. See https://docs.delta.io/latest/delta-storage.html for details.

4.  abstract class LogStore extends AnyRef

   :: DeveloperApi ::

   :: DeveloperApi ::

   General interface for all critical file system operations required to read and write the Delta logs. The correctness is predicated on the atomicity and durability guarantees of the implementation of this interface. Specifically,

   1. Atomic visibility of files: If isPartialWriteVisible is false, any file written through this store must be made visible atomically. In other words, this should not generate partial files.

   2. Mutual exclusion: Only one writer must be able to create (or rename) a file at the final destination.

   3. Consistent listing: Once a file has been written in a directory, all future listings for that directory must return that file.

   All subclasses of this interface is required to have a constructor that takes Configuration as a single parameter. This constructor is used to dynamically create the LogStore.

   LogStore and its implementations are not meant for direct access but for configuration based on storage system. See https://docs.delta.io/latest/delta-storage.html for details.

   Annotations

   @DeveloperApi()

   Since

   1.0.0

5.  class OracleCloudLogStore extends HadoopFileSystemLogStore

   :: Unstable ::

   :: Unstable ::

   LogStore implementation for OCI (Oracle Cloud Infrastructure).

   We assume the following from OCI (Oracle Cloud Infrastructure)'s BmcFilesystem implementations: - Rename without overwrite is atomic. - List-after-write is consistent.

   Regarding file creation, this implementation: - Uses atomic rename when overwrite is false; if the destination file exists or the rename fails, throws an exception. - Uses create-with-overwrite when overwrite is true. This does not make the file atomically visible and therefore the caller must handle partial files.

   Annotations

   @Unstable()

   Note

   This class is not meant for direct access but for configuration based on storage system. See https://docs.delta.io/latest/delta-storage.html for details.