abstract class LogStore extends AnyRef
:: DeveloperApi ::
General interface for all critical file system operations required to read and write the Delta logs. The correctness is predicated on the atomicity and durability guarantees of the implementation of this interface. Specifically,
1. Atomic visibility of files: If isPartialWriteVisible is false, any file written through this store must be made visible atomically. In other words, this should not generate partial files.
2. Mutual exclusion: Only one writer must be able to create (or rename) a file at the final destination.
3. Consistent listing: Once a file has been written in a directory, all future listings for that directory must return that file.
All subclasses of this interface is required to have a constructor that takes Configuration as a single parameter. This constructor is used to dynamically create the LogStore.
LogStore and its implementations are not meant for direct access but for configuration based on storage system. See https://docs.delta.io/latest/delta-storage.html for details.
- Since
1.0.0
- Alphabetic
- By Inheritance
- LogStore
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
- new LogStore(initHadoopConf: Configuration)
Abstract Value Members
-
abstract
def
isPartialWriteVisible(path: Path, hadoopConf: Configuration): Boolean
:: DeveloperApi ::
:: DeveloperApi ::
Whether a partial write is visible for the underlying file system of
path
.- Since
1.0.0
-
abstract
def
listFrom(path: Path, hadoopConf: Configuration): Iterator[FileStatus]
:: DeveloperApi ::
:: DeveloperApi ::
List the paths in the same directory that are lexicographically greater or equal to (UTF-8 sorting) the given
path
. The result should also be sorted by the file name.- Since
1.0.0
-
abstract
def
read(path: Path, hadoopConf: Configuration): CloseableIterator[String]
:: DeveloperApi ::
:: DeveloperApi ::
Load the given file and return an
Iterator
of lines, with line breaks removed from each line. Callers of this function are responsible to close the iterator if they are done with it.- Since
1.0.0
-
abstract
def
resolvePathOnPhysicalStorage(path: Path, hadoopConf: Configuration): Path
:: DeveloperApi ::
:: DeveloperApi ::
Resolve the fully qualified path for the given
path
.- Since
1.0.0
-
abstract
def
write(path: Path, actions: Iterator[String], overwrite: Boolean, hadoopConf: Configuration): Unit
:: DeveloperApi ::
:: DeveloperApi ::
Write the given
actions
to the givenpath
with or without overwrite as indicated. Implementation must throw java.nio.file.FileAlreadyExistsException exception if the file already exists and overwrite = false. Furthermore, if isPartialWriteVisible returns false, implementation must ensure that the entire file is made visible atomically, that is, it should not generate partial files.- Since
1.0.0
Concrete Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
initHadoopConf(): Configuration
:: DeveloperApi ::
:: DeveloperApi ::
Hadoop configuration that should only be used during initialization of LogStore. Each method should use their
hadoopConf
parameter rather than this (potentially outdated) hadoop configuration. -
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()