io.delta.kernel.defaults.engine

Class DefaultFileSystemClient

Object

io.delta.kernel.defaults.engine.DefaultFileSystemClient

All Implemented Interfaces:

FileSystemClient

public class DefaultFileSystemClient
extends Object
implements FileSystemClient

Default implementation of FileSystemClient based on Hadoop APIs. It takes a Hadoop Configuration object to interact with the file system. The following optional configurations can be set to customize the behavior of the client:

• io.delta.kernel.logStore.<scheme>.impl - The class name of the custom LogStore implementation to use for operations on storage systems with the specified scheme. For example, to use a custom LogStore for S3 storage objects:

  
   <property>
     <name>io.delta.kernel.logStore.s3.impl</name>
     <value>com.example.S3LogStore</value>
   </property>
  
   

  If not set, the default LogStore implementation for the scheme will be used.
• delta.enableFastS3AListFrom - Set to true to enable fast listing functionality when using a LogStore created for S3 storage objects.

The above list of options is not exhaustive. For a complete list of options, refer to the specific implementation of FileSystem.

Constructor Summary

Constructors

Constructor and Description
DefaultFileSystemClient(org.apache.hadoop.conf.Configuration hadoopConf) Create an instance of the default FileSystemClient implementation.

Method Summary

Modifier and Type	Method and Description
boolean	delete(String path) Delete the file at given path.
CloseableIterator<FileStatus>	listFrom(String filePath) List the paths in the same directory that are lexicographically greater or equal to (UTF-8 sorting) the given `path`.
boolean	mkdirs(String path) Create a directory at the given path including parent directories.
CloseableIterator<java.io.ByteArrayInputStream>	readFiles(CloseableIterator<FileReadRequest> readRequests) Return an iterator of byte streams one for each read request in readRequests.
String	resolvePath(String path) Resolve the given path to a fully qualified path.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DefaultFileSystemClient

public DefaultFileSystemClient(org.apache.hadoop.conf.Configuration hadoopConf)

Create an instance of the default FileSystemClient implementation.

Parameters:

hadoopConf - Configuration to use. List of options to customize the behavior of the client can be found in the class documentation.

Method Detail

listFrom

public CloseableIterator<FileStatus> listFrom(String filePath)
                                       throws java.io.IOException

Description copied from interface: FileSystemClient

List the paths in the same directory that are lexicographically greater or equal to (UTF-8 sorting) the given `path`. The result should also be sorted by the file name.

Specified by:

listFrom in interface FileSystemClient

Parameters:

filePath - Fully qualified path to a file

Returns:

Closeable iterator of files. It is the responsibility of the caller to close the iterator.

Throws:

java.io.FileNotFoundException - if the file at the given path is not found

java.io.IOException - for any other IO error.

resolvePath

public String resolvePath(String path)
                   throws java.io.IOException

Description copied from interface: FileSystemClient

Resolve the given path to a fully qualified path.

Specified by:

resolvePath in interface FileSystemClient

Parameters:

path - Input path

Returns:

Fully qualified path.

Throws:

java.io.FileNotFoundException - If the given path doesn't exist.

java.io.IOException - for any other IO error.

readFiles

public CloseableIterator<java.io.ByteArrayInputStream> readFiles(CloseableIterator<FileReadRequest> readRequests)

Description copied from interface: FileSystemClient

Return an iterator of byte streams one for each read request in readRequests. The returned streams are in the same order as the given FileReadRequests. It is the responsibility of the caller to close each returned stream.

Specified by:

readFiles in interface FileSystemClient

Parameters:

readRequests - Iterator of read requests

Returns:

Data for each request as one ByteArrayInputStream.

mkdirs

public boolean mkdirs(String path)
               throws java.io.IOException

Description copied from interface: FileSystemClient

Create a directory at the given path including parent directories. This mimicks the behavior of `mkdir -p` in Unix.

Specified by:

mkdirs in interface FileSystemClient

Parameters:

path - Full qualified path to create a directory at.

Returns:

true if the directory was created successfully, false otherwise.

Throws:

java.io.IOException - for any IO error.

delete

public boolean delete(String path)
               throws java.io.IOException

Description copied from interface: FileSystemClient

Delete the file at given path.

Specified by:

delete in interface FileSystemClient

Parameters:

path - the path to delete. If path is a directory throws an exception.

Returns:

true if delete is successful else false.

Throws:

java.io.IOException - for any IO error.