Package io.delta.kernel.defaults.engine.hadoopio

Class HadoopFileIO

Object
io.delta.kernel.defaults.engine.hadoopio.HadoopFileIO
All Implemented Interfaces:
FileIO
public class HadoopFileIO extends Object implements FileIO
Implementation of FileIO based on Hadoop APIs.

  • Constructor Details

    • HadoopFileIO

      public HadoopFileIO(org.apache.hadoop.conf.Configuration hadoopConf)

  • Method Details

    • listFrom

      public CloseableIterator<FileStatus> listFrom(String filePath) throws IOException
      Description copied from interface: FileIO
      List the paths in the same directory that are lexicographically greater or equal to (UTF-8 sorting) the given `path`. The result should also be sorted by the file name.
      Specified by:
      listFrom in interface FileIO
      Parameters:
      filePath - Fully qualified path to a file
      Returns:
      Closeable iterator of files. It is the responsibility of the caller to close the iterator.
      Throws:
      FileNotFoundException - if the file at the given path is not found
      IOException - for any other IO error.

    • getFileStatus

      public FileStatus getFileStatus(String path) throws IOException
      Description copied from interface: FileIO
      Get the metadata of the file at the given path.
      Specified by:
      getFileStatus in interface FileIO
      Parameters:
      path - Fully qualified path to the file.
      Returns:
      Metadata of the file.
      Throws:
      IOException - for any IO error.

    • resolvePath

      public String resolvePath(String path) throws IOException
      Description copied from interface: FileIO
      Resolve the given path to a fully qualified path.
      Specified by:
      resolvePath in interface FileIO
      Parameters:
      path - Input path
      Returns:
      Fully qualified path.
      Throws:
      FileNotFoundException - If the given path doesn't exist.
      IOException - for any other IO error.

    • mkdirs

      public boolean mkdirs(String path) throws IOException
      Description copied from interface: FileIO
      Create a directory at the given path including parent directories. This mimics the behavior of `mkdir -p` in Unix.
      Specified by:
      mkdirs in interface FileIO
      Parameters:
      path - Full qualified path to create a directory at.
      Returns:
      true if the directory was created successfully, false otherwise.
      Throws:
      IOException - for any IO error.

    • newInputFile

      public InputFile newInputFile(String path, long fileSize)
      Description copied from interface: FileIO
      Get an InputFile for file at given path which can be used to read the file from any arbitrary position in the file.
      Specified by:
      newInputFile in interface FileIO
      Parameters:
      path - Fully qualified path to the file.
      fileSize - Size of the file in bytes.
      Returns:
      InputFile instance.

    • newOutputFile

      public OutputFile newOutputFile(String path)
      Description copied from interface: FileIO
      Create a OutputFile to write new file at the given path.
      Specified by:
      newOutputFile in interface FileIO
      Parameters:
      path - Fully qualified path to the file.
      Returns:
      OutputFile instance which can be used to write to the file.

    • delete

      public boolean delete(String path) throws IOException
      Description copied from interface: FileIO
      Delete the file at given path.
      Specified by:
      delete in interface FileIO
      Parameters:
      path - the path to delete. If path is a directory throws an exception.
      Returns:
      true if delete is successful else false.
      Throws:
      IOException - for any IO error.

    • getConf

      public Optional<String> getConf(String confKey)
      Description copied from interface: FileIO
      Get the configuration value for the given key.

      TODO: should be in a separate interface? may be called ConfigurationProvider?

      Specified by:
      getConf in interface FileIO
      Parameters:
      confKey - configuration key name
      Returns:
      If no such value is present, an Optional.empty() is returned.