Class AzureFileSystemProvider

java.lang.Object
java.nio.file.spi.FileSystemProvider
com.azure.storage.blob.nio.AzureFileSystemProvider
public final class AzureFileSystemProvider extends FileSystemProvider
The AzureFileSystemProvider is Azure Storage's implementation of the nio interface on top of Azure Blob Storage.

Particular care should be taken when working with a remote storage service. This implementation makes no guarantees on behavior or state should other processes operate on the same data concurrently; file systems from this provider will assume they have exclusive access to their data and will behave without regard for potential of interfering applications. Moreover, remote file stores introduce higher latencies. Therefore, additional consideration should be given to managing concurrency: race conditions are more likely to manifest and network failures occur more frequently than disk failures. These and other such distributed application scenarios must be considered when working with this file system. While the AzureFileSystem will ensure it takes appropriate steps towards robustness and reliability, the application developer must design around these failure scenarios and have fallback and retry options available.

The Azure Blob Storage service backing these APIs is not a true FileSystem, nor is it the goal of this implementation to force Azure Blob Storage to act like a full-fledged file system. Some APIs and scenarios will remain unsupported indefinitely until they may be sensibly implemented. Other APIs may experience lower performance than is expected because of the number of network requests needed to ensure correctness. The javadocs for each type and method should also be read carefully to understand what guarantees are made and how they may differ from the contract defined by FileSystemProvider.

The scheme for this provider is "azb", and the format of the URI to identify an AzureFileSystem is "azb://?endpoint=<endpoint>". The endpoint of the Storage account is used to uniquely identify the filesystem.

An AzureFileSystem is backed by an account. An AzureFileStore is backed by a container. Any number of containers may be specified as file stores upon creation of the file system. When a file system is created, it will try to retrieve the properties of each container to ensure connection to the account. If any of the containers does not exist, it will be created. Failure to access or create containers as necessary will result in an exception and failure to create the file system. Any data existing in the containers will be preserved and accessible via the file system, though customers should be aware that it must be in a format understandable by the types in this package or behavior will be undefined.

newFileSystem(URI, Map) will check for the following keys in the configuration map and expect the named types. Any entries not listed here will be ignored. Note that AzureFileSystem has public constants defined for each of the keys for convenience. Most values are documented in the blob package. Any values which are unique to nio will be documented here.

  • AzureStorageSharedKeyCredential:StorageSharedKeyCredential
  • AzureStorageSasTokenCredential:AzureSasCredential
  • AzureStorageHttpLogDetailLevel:HttpLogDetailLevel
  • AzureStorageMaxTries:Integer
  • AzureStorageTryTimeout:Integer
  • AzureStorageRetryDelayInMs:Long
  • AzureStorageMaxRetryDelayInMs:Long
  • AzureStorageRetryPolicyType:RetryPolicyType
  • AzureStorageSecondaryHost:String
  • AzureStorageSecondaryHost:Integer
  • AzureStorageBlockSize:Long
  • AzureStoragePutBlobThreshold:Long
  • AzureStorageMaxConcurrencyPerRequest:Integer
  • AzureStorageDownloadResumeRetries:Integer
  • AzureStorageFileStores:String
  • AzureStorageSkipInitialContainerCheck:Boolean. Indicates that the initial check which confirms the existence of the containers meant to act as file stores should be skipped. This can be useful in cases where a sas token that is scoped to only one file is used to authenticate.

Either an account key or a sas token must be specified. If both are provided, the account key will be preferred. If a sas token is specified, the customer must take care that it has appropriate permissions to perform the actions demanded of the file system in a given workflow, including the initial connection check specified above. The same token will be applied to all operations.

An iterable of file stores must also be provided; each entry should simply be the name of a container. The first container listed will be considered the default file store and the root directory of which will be the file system's default directory. All other values listed are used to configure the underlying BlobServiceClient. Please refer to that type for more information on these values.

See Also:

  • Field Details

    • CONTENT_TYPE

      public static final String CONTENT_TYPE
      A helper for setting the HTTP properties when creating a directory.
      See Also:

    • CONTENT_DISPOSITION

      public static final String CONTENT_DISPOSITION
      A helper for setting the HTTP properties when creating a directory.
      See Also:

    • CONTENT_LANGUAGE

      public static final String CONTENT_LANGUAGE
      A helper for setting the HTTP properties when creating a directory.
      See Also:

    • CONTENT_ENCODING

      public static final String CONTENT_ENCODING
      A helper for setting the HTTP properties when creating a directory.
      See Also:

    • CONTENT_MD5

      public static final String CONTENT_MD5
      A helper for setting the HTTP properties when creating a directory.
      See Also:

    • CACHE_CONTROL

      public static final String CACHE_CONTROL
      A helper for setting the HTTP properties when creating a directory.
      See Also:

  • Constructor Details

    • AzureFileSystemProvider

      public AzureFileSystemProvider()
      Creates an AzureFileSystemProvider.

  • Method Details

    • getScheme

      public String getScheme()
      Returns the URI scheme that identifies this provider: "azb".
      Specified by:
      getScheme in class FileSystemProvider
      Returns:
      "azb"

    • newFileSystem

      public FileSystem newFileSystem(URI uri, Map<String,?> config) throws IOException
      Constructs a new FileSystem object identified by a URI.

      The format of a URI identifying a file system is "azb://?endpoint=<endpoint>".

      Once closed, a file system with the same identifier may be reopened.

      Specified by:
      newFileSystem in class FileSystemProvider
      Parameters:
      uri - URI reference
      config - A map of provider specific properties to configure the file system
      Returns:
      a new file system.
      Throws:
      IllegalArgumentException - If the pre-conditions for the uri parameter aren't met, or the env parameter does not contain properties required by the provider, or a property value is invalid.
      IOException - If an I/O error occurs.
      SecurityException - never
      FileSystemAlreadyExistsException - If the file system has already been created.

    • getFileSystem

      public FileSystem getFileSystem(URI uri)
      Returns an existing FileSystem created by this provider.

      The format of a URI identifying a file system is "azb://?endpoint=&lt;endpoint&gt;".

      Trying to retrieve a closed file system will throw a FileSystemNotFoundException. Once closed, a file system with the same identifier may be reopened.

      Specified by:
      getFileSystem in class FileSystemProvider
      Parameters:
      uri - URI reference
      Returns:
      the file system
      Throws:
      IllegalArgumentException - If the pre-conditions for the uri parameter aren't met
      FileSystemNotFoundException - If the file system already exists
      SecurityException - never

    • getPath

      public Path getPath(URI uri)
      Return a Path object by converting the given URI. The resulting Path is associated with a FileSystem that already exists.
      Specified by:
      getPath in class FileSystemProvider
      Parameters:
      uri - The URI to convert
      Returns:
      The path identified by the URI.
      Throws:
      IllegalArgumentException - If the URI scheme does not identify this provider or other preconditions on the uri parameter do not hold
      FileSystemNotFoundException - if the file system identified by the query does not exist
      SecurityException - never
      See Also:

    • newByteChannel

      public SeekableByteChannel newByteChannel(Path path, Set<? extends OpenOption> set, FileAttribute<?>... fileAttributes) throws IOException
      Opens or creates a file, returning a seekable byte channel to access the file.

      This method is primarily offered to support some jdk convenience methods such as Files.createFile(Path, FileAttribute[]) which requires opening a channel and closing it. A channel may only be opened in read mode OR write mode. It may not be opened in read/write mode. Seeking is supported for reads, but not for writes. Modifications to existing files is not permitted--only creating new files or overwriting existing files.

      This type is not threadsafe to prevent having to hold locks across network calls.

      Specified by:
      newByteChannel in class FileSystemProvider
      Parameters:
      path - the path of the file to open
      set - options specifying how the file should be opened
      fileAttributes - an optional list of file attributes to set atomically when creating the directory
      Returns:
      a new seekable byte channel
      Throws:
      UnsupportedOperationException - Operation is not supported.
      IllegalArgumentException - if the set contains an invalid combination of options
      FileAlreadyExistsException - if a file of that name already exists and the CREATE_NEW option is specified (optional specific exception)
      IOException - If an I/O error occurs.
      SecurityException - never

    • newInputStream

      public InputStream newInputStream(Path path, OpenOption... options) throws IOException
      Opens an InputStream to the given path.

      The stream will not attempt to read or buffer the entire file. However, when fetching data, it will always request the same size chunk of several MB to prevent network thrashing on small reads. Mark and reset are supported.

      Only StandardOpenOption.READ is supported. Any other option will throw.

      Overrides:
      newInputStream in class FileSystemProvider
      Parameters:
      path - the path to the file to open
      options - options specifying how the file is opened
      Returns:
      a new input stream
      Throws:
      IllegalArgumentException - if an invalid combination of options is specified
      UnsupportedOperationException - if an unsupported option is specified
      IOException - If an I/O error occurs.
      SecurityException - never

    • newOutputStream

      public OutputStream newOutputStream(Path path, OpenOption... options) throws IOException
      Opens an OutputStream to the given path. The resulting file will be stored as a block blob.

      The only supported options are StandardOpenOption.CREATE, StandardOpenOption.CREATE_NEW, StandardOpenOption.WRITE, StandardOpenOption.TRUNCATE_EXISTING. Any other options will throw an UnsupportedOperationException. WRITE and TRUNCATE_EXISTING must be specified or an IllegalArgumentException will be thrown. Hence, files cannot be updated, only overwritten completely.

      This stream will not attempt to buffer the entire file, however some buffering will be done for potential optimizations and to avoid network thrashing. Specifically, up to AzureFileSystem.AZURE_STORAGE_PUT_BLOB_THRESHOLD bytes will be buffered initially. If that threshold is exceeded, the data will be broken into chunks and sent in blocks, and writes will be buffered into sizes of AzureFileSystem.AZURE_STORAGE_UPLOAD_BLOCK_SIZE. The maximum number of buffers of this size to be allocated is defined by AzureFileSystem.AZURE_STORAGE_MAX_CONCURRENCY_PER_REQUEST, which also configures the level of parallelism with which we may write and thus may affect write speeds as well.

      The data is only committed when the steam is closed. Hence, data cannot be read from the destination until the stream is closed. When the close method returns, it is guaranteed that, barring any errors, the data is finalized and available for reading.

      Writing happens asynchronously. Bytes passed for writing are stored until either the threshold or block size are met at which time they are sent to the service. When the write method returns, there is no guarantee about which phase of this process the data is in other than it has been accepted and will be written. Again, closing will guarantee that the data is written and available.

      Flush is a no-op as regards data transfers, but it can be used to check the state of the stream for errors. This can be a useful tool because writing happens asynchronously, and therefore an error from a previous write may not otherwise be thrown unless the stream is flushed, closed, or written to again.

      Overrides:
      newOutputStream in class FileSystemProvider
      Parameters:
      path - the path to the file to open or create
      options - options specifying how the file is opened
      Returns:
      a new output stream
      Throws:
      IllegalArgumentException - if an invalid combination of options is specified
      UnsupportedOperationException - if an unsupported option is specified
      IOException - If an I/O error occurs.
      SecurityException - never

    • newDirectoryStream

      public DirectoryStream<Path> newDirectoryStream(Path path, DirectoryStream.Filter<? super Path> filter) throws IOException
      Returns an AzureDirectoryStream for iterating over the contents of a directory. The elements returned by the directory stream's iterator are of type Path, each one representing an entry in the directory. The Path objects are obtained as if by resolving the name of the directory entry against dir. The entries returned by the iterator are filtered by the given filter.

      When not using the try-with-resources construct, then directory stream's close method should be invoked after iteration is completed to free any resources held for the open directory.

      Where the filter terminates due to an uncaught error or runtime exception then it is propagated to the hasNext or next method. Where an IOException is thrown, it results in the hasNext or next method throwing a DirectoryIteratorException with the IOException as the cause.

      Specified by:
      newDirectoryStream in class FileSystemProvider
      Parameters:
      path - the path to the directory
      filter - the directory stream filter
      Returns:
      a new and open DirectoryStream object
      Throws:
      IllegalArgumentException - If the path type is not an instance of AzurePath.
      NotDirectoryException - if the file could not otherwise be opened because it is not a directory
      IOException - If an I/O error occurs.
      SecurityException - never

    • createDirectory

      public void createDirectory(Path path, FileAttribute<?>... fileAttributes) throws IOException
      Creates a new directory at the specified path.

      The existence of a directory in the AzureFileSystem is defined on two levels. Weak existence is defined by the presence of a non-zero number of blobs prefixed with the directory's path. This concept is also known as a virtual directory and enables the file system to work with containers that were pre-loaded with data by another source but need to be accessed by this file system. Strong existence is defined as the presence of an actual storage resource at the given path, which in the case of directories, is a zero-length blob whose name is the directory path with a particular metadata field indicating the blob's status as a directory. This is also known as a concrete directory. Directories created by this file system will strongly exist. Operations targeting directories themselves as the object (e.g. setting properties) will target marker blobs underlying concrete directories. Other operations (e.g. listing) will operate on the blob-name prefix.

      This method fulfills the nio contract of: "The check for the existence of the file and the creation of the directory if it does not exist are a single operation that is atomic with respect to all other filesystem activities that might affect the directory." More specifically, this method will atomically check for strong existence of another file or directory at the given path and fail if one is present. On the other hand, we only check for weak existence of the parent to determine if the given path is valid. Additionally, the action of checking whether the parent exists, is not atomic with the creation of the directory. Note that while it is possible that the parent may be deleted between when the parent is determined to exist and the creation of the child, the creation of the child will always ensure the existence of a virtual parent, so the child will never be left floating and unreachable. The different checks on parent and child is due to limitations in the Storage service API.

      There may be some unintuitive behavior when working with directories in this file system, particularly virtual directories (usually those not created by this file system). A virtual directory will disappear as soon as all its children have been deleted. Furthermore, if a directory with the given path weakly exists at the time of calling this method, this method will still return success and create a concrete directory at the target location. In other words, it is possible to "double create" a directory if it first weakly exists and then is strongly created. This is both because it is impossible to atomically check if a virtual directory exists while creating a concrete directory and because such behavior will have minimal side effects--no files will be overwritten and the directory will still be available for writing as intended, though it may not be empty. This is not a complete list of such unintuitive behavior.

      This method will attempt to extract standard HTTP content headers from the list of file attributes to set them as blob headers. All other attributes will be set as blob metadata. The value of every attribute will be converted to a String except the Content-MD5 attribute which expects a byte[]. When extracting the content headers, the following strings will be used for comparison (constants for these values can be found on this type):

      • Content-Type
      • Content-Disposition
      • Content-Language
      • Content-Encoding
      • Content-MD5
      • Cache-Control
      Note that these properties also have a particular semantic in that if one is specified, all are updated. In other words, if any of the above is set, all those that are not set will be cleared. See the Azure Docs for more information.
      Specified by:
      createDirectory in class FileSystemProvider
      Parameters:
      path - the directory to create
      fileAttributes - an optional list of file attributes to set atomically when creating the directory
      Throws:
      IllegalArgumentException - If the path type is not an instance of AzurePath.
      UnsupportedOperationException - if the array contains an attribute that cannot be set atomically when creating the directory
      FileAlreadyExistsException - if a directory could not otherwise be created because a file of that name already exists
      IOException - If an I/O error occurs.
      SecurityException - never

    • delete

      public void delete(Path path) throws IOException
      Deletes the specified resource.

      This method is not atomic with respect to other file system operations. It is possible to delete a file in use by another process, and doing so will not immediately invalidate any channels open to that file--they will simply start to fail. Root directories cannot be deleted even when empty.

      Specified by:
      delete in class FileSystemProvider
      Parameters:
      path - the path to the file to delete
      Throws:
      IllegalArgumentException - If the path type is not an instance of AzurePath.
      NoSuchFileException - if the file does not exist
      DirectoryNotEmptyException - if the file is a directory and could not otherwise be deleted because the directory is not empty
      IOException - If an I/O error occurs.
      SecurityException - never

    • copy

      public void copy(Path source, Path destination, CopyOption... copyOptions) throws IOException
      Copies the resource at the source location to the destination.

      This method is not atomic with respect to other file system operations. More specifically, the checks necessary to validate the inputs and state of the file system are not atomic with the actual copying of data. If the copy is triggered, the copy itself is atomic and only a complete copy will ever be left at the destination.

      In addition to those in the docs for FileSystemProvider.copy(Path, Path, CopyOption...), this method has the following requirements for successful completion. StandardCopyOption.COPY_ATTRIBUTES must be passed as it is impossible not to copy blob properties; if this option is not passed, an UnsupportedOperationException will be thrown. Neither the source nor the destination can be a root directory; if either is a root directory, an IllegalArgumentException will be thrown. The parent directory of the destination must at least weakly exist; if it does not, an IOException will be thrown. The only supported option other than StandardCopyOption.COPY_ATTRIBUTES is StandardCopyOption.REPLACE_EXISTING; the presence of any other option will result in an UnsupportedOperationException.

      This method supports both virtual and concrete directories as both the source and destination. Unlike when creating a directory, the existence of a virtual directory at the destination will cause this operation to fail. This is in order to prevent the possibility of overwriting a non-empty virtual directory with a file. Still, as mentioned above, this check is not atomic with the creation of the resultant directory.

      Specified by:
      copy in class FileSystemProvider
      Parameters:
      source - the path to the file to copy
      destination - the path to the target file
      copyOptions - specifying how the copy should be done
      Throws:
      UnsupportedOperationException - if the array contains a copy option that is not supported
      FileAlreadyExistsException - if the target file exists but cannot be replaced because the REPLACE_EXISTING option is not specified
      DirectoryNotEmptyException - the REPLACE_EXISTING option is specified but the file cannot be replaced because it is a non-empty directory
      IOException - If an I/O error occurs.
      IllegalArgumentException - If the path type is not an instance of AzurePath.
      SecurityException - never
      See Also:

    • move

      public void move(Path path, Path path1, CopyOption... copyOptions) throws IOException
      Unsupported.
      Specified by:
      move in class FileSystemProvider
      Parameters:
      path - path
      path1 - path
      copyOptions - options
      Throws:
      UnsupportedOperationException - Operation is not supported.
      IOException

    • isSameFile

      public boolean isSameFile(Path path, Path path1) throws IOException
      Unsupported.
      Specified by:
      isSameFile in class FileSystemProvider
      Parameters:
      path - path
      path1 - path
      Throws:
      UnsupportedOperationException - Operation is not supported.
      IOException

    • isHidden

      public boolean isHidden(Path path) throws IOException
      Always returns false as hidden files are not supported.
      Specified by:
      isHidden in class FileSystemProvider
      Parameters:
      path - the path
      Returns:
      false
      Throws:
      IOException - If an I/O error occurs.
      SecurityException - never

    • getFileStore

      public FileStore getFileStore(Path path) throws IOException
      Unsupported.
      Specified by:
      getFileStore in class FileSystemProvider
      Parameters:
      path - path
      Returns:
      the file store where the file is stored.
      Throws:
      UnsupportedOperationException - Operation is not supported.
      IOException - If an I/O error occurs.
      SecurityException - never

    • checkAccess

      public void checkAccess(Path path, AccessMode... accessModes) throws IOException
      Checks the existence, and optionally the accessibility, of a file.

      This method may only be used to check the existence of a file. It is not possible to determine the permissions granted to a given client, so if any mode argument is specified, an UnsupportedOperationException will be thrown.

      Specified by:
      checkAccess in class FileSystemProvider
      Parameters:
      path - the path to the file to check
      accessModes - The access modes to check; may have zero elements
      Throws:
      NoSuchFileException - if a file does not exist
      AccessDeniedException - the requested access would be denied or the access cannot be determined because the Java virtual machine has insufficient privileges or other reasons
      IOException - If an I/O error occurs.
      SecurityException - never

    • getFileAttributeView

      public <V extends FileAttributeView> V getFileAttributeView(Path path, Class<V> type, LinkOption... linkOptions)
      Returns a file attribute view of a given type.

      See AzureBasicFileAttributeView and AzureBlobFileAttributeView for more information.

      Reading attributes on a virtual directory will return null for most properties other than AzureBlobFileAttributes.isVirtualDirectory(), which will return true. See createDirectory(Path, FileAttribute[]) for more information on virtual directories.

      Specified by:
      getFileAttributeView in class FileSystemProvider
      Parameters:
      path - the path to the file
      type - the Class object corresponding to the file attribute view
      linkOptions - ignored
      Returns:
      a file attribute view of the specified type, or null if the attribute view type is not available

    • readAttributes

      public <A extends BasicFileAttributes> A readAttributes(Path path, Class<A> type, LinkOption... linkOptions) throws IOException
      Reads a file's attributes as a bulk operation.

      See AzureBasicFileAttributes and AzureBlobFileAttributes for more information.

      Reading attributes on a virtual directory will return null for most properties other than AzureBlobFileAttributes.isVirtualDirectory(), which will return true. See createDirectory(Path, FileAttribute[]) for more information on virtual directories.

      Specified by:
      readAttributes in class FileSystemProvider
      Parameters:
      path - the path to the file
      type - the Class of the file attributes required to read
      linkOptions - ignored
      Returns:
      the file attributes
      Throws:
      UnsupportedOperationException - if an attributes of the given type are not supported
      IOException - If an I/O error occurs.
      SecurityException - never

    • readAttributes

      public Map<String,Object> readAttributes(Path path, String attributes, LinkOption... linkOptions) throws IOException
      Reads a set of file attributes as a bulk operation.

      See AzureBasicFileAttributes and AzureBlobFileAttributes for more information.

      Reading attributes on a virtual directory will return null for all properties other than AzureBlobFileAttributes.isVirtualDirectory(), which will return true. See createDirectory(Path, FileAttribute[]) for more information on virtual directories.

      Specified by:
      readAttributes in class FileSystemProvider
      Parameters:
      path - the path to the file
      attributes - the attributes to read
      linkOptions - ignored
      Returns:
      a map of the attributes returned; may be empty. The map's keys are the attribute names, its values are the attribute values
      Throws:
      UnsupportedOperationException - if an attributes of the given type are not supported
      IllegalArgumentException - if no attributes are specified or an unrecognized attributes is specified
      IOException - If an I/O error occurs.
      SecurityException - never

    • setAttribute

      public void setAttribute(Path path, String attributes, Object value, LinkOption... linkOptions) throws IOException
      Sets the value of a file attribute.

      See AzureBlobFileAttributeView for more information.

      Setting attributes on a virtual directory is not supported and will throw an IOException. See createDirectory(Path, FileAttribute[]) for more information on virtual directories.

      Specified by:
      setAttribute in class FileSystemProvider
      Parameters:
      path - the path to the file
      attributes - the attribute to set
      value - the attribute value
      linkOptions - ignored
      Throws:
      UnsupportedOperationException - if an attribute view is not available
      IllegalArgumentException - if the attribute name is not specified, or is not recognized, or the attribute value is of the correct type but has an inappropriate value
      ClassCastException - If the attribute value is not of the expected type or is a collection containing elements that are not of the expected type
      IOException - If an I/O error occurs.
      SecurityException - never