Documentation

Implementing your own File System

Introduction

If you need to implement your own file system for use in SFTP and SCP then you will need to create an AbstractFile implementation. AbstractFile deals with the mechanisms of accessing files in a simple interface, much like java.io.File or if you are familiar with Apache Commons VFS, like its FileObject. By implementing AbstractFile and its associated AbstractFileFactory you will be able to install a custom file system in your SSH server, and access these files over SFTP, SCP and even in the virtual shell. 

We recommend using your AbstractFile implementation as a mount on the VirtualFileFactory. VirtualFileFactory implements a virtual file system and therefore deals with some of the common problems of supporting file access over SFTP. It also, due to its mounting capabilities also deals with user default directories so eliminates the need for your own factory to manage these. With this in mind, this article assumes that this is the method being used.

To get started, use your IDE to create a MyAbstractFile class that implements AbstractFile, and MyAbstractFileFactory that implements AbstractFileFactory. Once we have the outline classes we can go ahead and configure the file system on the server. We will be creating an implementation that uses java.io.File as the backing file store so we will mount user directories under “tmp/${username}”. 

So what we are expecting here is that when a user logs into an SFTP session, they will see the root of their filesystem, which will map to a local folder under the current working directory of the process.

server.setFileFactory(new FileFactory() {
     public AbstractFileFactory<?> getFileFactory(SshConnection con) {
         return new VirtualFileFactory(
              new VirtualMountTemplate("/", "tmp/${username}",  
                  new MyAbstractFileFactory()));
});

Here we are using the FileFactory interface, which is responsible for creating AbstractFileFactory instances to install the file implementation. By default, this will cache each implementation so that only one instance gets created per SshConnection.

Implementing AbstractFileFactory

AbstractFileFactory creates instances of AbstractFile on demand for the SFTP and SCP subsystems. When the system needs a file it calls the factory to resolve the file. All the access, reading, writing and setting attributes is performed through AbstractFile. Therefore each instance of AbstractFile should reference a single path in your file system regardless of whether it is a directory or a folder, and also regardless of whether it actually exists or not.

Let’s jump into AbstractFIle and add a constructor for us to create the file. We will convert the string path to a java.io.File object and store this in the class. We also store some variables that will be useful later.

public class MyAbstractFile implements AbstractFile {
   File thisFile; 
   AbstractFileFactory<?> factory;
   SshConnection con;

   public MyAbstractFile(String path, AbstractFileFactory<?> factory, SshConnection con) { 
      this.thisFile = new File(path); 
      this.factory = factory;
      this.con = con;
   }

When the system needs a file it calls getFile. In our implementation, we just create a new instance of MyAbstractFile using the constructor we have just implemented and pass this back.

public class MyAbstractFileFactory 
         implements AbstractFileFactory<MyAbstractFile> {

   public MyAbstractFile getFile(String path, SshConnection con) 
         throws PermissionDeniedException, IOException { 
      return new MyAbstractFile(path, this, con); 
   }

There are other methods we have to implement too, populateEvent is called by the SFTP subsystem when an event on the file is generated. This gives you the opportunity to add any file system-specific attributes to the event. We don’t have any need for this right now so we just return the event that was passed in without modifying it:

public Event populateEvent(Event evt) { 
   return evt; 
}

There is also getDefaultPath, this requires that your implementation knows about user default directories. As we are implementing this for use with the VirtualFileFactory we do not need to do anything useful in this method because it will never be called by VirtualFileFactory. Just to be sure we will throw an UnsupportedOperationException from the method to ensure its never called:

public MyAbstractFile getDefaultPath() 
      throws PermissionDeniedException, IOException { 
   throw new UnsupportedOperationException();
}

Implementing AbstractFile

We now have to complete the remaining methods on AbstractFile. Some of these are more obvious than others.

First of all is getName, like java.io.File this method should return the name of the file, without any directories. For example, if the path of the file is “/home/admin/file.txt” the return value would be “file.txt”

public String getName() { 
   return thisFile.getName(); 
}

We also have getAbsolutePath and getCanonicalPath. As we have identical methods on java.io.File we can just return these, but in another implementation where these are not available, they should return a unique absolute path to the file, with the addition that getCanonicalPath removes any path elements like “.” and “..” and also following symbolic links. 

public String getCanonicalPath() throws IOException, PermissionDeniedException {
 return thisFile.getCanonicalPath(); 
}

public String getAbsolutePath() throws IOException, PermissionDeniedException {
   return thisFile.getAbsolutePath(); 
}

State and Attributes

Next, we have a method that returns whether the file exists or not:

public boolean exists() throws IOException { 
   return thisFile.exists(); 
}

There are a number of methods that help identify some of the attributes of the file.

public boolean isDirectory() throws IOException { 
   return thisFile.isDirectory(); 
}

public boolean isFile() throws IOException { 
   return thisFile.isFile(); 
}

public boolean isHidden() throws IOException { 
 return thisFile.isHidden(); 
}

public long lastModified() throws IOException { 
   return thisFile.lastModified(); 
}

public long length() throws IOException { 
   return thisFile.length(); 
}

And others that help identify the access permissions of the file:

public boolean isWritable() throws IOException { 
   return thisFile.canWrite(); 
}

public boolean isReadable() throws IOException { 
   return thisFile.canRead(); 
}

One of the important methods for SFTP is the getAttributes method. This returns an API structure that represents the attributes of the file which are passed around in the SFTP protocol. 

You must identify each file as an SFTP file type. There are many constants on SftpFileAttributes for this purpose, for a simple file system of files and folders we only need to deal with 3 different types, file, folder and unknown. Here I have created a utility method to get the file type for a java.io.File

private int getFileType(File f) { 
   if(f.isDirectory()) { 
      return SftpFileAttributes.SSH_FILEXFER_TYPE_DIRECTORY; 
   } else if(f.exists()) { 
      return SftpFileAttributes.SSH_FILEXFER_TYPE_REGULAR; 
   } else { 
      return SftpFileAttributes.SSH_FILEXFER_TYPE_UNKNOWN; 
   } 
}

As you can see here we return SSH_FILEXFER_TYPE_UNKNOWN if the file does not exist and a suitable type for file or directory.

With this utility method we can now implement getAttributes:

public SftpFileAttributes getAttributes() throws IOException { 
   if(!thisFile.exists()) { 
      throw new FileNotFoundException(); 
   } 
   SftpFileAttributes attrs = new SftpFileAttributes(getFileType(thisFile), "UTF-8");
   return attrs;
}

This is the minimum requirement for SftpFileAttributes, but we have some information from java.io.File that should be included too. We only have last modified from the File object but SFTP wants last accessed, and last modified. So we set the same value on both.

Note how we are modifying the value returned from Java, which is in milliseconds since Epoch, dividing it by 1000 to get seconds since Epoch which is the value required by SFTP.

attrs.setTimes(new UnsignedInteger64(thisFile.lastModified() / 1000),  
   new UnsignedInteger64(thisFile.lastModified() / 1000));

We set the permissions string using the isWritable/isReadable methods. You could extend this to support further permissions like executable state, group and other if you have access to these values.

attrs.setPermissions(String.format("%s%s-------", (isReadable() ? "r" : "-"), (isWritable() ? "w" : "-")));

If the file is not a directory, let’s also set the size

if(!isDirectory()) { 
   attrs.setSize(new UnsignedInteger64(thisFile.length())); 
}   

Setting Attributes

The setting of attributes is supported via SftpFileAttributes being passed to the setAttributes method. Here you should check for the existence of a setting on SftpFileAttributes and set accordingly (or ignore).  In our example, we only have access to one property in java.io.File and that’s the last modified attribute. Again remember that the SftpFileAttributes value will be seconds from Epoch, and Java is expecting milliseconds from Epoch so the value must be adjusted accordingly. 

public void setAttributes(SftpFileAttributes attrs) throws IOException { 
   if(attrs.hasModifiedTime()) { 
      thisFile.setLastModified(attrs.getModifiedTime().longValue() * 1000); 
   } 
}

Reading and Writing 

In most situations when reading and writing from or to a file, this is performed synchronously from the start of the file to the end of the file. Therefore you should provide as a minimum an InputStream and OutputStream implementation to perform these operations:

public InputStream getInputStream() throws IOException { 
   return new FileInputStream(thisFile); 
}

public OutputStream getOutputStream() throws IOException { 
   return new FileOutputStream(thisFile); 
}

There is also the option to allow appending to files. This is dealt with by the method

public OutputStream getOutputStream(boolean append) throws IOException;

It is advisable to provide this if you are able to. If you cannot, throw an IOException or you risk having files overwritten when an append operation is attempted.

Random Access

The SFTP protocol supports full random access to file content. Whilst not many clients actually use this to its fullest capacity it would be prudent to support random access if you are able to do so. Most SFTP clients will transfer a file synchronously from start to finish and if this is a restriction for you then you can skip this section.

Random access is implemented through an optional interface AbstractFileRandomAccess.

public interface AbstractFileRandomAccess { 
   public int read(byte[] buf, int off, int len) throws IOException; 
   public void write(byte[] buf, int off, int len) throws IOException; 
   public void setLength(long length) throws IOException; 
   public void seek(long position) throws IOException; 
   public void close() throws IOException; 
   public long getFilePointer() throws IOException;
}

Since we are implementing a local file system with java.io.File we can support random access through the RandomAccessFile. First, we will implement the supportsRandomAccess to tell the server we have random access implementation available.

public boolean supportsRandomAccess() { 
   return true; 
}

It’s then just a case of returning a new instance of your AbstractFileRandomAccess implementation from the openFile method.

public AbstractFileRandomAccess openFile(boolean writeAccess) throws IOException { 
   return new RandomAccessImpl(thisFile, writeAccess); 
}

public class RandomAccessImpl implements AbstractFileRandomAccess { 
   protected RandomAccessFile raf; 
   protected File f; 
   public RandomAccessImpl(File f, boolean writeAccess) throws IOException { 
      this.f = f; 
      String mode = "r" + (writeAccess ? "w" : ""); 
      raf = new RandomAccessFile(f, mode); 
   } 
   public void write(int b) throws IOException { 
      raf.write(b); 
   } 
   public void write(byte[] buf, int off, int len) throws IOException { 
      raf.write(buf, off, len); 
   }
   public void close() throws IOException { 
      raf.close(); 
   } 
   public void seek(long position) throws IOException { 
      raf.seek(position); 
   } 
   public int read(byte[] buf, int off, int len) throws IOException { 
      return raf.read(buf, off, len); 
   } 
   public void setLength(long length) throws IOException { 
      raf.setLength(length); 
   } 
   public long getFilePointer() throws IOException { 
      return raf.getFilePointer(); 
   }
}

Copying and Moving

There are dedicated methods on AbstractFile to copyFrom and moveTo another AbstractFile. The following methods are generic and could be used to copy any AbstractFile implementation to another. Remember, your file system may be installed within the virtual file system where there are other AbstractFile implementations. So you cannot assume that a copy or move will be from the same AbstractFile type.

If the AbstractFile types are the same it may be more efficient for you to perform a different copy or move operation. The best approach may be to extend this generic method by checking the types and either performing a generic copy/move if the types are not the same, or an efficient system-dependent copy/move if they are the same.

public void copyFrom(AbstractFile src) throws IOException, PermissionDeniedException {
   if(src.isDirectory()) { 
      createFolder(); 
      for(AbstractFile f : src.getChildren()) { 
         resolveFile(f.getName()).copyFrom(f); 
      } 
   } else if(src.isFile()) { 
      copy(src.getInputStream(), getOutputStream()); 
   } else { 
      throw new IOException("Cannot copy object that is not directory or a regular file"); 
   } 
}

public void moveTo(AbstractFile target) throws IOException, PermissionDeniedException {
   if(isDirectory()) { 
      target.createFolder(); 
      for(AbstractFile f : getChildren()) { 
         target.resolveFile(f.getName()).copyFrom(f); 
         f.delete(false); 
      } 
   } else if(isFile()) { 
      copy(getInputStream(),target.getOutputStream()); 
   } else { 
      throw new IOException("Cannot move object that is not directory or a regular file"); 
   } 
   delete(false); 
}

private void copy(InputStream in, OutputStream out) throws IOException { 
   try { 
      byte[] buf = new byte[4096]; 
      int r; 
      while((r = in.read(buf)) > -1) { 
         out.write(buf,0,r); 
      } 
   } catch(IOException ex) { 
      throw new IOException(ex.getMessage(), ex); 
   } finally { 
      out.close(); 
      in.close(); 
   } 
}

Creating and Deleting

When the path does not exist its possible for the system to call the following methods to create a file or a folder. Return boolean to indicate whether a file or folder was actually created. False can indicate an error condition, or if the file exists. You can throw an IOException if you want to generate a more verbose error message.

public boolean createFolder() throws PermissionDeniedException, IOException { 
   return thisFile.mkdirs(); 
}

public boolean createNewFile() throws PermissionDeniedException, IOException {  
   return thisFile.createNewFile(); 
}

And when either needs to be deleted the delete method is called. Note that a folder should only be deleted when there are no children within it. 

public boolean delete(boolean recursive) throws IOException, PermissionDeniedException { 
   return thisFile.delete(); 
}

It’s also possible for the system to request a file is truncated. 

public void truncate() throws PermissionDeniedException, IOException { 
   new FileOutputStream(thisFile).close(); 
} 

Listing Children

Where the AbstractFile instance points to a directory object then it should be able to list its children. We return a List of AbstractFile objects for each child. It’s advisable to check that the instance is a directory and throw an exception if not.

public List<AbstractFile> getChildren() throws IOException, PermissionDeniedException { 
   if(!isDirectory()) { 
      throw new IOException("Cannot list because the path is not a directory!"); 
   } 
   File[] files = thisFile.listFiles(); 
   List<AbstractFile> results = new ArrayList<AbstractFile>(); 
   for(File f : files) { 
      results.add(new MyAbstractFile(f), getFileFactory(), con); 
   } 
   return results; 
}

Note that here we have introduced a new constructor to take a java.io.File object.

public MyAbstractFile(File file, AbstractFileFactory<?> factory, SshConnection con) { 
   this.thisFile = file; 
   this.factory = factory;
   this.con = con;
}

It’s appropriate now to complete the getFileFactory method.

public AbstractFileFactory<? extends AbstractFile> getFileFactory() { 
   return factory; 
}

Miscellaneous 

There is a refresh method that you may want to implement. This depends on the implementation if retrieving attributes of the file is CPU or database intensive you may want to cache some details in the AbstractFile. The refresh method may be called to ask the system to update any cached information. In our example, we will leave this empty.

public void refresh() {
} 

There is also a handy resolve file method. You should check that the path is a child, if it is not you should delegate to the factory.

public AbstractFile resolveFile(String child) throws IOException, PermissionDeniedException { 
   if(child.startsWith("/")) {
      return factory.getFile(child, con);
   } else { 
      return new MyAbstractFile(new File(thisFile, child), factory, con);
   }
}