Start line:  
End line:  

Snippet Preview

Snippet HTML Code

Stack Overflow Questions
Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
  
  package org.apache.hadoop.hbase.util;
  
  import java.net.URI;
  import java.util.HashMap;
  import java.util.List;
  import java.util.Map;
  
  
Utility methods for interacting with the underlying file system.
  
  public abstract class FSUtils {
    private static final Log LOG = LogFactory.getLog(FSUtils.class);

  
Full access permissions (starting point for a umask)
  
    public static final String FULL_RWX_PERMISSIONS = "777";
    private static final String THREAD_POOLSIZE = "hbase.client.localityCheck.threadPoolSize";
    private static final int DEFAULT_THREAD_POOLSIZE = 2;

  
Set to true on Windows platforms
  
    public static final boolean WINDOWS = System.getProperty("os.name").startsWith("Windows");
 
   protected FSUtils() {
     super();
   }

  
Compare of path component. Does not consider schema; i.e. if schemas different but path starts with rootPath, then the function returns true

Parameters:
rootPath
path
Returns:
True if path starts with rootPath
 
   public static boolean isStartingWithPath(final Path rootPathfinal String path) {
     String uriRootPath = rootPath.toUri().getPath();
     String tailUriPath = (new Path(path)).toUri().getPath();
     return tailUriPath.startsWith(uriRootPath);
   }

  
Compare path component of the Path URI; e.g. if hdfs://a/b/c and /a/b/c, it will compare the '/a/b/c' part. Does not consider schema; i.e. if schemas different but path or subpath matches, the two will equate.

Parameters:
pathToSearch Path we will be trying to match.
pathTail
Returns:
True if pathTail is tail on the path of pathToSearch
 
   public static boolean isMatchingTail(final Path pathToSearchString pathTail) {
     return isMatchingTail(pathToSearchnew Path(pathTail));
   }

  
Compare path component of the Path URI; e.g. if hdfs://a/b/c and /a/b/c, it will compare the '/a/b/c' part. If you passed in 'hdfs://a/b/c and b/c, it would return true. Does not consider schema; i.e. if schemas different but path or subpath matches, the two will equate.

Parameters:
pathToSearch Path we will be trying to match.
pathTail
Returns:
True if pathTail is tail on the path of pathToSearch
 
   public static boolean isMatchingTail(final Path pathToSearchfinal Path pathTail) {
     if (pathToSearch.depth() != pathTail.depth()) return false;
     Path tailPath = pathTail;
     String tailName;
     Path toSearch = pathToSearch;
     String toSearchName;
     boolean result = false;
     do {
       tailName = tailPath.getName();
       if (tailName == null || tailName.length() <= 0) {
         result = true;
         break;
       }
       toSearchName = toSearch.getName();
       if (toSearchName == null || toSearchName.length() <= 0) break;
       // Move up a parent on each path for next go around.  Path doesn't let us go off the end.
       tailPath = tailPath.getParent();
       toSearch = toSearch.getParent();
     } while(tailName.equals(toSearchName));
     return result;
   }
 
   public static FSUtils getInstance(FileSystem fsConfiguration conf) {
     String scheme = fs.getUri().getScheme();
     if (scheme == null) {
       .warn("Could not find scheme for uri " +
           fs.getUri() + ", default to hdfs");
       scheme = "hdfs";
     }
     Class<?> fsUtilsClass = conf.getClass("hbase.fsutil." +
         scheme + ".impl"FSHDFSUtils.class); // Default to HDFS impl
     FSUtils fsUtils = (FSUtils)ReflectionUtils.newInstance(fsUtilsClassconf);
     return fsUtils;
   }

  
Delete if exists.

Parameters:
fs filesystem object
dir directory to delete
Returns:
True if deleted dir
Throws:
java.io.IOException e
 
   public static boolean deleteDirectory(final FileSystem fsfinal Path dir)
   throws IOException {
     return fs.exists(dir) && fs.delete(dirtrue);
   }

  
Delete the region directory if exists.

Parameters:
conf
hri
Returns:
True if deleted the region directory.
Throws:
java.io.IOException
 
   public static boolean deleteRegionDir(final Configuration conffinal HRegionInfo hri)
   throws IOException {
     Path rootDir = getRootDir(conf);
     FileSystem fs = rootDir.getFileSystem(conf);
     return deleteDirectory(fs,
       new Path(getTableDir(rootDirhri.getTable()), hri.getEncodedName()));
   }

  
Return the number of bytes that large input files should be optimally be split into to minimize i/o time. use reflection to search for getDefaultBlockSize(Path f) if the method doesn't exist, fall back to using getDefaultBlockSize()

Parameters:
fs filesystem object
Returns:
the default block size for the path's filesystem
Throws:
java.io.IOException e
 
   public static long getDefaultBlockSize(final FileSystem fsfinal Path paththrows IOException {
     Method m = null;
     Class<? extends FileSystemcls = fs.getClass();
     try {
       m = cls.getMethod("getDefaultBlockSize"new Class<?>[] { Path.class });
     } catch (NoSuchMethodException e) {
       .info("FileSystem doesn't support getDefaultBlockSize");
     } catch (SecurityException e) {
       .info("Doesn't have access to getDefaultBlockSize on FileSystems"e);
       m = null// could happen on setAccessible()
     }
     if (m == null) {
       return fs.getDefaultBlockSize();
     } else {
       try {
         Object ret = m.invoke(fspath);
         return ((Long)ret).longValue();
       } catch (Exception e) {
         throw new IOException(e);
       }
     }
   }
 
   /*
    * Get the default replication.
    *
    * use reflection to search for getDefaultReplication(Path f)
    * if the method doesn't exist, fall back to using getDefaultReplication()
    *
    * @param fs filesystem object
    * @param f path of file
    * @return default replication for the path's filesystem
    * @throws IOException e
    */
   public static short getDefaultReplication(final FileSystem fsfinal Path paththrows IOException {
     Method m = null;
     Class<? extends FileSystemcls = fs.getClass();
     try {
       m = cls.getMethod("getDefaultReplication"new Class<?>[] { Path.class });
     } catch (NoSuchMethodException e) {
       .info("FileSystem doesn't support getDefaultReplication");
     } catch (SecurityException e) {
       .info("Doesn't have access to getDefaultReplication on FileSystems"e);
       m = null// could happen on setAccessible()
     }
     if (m == null) {
       return fs.getDefaultReplication();
     } else {
       try {
         Object ret = m.invoke(fspath);
         return ((Number)ret).shortValue();
       } catch (Exception e) {
         throw new IOException(e);
       }
     }
   }

  
Returns the default buffer size to use during writes. The size of the buffer should probably be a multiple of hardware page size (4096 on Intel x86), and it determines how much data is buffered during read and write operations.

Parameters:
fs filesystem object
Returns:
default buffer size to use during writes
 
   public static int getDefaultBufferSize(final FileSystem fs) {
     return fs.getConf().getInt("io.file.buffer.size", 4096);
   }

  
Create the specified file on the filesystem. By default, this will:
  1. overwrite the file if it exists
  2. apply the umask in the configuration (if it is enabled)
  3. use the fs configured buffer size (or 4096 if not set)
  4. use the default replication
  5. use the default block size
  6. not track progress

Parameters:
fs org.apache.hadoop.fs.FileSystem on which to write the file
path org.apache.hadoop.fs.Path to the file to write
perm permissions
favoredNodes
Returns:
output stream to the created file
Throws:
java.io.IOException if the file cannot be created
 
   public static FSDataOutputStream create(FileSystem fsPath path,
       FsPermission permInetSocketAddress[] favoredNodesthrows IOException {
     if (fs instanceof HFileSystem) {
       FileSystem backingFs = ((HFileSystem)fs).getBackingFs();
       if (backingFs instanceof DistributedFileSystem) {
         // Try to use the favoredNodes version via reflection to allow backwards-
         // compatibility.
         try {
           return (FSDataOutputStream) (DistributedFileSystem.class
               .getDeclaredMethod("create"Path.classFsPermission.class,
                   boolean.classint.classshort.classlong.class,
                   Progressable.classInetSocketAddress[].class)
                   .invoke(backingFspathpermtrue,
                       getDefaultBufferSize(backingFs),
                       getDefaultReplication(backingFspath),
                       getDefaultBlockSize(backingFspath),
                       nullfavoredNodes));
         } catch (InvocationTargetException ite) {
           // Function was properly called, but threw it's own exception.
           throw new IOException(ite.getCause());
         } catch (NoSuchMethodException e) {
           .debug("DFS Client does not support most favored nodes create; using default create");
           if (.isTraceEnabled()) .trace("Ignoring; use default create"e);
         } catch (IllegalArgumentException e) {
           .debug("Ignoring (most likely Reflection related exception) " + e);
         } catch (SecurityException e) {
           .debug("Ignoring (most likely Reflection related exception) " + e);
         } catch (IllegalAccessException e) {
           .debug("Ignoring (most likely Reflection related exception) " + e);
         }
       }
     }
     return create(fspathpermtrue);
   }

  
Create the specified file on the filesystem. By default, this will:
  1. apply the umask in the configuration (if it is enabled)
  2. use the fs configured buffer size (or 4096 if not set)
  3. use the default replication
  4. use the default block size
  5. not track progress

Parameters:
fs org.apache.hadoop.fs.FileSystem on which to write the file
path org.apache.hadoop.fs.Path to the file to write
perm
overwrite Whether or not the created file should be overwritten.
Returns:
output stream to the created file
Throws:
java.io.IOException if the file cannot be created
 
   public static FSDataOutputStream create(FileSystem fsPath path,
       FsPermission permboolean overwritethrows IOException {
     if (.isTraceEnabled()) {
       .trace("Creating file=" + path + " with permission=" + perm + ", overwrite=" + overwrite);
     }
     return fs.create(pathpermoverwritegetDefaultBufferSize(fs),
         getDefaultReplication(fspath), getDefaultBlockSize(fspath), null);
   }

  
Get the file permissions specified in the configuration, if they are enabled.

Parameters:
fs filesystem that the file will be created on.
conf configuration to read for determining if permissions are enabled and which to use
permssionConfKey property key in the configuration to use when finding the permission
Returns:
the permission to use when creating a new file on the fs. If special permissions are not specified in the configuration, then the default permissions on the the fs will be returned.
 
   public static FsPermission getFilePermissions(final FileSystem fs,
       final Configuration conffinal String permssionConfKey) {
     boolean enablePermissions = conf.getBoolean(
         .false);
 
     if (enablePermissions) {
       try {
         FsPermission perm = new FsPermission();
         // make sure that we have a mask, if not, go default.
         String mask = conf.get(permssionConfKey);
         if (mask == null)
           return getFileDefault();
         // appy the umask
         FsPermission umask = new FsPermission(mask);
         return perm.applyUMask(umask);
       } catch (IllegalArgumentException e) {
         .warn(
             "Incorrect umask attempted to be created: "
                 + conf.get(permssionConfKey)
                 + ", using default file permissions."e);
         return getFileDefault();
       }
     }
     return getFileDefault();
   }

  
Get the default permission for file. This is the same method as FsPermission.getFileDefault() in Hadoop 2. We provide the method here to support compatibility with Hadoop 1. See HBASE-11061. Would be better to do this as Interface in hadoop-compat w/ hadoop1 and hadoop2 implementations but punting on this since small risk this will change in 0.96/0.98 timeframe (only committed to these branches).
 
   public static FsPermission getFileDefault() {
     return new FsPermission((short)00666);
   }

  
Checks to see if the specified file system is available

Parameters:
fs filesystem
Throws:
java.io.IOException e
 
   public static void checkFileSystemAvailable(final FileSystem fs)
   throws IOException {
     if (!(fs instanceof DistributedFileSystem)) {
       return;
     }
     IOException exception = null;
     try {
       if (dfs.exists(new Path("/"))) {
         return;
       }
     } catch (IOException e) {
       exception = RemoteExceptionHandler.checkIOException(e);
     }
     try {
       fs.close();
     } catch (Exception e) {
       .error("file system close failed: "e);
     }
     IOException io = new IOException("File system is not available");
     io.initCause(exception);
     throw io;
   }

  
We use reflection because org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(org.apache.hadoop.hdfs.protocol.HdfsConstants.SafeModeAction,boolean) is not in hadoop 1.1

Parameters:
dfs
Returns:
whether we're in safe mode
Throws:
java.io.IOException
 
   private static boolean isInSafeMode(DistributedFileSystem dfsthrows IOException {
     boolean inSafeMode = false;
     try {
       Method m = DistributedFileSystem.class.getMethod("setSafeMode"new Class<?> []{
           org.apache.hadoop.hdfs.protocol.FSConstants.SafeModeAction.classboolean.class});
       inSafeMode = (Booleanm.invoke(dfs,
     } catch (Exception e) {
       if (e instanceof IOExceptionthrow (IOExceptione;
 
       // Check whether dfs is on safemode.
       inSafeMode = dfs.setSafeMode(
     }
     return inSafeMode;
   }

  
Check whether dfs is in safemode.

Parameters:
conf
Throws:
java.io.IOException
 
   public static void checkDfsSafeMode(final Configuration conf)
   throws IOException {
     boolean isInSafeMode = false;
     FileSystem fs = FileSystem.get(conf);
     if (fs instanceof DistributedFileSystem) {
       isInSafeMode = isInSafeMode(dfs);
     }
     if (isInSafeMode) {
       throw new IOException("File system is in safemode, it can't be written now");
     }
   }

  
Verifies current version of file system

Parameters:
fs filesystem object
rootdir root hbase directory
Returns:
null if no version file exists, version string otherwise.
Throws:
java.io.IOException e
org.apache.hadoop.hbase.exceptions.DeserializationException
 
   public static String getVersion(FileSystem fsPath rootdir)
     Path versionFile = new Path(rootdir.);
     FileStatus[] status = null;
     try {
       // hadoop 2.0 throws FNFE if directory does not exist.
       // hadoop 1.0 returns null if directory does not exist.
       status = fs.listStatus(versionFile);
     } catch (FileNotFoundException fnfe) {
       return null;
     }
     if (status == null || status.length == 0) return null;
     String version = null;
     byte [] content = new byte [(int)status[0].getLen()];
     FSDataInputStream s = fs.open(versionFile);
     try {
       IOUtils.readFully(scontent, 0, content.length);
       if (ProtobufUtil.isPBMagicPrefix(content)) {
         version = parseVersionFrom(content);
       } else {
         // Presume it pre-pb format.
         InputStream is = new ByteArrayInputStream(content);
         DataInputStream dis = new DataInputStream(is);
         try {
           version = dis.readUTF();
         } finally {
           dis.close();
         }
       }
     } catch (EOFException eof) {
       .warn("Version file was empty, odd, will try to set it.");
     } finally {
       s.close();
     }
     return version;
   }

  
Parse the content of the ${HBASE_ROOTDIR}/hbase.version file.

Parameters:
bytes The byte content of the hbase.version file.
Returns:
The version found in the file as a String.
Throws:
org.apache.hadoop.hbase.exceptions.DeserializationException
 
   static String parseVersionFrom(final byte [] bytes)
     ProtobufUtil.expectPBMagicPrefix(bytes);
     int pblen = ProtobufUtil.lengthOfPBMagic();
       FSProtos.HBaseVersionFileContent.newBuilder();
     FSProtos.HBaseVersionFileContent fileContent;
     try {
       fileContent = builder.mergeFrom(bytespblenbytes.length - pblen).build();
       return fileContent.getVersion();
     } catch (InvalidProtocolBufferException e) {
       // Convert
       throw new DeserializationException(e);
     }
   }

  
Create the content to write into the ${HBASE_ROOTDIR}/hbase.version file.

Parameters:
version Version to persist
Returns:
Serialized protobuf with version content and a bit of pb magic for a prefix.
 
   static byte [] toVersionByteArray(final String version) {
       FSProtos.HBaseVersionFileContent.newBuilder();
     return ProtobufUtil.prependPBMagic(builder.setVersion(version).build().toByteArray());
   }

  
Verifies current version of file system

Parameters:
fs file system
rootdir root directory of HBase installation
message if true, issues a message on System.out
Throws:
java.io.IOException e
org.apache.hadoop.hbase.exceptions.DeserializationException
 
   public static void checkVersion(FileSystem fsPath rootdirboolean message)
     checkVersion(fsrootdirmessage, 0, .);
   }

  
Verifies current version of file system

Parameters:
fs file system
rootdir root directory of HBase installation
message if true, issues a message on System.out
wait wait interval
retries number of times to retry
Throws:
java.io.IOException e
org.apache.hadoop.hbase.exceptions.DeserializationException
 
   public static void checkVersion(FileSystem fsPath rootdir,
       boolean messageint waitint retries)
     String version = getVersion(fsrootdir);
     if (version == null) {
       if (!metaRegionExists(fsrootdir)) {
         // rootDir is empty (no version file and no root region)
         // just create new version file (HBASE-1195)
         setVersion(fsrootdirwaitretries);
         return;
       }
     } else if (version.compareTo(.) == 0) return;
 
     // version is deprecated require migration
     // Output on stdout so user sees it in terminal.
     String msg = "HBase file layout needs to be upgraded."
       + " You have version " + version
       + " and I want version " + .
       + ". Consult http://hbase.apache.org/book.html for further information about upgrading HBase."
       + " Is your hbase.rootdir valid? If so, you may need to run "
       + "'hbase hbck -fixVersionFile'.";
     if (message) {
       ..println("WARNING! " + msg);
     }
     throw new FileSystemVersionException(msg);
   }

  
Sets version of file system

Parameters:
fs filesystem object
rootdir hbase root
Throws:
java.io.IOException e
 
   public static void setVersion(FileSystem fsPath rootdir)
   throws IOException {
     setVersion(fsrootdir., 0,
   }

  
Sets version of file system

Parameters:
fs filesystem object
rootdir hbase root
wait time to wait for retry
retries number of times to retry before failing
Throws:
java.io.IOException e
 
   public static void setVersion(FileSystem fsPath rootdirint waitint retries)
   throws IOException {
     setVersion(fsrootdir.waitretries);
   }


  
Sets version of file system

Parameters:
fs filesystem object
rootdir hbase root directory
version version to set
wait time to wait for retry
retries number of times to retry before throwing an IOException
Throws:
java.io.IOException e
 
   public static void setVersion(FileSystem fsPath rootdirString version,
       int waitint retriesthrows IOException {
     Path versionFile = new Path(rootdir.);
     Path tempVersionFile = new Path(rootdir. + . +
     while (true) {
       try {
         // Write the version to a temporary file
         FSDataOutputStream s = fs.create(tempVersionFile);
         try {
           s.write(toVersionByteArray(version));
           s.close();
           s = null;
           // Move the temp version file to its normal location. Returns false
           // if the rename failed. Throw an IOE in that case.
           if (!fs.rename(tempVersionFileversionFile)) {
             throw new IOException("Unable to move temp version file to " + versionFile);
           }
         } finally {
           // Cleaning up the temporary if the rename failed would be trying
           // too hard. We'll unconditionally create it again the next time
           // through anyway, files are overwritten by default by create().
 
           // Attempt to close the stream on the way out if it is still open.
           try {
             if (s != nulls.close();
           } catch (IOException ignore) { }
         }
         .info("Created version file at " + rootdir.toString() + " with version=" + version);
         return;
       } catch (IOException e) {
         if (retries > 0) {
           .debug("Unable to create version file at " + rootdir.toString() + ", retrying"e);
           fs.delete(versionFilefalse);
           try {
             if (wait > 0) {
               Thread.sleep(wait);
             }
           } catch (InterruptedException ex) {
             // ignore
           }
           retries--;
         } else {
           throw e;
         }
       }
     }
   }

  
Checks that a cluster ID file exists in the HBase root directory

Parameters:
fs the root directory FileSystem
rootdir the HBase root directory in HDFS
wait how long to wait between retries
Returns:
true if the file exists, otherwise false
Throws:
java.io.IOException if checking the FileSystem fails
 
   public static boolean checkClusterIdExists(FileSystem fsPath rootdir,
       int waitthrows IOException {
     while (true) {
       try {
         Path filePath = new Path(rootdir.);
         return fs.exists(filePath);
       } catch (IOException ioe) {
         if (wait > 0) {
           .warn("Unable to check cluster ID file in " + rootdir.toString() +
               ", retrying in "+wait+"msec: "+StringUtils.stringifyException(ioe));
           try {
             Thread.sleep(wait);
           } catch (InterruptedException ie) {
             throw (InterruptedIOException)new InterruptedIOException().initCause(ie);
           }
         } else {
           throw ioe;
         }
       }
     }
   }

  
Returns the value of the unique cluster ID stored for this HBase instance.

Parameters:
fs the root directory FileSystem
rootdir the path to the HBase root directory
Returns:
the unique cluster identifier
Throws:
java.io.IOException if reading the cluster ID file fails
 
   public static ClusterId getClusterId(FileSystem fsPath rootdir)
   throws IOException {
     Path idPath = new Path(rootdir.);
     ClusterId clusterId = null;
     FileStatus status = fs.exists(idPath)? fs.getFileStatus(idPath):  null;
     if (status != null) {
       int len = Ints.checkedCast(status.getLen());
       byte [] content = new byte[len];
       FSDataInputStream in = fs.open(idPath);
       try {
         in.readFully(content);
       } catch (EOFException eof) {
         .warn("Cluster ID file " + idPath.toString() + " was empty");
       } finally{
         in.close();
       }
       try {
         clusterId = ClusterId.parseFrom(content);
       } catch (DeserializationException e) {
         throw new IOException("content=" + Bytes.toString(content), e);
       }
       // If not pb'd, make it so.
       if (!ProtobufUtil.isPBMagicPrefix(content)) {
         String cid = null;
         in = fs.open(idPath);
         try {
           cid = in.readUTF();
           clusterId = new ClusterId(cid);
         } catch (EOFException eof) {
           .warn("Cluster ID file " + idPath.toString() + " was empty");
         } finally {
           in.close();
         }
         rewriteAsPb(fsrootdiridPathclusterId);
       }
       return clusterId;
     } else {
       .warn("Cluster ID file does not exist at " + idPath.toString());
     }
     return clusterId;
   }

  

Parameters:
cid
Throws:
java.io.IOException
 
   private static void rewriteAsPb(final FileSystem fsfinal Path rootdirfinal Path p,
       final ClusterId cid)
   throws IOException {
     // Rewrite the file as pb.  Move aside the old one first, write new
     // then delete the moved-aside file.
     Path movedAsideName = new Path(p + "." + System.currentTimeMillis());
     if (!fs.rename(pmovedAsideName)) throw new IOException("Failed rename of " + p);
     setClusterId(fsrootdircid, 100);
     if (!fs.delete(movedAsideNamefalse)) {
       throw new IOException("Failed delete of " + movedAsideName);
     }
     .debug("Rewrote the hbase.id file as pb");
   }

  
Writes a new unique identifier for this cluster to the "hbase.id" file in the HBase root directory

Parameters:
fs the root directory FileSystem
rootdir the path to the HBase root directory
clusterId the unique identifier to store
wait how long (in milliseconds) to wait between retries
Throws:
java.io.IOException if writing to the FileSystem fails and no wait value
 
   public static void setClusterId(FileSystem fsPath rootdirClusterId clusterId,
       int waitthrows IOException {
     while (true) {
       try {
         Path idFile = new Path(rootdir.);
         Path tempIdFile = new Path(rootdir. +
           . + .);
         // Write the id file to a temporary location
         FSDataOutputStream s = fs.create(tempIdFile);
         try {
           s.write(clusterId.toByteArray());
           s.close();
           s = null;
           // Move the temporary file to its normal location. Throw an IOE if
           // the rename failed
           if (!fs.rename(tempIdFileidFile)) {
             throw new IOException("Unable to move temp version file to " + idFile);
           }
         } finally {
           // Attempt to close the stream if still open on the way out
           try {
             if (s != nulls.close();
           } catch (IOException ignore) { }
         }
         if (.isDebugEnabled()) {
           .debug("Created cluster ID file at " + idFile.toString() + " with ID: " + clusterId);
         }
         return;
       } catch (IOException ioe) {
         if (wait > 0) {
           .warn("Unable to create cluster ID file in " + rootdir.toString() +
               ", retrying in " + wait + "msec: " + StringUtils.stringifyException(ioe));
           try {
             Thread.sleep(wait);
           } catch (InterruptedException ie) {
             Thread.currentThread().interrupt();
             break;
           }
         } else {
           throw ioe;
         }
       }
     }
   }

  
Verifies root directory path is a valid URI with a scheme

Parameters:
root root directory path
Returns:
Passed root argument.
Throws:
java.io.IOException if not a valid URI with a scheme
 
   public static Path validateRootPath(Path rootthrows IOException {
     try {
       URI rootURI = new URI(root.toString());
       String scheme = rootURI.getScheme();
       if (scheme == null) {
         throw new IOException("Root directory does not have a scheme");
       }
       return root;
     } catch (URISyntaxException e) {
       IOException io = new IOException("Root directory path is not a valid " +
         "URI -- check your " + . + " configuration");
       io.initCause(e);
       throw io;
     }
   }

  
Checks for the presence of the root path (using the provided conf object) in the given path. If it exists, this method removes it and returns the String representation of remaining relative path.

Parameters:
path
conf
Returns:
String representation of the remaining relative path
Throws:
java.io.IOException
 
   public static String removeRootPath(Path pathfinal Configuration confthrows IOException {
     Path root = FSUtils.getRootDir(conf);
     String pathStr = path.toString();
     // check that the path is absolute... it has the root path in it.
     if (!pathStr.startsWith(root.toString())) return pathStr;
     // if not, return as it is.
     return pathStr.substring(root.toString().length() + 1);// remove the "/" too.
   }

  
If DFS, check safe mode and if so, wait until we clear it.

Parameters:
conf configuration
wait Sleep between retries
Throws:
java.io.IOException e
 
   public static void waitOnSafeMode(final Configuration conf,
     final long wait)
   throws IOException {
     FileSystem fs = FileSystem.get(conf);
     if (!(fs instanceof DistributedFileSystem)) return;
     // Make sure dfs is not in safe mode
     while (isInSafeMode(dfs)) {
       .info("Waiting for dfs to exit safe mode...");
       try {
         Thread.sleep(wait);
       } catch (InterruptedException e) {
         //continue
       }
     }
   }

  
Return the 'path' component of a Path. In Hadoop, Path is an URI. This method returns the 'path' component of a Path's URI: e.g. If a Path is hdfs://example.org:9000/hbase_trunk/TestTable/compaction.dir, this method returns /hbase_trunk/TestTable/compaction.dir. This method is useful if you want to print out a Path without qualifying Filesystem instance.

Parameters:
p Filesystem Path whose 'path' component we are to return.
Returns:
Path portion of the Filesystem
 
   public static String getPath(Path p) {
     return p.toUri().getPath();
   }

  

Parameters:
c configuration
Returns:
Path to hbase root directory: i.e. hbase.rootdir from configuration as a qualified Path.
Throws:
java.io.IOException e
 
   public static Path getRootDir(final Configuration cthrows IOException {
     Path p = new Path(c.get(.));
     FileSystem fs = p.getFileSystem(c);
     return p.makeQualified(fs);
   }
 
   public static void setRootDir(final Configuration cfinal Path rootthrows IOException {
     c.set(.root.toString());
   }
 
   public static void setFsDefault(final Configuration cfinal Path rootthrows IOException {
     c.set("fs.defaultFS"root.toString());    // for hadoop 0.21+
     c.set("fs.default.name"root.toString()); // for hadoop 0.20
   }

  
Checks if meta region exists

Parameters:
fs file system
rootdir root directory of HBase installation
Returns:
true if exists
Throws:
java.io.IOException e
 
   @SuppressWarnings("deprecation")
   public static boolean metaRegionExists(FileSystem fsPath rootdir)
   throws IOException {
     Path metaRegionDir =
       HRegion.getRegionDir(rootdir.);
     return fs.exists(metaRegionDir);
   }

  
Compute HDFS blocks distribution of a given file, or a portion of the file

Parameters:
fs file system
status file status of the file
start start position of the portion
length length of the portion
Returns:
The HDFS blocks distribution
 
     final FileSystem fsFileStatus statuslong startlong length)
     throws IOException {
     HDFSBlocksDistribution blocksDistribution = new HDFSBlocksDistribution();
     BlockLocation [] blockLocations =
       fs.getFileBlockLocations(statusstartlength);
     for(BlockLocation bl : blockLocations) {
       String [] hosts = bl.getHosts();
       long len = bl.getLength();
       blocksDistribution.addHostsAndBlockWeight(hostslen);
     }
 
     return blocksDistribution;
   }



  
Runs through the hbase rootdir and checks all stores have only one file in them -- that is, they've been major compacted. Looks at root and meta tables too.

Parameters:
fs filesystem
hbaseRootDir hbase root directory
Returns:
True if this hbase install is major compacted.
Throws:
java.io.IOException e
  public static boolean isMajorCompacted(final FileSystem fs,
      final Path hbaseRootDir)
  throws IOException {
    List<PathtableDirs = getTableDirs(fshbaseRootDir);
    PathFilter regionFilter = new RegionDirFilter(fs);
    PathFilter familyFilter = new FamilyDirFilter(fs);
    for (Path d : tableDirs) {
      FileStatus[] regionDirs = fs.listStatus(dregionFilter);
      for (FileStatus regionDir : regionDirs) {
        Path dd = regionDir.getPath();
        // Else its a region name.  Now look in region for families.
        FileStatus[] familyDirs = fs.listStatus(ddfamilyFilter);
        for (FileStatus familyDir : familyDirs) {
          Path family = familyDir.getPath();
          // Now in family make sure only one file.
          FileStatus[] familyStatus = fs.listStatus(family);
          if (familyStatus.length > 1) {
            .debug(family.toString() + " has " + familyStatus.length +
                " files.");
            return false;
          }
        }
      }
    }
    return true;
  }
  // TODO move this method OUT of FSUtils. No dependencies to HMaster
  
Returns the total overall fragmentation percentage. Includes hbase:meta and -ROOT- as well.

Parameters:
master The master defining the HBase root and file system.
Returns:
A map for each table and its percentage.
Throws:
java.io.IOException When scanning the directory fails.
  public static int getTotalTableFragmentation(final HMaster master)
  throws IOException {
    Map<StringIntegermap = getTableFragmentation(master);
    return map != null && map.size() > 0 ? map.get("-TOTAL-") : -1;
  }

  
Runs through the HBase rootdir and checks how many stores for each table have more than one file in them. Checks -ROOT- and hbase:meta too. The total percentage across all tables is stored under the special key "-TOTAL-".

Parameters:
master The master defining the HBase root and file system.
Returns:
A map for each table and its percentage.
Throws:
java.io.IOException When scanning the directory fails.
  public static Map<StringIntegergetTableFragmentation(
    final HMaster master)
  throws IOException {
    Path path = getRootDir(master.getConfiguration());
    // since HMaster.getFileSystem() is package private
    FileSystem fs = path.getFileSystem(master.getConfiguration());
    return getTableFragmentation(fspath);
  }

  
Runs through the HBase rootdir and checks how many stores for each table have more than one file in them. Checks -ROOT- and hbase:meta too. The total percentage across all tables is stored under the special key "-TOTAL-".

Parameters:
fs The file system to use.
hbaseRootDir The root directory to scan.
Returns:
A map for each table and its percentage.
Throws:
java.io.IOException When scanning the directory fails.
  public static Map<StringIntegergetTableFragmentation(
    final FileSystem fsfinal Path hbaseRootDir)
  throws IOException {
    Map<StringIntegerfrags = new HashMap<StringInteger>();
    int cfCountTotal = 0;
    int cfFragTotal = 0;
    PathFilter regionFilter = new RegionDirFilter(fs);
    PathFilter familyFilter = new FamilyDirFilter(fs);
    List<PathtableDirs = getTableDirs(fshbaseRootDir);
    for (Path d : tableDirs) {
      int cfCount = 0;
      int cfFrag = 0;
      FileStatus[] regionDirs = fs.listStatus(dregionFilter);
      for (FileStatus regionDir : regionDirs) {
        Path dd = regionDir.getPath();
        // else its a region name, now look in region for families
        FileStatus[] familyDirs = fs.listStatus(ddfamilyFilter);
        for (FileStatus familyDir : familyDirs) {
          cfCount++;
          cfCountTotal++;
          Path family = familyDir.getPath();
          // now in family make sure only one file
          FileStatus[] familyStatus = fs.listStatus(family);
          if (familyStatus.length > 1) {
            cfFrag++;
            cfFragTotal++;
          }
        }
      }
      // compute percentage per table and store in result list
      frags.put(FSUtils.getTableName(d).getNameAsString(),
          Math.round((floatcfFrag / cfCount * 100));
    }
    // set overall percentage for all tables
    frags.put("-TOTAL-", Math.round((floatcfFragTotal / cfCountTotal * 100));
    return frags;
  }

  
Returns the org.apache.hadoop.fs.Path object representing the table directory under path rootdir

Parameters:
rootdir qualified path of HBase root directory
tableName name of table
Returns:
org.apache.hadoop.fs.Path for table
  public static Path getTableDir(Path rootdirfinal TableName tableName) {
    return new Path(getNamespaceDir(rootdirtableName.getNamespaceAsString()),
        tableName.getQualifierAsString());
  }

  
Returns the org.apache.hadoop.hbase.TableName object representing the table directory under path rootdir

Parameters:
tablePath path of table
Returns:
org.apache.hadoop.fs.Path for table
  public static TableName getTableName(Path tablePath) {
    return TableName.valueOf(tablePath.getParent().getName(), tablePath.getName());
  }

  
Returns the org.apache.hadoop.fs.Path object representing the namespace directory under path rootdir

Parameters:
rootdir qualified path of HBase root directory
namespace namespace name
Returns:
org.apache.hadoop.fs.Path for table
  public static Path getNamespaceDir(Path rootdirfinal String namespace) {
    return new Path(rootdirnew Path(.,
        new Path(namespace)));
  }

  
A org.apache.hadoop.fs.PathFilter that returns only regular files.
  static class FileFilter implements PathFilter {
    private final FileSystem fs;
    public FileFilter(final FileSystem fs) {
      this. = fs;
    }
    @Override
    public boolean accept(Path p) {
      try {
        return .isFile(p);
      } catch (IOException e) {
        .debug("unable to verify if path=" + p + " is a regular file"e);
        return false;
      }
    }
  }

  
Directory filter that doesn't include any of the directories in the specified blacklist
  public static class BlackListDirFilter implements PathFilter {
    private final FileSystem fs;
    private List<Stringblacklist;

    
Create a filter on the give filesystem with the specified blacklist

Parameters:
fs filesystem to filter
directoryNameBlackList list of the names of the directories to filter. If null, all directories are returned
    @SuppressWarnings("unchecked")
    public BlackListDirFilter(final FileSystem fsfinal List<StringdirectoryNameBlackList) {
      this. = fs;
       =
        (List<String>) (directoryNameBlackList == null ? Collections.emptyList()
          : directoryNameBlackList);
    }
    @Override
    public boolean accept(Path p) {
      boolean isValid = false;
      try {
        if (isValidName(p.getName())) {
          isValid = .getFileStatus(p).isDir();
        } else {
          isValid = false;
        }
      } catch (IOException e) {
        .warn("An error occurred while verifying if [" + p.toString()
            + "] is a valid directory. Returning 'not valid' and continuing."e);
      }
      return isValid;
    }
    protected boolean isValidName(final String name) {
      return !.contains(name);
    }
  }

  
A org.apache.hadoop.fs.PathFilter that only allows directories.
  public static class DirFilter extends BlackListDirFilter {
    public DirFilter(FileSystem fs) {
      super(fsnull);
    }
  }

  
A org.apache.hadoop.fs.PathFilter that returns usertable directories. To get all directories use the FSUtils.BlackListDirFilter with a null blacklist
  public static class UserTableDirFilter extends BlackListDirFilter {
    public UserTableDirFilter(FileSystem fs) {
      super(fs.);
    }
    protected boolean isValidName(final String name) {
      if (!super.isValidName(name))
        return false;
      try {
        TableName.isLegalTableQualifierName(Bytes.toBytes(name));
      } catch (IllegalArgumentException e) {
        .info("INVALID NAME " + name);
        return false;
      }
      return true;
    }
  }

  
Heuristic to determine whether is safe or not to open a file for append Looks both for dfs.support.append and use reflection to search for SequenceFile.Writer.syncFs() or FSDataOutputStream.hflush()

Parameters:
conf
Returns:
True if append support
  public static boolean isAppendSupported(final Configuration conf) {
    boolean append = conf.getBoolean("dfs.support.append"false);
    if (append) {
      try {
        // TODO: The implementation that comes back when we do a createWriter
        // may not be using SequenceFile so the below is not a definitive test.
        // Will do for now (hdfs-200).
        SequenceFile.Writer.class.getMethod("syncFs"new Class<?> []{});
        append = true;
      } catch (SecurityException e) {
      } catch (NoSuchMethodException e) {
        append = false;
      }
    }
    if (!append) {
      // Look for the 0.21, 0.22, new-style append evidence.
      try {
        FSDataOutputStream.class.getMethod("hflush"new Class<?> []{});
        append = true;
      } catch (NoSuchMethodException e) {
        append = false;
      }
    }
    return append;
  }

  

Parameters:
conf
Returns:
True if this filesystem whose scheme is 'hdfs'.
Throws:
java.io.IOException
  public static boolean isHDFS(final Configuration confthrows IOException {
    FileSystem fs = FileSystem.get(conf);
    String scheme = fs.getUri().getScheme();
    return scheme.equalsIgnoreCase("hdfs");
  }

  
Recover file lease. Used when a file might be suspect to be had been left open by another process.

Parameters:
fs FileSystem handle
p Path of file to recover lease
conf Configuration handle
Throws:
java.io.IOException
  public abstract void recoverFileLease(final FileSystem fsfinal Path p,
      Configuration confCancelableProgressable reporterthrows IOException;
  public static List<PathgetTableDirs(final FileSystem fsfinal Path rootdir)
      throws IOException {
    List<PathtableDirs = new LinkedList<Path>();
    for(FileStatus status :
        fs.globStatus(new Path(rootdir,
            new Path(."*")))) {
      tableDirs.addAll(FSUtils.getLocalTableDirs(fsstatus.getPath()));
    }
    return tableDirs;
  }

  

Parameters:
fs
rootdir
Returns:
All the table directories under rootdir. Ignore non table hbase folders such as .logs, .oldlogs, .corrupt folders.
Throws:
java.io.IOException
  public static List<PathgetLocalTableDirs(final FileSystem fsfinal Path rootdir)
      throws IOException {
    // presumes any directory under hbase.rootdir is a table
    FileStatus[] dirs = fs.listStatus(rootdirnew UserTableDirFilter(fs));
    List<Pathtabledirs = new ArrayList<Path>(dirs.length);
    for (FileStatus dirdirs) {
      tabledirs.add(dir.getPath());
    }
    return tabledirs;
  }

  
Checks if the given path is the one with 'recovered.edits' dir.

Parameters:
path
Returns:
True if we recovered edits
  public static boolean isRecoveredEdits(Path path) {
  }

  
Filter for all dirs that don't start with '.'
  public static class RegionDirFilter implements PathFilter {
    // This pattern will accept 0.90+ style hex region dirs and older numeric region dir names.
    final public static Pattern regionDirPattern = Pattern.compile("^[0-9a-f]*$");
    final FileSystem fs;
    public RegionDirFilter(FileSystem fs) {
      this. = fs;
    }
    @Override
    public boolean accept(Path rd) {
      if (!.matcher(rd.getName()).matches()) {
        return false;
      }
      try {
        return .getFileStatus(rd).isDir();
      } catch (IOException ioe) {
        // Maybe the file was moved or the fs was disconnected.
        .warn("Skipping file " + rd +" due to IOException"ioe);
        return false;
      }
    }
  }

  
Given a particular table dir, return all the regiondirs inside it, excluding files such as .tableinfo

Parameters:
fs A file system for the Path
tableDir Path to a specific table directory <hbase.rootdir>/<tabledir>
Returns:
List of paths to valid region directories in table dir.
Throws:
java.io.IOException
  public static List<PathgetRegionDirs(final FileSystem fsfinal Path tableDirthrows IOException {
    // assumes we are in a table dir.
    FileStatus[] rds = fs.listStatus(tableDirnew RegionDirFilter(fs));
    List<PathregionDirs = new ArrayList<Path>(rds.length);
    for (FileStatus rdfsrds) {
      Path rdPath = rdfs.getPath();
      regionDirs.add(rdPath);
    }
    return regionDirs;
  }

  
Filter for all dirs that are legal column family names. This is generally used for colfam dirs <hbase.rootdir>/<tabledir>/<regiondir>/<colfamdir>.
  public static class FamilyDirFilter implements PathFilter {
    final FileSystem fs;
    public FamilyDirFilter(FileSystem fs) {