Start line:  
End line:  

Snippet Preview

Snippet HTML Code

Stack Overflow Questions
Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
 
 package org.apache.hadoop.hbase.util;
 
 import java.util.List;
 import java.util.Map;
 
 
Implementation of org.apache.hadoop.hbase.TableDescriptors that reads descriptors from the passed filesystem. It expects descriptors to be in a file in the TABLEINFO_DIR subdir of the table's directory in FS. Can be read-only -- i.e. does not modify the filesystem or can be read and write.

Also has utility for keeping up the table descriptors tableinfo file. The table schema file is kept in the TABLEINFO_DIR subdir of the table directory in the filesystem. It has a TABLEINFO_FILE_PREFIX and then a suffix that is the edit sequenceid: e.g. .tableinfo.0000000003. This sequenceid is always increasing. It starts at zero. The table schema file with the highest sequenceid has the most recent schema edit. Usually there is one file only, the most recent but there may be short periods where there are more than one file. Old files are eventually cleaned. Presumption is that there will not be lots of concurrent clients making table schema edits. If so, the below needs a bit of a reworking and perhaps some supporting api in hdfs.

 
 public class FSTableDescriptors implements TableDescriptors {
   private static final Log LOG = LogFactory.getLog(FSTableDescriptors.class);
   private final FileSystem fs;
   private final Path rootdir;
   private final boolean fsreadonly;
   private volatile boolean usecache;
   private volatile boolean fsvisited;
 
   @VisibleForTesting long cachehits = 0;
   @VisibleForTesting long invocations = 0;

  
The file name prefix used to store HTD in HDFS
 
   static final String TABLEINFO_FILE_PREFIX = ".tableinfo";
   static final String TABLEINFO_DIR = ".tabledesc";
   static final String TMP_DIR = ".tmp";
 
   // This cache does not age out the old stuff.  Thinking is that the amount
   // of data we keep up in here is so small, no need to do occasional purge.
   // TODO.
   private final Map<TableNameHTableDescriptorcache =
Table descriptor for hbase:meta catalog table
 
   private final HTableDescriptor metaTableDescriptor;

  
Construct a FSTableDescriptors instance using the hbase root dir of the given conf and the filesystem where that root dir lives. This instance can do write operations (is not read only).
  public FSTableDescriptors(final Configuration confthrows IOException {
    this(conf, FSUtils.getCurrentFileSystem(conf), FSUtils.getRootDir(conf));
  }
  public FSTableDescriptors(final Configuration conffinal FileSystem fsfinal Path rootdir)
      throws IOException {
    this(conffsrootdirfalsetrue);
  }

  

Parameters:
fsreadonly True if we are read-only when it comes to filesystem operations; i.e. on remove, we do not do delete in fs.
  public FSTableDescriptors(final Configuration conffinal FileSystem fs,
    final Path rootdirfinal boolean fsreadonlyfinal boolean usecachethrows IOException {
    super();
    this. = fs;
    this. = rootdir;
    this. = fsreadonly;
    this. = usecache;
    this. = HTableDescriptor.metaTableDescriptor(conf);
  }
  public void setCacheOn() throws IOException {
    this..clear();
    this. = true;
  }
  public void setCacheOff() throws IOException {
    this. = false;
    this..clear();
  }
  public boolean isUsecache() {
    return this.;
  }

  
Get the current table descriptor for the given table, or null if none exists. Uses a local cache of the descriptor but still checks the filesystem on each call to see if a newer file has been created since the cached one was read.
  public HTableDescriptor get(final TableName tablename)
  throws IOException {
    ++;
    if (..equals(tablename)) {
      ++;
      return ;
    }
    // hbase:meta is already handled. If some one tries to get the descriptor for
    // .logs, .oldlogs or .corrupt throw an exception.
      throw new IOException("No descriptor found for non table = " + tablename);
    }
    if () {
      // Look in cache of descriptors.
      HTableDescriptor cachedtdm = this..get(tablename);
      if (cachedtdm != null) {
        ++;
        return cachedtdm;
      }
    }
    HTableDescriptor tdmt = null;
    try {
      tdmt = getTableDescriptorFromFs(tablename, !);
    } catch (NullPointerException e) {
      .debug("Exception during readTableDecriptor. Current table name = "
                  + tablenamee);
    } catch (IOException ioe) {
      .debug("Exception during readTableDecriptor. Current table name = "
                  + tablenameioe);
    }
    // last HTD written wins
    if ( && tdmt != null) {
      this..put(tablenametdmt);
    }
    return tdmt;
  }

  
Returns a map from table name to table descriptor for all tables.
  throws IOException {
    if ( && ) {
      for (Map.Entry<TableNameHTableDescriptorentrythis..entrySet()) {
        htds.put(entry.getKey().toString(), entry.getValue());
      }
      // add hbase:meta to the response
    } else {
      .debug("Fetching table descriptors from the filesystem.");
      boolean allvisited = true;
      for (Path d : FSUtils.getTableDirs()) {
        HTableDescriptor htd = null;
        try {
          htd = get(FSUtils.getTableName(d));
        } catch (FileNotFoundException fnfe) {
          // inability of retrieving one HTD shouldn't stop getting the remaining
          .warn("Trouble retrieving htd"fnfe);
        }
        if (htd == null) {
          allvisited = false;
          continue;
        } else {
          htds.put(htd.getTableName().getNameAsString(), htd);
        }
         = allvisited;
      }
    }
    return htds;
  }
  /* (non-Javadoc)
   * @see org.apache.hadoop.hbase.TableDescriptors#getTableDescriptors(org.apache.hadoop.fs.FileSystem, org.apache.hadoop.fs.Path)
   */
  throws IOException {
    List<PathtableDirs =
        FSUtils.getLocalTableDirs(, FSUtils.getNamespaceDir(name));
    for (Path dtableDirs) {
      HTableDescriptor htd = null;
      try {
        htd = get(FSUtils.getTableName(d));
      } catch (FileNotFoundException fnfe) {
        // inability of retrieving one HTD shouldn't stop getting the remaining
        .warn("Trouble retrieving htd"fnfe);
      }
      if (htd == nullcontinue;
      htds.put(FSUtils.getTableName(d).getNameAsString(), htd);
    }
    return htds;
  }

  
Adds (or updates) the table descriptor to the FileSystem and updates the local cache with it.
  public void add(HTableDescriptor htdthrows IOException {
    if () {
      throw new NotImplementedException("Cannot add a table descriptor - in read only mode");
    }
      throw new NotImplementedException();
    }
      throw new NotImplementedException(
        "Cannot add a table descriptor for a reserved subdirectory name: " + htd.getNameAsString());
    }
  }

  
Removes the table descriptor from the local cache and returns it. If not in read only mode, it also deletes the entire table directory(!) from the FileSystem.
  public HTableDescriptor remove(final TableName tablename)
  throws IOException {
    if () {
      throw new NotImplementedException("Cannot remove a table descriptor - in read only mode");
    }
    Path tabledir = getTableDir(tablename);
    if (this..exists(tabledir)) {
      if (!this..delete(tabledirtrue)) {
        throw new IOException("Failed delete of " + tabledir.toString());
      }
    }
    HTableDescriptor descriptor = this..remove(tablename);
    if (descriptor == null) {
      return null;
    } else {
      return descriptor;
    }
  }

  
Checks if a current table info file exists for the given table

Parameters:
tableName name of table
Returns:
true if exists
Throws:
java.io.IOException
  public boolean isTableInfoExists(TableName tableNamethrows IOException {
    return getTableInfoPath(tableName) != null;
  }

  
Find the most current table info file for the given table in the hbase root directory.

Returns:
The file status of the current table info file or null if it does not exist
  private FileStatus getTableInfoPath(final TableName tableNamethrows IOException {
    Path tableDir = getTableDir(tableName);
    return getTableInfoPath(tableDir);
  }
  private FileStatus getTableInfoPath(Path tableDir)
  throws IOException {
    return getTableInfoPath(tableDir, !);
  }

  
Find the most current table info file for the table located in the given table directory. Looks within the TABLEINFO_DIR subdirectory of the given directory for any table info files and takes the 'current' one - meaning the one with the highest sequence number if present or no sequence number at all if none exist (for backward compatibility from before there were sequence numbers).

Returns:
The file status of the current table info file or null if it does not exist
Throws:
java.io.IOException
  public static FileStatus getTableInfoPath(FileSystem fsPath tableDir)
  throws IOException {
    return getTableInfoPath(fstableDirfalse);
  }

  
Find the most current table info file for the table in the given table directory. Looks within the TABLEINFO_DIR subdirectory of the given directory for any table info files and takes the 'current' one - meaning the one with the highest sequence number if present or no sequence number at all if none exist (for backward compatibility from before there were sequence numbers). If there are multiple table info files found and removeOldFiles is true it also deletes the older files.

Returns:
The file status of the current table info file or null if none exist
Throws:
java.io.IOException
  private static FileStatus getTableInfoPath(FileSystem fsPath tableDirboolean removeOldFiles)
  throws IOException {
    Path tableInfoDir = new Path(tableDir);
    return getCurrentTableInfoStatus(fstableInfoDirremoveOldFiles);
  }

  
Find the most current table info file in the given directory Looks within the given directory for any table info files and takes the 'current' one - meaning the one with the highest sequence number if present or no sequence number at all if none exist (for backward compatibility from before there were sequence numbers). If there are multiple possible files found and the we're not in read only mode it also deletes the older files.

Returns:
The file status of the current table info file or null if it does not exist
Throws:
java.io.IOException
  // only visible for FSTableDescriptorMigrationToSubdir, can be removed with that
  static FileStatus getCurrentTableInfoStatus(FileSystem fsPath dirboolean removeOldFiles)
  throws IOException {
    FileStatus [] status = FSUtils.listStatus(fsdir);
    if (status == null || status.length < 1) return null;
    FileStatus mostCurrent = null;
    for (FileStatus file : status) {
      if (mostCurrent == null || .compare(filemostCurrent) < 0) {
        mostCurrent = file;
      }
    }
    if (removeOldFiles && status.length > 1) {
      // Clean away old versions
      for (FileStatus file : status) {
        Path path = file.getPath();
        if (file != mostCurrent) {
          if (!fs.delete(file.getPath(), false)) {
            .warn("Failed cleanup of " + path);
          } else {
            .debug("Cleaned up old tableinfo file " + path);
          }
        }
      }
    }
    return mostCurrent;
  }

  
Compare org.apache.hadoop.fs.FileStatus instances by org.apache.hadoop.fs.Path.getName(). Returns in reverse order.
  new Comparator<FileStatus>() {
    @Override
    public int compare(FileStatus leftFileStatus right) {
      return right.compareTo(left);
    }};

  
Return the table directory in HDFS
  @VisibleForTesting Path getTableDir(final TableName tableName) {
    return FSUtils.getTableDir(tableName);
  }
  private static final PathFilter TABLEINFO_PATHFILTER = new PathFilter() {
    @Override
    public boolean accept(Path p) {
      // Accept any file that starts with TABLEINFO_NAME
      return p.getName().startsWith();
    }};

  
Width of the sequenceid that is a suffix on a tableinfo file.
  @VisibleForTesting static final int WIDTH_OF_SEQUENCE_ID = 10;
  /*
   * @param number Number to use as suffix.
   * @return Returns zero-prefixed decimal version of passed
   * number (Does absolute in case number is negative).
   */
  private static String formatTableInfoSequenceId(final int number) {
    byte [] b = new byte[];
    int d = Math.abs(number);
    for (int i = b.length - 1; i >= 0; i--) {
      b[i] = (byte)((d % 10) + '0');
      d /= 10;
    }
    return Bytes.toString(b);
  }

  
Regex to eat up sequenceid suffix on a .tableinfo file. Use regex because may encounter oldstyle .tableinfos where there is no sequenceid on the end.
  private static final Pattern TABLEINFO_FILE_REGEX =
    Pattern.compile( + "(\\.([0-9]{" +  + "}))?$");

  

Parameters:
p Path to a .tableinfo file.
Returns:
The current editid or 0 if none found.
  @VisibleForTesting static int getTableInfoSequenceId(final Path p) {
    if (p == nullreturn 0;
    if (!m.matches()) throw new IllegalArgumentException(p.toString());
    String suffix = m.group(2);
    if (suffix == null || suffix.length() <= 0) return 0;
    return Integer.parseInt(m.group(2));
  }

  

Parameters:
sequenceid
Returns:
Name of tableinfo file.
  @VisibleForTesting static String getTableInfoFileName(final int sequenceid) {
    return  + "." + formatTableInfoSequenceId(sequenceid);
  }

  
Returns the latest table descriptor for the given table directly from the file system if it exists, bypassing the local cache. Returns null if it's not found.
    Path hbaseRootDirTableName tableNamethrows IOException {
    Path tableDir = FSUtils.getTableDir(hbaseRootDirtableName);
    return getTableDescriptorFromFs(fstableDir);
  }

  
Returns the latest table descriptor for the table located at the given directory directly from the file system if it exists.

Throws:
org.apache.hadoop.hbase.TableInfoMissingException if there is no descriptor
    Path hbaseRootDirTableName tableNameboolean rewritePbthrows IOException {
    Path tableDir = FSUtils.getTableDir(hbaseRootDirtableName);
    return getTableDescriptorFromFs(fstableDirrewritePb);
  }

  
Returns the latest table descriptor for the table located at the given directory directly from the file system if it exists.

Throws:
org.apache.hadoop.hbase.TableInfoMissingException if there is no descriptor
  public static HTableDescriptor getTableDescriptorFromFs(FileSystem fsPath tableDir)
  throws IOException {
    return getTableDescriptorFromFs(fstableDirfalse);
  }

  
Returns the latest table descriptor for the table located at the given directory directly from the file system if it exists.

Throws:
org.apache.hadoop.hbase.TableInfoMissingException if there is no descriptor
  public static HTableDescriptor getTableDescriptorFromFs(FileSystem fsPath tableDir,
    boolean rewritePb)
  throws IOException {
    FileStatus status = getTableInfoPath(fstableDirfalse);
    if (status == null) {
      throw new TableInfoMissingException("No table descriptor file under " + tableDir);
    }
    return readTableDescriptor(fsstatusrewritePb);
  }
  private static HTableDescriptor readTableDescriptor(FileSystem fsFileStatus status,
    boolean rewritePb)
  throws IOException {
    int len = Ints.checkedCast(status.getLen());
    byte [] content = new byte[len];
    FSDataInputStream fsDataInputStream = fs.open(status.getPath());
    try {
      fsDataInputStream.readFully(content);
    } finally {
      fsDataInputStream.close();
    }
    HTableDescriptor htd = null;
    try {
      htd = HTableDescriptor.parseFrom(content);
    } catch (DeserializationException e) {
      // we have old HTableDescriptor here
      try {
        HTableDescriptor ohtd = HTableDescriptor.parseFrom(content);
        .warn("Found old table descriptor, converting to new format for table " +
                   ohtd.getTableName());
        htd = new HTableDescriptor(ohtd);
        if (rewritePbrewriteTableDescriptor(fsstatushtd);
      } catch (DeserializationException e1) {
        throw new IOException("content=" + Bytes.toShort(content), e1);
      }
    }
    if (rewritePb && !ProtobufUtil.isPBMagicPrefix(content)) {
      // Convert the file over to be pb before leaving here.
      rewriteTableDescriptor(fsstatushtd);
    }
    return htd;
  }
  private static void rewriteTableDescriptor(final FileSystem fsfinal FileStatus status,
    final HTableDescriptor td)
  throws IOException {
    Path tableInfoDir = status.getPath().getParent();
    Path tableDir = tableInfoDir.getParent();
    writeTableDescriptor(fstdtableDirstatus);
  }

  
Update table descriptor on the file system

Throws:
java.io.IOException Thrown if failed update.
org.apache.commons.lang.NotImplementedException if in read only mode
  throws IOException {
    if () {
      throw new NotImplementedException("Cannot update a table descriptor - in read only mode");
    }
    Path tableDir = getTableDir(htd.getTableName());
    Path p = writeTableDescriptor(htdtableDirgetTableInfoPath(tableDir));
    if (p == nullthrow new IOException("Failed update");
    .info("Updated tableinfo=" + p);
    if () {
      this..put(htd.getTableName(), htd);
    }
    return p;
  }

  
Deletes all the table descriptor files from the file system. Used in unit tests only.

  public void deleteTableDescriptorIfExists(TableName tableNamethrows IOException {
    if () {
      throw new NotImplementedException("Cannot delete a table descriptor - in read only mode");
    }
    Path tableDir = getTableDir(tableName);
    Path tableInfoDir = new Path(tableDir);
  }

  
Deletes files matching the table info file pattern within the given directory whose sequenceId is at most the given max sequenceId.
  private static void deleteTableDescriptorFiles(FileSystem fsPath dirint maxSequenceId)
  throws IOException {
    FileStatus [] status = FSUtils.listStatus(fsdir);
    for (FileStatus file : status) {
      Path path = file.getPath();
      int sequenceId = getTableInfoSequenceId(path);
      if (sequenceId <= maxSequenceId) {
        boolean success = FSUtils.delete(fspathfalse);
        if (success) {
          .debug("Deleted table descriptor at " + path);
        } else {
          .error("Failed to delete descriptor at " + path);
        }
      }
    }
  }

  
Attempts to write a new table descriptor to the given table's directory. It first writes it to the .tmp dir then uses an atomic rename to move it into place. It begins at the currentSequenceId + 1 and tries 10 times to find a new sequence number not already in use. Removes the current descriptor file if passed in.

Returns:
Descriptor file or null if we failed write.
  private static Path writeTableDescriptor(final FileSystem fs,
    final HTableDescriptor htdfinal Path tableDir,
    final FileStatus currentDescriptorFilethrows IOException {
    // Get temporary dir into which we'll first write a file to avoid half-written file phenomenon.
    // This directory is never removed to avoid removing it out from under a concurrent writer.
    Path tmpTableDir = new Path(tableDir);
    Path tableInfoDir = new Path(tableDir);
    // What is current sequenceid?  We read the current sequenceid from
    // the current file.  After we read it, another thread could come in and
    // compete with us writing out next version of file.  The below retries
    // should help in this case some but its hard to do guarantees in face of
    // concurrent schema edits.
    int currentSequenceId = currentDescriptorFile == null ? 0 :
      getTableInfoSequenceId(currentDescriptorFile.getPath());
    int newSequenceId = currentSequenceId;
    // Put arbitrary upperbound on how often we retry
    int retries = 10;
    int retrymax = currentSequenceId + retries;
    Path tableInfoDirPath = null;
    do {
      newSequenceId += 1;
      String filename = getTableInfoFileName(newSequenceId);
      Path tempPath = new Path(tmpTableDirfilename);
      if (fs.exists(tempPath)) {
        .debug(tempPath + " exists; retrying up to " + retries + " times");
        continue;
      }
      tableInfoDirPath = new Path(tableInfoDirfilename);
      try {
        writeHTD(fstempPathhtd);
        fs.mkdirs(tableInfoDirPath.getParent());
        if (!fs.rename(tempPathtableInfoDirPath)) {
          throw new IOException("Failed rename of " + tempPath + " to " + tableInfoDirPath);
        }
        .debug("Wrote descriptor into: " + tableInfoDirPath);
      } catch (IOException ioe) {
        // Presume clash of names or something; go around again.
        .debug("Failed write and/or rename; retrying"ioe);
        if (!FSUtils.deleteDirectory(fstempPath)) {
          .warn("Failed cleanup of " + tempPath);
        }
        tableInfoDirPath = null;
        continue;
      }
      break;
    } while (newSequenceId < retrymax);
    if (tableInfoDirPath != null) {
      // if we succeeded, remove old table info files.
      deleteTableDescriptorFiles(fstableInfoDirnewSequenceId - 1);
    }
    return tableInfoDirPath;
  }
  private static void writeHTD(final FileSystem fsfinal Path pfinal HTableDescriptor htd)
  throws IOException {
    FSDataOutputStream out = fs.create(pfalse);
    try {
      // We used to write this file out as a serialized HTD Writable followed by two '\n's and then
      // the toString version of HTD.  Now we just write out the pb serialization.
      out.write(htd.toByteArray());
    } finally {
      out.close();
    }
  }

  
Create new HTableDescriptor in HDFS. Happens when we are creating table. Used by tests.

Returns:
True if we successfully created file.
  public boolean createTableDescriptor(HTableDescriptor htdthrows IOException {
    return createTableDescriptor(htdfalse);
  }

  
Create new HTableDescriptor in HDFS. Happens when we are creating table. If forceCreation is true then even if previous table descriptor is present it will be overwritten

Returns:
True if we successfully created file.
  public boolean createTableDescriptor(HTableDescriptor htdboolean forceCreation)
  throws IOException {
    Path tableDir = getTableDir(htd.getTableName());
    return createTableDescriptorForTableDirectory(tableDirhtdforceCreation);
  }

  
Create a new HTableDescriptor in HDFS in the specified table directory. Happens when we create a new table or snapshot a table.

Parameters:
tableDir table directory under which we should write the file
htd description of the table to write
forceCreation if true,then even if previous table descriptor is present it will be overwritten
Returns:
true if the we successfully created the file, false if the file already exists and we weren't forcing the descriptor creation.
Throws:
java.io.IOException if a filesystem error occurs
  public boolean createTableDescriptorForTableDirectory(Path tableDir,
      HTableDescriptor htdboolean forceCreationthrows IOException {
    if () {
      throw new NotImplementedException("Cannot create a table descriptor - in read only mode");
    }
    FileStatus status = getTableInfoPath(tableDir);
    if (status != null) {
      .debug("Current tableInfoPath = " + status.getPath());
      if (!forceCreation) {
        if (.exists(status.getPath()) && status.getLen() > 0) {
          if (readTableDescriptor(statusfalse).equals(htd)) {
            .debug("TableInfo already exists.. Skipping creation");
            return false;
          }
        }
      }
    }
    Path p = writeTableDescriptor(htdtableDirstatus);
    return p != null;
  }
New to GrepCode? Check out our FAQ X