Start line:  
End line:  

Snippet Preview

Snippet HTML Code

Stack Overflow Questions
Copyright 2010 Google Inc. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS-IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
 import java.util.List;
AppsLogWriter is responsible for batching application logs for a single request and sending them back to the AppServer via the LogService.Flush API call.

The current algorithm used to send logs is as follows:

  • The code never allows more than byteCountBeforeFlush bytes of log data to accumulate in the buffer. If adding a new log line would exceed that limit, the current set of logs are removed from it and an asynchronous API call is started to flush the logs before buffering the new line.
  • If another flush occurs while a previous flush is still pending, the caller will block synchronously until the previous call completed.
  • When the overall request completes is should call

    waitForCurrentFlushAndStartNewFlush} and report the flush count as a HTTP response header. The vm_runtime on the appserver will wait for the reported number of log flushes before forwarding the HTTP response to the user.

This class is also responsible for splitting large log entries into smaller fragments, which is unrelated to the batching mechanism described above but is necessary to prevent the AppServer from truncating individual log entries.

This class is thread safe and all methods accessing local state are synchronized. Since each request have their own instance of this class the only contention possible is between the original request thread and and any child RequestThreads created by the request through the threading API.

 class VmAppLogsWriter {
     private static final Logger logger =
     static final String LOG_CONTINUATION_SUFFIX = "\n<continued in next message>";
     static final String LOG_CONTINUATION_PREFIX = "<continued from previous message>\n";
     static final int MIN_MAX_LOG_MESSAGE_LENGTH = 1024;
     static final int LOG_FLUSH_TIMEOUT_MS = 2000;
     private final int maxLogMessageLength;
     private final int logCutLength;
     private final int logCutLengthDiv10;
     private final List<UserAppLogLine> buffer;
     private final long maxBytesToFlush;
     private long currentByteCount;
     private final int maxSecondsBetweenFlush;
     private int flushCount = 0;
     private Future<byte[]> currentFlush;
     private Stopwatch stopwatch;

Construct an AppLogsWriter instance.

buffer Buffer holding messages between flushes.
maxBytesToFlush The maximum number of bytes of log message to allow in a single flush. The code flushes any cached logs before reaching this limit. If this is 0, AppLogsWriter will not start an intermediate flush based on size.
maxLogMessageLength The maximum length of an individual log line. A single log line longer than this will be written as multiple log entries (with the continuation prefix/suffixes added to indicate this).
maxFlushSeconds The amount of time to allow a log line to sit cached before flushing. Once a log line has been sitting for more than the specified time, all currently cached logs are flushed. If this is 0, no time based flushing occurs. N.B. because we only check the time on a log call, it is possible for a log to stay cached long after the specified time has been reached. Consider this example (assume maxFlushSeconds=60): the app logs a message when the handler starts but then does not log another message for 10 minutes. The initial log will stay cached until the second message is logged.
    public VmAppLogsWriter(List<UserAppLogLine> bufferlong maxBytesToFlushint maxLogMessageLength,
                           int maxFlushSeconds) {
        this. = buffer;
        this. = maxFlushSeconds;
        if (maxLogMessageLength < ) {
            String message = String.format(
                    "maxLogMessageLength sillily small (%s); setting maxLogMessageLength to %s",
            this. = ;
        } else {
            this. = maxLogMessageLength;
         = maxLogMessageLength - ;
         =  / 10;
        if (maxBytesToFlush < this.) {
            String message = String.format(
                    "maxBytesToFlush (%s) smaller than  maxLogMessageLength (%s)",
            this. = this.;
        } else {
            this. = maxBytesToFlush;
         = Stopwatch.createUnstarted();

Add the specified LogRecord for the current request. If enough space (or in the future, time) has accumulated, an asynchronous flush may be started. If flushes are backed up, this method may block.
    synchronized void addLogRecordAndMaybeFlush(LogRecord fullRecord) {
        for (LogRecord record : split(fullRecord)) {
            UserAppLogLine logLine = new UserAppLogLine();
            int maxEncodingSize = logLine.maxEncodingSize();
            if ( > 0 &&
                    ( + maxEncodingSize) > ) {
                .info( + " bytes of app logs pending, starting flush...");
            if (.size() == 0) {
             += maxEncodingSize;
        if ( > 0 &&
                .elapsed(.) >= ) {

Starts an asynchronous flush. This method may block if flushes are backed up.

The number of times this AppLogsWriter has initiated a flush.
    synchronized int waitForCurrentFlushAndStartNewFlush() {
        if (.size() > 0) {
             = doFlush();
        return ;

Initiates a synchronous flush. This method will always block until any pending flushes and its own flush completes.
    synchronized void flushAndWait() {
        if (.size() > 0) {
             = doFlush();

This method blocks until any outstanding flush is completed. This method should be called prior to doFlush() so that it is impossible for the appserver to process logs out of order.
    private void waitForCurrentFlush() {
        if ( != null) {
            .info("Previous flush has not yet completed, blocking.");
            try {
                        . + ,
            } catch (InterruptedException ex) {
                .warning("Interruped while blocking on a log flush, setting interrupt bit and " +
                        "continuing.  Some logs may be lost or occur out of order!");
            } catch (TimeoutException e) {
                .log(."Timeout waiting for log flush to complete. "
                        + "Log messages may have been lost/reordered!"e);
            } catch (ExecutionException ex) {
                        "A log flush request failed.  Log messages may have been lost!"ex);
             = null;
    private Future<byte[]> doFlush() {
        UserAppLogGroup group = new UserAppLogGroup();
        for (UserAppLogLine logLine : ) {
         = 0;
        FlushRequest request = new FlushRequest();
        ApiConfig apiConfig = new ApiConfig();
        apiConfig.setDeadlineInSeconds( / 1000.0);
        return ApiProxy.makeAsyncCall("logservice""Flush"request.toByteArray(), apiConfig);

Because the App Server will truncate log messages that are too long, we want to split long log messages into mutliple messages. This method returns a List of LogRecords, each of which have the same LogRecord.getLevel() and LogRecord.getTimestamp() as this one, and whose LogRecord.getMessage() is short enough that it will not be truncated by the App Server. If the message of this LogRecord is short enough, the list will contain only this LogRecord. Otherwise the list will contain multiple LogRecords each of which contain a portion of the message. Additionally, strings will be prepended and appended to each of the messages indicating that the message is continued in the following log message or is a continuation of the previous log mesage.
    List<LogRecord> split(LogRecord aRecord) {
        LinkedList<LogRecord> theList = new LinkedList<LogRecord>();
        String message = aRecord.getMessage();
        if (null == message || message.length() <= ) {
            return theList;
        String remaining = message;
        while (remaining.length() > 0) {
            String nextMessage;
            if (remaining.length() <= ) {
                nextMessage = remaining;
                remaining = "";
            } else {
                int cutLength = ;
                boolean cutAtNewline = false;
                int friendlyCutLength = remaining.lastIndexOf('\n');
                if (friendlyCutLength > ) {
                    cutLength = friendlyCutLength;
                    cutAtNewline = true;
                nextMessage = remaining.substring(0, cutLength) + ;
                remaining = remaining.substring(cutLength + (cutAtNewline ? 1 : 0));
                if (remaining.length() >  ||
                        remaining.length() +  <= ) {
                    remaining =  + remaining;
            theList.add(new LogRecord(aRecordnextMessage));
        return theList;
New to GrepCode? Check out our FAQ X