Title: GNU Flymake -- an on-the-fly syntax checker for GNU Emacs
Version: 0.3
Environment: GNU emacs, GNU make
Description: Performs on-the-fly syntax checks of the files being edited
using the external syntax check tool (usually the compiler).
Highlights erroneous lines and displays associated
error messages.
Switching to the GNU Emacs/GNU Make environment from the Microsoft Visual Studio 6.0/Visual Assist 6.0 pair had both positive and negative consequences. On one hand, memory consumption was dramatically reduced, making many things work much faster then they used to. On the other hand, such nice IDE features as context aware auto-completion and auto parameter info were no longer available. This resulted in an increased amount of typos and errors, not to be revealed until compilation time.
Furthermore, even Visual Assist was not able to correctly parse all of the C++ constructs I've been using. This is especially true for non-trivial templates as well as macros. In the end, the ultimate tool for checking syntax is the compiler, as it can easily handle any valid constructs and report errors if any. So, having compiler make on-the-fly syntax checks of the source code being edited seems like a good idea.
Users of some modern Java IDEs, like IntelliJ IDEA, enjoy this feature for a relatively long time. I don't know of any C++ IDE with this feature, and Visual C++ 6.0/.NET lack it as well. So came the idea of developing of an on-the-fly syntax checker for Emacs -- flymake.
Flymake is implemented as an Emacs minor mode. It runs the syntax check tool (the compiler for C++ files, perl for perl files, etc.) in the background, passing it a temporary copy of the current buffer and parses the output for known error/warning message patterns. Flymake then highlights erroneous lines (that is, lines for which at least one error or warning has been reported), and displays an overall buffer status in the mode line, as shown on the figure below.
File Func.c here has 2 errors and 1 warning. Using
flymake-goto-next-error, it is posible to jump to erroneous line 7,
highlighted in red, and calling
flymake-display-err-menu-for-current-line will popup a menu
containing error text provided by the compiler. Line 4 is highlighted in blue,
as it only has a warning attached to it. Line 1 is red, but as calling
flymake-display-err-menu-for-current-line shows, the error really
belongs to line 5 of Func.h.
Selecting the menu item will jump to line 5 of Func.h, which has
the same error, as shown on the next figure.
Notice that line 1 of Func.h is highlighted in red, as
Func.c, which was actually used to syntax check the .h file, still
contains 2 errors and 1 warning.
Syntax check is done 'on-the-fly', using some external check tool, which is
most commonly a compiler (gcc for the pictures above).
Syntax check is started whenever a) buffer is loaded, b) a newline character
is added to the buffer, and c) some changes were made to the buffer more than t
seconds ago (t is configurable).
When flymake mode is active, any of the 3 conditions stated above will cause flymake to try to syntax check the current buffer. Flymake first determines whether it is able to do syntax check. It then saves a copy of the buffer in a temporary file in the buffer's directory (or in the system temp directory -- for java files), creates a syntax check command and launches a process with this command. The output is parsed using known error message patterns, and error information (file name, line number, type and text) is saved. After the process has finished, flymake highlights erroneous lines in the buffer using the accumulated error information.
A simple algorithm based of buffer filename is used to determine whether a buffer can be syntax checked. There are 3 possible options:
Syntax check is configured by mapping a regular expression for a filename to the following 3 functions:
Flymake contains implementations of all functionality required to support different syntax check modes described above (making temporary copies, finding master files, etc.), as well as some tool-specific code.
Mapping of filename regexp to syntax check functions is contained in flymake-allowed-file-name-masks.
For example, the following entry is used for c sources:
(".+\\.c$" flymake-simple-make-init flymake-simple-cleanup flymake-get-real-file-name)
flymake-simple-make-init creates a temporary copy and configures make command line,
flymake-simple-cleanup deletes the temporary copy, and flymake-get-real-file-name is a universal
function for resolving file names.
Syntax check is considered possible for a buffer if there's a corresponding entry in
flymake-allowed-file-name-masks.
After the possibility of the syntax check has been determined, a temporary
copy of the current buffer is made so that the most recent unsaved changes could
be seen by the syntax check tool. Making a copy is quite straightforward in a
standalone case (mode 1), as it's just saving buffer contents in a
file. Things get trickier, however, when master file is involved, as it requires
to a) locate master file and b) patch it to include the current file using its
new (temporary) name. Locating master file is discussed in the following section.
Patching just changes all appropriate lines of the master file so that they
use the new (temporary) name of the current file. For example, suppose current
file name is File.h, the master file is File.cpp, and
it includes current file via #include "File.h". Current file's copy
is saved to file File_flymake.h, so the include line must be
changed to #include "File_flymake.h". Finally, patched master file
is saved to File_flymk_master.cpp, and the last one is passed to
the syntax check tool.
Master file is located in two steps.
First, a list of possible master files is built. Again, a simple name
matching is used to find the files. For a C++ header File.h,
flymake searches for all .cpp files in the directories whose relative paths are
stored in a customizable variable flymake-master-file-dirs, which
usually contains something like ("." "./src"). No more than
flymake-master-file-count-limit entries is added to the master file
list. The list is then sorted to move files with names File.cpp to
the top.
Next, each master file in a list is checked to contain the appropriate
include directives. No more than flymake-check-file-limit of each
file are parsed.
For File.h, the include directives to look for are
#include "File.h", #include "../File.h", etc. Each
include is checked via the include directories (next section) to be sure it
points to the correct File.h.
First matching master file found stops the search. The master file is then patched and saved to disk. In case no master file was found, syntax check is aborted, and corresponding status (!) is reported in the mode line.
Two sets of include directories are distinguished: system include directories
and project include directories. The former is just the contents of the
INCLUDE environment variable. The latter is not so easy to obtain,
and the way it can be obtained can vary greatly for different project.
Therefore, a customizable variable
flymake-get-project-include-dirs-function is used to provide the
way to implement the desired behavior.
The default implementation, flymake-get-project-include-dirs-imp,
uses a make call. This requires a correct base directory, that is, a
directory containing a correct Makefile, be determined.
As obtaining the project include directories might be a costly operation, its return value is cached in the hash table. The cache is cleared in the beginning of every syntax check attempt.
Flymake can be configured to use different tools for performing syntax checks. For example, direct compiler
call to syntax check a perl script or a call to make for a more complicated case of a C++ source.
The general idea is that simple files, like perl scripts and html pages, can be checked by directly invoking
a corresponding tool. Files that are usually more complex and generally used as part of larger projects, might require
non-trivial options be passed to the syntax check tool, like include directories for C++. The latter files are syntax
checked via a call to make.
All make configuration data is usually stored in a file called
Makefile. To allow for future extensions, flymake uses a notion of
buildfile to reference the 'project configuration' file.
Special function, flymake-find-buildfile is provided for locating buildfiles.
Searching for a buildfile is done in a manner similar to that of searching
for possible master files. A customizable variable
flymake-buildfile-dirs holds a list of relative paths to the
buildfile. They are checked sequentially until a buildfile is found. In case
there's no build file, syntax check is aborted. Buildfile values are also cached.
The command line for launching a process is returned by the initialization function. Flymake then just
calls start-process to start an asynchronous process and configures process filter and sentinel which
is used for processing the output of the syntax check tool.
The output generated by the syntax check tool is parsed in the process filter
using the error message patterns stored in the
flymake-err-line-patterns variable. This variable contains a list
of items of the form (regexp file-idx line-idx err-text-idx), used
to determine whether a particular line is an error message and extract file
name, line number and error text, respectively. Error type (error/warning) is
also guessed by matching error text with the '^[wW]arning' pattern. Anything that
was not classified as a warning is considered an error. Type is then used to
sort error menu items, which shows error messages first.
Flymake is also able to interpret error message patterns missing err-text-idx
information. This is done by merely taking the rest of the matched line
((substring line (match-end 0))) as error text. This trick allows
to make use of a huge collection of error message line patterns from
compile.el. All these error patterns are appended to
the end of flymake-err-line-patterns.
The error information obtained is saved in a buffer local variable. The buffer for which the process output belongs is determined from the process-id -- buffer mapping updated after every process launch/exit.
Highlighting is implemented with overlays and happens in the process
sentinel, after calling the cleanup function. 2 customizable faces are
used: flymake-errline-face and flymake-warnline-face.
Errors belonging outside the current buffer are considered to belong to line 1
of the current buffer.
After syntax check is finished, its results are displayed in the mode line. The following statuses are defined.
| Flymake* or Flymake:E/W* | Flymake is currently running. For the second case, E/W contains the error and warning count for the previous run. |
| Flymake | Syntax check is not running. Usually this means syntax check was
successfully passed (no errors, no warnings). Other possibilities are:
syntax check was killed as a result of executing
flymake-compile, or syntax check cannot start as compilation
is currently in progress. |
| Flymake:E/W | Number of errors/warnings found by the syntax check process. |
| Flymake:! | Flymake was unable to find master file for the current buffer. |
The following errors cause a warning message and switch flymake mode OFF for the buffer.
| CFGERR | Syntax check process returned nonzero exit code, but no errors/warnings were reported. This indicates a possible configuration error. |
| NOMASTER | Flymake was unable to find master file for the current buffer. |
| NOMK | Flymake was unable to find buildfile for the current buffer. |
| PROCERR | Flymake was unable to launch syntax check process. |
For each erroneous line, a popup menu with all errors reported for that line
can be displayed. It is built from the error information stored in the buffer
local variables. Selecting the item whose error belongs to another file brings
forward that file with the help of the flymake-goto-file-and-line
function.
The only mode flymake currently knows about is compile.
Flymake can be configured to not start syntax check if it thinks the
compilation is in progress. The check is made by the
flymake-compilation-is-running, which tests the
compilation-in-progress variable. The reason why this might be
useful is saving CPU time in case both syntax check and compilation are very CPU
intensive. The original reason for adding this feature, though, was working
around a locking problem with Visual C++ compiler.
Flymake also provides an alternative command for starting compilation,
flymake-compile:
(defun flymake-compile()
"kill all flymake syntax checks, start compilation"
(interactive)
(flymake-stop-all-syntax-checks)
(call-interactively 'compile)
)
It just kills all the active syntax check processes before calling
compile.
The whole idea of the on-the-fly syntax check will be compromised in case any single syntax check will require a significant amount of time to complete. So flymake is certainly meant to be used on a fast machine. Also, any possible optimizations, like switching on precompiled headers generation, will be of great help.
To give you some numbers, here's the check times for several C++ source files compiled on a Pentium-IV 1.7 GHz machine.
| Number of lines in a file | Precompiled headers | Syntax check time, in seconds |
| 1000 | ON | 1.0 |
| 1000 | OFF | 3.6 |
| 500 | ON | 0.7 |
| 500 | OFF | 3.1 |
| 10 | ON | 0.5 |
| 10 | OFF | 0.1 |
1 second delay is barely noticeable, and even with a 2-3 seconds delay on-the-fly syntax check might still be very useful.
One of ideas behind Flymake was to create a universal syntax checker. Basically, all that is necessary to use flymake with a certain syntax check tool is to provide the error message patterns for that tool (in case they aren't already there) and write the code to create the command line to be used to run the tool.
The following table provides a summary of languages/compilers flymake was tested with.
| Language | Compiler | OS |
| C/C++ | Microsoft Visual C++ 6.0 | Windows 2000 |
| C/C++ | gcc 2.9 | Windows 2000/XP/Linux |
| MS IDL | midl 5.01 | Windows 2000 |
| Java | IBM Jikes 1.17 | Windows 2000 |
| Latex | MikTeX 2.2 | Windows 2000 |
| HTML | HTML Tidy | Windows 2000/XP |
| XML | XMLStarlet Command Line XML Toolkit 0.7.0 | Windows 2000/XP |
| Perl | Perl 5.8 | Windows 2000/XP |
Flymake makes syntax check attempts quite often. It might be unable to launch syntax check process for some reasons (no master file, etc.). Nevertheless, the failed attempts themselves might take some time (getting include directories, looking for a master file), interfering with the editing process, which is sometimes annoying. The possible solution is to switch off flymake-mode for such buffers.
Algorithms used for locating buildfiles and master files are quite simple, and won't cover all the use cases.
Flymake was primarily tested in Windows (Windows 2000, XP) environment, with only a few tests made on Linux (KNOPPIX 3.2 RE). So, switching to other operating system/build tool/compiler might require some customizations or even fixes.
Flymake might be of great help for editing the source code. When seen immediately, errors and warnings are much easier to fix. Warnings issued during compilation are sometimes left alone, as the project still 'builds OK' (unless 'treat warnings as errors' option is used). With flymake, those warnings will be displayed immediately, and so more likely be fixed.
Flymake requires quite a fast machine to run, but fast machines are not so rare nowadays.