Here is a very rough outline of our schedule. This list is just an indication and will be updated on a regular basis. We have a few more intermediate stages in mind.
- Merging engines
- Objective: Merge the Aleph and pdfTeX codebase so that LuaTeX in DVI mode
behaves like Aleph, and in PDF mode like pdf(e)TeX. Also allow Lua callbacks
for file searching.
Status: This stage is mostly finished. LuaTeX 0.40 and higher start up as TeX82 and one has to explicitly enable extra primitives (normally this will be done by the macropackage).
- Objective: Some functionality is still tagged as pdfTeX or Omega while
in practice it concerns generic functionality. The low level code as well as some
primitives can be stripped from these properties. At the node level this means that
some whatsits will become first class nodes.
Status: This will happen from version 0.80 onwards and take some time. It is part of the general cleanup (and also involves removal of experimental code from older engines).
- OpenType fonts
- Objective: Make the Aleph functionality available in PDF mode and add
support for OpenType fonts. Allow Lua scripts to control all aspects of font
loading, font definition and manipulation. Core font support completely driven by
Lua remains a core objective (in addition to the traditional, downward compatible
Status: Most of this is finished. We use code from the FontForge project which has the advantage that one can use that open sourced tool alongside LuaTeX and get a similar view at the font. The already stripped FF library will be further stripped.
- Tokenizing and node lists
- Objective: Use Lua callbacks for various internals, complete access to
tokenizer and provide access to node lists at moments that make sense.
Status: This stage is to a large extent completed (around version 0.50) but you can expect a few more callbacks when the internals have been cleaned up. An additional node access model has been introduced in 2013 and a better tokenizer interface will show up end of 2014.
- Paragraph building
- Objective: Provide control over various aspects of paragraph building
(hyphenation, kerning, ligature building), dynamic loading loading of hyphenation
Status: Apart from some small details these objectives are met. Users can write their own paragraph builders (although this is a somewhat specialized task). At some point we will provide more context for each paragraph. We expect to have reorganized this in version 0.90.
- Metapost (mplib)
- Objective: Turn metapost into a library. Extend the library to arbitrary
precision (megapost) and add features on the wishlist. Advanced usage in LuaTeX,
like for instance runtime font generation and manipulation is not yet scheduled.
Status: The first stage of this project is finished with the first public beta and integration in LuaTeX. This subproject has been finished on schedule by the end of 2009. Future versions of LuaTeX will ship with the latest versions of the mplib library. For instance in 2013 we got support for doubles and in 2014 the decimal and binary number systems showed up. Later in 2014 there is access to Lua from this library.
- Image handling
- Objective: Image identification and loading in Lua including scaling and object management. Status: This first version of the img library showed up in the 0.22 beta and some more features have been added after that. Inclusion of PDF has been optimized in 0.50.
- Multi-directional typesetting
- Objective: Cleanup the internals, remove bugs in the Aleph code and review the backend part. The model stays mostly as in Omega. Status: This is an ongoing effort and the first major cleanup took place around versions 0.40 and 0.50. The code has already been debugged and simplified. We now only have four directions. Some aspects of the page, body, text and par model might change and information currently stored in whatsits might move to normal nodes. This is projected for versions 0.80 or later.
- Special features
- Objective: Cleaning up of HZ optimization and protruding, optimize resources in the PDF output. Get rid of remaining global font properties. Cleanup and optimize the backend code. Status: Much of this was done in 0.50 (more efficient pdf output). In 0.78 a major change took place when the front- and backend aspects of this mechanism were separated and overhauled.
- Objective: Extending math to Unicode and providing more control over rendering.
Support OpenType math cf the Microsoft specification. Provide primitive support for
some features that are currently dealt with by macros. Also support math table building as in
Status: This effort started at the end of 2008 and is mostly finished mid 2009 in version 0.40 and became quite stable in 0.50. We have a few extensions on the agenda but they are subprojects. These developments happen in parallel to the development of more math fonts in the TeXGyre project.
- Page building
- Objective: Control over page building and access to internals that matter. Access
Status: This is on the agenda for 2014 at the earliest (maybe version 0.90).
- TeX primitives
- Objective: Access to and control over most TeX primitives (and related mechanisms)
as well as registers will be provided. There will be a clear split between front and backend.
Status: This is an ongoing effort and much already has been done.
- PDF Backend
- Objective: Most backend related features will be opened up. Think of annotations
and object management.
Status: The first code will show up later in 2009. In version 0.50 already quite some functionality is accessible at the Lua end. In 0.79 a lot of code has been normalized and cleaned up. There is still a todo list, so expect more.
- Future stage
- Objective: Convert the TeX Pascal source into C (CWeb) and start using Lua as glue
language for components.
Status: This has been tested on MPlib. Parts of LuaTeX are already rewritten in C, for instance some memory management, IO, language support, par building, hyphenation, font loading and math. Version 0.60 is the first CWeb version because it is mandate for some of the things we have in mind. Improving the result is an ongoing effort.
In addition to the mentioned functionality we have a couple of ideas that we will evolve and be implement along the road. We had some releases and there will be more:
- Version 0.00
- We first demonstrated LuaTeX in TUG 2005 in Wuhan (China). At that time we had only the scripting engine available and access to some registers. We could print back to TeX.
- Version 0.10
- The first major (beta) release was at TUG 2007 in San Diego (USA). This version could handle fonts and we demonstrated the first attempts of dealing with Arabic typesetting as part of the Oriental TeX project.
- Version 0.20
- The second major release (still beta) was around TUG 2008 in Cork (Ireland).
- Version 0.30
- This version showed up as prelude to version 0.40 and provided preliminary support for OpenType math. This version was the basis for successive (incompatible) beta versions that got more and more implemented of our relatively long wishlist.
- Version 0.40
- This is the BachoTeX 2009 release that will also be shipped with TeXlive 2009. At BachoTeX we will officially release the new math enhancements as this is the place where discussions started with regards to the new OpenType math companion fonts.
- Version 0.50
- Also known as the halfway version, version 0.50 will show up around EuroTeX 2009 that takes place in Delft in the Netherlands. This version will be a polished version 0.40 with some pending cleanup being done. It is the prelude to 0.60, the CWeb version. For practical reasons the official release date has been moved forward to after TeXlive is frozen and distributed (TeXlive ships with version 0.4*). Version 0.50 has been released just before xmas 2009.
- Version 0.60
- This version does not offer fundamental new features compared to 0.50. Released begin April this iteration will go on TeXlive 2010 and for that stability is a condition. There are a few more callbacks and functions, some cleanups in the backend and the font inclusion has been improved. We are confident that this version is quite useable. At this point the source code has been converted back into web so that more detailed documentation can be added.
- Version 0.65
- This was an intermediate release with lots of small fixes and a few new features. The release was late 2010 (around the ConTeXt conference in Czech).
- Version 0.70
- This version came available around the BackoTeX (also EuroTeX) 2011 conference in Poland and will end up on TeXlive 2011. This is a stable version, suitable for production. No fundamental new features have been added. Macro (package) writers should be prepared to adapt (node and backend related Lua code) from version 0.71 onward.
- Version 0.75
- Around this version, which came available end 2012 we got Lua 5.2 on board. Experiments have shown us that this version runs somewhat faster than previous versions and also uses a bit less memory. Of course there are the usual bug fixed and some tracker items have been resolved. There is an updated MetaPost library that supports an additional number mode using doubles.At the same time the first beta of LuajitTeX was released (version 0.72, Lua 5.1). This version uses LuaJIT. This version is not portable to all platforms. Some benchmarks with ConTeXt have demonstrated that runs can be 20-35% faster than a regular 0.72 version. Most of the speedup is due to the faster virtual machine as well as the garbage collector (which is similar to the one in Lua 5.2). You can use both version alongside but LuaTeX with Lua 5.2 is the official standard. This is a stable release.
- Version 0.77
- This is the version that is on TeXlive 2013. There are no fundamental changes and additions, but bugs have been patched. Because the engine is now more widespread we cannot make big changes unless macro package writers have time to adapt. Of course, we can still change aspects of the interfaces.
- Version 0.78
- This is the end-of-year version in 2013. Apart from some bug fixes, the most important change is the (first) cleanup of the hz optimization (expansion), which is a prelude to a better separation between front- and backend. There are also some speedups in node processing (that have been tested for more than half a year now). The remains of AAT font technology have been removed and the font loader is somewhat more robust. As usual, the latest MetaPost library is included.
- Version 0.79
- This is mostly be a bug-fix release. An experimental new token scanner interface has been demonstrated in 2014 and added to 0.79.3 onwards beginning of 2015.
- Version 0.80
- This version is the offical TeXlive 2015 version. Apart from some bug fixed the major new thing is a (new)token scanner that will replace the old one, so we have a transition stage. Lua 5.2.4 is used now.
- Version 0.81-0.84
- These versions will drop the old token scanner and as side effects also have a cleaned up input handler. We're still not sure if and when we will move to 5.3: experiments have demonstrated that the new mixed integer/float model works okay but is somewhat slower. This update might show up around the 2015 ConTeXt conference.
- Version 0.85
- This version might end up on TeXlive 2016. We hope to have a cleaner separation of the front and backend too. Some currently whatsit nodes might have been migrated to core nodes.
- Version 0.90
- This version might end up on TeXlive 2017.
- Version 0.95
- This version might end up on TeXlive 2018.
- Version 1.00
- This version might end up on TeXlive 2020.
We report progress on a regular basis at usergroup meetings and in usergroup journals so if you want to be kept informed, you might consider becoming a member of a TeX usergroup or visiting one of the TeX conferences.
LuaTeX will stay beta till we are more or less functionally complete. It will take a couple of years before we reach that state. Our original estimate was that we could provide version 1.0 in 2012 around TeX's 32th birthday, but we ended up with a lower number. Although major releases (x.x0 versions) are stable enough to be used for production work you need to keep in mind that only at version 1.00 most of the functionality and interfaces will be stable. Don't automatically expect your old code to work with newer versions of LuaTeX: best update on a regular basis. Starting with version 0.50 the manual will tell what features are stable.
It is important to know that starting with version 0.40 LuaTeX will start up with only TeX82 functionality. The \directlua primitive will be available for testing the available functionality and further initialization of features.
For testing we use an experimental version of ConTeXt tagged MKiV. Support is provided on the wiki and mailing list. ConTeXt users are advised to keep their LuaTeX binary in sync with the latest ConTeXt MKiV. The best way to do this is to use the minimals from the contextgarden as binaries are automatically updated there. Users of other macro packages can best consult the package related sites and lists.
- The landscape
- There are currently three engines in use in the TeX community: pdfTeX, XeTeX and LuaTeX and they all serve an audience.
- The pdfTeX engine is a stable extension to TeX that has eTeX on board as well as a few extensions. It provides hz and protruding and supports advanced PDF output. It is fast and reliable and sort of frozen with respect to functionality. It will stay around for ever but has only limited provisions for dealing with OpenType fonts.
- Then there is XeTeX which supports Unicode as well as OpenType by means of third party libraries. It integrates nicely into the current infrastructure and support from macro packages is easy as there are no fundamental changes in interfaces and functionality. It uses the eTeX r2l typesetting while LuaTeX uses Omega's (Alephs). Support from macro packages does not demand changes in the core.
- The LuaTeX project uses a different approach. Instead of using libraries it provides an extension mechanism. This keeps the program independent and permits the flexibility that we like so much. However, it comes at a price. First of all LuaTeX does not have the performance of its cousins, although for instance the average ConTeXt MkIV run can be faster in LuaTeX. LuaTeX also demands extensive support at the macro level if you want to profit from its benefits. Just adding some support for scripting is nice but the power of LuaTeX only shows when it's tightly integrated. This integration is also needed in order to provide macro users some stability as most of them are no TeX programmers. If such integration is not what you want, you might consider sticking to the other engines as there are no real advantages to using LuaTeX then.
- As it possible to replace the current paragraph builder by a Lua function and this is the direction that we move to: LuaTeX as framework with only built in TeX82 support, some extended core functions (hyphenation and math) and provisions to extend the more debatable bit and pieces with your own code. One of the main objectives of the oriental TeX project is to add advanced r2l typesetting to ConTeXt MkIV and part of that extensions to the par builder written in Lua. There will be no extensions to the built in typesetting engine as we are confident that the Lua interface is flexible enough. An exception is math, where opening up as well as support for OpenType math demands some more control.
- We ship with a minimal number of built in libraries in addition to the standard libraries: a fontloading library cf. FontForge, a few basic IO related libraries, some Unicode helper code. These have been adapted to our needs and are statically linked into LuaTeX. We try to keep the default featureset small and this is already a bit larger than we had in mind.
- The 0.60 release provides can load additional libraries but these are not part of the projects as there are multiple choices as well as interfaces to these libraries possible. Demand and support is probably quite related to macro package demands and LuaTeX is (to be) independent of macro packages. There are some known limitation with respect to library versions related to the embedded Lua variant. These will be sorted out later.
- We expect version 0.80 to use stock Lua 5.2 which will make it possible to provide stable library loading in 0.90. By then, some currently preloaded libraries can become delayed as well.
- We have chosen for a Lua based support for fonts. There are two solutions for handling fonts: using the internal functions that do what TeX has always done, or write a Lua function that does a different job. As there are multiple solutions possible and as we expect macro packages to have their own ways of dealing with fonts, there is not one solution for dealing with fonts anyway. Also, TeXies have always wanted full control over matters, and this is provided by the Lua solution.
- There will be no hard coded solutions and experiments with helper code as show us that this is of no use as switching between Lua and TeX adds enough overhead to nil the advantage of such helper code. It's more efficient to stay either at the TeX end or at the Lua end. Of course we will continue to experiment with alternative ways but this is unlikely to happen before version 1.0 (after that we will formulate our LuaTeX 2.0 objectives and set a new agenda).
- Plain TeX
- As part of the project we provide some basic (non-intrusive) functionality like for instance OpenType support for plain TeX as we use this for testing some code. This basic code is derived from ConTeXT MkIV code which serves as our benchmark (the larger testbed is the Oriental TeX project). The code is part of the ConTeXt distribution but can be used independently. The first code showed up alongside version 0.40 (in TeXlive 2009) and is kept in sync with the ConTeXt code. Based on that support for LaTeX is provided.
- You might have noticed that we really take our time to complete this project. This is partly because it is non-trivial to rewrite TeX in such a way that existing macro code still works. As LuaTeX development goes in parallel with rewriting ConTeXt (MkIV) we test everything new and opened up extensively. We spend much time discussing and trying out solutions. We do and redo when needed and try to stay generic as much as possible. We also document what we've done and report on it as well as maintain a manual. This simply takes time and we're talking of many (mostly unpaid) man-years here. We feel that this is the only way to arrive at what we aim at with LuaTeX, which is important as we ourselves are among its users. In the perspective of 30 years of TeX the timeline is quite feasible. Of course, as intermediate versions are quite useable, you don't have to wait till 2012 to start using LuaTeX.
- Traditional TeX has its own output format (DVI) which is then postprocessed into for instance specific printer code, the more generic PostScript or PDF. With pdfTeX one can directly produce PDF and as LuaTeX is derived from pdfTeX is also produced PDF. Because one has access to the internals additional backeds are possible (using Lua).
This page will be updated, extended and cleaned up occasionally. Not all objectives and developments are described here. As part of ConTeXt MkIV development, the most important LuaTeX developments are reported at user group meetings and reported in articles.