Here is a very rough outline of our schedule. This list is just an indication and will be updated on a regular basis. We have a few more intermediate stages in mind.
- Merging engines
- Objective: Merge the Aleph and pdfTeX codebase so that LuaTeX in DVI mode
behaves like Aleph, and in PDF mode like pdf(e)TeX. Also allow Lua callbacks
for file searching.
Status: This stage is mostly finished. LuaTeX 0.40 and higher start up as TeX82 and one has to explicitly enable extra primitives (normally this will be done by the macropackage).
- OpenType fonts
- Objective: Make the Aleph functionality available in PDF mode and add
support for OpenType fonts. Allow Lua scripts to control all aspects of font
loading, font definition and manipulation.
Status: Most of this is finished. We use code from the FontForge project which has the advantage that one can use that open sourced tool alongside LuaTeX and get a similar view at the font.
- Tokenizing and node lists
- Objective: Use Lua callbacks for various internals, complete access to
tokenizer and provide access to node lists at moments that make sense.
Status: This stage is completed but you can expect a few more goodies as we open up more of TeX after 0.50.
- Paragraph building
- Objective: Provide control over various aspects of paragraph building
(hyphenation, kerning, ligature building), dynamic loading loading of hyphenation
Status: Apart from some small details these objectives are met. Users can write their own paragraph builders (although this is a somewhat specialized task). At some point we will provide more context for each paragraph. We expect to have reorganized this in version 0.70 or 0.80.
- Metapost (mplib)
- Objective: Turn metapost into a library. Extend the library to arbitrary
precision (megapost) and add features on the wishlist. Advanced usage in LuaTeX,
like for instance runtime font generation and manipulation is not yet scheduled.
Status: The first stage of this project is finished with the first public beta and integration in LuaTeX. This subproject is on schedule and the megapost stage will be finished by the end of 2009. Future versions of LuaTeX will ship with the upgraded mplib 2 library. The first outcome of this project will be in 2010.
- Image handling
- Objective: Image identification and loading in Lua including scaling and object management. Status: Nicely on schedule, the first version of the img library showed up in the 0.22 beta and some more features are planned. Inclusion of PDF has been optimized in 0.50 and further optimizations will be available in 0.60.
- Multi-directional typesetting
- Objective: Cleanup the internals, remove bugs in the Aleph code and review the backend part. The model stays mostly as in Omega. Status: This is an ongoing effort and the first major cleanup takes place around versions 0.40 and 0.50. After 0.50 we will revision the directional model. The code has already be debugged and simplified. We now only have 4 directions. Some aspects of the page, body, text and par model might change and information currently stored in whatsits might move to nodes. This is projected for versions 0.70 and 0.80.
- Special features
- Objective: Cleaning up of HZ optimization and protruding, optimize resources in the PDF output. Get rid of remaining global font properties. Cleanup and optimize the backend code. Much of this is done in 0.50. Status: This is an ongoing effort.
- Objective: Extending math to Unicode and providing more control over rendering.
Support OpenType math cf the Microsoft specification. Provide primitive support for
some features that are currently dealt with by macros. Also support math table building as in
Status: This effort started at the end of 2008 and is mostly finished mid 2009 (version 0.40) and is quite stable in 0.50. We have a few extensions on the agenda but they are subprojects.
- Page building
- Objective: Control over page building and access to internals that matter. Access
Status: This is on the agenda for 2010 at the earliest (maybe version 0.70).
- TeX primitives
- Objective: Access to and control over most TeX primitives (and related mechanisms)
as well as registers will be provided. There will be a clear split between front and backend.
Status: This is an ongoing effort.
- PDF Backend
- Objective: Most backend related features will be opened up. Think of annotations
and object management.
Status: The first code will show up later in 2009. In version 0.50 already quite some functionality is accessible at the Lua end.
- Future stage
- Objective: Convert the TeX Pascal source into C (CWeb) and start using Lua as glue
language for components.
Status: This has been tested on MPlib. Parts of LuaTeX are already rewritten in C, for instance some memory management, IO, language support, par building, hyphenation, font loading and math. Version 0.60 is the first CWeb version because it is mandate for some of the things we have in mind.
In addition to the mentioned functionality we have a couple of ideas that we will evolve and be implement along the road. We had some releases and there will be more:
- Version 0.00
- We first demonstrated LuaTeX in TUG 2005 in Wuhan (China). At that time we had only the scripting engine available and access to some registers. We could print back to TeX.
- Version 0.10
- The first major (beta) release was at TUG 2007 in San Diego (USA). This version could handle fonts and we demonstrated the first attempts of dealing with Arabic typesetting as part of the Oriental TeX project.
- Version 0.20
- The second major release (still beta) was around TUG 2008 in Cork (Ireland).
- Version 0.30
- This version showed up as prelude to version 0.40 and provided preliminary support for OpenType math. This version was the basis for successive (incompatible) beta versions that got more and more implemented of our relatively long wishlist.
- Version 0.40
- This is the BachoTeX 2009 release that will also be shipped with TeXlive 2009. At BachoTeX we will officially release the new math enhancements as this is the place where discussions started with regards to the new OpenType math companion fonts.
- Version 0.50
- Also known as the halfway version, version 0.50 will show up around EuroTeX 2009 that takes place in Delft in the Netherlands. This version will be a polished version 0.40 with some pending cleanup being done. It is the prelude to 0.60, the CWeb version. For practical reasons the official release date has been moved forward to after TeXlive is frozen and distributed (TeXlive ships with version 0.4*). Version 0.50 has been released just before xmas 2009.
- Version 0.60
- This version does not offer fundamental new features compared to 0.50. Released begin April this iteration will go on TeXlive 2010 and for that stability is a condition. There are a few more callbacks and functions, some cleanups in the backend and the font inclusion has been improved. We are confident that this version is quite useable. At this point the source code has been converted back into web so that more detailed documentation can be added.
- Version 0.65
- This was an intermediate release with lots of small fixes and a few new features. The release was late 2010 (around the ConTeXt conference in Czech).
- Version 0.70
- This version came available around the BackoTeX (also EuroTeX) 2011 conference in Poland and will end up on TeXlive 2011. This is a stable version, suitable for production. No fundamental new features have been added. Macro (package) writers should be prepared to adapt (node and backend related Lua code) from version 0.71 onward.
- Version 0.75
- Around this version, which is available end 2012 we will have Lua 5.2 on board. Experiments have shown us that this version runs somewhat faster than previous versions and also uses a bit less memory. Of course there are the usual bug fixed and some tracker items have been resolved. There is an updated MetaPost library that supports an additional number mode using doubles.At the same time the first beta of LuajitTeX is released (version 0.72, Lua 5.1). This version uses LuaJIT. This version is not portable to all platforms. Some benchmarks with ConTeXt have demonstrated that runs can be 20-35% faster than a regular 0.72 version. Most of the speedup is due to the faster virtual machine as well as the garbage collector (which is similar to the one in Lua 5.2). You can use both version alongside but LuaTeX with Lua 5.2 is the official standard. This is a stable release.
- Version 0.80
- (to be decided)
- Version 0.90
- (to be decided)
- Version 1.00
- (to be decided)
We report progress on a regular basis at usergroup meetings and in usergroup journals so if you want to be kept informed, you might consider becoming a member of a TeX usergroup or visiting one of the TeX conferences.
LuaTeX will stay beta till we are more or less functionally complete. It will take a couple of years before we reach that state. Our current estimate is that we will provide version 1.0 in 2012 around TeX's 32th birthday. Although major releases (x.x0 versions) are stable enough to be used for production work you need to keep in mind that only at version 1.00 most of the functionality and interfaces will be stable. Don't automatically expect your old code to work with newer versions of LuaTeX: best update on a regular basis. Starting with version 0.50 the manual will tell what features are stable.
It is important to know that starting with version 0.40 LuaTeX will start up with only TeX82 functionality. The \directlua primitive will be available for testing the available functionality and further initialization of features.
For testing we use an experimental version of ConTeXt tagged MKiV. Support is provided on the wiki and mailing list. ConTeXt users are advised to keep their LuaTeX binary in sync with the latest ConTeXt MKiV. The best way to do this is to use the minimals from the contextgarden as binaries are automatically updated there. Users of other macro packages can best consult the package related sites and lists.
- The landscape
- There are currently three engines in use in the TeX cummunity: pdfTeX, XeTeX and LuaTeX and they all serve an audience.
- The pdfTeX engine is a stable extension to TeX that has eTeX on board as well as a few extensions. It provides hz and protruding and supports advanced PDF output. It is fast and reliable and sort of frozen with respect to functionality. It will stay around for ever but has only limited provisions for dealing with OpenType fonts.
- Then there is XeTeX which supports Unicode as well as OpenType by means of third party libraries. It integrates nicely into the current infrastructure and support from macro packages is easy as there are no fundamental changes in interfaces and functionality. It uses the eTeX r2l typesetting while LuaTeX uses Omega's (Alephs). Support from macro packages does not demand changes in the core.
- The LuaTeX project uses a different approach. Instead of using libraries it provides an extension mechanism. This keeps the program independent and permits the flexibility that we like so much. However, it comes at a price. First of all LuaTeX does not have the performance of its cousins. It also demands extensive support at the macro level if you want to profit from its benefits. Just adding some support for scripting is nice but the power of LuaTeX only shows when it's tightly integrated. This integration is also needed in order to provide macro users some stability as most of them are no TeX programmers. If such integration is not what you want, you might consider sticking to the other engines as there are no real advantages to using LuaTeX then.
- As it possible to replace the current paragraph builder by a Lua function and this is the direction that we move to: LuaTeX as framework with only built in TeX82 support, some extended core functions (hyphenation and math) and provisions to extend the more debatable bit and pieces with your own code. One of the main objectives of the oriental TeX project is to add advanced r2l typesetting to ConTeXt MkIV and part of that extensions to the par builder written in Lua. There will be no extensions to the built in typesetting engine as we are confident that the Lua interface is flexible enough. An exception is math, where opening up as well as support for OpenType math demands some more control.
- We ship with a minimal number of built in libraries in addition to the standard libraries: a fontloading library cf. FontForge, a few basic IO related libraries, some Unicode helper code. These have been adapted to our needs and are statically linked into LuaTeX. We try to keep the default featureset small and this is already a bit larger than we had in mind.
- The 0.60 release provides can load additional libraries but these are not part of the projects as there are multiple choices as well as interfaces to these libraries possible. Demand and support is probably quite related to macro package demands and LuaTeX is (to be) independent of macro packages. There are some known limitation with respect to library versions related to the embedded Lua variant. These will be sorted out later.
- We expect version 0.80 to use stock Lua 5.2 which will make it possible to provide stable library loading in 0.90. By then, some currently preloaded libraries can become delayed as well.
- We have chosen for a Lua based support for fonts. There are two solutions for handling fonts: using the internal functions that do what TeX has always done, or write a Lua function that does a different job. As there are multiple solutions possible and as we expect macro packages to have their own ways of dealing with fonts, there is not one solution for dealing with fonts anyway. Also, TeXies have always wanted full control over matters, and this is provided by the Lua solution.
- There will be no hard coded solutions and experiments with helper code as show us that this is of no use as switching between Lua and TeX adds enough overhead to nil the advantage of such helper code. It's more efficient to stay either at the TeX end or at the Lua end. Of course we will continue to experiment with alternative ways but this is unlikely to happen before version 1.0 (after that we will formulate our LuaTeX 2.0 objectives and set a new agenda).
- Plain TeX
- As part of the project we will provide some basic (non-intrusive) functionality like for instance OpenType support for plain TeX as we use this for testing some code. This basic code is derived from ConTeXT MkIV code which serves as our benchmark (the larger testframe is the Oriental TeX project). The code is part of the ConTeXt distribution but can be used independently. The first code showed up alongside version 0.40 (in TeXlive 2009).
- You might have noticed that we really take our time to complete this project. This is partly because it is non-trivial to rewrite TeX in such a way that existing macro code still works. As LuaTeX development goes in parallel with rewriting ConTeXt (MkIV) we test everything new and opened up extensively. We spend much time discussing and trying out solutions. We do and redo when needed and try to stay generic as much as possible. We also document what we've done and report on it as well as maintain a manual. This simply takes time and we're talking of many (mostly unpaid) man-years here. We feel that this is the only way to arrive at what we aim at with LuaTeX, which is important as we ourselves are among its users. In the perspecitive of 30 years of TeX the timeline is quite feasible. Of course, as intermediate versions are quite useable, you don't have to wait till 2012 to start using LuaTeX.
- Traditional TeX has its own output format (DVI) which is then postprocessed into for instance specific printer code, the more generic PostScript or PDF. With pdfTeX one can directly produce PDF and as LuaTeX is derived from pdfTeX is also produced PDF. In the LuaTeX project we will rewrite the backend code in such a way that eventually we can provide more backends. This is a mayor undertaking that will happen stepwise. Of course this has to happen in such a way that we don't loose functionality as pdfTeX is pretty good in producing advanced PDF. In version 0.80 of 0.90 we expect the new backend (epdf) library to be stable. Till then many aspects can change. This will not influence the usability.
This page will be updated, extended and cleaned up occasionally.