Jinja 2.9 Released

written by Armin Ronacher on 2017-01-07 in Releases

After a long time of no changes a new release, 2.9 codename “Derivation” of the Jinja template engine for Python is out. This change is probably the biggest release in Jinja2's history and it requires a bit of explanation of why it happened and why it happened now. But first let's cover the big changes in it.

New Identifier Tracking

This appears to be an under the hood change because it effectively just changes the internal algorithm that Jinja2 uses to map it's own scoping rules onto the Python interpreter's scoping rules but it has wide ranging consequences. The original tracking issue goes back to 2011 and was created as a long running TODO item. There were a few reasons why it was implemented this way but over the years it became clearer and clearer that the method chosen for mapping identifiers is just causing too many issues. The reason it was not changed was that the worries were too big that people accidentally relied on such bugs.

However with improved support for pinning to older versions of Python packages and the fact that people no longer use linux distribution provided packages much the worries about greater changes are heavily reduced. With Jinja 2.9 the identifier tracking was completely changed which should squash pretty much all the weird behaviors developers might have encountered and also enabled support for related improvements. In particular the new identifier tracking generalized the context behavior for includes and imports.

In the ideal case you won't notice anything other than that some constructs are possible now that caused errors in the past.

Python 3 Feature Support

While Jinja2 supported Python 3 for years now it never really embraced functionality that the language provides on 3.x that it does not do on 2.x. Largely that is caused by the fact that we want templates to be compatible between 2.x and 3.x (this is true only for as long as template variables are restricted to ASCII characters). However 3.6 now added async generators which permits Jinja2 to fully support the async and await keywords on 3.6 and later.

In particular it means that you can now return coroutines from functions passed to Jinja2 templates and the template engine will automatically await them. Likewise all filters were updated to work with iterators as well as async iterators alike.

Additionally Jinja2 now emits generator_stop as a future flag which means that accidentally emitting StopIteration from a filter or other code will no longer cause the template to just stop rendering silently. This was enabled by PEP 479.

Policy Framework

Jinja2 now has an internal policy configuration which permits one to centrally reconfigure behavior of filters and other things. While so far not much can be reconfigured it will greatly support future improvements. This also finally allows us to provide a default tojson filter which previously was only available through Flask or other frameworks. Through the policy configuration the particular form of JSON serialization can be customized and replaced.

Full Changelog

  • Change cache key definition in environment. This fixes a performance regression introduced in 2.8.
  • Added support for generator_stop on supported Python versions (Python 3.5 and later)
  • Corrected a long standing issue with operator precedence of math operations not being what was expected.
  • Added support for Python 3.6 async iterators through a new async mode.
  • Added policies for filter defaults and similar things.
  • urlize now sets "rel noopener" by default.
  • Support attribute fallback for old-style classes in 2.x.
  • Support toplevel set statements in extend situations.
  • Restored behavior of Cycler for Python 3 users.
  • Subtraction now follows the same behavior as other operators on undefined values.
  • map and friends will now give better error messages if you forgot to quote the parameter.
  • Depend on MarkupSafe 0.23 or higher.
  • Improved the truncate filter to support better truncation in case the string is barely truncated at all.
  • Change the logic for macro autoescaping to be based on the runtime autoescaping information at call time instead of macro define time.
  • Ported a modified version of the tojson filter from Flask to Jinja2 and hooked it up with the new policy framework.
  • Block sets are now marked safe by default.
  • On Python 2 the asciification of ASCII strings can now be disabled with the compiler.ascii_str policy.
  • Tests now no longer accept an arbitrary expression as first argument but a restricted one. This means that you can now properly use multiple tests in one expression without extra parentheses. In particular you can now write foo is divisibleby 2 or foo is divisibleby 3 as you would expect.
  • Greatly changed the scoping system to be more consistent with what template designers and developers expect. There is now no more magic difference between the different include and import constructs. Context is now always propagated the same way. The only remaining differences is the defaults for with context and without context.
  • The with and autoescape tags are now built-in.
  • Added the new select_autoescape function which helps configuring better autoescaping easier.

Why an Update now?

One of the reasons Jinja2 was lying largely dormant is that it's always scary to touch something so many people use. While bugs were known for quite some time it requires careful changes to not change all the template code out there. In particular once salt and ansible started using Jinja it became a big emotional burden to do changes on it.

However it turns out that emotional burden does not get smaller for as long as those bugs are in there. Going in and cleaning up the behavior actually turns out to be healthier in the long run. Now even if this release introduces regressions in behavior, the internal code quality is much improved and reasoning about the runtime and compiler is a lot easier now.

On a personal note I'm happy to see how popular Jinja has become and I hope that with this release it becomes more enjoyable to write Jinja templates.

Aside: if you are downloading Jinja2 you will actually pull 2.9.1 because there was a regression in 2.9 that was only found after releasing it. In an ironic twist it was found when this website attempted to render a complex template with Jinja 2.9 on the build server.

Jinja 2.8.1 Security Release

written by Armin Ronacher on 2016-12-29 in Releases , Security

We just pushed out a new release for Jinja (2.8.1) which includes a security related fix. If you are using the Jinja2 sandbox you are encouraged to upgrade or alternatively manually further lock down the sandbox.

The core of the issue is that Python's string format method that was added to strings can be used to discover potentially dangerous values including configuration values:

>>> config = {'SECRET_KEY': '12345'}
>>> class User(object):
...  def __init__(self, name):
...   self.name = name
...
>>> user = User('joe')
>>> '{0.__class__.__init__.__globals__[config]}'.format(user)
"{'SECRET_KEY': '12345'}"

For this reason you must never let user supply format strings in raw Python as its too easy to escape them. However specifically for the Jinja2 sandbox we changed the behavior now that we're using the same sandboxing functionality that Jinja2 uses for its own runtime also for Python string formatting.

This means that with 2.8.1 and higher templates from sandboxed environments will intercept format strings the same way as with normal cases:

>>> from jinja2.sandbox import SandboxedEnvironment
>>> env = SandboxedEnvironment()
>>> class User(object):
...  def __init__(self, name):
...   self.name = name
...
>>> t = env.from_string(
...  '{{ "{0.__class__.__init__.__globals__}".format(user) }}')
>>> t.render(user=User('joe'))
Traceback (most recent call last):
  ...
jinja2.exceptions.SecurityError: ...

If you don't want or you cannot upgrade Jinja2, you can override the is_safe_attribute method on the sandbox and explicitly disallow all format attributes on strings.

Thank you to Olivier Dony for reporting the issue.

Flask 0.12 released

written by Markus Unterwaditzer on 2016-12-21 in Releases

Flask 0.12 has been released. This is not as big a step as 0.11 has been, the only intentionally backwards-incompatible change is with regards to send_file's behavior, which, when invoked as send_file(f) with f being a file-like object, no longer guesses the MIME-type from f.name. You have to pass a filepath instead.

Special thanks to Kyle Lawlor for fixing up most documentation after the flask CLI got introduced.

Changelog

  • the cli command now responds to --version.
  • Mimetype guessing and ETag generation for file-like objects in send_file has been removed, as per issue #104. See pull request #1849.
  • Mimetype guessing in send_file now fails loudly and doesn't fall back to application/octet-stream. See pull request #1988.
  • Make flask.safe_join able to join multiple paths like os.path.join (pull request #1730).
  • Revert a behavior change that made the dev server crash instead of returning a Internal Server Error (pull request #2006).
  • Correctly invoke response handlers for both regular request dispatching as well as error handlers.
  • Disable logger propagation by default for the app logger.
  • Add support for range requests in send_file.
  • app.test_client includes preset default environment, which can now be directly set, instead of per client.get.

Flask 0.11 Released

written by Armin Ronacher on 2016-05-29 in Releases

After a very long, long waiting time Flask finally got a new release. There really was no good reason that there has not been a release in such a long time but unfortunately once things are postponed for too long a certain release anxiety kicks in.

In this case this was long tagged as 1.0 but we decided for renaming it to 0.11 and back out some of the more controversial changes. In particular the new command line interface for Flask was modified a bit to not depend on some specific functionality in the supporting Click library.

Highlights of this release are the improved development experience which now stalls the browser on reload instead of bringing up a "connection reset" page and the new command line support.

Future Plans

This is also the first Flask release under the new pallets organization and from now on we hope to bring you releases more frequently. Ideally with the next release we also update the website to find a new home for the showcase, flask extension list as well as the snippet section to allow the community to take care of those things themselves.

Changelog

  • Added support to serializing top-level arrays to flask.jsonify. This introduces a security risk in ancient browsers. See json-security for details.
  • Added before_render_template signal.
  • Added **kwargs to flask.Test.test_client to support passing additional keyword arguments to the constructor of flask.Flask.test_client_class.
  • Added SESSION_REFRESH_EACH_REQUEST config key that controls the set-cookie behavior. If set to True a permanent session will be refreshed each request and get their lifetime extended, if set to False it will only be modified if the session actually modifies. Non permanent sessions are not affected by this and will always expire if the browser window closes.
  • Made Flask support custom JSON mimetypes for incoming data.
  • Added support for returning tuples in the form (response, headers) from a view function.
  • Added flask.Config.from_json.
  • Added flask.Flask.config_class.
  • Added flask.config.Config.get_namespace.
  • Templates are no longer automatically reloaded outside of debug mode. This can be configured with the new TEMPLATES_AUTO_RELOAD config key.
  • Added a workaround for a limitation in Python 3.3's namespace loader.
  • Added support for explicit root paths when using Python 3.3's namespace packages.
  • Added the flask command and the flask.cli module to start the local debug server through the click CLI system. This is recommended over the old flask.run() method as it works faster and more reliable due to a different design and also replaces Flask-Script.
  • Error handlers that match specific classes are now checked first, thereby allowing catching exceptions that are subclasses of HTTP exceptions (in werkzeug.exceptions). This makes it possible for an extension author to create exceptions that will by default result in the HTTP error of their choosing, but may be caught with a custom error handler if desired.
  • Added flask.Config.from_mapping.
  • Flask will now log by default even if debug is disabled. The log format is now hardcoded but the default log handling can be disabled through the LOGGER_HANDLER_POLICY configuration key.
  • Removed deprecated module functionality.
  • Added the EXPLAIN_TEMPLATE_LOADING config flag which when enabled will instruct Flask to explain how it locates templates. This should help users debug when the wrong templates are loaded.
  • Enforce blueprint handling in the order they were registered for template loading.
  • Ported test suite to py.test.
  • Deprecated request.json in favour of request.get_json().
  • Add "pretty" and "compressed" separators definitions in jsonify() method. Reduces JSON response size when JSONIFY_PRETTYPRINT_REGULAR=False by removing unnecessary white space included by default after separators.
  • JSON responses are now terminated with a newline character, because it is a convention that UNIX text files end with a newline and some clients don't deal well when this newline is missing. See https://github.com/pallets/flask/pull/1262 -- this came up originally as a part of https://github.com/kennethreitz/httpbin/issues/168
  • The automatically provided OPTIONS method is now correctly disabled if the user registered an overriding rule with the lowercase-version options (issue #1288).
  • flask.json.jsonify now supports the datetime.date type (pull request #1326).
  • Don't leak exception info of already caught exceptions to context teardown handlers (pull request #1393).
  • Allow custom Jinja environment subclasses (pull request #1422).
  • flask.g now has pop() and setdefault methods.
  • Turn on autoescape for flask.templating.render_template_string by default (pull request #1515).
  • flask.ext is now deprecated (pull request #1484).
  • send_from_directory now raises BadRequest if the filename is invalid on the server OS (pull request #1763).
  • Added the JSONIFY_MIMETYPE configuration variable (pull request #1728).
  • Exceptions during teardown handling will no longer leave bad application contexts lingering around.

Werkzeug 0.11.6 Security Release

written by Armin Ronacher on 2016-04-14 in Releases , Security

Today we pushed out a Werkzeug bugfix release which contains a security relevant fix. It has come to our attention (reported by Jordan Milne) that the PIN brute-force protection in the Werkzeug debugger could be bypassed by attacking the cookie rather than the PIN. While this is generally not easily fixable we improved the situation by mixing in higher quality secret data into the cookie name and made it more complex. We now include a UUID of the machine the code is running on.

This should make it significantly more complex to bypass the PIN check. That said we want to reiterate that the PIN protection for the debugger is not a suitable protection to run the debugger in production. It's a basic security feature to make it less likely to use an accidentally enabled debugger. Please ensure that you never enable the debugger in production environments regardless of this feature.

Hello Pallets Users

written by Armin Ronacher on 2016-04-01 in Meta

On the first of April 2010, I released a joke microframework called denied which made fun of the fact that all microframeworks at the time decided to forgo with dependencies and bundle up everything they need in a single Python file. What I did was embed all of Jinja2 and Werkzeug in a base64 encoded zip file within the framework's only Python file. The response to it was interesting in a few ways because on the one hand quite a few people did not really understand that it was an April fools joke to begin with and on the other, there was a discussion where there were no microframeworks that actually did use dependencies and encouraged it.

One month later there was a new project by the name of "Flask" which actually gave this concept a real shot. It launched with the tagline "a microframework for Python based on Werkzeug, Jinja 2 and good intentions" and six years later it's the most starred Python framework on GitHub.

What's interesting about Flask is that its success just happened. There are no conferences about it, no society or foundation and still today much of it heavily depends on me directly. I'm not even sure why it became this successful but I attribute a lot of it to the fact that it's easy to get started and the full footprint of the framework is small enough that it becomes easy enough to understand.

So what's changing now? Today we launch the Pallets Projects. What is it? Primarily it's a GitHub organization which will be the home of Flask and all the associated projects. It will be a new home for those libraries and the first step to give the community more impact on the development of Flask and all libraries. In addition there will be a new release of Flask very soon after a thorough check that we do not break anything.

The people behind the Pallets Projects are me, Markus Unterwaditzer, David Lord and Adrian Mönnich with the organization being open for newcomers to help and drive the projects forward.

We will spend the next few weeks adding as much organizational information on the project's website to ensure that what often currently only exists in my head is brought down to text.

I'm amazed how many people use and love Flask and my libraries and I hope that this organization will be a good step towards making this scale past me. It's humbling how big all of this became.