Modern Javascript in Django Templates
Originally posted at Medium.com
Since the beginning, Oda has benefited greatly from building its webshop at high pace using Django templates. However, in order to tap into the mainstream market, a fluent customer experience is important, which calls for more JavaScript in the client. It also makes sense to align our web and native apps into using the same API’s, and provide the same features to avoid having to maintain multiple. The biggest issue is that it is not straightforward to just add more modern JavaScript to a template rendered app gradually, and a complete single-page application rebuild would be an immense effort.
While we recently also have started transitioning a few pieces to use Next.js, we still need to be able to provide a modern stack for our existing site since it will probably live on for a while. The approach and tooling we built is a good example of how one could do that — hence this blog post.
Example: Oda’s frontpage feed
Let’s look at an example that showcases this new tooling: our frontpage “feed”. Our “feed” contains personalized product lists, FAQ for new users, and things like an order-tracker and shopping tips. It is composed of server-driven components represented in JSON, which render natively in our apps. Historically this has only been available for our native apps, but now it is finally built for web using the same APIs and components.
Our total feed bundle size? 55.6kb gzipped..!
One of the main concerns about adding non-server-rendered content on our front page was performance, the potential for delayed rendering, and page-flickering. Without server-rendered JavaScript we have to live with the limitation of rendering our HTML on the client, however, we can optimize for quick access of the content to get fast renders without making it so noticeable.
The idea is basically to avoid the initial API call, by preloading the API response and add it as json in the template. Django enables us to do this with it |json and |json_script:"__NAME__" builtins, and then we either add it on the window, or do a simple DOM operation in our JS document.getElementById("__NAME__").
With libraries like react-query one can even provide this json as initial data and still have rigged API queries.
Refresh and hard-refresh.
This tooling was also the basis for the shopping-assistant I demoed earlier, and it is becoming a company success story as we are already up to 10 internal and external apps built using tooling — all of them code-splitted across each other!
Some background
Our goal was to inject pieces of JavaScript into our existing Django site, and there are numerous ways one can accomplish this. One important factor for us was to keep our bundle sizes to a minimum, while still providing the ability to leverage the benefits of JavaScript rendered apps. Injecting JavaScript in templates won’t change the fact that our templates are rendered on the server, and our site is refreshed on every navigation. Thus, adding JavaScript on multiple pages poses a performance problem. Handling bundling naively, it is not unlikely to end up with either one giant bundle, or multiple big bundles all containing a lot of the same packages — ultimately sacrificing performance and shopping experience.
When we started the effort of adding JavaScript, we explored different bundlers. Our choice landed on using Rollup, since it is pretty much the de-facto standard for outputting ES modules — which is also referred to as “native JavaScript modules”. Since ES modules are not supported by all browsers, we also need to provide a legacy browser build — which we support using SystemJS.
So why ES modules and SystemJS?
ES modules give us a lot of benefits, and it is the work of JavaScript standardization over 10 years.
-
They are supported by all modern browsers
-
Statically analyzable making them more viable for code-splitting
-
Smaller footprint through great dead-code elimination ability
-
Can load modules dynamically on-demand, only fetching pieces when you need them
-
Enables easier browser caching
-
Modern JavaScript features
and a lot more…
SystemJS is best described by their docs:
SystemJS is a hookable, standards-based module loader. It provides a workflow where code written for production workflows of native ES modules in browsers (like Rollup code-splitting builds), can be transpiled to the System.register module format to work in older browsers that don’t support native modules, running almost-native module speeds while supporting top-level await, dynamic import, circular references and live bindings, import.meta.url, module types, import maps, integrity and Content Security Policy with compatibility in older browsers back to IE11.
This means we can provide code-splitted builds even for IE11! 🤯
What about bundlers?
The sad thing is that the most popular app bundler around, Webpack, does not support outputting ES modules — it only supports them as inputs like libraries. Meaning the JavaScript web today mostly consists of commonjs bundles, however, it has been requested since 2016, and is now intended that Webpack will start supporting ES module outputs in v5.
As mentioned, it exists bundlers that support ES modules today. The most noteworthy is rollup which has generally only been used for library bundling, mostly due to lacking code-splitting and developer tooling. Now, we do have code-splitting and decent developer tooling, making it a really viable choice for app development as well. A recent and highly regarded alternative is snowpack which provides pretty much all you need (that uses rollup under the hood). However for our Django case, we needed granular control over our manifest and stuff like that, so we opted for rollup, alongside nollup which supports rollup configs, provides hot-module-reloading and development performance optimizations.
Also, the author of nollup wrote a great blog post on why he prefers rollup over webpack, which is a great read.
What does it not give us?
I guess the most prominent and lacking features of a rollup configuration like ours are:
-
Server-side rendering of JavaScript content
-
“No-config” bundler
Despite that, given that we are incrementally adding modern features in a template rendered site — I feel that we strike an interesting balance between pragmatism and modern tech. For our case, going from jQuery sprinkling to adding modern JavaScript capabilities without a full SPA rewrite.
Injection of JS
Similar to a few Django JavaScript loader packages, we use a custom template tag rollup_bundle to inject a given file based on a key to the template.
{% rollup_bundle 'frontend/feed/feed.tsx' %}
The logic in python land is pretty simple.
-
We keep a source of truth in using a
rollup-manifest.jsonfile. -
The manifest is built when bundling, meaning Django does not have to care about anything else.
The generated manifest looks like this:
{
"frontend/feed/feed.tsx": {
"esm": "feed.8e122f39.mjs",
"imports": [
"regexp-sticky-helpers.7e14e245.mjs",
...
],
"dynamicImports": [],
"stylesheet": "feed.3c8ccaf9.css",
"systemjs": "feed.ee3bbd52.js",
"systemjsImports": [
"es.regexp.exec.d166fe35.js",
...
],
"systemjsDynamicImports": []
},
...
}
Using that we can pretty much return
file = manifest[<key>]['esm']
return f"<script src='{file}'>"
in the template tag render method and have JavaScript inserted. 🎉
The manifest is built when bundling, meaning Django does not have to care about anything else.
Okay, in our case a few more things:
Since we build two different versions, one esm and one systemjs, we have to handle those separately. Due to systemjs being a module loader, we do not use the script module/nomodule approach. Instead, we provide two different script loader functions from esmPolyfills and systemJSPolyfills, which exposes injectESModule and injectSystemJSModule on window.
Inspired by Philip Walton's blog post, rollup-starter-code-splitting, and systemjs docs
esmPolyfills:
dynamicImportPolyfill from 'dynamic-import-polyfill';
const injectESModule = (esModulePath: string) => {
dynamicImportPolyfill.initialize({ modulePath: '/static/' });
import(esModulePath);
window.supportsDynamicImport = true;
};
window.injectESModule = injectESModule;
In Django we use a simple html_tag helper and do:
'\n'.join([
html_tag("script", src="esmPolyfills.js"),
html_tag("script", content=f"window.injectESModule('{esm}');"),
])
rendering:
<script src="esmPolyfills.js"></script>
<script>window.injectESModule('file.mjs')</script>
We always try to import the es module first, and the logic falls sets a supportDynamicImport flag that is used to evaluate falling back to systemjs or not. The benefit is that we only load the es module bundle as long as it works, otherwise fetching and loading the systemjs bundle.
Browser support for es modules and dynamic imports is pretty big with ~92% https://caniuse.com/es6-module-dynamic-import, so we feel we are better off helping e.g. mobile devices not fetch more than necessary rather than optimizing for legacy that is often desktop computers.
systemJSPolyfills:
browserSupportsAllFeatures = () =>
window.Promise && window.fetch && window.Symbol;
const systemJSPolyfills = (path: string, err?: Error) => {
if (err) {
return;
}
if (!window.supportsDynamicImport) {
const systemJsLoaderTag = document.createElement('script');
systemJsLoaderTag.src =
'https://unpkg.com/[email protected]/dist/s.min.js';
systemJsLoaderTag.addEventListener('load', () => {
System.import(path);
});
document.head.appendChild(systemJsLoaderTag);
}
};
const loadScript = (src: string, done?: (err?: Error) => void) => {
const js = document.createElement('script');
js.src = src;
js.onload = () => {
if (done) {
done();
}
};
js.onerror = () => {
if (done) {
done(new Error('Failed to load script ' + src));
}
};
document.head.appendChild(js);
};
const injectSystemJSModule = (systemModulePath: string) => {
const callback = (err?: Error) => systemJSPolyfills(systemModulePath, err);
if (browserSupportsAllFeatures()) {
callback();
} else {
loadScript(
'https://cdn.polyfill.io/v2/polyfill.min.js?features=Promise,fetch,Symbol,Array.prototype.@@iterator',
callback
);
}
};
window.injectSystemJSModule = injectSystemJSModule;
Again inspired by Philip Walton, this script allows us polyfilling basics and loading systemjs bundles. It might look daunting, but what it does is:
-
Check and add basic Promise, fetch, and Symbol polyfills.
-
Check the supportDynamicImport flag set in esmPolyfills - skip if true.
-
DOM-operation adding a systemjs loader script tag that loads our provided systemjs module on the load callback.
Likewise esmPolyfills we do in Django:
'\n'.join([
html_tag("script", src="systemJSPolyfills.js"),
html_tag("script", content=f"window.injectSystemJSModule('{systemjs}');"),
])
rendering:
<script src="systemJSPolyfills.js"></script>
<script>window.injectSystemJSModule("file.js")</script>
Preloading JS
A benefit of having control over the injection logic is that we easily could add a second parameter to the template tag, giving us a type that can influence how we inject things.
By adding head_js we can inform the template tag to inject JavaScript in a way that is supported in the HTML head while accessing the manifest, this means we are now capable of leveraging capabilities of the link tag's preload and modulepreload. Having grouped the imports from our bundler into imports and dynamicImports we can then decide if and how we want to load them. Allowing us to use things like React.lazy properly.
Exploiting this, the result is quicker rendering times!
{% rollup_bundle 'frontend/feed/feed.tsx' 'head_js' %}
html_tag(
"link",
self_closing=True,
**{
"rel": "modulepreload",
"as": "script",
"type": "module",
"href": import_path,
},
)
<head>
<link rel="modulepreload" as="script" type="module" href="file.mjs" />
...
</head>
Building the JS
As mentioned we use rollup to bundle our JavaScript, alongside nollup for dev tooling. Let me show you how we bundle!
First, instead of manually updating an input list, we automatically detect what code entries that are subject for bundling. We grep for the {% rollup_bundle ... %} tags in our .html templates. We use this output as input to rollup, which naturally will crash if the entrypoint’s file path does not exist. This allows us to add a new entry point, and add the template tag — then it will automatically get bundled!
const exec = require('child_process').exec;
const getInputMapping = async () => {
/**
* Function that returns object from templatetag literals in tienda required for bundling
* => { "js": ["frontend/feed", ...], "css": ["frontend/feed"] }
*/
const grepCommand = 'grep -hro --include "*.html" "{% rollup_bundle.*%}" tienda';
const bundleRegex = /"(.*?)"\W"(.*?)"/;
return new Promise((resolve, reject) => {
exec(grepCommand, (error, stdout, stderr) => {
if (error !== null) {
reject(stderr);
}
const inputs = stdout
.split('\n')
.filter(Boolean)
.map((str) => str.replace(/'/g, '"'));
let inputMapping = {};
for (const str of inputs) {
const [, appName, type] = str.match(bundleRegex); // Grabs [_, frontend/feed, js] from regex
inputMapping[type] = [
...(inputMapping[type] ? inputMapping[type] : []),
appName,
];
}
resolve(inputMapping);
});
});
};
This could probably be written easier without the grep command fork, but it does the job grepping and outputting an object grouped by the second argument, e.g. “js” as keys with arrays of entries.
Rollup configuration
Configurations for rollup are relatively simple, they require input, output, and optional plugins, however, we do specify a few extra options like source maps and naming structure. Rollup itself supports both outputs as a single object or a list of objects, and in our case where we want to bundle two variants, esm, and systemjs, we return a list with different formats.
rollup.config.js:
// Async method for generating rollup config to support `getInputMapping` async nature.
const generateRollupConfig = async () => {
const { js = [] } = await getInputMapping();
const configs = [
{
input: js,
output: [
{
dir: path.resolve(__dirname, BUNDLE_DIR),
format: 'es',
sourcemap: true,
compact: true,
entryFileNames: '[name].[hash].mjs',
chunkFileNames: '[name].[hash].mjs',
assetFileNames: '[name].[hash][extname]',
dynamicImportFunction: '__import__',
},
],
plugins: basePlugins({ module: true }),
preserveEntrySignatures: false, // Recommended for web apps
},
];
if (isProd) {
const systemJSConfig = {
input: js,
output: [
{
dir: path.resolve(__dirname, BUNDLE_DIR),
format: 'system',
sourcemap: true,
compact: true,
entryFileNames: '[name].[hash].js',
chunkFileNames: '[name].[hash].js',
assetFileNames: '[name].[hash][extname]',
},
],
plugins: basePlugins({ module: false }),
preserveEntrySignatures: false, // Recommended for web apps
};
configs.push(systemJSConfig);
}
return configs;
};
export default generateRollupConfig()
Plugins
You might notice that we do provide plugins using a function to avoid duplication, as a lot of the plugins are pretty essential and it quickly becomes a few lines.
import nodeResolve from '@rollup/plugin-node-resolve';
import commonjs from '@rollup/plugin-commonjs';
import replace from '@rollup/plugin-replace';
import typescript from '@rollup/plugin-typescript';
import babel from '@rollup/plugin-babel';
import { terser } from 'rollup-plugin-terser';
import styles from 'rollup-plugin-styles';
import postcssPresetEnv from 'postcss-preset-env';
const extensions = ['.mjs', '.js', '.jsx', '.ts', '.tsx'];
// Rollup plugins used for legacy iife bundling and modern es-modules
const basePlugins = ({ module = true } = {}) => {
return [
styles({
mode: 'extract',
plugins: [postcssPresetEnv({ browsers: ['ie 11'] })],
minimize: isProd,
}), // Add CSS and SCSS support
typescript(), // Add typescript support
replace({
'process.env.NODE_ENV': JSON.stringify(
process.env.NODE_ENV || 'production'
),
}), // NODE_ENV for react
nodeResolve({ extensions }), // Grabs libraries from node modules
babel({
babelHelpers: 'bundled',
extensions,
exclude: /node_modules/,
presets: [
[
'@babel/preset-env',
{
targets: module
? { esmodules: true },
: { browsers: ['ie 11'] }
useBuiltIns: 'usage', // Add polyfills only by usage
corejs: 3,
// es.promise does not contain Promise.finally which breaks IE11 Promises
// Promise in IE11 is already polyfilled through `systemJSPolyfills`, and this would overwrite it.
exclude: ['es.promise'],
},
],
],
}), // Transpile code depending on es module support
commonjs(), // Bundles commonjs libraries to esm compatible ones for rollup to handle
isProd && manifestPlugin(), // Secret plugin that generates our manifest.json
isProd && terser({ module: module }), // Minifying code
];
};
I won’t dive into details of all these plugins, but their purpose is written in comments inline and are pretty common in the rollup community, except for the custom babel and our manifestPlugin.
Babel is well-known in frontend development, and it is hard to get along without it. For modern things, it should be pretty straightforward, target preset esmodules: true and you should generally be good. But if you care for older browsers like IE11, at least one bug exists today. The Promise.finally prototype does not exist in corejs' es.promise (issue #1 and #2), resulting in red text in the console and blank pages in older browsers when using it 😅 We exclude it, since it is handled in the systemjs polyfill anyways.
Manifest plugin
Remember I spoke of the manifest Django accesses that is provided from build-step? The manifestPlugin is our custom plugin that serves one purpose. Take the output of rollup builds and place it in an object with the entry path as key, structured by output-type. Having a manifest for chunks is a common pattern in other bundlers as well, but in our situation, the flexibility is important as it allows us to do our Django template tag tricks.
// Rollup plugin to write rollup-manifest.json to hold a mapping of app => bundle information. See rollup-manifest.json
const manifest = {};
const manifestPlugin = () => ({
name: 'manifest',
generateBundle(options, bundle) {
for (const [
name, { facadeModuleId, isEntry, imports, dynamicImports, type },
] of Object.entries(bundle)) {
const isCSSAsset = type === 'asset' && name.endsWith('.css');
if (!isEntry && !isCSSAsset) {
continue;
}
let key;
if (facadeModuleId) {
// Get "frontend/feed/index.ts" from absolute path
key = path.relative(path.resolve(__dirname), facadeModuleId);
} else if (isCSSAsset) {
// Find manifest key for css asset
key = Object.keys(manifest).find((manifestKey) =>
manifestKey.includes(name.split('.')[0])
);
if (!key) {
continue;
}
}
const manifestValue = { ...manifest[key] };
if (name.endsWith('.mjs')) {
manifestValue.esm = name;
manifestValue.imports = imports;
manifestValue.dynamicImports = dynamicImports;
} else if (name.endsWith('css')) {
manifestValue.stylesheet = name;
} else if (name.endsWith('.js')) {
manifestValue.systemjs = name;
manifestValue.systemjsImports = imports;
manifestValue.systemjsDynamicImports = dynamicImports;
}
manifest[key] = manifestValue;
}
this.emitFile({
type: 'asset',
fileName: 'rollup-manifest.json',
source: JSON.stringify(manifest, null, 2),
});
},
});
In contrast to how it might look on the surface, when breaking down this code it is quite simple. There are only a few rollup specifics one needs to understand. Rollup exposes a few lifecycle methods for plugins, and we leverage the generateBundle step in that lifecycle.
Our plugin receives all bundles that rollup are generating, and we use the different properties like name, facadeModuleId, and whether it is an entry or .css file to determine whether to add it to our global manifest object. .mjs is esm, .js is system — naive and simple. Then we emit the file using rollup's API and it becomes available alongside the bundled js.
If you like this or have questions — feel free to reach out!