# i18next-scanner [![build status](https://travis-ci.org/i18next/i18next-scanner.svg?branch=master)](https://travis-ci.org/i18next/i18next-scanner) [![Coverage Status](https://coveralls.io/repos/i18next/i18next-scanner/badge.svg?branch=master&service=github)](https://coveralls.io/github/i18next/i18next-scanner?branch=master)

[![NPM](https://nodei.co/npm/i18next-scanner.png?downloads=true&stars=true)](https://www.npmjs.com/package/i18next-scanner)

Scan your code, extract translation keys/values, and merge them into i18n resource files.

Turns your code
```js
i18n._('Loading...');
i18n._('Backslashes in single quote: \' \\ \'');
i18n._('This is \
a multiline \
string');

i18n.t('car', { context: 'blue', count: 1 }); // output: 'One blue car'
i18n.t('car', { context: 'blue', count: 2 }); // output: '2 blue cars'

<Trans i18nKey="some.key">Default text</Trans>
```

into resource files
```js
{
  "Loading...": "Wird geladen...", // uses existing translation
  "Backslashes in single quote: ' \\ '": "__NOT_TRANSLATED__", // returns a custom string
  "This is a multiline string": "this is a multiline string", // returns the key as the default value 
  "car": "car",
  "car_blue": "One blue car",
  "car_blue_plural": "{{count}} blue cars",
  "some": {
    "key": "Default text"
  }
}
```


## Notice
There is a major breaking change since v1.0, and the API interface and options are not compatible with v0.x.

Checkout [Migration Guide](https://github.com/i18next/i18next-scanner/wiki/Migration-Guide) while upgrading from earlier versions.

## Features
* Fully compatible with [i18next](https://github.com/i18next/i18next) - a full-featured i18n javascript library for translating your webapplication.
* Support [react-i18next](https://github.com/i18next/react-i18next) for parsing the <b>Trans</b> component
* Support [Key Based Fallback](https://www.i18next.com/principles/fallback#key-fallback/) to write your code without the need to maintain i18n keys. This feature is available since [i18next@^2.1.0](https://github.com/i18next/i18next/blob/master/CHANGELOG.md#210)
* A standalone parser API
* A transform stream that works with both Gulp and Grunt task runner.
* Support custom transform and flush functions.

## Installation

```sh
npm install --save-dev i18next-scanner
```

or

```sh
npm install -g i18next-scanner
```

## Usage

### CLI Usage

```sh
$ i18next-scanner

  Usage: i18next-scanner [options] <file ...>


  Options:

    -V, --version      output the version number
    --config <config>  Path to the config file (default: i18next-scanner.config.js)
    --output <path>    Path to the output directory (default: .)
    -h, --help         output usage information

  Examples:

    $ i18next-scanner --config i18next-scanner.config.js --output /path/to/output 'src/**/*.{js,jsx}'
    $ i18next-scanner --config i18next-scanner.config.js 'src/**/*.{js,jsx}'
    $ i18next-scanner '/path/to/src/app.js' '/path/to/assets/index.html'
```

Globbing patterns are supported for specifying file paths:
* `*` matches any number of characters, but not `/`
* `?` matches a single character, but not `/`
* `**` matches any number of characters, including `/`, as long as it's the only thing in a path part
* `{}` allows for a comma-separated list of "or" expressions
* `!` at the beginning of a pattern will negate the match

_Note: Globbing patterns should be wrapped in single quotes._

#### Examples

* [examples/i18next-scanner.config.js](https://github.com/i18next/i18next-scanner/blob/master/examples/i18next-scanner.config.js)

```js
const fs = require('fs');
const chalk = require('chalk');

module.exports = {
    input: [
        'app/**/*.{js,jsx}',
        // Use ! to filter out files or directories
        '!app/**/*.spec.{js,jsx}',
        '!app/i18n/**',
        '!**/node_modules/**',
    ],
    output: './',
    options: {
        debug: true,
        func: {
            list: ['i18next.t', 'i18n.t'],
            extensions: ['.js', '.jsx']
        },
        trans: {
            component: 'Trans',
            i18nKey: 'i18nKey',
            defaultsKey: 'defaults',
            extensions: ['.js', '.jsx'],
            fallbackKey: function(ns, value) {
                return value;
            },
            acorn: {
                ecmaVersion: 10, // defaults to 10
                sourceType: 'module', // defaults to 'module'
                // Check out https://github.com/acornjs/acorn/tree/master/acorn#interface for additional options
            }
        },
        lngs: ['en','de'],
        ns: [
            'locale',
            'resource'
        ],
        defaultLng: 'en',
        defaultNs: 'resource',
        defaultValue: '__STRING_NOT_TRANSLATED__',
        resource: {
            loadPath: 'i18n/{{lng}}/{{ns}}.json',
            savePath: 'i18n/{{lng}}/{{ns}}.json',
            jsonIndent: 2,
            lineEnding: '\n'
        },
        nsSeparator: false, // namespace separator
        keySeparator: false, // key separator
        interpolation: {
            prefix: '{{',
            suffix: '}}'
        }
    },
    transform: function customTransform(file, enc, done) {
        "use strict";
        const parser = this.parser;
        const content = fs.readFileSync(file.path, enc);
        let count = 0;

        parser.parseFuncFromString(content, { list: ['i18next._', 'i18next.__'] }, (key, options) => {
            parser.set(key, Object.assign({}, options, {
                nsSeparator: false,
                keySeparator: false
            }));
            ++count;
        });

        if (count > 0) {
            console.log(`i18next-scanner: count=${chalk.cyan(count)}, file=${chalk.yellow(JSON.stringify(file.relative))}`);
        }

        done();
    }
};
```

### Standard API
```js
const fs = require('fs');
const Parser = require('i18next-scanner').Parser;

const customHandler = function(key) {
    parser.set(key, '__TRANSLATION__');
};

const parser = new Parser();
const content = '';

// Parse Translation Function
// i18next.t('key');
content = fs.readFileSync('/path/to/app.js', 'utf-8');
parser
    .parseFuncFromString(content, customHandler) // pass a custom handler
    .parseFuncFromString(content, { list: ['i18next.t']}) // override `func.list`
    .parseFuncFromString(content, { list: ['i18next.t']}, customHandler)
    .parseFuncFromString(content); // use default options and handler

// Parse Trans component
content = fs.readFileSync('/path/to/app.jsx', 'utf-8');
parser
    .parseTransFromString(content, customHandler) // pass a custom handler
    .parseTransFromString(content, { component: 'Trans', i18nKey: 'i18nKey', defaultsKey: 'defaults' })
    .parseTransFromString(content, { fallbackKey: true }) // Uses defaultValue as the fallback key when the i18nKey attribute is missing
    .parseTransFromString(content); // use default options and handler

// Parse HTML Attribute
// <div data-i18n="key"></div>
content = fs.readFileSync('/path/to/index.html', 'utf-8');
parser
    .parseAttrFromString(content, customHandler) // pass a custom handler
    .parseAttrFromString(content, { list: ['data-i18n'] }) // override `attr.list`
    .parseAttrFromString(content, { list: ['data-i18n'] }, customHandler)
    .parseAttrFromString(content); // using default options and handler

console.log(parser.get());
console.log(parser.get({ sort: true }));
console.log(parser.get('translation:key', { lng: 'en'}));
```

### Transform Stream API
The main entry function of [i18next-scanner](https://github.com/i18next/i18next-scanner) is a transform stream. You can use [vinyl-fs](https://github.com/wearefractal/vinyl) to create a readable stream, pipe the stream through [i18next-scanner](https://github.com/i18next/i18next-scanner) to transform your code into an i18n resource object, and write to a destination folder.

Here is a simple example showing how that works:
```js
const scanner = require('i18next-scanner');
const vfs = require('vinyl-fs');
const sort = require('gulp-sort');
const options = {
    // See options at https://github.com/i18next/i18next-scanner#options
};
vfs.src(['/path/to/src'])
    .pipe(sort()) // Sort files in stream by path
    .pipe(scanner(options))
    .pipe(vfs.dest('/path/to/dest'));
```

Alternatively, you can get a transform stream by calling createStream() as show below:
```js
vfs.src(['/path/to/src'])
    .pipe(sort()) // Sort files in stream by path
    .pipe(scanner.createStream(options))
    .pipe(vfs.dest('/path/to/dest'));
```

### Gulp
Now you are ready to set up a minimal configuration, and get started with Gulp. For example:
```js
const gulp = require('gulp');
const sort = require('gulp-sort');
const scanner = require('i18next-scanner');

gulp.task('i18next', function() {
    return gulp.src(['src/**/*.{js,html}'])
        .pipe(sort()) // Sort files in stream by path
        .pipe(scanner({
            lngs: ['en', 'de'], // supported languages
            resource: {
                // the source path is relative to current working directory
                loadPath: 'assets/i18n/{{lng}}/{{ns}}.json',
                
                // the destination path is relative to your `gulp.dest()` path
                savePath: 'i18n/{{lng}}/{{ns}}.json'
            }
        }))
        .pipe(gulp.dest('assets'));
});
```

### Grunt
Once you've finished the installation, add this line to your project's Gruntfile:
```js
grunt.loadNpmTasks('i18next-scanner');
```

In your project's Gruntfile, add a section named `i18next` to the data object passed into `grunt.initConfig()`, like so:
```js
grunt.initConfig({
    i18next: {
        dev: {
            src: 'src/**/*.{js,html}',
            dest: 'assets',
            options: {
                lngs: ['en', 'de'],
                resource: {
                    loadPath: 'assets/i18n/{{lng}}/{{ns}}.json',
                    savePath: 'i18n/{{lng}}/{{ns}}.json'
                }
            }
        }
    }
});
```

## API

There are two ways to use i18next-scanner:

### Standard API
```js
const Parser = require('i18next-scanner').Parser;
const parser = new Parser(options);

const code = "i18next.t('key'); ...";
parser.parseFuncFromString(code); 

const jsx = '<Trans i18nKey="some.key">Default text</Trans>';
parser.parseTransFromString(jsx);

const html = '<div data-i18n="key"></div>';
parser.parseAttrFromString(html);

parser.get();
```

#### parser.parseFuncFromString
Parse translation key from JS function
```js
parser.parseFuncFromString(content)

parser.parseFuncFromString(content, { list: ['_t'] });

parser.parseFuncFromString(content, function(key, options) {
    options.defaultValue = key; // use key as the value
    parser.set(key, options);
});

parser.parseFuncFromString(content, { list: ['_t'] }, function(key, options) {
    parser.set(key, options); // use defaultValue
});
```

#### parser.parseTransFromString
Parse translation key from the [Trans component](https://github.com/i18next/react-i18next)
```js
parser.parseTransFromString(content);

parser.parseTransFromString(context, { component: 'Trans', i18nKey: 'i18nKey' });

// Uses defaultValue as the fallback key when the i18nKey attribute is missing
parser.parseTransFromString(content, { fallbackKey: true });

parser.parseTransFromString(content, {
    fallbackKey: function(ns, value) {
        // Returns a hash value as the fallback key
        return sha1(value);
    }
});

parser.parseTransFromString(content, function(key, options) {
    options.defaultValue = key; // use key as the value
    parser.set(key, options);
});
```

#### parser.parseAttrFromString
Parse translation key from HTML attribute
```js
parser.parseAttrFromString(content)

parser.parseAttrFromString(content, { list: ['data-i18n'] });

parser.parseAttrFromString(content, function(key) {
    const defaultValue = key; // use key as the value
    parser.set(key, defaultValue);
});

parser.parseAttrFromString(content, { list: ['data-i18n'] }, function(key) {
    parser.set(key); // use defaultValue
});
```

#### parser.get
Get the value of a translation key or the whole i18n resource store
```js
// Returns the whole i18n resource store
parser.get();

// Returns the resource store with the top-level keys sorted by alphabetical order 
parser.get({ sort: true });

// Returns a value in fallback language (@see options.fallbackLng) with namespace and key
parser.get('ns:key');

// Returns a value with namespace, key, and lng
parser.get('ns:key', { lng: 'en' });
```

#### parser.set
Set a translation key with an optional defaultValue to i18n resource store

```js
// Set a translation key
parser.set(key);

// Set a translation key with default value
parser.set(key, defaultValue);

// Set a translation key with default value using options
parser.set(key, {
    defaultValue: defaultValue
});
```

### Transform Stream API
```js
const scanner = require('i18next-scanner');
scanner.createStream(options, customTransform /* optional */, customFlush /* optional */);
```

#### customTransform
The optional `customTransform` function is provided as the 2nd argument for the transform stream API. It must have the following signature: `function (file, encoding, done) {}`. A minimal implementation should call the `done()` function to indicate that the transformation is done, even if that transformation means discarding the file.
For example:
```js
const scanner = require('i18next-scanner');
const vfs = require('vinyl-fs');
const customTransform = function _transform(file, enc, done) {
    const parser = this.parser;
    const content = fs.readFileSync(file.path, enc);

    // add your code
    done();
};

vfs.src(['/path/to/src'])
    .pipe(scanner(options, customTransform))
    .pipe(vfs.dest('path/to/dest'));
```

To parse a translation key, call `parser.set(key, defaultValue)` to assign the key with an optional `defaultValue`.
For example:
```js
const customTransform = function _transform(file, enc, done) {
    const parser = this.parser;
    const content = fs.readFileSync(file.path, enc);
    
    parser.parseFuncFromString(content, { list: ['i18n.t'] }, function(key) {
        const defaultValue = '__L10N__';
        parser.set(key, defaultValue);
    });
    
    done();
};
```

Alternatively, you may call `parser.set(defaultKey, value)` to assign the value with a default key. The `defaultKey` should be unique string and can never be `null`, `undefined`, or empty.
For example:
```js
const hash = require('sha1');
const customTransform = function _transform(file, enc, done) {
    const parser = this.parser;
    const content = fs.readFileSync(file.path, enc);
    
    parser.parseFuncFromString(content, { list: ['i18n._'] }, function(key) {
        const value = key;
        const defaultKey = hash(value);
        parser.set(defaultKey, value);
    });
    
    done();
};
```

#### customFlush
The optional `customFlush` function is provided as the last argument for the transform stream API, it is called just prior to the stream ending. You can implement your `customFlush` function to override the default `flush` function. When everything's done, call the `done()` function to indicate the stream is finished.
For example:
```js
const scanner = require('i18next-scanner');
const vfs = require('vinyl-fs');
const customFlush = function _flush(done) {
    const parser = this.parser;
    const resStore = parser.getResourceStore();

    // loop over the resStore
    Object.keys(resStore).forEach(function(lng) {
        const namespaces = resStore[lng];
        Object.keys(namespaces).forEach(function(ns) {
            const obj = namespaces[ns];
            // add your code
        });
    });
    
    done();
};

vfs.src(['/path/to/src'])
    .pipe(scanner(options, customTransform, customFlush))
    .pipe(vfs.dest('/path/to/dest'));
```


## Default Options

Below are the configuration options with their default values:

```javascript
{
    debug: false,
    removeUnusedKeys: false,
    sort: false,
    attr: {
        list: ['data-i18n'],
        extensions: ['.html', '.htm']
    },
    func: {
        list: ['i18next.t', 'i18n.t'],
        extensions: ['.js', '.jsx']
    },
    trans: {
        component: 'Trans',
        i18nKey: 'i18nKey',
        defaultsKey: 'defaults',
        extensions: ['.js', '.jsx'],
        fallbackKey: false
    },
    lngs: ['en'],
    ns: ['translation'],
    defaultLng: 'en',
    defaultNs: 'translation',
    defaultValue: '',
    resource: {
        loadPath: 'i18n/{{lng}}/{{ns}}.json',
        savePath: 'i18n/{{lng}}/{{ns}}.json',
        jsonIndent: 2,
        lineEnding: '\n'
    },
    nsSeparator: ':',
    keySeparator: '.',
    pluralSeparator: '_',
    contextSeparator: '_',
    contextDefaultValues: [],
    interpolation: {
        prefix: '{{',
        suffix: '}}'
    }
}
```

#### debug

Type: `Boolean` Default: `false`

Set to `true` to turn on debug output.

#### removeUnusedKeys

Type: `Boolean` Default: `false`

Set to `true` to remove unused translation keys from i18n resource files.

#### sort

Type: `Boolean` Default: `false`

Set to `true` if you want to sort translation keys in ascending order.

#### attr

Type: `Object` or `false`

If an `Object` is supplied, you can either specify a list of attributes and extensions, or override the default.
```js
{ // Default
    attr: {
        list: ['data-i18n'],
        extensions: ['.html', '.htm']
    }
}
```

You can set attr to `false` to disable parsing attribute as below:
```js
{
    attr: false
}
```

#### func

Type: `Object` or `false`

If an `Object` is supplied, you can either specify a list of translation functions and extensions, or override the default.
```js
{ // Default
    func: {
        list: ['i18next.t', 'i18n.t'],
        extensions: ['.js', '.jsx']
    }
}
```

You can set func to `false` to disable parsing translation function as below:
```js
{
    func: false
}
```

#### trans

Type: `Object` or `false`

If an `Object` is supplied, you can specify a list of extensions, or override the default.
```js
{ // Default
    trans: {
        component: 'Trans',
        i18nKey: 'i18nKey',
        defaultsKey: 'defaults',
        extensions: ['.js', '.jsx'],
        fallbackKey: false
    }
}
```

You can set trans to `false` to disable parsing Trans component as below:
```js
{
    trans: false
}
```

The fallbackKey can either be a boolean value, or a function like so:
```js
fallbackKey: function(ns, value) {
    // Returns a hash value as the fallback key
    return sha1(value);
}
```

#### lngs

Type: `Array` Default: `['en']`

An array of supported languages.

#### ns

Type: `String` or `Array` Default: `['translation']`

A namespace string or an array of namespaces.

#### defaultLng

Type: `String` Default: `'en'`

The default language used for checking default values.

#### defaultNs

Type: `String` Default: `'translation'`

The default namespace used if not passed to translation function.

#### defaultValue

Type: `String` or `Function` Default: `''`

The default value used if not passed to `parser.set`.

##### Examples
Provides the default value with a string:
```js
{
    defaultValue: '__NOT_TRANSLATED__'
}
```

Provides the default value as a callback function:
```js
{
    // @param {string} lng The language currently used.
    // @param {string} ns The namespace currently used.
    // @param {string} key The translation key.
    // @return {string} Returns a default value for the translation key.
    defaultValue: function(lng, ns, key) {
        if (lng === 'en') {
            // Return key as the default value for English language
            return key;
        }
        // Return the string '__NOT_TRANSLATED__' for other languages
        return '__NOT_TRANSLATED__';
    }
}
```

#### resource

Type: `Object`

Resource options:
```js
{ // Default
    resource: {
        // The path where resources get loaded from. Relative to current working directory.
        loadPath: 'i18n/{{lng}}/{{ns}}.json',

        // The path to store resources. Relative to the path specified by `gulp.dest(path)`.
        savePath: 'i18n/{{lng}}/{{ns}}.json',

        // Specify the number of space characters to use as white space to insert into the output JSON string for readability purpose.
        jsonIndent: 2,

        // Normalize line endings to '\r\n', '\r', '\n', or 'auto' for the current operating system. Defaults to '\n'.
        // Aliases: 'CRLF', 'CR', 'LF', 'crlf', 'cr', 'lf'
        lineEnding: '\n'
    }
}
```

#### keySeparator

Type: `String` or `false` Default: `'.'`

Key separator used in translation keys.

Set to `false` to disable key separator if you prefer having keys as the fallback for translation (e.g. gettext). This feature is supported by [i18next@2.1.0](https://github.com/i18next/i18next/blob/master/CHANGELOG.md#210). Also see <strong>Key based fallback</strong> at https://www.i18next.com/principles/fallback#key-fallback.

#### nsSeparator

Type: `String` or `false` Default: `':'`

Namespace separator used in translation keys.

Set to `false` to disable namespace separator if you prefer having keys as the fallback for translation (e.g. gettext). This feature is supported by [i18next@2.1.0](https://github.com/i18next/i18next/blob/master/CHANGELOG.md#210). Also see <strong>Key based fallback</strong> at https://www.i18next.com/principles/fallback#key-fallback.

#### context

Type: `Boolean` or `Function` Default: `true`

Whether to add context form key.

```js
context: function(lng, ns, key, options) {
    return true;
}
```

#### contextFallback

Type: `Boolean` Default: `true`

Whether to add a fallback key as well as the context form key.

#### contextSeparator

Type: `String` Default: `'_'`

The character to split context from key.

#### contextDefaultValues

Type: `Array` Default: `[]`

A list of default context values, used when the scanner encounters dynamic value as a `context`.
For a list of `['male', 'female']` the scanner will generate an entry for each value.

#### plural

Type: `Boolean` or `Function` Default: `true`

Whether to add plural form key.

```js
plural: function(lng, ns, key, options) {
    return true;
}
```

#### pluralFallback

Type: `Boolean` Default: `true`

Whether to add a fallback key as well as the plural form key.

#### pluralSeparator

Type: `String` Default: `'_'`

The character to split plural from key.

#### interpolation

Type: `Object`

interpolation options
```js
{ // Default
    interpolation: {
        // The prefix for variables
        prefix: '{{',

        // The suffix for variables
        suffix: '}}'
    }
}
```

## Integration Guide
Checkout [Integration Guide](https://github.com/i18next/i18next-scanner/wiki/Integration-Guide) to learn how to integrate with [React](https://github.com/i18next/i18next-scanner/wiki/Integration-Guide#react), [Gettext Style I18n](https://github.com/i18next/i18next-scanner/wiki/Integration-Guide#gettext-style-i18n), and [Handlebars](https://github.com/i18next/i18next-scanner/wiki/Integration-Guide#handlebars).

## License

MIT