Regular expression for tokenize request

Question

This question is a rather theoretical continuation of this question.

I need to tokenize the request (split the request into parts), suppose, as an example:

These are the separate query elements "This is composite term" 

The regular expression point for the specified request is the result with an array of 7 tokens:

 1) These 2) are 3) the 4) separate 5) query 6) elements 7) These are compound composite term 

The answers given in the previous question tend to use a split or for/while to get values, instead of what I wanted, this is one reasonably quick call to /.../g i.e. global regex that will result in a set of values.

What I chose is regex

 (?:")(?:\w+\W*)+(?:")|\w+ 

which can be called like this:

 var regex = /(?:")(?:\w+\W*)+(?:")|\w+/g regex.exec(s) 

But for some reason, the global keyword does not work with it, and therefore it does not lead to an array of all values, but returns one match at a time and requires iteration through a whole string.

How to make it work on this regular expression as global and make it return an array of values, rather than individual components of the array?

Thank you in advance!

Answer 1

Use match () instead of exec () to get all matches in the array:

 s.match(regex) 

Answer 2

You can use exec in a loop like this:

 var re = /"[^"]+"|\w+/g; var str = 'These are the separate query elements "This is composite term"'; var m; var arr = [] while ((m = re.exec(str)) !== null) { if (m.index === re.lastIndex) re.lastIndex++; arr.push(m[0]); } console.log(arr); //=> ["These", "are", "the", "separate", "query", "elements", ""This is composite term""] 

RegEx Demo