When does Mathematica create a new character?

Question

Good afternoon,

Earlier, I thought that Mathematica creates new characters in the current $Context at the stage of converting the input string (which is assigned to InString ) to enter an expression (which is assigned to In ). But one simple example violated this explanation:

 In[1]:= ?f During evaluation of In[1]:= Information::notfound: Symbol f not found. >> In[2]:= Names["`*"] Out[2]= {} In[3]:= DownValues[In]//First InString[1] Names["`*"] Out[3]= HoldPattern[In[1]]:>Information[f,LongForm->False] Out[4]= \(? f\) Out[5]= {} 

You can see that there is no f character in $ContextPath , although it is already used inside the definition for In[1] .

This example shows that in Mathematica it is in principle possible to create definitions with characters that do not exist in $ContextPath without creating them. This can be an interesting alternative to preventing the creation of a symbol using Symbol :

 In[9]:= ff := Symbol["f"] Names["`*"] Out[10]= {"ff"} 

Can someone explain under what conditions and at what stage in the evaluation process Mathematica creates new characters?

EDIT

As Sasha noted in a comment on this question, in fact, I was faked with the default ShowStringCharacters->False settings for output cells in the default stylesheet Core.nb and skipped FullForm for output for DownValues[In]//First . In fact, the character f not used in the definition for In[1] , as we see using InputForm :

 In[1]:= ?f DownValues[In]//First//InputForm During evaluation of In[1]:= Information::notfound: Symbol f not found. >> Out[2]//InputForm= HoldPattern[In[1]] :> Information["f", LongForm -> False] 

Sorry for the hasty statement.

So, now the question is about the stage at which Mathematica decides to create a new Symbol and how can we prevent this? For example, in the above example, we introduce f as Symbol , but Mathematica converts it to String without creating a new symbol. This is the built-in behavior of MakeExpression :

 In[1]:= ?f InputForm[MakeExpression[ ToExpression@InString [1], StandardForm]] During evaluation of In[1]:= Information::notfound: Symbol f not found. >> Out[2]//InputForm= HoldComplete[Information["f", LongForm -> False]] 

Perhaps it is possible to define some type of syntactic construct that will prevent the creation of a character until it is evaluated.

About the assessment stage when creating a new character

We see that the increment of $Line occurs before calling MakeExpression , but after creating MakeExpression :

there is a new Symbol creating and assigning a new value for the variables InString and In

 In[1]:= MakeExpression[My`boxes_,My`f_]/;!TrueQ[My`$InsideMakeExpression]:=Block[{My`$InsideMakeExpression=True},Print[$Line];Print[DownValues[InString][[All,1]]];Print[DownValues[In][[All,1]]];Print[Names["`*"]];MakeExpression[My`boxes,My`f]]; In[2]:= a During evaluation of In[2]:= 2 During evaluation of In[2]:= {HoldPattern[InString[1]]} During evaluation of In[2]:= {HoldPattern[In[1]]} During evaluation of In[2]:= {} Out[2]= a 

The same can be said of $PreRead and $NewSymbol call time:

 In[1]:= $NewSymbol:=Print["Names[\"`*\"]=",Names["`*"],"\nDownValues[InString]=",DownValues[InString][[All,1]],"\nDownValues[In]=",DownValues[In][[All,1]],"\nName: ",#1,"\tContext: ",#2]& In[2]:= a During evaluation of In[2]:= Names["`*"]={} DownValues[InString]={HoldPattern[InString[1]]} DownValues[In]={HoldPattern[In[1]]} Name: a Context: Global` Out[2]= a 

$Pre is executed after creating a new assignment in In and after creating an entire new Symbol in the current $Context :

 In[1]:= $Pre := (Print[Names["`*"]]; Print[DownValues[In][[All, 1]]]; ##) & In[2]:= a During evaluation of In[2]:= {a} During evaluation of In[2]:= {HoldPattern[In[1]],HoldPattern[In[2]]} Out[2]= a 

It seems impossible to intercept the assignment of a new value for the variable In .

---

Conclusion: new Symbol are created after calling $PreRead , MakeExpression and $NewSymbol , but before calling $Pre .

Answer 1

Regarding your question regarding editing: not sure if this is what you had in mind, but in $PreRead sessions you can use $PreRead to save characters as strings during the parsing step. Here is one of the possible hackers:

 symbolQ = StringMatchQ[#, RegularExpression["[a-zA-Z$][a-zA-Z$`0-9]*"]] &; ClearAll[keepSymbolsAsStrings]; SetAttributes[keepSymbolsAsStrings, HoldAllComplete]; $PreRead = # //. RowBox[{"keepSymbolsAsStrings", rest___}] :> RowBox[{"keepSymbolsAsStrings", Sequence @@ ({rest} //. x_String?symbolQ :> With[{context = Quiet[Context[x]]}, StringJoin["\"", x, "\""] /; Head[context] === Context])}] &; 

A symbol will be converted to a string only if it does not already exist (which is checked through Context[symbol_string_name] ). for instance

 In[4]:= keepSymbolsAsStrings[a+b*Sin[c]]//FullForm Out[4]//FullForm= keepSymbolsAsStrings[Plus["a",Times["b",Sin["c"]]]] 

It is important that keepSymbolsAsStrings defined keepSymbolsAsStrings so that this character is created. This makes it repeat:

 In[6]:= keepSymbolsAsStrings[a+b*Sin[c]*keepSymbolsAsStrings[d+e*Sin[f]]]//FullForm Out[6]//FullForm= keepSymbolsAsStrings[Plus["a",Times["b",Sin["c"], keepSymbolsAsStrings[Plus["d",Times["e",Sin["f"]]]]]]] 

Now you can process these characters (stored as strings) after your code has been parsed the way you like. You can also use another symbolQ function - for example, I just use simple-mindedness.

However, this will not work for packages. I do not see a direct way to do this for packages. One simplified approach would be to dynamically redefine Needs to change the source at the row level in the same way as a kind of preprocessing step, and effectively invoke Needs on the modified source. But the original row-level modifications are usually fragile.

NTN

Edit

The above code has the disadvantage that it is difficult to distinguish which lines should have been lines and which characters were converted by the above function. You can change the code above by changing ClearAll[keepSymbolsAsStrings] to ClearAll[keepSymbolsAsStrings, symbol] and StringJoin["\"", x, "\""] to RowBox[{"symbol", "[", StringJoin["\"", x, "\""], "]"}] to keep track of which lines in the resulting expression match the converted characters.

Edit 2

Here is a modified code based on MakeExpression , not $PreRead , as suggested by @Alexey:

 symbolQ = StringMatchQ[#, RegularExpression["[a-zA-Z$][a-zA-Z$0-9`]*"]] &; ClearAll[keepSymbolsAsStrings, symbol]; SetAttributes[keepSymbolsAsStrings, HoldAllComplete]; Module[{tried}, MakeExpression[RowBox[{"keepSymbolsAsStrings", rest___}], form_] := Block[{tried = True}, MakeExpression[ RowBox[{"keepSymbolsAsStrings", Sequence @@ ({rest} //. x_String?symbolQ :> With[{context = Quiet[Context[x]]}, RowBox[{"symbol", "[", StringJoin["\"", x, "\""], "]"}] /; Head[context] === Context])}], form] ] /;!TrueQ[tried] ] 

We need Todd Giley's trick to break out of infinite recursion in MakeExpression definitions. Here are some examples:

 In[7]:= keepSymbolsAsStrings[a+b*Sin[c]]//FullForm Out[7]//FullForm= keepSymbolsAsStrings[Plus[symbol["a"],Times[symbol["b"],Sin[symbol["c"]]]]] In[8]:= keepSymbolsAsStrings[a+b*Sin[c]*keepSymbolsAsStrings[d+e*Sin[f]]]//FullForm Out[8]//FullForm= keepSymbolsAsStrings[Plus[symbol["a"],Times[symbol["b"],Sin[symbol["c"]], keepSymbolsAsStrings[Plus[symbol["d"],Times[symbol["e"],Sin[symbol["f"]]]]]]]] 

This method is $PreRead since $PreRead is still available to the end user.

Answer 2

You can use $NewSymbol and $NewMessage have a better understanding of character creation. But from a virtual book, a symbol is created in $Context when it cannot be found in either $Context or $ContextPath .

Answer 3

I think that your basic understanding of what characters are created when input is parsed in an expression is correct.

The subtle part is what ? at the beginning of the line (and << and >> ) are parsed specifically to allow lines without the use of quotation marks. (The implicit lines here are patterns like *Min* for ? And file names for << and >> .)