How can I use the Rust (libsyntax) parser myself?

Question

I want to use the Rust parser (libsyntax) to parse a Rust file and extract information from it, such as function names. I started digging in documents and code, so my first goal is a program that prints all the function names of autonomous functions in a .rs file.

A program must expand all macros before printing function names, so functions declared with a macro are not skipped. That's why I can’t write some crappy little parser myself to do this job.

I have to admit that I'm not quite good at Rust yet, so I apologize in advance for any stupid statements in this matter.

As I understand it, I need to take the following steps:

• Parse a file with a Parser struct
• Expand Macros with MacroExpander
• ???
• Use Visitor to go through the AST and retrieve the information I need (e.g. via visit_fn )

So here are my questions:

• How to use MacroExpander ?
• How do I go through an advanced AST with a user visitor?

I had an idea to use a regular lint check instead of a full-fledged parser. I am exploring this option.

If that matters, I use rustc 0.13.0-nightly (f168c12c5 2014-10-25 20:57:10 +0000)

Answer 1

I'm afraid I cannot answer your question directly; but I can provide an alternative that could help.

If you only need AST, you can get it in JSON format with rustc -Z ast-json . Then use your favorite language (Python is excellent) to handle the output.

You can also get the print source with rustc --pretty=(expanded|normal|typed) .

For example, given this hello.rs :

 fn main() { println!("hello world"); } 

We get:

 $ rustc -Z ast-json hello.rs {"module":{"inner":null,"view_items":[{"node":{"va... (etc.) 

 $ rustc --pretty=normal hello.rs #![no_std] #[macro_use] extern crate "std" as std; #[prelude_import] use std::prelude::v1::*; fn main() { println!("hello world"); } 

 $ rustc --pretty=expanded hello.rs #![no_std] #[macro_use] extern crate "std" as std; #[prelude_import] use std::prelude::v1::*; fn main() { ::std::io::stdio::println_args(::std::fmt::Arguments::new({ #[inline] #[allow(dead_code)] static __STATIC_FMTSTR: &'static [&'static str] = &["hello world"]; __STATIC_FMTSTR }, &match () { () => [], })); } 

If you need more, but a plugin with a lens would be a better option. config flags , a modular system and everything else that fits is pretty nontrivial. With the lint plugin, you can immediately get the AST test without fuss. Cargo also supports compiler plugins, so your tool will fit nicely with other people's projects.

Answer 2

You can use syntex to parse Rust, so you don't need to use unstable Rust.

Here is a simple example:

 // Tested against syntex_syntax v0.33 extern crate syntex_syntax as syntax; use std::rc::Rc; use syntax::codemap::{CodeMap}; use syntax::errors::{Handler}; use syntax::errors::emitter::{ColorConfig}; use syntax::parse::{self, ParseSess}; fn main() { let codemap = Rc::new(CodeMap::new()); let tty_handler = Handler::with_tty_emitter(ColorConfig::Auto, None, true, false, codemap.clone()); let parse_session = ParseSess::with_span_handler(tty_handler, codemap.clone()); let src = "fn foo(x: i64) { let y = x + 1; return y; }".to_owned(); let result = parse::parse_crate_from_source_str(String::new(), src, Vec::new(), &parse_session); println!("parse result: {:?}", result); } 

This prints the entire AST:

 parse result: Ok(Crate { module: Mod { inner: Span { lo: BytePos(0), hi: BytePos(43), expn_id: ExpnId(4294967295) }, items: [Item { ident: foo#0, attrs: [], id: 4294967295, node: Fn(FnDecl { inputs: [Arg { ty: type(i64), pat: pat(4294967295: x), id: 4294967295 }], output: Default(Span { lo: BytePos(15), hi: BytePos(15), expn_id: ExpnId(4294967295) }), variadic: false }, Normal, NotConst, Rust, Generics { lifetimes: [], ty_params: [], where_clause: WhereClause { id: 4294967295, predicates: [] } }, Block { stmts: [stmt(4294967295: let y = x + 1;), stmt(4294967295: return y;)], expr: None, id: 4294967295, rules: Default, span: Span { lo: BytePos(15), hi: BytePos(43), expn_id: ExpnId(4294967295) } }), vis: Inherited, span: Span { lo: BytePos(0), hi: BytePos(43), expn_id: ExpnId(4294967295) } }] }, attrs: [], config: [], span: Span { lo: BytePos(0), hi: BytePos(42), expn_id: ExpnId(4294967295) }, exported_macros: [] })