You are only looking at the grammar for format_spec, the full grammar is listed above on the same page :
replacement_field ::= "{" [field_name] ["!" conversion] [":" format_spec] "}"
field_name ::= arg_name ("." attribute_name | "[" element_index "]")*
arg_name ::= [identifier | integer]
attribute_name ::= identifier
element_index ::= integer | index_string
index_string ::= <any source character except "]"> +
conversion ::= "r" | "s"
format_spec ::= <described in the next section>
In the syntax, replacement_fieldpay attention to the :foregoing format_spec.
field_name , , '!', a format_spec, ':'
field_name / conversion, : format_spec.
>>> "{0:b}".format(100)
'1100100'
zero field_name, , ; , .