How do I understand what kind of character this is?

Question

Update: Obviously, these are control characters, not Unicode characters.

I am trying to parse an XML file with an odd character in it, which makes it invalid and makes my tools (Firefox, Nokogiri) complain.

This is what the character looks like in Firefox, and what it looks like when I copy and paste it into Textmate (obviously I'm on OS X).

crazy characters http://img.skitch.com/20090811-ghu43k5u9nhpcjmh443dpq76jp.preview.jpg

Instead of just cryptic icons and small gray diamonds, I would really like to know what these symbols are (for example, hexadecimal / decoding codes), but I'm not sure how to figure it out.

Answer 1

I would save the page in Firefox to a file and pass it to hexdump -C . Find the HTML fragment around it in the ASCII part, then find the hexadecimal bytes. This is most likely UTF-8, so expect a multibyte code.

Answer 2

Your screenshot is tiny, but does the Firefox sample contain a glyph with four hexadecimal characters? If so, this is the Unicode character code number. You can also hunt for this diamond glyph on the Unicode code chart or simply copy the diamond into a Google search and the character’s name should appear next to the top.

But the real question is how to handle Unicode input in your program. This must be done correctly if you are processing XML. Nokogiri - Ruby library? I am surprised to hear that it does not automatically process Unicode.

Answer 3

The search query you are looking for is U+2603 or U2603 , explicitly substituting the numbers from your deplorably blurred “unknown glyph”. The first few results will concern this Unicode character.

Answer 4

Copy it to emacs and run hexl-mode.

Answer 5

Just open the file using hexeditor, for example xvi32 .

Answer 6

Open the hex file and extract the hexadecimal representation of the character. Then find the code http://unicode.org to find out the name of the symbol.

Answer 7

hexdump -c from the command line of the terminal you will see the character code.

Answer 8

Save the file and then use the terminal:

od (octal dump)

Answer 9

If you are using Vim, then move the cursor over the character and type ga to display the hex in the status area

Answer 10

you can download the Ruby hexdump extension for the String class and directly print the hexdump from Ruby:

 require 'hexdump' #... whatever you do in your program puts your_string.hexdump 

Result

looks like you get from hexdump -C in shell

Cm:

http://www.unixgods.org/~tilo/Ruby/hexdump.html