"utf8" \ x96 "does not appear in Unicode in <somefile.pl> in line no - 321" Error in Perl

Question

I am setting up a Perl application. I get this error. "utf8" \ x96 "does not appear in Unicode on the page" Can anyone tell me the reason and solution. Am I missing any configuration or is this my installation problem?

Below is the code:

open(FILE,"<:encoding(UTF-8)",$self->{BASEDIR}.$self->{FILENAME}) || die "could not open file $basedir$filename - $!"; 

Answer 1

The character 0x96 is not a valid UTF-8 encoding. There is a block of code points just above 0x80, which in UTF-8 encodes the start of a two- or three-byte character.

The input you are reading should not be UTF-8 and most likely Latin1 or CP1252.

You will need to convert the input to UTF-8, however this is done in Perl (it has been a long time since I did any Perl and it has not used UTF-8 by default when writing Perl :-)

Answer 2

I suspect that what you consider encoded in UTF-8 is not actually encoded with UTF-8.

Answer 3

Just post this information there if it helps someone in the future.

If you are working with a Microsoft product, it may be caused by non-American characters (European, Chinese, etc.). For example, if someone sends you an Excel data table that you need to process, and it is saved in .csv format, these characters may be out of the utf-8 range if they were not saved properly.

Fortunately, at least in Excel for Mac v.15, you can take this data and "save as" in particular the CSV UTF-8 file - it is in the list of parameters. This is a separate parameter from another CSV file. This converts non-US characters to UTF-8 encoding and solves this problem.