8027.org

In order for text editors to accurately detect Unicode text encodings in files, there exists a multibyte signature called BOM which, located at the very beginning of a file, tells the application which UTF the text is encoded in.

Now, some editors place this signature automatically, and it is quite possible that it is not desirable to have it placed, say, for example, in HTML or PHP files. If the webserver serving those files is not able to detect and remove those bytes, they will get displayed at the beginning of the page, maybe causing errors to happen, for example if they get output before session starting in PHP.

That was precisely the situation at our webservers so I wrote a little perl script to remove those BOMs. You can read the code below if you feel like using it. I tried to comment the code as to make it as clear as possible.

1. #!/usr/bin/perl
2. #
3. # [ Miguel Thu Feb 17 15:23:32 CET 2005 (@colinux) ]
4. #
5. # Finds BOM signatures and strips them from files.
6. #
7. # More info regarding BOM at http://www.unicode.org/faq/utf_bom.html
8. #
9.  
10. use strict;
11. use warnings;
12.  
13. sub remove_bom($);
14.  
15. if (@ARGV < 2) {
16. print "removebom.pl: removes UTF Byte Order Marks from files.\n\n";
17. print " Usage: removebom.pl [path] [pattern]\n\n";
18. print " You must specify both a target directory to be traversed and a file pattern.\n";
19. exit 1;
20. }
21.  
22. my ($dir, $patt) = ($ARGV[0], $ARGV[1]);
23. my @list = `find $dir -iname "*$patt"`;
24. my $enc;
25.  
26. open LOG, ">>", "/tmp/removebom.log"
27. or die "Unable to open log file /tmp/removebom.log\n";
28.  
29. chomp @list;
30.  
31. print LOG "\n\n\n-------------- removebom.pl $dir $patt --------------\n\n";
32.  
33. foreach my $file (@list) {
34. print LOG remove_bom($file);
35. }
36.  
37. close LOG;
38.  
39.  
40. ##############################################################################
41. # Removes BOMs
42. ##############################################################################
43. sub remove_bom($)
44. {
45. my $fname = shift;
46. my $str;
47. my $enc = "No";
48.  
49. open (FH, "<", $fname)
50. or return "Unable to open file $fname for reading.\n";
51.  
52. {
53. $/ = undef; # only in this scope.
54. $str = <FH>; # read the entire input.
55. }
56.  
57. close FH;
58.  
59.  
60.  
61. # Remove UTF-8 Byte Order Mark
62. ($str =~ s/^\xEF\xBB\xBF//) && ($enc = "UTF-8");
63.  
64. # Remove UTF-16 Big Endian Byte Order Mark
65. ($str =~ s/^\xFE\xFF//) && ($enc = "UTF-16BE");
66.  
67. # Remove UTF-16 Little Endian Byte Order Mark
68. ($str =~ s/^\xFF\x/FE/) && ($enc = "UTF-16LE");
69.  
70. # Remove UTF-32 Big Endian Byte Order Mark
71. ($str =~ s/^\x00\x00\xFE\xFF//) && ($enc = "UTF-32BE");
72.  
73. # Remove UTF-32 Little Endian Byte Order Mark
74. ($str =~ s/^\xFF\xFE\x00\x00//) && ($enc = "UTF-32LE");
75.  
76.  
77. #
78. # Reopen and truncate file.
79. #
80.  
81. open (FH, ">", $fname)
82. or return "Unable to open file $fname for writing.\n";
83.  
84. print FH $str;
85.  
86. close FH;
87.  
88. return "$enc BOM removed from $fname\n";
89. }
90. Download this code: removebom.txt

You can read more on Unicode here, and, more specifically, on UTF and BOM here.

This entry was posted on Friday, February 18th, 2005 at 10:11 am and is filed under Programming, Perl. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.